hwasan: add -fsanitize=kernel-hwaddress

[andreyknvl]

Summary

Add -fsanitize=kernel-hwaddress flag, that enables -hwasan-kernel=1 -hwasan-recover=1 -hwasan-match-all-tag=0xff.

[andreyknvl]

Note: goes on top of https://reviews.llvm.org/D44827 and https://reviews.llvm.org/D44981

[andreyknvl]

Rebased onto https://reviews.llvm.org/D44699

[eugenis]

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

733

I'd prefer a CHECK(!CompileKernel || Recover).

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

345

How about matchAllTag = ClMatchAllTag.getNumOccurrences() > 0 ? ClMatchAllTag : (CompileKernel ? 0xFF : -1) ?

In fact, the same logic could be applied to CompileKernel and Recover in the constructor.

llvm/test/Instrumentation/HWAddressSanitizer/X86/kernel.ll

21

what happened here? Recover vs trap difference?

[andreyknvl]

Addressed @eugenis comments.

[andreyknvl]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

345

getNumOccurrences is what I was looking for!

Done for ClMatchAllTag and ClRecover.

For ClRecover the behavior now is: when ClRecover is provided, use it's value; otherwise use True if CompileKernel and the value passed from CGOpts if not CompileKernel.

llvm/test/Instrumentation/HWAddressSanitizer/X86/kernel.ll

21

Had to change that since I forced ClRecover when ClKhwasan. Fixed now.

[eugenis]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

345

Looking at this:

this->Recover = ClRecover.getNumOccurrences() > 0 ? ClRecover : Recover;

Recover does not depend on CompileKernel at all.

Why not do the same getNumOccurrences() thing for CompileKernel?

What happens if the user asks for -fsanitize=kernel-hwaddress -fsanitize-recover=kernel-hwaddress? Looks like recover will be silently ignored. Could this be fixed by declaring it non-recoverable in SanitizerArgs?

[andreyknvl]

Check ClEnableK[hw]asan.getNumOccurrences().

[andreyknvl]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

345

Why not do the same getNumOccurrences() thing for CompileKernel?

Done

What happens if the user asks for -fsanitize=kernel-hwaddress -fsanitize-recover=kernel-hwaddress? Looks like recover will be silently ignored. Could this be fixed by declaring it non-recoverable in SanitizerArgs?

Ah, so there's -fsanitize-recover and that's where those CodeGenOptions come from :)

You probably meant -fsanitize=kernel-hwaddress -fnosanitize-recover=kernel-hwaddress (-fsanitize=kernel-[hw]address always enables recover right now)? Yes, in this case it the latter flag will be ignored. Is there a way to mark -fsanitize=kernel-address non-unrecoverable?

There's a little chance that we might want to change that in the future and make KASAN/KHWASAN support the recover=0 mode.

[eugenis]

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

345

It would need to be done in SanitizerArgs.cpp (see ex. Unrecoverable). It appears that we don't have any sanitizers that are not-not-recoverable yet.

[andreyknvl]

PTAL. Added AlwaysRecoverable to SanitizerArgs.cpp. Right now recovery is disallowed for both KASan and KHWASan, but all the appropriate flags seem to be passed properly to the instrumentation pass, so we can easily allow it in the future in we want.

[andreyknvl]

Also rebased onto the latest tip.

[eugenis]

clang/lib/Driver/SanitizerArgs.cpp

451

Looks great, but this new logic needs a test.

[andreyknvl]

Added a test.

[andreyknvl]

Thanks! Could you commit this as well?

[eugenis]

the tests fail with:
clang: /code/llvm-project/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp:1064: llvm::FunctionPass *llvm::createAddressSanitizerFunctionPass(bool, bool, bool): Assertion `!CompileKernel || Recover' failed.

[andreyknvl]

Hm, I don't see this failure. Just in case I rebased onto the latest tip.

How do you run the tests? I tried both make -j48 check-clang check-llvm and make -j48 check-all. I do see some tests failing, but they are not related to Sanitizers.

[eugenis]

Simply ninja check-all. Maybe the patch got applied incorrectly. Phabricator is really bad with file copies.

[eugenis]

clang/test/CodeGen/address-safety-attr-flavors.cpp

17–1

^^^ this line expands to this:
/code/build-llvm/bin/clang -cc1 -internal-isystem /code/build-llvm/lib/clang/7.0.0/include -nostdsysteminc -triple i386-unknown-linux -fsanitize=kernel-address -disable-O0-optnone -emit-llvm -o - /code/llvm-project/clang/test/CodeGen/address-safety-attr-flavors.cpp

and

clang: /code/llvm-project/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp:1064: llvm::FunctionPass *llvm::createAddressSanitizerFunctionPass(bool, bool, bool): Assertion `!CompileKernel || Recover' failed.
#0 0x00000000020af584 PrintStackTraceSignalHandler(void*) (/code/build-llvm/bin/clang+0x20af584)
#1 0x00000000020af8c6 SignalHandler(int) (/code/build-llvm/bin/clang+0x20af8c6)
#2 0x00007f53abd690c0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x110c0)
#3 0x00007f53aa8fafcf gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x32fcf)
#4 0x00007f53aa8fc3fa abort (/lib/x86_64-linux-gnu/libc.so.6+0x343fa)
#5 0x00007f53aa8f3e37 (/lib/x86_64-linux-gnu/libc.so.6+0x2be37)
#6 0x00007f53aa8f3ee2 (/lib/x86_64-linux-gnu/libc.so.6+0x2bee2)
#7 0x0000000002c5e817 (/code/build-llvm/bin/clang+0x2c5e817)
#8 0x000000000228b0e1 addKernelAddressSanitizerPasses(llvm::PassManagerBuilder const&, llvm::legacy::PassManagerBase&) (/code/build-llvm/bin/clang+0x228b0e1)
#9 0x0000000001c37c69 llvm::PassManagerBuilder::addExtensionsToPM(llvm::PassManagerBuilder::ExtensionPointTy, llvm::legacy::PassManagerBase&) const (/code/build-llvm/bin/clang+0x1c37c69)
#10 0x0000000001c38fb7 llvm::PassManagerBuilder::populateModulePassManager(llvm::legacy::PassManagerBase&) (/code/build-llvm/bin/clang+0x1c38fb7)
#11 0x000000000228093b clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) (/code/build-llvm/bin/clang+0x228093b)
#12 0x0000000002afb83c clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/code/build-llvm/bin/clang+0x2afb83c)
#13 0x0000000003368e24 clang::ParseAST(clang::Sema&, bool, bool) (/code/build-llvm/bin/clang+0x3368e24)
#14 0x0000000002758f38 clang::FrontendAction::Execute() (/code/build-llvm/bin/clang+0x2758f38)
#15 0x0000000002716451 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/code/build-llvm/bin/clang+0x2716451)
#16 0x000000000280d0c1 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/code/build-llvm/bin/clang+0x280d0c1)
#17 0x00000000008ff540 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/code/build-llvm/bin/clang+0x8ff540)

[andreyknvl]

Fixed the assert. PTAL.

The reason tests didn't work for me was Release build mode. Tried Debug mode - apparently 64 GB RAM is not enough to build llvm =/ Ended up using Release build with asserts enabled.

I've also got commit access now, so I'll try to commit this myself.

[eugenis]

Could be too many parallel links, try -DLLVM_PARALLEL_LINK_JOBS=3.

Anyway, Debug build is way too slow. I normally use Release with asserts, too.

[eugenis]

LGTM (+missing *-commits MLs)

clang/include/clang/Basic/Sanitizers.def

 
 
 // AddressSanitizer
 SANITIZER("address", Address)
 
 // Kernel AddressSanitizer (KASan)
 SANITIZER("kernel-address", KernelAddress)
 
+// Hardware-assisted AddressSanitizer
 SANITIZER("hwaddress", HWAddress)
 
+// Kernel Hardware-assisted AddressSanitizer (KHWASan)
+SANITIZER("kernel-hwaddress", KernelHWAddress)
+
 // MemorySanitizer
 SANITIZER("memory", Memory)
 
 // libFuzzer
 SANITIZER("fuzzer", Fuzzer)
 
 // libFuzzer-required instrumentation, no linking.
 SANITIZER("fuzzer-no-link", FuzzerNoLink)

clang/lib/CodeGen/BackendUtil.cpp

PM.add(createAddressSanitizerFunctionPass(/*CompileKernel*/ false, Recover,
                                             UseAfterScope));
   PM.add(createAddressSanitizerModulePass(/*CompileKernel*/ false, Recover,
                                           UseGlobalsGC));
 }
 
 static void addKernelAddressSanitizerPasses(const PassManagerBuilder &Builder,
                                             legacy::PassManagerBase &PM) {
   PM.add(createAddressSanitizerFunctionPass(
-      /*CompileKernel*/ true,
-      /*Recover*/ true, /*UseAfterScope*/ false));
-  PM.add(createAddressSanitizerModulePass(/*CompileKernel*/true,
-                                          /*Recover*/true));
+      /*CompileKernel*/ true, /*Recover*/ true, /*UseAfterScope*/ false));
+  PM.add(createAddressSanitizerModulePass(
+      /*CompileKernel*/ true, /*Recover*/ true));
 }
 
 static void addHWAddressSanitizerPasses(const PassManagerBuilder &Builder,
                                             legacy::PassManagerBase &PM) {
   const PassManagerBuilderWrapper &BuilderWrapper =
       static_cast<const PassManagerBuilderWrapper &>(Builder);
   const CodeGenOptions &CGOpts = BuilderWrapper.getCGOpts();
   bool Recover = CGOpts.SanitizeRecover.has(SanitizerKind::HWAddress);
-  PM.add(createHWAddressSanitizerPass(Recover));
+  PM.add(createHWAddressSanitizerPass(/*CompileKernel*/ false, Recover));
+}
+
+static void addKernelHWAddressSanitizerPasses(const PassManagerBuilder &Builder,
+                                            legacy::PassManagerBase &PM) {
+  PM.add(createHWAddressSanitizerPass(
+      /*CompileKernel*/ true, /*Recover*/ true));
 }
 
 static void addMemorySanitizerPass(const PassManagerBuilder &Builder,
                                    legacy::PassManagerBase &PM) {
   const PassManagerBuilderWrapper &BuilderWrapper =
       static_cast<const PassManagerBuilderWrapper&>(Builder);
   const CodeGenOptions &CGOpts = BuilderWrapper.getCGOpts();
   int TrackOrigins = CGOpts.SanitizeMemoryTrackOrigins;
void EmitAssemblyHelper::CreatePasses(legacy::PassManager &MPM,
 
   if (LangOpts.Sanitize.has(SanitizerKind::HWAddress)) {
     PMBuilder.addExtension(PassManagerBuilder::EP_OptimizerLast,
                            addHWAddressSanitizerPasses);
     PMBuilder.addExtension(PassManagerBuilder::EP_EnabledOnOptLevel0,
                            addHWAddressSanitizerPasses);
   }
 
+  if (LangOpts.Sanitize.has(SanitizerKind::KernelHWAddress)) {
+    PMBuilder.addExtension(PassManagerBuilder::EP_OptimizerLast,
+                           addKernelHWAddressSanitizerPasses);
+    PMBuilder.addExtension(PassManagerBuilder::EP_EnabledOnOptLevel0,
+                           addKernelHWAddressSanitizerPasses);
+  }
+
   if (LangOpts.Sanitize.has(SanitizerKind::Memory)) {
     PMBuilder.addExtension(PassManagerBuilder::EP_OptimizerLast,
                            addMemorySanitizerPass);
     PMBuilder.addExtension(PassManagerBuilder::EP_EnabledOnOptLevel0,
                            addMemorySanitizerPass);
   }
 
   if (LangOpts.Sanitize.has(SanitizerKind::Thread)) {

clang/lib/CodeGen/CGDeclCXX.cpp

llvm::Function *CodeGenModule::CreateGlobalInitOrDestructFunction(
   if (getLangOpts().Sanitize.has(SanitizerKind::KernelAddress) &&
       !isInSanitizerBlacklist(SanitizerKind::KernelAddress, Fn, Loc))
     Fn->addFnAttr(llvm::Attribute::SanitizeAddress);
 
   if (getLangOpts().Sanitize.has(SanitizerKind::HWAddress) &&
       !isInSanitizerBlacklist(SanitizerKind::HWAddress, Fn, Loc))
     Fn->addFnAttr(llvm::Attribute::SanitizeHWAddress);
 
+  if (getLangOpts().Sanitize.has(SanitizerKind::KernelHWAddress) &&
+      !isInSanitizerBlacklist(SanitizerKind::KernelHWAddress, Fn, Loc))
+    Fn->addFnAttr(llvm::Attribute::SanitizeHWAddress);
+
   if (getLangOpts().Sanitize.has(SanitizerKind::Thread) &&
       !isInSanitizerBlacklist(SanitizerKind::Thread, Fn, Loc))
     Fn->addFnAttr(llvm::Attribute::SanitizeThread);
 
   if (getLangOpts().Sanitize.has(SanitizerKind::Memory) &&
       !isInSanitizerBlacklist(SanitizerKind::Memory, Fn, Loc))
     Fn->addFnAttr(llvm::Attribute::SanitizeMemory);
 

clang/lib/CodeGen/CodeGenFunction.cpp

if (D) {
     // Apply the no_sanitize* attributes to SanOpts.
     for (auto Attr : D->specific_attrs<NoSanitizeAttr>()) {
       SanitizerMask mask = Attr->getMask();
       SanOpts.Mask &= ~mask;
       if (mask & SanitizerKind::Address)
         SanOpts.set(SanitizerKind::KernelAddress, false);
       if (mask & SanitizerKind::KernelAddress)
         SanOpts.set(SanitizerKind::Address, false);
+      if (mask & SanitizerKind::HWAddress)
+        SanOpts.set(SanitizerKind::KernelHWAddress, false);
+      if (mask & SanitizerKind::KernelHWAddress)
+        SanOpts.set(SanitizerKind::HWAddress, false);
     }
   }
 
   // Apply sanitizer attributes to the function.
   if (SanOpts.hasOneOf(SanitizerKind::Address | SanitizerKind::KernelAddress))
     Fn->addFnAttr(llvm::Attribute::SanitizeAddress);
-  if (SanOpts.hasOneOf(SanitizerKind::HWAddress))
+  if (SanOpts.hasOneOf(SanitizerKind::HWAddress | SanitizerKind::KernelHWAddress))
     Fn->addFnAttr(llvm::Attribute::SanitizeHWAddress);
   if (SanOpts.has(SanitizerKind::Thread))
     Fn->addFnAttr(llvm::Attribute::SanitizeThread);
   if (SanOpts.has(SanitizerKind::Memory))
     Fn->addFnAttr(llvm::Attribute::SanitizeMemory);
   if (SanOpts.has(SanitizerKind::SafeStack))
     Fn->addFnAttr(llvm::Attribute::SafeStack);
 

clang/lib/CodeGen/CodeGenModule.cpp

bool CodeGenModule::isInSanitizerBlacklist(SanitizerMask Kind,
   return false;
 }
 
 bool CodeGenModule::isInSanitizerBlacklist(llvm::GlobalVariable *GV,
                                            SourceLocation Loc, QualType Ty,
                                            StringRef Category) const {
   // For now globals can be blacklisted only in ASan and KASan.
   const SanitizerMask EnabledAsanMask = LangOpts.Sanitize.Mask &
-      (SanitizerKind::Address | SanitizerKind::KernelAddress | SanitizerKind::HWAddress);
+      (SanitizerKind::Address | SanitizerKind::KernelAddress |
+       SanitizerKind::HWAddress | SanitizerKind::KernelHWAddress);
   if (!EnabledAsanMask)
     return false;
   const auto &SanitizerBL = getContext().getSanitizerBlacklist();
   if (SanitizerBL.isBlacklistedGlobal(EnabledAsanMask, GV->getName(), Category))
     return true;
   if (SanitizerBL.isBlacklistedLocation(EnabledAsanMask, Loc, Category))
     return true;
   // Check global type.

clang/lib/CodeGen/SanitizerMetadata.cpp

 SanitizerMetadata::SanitizerMetadata(CodeGenModule &CGM) : CGM(CGM) {}
 
 void SanitizerMetadata::reportGlobalToASan(llvm::GlobalVariable *GV,
                                            SourceLocation Loc, StringRef Name,
                                            QualType Ty, bool IsDynInit,
                                            bool IsBlacklisted) {
   if (!CGM.getLangOpts().Sanitize.hasOneOf(SanitizerKind::Address |
                                            SanitizerKind::KernelAddress |
-                                           SanitizerKind::HWAddress))
+                                           SanitizerKind::HWAddress |
+                                           SanitizerKind::KernelHWAddress))
     return;
   IsDynInit &= !CGM.isInSanitizerBlacklist(GV, Loc, Ty, "init");
   IsBlacklisted |= CGM.isInSanitizerBlacklist(GV, Loc, Ty);
 
   llvm::Metadata *LocDescr = nullptr;
   llvm::Metadata *GlobalName = nullptr;
   llvm::LLVMContext &VMContext = CGM.getLLVMContext();
   if (!IsBlacklisted) {
llvm::NamedMDNode *AsanGlobals =
       CGM.getModule().getOrInsertNamedMetadata("llvm.asan.globals");
   AsanGlobals->addOperand(ThisGlobal);
 }
 
 void SanitizerMetadata::reportGlobalToASan(llvm::GlobalVariable *GV,
                                            const VarDecl &D, bool IsDynInit) {
   if (!CGM.getLangOpts().Sanitize.hasOneOf(SanitizerKind::Address |
                                            SanitizerKind::KernelAddress |
-                                           SanitizerKind::HWAddress))
+                                           SanitizerKind::HWAddress |
+                                           SanitizerKind::KernelHWAddress))
     return;
   std::string QualName;
   llvm::raw_string_ostream OS(QualName);
   D.printQualifiedName(OS);
 
   bool IsBlacklisted = false;
   for (auto Attr : D.specific_attrs<NoSanitizeAttr>())
     if (Attr->getMask() & SanitizerKind::Address)
       IsBlacklisted = true;
   reportGlobalToASan(GV, D.getLocation(), OS.str(), D.getType(), IsDynInit,
                      IsBlacklisted);
 }
 
 void SanitizerMetadata::disableSanitizerForGlobal(llvm::GlobalVariable *GV) {
   // For now, just make sure the global is not modified by the ASan
   // instrumentation.
   if (CGM.getLangOpts().Sanitize.hasOneOf(SanitizerKind::Address |
                                           SanitizerKind::KernelAddress |
-                                          SanitizerKind::HWAddress))
+                                          SanitizerKind::HWAddress |
+                                          SanitizerKind::KernelHWAddress))
     reportGlobalToASan(GV, SourceLocation(), "", QualType(), false, true);
 }
 
 void SanitizerMetadata::disableSanitizerForInstruction(llvm::Instruction *I) {
   I->setMetadata(CGM.getModule().getMDKindID("nosanitize"),
                  llvm::MDNode::get(CGM.getLLVMContext(), None));
 }
 

clang/lib/Driver/SanitizerArgs.cpp

 
 enum : SanitizerMask {
   NeedsUbsanRt = Undefined | Integer | Nullability | CFI,
   NeedsUbsanCxxRt = Vptr | CFI,
   NotAllowedWithTrap = Vptr,
   NotAllowedWithMinimalRuntime = Vptr,
   RequiresPIE = DataFlow | HWAddress | Scudo,
   NeedsUnwindTables = Address | HWAddress | Thread | Memory | DataFlow,
-  SupportsCoverage = Address | HWAddress | KernelAddress | Memory | Leak |
-                     Undefined | Integer | Nullability | DataFlow | Fuzzer |
-                     FuzzerNoLink,
+  SupportsCoverage = Address | HWAddress | KernelAddress | KernelHWAddress |
+                     Memory | Leak | Undefined | Integer | Nullability |
+                     DataFlow | Fuzzer | FuzzerNoLink,
   RecoverableByDefault = Undefined | Integer | Nullability,
   Unrecoverable = Unreachable | Return,
   LegacyFsanitizeRecoverMask = Undefined | Integer,
   NeedsLTO = CFI,
   TrappingSupported = (Undefined & ~Vptr) | UnsignedIntegerOverflow |
                       Nullability | LocalBounds | CFI,
   TrappingDefault = CFI,
   CFIClasses = CFIVCall | CFINVCall | CFIDerivedCast | CFIUnrelatedCast,
std::pair<SanitizerMask, SanitizerMask> IncompatibleGroups[] = {
       std::make_pair(Leak, Thread | Memory),
       std::make_pair(KernelAddress, Address | Leak | Thread | Memory),
       std::make_pair(HWAddress, Address | Thread | Memory | KernelAddress),
       std::make_pair(Efficiency, Address | HWAddress | Leak | Thread | Memory |
                                      KernelAddress),
       std::make_pair(Scudo, Address | HWAddress | Leak | Thread | Memory |
                                 KernelAddress | Efficiency),
       std::make_pair(SafeStack, Address | HWAddress | Leak | Thread | Memory |
-                                    KernelAddress | Efficiency)};
+                                    KernelAddress | Efficiency),
+      std::make_pair(KernelHWAddress, Address | HWAddress | Leak | Thread |
+                        Memory | KernelAddress | Efficiency | SafeStack)};
 
   // Enable toolchain specific default sanitizers if not explicitly disabled.
   SanitizerMask Default = TC.getDefaultSanitizers() & ~AllRemove;
 
   // Disable default sanitizers that are incompatible with explicitly requested
   // ones.
   for (auto G : IncompatibleGroups) {
     SanitizerMask Group = G.first;
if (Arg->getOption().matches(options::OPT_fsanitize_recover)) {
       RecoverableKinds &= ~expandSanitizerGroups(parseArgValues(D, Arg, true));
       Arg->claim();
     }
     if (DeprecatedReplacement) {
       D.Diag(diag::warn_drv_deprecated_arg) << Arg->getAsString(Args)
                                             << DeprecatedReplacement;
     }
   }
   RecoverableKinds &= Kinds;
   RecoverableKinds &= ~Unrecoverable;
 
   TrappingKinds &= Kinds;
   RecoverableKinds &= ~TrappingKinds;
 
   // Setup blacklist files.
   // Add default blacklist from resource directory.
   addDefaultBlacklists(D, Kinds, BlacklistFiles);

clang/lib/Driver/ToolChains/Linux.cpp

SanitizerMask Linux::getSupportedSanitizers() const {
   if (IsX86_64 || IsMIPS64 || IsPowerPC64 || IsAArch64)
     Res |= SanitizerKind::Memory;
   if (IsX86_64 || IsMIPS64)
     Res |= SanitizerKind::Efficiency;
   if (IsX86 || IsX86_64)
     Res |= SanitizerKind::Function;
   if (IsX86_64 || IsMIPS64 || IsAArch64 || IsX86 || IsMIPS || IsArmArch)
     Res |= SanitizerKind::Scudo;
-  if (IsX86_64 || IsAArch64)
+  if (IsX86_64 || IsAArch64) {
     Res |= SanitizerKind::HWAddress;
+    Res |= SanitizerKind::KernelHWAddress;
+  }
   return Res;
 }
 
 void Linux::addProfileRTLibs(const llvm::opt::ArgList &Args,
                              llvm::opt::ArgStringList &CmdArgs) const {
   if (!needsProfileRT(Args)) return;
 
   // Add linker option -u__llvm_runtime_variable to cause runtime
   // initialization module to be linked in.
   if (!Args.hasArg(options::OPT_coverage))
     CmdArgs.push_back(Args.MakeArgString(
         Twine("-u", llvm::getInstrProfRuntimeHookVarName())));
   ToolChain::addProfileRTLibs(Args, CmdArgs);
 }

clang/lib/Lex/PPMacroExpansion.cpp

static bool HasFeature(const Preprocessor &PP, StringRef Feature) {
   if (Feature.startswith("__") && Feature.endswith("__") && Feature.size() >= 4)
     Feature = Feature.substr(2, Feature.size() - 4);
 
   return llvm::StringSwitch<bool>(Feature)
       .Case("address_sanitizer",
             LangOpts.Sanitize.hasOneOf(SanitizerKind::Address |
                                        SanitizerKind::KernelAddress))
       .Case("hwaddress_sanitizer",
-            LangOpts.Sanitize.hasOneOf(SanitizerKind::HWAddress))
+            LangOpts.Sanitize.hasOneOf(SanitizerKind::HWAddress |
+                                       SanitizerKind::KernelHWAddress))
       .Case("assume_nonnull", true)
       .Case("attribute_analyzer_noreturn", true)
       .Case("attribute_availability", true)
       .Case("attribute_availability_with_message", true)
       .Case("attribute_availability_app_extension", true)
       .Case("attribute_availability_with_version_underscores", true)
       .Case("attribute_availability_tvos", true)
       .Case("attribute_availability_watchos", true)

clang/test/CodeGen/address-safety-attr-flavors.cpp

-// Make sure the sanitize_address attribute is emitted when using ASan, KASan or HWASan.
+// Make sure the sanitize_address attribute is emitted when using ASan, KASan, HWASan or KHWASan.
 // Either __attribute__((no_sanitize("address")) or __attribute__((no_sanitize("kernel-address")) disables both ASan and KASan instrumentation.
+// Same for __attribute__((no_sanitize("hwaddress")) and __attribute__((no_sanitize("kernel-hwddress")) and HWASan and KHWASan.
 
 /// RUN: %clang_cc1 -triple i386-unknown-linux -disable-O0-optnone -emit-llvm -o - %s | FileCheck -check-prefix=CHECK-NOASAN %s
 /// RUN: %clang_cc1 -triple i386-unknown-linux -fsanitize=address -disable-O0-optnone -emit-llvm -o - %s | FileCheck -check-prefix=CHECK-ASAN %s
 /// RUN: %clang_cc1 -triple i386-unknown-linux -fsanitize=kernel-address -disable-O0-optnone -emit-llvm -o - %s | FileCheck -check-prefix=CHECK-KASAN %s
 /// RUN: %clang_cc1 -triple i386-unknown-linux -fsanitize=hwaddress -disable-O0-optnone -emit-llvm -o - %s | FileCheck -check-prefix=CHECK-HWASAN %s
+/// RUN: %clang_cc1 -triple i386-unknown-linux -fsanitize=kernel-hwaddress -disable-O0-optnone -emit-llvm -o - %s | FileCheck -check-prefix=CHECK-KHWASAN %s
 
 int HasSanitizeAddress() {
   return 1;
 }
 // CHECK-NOASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-ASAN: Function Attrs: noinline nounwind sanitize_address
 // CHECK-KASAN: Function Attrs: noinline nounwind sanitize_address
 // CHECK-HWASAN: Function Attrs: noinline nounwind sanitize_hwaddress
+// CHECK-KHWASAN: Function Attrs: noinline nounwind sanitize_hwaddress
 
 __attribute__((no_sanitize("address")))
 int NoSanitizeQuoteAddress() {
   return 0;
 }
 // CHECK-NOASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-ASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-KASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-HWASAN: {{Function Attrs: noinline nounwind sanitize_hwaddress$}}
+// CHECK-KHWASAN: {{Function Attrs: noinline nounwind sanitize_hwaddress$}}
 
 __attribute__((no_sanitize_address))
 int NoSanitizeAddress() {
   return 0;
 }
 // CHECK-NOASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-ASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-KASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-HWASAN: {{Function Attrs: noinline nounwind sanitize_hwaddress$}}
+// CHECK-KHWASAN: {{Function Attrs: noinline nounwind sanitize_hwaddress$}}
 
 __attribute__((no_sanitize("kernel-address")))
 int NoSanitizeKernelAddress() {
   return 0;
 }
 // CHECK-NOASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-ASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-KASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-HWASAN: {{Function Attrs: noinline nounwind sanitize_hwaddress$}}
+// CHECK-KHWASAN: {{Function Attrs: noinline nounwind sanitize_hwaddress$}}
 
 __attribute__((no_sanitize("hwaddress")))
 int NoSanitizeHWAddress() {
   return 0;
 }
 // CHECK-NOASAN: {{Function Attrs: noinline nounwind$}}
 // CHECK-ASAN: {{Function Attrs: noinline nounwind sanitize_address$}}
 // CHECK-KASAN: {{Function Attrs: noinline nounwind sanitize_address$}}
 // CHECK-HWASAN: {{Function Attrs: noinline nounwind$}}
+// CHECK-KHWASAN: {{Function Attrs: noinline nounwind$}}
+
+__attribute__((no_sanitize("kernel-hwaddress")))
+int NoSanitizeKernelHWAddress() {
+  return 0;
+}
+// CHECK-NOASAN: {{Function Attrs: noinline nounwind$}}
+// CHECK-ASAN: {{Function Attrs: noinline nounwind sanitize_address$}}
+// CHECK-KASAN: {{Function Attrs: noinline nounwind sanitize_address$}}
+// CHECK-HWASAN: {{Function Attrs: noinline nounwind$}}
+// CHECK-KHWASAN: {{Function Attrs: noinline nounwind$}}

clang/test/Driver/asan.c

 // RUN: %clang     -target i386-unknown-linux -fsanitize=address %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-ASAN
 // RUN: %clang -O1 -target i386-unknown-linux -fsanitize=address %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-ASAN
 // RUN: %clang -O2 -target i386-unknown-linux -fsanitize=address %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-ASAN
 // RUN: %clang -O3 -target i386-unknown-linux -fsanitize=address %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-ASAN
 // RUN: %clang     -target i386-unknown-linux -fsanitize=kernel-address %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-KASAN
 // RUN: %clang -O1 -target i386-unknown-linux -fsanitize=kernel-address %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-KASAN
 // RUN: %clang -O2 -target i386-unknown-linux -fsanitize=kernel-address %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-KASAN
 // RUN: %clang -O3 -target i386-unknown-linux -fsanitize=kernel-address %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-KASAN
 // RUN: %clang     -target aarch64-unknown-linux -fsanitize=hwaddress %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-HWASAN
 // RUN: %clang -O1 -target aarch64-unknown-linux -fsanitize=hwaddress %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-HWASAN
 // RUN: %clang -O2 -target aarch64-unknown-linux -fsanitize=hwaddress %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-HWASAN
 // RUN: %clang -O3 -target aarch64-unknown-linux -fsanitize=hwaddress %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-HWASAN
-// Verify that -fsanitize={address,kernel-address} invoke ASan and KASan instrumentation.
+// RUN: %clang     -target aarch64-unknown-linux -fsanitize=kernel-hwaddress %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-KHWASAN
+// RUN: %clang -O1 -target aarch64-unknown-linux -fsanitize=kernel-hwaddress %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-KHWASAN
+// RUN: %clang -O2 -target aarch64-unknown-linux -fsanitize=kernel-hwaddress %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-KHWASAN
+// RUN: %clang -O3 -target aarch64-unknown-linux -fsanitize=kernel-hwaddress %s -S -emit-llvm -o - | FileCheck %s --check-prefix=CHECK-KHWASAN
+// Verify that -fsanitize={address,hwaddres,kernel-address,kernel-hwaddress} invokes ASan, HWAsan, KASan or KHWASan instrumentation.
 
 int foo(int *a) { return *a; }
 // CHECK-ASAN: __asan_init
 // CHECK-KASAN: __asan_load4_noabort
 // CHECK-HWASAN: __hwasan_init
+// CHECK-KHWASAN: __hwasan_load4_noabort

clang/test/Driver/fsanitize-coverage.c

 // RUN: %clang -target x86_64-linux-gnu -fsanitize=address -fsanitize-coverage=0 %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-0
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=address -fsanitize-coverage=edge -fsanitize-coverage=0 %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-0
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=address %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-0
 // CHECK-SANITIZE-COVERAGE-0-NOT: fsanitize-coverage-type
 // CHECK-SANITIZE-COVERAGE-0: -fsanitize=address
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=address -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-address -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-hwaddress -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=memory -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=leak -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=undefined -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=bool -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=dataflow -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // RUN: %clang -target x86_64-linux-gnu                     -fsanitize-coverage=func,trace-pc %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANITIZE-COVERAGE-FUNC
 // CHECK-SANITIZE-COVERAGE-FUNC: fsanitize-coverage-type=1
 

clang/test/Driver/fsanitize.c

 // CHECK-SANKA-SANT: '-fsanitize=kernel-address' not allowed with '-fsanitize=thread'
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-address,memory -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKA-SANM
 // CHECK-SANKA-SANM: '-fsanitize=kernel-address' not allowed with '-fsanitize=memory'
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-address,address -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKA-SANA
 // CHECK-SANKA-SANA: '-fsanitize=kernel-address' not allowed with '-fsanitize=address'
 
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-address,leak -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKA-SANL
+// CHECK-SANKA-SANL: '-fsanitize=kernel-address' not allowed with '-fsanitize=leak'
+
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-hwaddress,thread -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKHA-SANT
+// CHECK-SANKHA-SANT: '-fsanitize=kernel-hwaddress' not allowed with '-fsanitize=thread'
+
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-hwaddress,memory -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKHA-SANM
+// CHECK-SANKHA-SANM: '-fsanitize=kernel-hwaddress' not allowed with '-fsanitize=memory'
+
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-hwaddress,address -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKHA-SANA
+// CHECK-SANKHA-SANA: '-fsanitize=kernel-hwaddress' not allowed with '-fsanitize=address'
+
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-hwaddress,leak -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKHA-SANL
+// CHECK-SANKHA-SANL: '-fsanitize=kernel-hwaddress' not allowed with '-fsanitize=leak'
+
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-hwaddress,hwaddress -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKHA-SANHA
+// CHECK-SANKHA-SANHA: '-fsanitize=kernel-hwaddress' not allowed with '-fsanitize=hwaddress'
+
+// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-hwaddress,kernel-address -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKHA-SANKA
+// CHECK-SANKHA-SANKA: '-fsanitize=kernel-hwaddress' not allowed with '-fsanitize=kernel-address'
+
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress,thread -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANHA-SANT
 // CHECK-SANHA-SANT: '-fsanitize=hwaddress' not allowed with '-fsanitize=thread'
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress,memory -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANHA-SANM
 // CHECK-SANHA-SANM: '-fsanitize=hwaddress' not allowed with '-fsanitize=memory'
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=hwaddress,address -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANHA-SANA
 // CHECK-SANHA-SANA: '-fsanitize=hwaddress' not allowed with '-fsanitize=address'
 
-// RUN: %clang -target x86_64-linux-gnu -fsanitize=kernel-address,leak -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANKA-SANL
-// CHECK-SANKA-SANL: '-fsanitize=kernel-address' not allowed with '-fsanitize=leak'
-
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=efficiency-cache-frag,address -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANE-SANA
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=efficiency-working-set,address -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANE-SANA
 // CHECK-SANE-SANA: '-fsanitize=efficiency-{{.*}}' not allowed with '-fsanitize=address'
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=efficiency-cache-frag,leak -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANE-SANL
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=efficiency-working-set,leak -pie -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANE-SANL
 // CHECK-SANE-SANL: '-fsanitize=efficiency-{{.*}}' not allowed with '-fsanitize=leak'
 

clang/test/Lexer/has_feature_address_sanitizer.cpp

 // RUN: %clang_cc1 -E -fsanitize=address %s -o - | FileCheck --check-prefix=CHECK-ASAN %s
 // RUN: %clang_cc1 -E -fsanitize=kernel-address %s -o - | FileCheck --check-prefix=CHECK-ASAN %s
 // RUN: %clang_cc1 -E -fsanitize=hwaddress %s -o - | FileCheck --check-prefix=CHECK-HWASAN %s
+// RUN: %clang_cc1 -E -fsanitize=kernel-hwaddress %s -o - | FileCheck --check-prefix=CHECK-HWASAN %s
 // RUN: %clang_cc1 -E  %s -o - | FileCheck --check-prefix=CHECK-NO-ASAN %s
 
 #if __has_feature(address_sanitizer)
 int AddressSanitizerEnabled();
 #else
 int AddressSanitizerDisabled();
 #endif
 

llvm/include/llvm/Transforms/Instrumentation.h

 ModulePass *createAddressSanitizerModulePass(bool CompileKernel = false,
                                              bool Recover = false,
                                              bool UseGlobalsGC = true);
 
 // Insert MemorySanitizer instrumentation (detection of uninitialized reads)
 FunctionPass *createMemorySanitizerPass(int TrackOrigins = 0,
                                         bool Recover = false);
 
-FunctionPass *createHWAddressSanitizerPass(bool Recover = false);
+FunctionPass *createHWAddressSanitizerPass(bool CompileKernel = false,
+                                           bool Recover = false);
 
 // Insert ThreadSanitizer (race detection) instrumentation
 FunctionPass *createThreadSanitizerPass();
 
 // Insert DataFlowSanitizer (dynamic data flow analysis) instrumentation
 ModulePass *createDataFlowSanitizerPass(
     const std::vector<std::string> &ABIListFiles = std::vector<std::string>(),
     void *(*getArgTLS)() = nullptr, void *(*getRetValTLS)() = nullptr);

llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp

 /// AddressSanitizer: instrument the code in module to find memory bugs.
 struct AddressSanitizer : public FunctionPass {
   // Pass identification, replacement for typeid
   static char ID;
 
   explicit AddressSanitizer(bool CompileKernel = false, bool Recover = false,
                             bool UseAfterScope = false)
       : FunctionPass(ID), CompileKernel(CompileKernel || ClEnableKasan),
-        Recover(Recover || ClRecover),
+  Recover(Recover || ClRecover),
         UseAfterScope(UseAfterScope || ClUseAfterScope) {
+    assert(!CompileKernel || Recover);
     initializeAddressSanitizerPass(*PassRegistry::getPassRegistry());
   }
 
   StringRef getPassName() const override {
     return "AddressSanitizerFunctionPass";
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {

 public:
   // Pass identification, replacement for typeid
   static char ID;
 
   explicit AddressSanitizerModule(bool CompileKernel = false,
                                   bool Recover = false,
                                   bool UseGlobalsGC = true)
       : ModulePass(ID), CompileKernel(CompileKernel || ClEnableKasan),
-        Recover(Recover || ClRecover),
+  Recover(Recover || ClRecover),
         UseGlobalsGC(UseGlobalsGC && ClUseGlobalsGC),
         // Not a typo: ClWithComdat is almost completely pointless without
         // ClUseGlobalsGC (because then it only works on modules without
         // globals, which are rare); it is a prerequisite for ClUseGlobalsGC;
         // and both suffer from gold PR19002 for which UseGlobalsGC constructor
         // argument is designed as workaround. Therefore, disable both
         // ClWithComdat and ClUseGlobalsGC unless the frontend says it's ok to
         // do globals-gc.
-        UseCtorComdat(UseGlobalsGC && ClWithComdat) {}
+        UseCtorComdat(UseGlobalsGC && ClWithComdat) {
+          assert(!CompileKernel || Recover);
+  }
 
   bool runOnModule(Module &M) override;
   StringRef getPassName() const override { return "AddressSanitizerModule"; }
 
 private:
   void initializeCallbacks(Module &M);
 
   bool InstrumentGlobals(IRBuilder<> &IRB, Module &M, bool *CtorComdat);

llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp

 
 /// \brief An instrumentation pass implementing detection of addressability bugs
 /// using tagged pointers.
 class HWAddressSanitizer : public FunctionPass {
 public:
   // Pass identification, replacement for typeid.
   static char ID;
 
-  HWAddressSanitizer(bool Recover = false)
-      : FunctionPass(ID), Recover(Recover || ClRecover) {}
+  explicit HWAddressSanitizer(bool CompileKernel = false, bool Recover = false)
+      : FunctionPass(ID), CompileKernel(CompileKernel || ClEnableKhwasan),
+        Recover(Recover || ClRecover) {
+    assert(!CompileKernel || Recover);
+  }
 
   StringRef getPassName() const override { return "HWAddressSanitizer"; }
 
   bool runOnFunction(Function &F) override;
   bool doInitialization(Module &M) override;
 
   void initializeCallbacks(Module &M);
   void untagPointerOperand(Instruction *I, Value *Addr);

 
 private:
   LLVMContext *C;
   Triple TargetTriple;
 
   Type *IntptrTy;
   Type *Int8Ty;
 
+  bool CompileKernel;
   bool Recover;
 
   Function *HwasanCtorFunction;
 
   Function *HwasanMemoryAccessCallback[2][kNumberOfAccessSizes];
   Function *HwasanMemoryAccessCallbackSized[2];
 
   Function *HwasanTagMemoryFunc;
   Function *HwasanGenerateTagFunc;
 };
 
 } // end anonymous namespace
 
 char HWAddressSanitizer::ID = 0;
 
 INITIALIZE_PASS_BEGIN(
     HWAddressSanitizer, "hwasan",
     "HWAddressSanitizer: detect memory bugs using tagged addressing.", false, false)
 INITIALIZE_PASS_END(
     HWAddressSanitizer, "hwasan",
     "HWAddressSanitizer: detect memory bugs using tagged addressing.", false, false)
 
-FunctionPass *llvm::createHWAddressSanitizerPass(bool Recover) {
-  return new HWAddressSanitizer(Recover);
+FunctionPass *llvm::createHWAddressSanitizerPass(bool CompileKernel,
+                                                 bool Recover) {
+  return new HWAddressSanitizer(CompileKernel, Recover);
 }
 
 /// \brief Module-level initialization.
 ///
 /// inserts a call to __hwasan_init to the module's constructor list.
 bool HWAddressSanitizer::doInitialization(Module &M) {
   DEBUG(dbgs() << "Init " << M.getName() << "\n");
   auto &DL = M.getDataLayout();
 
   TargetTriple = Triple(M.getTargetTriple());
 
   C = &(M.getContext());
   IRBuilder<> IRB(*C);
   IntptrTy = IRB.getIntPtrTy(DL);
   Int8Ty = IRB.getInt8Ty();
 
   HwasanCtorFunction = nullptr;
-  if (!ClEnableKhwasan) {
+  if (!CompileKernel) {
     std::tie(HwasanCtorFunction, std::ignore) =
         createSanitizerCtorAndInitFunctions(M, kHwasanModuleCtorName,
                                             kHwasanInitName,
                                             /*InitArgTypes=*/{},
                                             /*InitArgs=*/{});
     appendToGlobalCtors(M, HwasanCtorFunction, 0);
   }
   return true;
void HWAddressSanitizer::instrumentMemAccessInline(Value *PtrLong, bool IsWrite,
   if (ClMappingOffset)
     ShadowLong = IRB.CreateAdd(
         ShadowLong, ConstantInt::get(PtrLong->getType(), ClMappingOffset,
                                      /*isSigned=*/false));
   Value *MemTag =
       IRB.CreateLoad(IRB.CreateIntToPtr(ShadowLong, IRB.getInt8PtrTy()));
   Value *TagMismatch = IRB.CreateICmpNE(PtrTag, MemTag);
 
-  if (ClMatchAllTag != -1) {
+  int matchAllTag = ClMatchAllTag.getNumOccurrences() > 0 ?
+      ClMatchAllTag : (CompileKernel ? 0xFF : -1);
+  if (matchAllTag != -1) {
     Value *TagNotIgnored = IRB.CreateICmpNE(PtrTag,
-        ConstantInt::get(PtrTag->getType(), ClMatchAllTag));
+        ConstantInt::get(PtrTag->getType(), matchAllTag));
     TagMismatch = IRB.CreateAnd(TagMismatch, TagNotIgnored);
   }
 
   TerminatorInst *CheckTerm =
       SplitBlockAndInsertIfThen(TagMismatch, InsertBefore, !Recover,
                                 MDBuilder(*C).createBranchWeights(1, 100000));
 
   IRB.SetInsertPoint(CheckTerm);
if (ClGenerateTagsWithCalls)
     return getNextTagWithCall(IRB);
   return IRB.CreateXor(StackTag, ConstantInt::get(IntptrTy, 0xFFU));
 }
 
 // Add a tag to an address.
 Value *HWAddressSanitizer::tagPointer(IRBuilder<> &IRB, Type *Ty, Value *PtrLong,
                                       Value *Tag) {
   Value *TaggedPtrLong;
-  if (ClEnableKhwasan) {
+  if (CompileKernel) {
     // Kernel addresses have 0xFF in the most significant byte.
     Value *ShiftedTag = IRB.CreateOr(
         IRB.CreateShl(Tag, kPointerTagShift),
         ConstantInt::get(IntptrTy, (1ULL << kPointerTagShift) - 1));
     TaggedPtrLong = IRB.CreateAnd(PtrLong, ShiftedTag);
   } else {
     // Userspace can simply do OR (tag << 56);
     Value *ShiftedTag = IRB.CreateShl(Tag, kPointerTagShift);
     TaggedPtrLong = IRB.CreateOr(PtrLong, ShiftedTag);
   }
   return IRB.CreateIntToPtr(TaggedPtrLong, Ty);
 }
 
 // Remove tag from an address.
 Value *HWAddressSanitizer::untagPointer(IRBuilder<> &IRB, Value *PtrLong) {
   Value *UntaggedPtrLong;
-  if (ClEnableKhwasan) {
+  if (CompileKernel) {
     // Kernel addresses have 0xFF in the most significant byte.
     UntaggedPtrLong = IRB.CreateOr(PtrLong,
         ConstantInt::get(PtrLong->getType(), 0xFFULL << kPointerTagShift));
   } else {
     // Userspace addresses have 0x00.
     UntaggedPtrLong = IRB.CreateAnd(PtrLong,
         ConstantInt::get(PtrLong->getType(), ~(0xFFULL << kPointerTagShift)));
   }

llvm/test/Instrumentation/HWAddressSanitizer/X86/kernel.ll

-; Test kernel hwasan instrumentation.
+; Test KHWASan instrumentation.
 ; Generic code is covered by ../kernel.ll, only the x86_64 specific code is
 ; tested here.
 ;
 ; RUN: opt < %s -hwasan -hwasan-kernel=1 -S | FileCheck %s --allow-empty --check-prefixes=INIT
 ; RUN: opt < %s -hwasan -hwasan-kernel=1 -S | FileCheck %s
 ; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-mapping-offset=12345678 -S | FileCheck %s
 ; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-recover=0 -S | FileCheck %s --check-prefixes=CHECK,ABORT
 ; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-recover=1 -S | FileCheck %s --check-prefixes=CHECK,RECOVER
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i8 @test_load(i8* %a) sanitize_hwaddress {
 ; CHECK-LABEL: @test_load(
 ; CHECK: %[[A:[^ ]*]] = ptrtoint i8* %a to i64
 
 ; ABORT: call void asm sideeffect "int3\0Anopl 64(%rax)", "{rdi}"(i64 %[[A]])
 ; ABORT: unreachable
 ; RECOVER: call void asm sideeffect "int3\0Anopl 96(%rax)", "{rdi}"(i64 %[[A]])
 ; RECOVER: br label
 
 ; CHECK: %[[A:[^ ]*]] = ptrtoint i8* %a to i64
 ; CHECK: %[[UNTAGGED:[^ ]*]] = or i64 %[[A]], -72057594037927936
 ; CHECK: %[[UNTAGGED_PTR:[^ ]*]] = inttoptr i64 %[[UNTAGGED]] to i8*
 ; CHECK: %[[G:[^ ]*]] = load i8, i8* %[[UNTAGGED_PTR]], align 4
 ; CHECK: ret i8 %[[G]]
 
 entry:
   %b = load i8, i8* %a, align 4
   ret i8 %b
 }
 
 ; INIT-NOT: call void @__hwasan_init

llvm/test/Instrumentation/HWAddressSanitizer/kernel.ll

-; Test kernel hwasan instrumentation.
+; Test KHWASan instrumentation.
 ;
 ; RUN: opt < %s -hwasan -hwasan-kernel=1 -S | FileCheck %s --allow-empty --check-prefixes=INIT
-; RUN: opt < %s -hwasan -hwasan-kernel=1 -S | FileCheck %s  --check-prefixes=CHECK,NOOFFSET,NO-MATCH-ALL
-; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-mapping-offset=12345678 -S | FileCheck %s  --check-prefixes=CHECK,OFFSET,NO-MATCH-ALL
-; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-recover=0 -S | FileCheck %s  --check-prefixes=CHECK,NOOFFSET,ABORT,NO-MATCH-ALL
-; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-recover=1 -S | FileCheck %s  --check-prefixes=CHECK,NOOFFSET,RECOVER,NO-MATCH-ALL
-; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-recover=1 -hwasan-match-all-tag=0xff -S | FileCheck %s  --check-prefixes=CHECK,NOOFFSET,RECOVER,MATCH-ALL
+; RUN: opt < %s -hwasan -hwasan-kernel=1 -S | FileCheck %s  --check-prefixes=CHECK,NOOFFSET,MATCH-ALL
+; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-mapping-offset=12345678 -S | FileCheck %s  --check-prefixes=CHECK,OFFSET,MATCH-ALL
+; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-recover=0 -S | FileCheck %s  --check-prefixes=CHECK,NOOFFSET,ABORT,MATCH-ALL
+; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-recover=1 -S | FileCheck %s  --check-prefixes=CHECK,NOOFFSET,RECOVER,MATCH-ALL
+; RUN: opt < %s -hwasan -hwasan-kernel=1 -hwasan-recover=1 -hwasan-match-all-tag=-1 -S | FileCheck %s  --check-prefixes=CHECK,NOOFFSET,RECOVER,NO-MATCH-ALL
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64--linux-android"
 
 define i8 @test_load(i8* %a) sanitize_hwaddress {
 ; CHECK-LABEL: @test_load(
 ; CHECK: %[[A:[^ ]*]] = ptrtoint i8* %a to i64
 ; CHECK: %[[B:[^ ]*]] = lshr i64 %[[A]], 56