[AArch64] Support customizing stack protector guard

[nickdesaulniers]

Summary

Follow up to D88631 but for aarch64; the Linux kernel uses the command
line flags:

1. -mstack-protector-guard=sysreg
2. -mstack-protector-guard-reg=sp_el0
3. -mstack-protector-guard-offset=0

to use the system register sp_el0 for the stack canary, enabling the
kernel to have a unique stack canary per task (like a thread, but not
limited to userspace as the kernel can preempt itself).

Address pr/47341 for aarch64.

Fixes: https://github.com/ClangBuiltLinux/linux/issues/289
Signed-off-by: Nick Desaulniers <[e-mail]>

[nickdesaulniers]

I still have some todos that I will get to tomorrow, but posting this early for review.

[nickdesaulniers]

clang/lib/Driver/ToolChains/Clang.cpp

3150

TODO: can we re-use AArch64SysReg::lookupSysRegByName in the frontend?

[xiangzhangllvm]

I didn't find any problem in the main context of the patch, +1 first.

[nickdesaulniers]

Pretty sure I need to do something with non-zero values of -mstack-protector-guard-offset=0.

[kristof.beyls]

clang/test/Driver/stack-protector-guard.c

26

I guess you meant to type "-target arm-linux-gnueabi"?

[nickdesaulniers]

clang/lib/Driver/ToolChains/Clang.cpp

3111

TODO: tls is not supported on aarch64 in GCC.

3126

TODO: GCC treats this as mutually exclusive when OPT_mstack_protector_guard_EQ == global.

[nickdesaulniers]

• fix 32b ARM triple, add test for non-zero offset, actually load ptr

[nickdesaulniers]

llvm/test/CodeGen/AArch64/stack-guard-sysreg-nonzero-offset.ll

1 ↗

(On Diff #340192)

This can be combined with the other test by using different CHECK prefixes.

[nickdesaulniers]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1914–1915

@efriedma can you please check that I have these flags correct? I'm not sure that I do. Also, I suspect that I can't allocated a virtual register for the result of the LDR, since this is post-RA?

[nickdesaulniers]

• combine tests, clean up Clang drivers slightly

[nickdesaulniers]

clang/lib/Driver/ToolChains/Clang.cpp

3126

And it does so inconsistently; it errors for aarch64, but does literally nothing (no warning, no change to codegen) for x86. I don't plan to be bug compatible here.

3150

I don't think so because llvm/lib/Target/AArch64/Utils/AArch64BaseInfo.h is under lib/ not include/. Not sure if I should just remove reg validation?

[nickdesaulniers]

clang/lib/Driver/ToolChains/Clang.cpp

3150

Guidance provided by @echristo and @jyknight was that we should avoid such linkage requirements on Target/, so instead I'll work on adding a helper to clang/lib/Driver/ToolChains/Arch/AArch64.cpp that duplicates logic from AArch64SysReg::lookupSysRegByName.

[nickdesaulniers]

clang/lib/Driver/ToolChains/Clang.cpp

3150

It looks like there's ~1000 possible sysregs for aarch64 ATM; do we really want to add all of those to clang?

[nickdesaulniers]

• add support for NPOT and negative offsets

[nickdesaulniers]

clang/lib/Driver/ToolChains/Clang.cpp

3150

I'm going to post that as a separate commit/review on top of this series, that way it doesn't pollute this code review. This is ready to be reviewed.

[DavidSpickett]

clang/lib/Driver/ToolChains/Clang.cpp

3117

Shouldn't this also allow "tls"? At least that's what the previous code works out to, I don't know if that actually works on AArch64 or if it just didn't error.

3150

If the number of different registers people actually use with this option is somewhere < 10 I'd just hardcode the names here as needed. (a large amount of those sysregs won't be suitable for this purpose anyway)

[DavidSpickett]

clang/test/Driver/stack-protector-guard.c

59

I'm not sure if this is due to your code or the error machinery itself but these errors are strangely written.

I'd expect:

error: invalid value 'tls' in 'mstack-protector-guard='tls', valid arguments to '-mstack-protector-guard=' are:sysreg global'

Maybe it's assuming that there could be multiple values and ',' means that list option treats the value as a list? Or it's not using the right value and the comma is meant to be after the ='' as in my example.

[nickdesaulniers]

• prefer err_drv_invalid_value

clang/lib/Driver/ToolChains/Clang.cpp

3117

I don't think so; GCC seems to support tls for x86 but not for aarch64.
https://godbolt.org/z/6WjEPfhT5

3150

Right, I figure that can be a separate decision from the rest of the implementation, I've forked that off in https://reviews.llvm.org/D101327. Whether it lands or not doesn't matter to me.

clang/test/Driver/stack-protector-guard.c

59

Sure, err_drv_invalid_value would be better.

[nickdesaulniers]

clang/test/Driver/stack-protector-guard.c

59

Ah, err_drv_invalid_value_with_suggestion puts single quotes around the third parameter %2. Let me send a patch cleaning that up, then I'll rebase this on that.

[nickdesaulniers]

• rebase on D101387

[DavidSpickett]

clang/include/clang/Basic/CodeGenOptions.h

385

This is now used for TLS on x86 and sysreg on AArch64 so how about:

/// The TLS base register when StackProtectorGuard is "tls", or register
/// used to store the stack canary for "sysreg".
/// On x86 this can be "fs" or "gs".
/// On AArch64 this can only be "sp_el0".

clang/lib/Driver/ToolChains/Clang.cpp

3117

Fair enough, seems unlikely it would work in any case.

llvm/test/CodeGen/AArch64/stack-guard-sysreg.ll

21

What does NPOT stand for here, something to do with the offset not being a multiple of 8 bytes?

[nickdesaulniers]

• rebase, update comment

[nickdesaulniers]

llvm/test/CodeGen/AArch64/stack-guard-sysreg.ll

21

Non Power Of Two

[nickdesaulniers]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1914–1915

@efriedma bumping for review; otherwise can you suggest another reviewer?

[nickdesaulniers]

• prefer addDef, explicit register kills

[nickdesaulniers]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1914–1915

In particular, I'm most curious about the use of RegState::Renamable; the result of reading via mrs is a temporary; but this is post-RA, so isn't it too late for further register renaming?

[nickdesaulniers]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1914–1915

Additionally, if it's a temporary, should it be RegState::Implicit?

[DavidSpickett]

All my comments have been answered. Perhaps @ostannard can check AArch64InstrInfo.cpp ?

[nickdesaulniers]

bumping for review

[DavidSpickett]

This is probably not showing up in review queues as it got accepted accidentally at some point. I'm going to "Request Changes" then I think you'll need to refresh the diff to get it back into active review.
(I don't know of any other way to "give up" an accept revision)

[dmgreen]

I don't know a huge amount about stack protectors.

It's worth adding -verify-machineinstrs to the tests, to check the code passes the internal checks.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1917

What is StackProtectorGuardOffset, and could is be out of range for the load? That is something that verify-machineinstrs won't currently check for.

1920

Defs come before Uses in machine instructions.

[nickdesaulniers]

• reorder USE/DEF, add -verify-machineinstrs

[nickdesaulniers]

In D100919#2750125, @dmgreen wrote:

It's worth adding -verify-machineinstrs to the tests, to check the code passes the internal checks.

Done.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1917

It's passed from the front end; it's an offset added to the pointer load. The Linux kernel uses a value in 1100-1200 range. GCC will let you use whatever large value and keep appending adds for constants greater than 32200: https://godbolt.org/z/1xhbEPbKG, but YAGNI.

[dmgreen]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1917

OK. It might be worth at least issuing a report_fatal_error if it is out of the range of the instructions handled here. Otherwise the error might be non-obvious, it just truncating the immediate during codegen.

[nickdesaulniers]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1917

Indeed, in particular it looks like the range of LDR is 0-32760 which is sufficient for the kernel, but the range for LDUR is only -256-255, which is not.

[nickdesaulniers]

• support up to +/- 4096 non-multiples of 8. We could get crazy with psuedo instructions and register scavenging, but this is really overkill; the very first case of positive multiple's of 8 < 32670 will be 99% of the kernel's use case.

[dmgreen]

Thanks. This sounds good to me, if David agrees.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1909

-> Unknown

1951

It may be possible to do something earlier, where a we add the MOVi32imm earlier before registry allocation and use that as the register for the ldr offset. That would keep it simpler than trying to find scratch registers, and may be optimized better by the pipeline.

[DavidSpickett]

LGTM with one nit.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1916

For all the if/else here I'd include the braces. I know that once the preprocessor is done it'll be a one line if but visually it's confusing.

[nickdesaulniers]

• braces, typo fix, add comment about additional approaches

[nickdesaulniers]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

1951

Sure; added that as a comment; hopefully we wont need it, but "never say never (again)."

[nickdesaulniers]

Appreciate the reviews; ++beers_owed;

clang/include/clang/Basic/CodeGenOptions.h

public:
 
   /// The guard style used for stack protector to get a initial value, this
   /// value usually be gotten from TLS or get from __stack_chk_guard, or some
   /// other styles we may implement in the future.
   std::string StackProtectorGuard;
 
   /// The TLS base register when StackProtectorGuard is "tls".
   /// On x86 this can be "fs" or "gs".
+  /// On AArch64 this can be any value of llvm::AArch64SysReg::SysReg.
   std::string StackProtectorGuardReg;
 
   /// Path to blocklist file specifying which objects
   /// (files, functions) listed for instrumentation by sanitizer
   /// coverage pass should actually not be instrumented.
   std::vector<std::string> SanitizeCoverageBlocklistFiles;
 
   /// Executable and command-line used to create a given CompilerInvocation.

clang/lib/CodeGen/BackendUtil.cpp

static bool initTargetOptions(DiagnosticsEngine &Diags,
   Options.EnableMachineFunctionSplitter = CodeGenOpts.SplitMachineFunctions;
   Options.FunctionSections = CodeGenOpts.FunctionSections;
   Options.DataSections = CodeGenOpts.DataSections;
   Options.IgnoreXCOFFVisibility = LangOpts.IgnoreXCOFFVisibility;
   Options.UniqueSectionNames = CodeGenOpts.UniqueSectionNames;
   Options.UniqueBasicBlockSectionNames =
       CodeGenOpts.UniqueBasicBlockSectionNames;
   Options.StackProtectorGuard =
-      llvm::StringSwitch<llvm::StackProtectorGuards>(CodeGenOpts
-          .StackProtectorGuard)
+      llvm::StringSwitch<llvm::StackProtectorGuards>(
+          CodeGenOpts.StackProtectorGuard)
           .Case("tls", llvm::StackProtectorGuards::TLS)
           .Case("global", llvm::StackProtectorGuards::Global)
+          .Case("sysreg", llvm::StackProtectorGuards::SysReg)
           .Default(llvm::StackProtectorGuards::None);
   Options.StackProtectorGuardOffset = CodeGenOpts.StackProtectorGuardOffset;
   Options.StackProtectorGuardReg = CodeGenOpts.StackProtectorGuardReg;
   Options.TLSSize = CodeGenOpts.TLSSize;
   Options.EmulatedTLS = CodeGenOpts.EmulatedTLS;
   Options.ExplicitEmulatedTLS = CodeGenOpts.ExplicitEmulatedTLS;
   Options.DebuggerTuning = CodeGenOpts.getDebuggerTuning();
   Options.EmitStackSizeSection = CodeGenOpts.StackSizeSection;

clang/lib/Driver/ToolChains/Clang.cpp

if (Str.startswith("ssp-buffer-size=")) {
         CmdArgs.push_back("-stack-protector-buffer-size");
         // FIXME: Verify the argument is a valid integer.
         CmdArgs.push_back(Args.MakeArgString(Str.drop_front(16)));
       }
       A->claim();
     }
   }
 
-  // First support "tls" and "global" for X86 target.
-  // TODO: Support "sysreg" for AArch64.
   const std::string &TripleStr = EffectiveTriple.getTriple();
   if (Arg *A = Args.getLastArg(options::OPT_mstack_protector_guard_EQ)) {
     StringRef Value = A->getValue();
     if (!EffectiveTriple.isX86() && !EffectiveTriple.isAArch64())
       D.Diag(diag::err_drv_unsupported_opt_for_target)
           << A->getAsString(Args) << TripleStr;
-    if (Value != "tls" && Value != "global") {
+    if (EffectiveTriple.isX86() && Value != "tls" && Value != "global") {
       D.Diag(diag::err_drv_invalid_value_with_suggestion)
           << A->getOption().getName() << Value
           << "valid arguments to '-mstack-protector-guard=' are:tls global";
       return;
     }
+    if (EffectiveTriple.isAArch64() && Value != "sysreg" && Value != "global") {
+      D.Diag(diag::err_drv_invalid_value_with_suggestion)
+          << A->getOption().getName() << Value
+          << "valid arguments to '-mstack-protector-guard=' are:sysreg global";
+      return;
+    }
     A->render(Args, CmdArgs);
   }
 
   if (Arg *A = Args.getLastArg(options::OPT_mstack_protector_guard_offset_EQ)) {
     StringRef Value = A->getValue();
-    if (!EffectiveTriple.isX86())
+    if (!EffectiveTriple.isX86() && !EffectiveTriple.isAArch64())
       D.Diag(diag::err_drv_unsupported_opt_for_target)
           << A->getAsString(Args) << TripleStr;
     int Offset;
     if (Value.getAsInteger(10, Offset)) {
       D.Diag(diag::err_drv_invalid_value) << A->getOption().getName() << Value;
       return;
     }
     A->render(Args, CmdArgs);
   }
 
   if (Arg *A = Args.getLastArg(options::OPT_mstack_protector_guard_reg_EQ)) {
     StringRef Value = A->getValue();
-    if (!EffectiveTriple.isX86())
+    if (!EffectiveTriple.isX86() && !EffectiveTriple.isAArch64())
       D.Diag(diag::err_drv_unsupported_opt_for_target)
           << A->getAsString(Args) << TripleStr;
     if (EffectiveTriple.isX86() && (Value != "fs" && Value != "gs")) {
       D.Diag(diag::err_drv_invalid_value_with_suggestion)
       << A->getOption().getName() << Value
       << "for X86, valid arguments to '-mstack-protector-guard-reg=' are:fs gs";
       return;
     }
+    if (EffectiveTriple.isAArch64() && Value != "sp_el0") {
+      D.Diag(diag::err_drv_invalid_value) << A->getOption().getName() << Value;
+      return;
+    }
     A->render(Args, CmdArgs);
   }
 }
 
 static void RenderSCPOptions(const ToolChain &TC, const ArgList &Args,
                              ArgStringList &CmdArgs) {
   const llvm::Triple &EffectiveTriple = TC.getEffectiveTriple();
 

clang/test/Driver/stack-protector-guard.c

 // RUN: not %clang -target arm-eabi-c -mstack-protector-guard=tls %s 2>&1 | \
 // RUN:   FileCheck -check-prefix=INVALID-ARCH %s
 // INVALID-ARCH: unsupported option '-mstack-protector-guard=tls' for target
 
 // RUN: not %clang -target powerpc64le-linux-gnu -mstack-protector-guard-reg=fs %s 2>&1 | \
 // RUN:   FileCheck -check-prefix=INVALID-ARCH2 %s
 // INVALID-ARCH2: unsupported option '-mstack-protector-guard-reg=fs' for target
 
-// RUN: not %clang -target aarch64-linux-gnu -mstack-protector-guard-offset=10 %s 2>&1 | \
+// RUN: not %clang -target arm-linux-gnueabi -mstack-protector-guard-offset=10 %s 2>&1 | \
 // RUN:   FileCheck -check-prefix=INVALID-ARCH3 %s
 // INVALID-ARCH3: unsupported option '-mstack-protector-guard-offset=10' for target
 
 // Invalid option value
 // RUN: not %clang -target x86_64-unknown-unknown -c -mstack-protector-guard-reg=cs %s 2>&1 | \
 // RUN:   FileCheck -check-prefix=INVALID-REG %s
 // RUN: not %clang -target x86_64-unknown-unknown -c -mstack-protector-guard-reg=ds %s 2>&1 | \
 // RUN:   FileCheck -check-prefix=INVALID-REG %s
 
 // CHECK-FS: "-cc1" {{.*}}"-mstack-protector-guard-reg=fs"
 // CHECK-GS: "-cc1" {{.*}}"-mstack-protector-guard-reg=gs"
 // INVALID-REG: error: invalid value {{.*}} in 'mstack-protector-guard-reg=','for X86, valid arguments to '-mstack-protector-guard-reg=' are:fs gs'
 
 // RUN: %clang -### -target x86_64-unknown-unknown -mstack-protector-guard-offset=30 %s 2>&1 | \
 // RUN:   FileCheck -check-prefix=CHECK-OFFSET %s
 
 // CHECK-OFFSET: "-cc1" {{.*}}"-mstack-protector-guard-offset=30"
+
+// RUN: %clang -### -target aarch64-linux-gnu -mstack-protector-guard=sysreg \
+// RUN:   -mstack-protector-guard-reg=sp_el0 \
+// RUN:   -mstack-protector-guard-offset=0 %s 2>&1 | \
+// RUN: FileCheck -check-prefix=CHECK-AARCH64 %s
+// RUN: %clang -### -target aarch64-linux-gnu \
+// RUN:   -mstack-protector-guard=tls %s 2>&1 | \
+// RUN:   FileCheck -check-prefix=INVALID-VALUE-AARCH64 %s
+// RUN: %clang -### -target aarch64-linux-gnu -mstack-protector-guard=sysreg \
+// RUN:   -mstack-protector-guard-reg=foo \
+// RUN:   -mstack-protector-guard-offset=0 %s 2>&1 | \
+// RUN: FileCheck -check-prefix=INVALID-REG-AARCH64 %s
+
+// CHECK-AARCH64: "-cc1" {{.*}}"-mstack-protector-guard=sysreg" "-mstack-protector-guard-offset=0" "-mstack-protector-guard-reg=sp_el0"
+// INVALID-VALUE-AARCH64: error: invalid value 'tls' in 'mstack-protector-guard=','valid arguments to '-mstack-protector-guard=' are:sysreg global'
+// INVALID-REG-AARCH64: error: invalid value 'foo' in 'mstack-protector-guard-reg='

llvm/include/llvm/Target/TargetOptions.h

Labels, // Do not use Basic Block Sections but label basic blocks.  This
             // is useful when associating profile counts from virtual addresses
             // to basic blocks.
     Preset, // Similar to list but the blocks are identified by passes which
             // seek to use Basic Block Sections, e.g. MachineFunctionSplitter.
             // This option cannot be set via the command line.
     None    // Do not use Basic Block Sections.
   };
 
-  enum class StackProtectorGuards {
-    None,
-    TLS,
-    Global
-  };
+  enum class StackProtectorGuards { None, TLS, Global, SysReg };
 
   enum class EABI {
     Unknown,
     Default, // Default means not specified
     EABI4,   // Target-specific (either 4, 5 or gnu depending on triple).
     EABI5,
     GNU
   };
public:
     unsigned ForceDwarfFrameSection : 1;
 
     /// Emit XRay Function Index section
     unsigned XRayOmitFunctionIndex : 1;
 
     /// Stack protector guard offset to use.
     int StackProtectorGuardOffset = INT_MAX;
 
-    /// Stack protector guard mode to use, e.g. tls, global.
+    /// Stack protector guard mode to use, e.g. tls, global, sysreg.
     StackProtectorGuards StackProtectorGuard =
                                          StackProtectorGuards::None;
 
     /// Stack protector guard reg to use, e.g. usually fs or gs in X86.
     std::string StackProtectorGuardReg = "None";
 
     /// FloatABIType - This setting is set by -float-abi=xxx option is specfied
     /// on the command line. This setting may either be Default, Soft, or Hard.

llvm/lib/CodeGen/CommandFlags.cpp

 }
 
 llvm::StackProtectorGuards
 codegen::getStackProtectorGuardMode(llvm::TargetOptions &Options) {
   if (getStackProtectorGuard() == "tls")
     return StackProtectorGuards::TLS;
   if (getStackProtectorGuard() == "global")
     return StackProtectorGuards::Global;
+  if (getStackProtectorGuard() == "sysreg")
+    return StackProtectorGuards::SysReg;
   if (getStackProtectorGuard() != "none") {
     ErrorOr<std::unique_ptr<MemoryBuffer>> MBOrErr =
         MemoryBuffer::getFile(getStackProtectorGuard());
     if (!MBOrErr)
       errs() << "error illegal stack protector guard mode: "
              << MBOrErr.getError().message() << "\n";
     else
       Options.BBSectionsFuncListBuf = std::move(*MBOrErr);

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

BuildMI(MBB, FirstEpilogSEH, DL, TII->get(AArch64::ADDXri))
         .addReg(AArch64::X0, RegState::Define)
         .addReg(AArch64::X0)
         .addMBB(TargetMBB)
         .addImm(0);
     return true;
   }
 
   Register Reg = MI.getOperand(0).getReg();
+  if (MI.getOpcode() == AArch64::LOAD_STACK_GUARD) {
+    TargetOptions Options = MI.getParent()->getParent()->getTarget().Options;
+    if (Options.StackProtectorGuard == StackProtectorGuards::SysReg) {
+      const AArch64SysReg::SysReg *SrcReg =
+          AArch64SysReg::lookupSysRegByName(Options.StackProtectorGuardReg);
+      assert(SrcReg && "Unable to encode SysReg");
+      BuildMI(MBB, MI, DL, get(AArch64::MRS))
+          .addReg(Reg, RegState::Define)
+          .addImm(SrcReg->Encoding);
+      if (Options.StackProtectorGuardOffset % 8 == 0)
+        BuildMI(MBB, MI, DL, get(AArch64::LDRXui))
+            .addReg(Reg, getKillRegState(false))
+            .addReg(Reg, RegState::Define)
+            .addImm(Options.StackProtectorGuardOffset >> 3);
+      else
+        BuildMI(MBB, MI, DL, get(AArch64::LDURXi))
+            .addReg(Reg, getKillRegState(false))
+            .addReg(Reg, RegState::Define)
+            .addImm(Options.StackProtectorGuardOffset);
+      MBB.erase(MI);
+      return true;
+    }
+  }
+
   const GlobalValue *GV =
       cast<GlobalValue>((*MI.memoperands_begin())->getValue());
   const TargetMachine &TM = MBB.getParent()->getTarget();
   unsigned OpFlags = Subtarget.ClassifyGlobalReference(GV, TM);
   const unsigned char MO_NC = AArch64II::MO_NC;
 
   if ((OpFlags & AArch64II::MO_GOT) != 0) {
     BuildMI(MBB, MI, DL, get(AArch64::LOADgot), Reg)
         .addGlobalAddress(GV, 0, OpFlags);
     if (Subtarget.isTargetILP32()) {
       unsigned Reg32 = TRI->getSubReg(Reg, AArch64::sub_32);
       BuildMI(MBB, MI, DL, get(AArch64::LDRWui))
           .addDef(Reg32, RegState::Dead)
           .addUse(Reg, RegState::Kill)
           .addImm(0)
           .addMemOperand(*MI.memoperands_begin())
           .addDef(Reg, RegState::Implicit);
     } else {
       BuildMI(MBB, MI, DL, get(AArch64::LDRXui), Reg)
           .addReg(Reg, RegState::Kill)
           .addImm(0)
           .addMemOperand(*MI.memoperands_begin());
     }
   } else if (TM.getCodeModel() == CodeModel::Large) {
     assert(!Subtarget.isTargetILP32() && "how can large exist in ILP32?");
     BuildMI(MBB, MI, DL, get(AArch64::MOVZXi), Reg)
         .addGlobalAddress(GV, 0, AArch64II::MO_G0 | MO_NC)
         .addImm(0);
     BuildMI(MBB, MI, DL, get(AArch64::MOVKXi), Reg)
         .addReg(Reg, RegState::Kill)
         .addGlobalAddress(GV, 0, AArch64II::MO_G1 | MO_NC)
         .addImm(16);

llvm/test/CodeGen/AArch64/stack-guard-sysreg.ll

+; RUN: llc %s --stack-protector-guard=sysreg \
+; RUN:   --stack-protector-guard-reg=sp_el0 \
+; RUN:   --stack-protector-guard-offset=0 -o - | \
+; RUN: FileCheck --check-prefix=CHECK --check-prefix=CHECK-NO-OFFSET %s
+; RUN: llc %s --stack-protector-guard=sysreg \
+; RUN:   --stack-protector-guard-reg=sp_el0 \
+; RUN:   --stack-protector-guard-offset=8 -o - | \
+; RUN: FileCheck --check-prefix=CHECK --check-prefix=CHECK-POSITIVE-OFFSET %s
+; RUN: llc %s --stack-protector-guard=sysreg \
+; RUN:   --stack-protector-guard-reg=sp_el0 \
+; RUN:   --stack-protector-guard-offset=-8 -o - | \
+; RUN: FileCheck --check-prefix=CHECK --check-prefix=CHECK-NEGATIVE-OFFSET %s
+; RUN: llc %s --stack-protector-guard=sysreg \
+; RUN:   --stack-protector-guard-reg=sp_el0 \
+; RUN:   --stack-protector-guard-offset=1 -o - | \
+; RUN: FileCheck --check-prefix=CHECK --check-prefix=CHECK-NPOT-OFFSET %s
+; RUN: llc %s --stack-protector-guard=sysreg \
+; RUN:   --stack-protector-guard-reg=sp_el0 \
+; RUN:   --stack-protector-guard-offset=-1 -o - | \
+; RUN: FileCheck --check-prefix=CHECK --check-prefix=CHECK-NPOT-NEG-OFFSET %s
+
+target triple = "aarch64-unknown-linux-gnu"
+
+; Verify that we `mrs` from `SP_EL0` twice, rather than load from
+; __stack_chk_guard.
+define dso_local void @foo(i64 %t) local_unnamed_addr #0 {
+; CHECK-LABEL:   foo:
+; CHECK:         // %bb.0: // %entry
+; CHECK-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    mov x29, sp
+; CHECK-NEXT:    sub sp, sp, #16 // =16
+; CHECK-NEXT:    .cfi_def_cfa w29, 16
+; CHECK-NEXT:    .cfi_offset w30, -8
+; CHECK-NEXT:    .cfi_offset w29, -16
+; CHECK-NEXT:    mrs x8, SP_EL0
+; CHECK-NO-OFFSET:       ldr x8, [x8]
+; CHECK-POSITIVE-OFFSET: ldr x8, [x8, #8]
+; CHECK-NEGATIVE-OFFSET: ldr x8, [x8, #-8]
+; CHECK-NPOT-OFFSET:     ldur x8, [x8, #1]
+; CHECK-NPOT-NEG-OFFSET: ldur x8, [x8, #-1]
+; CHECK-NEXT:    lsl x9, x0, #2
+; CHECK-NEXT:    add x9, x9, #15 // =15
+; CHECK-NEXT:    and x9, x9, #0xfffffffffffffff0
+; CHECK-NEXT:    stur x8, [x29, #-8]
+; CHECK-NEXT:    mov x8, sp
+; CHECK-NEXT:    sub x0, x8, x9
+; CHECK-NEXT:    mov sp, x0
+; CHECK-NEXT:    bl baz
+; CHECK-NEXT:    ldur x8, [x29, #-8]
+; CHECK-NEXT:    mrs x9, SP_EL0
+; CHECK-NO-OFFSET:       ldr x9, [x9]
+; CHECK-POSITIVE-OFFSET: ldr x9, [x9, #8]
+; CHECK-NEGATIVE-OFFSET: ldr x9, [x9, #-8]
+; CHECK-NPOT-OFFSET:     ldur x9, [x9, #1]
+; CHECK-NPOT-NEG-OFFSET: ldur x9, [x9, #-1]
+; CHECK-NEXT:    cmp x9, x8
+; CHECK-NEXT:    b.ne .LBB0_2
+; CHECK-NEXT:  // %bb.1: // %entry
+; CHECK-NEXT:    mov sp, x29
+; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB0_2: // %entry
+; CHECK-NEXT:    bl __stack_chk_fail
+; CHECK-NOT: __stack_chk_guard
+entry:
+  %vla = alloca i32, i64 %t, align 4
+  call void @baz(i32* nonnull %vla)
+  ret void
+}
+
+declare dso_local void @baz(i32*) local_unnamed_addr
+
+attributes #0 = { sspstrong }