This is an archive of the discontinued LLVM Phabricator instance.

[CodeGen][RISCV] Change Shadow Call Stack Register to X3
ClosedPublic

Authored by paulkirth on Mar 20 2023, 3:08 PM.

Details

Summary

ShadowCallStack implementation uses s2 register on RISC-V, but that
choice is problematic for reasons described in:

https://lists.riscv.org/g/sig-toolchains/message/544,
https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/370, and
https://github.com/google/android-riscv64/issues/72

The concern over the register choice was also brought up in
https://reviews.llvm.org/D84414.

https://reviews.llvm.org/D84414#2228666 said:

"If the register choice is the only concern about this work, then I think
we can probably land it as-is and fixup the register choice if we see
major drawbacks later. Yes, it's an ABI issue, but on the other hand the
shadow call stack is not a standard ABI anyway.""

Since we have now found a sufficient reason to fixup the register
choice, we should go ahead and update the implementation. We propose
using x3(gp) which is now the platform register in the RISC-V ABI.

Diff Detail

Event Timeline

paulkirth created this revision.Mar 20 2023, 3:08 PM
Herald added a project: Restricted Project. · View Herald TranscriptMar 20 2023, 3:08 PM
paulkirth requested review of this revision.Mar 20 2023, 3:08 PM
Herald added projects: Restricted Project, Restricted Project, Restricted Project. · View Herald TranscriptMar 20 2023, 3:08 PM
Herald added subscribers: llvm-commits, Restricted Project, cfe-commits and 3 others. · View Herald Transcript
craig.topper added inline comments.
llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
50–51

Can you add a FIXME here? Using x27 should hopefully remove this restriction

llvm/lib/TargetParser/RISCVTargetParser.cpp
87–88

X18->X27

paulkirth updated this revision to Diff 506762.Mar 20 2023, 4:12 PM

Fix comment and add FIXME

paulkirth marked 2 inline comments as done.Mar 20 2023, 4:13 PM
paulkirth added inline comments.
llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
50–51

I've added the FIXME, but do we just want to remove this now? or is there some issues or chang under review I can ref in the FIXME?

craig.topper added inline comments.Mar 20 2023, 4:20 PM
llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
50–51

We can remove it, but we should add a test for it.

paulkirth marked an inline comment as done.Mar 20 2023, 4:33 PM
paulkirth added inline comments.
llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
50–51

That makes sense. Thanks.

paulkirth updated this revision to Diff 506790.Mar 20 2023, 5:23 PM

Remove outdated diagnostics and add tests for SCS + save/restore.

I'm not 100% on if this is the right approach for save/restore testing, so I'm hapy to back this bit out or add more cases.

hiraditya accepted this revision.Mar 20 2023, 5:45 PM

Thanks for putting the patch.

This revision is now accepted and ready to land.Mar 20 2023, 5:45 PM
craig.topper added inline comments.Mar 20 2023, 5:55 PM
llvm/test/CodeGen/RISCV/shadowcallstack.ll
184

Can we split this to a separate file? It doesn't look like it was generated by update_llc_test_checks.py and we don't want these checks on the test cases above.

paulkirth added inline comments.Mar 20 2023, 9:23 PM
llvm/test/CodeGen/RISCV/shadowcallstack.ll
184

Good point. This was taken directly from what looked to be the only relevant case in saverestore.ll. Tomorrow I will split this off into a standalone file. Do you know if there would be other relevant cases to consider from the save/restore tests?

Split out save/restore tests for SCS into its own file

paulkirth marked an inline comment as done.Mar 21 2023, 10:45 AM
paulkirth marked an inline comment as done.
hiraditya added inline comments.Mar 21 2023, 11:06 AM
llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
62

asb added a comment.Mar 21 2023, 11:27 AM

In principle I think this is good, but I see two considerations:

  1. Sam was right that there's an ability to change the register after the fact, but 2.5 years have passed since then. We should make a good faith attempt to see if any downstream users are relying on the current ABI (e.g. the original contributors - @zzheng and @apazos).
  2. Now that the issue has been raised in the psABI / toolchain-sig, it would be good if there were consensus on the "platform register" rather than changing this multiple times if some different register is decided.

I'd hope we can resolve this quickly enough that we don't need to consider e.g. adding a way to select the shadow call stack register, but if you _need_ this quickly to iterate on Android/Fuchsia then that might be the easiest route to land something very rapidly.

https://lists.riscv.org/g/sig-toolchains/message/548 has a proposal where X27 is one of the registers.

MaskRay added inline comments.Mar 22 2023, 2:49 PM
llvm/lib/Target/RISCV/RISCVFrameLowering.cpp
50–51

Nit: Drop the period while updating the message. See https://llvm.org/docs/CodingStandards.html#error-and-warning-messages

105–106

Nit: Drop the period while updating the message. See https://llvm.org/docs/CodingStandards.html#error-and-warning-messages

llvm/test/CodeGen/RISCV/saverestore-scs.ll
2

;; appears a useful prefix to highlight non-RUN-non-CHECK comments. It also makes it clear Check is not a typo for CHECK: when we have some automation tools to detect stale CHECK lines.

Address comments.

  • Remove . from warning message
  • use ;; prefix in test
paulkirth marked 3 inline comments as done.Mar 23 2023, 10:32 AM

ugh. I accidentally commited some temporary work here.

Remove unrelated changes, due to bad commit.

Restore check lines in test.

asb added a comment.Mar 27 2023, 2:27 AM

I've flagged this proposed change on Discourse (more the fact we're looking to change it, rather than the precise register we end up with).

Sorry for cross post, but I guess some people might not follow closely in discourse (like me :P):

Another proposal from me is using gp as platform register: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/pull/371

Some advantage on taking gp as platform register rather than other GPRs:

  • Compiler doesn’t use gp register anywhere for now.
  • All assembly files (which conform with current ABI) didn’t use that except the __global_pointer$ initialization code in CRT files.
  • The main user is linker, linker will use that to perform linker relaxation, and we already have the command option to tune that off.

Potential issues:

  • Loss the code size and performance gain from gp relaxation
    • The most gain from gp relaxation is embedded application, it’s different target audience as SCS, so this should not blocker issues.
    • Android is an example, it’s already disable GP relaxation at all, so we don’t have any loss for this case.
  • Will it break any existing platform?
    • Treat gp as platform register is optional, it’s still default use as gp relaxation, so NO breakage on existing platform, but give the freedom of the platform to use gp register as other purpose if they don’t want gp relxation.
    • Added an attribute to let linker to help mixing up different gp usage objects, also linker could check that attribute to make sure gp relaxation is do-able or not.
paulkirth updated this revision to Diff 509508.Mar 29 2023, 5:14 PM
paulkirth retitled this revision from [CodeGen][RISCV] Change Shadow Call Stack Register to S11 to [CodeGen][RISCV] Change Shadow Call Stack Register to X3.
paulkirth edited the summary of this revision. (Show Details)

update to use gp since it was chosen in psABI and sig-toolchain.

Does the kernel populate the initial value of the register for the shadow stack or is that done by a runtime inside the application?

Does the kernel populate the initial value of the register for the shadow stack or is that done by a runtime inside the application?

This is a change to code generation and is orthogonal to how the runtime environment supports the ABI requirement. The details depend on the particular runtime environment, which is not the province of the compiler backend.

Does the kernel populate the initial value of the register for the shadow stack or is that done by a runtime inside the application?

This is a change to code generation and is orthogonal to how the runtime environment supports the ABI requirement. The details depend on the particular runtime environment, which is not the province of the compiler backend.

I ask because GP is set in crtbegin.o on Linux today. So if its the kernel that populates it, then GP isn't valid choice for Linux without changing crtbegin.

Does the kernel populate the initial value of the register for the shadow stack or is that done by a runtime inside the application?

This is a change to code generation and is orthogonal to how the runtime environment supports the ABI requirement. The details depend on the particular runtime environment, which is not the province of the compiler backend.

I ask because GP is set in crtbegin.o on Linux today. So if its the kernel that populates it, then GP isn't valid choice for Linux without changing crtbegin.

To use ShadowCallStack, executables need to be built with -msmall-data-limit=0 and the runtime environment they use needs to meet the ABI. Neither of those things is provided out of the box for Linux, just as on AArch64 neither building with -ffixed-x18 nor a runtime environment meeting the ABI exists out of the box on Linux. If a runtime environment for Linux were to support the ShadowCallStack ABI, that would be done in startup code that runs after the code injected by crtbegin.o and before any code built for the ShadowCallStack is expected to be usable, such as in the libc startup code.

paulkirth updated this revision to Diff 509811.Mar 30 2023, 2:08 PM

Remove blank lines added to test file.

@asb, @craig.topper, @jrtc27 Are there any remaining considerations for us here? From the discussions in psABI and sig-toolchain, I think we have a consensus that this is the approach we'll be taking for RISC-V. We'd prefer to correct this ASAP, so as to prevent future incompatibility/continuing to use a non-standard register.

craig.topper accepted this revision.Apr 5 2023, 3:35 PM

LGTM other than that one comment.

llvm/lib/Target/RISCV/RISCVSubtarget.cpp
85–86

Drop the blank line here.

I don't see changes here to make the Fuchsia & Android targets stop doing implicit -ffixed-x18 and to make them start doing implicit -msmall-data-limit=0 for non-PIC modes.

clang/lib/Driver/SanitizerArgs.cpp
547–548

For RISC-V this needs a similar check that the equivalent of -msmall-data-limit=0 is in force (that's the default under -fPIC but not other modes).

llvm/test/CodeGen/RISCV/reserved-regs.ll
59–60

gp is a fixed register for all RISC-V targets. What's important to check is that Fuchsia and Android disable the small-data-section behavior.

paulkirth added a comment.EditedApr 5 2023, 6:43 PM
This comment has been deleted.
clang/lib/Driver/SanitizerArgs.cpp
547–548

I'll look into adding support for that.

llvm/lib/Target/RISCV/RISCVSubtarget.cpp
86

@mcgrathr This stops Fuchsia and Android from reserving X18. I'll look at doing something similar for -msmall-data-limit=0.

jrtc27 added a comment.Apr 5 2023, 9:22 PM

From an ABI/codegen perspective the small data limit doesn’t matter here, all it does is govern whether things go in .sdata or not. The linker is what chooses whether to use GP, which happens regardless of whether something is in .sdata, just things in .sdata are more likely to be picked, since GP is defined to be 0x800 from its start. What matters is that the linker is told not to do GP relaxation.

asb added a comment.Apr 5 2023, 10:58 PM

@asb, @craig.topper, @jrtc27 Are there any remaining considerations for us here? From the discussions in psABI and sig-toolchain, I think we have a consensus that this is the approach we'll be taking for RISC-V. We'd prefer to correct this ASAP, so as to prevent future incompatibility/continuing to use a non-standard register.

It looks like we have strong consensus here. Could you update the patch to add a release note about this change please? (I guess it might merit inclusion in both the LLVM and the Clang release notes).

The RFC to check nobody downstream is concerned about changing the SCS register has been up for a week and a half with no concerns. You might consider leaving merging this until Monday to give it a full 2 week period, but I'm not opposed to going ahead before that. With the level of agreement we have it's hard to imagine changing direction and we want to change this default anyway - if there's a downstream user who needs to continue supporting the old SCS register then the logical path of action would be to add a new compiler option to enable that.

asb added a comment.Apr 6 2023, 8:46 AM

Ok, looks like the consensus I thought we had isn't quite there on the psABI thread - Andrew Waterman has some concerns. We should let that discussion play out some more.

paulkirth updated this revision to Diff 511458.Apr 6 2023, 9:51 AM

Fix formatting and update documentation.

paulkirth updated this revision to Diff 511466.Apr 6 2023, 10:19 AM
paulkirth marked 2 inline comments as done.

Remove extra line.

paulkirth updated this revision to Diff 511575.Apr 6 2023, 5:51 PM
paulkirth marked an inline comment as done.

Add default small-data-limit for Android and Fuchsia.

  • update tests
  • update clang driver
paulkirth updated this revision to Diff 511704.Apr 7 2023, 9:13 AM
paulkirth marked an inline comment as done.

Remove unrelated whitespace changes.

@asb Are we happy with the state of consensus w.r.t. using x3? I think the lingering concerns from the psABI discussion have been resolved.

asb accepted this revision.Apr 10 2023, 11:53 AM

@asb Are we happy with the state of consensus w.r.t. using x3? I think the lingering concerns from the psABI discussion have been resolved.

Yes, all LGTM now.

clang/lib/Driver/ToolChains/Clang.cpp
2062 ↗(On Diff #511704)

Nit: Missing space after platforms

mcgrathr accepted this revision.Apr 10 2023, 1:44 PM

lgtm with some nits and a few more test cases.

clang/docs/ShadowCallStack.rst
61

I think it would be worthwhile to say x3 (gp) here and perhaps everywhere that x3 is mentioned.
Since gp is always a reserved register from the compiler point of view, I think some different verbiage here about the RISC-V case and the interactions with -msmall-data-limit and linker behavior make sense.

clang/lib/Driver/SanitizerArgs.cpp
308

I'm guessing that "DSL" stands for "data small limit" or something, which is confusing since the thing is called small-data-limit, not data-small-limit. Anyway, this is the first use I've seen of DSL as an abbreviation related to any of this. I think it would be clear enough to just call it shadowCallStackConflicts since it's responsible for detecting conflicts between shadow-call-stack and whatever is relevant to that in the particular target.

315

This seems like it ought to parse the integer and then compare it to zero as an integer, rather than relying on one exact string representation of zero.

553

I'd just move all this per-target logic inside shadowCallStackConflicts and maybe just make it a void function called checkShadowCallStackConflcits or something.

clang/lib/Driver/ToolChains/Clang.cpp
2063 ↗(On Diff #511704)

typo: "it's"

2065 ↗(On Diff #511704)

I wonder if it wouldn't be more natural for the target hook to be defaultSmallDataLimit and return an integer where the base class returns 8 and the android/fuchsia subclasses return 0. It seems quite plausible that eventually different subtargets might have other tunings (for subtargets that do want to use gp relaxation).

clang/test/Driver/sanitizer-ld.c
746

I'd also add tests here for android and fuchsia targets with explicit -msmall-data-limit=<nonzero> being rejected by default but being allowed with -fno-sanitize=shadow-call-stack.

There should also be tests independent of shadow-call-stack that just verify at least for fuchsia targets that -msmall-data-limit=0 is passed by default for the default (PIE) case. Actually, it should also test that this remains so when -fno-sanitize=shadow-call-stack is passed, because it's still the standard target ABI that gp might be used for shadow-call-stack by interoperating code and so gp should never be used for something else without explicit -msmall-data-limit=... on fuchsia.

llvm/docs/ReleaseNotes.rst
158

Spaces before the parenthetical clauses.

llvm/include/llvm/TargetParser/RISCVTargetParser.h
42–43

Why not return an integer value instead of "is zero"?

jrtc27 added inline comments.Apr 10 2023, 1:51 PM
clang/docs/ShadowCallStack.rst
155

This doesn't really achieve anything other than not do one optimisation (which may or not matter; the grouping of small data may have a noticeable cache locality effect)

clang/lib/Driver/SanitizerArgs.cpp
566

Why is this an error? It may be a misguided thing to enable but it is 100% supported to combine this. All the limit does is put things in .sdata, but they can still be addressed just fine.

clang/lib/Driver/ToolChains/Clang.cpp
2072 ↗(On Diff #511704)

If you're modelling it on this warning (note, not an error) then this is crap too, there are legitimate reasons to use -G with -fPIC (and the -shared handling is even more questionable).

paulkirth added inline comments.Apr 10 2023, 2:47 PM
clang/lib/Driver/SanitizerArgs.cpp
308

Oops. Seems to be a typo. Thank you.

566

My understanding is that (whether they should or not) linkers are using the presence of the .sdata section to enable gp relaxation.

jrtc27 added inline comments.Apr 10 2023, 2:49 PM
clang/lib/Driver/SanitizerArgs.cpp
566

That is not true

jrtc27 added inline comments.Apr 10 2023, 2:53 PM
clang/lib/Driver/SanitizerArgs.cpp
566

Putting variables in .sdata merely makes it more likely that the linker will be able to relax accesses to them to use GP. Variables outside .sdata still can and will be relaxed if within range of whatever gets picked for __global_pointer$.

paulkirth updated this revision to Diff 512276.Apr 10 2023, 3:51 PM
paulkirth marked 7 inline comments as done.

Remove -msmall-data-limit related changes. There was a misunderstanding on my part about default linker behavior.

clang/lib/Driver/SanitizerArgs.cpp
566

Yes, I seem to have had been under the wrong impression. I've dropped everything regarding -msmall-data-limit.

paulkirth updated this revision to Diff 512277.Apr 10 2023, 3:58 PM

Remove dead declaration.

paulkirth updated this revision to Diff 512489.Apr 11 2023, 9:14 AM
paulkirth marked 2 inline comments as done.

Add spaces to parantheticals. Shorten link to discussion on psABI.