docs: Update SafeStack docs with separate-stack-seg feature and various USP storage modes

[mlemay-intel]

Summary

This patch updates the SafeStack documentation to describe the
separate-stack-seg feature for helping to enable hardware segmentation on
x86-32.

It also describes the following options:

• -safe-stack-usp-storage=single-thread
• -safe-stack-usp-storage=tcb
• -safe-stack-usp-tcb-offset=<offset>
• -safe-stack-usp-init

[mlemay-intel]

Add documentation for -safe-stack-usp-storage=tcb, -safe-stack-usp-tcb-offset=<offset>, and -safe-stack-usp-init.

[pcc]

We shouldn't be adding (much less documenting) -mllvm flags. Is there any reason why this behavior can't be gated on the OS?

[mlemay-intel]

In D19483#417844, @pcc wrote:

We shouldn't be adding (much less documenting) -mllvm flags. Is there any reason why this behavior can't be gated on the OS?

We actually already have added at least one -mllvm flag: -mllvm -safe-stack-usp-storage=single-thread. You can see an example of how it can be used here: https://github.com/contiki-os/contiki/pull/1642/commits/f13d1e22669d5526f2a721a337acd06f23a8d49d Can you please provide an example of how you think that setting should be specified without the use of the -mllvm flag?

[pcc]

You should be using -target x86-unknown-contiki or similar. That should tune the behaviour to what is required for that OS. See what we have in TargetLoweringBase::getSafeStackPointerLocation to provide Android-specific behaviour for example.

The existence of flags is not justification to add more. Besides, it appears that the -safe-stack-usp-storage flag was added without proper review. It was reviewed in D15673, but the mailing list was not added as a subscriber. If I had been aware of that review I would have made the same objections at that time.

[mlemay-intel]

In D19483#417857, @pcc wrote:

You should be using -target x86-unknown-contiki or similar. That should tune the behaviour to what is required for that OS. See what we have in TargetLoweringBase::getSafeStackPointerLocation to provide Android-specific behaviour for example.

This makes sense for the example I gave. However, there are also more complicated situations. Sometimes it is necessary to specify different options for different files that are compiled for the same OS. For example, early during the initialization of a dynamic linker or C library, a single-threaded mode of USP storage needs to be supported. TLS is not available at that time. How should requirements like that be conveyed to the compiler?

The existence of flags is not justification to add more. Besides, it appears that the -safe-stack-usp-storage flag was added without proper review. It was reviewed in D15673, but the mailing list was not added as a subscriber. If I had been aware of that review I would have made the same objections at that time.

Sorry, I only recently learned that the mailing list should be added as a subscriber. Prior to that, I thought that patches were automatically sent to the appropriate mailing lists.

[pcc]

This makes sense for the example I gave. However, there are also more complicated situations. Sometimes it is necessary to specify different options for different files that are compiled for the same OS. For example, early during the initialization of a dynamic linker or C library, a single-threaded mode of USP storage needs to be supported. TLS is not available at that time. How should requirements like that be conveyed to the compiler?

In cases like this, we should introduce a new -m flag in the Clang driver and plumb that through to the SafeStack pass using a target-specific function attribute.

[mlemay-intel]

In D19483#417865, @pcc wrote:

This makes sense for the example I gave. However, there are also more complicated situations. Sometimes it is necessary to specify different options for different files that are compiled for the same OS. For example, early during the initialization of a dynamic linker or C library, a single-threaded mode of USP storage needs to be supported. TLS is not available at that time. How should requirements like that be conveyed to the compiler?

In cases like this, we should introduce a new -m flag in the Clang driver and plumb that through to the SafeStack pass using a target-specific function attribute.

It appears that Clang already supports an -mthread_model=single option, so do you recommend that I remove the -mllvm -safe-stack-usp-storage=single-thread option and rely on -mthread_model instead? I added @eugenis to get his feedback as well, since he approved my patch that added the new flag. -mllvm -safe-stack-usp-storage=single-thread would have been documented for the first time by this patch, and I have not heard of anyone else using it yet.

I don't see any existing option that seems suitable for indicating that a file may be used during dynamic linker or C library initialization, so perhaps we should add an -mruntime-init flag? I welcome other naming suggestions.

[pcc]

It appears that Clang already supports an -mthread_model=single option, so do you recommend that I remove the -mllvm -safe-stack-usp-storage=single-thread option and rely on -mthread_model instead?

Makes sense to me. We're free to break -mllvm flags, they are for development/debugging purposes only.

Of course, the global variable USP requires runtime support (which I assume Contiki provides), so we can probably gate this on OS + thread model.

I don't see any existing option that seems suitable for indicating that a file may be used during dynamic linker or C library initialization, so perhaps we should add an -mruntime-init flag?

Yes, I'm not sure about the name but we should do something like this.

docs/SafeStack.rst

 region. The safe stack is automatically protected against stack-based buffer
 overflows, since it is disjoint from the unsafe stack in memory, and it itself
 is always accessed in a safe way. In the current implementation, the safe stack
 is protected against arbitrary memory write vulnerabilities though
 randomization and information hiding: the safe stack is allocated at a random
 address and the instrumentation ensures that no pointers to the safe stack are
 ever stored outside of the safe stack itself (see limitations below).
 
+SafeStack can be combined with the ``separate-stack-seg`` feature (using the
+``-mseparate-stack-seg`` command line option) for x86-32. Each x86 memory
+operand has an effective segment. The ``separate-stack-seg`` feature emits
+segment override prefixes as necessary to direct safe stack accesses to the SS
+segment and ordinary data accesses to the DS and ES segments. That eliminates
+one obstacle preventing the limits of the DS and ES segments from being set
+below the start of the safe stacks. However, additional runtime support is
+needed to actually perform that segment configuration. Furthermore, all code in
+the program, including library routines, that may be run with such a segment
+configuration must be compiled to support it. This is a more extensive level of
+runtime support than is supplied by the compiler-rt library.  Thus, the
+compiler-rt library is not automatically linked when the ``separate-stack-seg``
+feature is enabled.
+
+The ``separate-stack-seg`` feature only performs intra-procedural analysis.  It
+assumes that only the stack register points to the safe stack at function
+entry. It assumes that no pointers passed into the function or stored as global
+variables point to the safe stack. SafeStack helps to satisfy this assumption
+when the ``separate-stack-seg`` feature is enabled by moving additional types of
+allocations to the unsafe stack if pointers to them may be passed to
+subroutines. Note that variables can be annotated as pointing to address space
+258 independent of the ``separate-stack-seg`` feature, and the compiler will
+generate code to access the referenced data in the SS segment.
+
+Special handling is needed for variadic argument lists when the
+``separate-stack-seg`` feature is enabled. Certain variadic argument intrinsics
+(e.g. ``va_start`` and ``va_arg``) may store the addresses of variadic arguments
+to memory. Variadic arguments are passed on the safe stack. Ordinarily, the
+``separate-stack-seg`` feature attempts to block most pointers to the safe stack
+from being stored, unless it is able to track them as register spills. The
+``separate-stack-seg`` feature does not support tracking other types of stores
+of safe stack pointers and pairing them with subsequent loads (i.e. register
+fills) so that it can emit the necessary segment override prefixes for safe
+stack accesses. However, the ``separate-stack-seg`` feature attempts to identify
+and permit instructions that store safe stack pointers to support variadic
+arguments. Its heuristics are only capable of identifying simple usages of the
+variadic argument intrinsics. More complex usages may result in assertion
+failures. The relevant assertion can be bypassed just for functions with a
+``va_start`` by passing an argument to LLVM: ``-mllvm
+-sep-stk-seg-sp-store-verif=ignore-va``. That assertion can be bypassed for all
+functions by passing ``-mllvm -sep-stk-seg-sp-store-verif=none``. However,
+passing either of these arguments may lead to incorrect code being generated.
+
 Known security limitations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 A complete protection against control-flow hijack attacks requires combining
 SafeStack with another mechanism that enforces the integrity of code pointers
 that are stored on the heap or the unsafe stack, such as `CPI
 <http://dslab.epfl.ch/proj/cpi/>`_, or a forward-edge control flow integrity
 mechanism that enforces correct calling conventions at indirect call sites,

 future, such leaks could be detected by static or dynamic analysis tools and
 prevented by adjusting such functions to either encrypt the stack pointer when
 storing it in the heap (as already done e.g., by ``setjmp``/``longjmp``
 implementation in glibc), or store it in a safe region instead.
 
 The `CPI paper <http://dslab.epfl.ch/pubs/cpi.pdf>`_ describes two alternative,
 stronger safe stack protection mechanisms, that rely on software fault
 isolation, or hardware segmentation (as available on x86-32 and some x86-64
-CPUs).
+CPUs). The ``separate-stack-seg`` feature can be used with appropriate runtime
+support to enable hardware segmentation on x86-32.
 
 At the moment, SafeStack assumes that the compiler's implementation is correct.
 This has not been verified except through manual code inspection, and could
 always regress in the future. It's therefore desirable to have a separate
 static or dynamic binary verification tool that would check the correctness of
 the SafeStack instrumentation in final binaries.
 
 Usage
 =====
 
 To enable SafeStack, just pass ``-fsanitize=safe-stack`` flag to both compile
 and link command lines.
 
+To store the unsafe stack pointer in an ordinary global variable instead of a
+thread-local variable, add the ``-mllvm -safe-stack-usp-storage=single-thread``
+flag. This is only suitable for single-threaded programs and is incompatible
+with the compiler-rt library. It was originally intended for certain embedded
+systems. A linked object or library must define and initialize
+``__safestack_unsafe_stack_ptr`` with type ``void*`` as well as associated
+storage for the unsafe stack.
+
+A single-threaded unsafe stack pointer can also be useful during program
+initialization in the dynamic linker and C library startup routines, before
+thread-local storage is available. Some functions may need to be invoked both
+during this stage and later after the program has switched to using thread-local
+unsafe stack pointers. The ``-mllvm -safe-stack-usp-init`` flag adds support
+both of these modes, assuming that appropriate runtime support is available.
+Each such function that accesses the unsafe stack pointer first attempts to use
+the single-threaded ``__safestack_unsafe_stack_ptr_init`` with type ``void*`` if
+it is not null. If that variable is null, the function falls back to using
+``__safestack_unsafe_stack_ptr`` wherever that is configured to be stored.
+
+It can be more efficient to store the unsafe stack pointer in the Thread Control
+Block (TCB) rather than ordinary thread-local storage. The
+``-mllvm -safe-stack-usp-storage=tcb`` flag selects this type of storage. The
+``-mllvm -safe-stack-usp-tcb-offset=<offset>`` flag must also be used to specify
+the byte offset within the TCB where the unsafe stack pointer should be stored.
+This requires runtime support.
+
 Supported Platforms
 -------------------
 
 SafeStack was tested on Linux, FreeBSD and MacOSX.
 
 Low-level API
 -------------