This is an archive of the discontinued LLVM Phabricator instance.

[C2x] Support <string.h> in freestanding
Needs RevisionPublic

Authored by aaron.ballman on Feb 27 2023, 8:57 AM.

Details

Reviewers
jyknight
efriedma
joerg
Group Reviewers
Restricted Project
Summary

This implements most of the support for WG14 N2524 (https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2524.htm), which adds <string.h> to the list of headers required by a freestanding implementation.

We do not currently have builtins for memccpy, memset_explicit, or strtok, so those APIs are not yet exposed via our freestanding string.h. Further, we do not yet implement WG14 N3020 which changes the return type for some of the interfaces in string.h. So this patch does not claim full support for string.h yet (it adds the __STDC_VERSION_STRING_H__ macro but leaves its value as 202300L instead of 202311L as required by the standard).

Diff Detail

Event Timeline

aaron.ballman created this revision.Feb 27 2023, 8:57 AM
Herald added a project: Restricted Project. · View Herald TranscriptFeb 27 2023, 8:57 AM
aaron.ballman requested review of this revision.Feb 27 2023, 8:57 AM
Herald added a project: Restricted Project. · View Herald TranscriptFeb 27 2023, 8:57 AM
efriedma requested changes to this revision.Feb 27 2023, 9:13 AM

Providing this header doesn't do anything useful without an actual implementation; all of these "__builtin" calls just lower to libc calls in the general case. How do you plan to provide implementations?

This revision now requires changes to proceed.Feb 27 2023, 9:13 AM

Providing this header doesn't do anything useful without an actual implementation; all of these "__builtin" calls just lower to libc calls in the general case. How do you plan to provide implementations?

I *thought* that the difference between:
https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Basic/Builtins.def#L1038
and
https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Basic/Builtins.def#L559
was that the first one is lowered to a potential library call while the second one is is lowered to a backend implementation that performs the work. (This is why I thought we would not be able to support memccpy or strtok -- they were missing the __builtin_ variants.) If the backend is just going to generate a library call that requires an external library when calling a __builtin_, then I agree that there's more here that needs to be done. How do we typically handle it for freestanding functions? Do we implement something in compiler-rt as a fallback, or do we emit LLVM IR for the function implementation, etc?

Historically, the required functions for a "freestanding" C implementation were very restricted. Freestanding headers didn't export library functions, just constants and types.

As a practical matter, we actually do need a few functions to support code generation for C even if they don't include any standard library headers. We expect "freestanding" users to provide the three functions memcpy, memmove, and memset. Other misc. functions like large integer operations and floating point are provided by compiler-rt.builtins (which is ABI-compatible with libgcc).

Note that we can't easily expand the API surface of compiler-rt.builtins; see https://discourse.llvm.org/t/proposal-split-built-ins-from-the-rest-of-compiler-rt/67978 .

was that the first one is lowered to a potential library call while the second one is is lowered to a backend implementation that performs the work

The difference is really that the __builtin_* functions are recognized as builtins even if the the user is using -fno-builtin/-ffreestanding. It doesn't mean we actually have dedicated codegen support.

We do not currently have builtins for memccpy, memset_explicit, or strtok

I'm not sure it makes sense to provide a "freestanding" strtok; it requires global state.

A freestanding implementation doesn't necessarily mean that everything is header-only. It's fine to require linking against a (freestanding) C runtime library. All header-only is fine too though, if you want to make that work.

Architecturally, I don't feel it is required that Clang be the location of all the freestanding headers for C. I think that's fine for the C library in the final toolchain to own those headers. I'm not super familiar with what interface contracts you have between the clang-provided C headers and the C-library provided C headers though.

I'm not sure it makes sense to provide a "freestanding" strtok; it requires global state.

I agree with this, but the C committee felt otherwise. C++26 freestanding is most likely including strtok too, to stay consistent with C (the freestanding C library paper is through LWG and awaiting a C++26 plenary vote). It shouldn't be hard to implement in an OS-independent way, but it will be a low quality implementation with cross-thread races.

Historically, the required functions for a "freestanding" C implementation were very restricted. Freestanding headers didn't export library functions, just constants and types.

<nods>

As a practical matter, we actually do need a few functions to support code generation for C even if they don't include any standard library headers. We expect "freestanding" users to provide the three functions memcpy, memmove, and memset. Other misc. functions like large integer operations and floating point are provided by compiler-rt.builtins (which is ABI-compatible with libgcc).

Note that we can't easily expand the API surface of compiler-rt.builtins; see https://discourse.llvm.org/t/proposal-split-built-ins-from-the-rest-of-compiler-rt/67978 .

Oof.

was that the first one is lowered to a potential library call while the second one is is lowered to a backend implementation that performs the work

The difference is really that the __builtin_* functions are recognized as builtins even if the the user is using -fno-builtin/-ffreestanding. It doesn't mean we actually have dedicated codegen support.

Ah, that's good to know. Thanks!

We do not currently have builtins for memccpy, memset_explicit, or strtok

I'm not sure it makes sense to provide a "freestanding" strtok; it requires global state.

I'm happy to file a CD2 comment to this effect.

A freestanding implementation doesn't necessarily mean that everything is header-only. It's fine to require linking against a (freestanding) C runtime library. All header-only is fine too though, if you want to make that work.

Architecturally, I don't feel it is required that Clang be the location of all the freestanding headers for C. I think that's fine for the C library in the final toolchain to own those headers. I'm not super familiar with what interface contracts you have between the clang-provided C headers and the C-library provided C headers though.

Okay, so this is potentially ignorance on my part. I was under the impression that folks using freestanding mode did not want any library to be linked in. Are there freestanding libc implementations out there that the user would link in instead that we defer to (same as we defer hosted C standard library interfaces to the CRT)? Basically, do we need to do nothing in Clang to support N2524?

I'm not sure it makes sense to provide a "freestanding" strtok; it requires global state.

I agree with this, but the C committee felt otherwise. C++26 freestanding is most likely including strtok too, to stay consistent with C (the freestanding C library paper is through LWG and awaiting a C++26 plenary vote). It shouldn't be hard to implement in an OS-independent way, but it will be a low quality implementation with cross-thread races.

FWIW, WG14 is having a second round of CD ballot comments. If you'd like, I'm happy to file a comment asking to remove strtok from the required set for freestanding conformance.

Okay, so this is potentially ignorance on my part. I was under the impression that folks using freestanding mode did not want any library to be linked in. Are there freestanding libc implementations out there that the user would link in instead that we defer to (same as we defer hosted C standard library interfaces to the CRT)? Basically, do we need to do nothing in Clang to support N2524?

Well, people want all sorts of things :)

(Gross generalizations coming) The compiler writer view of freestanding that I've heard in the past is that the compiler writer views freestanding as the necessary parts of the library needed to support the language. This is mostly accurate when describing freestanding prior to C++23 and C23. Most of that is either header-only, in libc++abi, or in compiler-rt. This makes testing the compiler a bit easier, as you don't need a full stdlib. That's not a design goal of freestanding post C/C++23 though. It's fine for Clang to provide half of a stdlib for internal testing or integration with other C libraries, so long as the final toolchain has everything.

One challenge for some customers will be that they like to build with -nodefaultlibs, but that's not a deal breaker here, it just means that they would need to manually put their C library on the link line. Alternatively, they could start passing other linker flags, like -nostartfiles instead, but that will be use case dependent. These kinds of hoops have historically been needed because a customer may use a "stock" ARM Clang, but not want to use the libc that came with that toolchain, so the big linker flags would come out to give the user control.

I don't think that Clang-the-compiler needs to do anything to support the new C/C++ freestanding features. The main caveat being that you still need to be able to ensure the provided memcpy gets used instead of a codegenned (possibly vectorized) memcpy, but I think -ffreestanding already forces the compiler to generate calls and not use in-compiler codegen.

FWIW, WG14 is having a second round of CD ballot comments. If you'd like, I'm happy to file a comment asking to remove strtok from the required set for freestanding conformance.

That would be much appreciated. I can try to get my INCITS / ISO status squared away so that I can represent that issue at the appropriate meeting.

Okay, so this is potentially ignorance on my part. I was under the impression that folks using freestanding mode did not want any library to be linked in. Are there freestanding libc implementations out there that the user would link in instead that we defer to (same as we defer hosted C standard library interfaces to the CRT)? Basically, do we need to do nothing in Clang to support N2524?

Oh, and I forgot GPUs. GPUs do like having everything be header-only, but in my opinion, that's still a requirement / request for C library authors rather than the compiler.

A freestanding implementation doesn't necessarily mean that everything is header-only. It's fine to require linking against a (freestanding) C runtime library. All header-only is fine too though, if you want to make that work.

Architecturally, I don't feel it is required that Clang be the location of all the freestanding headers for C. I think that's fine for the C library in the final toolchain to own those headers. I'm not super familiar with what interface contracts you have between the clang-provided C headers and the C-library provided C headers though.

In practice, there are basically three ways people can use clang:

  1. A normal operating system target with a "hosted" libc.
  2. A baremetal target with an "embedded" libc; basically a stripped down libc which only includes stuff that doesn't require an operating system. What exactly this includes varies; may include some form of "malloc", some "POSIX" APIs, semihosting, etc.
  3. A baremetal target with no libc, using custom implementations for everything. This is what we conventionally referred to as "freestanding", and what -ffreestanding implements. The user provides memcpy/memmove/memset, and uses the compiler-provided "builtins" lib if neceessary, but nothing we would traditionally call "libc".

We could just refuse to provide any APIs outside of our conventional notion of "freestanding", I guess, and say users that want a C23 freestanding environment need to find their own mini-libc. Or just provide the header, and say users are responsible for providing implementations, like they currently are for memcpy etc. That doesn't really align with what the committee was hoping for, though...

Maybe eventually we can come up with some way to use implementations provided by LLVM libc, but I'm not sure what exactly that looks like.

  1. A normal operating system target with a "hosted" libc.
  2. A baremetal target with an "embedded" libc; basically a stripped down libc which only includes stuff that doesn't require an operating system. What exactly this includes varies; may include some form of "malloc", some "POSIX" APIs, semihosting, etc.
  3. A baremetal target with no libc, using custom implementations for everything. This is what we conventionally referred to as "freestanding", and what -ffreestanding implements. The user provides memcpy/memmove/memset, and uses the compiler-provided "builtins" lib if neceessary, but nothing we would traditionally call "libc".

In my committee work, I'm targeting option 2. I think that's what should be considered when talking about freestanding in a standards context.

Option 3 is valuable and should continue to exist, but it is non-conforming pre-C23 and post-C23. The standards don't have any provision for requiring the user to implement mem* functions that the implementation chooses to invoke without the user ever spelling that function call (perhaps because they did char buf[256] = {0} though). Post-C23, it doesn't provide enough. The C library authors will likely need this mode to author the C library itself though. I'll suggest calling this mode something like "nodefaultlibs".

perhaps because they did char buf[256] = {0} though

Right, we need memcpy and memset to emit reasonable code for struct/array initialization and assignment.

We don't provide any library that includes memcpy/memset/memmove because of a combination of legacy, and that writing well-optimized versions is hard. Ideally, it probably makes sense.

rsmith added a subscriber: rsmith.Feb 27 2023, 3:48 PM

Do we know what GCC intends to do about this C change? Per their documentation, they intend for GCC to be / eventually become a complete freestanding implementation without need for a separate libc:

GCC aims towards being usable as a conforming freestanding implementation, or as the compiler for a conforming hosted implementation.
GCC does not provide the library facilities required only of hosted implementations, nor yet all the facilities required by C99 of freestanding implementations on all platforms.

(Emphasis mine.) Likely because of GCC's perspective on this, the set of C headers provided by GCC, Clang, ICC, etc. has included the complete list of freestanding headers and more or less no others, with some libc implementations providing the full set and some providing just the non-freestanding ones. If we're moving away from that, we'll presumably need a new agreement between compiler folks and libc folks about which headers are provided by whom (and I expect it'll be "compilers provide what the C standard said before this change to freestanding and we pretend the change never happened").

Likely because of GCC's perspective on this, the set of C headers provided by GCC, Clang, ICC, etc. has included the complete list of freestanding headers and more or less no others, with some libc implementations providing the full set and some providing just the non-freestanding ones. If we're moving away from that, we'll presumably need a new agreement between compiler folks and libc folks about which headers are provided by whom (and I expect it'll be "compilers provide what the C standard said before this change to freestanding and we pretend the change never happened").

Yeah...this seems like a pretty big divergence from the historical view compilers have for "freestanding" mode, and I think the implementation should be more along the lines of a "micro-libc", than something built into clang. Especially considering the potential future additions e.g. errno.h. If we look at this as a specification for what to provide in a "micro-libc", having such global state is not a major issue, but if we look at this as a "compiler builtin" sort of thing (ala traditional freestanding), then it's quite a problem.

So, I think the correct solution here is to just document what parts of a C2x-conforming "freestanding" Clang provides itself, and what remains for an external "mini-libc" implementation to provide.

It probably makes sense for one such external implementation to be provided by llvm-libc in the future -- I suspect it should have a separate "freestanding" target mode, which would omit all the runtime features like memory allocation and threads, and provide just the pure library functionality. But I'd also expect such an llvm-libc freestanding mode would also provide a lot more functionality than just the minimum requirements -- there's a whole lot more "pure library" functions in C than just the ones required in the new "freestanding" standard.

Likely because of GCC's perspective on this, the set of C headers provided by GCC, Clang, ICC, etc. has included the complete list of freestanding headers and more or less no others, with some libc implementations providing the full set and some providing just the non-freestanding ones. If we're moving away from that, we'll presumably need a new agreement between compiler folks and libc folks about which headers are provided by whom (and I expect it'll be "compilers provide what the C standard said before this change to freestanding and we pretend the change never happened").

Yeah...this seems like a pretty big divergence from the historical view compilers have for "freestanding" mode, and I think the implementation should be more along the lines of a "micro-libc", than something built into clang. Especially considering the potential future additions e.g. errno.h. If we look at this as a specification for what to provide in a "micro-libc", having such global state is not a major issue, but if we look at this as a "compiler builtin" sort of thing (ala traditional freestanding), then it's quite a problem.

On the one hand, I think many users will want a libc that targets their particular freestanding environment so that they get the best performance characteristics (and other considerations) for their target. This suggests the micro-libc approach. On the other hand, I think a not-insignificant number of users are interested in freestanding environments for one-off/fun/experimental projects where ease of access is more important than performance characteristics -- think: users who are playing around with an arduino, etc. This suggests a complete freestanding implementation may be preferable. It's somewhat hard to swallow to ask someone to go find a freestanding C library for a fun-time hardware project that might not have fantastic documentation just so the user can use strlen -- I suspect a nontrivial number of those folks are going to write their own strlen function and be done with it rather than try to solve C and C++'s terrible lack of package management. But it's also an awful lot of effort to write a custom implementation of strtok on the off chance a fun-time hardware project might want to use it.

tl;dr: I don't know there's a clear "winner" approach, but I want to make sure we keep in mind the user experience for the full breadth of users we support.

So, I think the correct solution here is to just document what parts of a C2x-conforming "freestanding" Clang provides itself, and what remains for an external "mini-libc" implementation to provide.

I think that makes sense. From what I'm reading, it sounds like that documentation is effectively pointing out that -ffreestanding mode in Clang will provide headers containing type definitions and macros for target-specific information that only the compiler knows about, expects the user to provide an implementation of memcpy, memmove, and memset (others?), but otherwise leaves all freestanding C standard library functions up to a user-provided library. Is that about what we'd want to document or do you expect more? (I can try to put together some docs we can iterate on.)

It probably makes sense for one such external implementation to be provided by llvm-libc in the future -- I suspect it should have a separate "freestanding" target mode, which would omit all the runtime features like memory allocation and threads, and provide just the pure library functionality. But I'd also expect such an llvm-libc freestanding mode would also provide a lot more functionality than just the minimum requirements -- there's a whole lot more "pure library" functions in C than just the ones required in the new "freestanding" standard.

+1

Let's talk about these use cases.

On the one hand, I think many users will want a libc that targets their particular freestanding environment so that they get the best performance characteristics (and other considerations) for their target.

I think this fits kernel driver environments, GPUs, and other environments where the vendor is heavily involved in compiler development. The vendor is likely to provide extensions to their freestanding implementation as well (like math.h functions for GPUs).

On the other hand, I think a not-insignificant number of users are interested in freestanding environments for one-off/fun/experimental projects where ease of access is more important than performance characteristics -- think: users who are playing around with an arduino, etc.

This is also an important use case. I don't think it rules out a "stock" freestanding implementation though. A toolchain could reasonably ship one freestanding static library for each ISA-subset / target-triple they want to support, and it could work regardless of the target operating system. memcpy has the same assembly whether it's on OpenRTOS or threadx or MyHomeGrownOS, so long as all are on the same processor with the same calling conventions. A toolchain could also try the header-only approach and lean on the optimizer to turn strcpy into reasonable assembly.

I will definitely agree on the big point with this use case though. I don't want to force every OS vendor to also need to be a compiler / libc vendor, so long as they are willing to settle for the freestanding subset of the language. I also don't want to force them to rewrite strcpy and friends.

On the other hand, I think a not-insignificant number of users are interested in freestanding environments for one-off/fun/experimental projects where ease of access is more important than performance characteristics -- think: users who are playing around with an arduino, etc.

I agree. But, I see no reason that building the micro-libc for a given target should be any more onerous than building the compiler-rt builtins for said target -- which you already need to do! And it doesn't actually help to make micro-libc _less_ onerous to build than compiler-rt builtins, since you also need that...

Of course, building compiler-rt builtins actually IS rather onerous for users at the moment. This is quite unfortunate, and we should definitely improve the experience of that. And then do the same thing for any micro-libc target. (I think it might go a long way to do something along the lines of distributing target-independent sources and a _trivial_ standalone builtins build script with Clang, such that a user could on-demand build a libclang_builtins.a for any target configuration they wish to use. But that's really a topic for a different thread.)

On the other hand, I think a not-insignificant number of users are interested in freestanding environments for one-off/fun/experimental projects where ease of access is more important than performance characteristics -- think: users who are playing around with an arduino, etc.

I agree. But, I see no reason that building the micro-libc for a given target should be any more onerous than building the compiler-rt builtins for said target -- which you already need to do! And it doesn't actually help to make micro-libc _less_ onerous to build than compiler-rt builtins, since you also need that...

In my ideal world (which might not be practical!), the user doesn't build compiler-rt at all (they don't even need to know it's a thing) -- they use whatever means they usually do to get clang installed on their system and that's sufficient for them to work on their project. So, to me, asking the user to build a micro-libc or compiler-rt is somewhat onerous in and of itself.