[libc] Header generation scheme.

[sivachandra]

Summary

This patch has been prepared to showcase the header generation proposal posted on libc-dev.

It is incomplete and will not be submitted or sent for formal review.

[MaskRay]

libc/spec/stdc.td

2

Can you explain a bit more why we need a header generator?

If you want to make some declarations condition on feature macros, I think the usual approach is:

int foo(int);
int bar(int);

#if defined(_BSD_SOURCE) || defined(_XOPEN_SOURCE)
int qux(int);
#endif

[sivachandra]

libc/spec/stdc.td

2

Yes, that is how current implementations do it. While it is not necessarily bad, we want to avoid such patterns as much as possible so that we do not end up with the macro mess that current libcs ended up having. If you always generate, neither the sources, nor the generated headers have the macro mess. More importantly though, generating headers serves the use case of being able to selectively pick and choose what one wants to expose on their platform. Down stream configs become easier to manage.

[abrachet]

libc/spec/stdc.td

2

FWIW I don’t think conditionally exposing function definitions with the *_SOURCE macros is really a macro mess, it’s a pretty common practice among most (all?) libcs.

How far does this picking and choosing go? Would I be able to choose not to have the qux symbol in my libc.so at all? If so then it would be pretty bad to have it in the header at all even if it’s behind _GNU_SOURCE or similar I think.

[sivachandra]

libc/spec/stdc.td

2

Yes, that is point. It will neither be in the header nor in the library files like libc.a.

[abrachet]

LGTM I see no problems from what is here. I'd also like to say that the emphasis on writing docs for all of this is something I'm really happy to see.
I look forward to seeing how the generator works. I am also cautiously optimistic about how well this mechanism will work to create more exotic functions and typedefs. signal(3) and jmp_buf will surely be interesting interesting :)

libc/spec/stdc.td

12

Nit: Might be worth it to add strcat and acosl because they are in header_conf.td.

[theraven]

libc/spec/stdc.td

2

I think that it will still be important to have the ability to subset dynamically based on macros. Consider a project that is 99% portable POSIX code but has a platform abstraction layer that uses per-platform extensions. When building this project on Linux, for example, I would like to be able to expose everything in the platform abstraction layer, but restrict every other compilation unit to only POSIX functions. This is a fairly common idiom and helps catch portability issues early. Requiring every project with this structure to add a tablegen pass to its build system to generate new libc headers (and handle all of the -isystem flags correctly so that they override the default ones) seems a lot to ask.

That said, I'm fairly confident that this is something that can be added later in the TableGen back end, if the sources contain the information about the set of standards that a function conforms to. It should be easy to generate a .h that exposes everything that will work on a given platform inside #ifdefs for the relevant standards. Generating this from TableGen is a lot less horrible than maintaining it manually.

6

A minor issue, but it would be nice if the argument types were TableGen types and not strings, so that we can avoid accidental header pollution. For example, there are a number of libc headers that define interfaces that use size_t, but that do not export size_t. To make these work, implementations typically define __size_t in an internal header and then typedef __size_t size_t (guarded against multiple definition) in the headers that are expected by the standard to export it.

8

math.h is expected to define float_t and double_t. The values of these (float and double, double and double, or something else) depends on the value of the FLT_EVAL_METHOD macro. Please can you at least sketch out how this would be implemented? I presume that we'd want to pick a single value for FLT_EVAL_METHOD (most likely 1) and support only that, but we'd still need to export the two types and the macro from this header.

[theraven]

libc/spec/stdc.td

6

Additionally, it would be nice to decouple attributes here. For example, restrict and _Nonnull can be specified inline (though the GCC equivalent can't), but things like the printf-like attributes are properties of the arguments that need to be specified as function attributes.

In an ideal world, we'd also have SAL annotations, but these are currently lacking a clang implementation.

[sivachandra]

Add more examples.

[sivachandra]

libc/spec/stdc.td

2

I agree that we will end up with a mix of things. Where exactly do we draw a line, I am not sure at this point; Learn as we go may be.

And yes, we can adjust the backend and add additional attributes to get the header generator to put in the excluding/including macros.

6

About "decouple attributes", Can you give some examples of what you are describing here? I probably understand vaguely, but I want to see an example to make sure.

6

[Side note: We have decided, on another patch, that items like size_t come from freestanding headers and so we will make use of them instead of making LLVM-libc re-implement/re-define them.]

I understand and agree that it would be good to have TableGen types for arg types But, how deep/far do we go? For example, should there be a table gen type size_t and another one for size_t *?

What exact problems do you foresee with using strings instead of TableGen types? One obvious problem is typos. Are there any other?

8

I have added something to show how items like float_t/double_t/size_t would be handled.

[sivachandra]

Use TableGen types instead of strings.

libc/spec/stdc.td

6

This is not to say we should not do TableGen types. I have now uploaded a new diff with only TableGen types.

[theraven]

libc/spec/stdc.td

6

As a concrete example, this is FreeBSD libc's declaration of snprintf:

int snprintf(char * __restrict, size_t, const char * __restrict, ...) __printflike(3, 4);

The third argument has two attributes:

1. It is a restrict-qualified pointer, so the caller guarantees that it does not alias any other arguments.
2. It is a printf-compatible format string, with the formatted values starting in argument 4.

The Windows version of snprintf has the following declaration with a much richer set of metadata:

_Success_(return >= 0)
_Check_return_opt_
_CRT_STDIO_INLINE int __CRTDECL snprintf(
    _Out_writes_opt_(_BufferCount) _Always_(_Post_z_) char*       const _Buffer,
    _In_                                              size_t      const _BufferCount,
    _In_z_ _Printf_format_string_                     char const* const _Format,
    ...)

Beyond the C standard definitions, these indicate:

1. That the return value does not need to be checked.
2. That the calling convention is the one used for the standard library (not necessarily the default calling convention).
3. That the first argument is either a null pointer or a pointer that is written through and must be (at least) as long as the _BufferCount bytes and, on output, will always be null terminated.
4. That the second argument is an input parameter (redundant: it's a value type)
5. That the third argument is an input parameter (not written through) and is a null-terminated string.
6. That the third argument is a printf-compatible format string.

It would be nice if those were attributes could be expressed uniformly. Visual Studio, clang, and gcc all have subtly different ways of expressing some of these (as an aside: I'd love to see clang gain support for SAL annotations).

6

Using the definition of size_t from another header makes avoiding header pollution difficult. How do you expect to be able to implement headers that are expected to expose interfaces that use size_t but not provide size_t? Other libc implementations define something like __size_t for use in these headers. Clang's freestanding headers don't have to address this problem, but a full libc does.

I would expect size_t* to be something like Pointer<size_t>, which would be size_t* in the implementation but potentially __size_t* or size_t* in the header, depending on whether the header is supposed to export size_t. This is very hard to do with strings (it's possible, but it involves a load of parsing). Ideally, I'd like it to be something like Pointer<size_t>,In,NotNull.

8

I'm not sure how this expresses their conditional declaration or the relationship between them and the macro definition. Are these simply informal contracts that we write in comments, or do you imagine a scheme for expressing them directly in the TableGen?

31

What is the definition of NULL? This is one of the most horrible macros to define in a C standard library. Depending on the language dialect and supported extensions, it is either:

• __null
• nullptr
• 0
• (void*)0

[sivachandra]

libc/spec/stdc.td

6

GCC and Clang's free standing headers give us a way to expose the type size_t and the macro NULL without causing header pollution. See the file config/linux/header_conf.td in this patch. It shows how size_t and NULL are "implemented" without causing header pollution.

6

Thanks for the detailed reply. This file is a list of items the standard says should be provided. Things like function attributes and argument annotations are implementation detail and should be captured in the platform config file like the file config/linux/header_conf.td in this patch.

I will soon upload a diff with an example of how this can be done.

8

This file is merely a list of what the standard says should be provided. How it is implemented is an implementation detail captured in the platform config file. I have an example of the Linux config file in this patch at config/linux/header_conf.td. The relationship between float_t, double_t and the FLT_EVAL_METHOD macro are captured in that platform config file.

31

Again, see config/linux/header_conf.td for the definition of NULL: It is not really defined, but just pulled out from the stddef.h freestanding header.

[sivachandra]

Add example of function attrbutes and argument annotations.

[sivachandra]

My latest diff showcases function attributes and also argument annotations.

About size_t like items which standard says that multiple headers files should provide: As theraven@ suggests, we will in general have an internal header file. For size_t itself, the freestanding headers provide us a convenient way to do it without causing header pollution.

[sivachandra]

libc/spec/stdc.td

6

I thought about this a little more and may be what you are saying is that, we should "invent" a way to specify argument annotations in a platform independent fashion and have the header generator generate the annotations suitable for the target platform. I agree that such a mechanism would be ideal and also be much superior to what I have proposed.

For the sake of progress now, I am OK to leave a place holder for argument annotations, and in a later round add annotations wherever suitable.

[theraven]

libc/spec/stdc.td

6

I am happy as long as we don't do anything that precludes adding this later. I think the current scheme is sufficient to be refactored into the right thing later.

Ideally, I would like to see a SAL implementation for clang, but that will require a nontrivial amount of work and shouldn't be a blocker for this.

[sivachandra]

Abandoning this change now as the patch implementing the ideas showcased here has landed.

libc/config/linux/header_conf.td

+include "config/public_api.td"
+
+def LinuxStandards : Standards {
+  // On Linux, we support Public API specified by
+  // the C standard library and the POSIX extensions.
+  let Standards = [
+    "stdc",
+    "posix",
+  ];
+}
+
+def FloatT : TypeDecl<"float_t"> {
+  let Decl = [{
+    #if __FLT_EVAL_METHOD__ == 1
+      typedef float float_t
+    #elseif __FLT_EVAL_METHOD__ == 2
+      ...
+    #else
+      ...
+    #endif
+  }]; // This is only an example and not exactly how it will appear
+}
+
+def SizeT : TypeDecl<"size_t"> {
+  let Decl = [{
+    #define __need_size_t
+    #include <stddef.h>
+  }];
+}
+
+def NullMacro : MacroDef<"NULL"> {
+  let Defn = [{
+    #define __need_NULL
+    #include <stddef.h>
+  }];
+}
+
+def Math : PublicAPI<"math.h"> {
+  let Functions = [
+    FunctionDecl<"acos">,
+    FunctionDecl<"acosl">,
+  ];
+
+  let TypeDeclarations = [
+    FloatT,
+  ];
+}
+
+def String : PublicAPI<"string.h"> {
+  let Functions = [
+    FunctionDecl<"strcat">,
+    FunctionDecl<"strcpy">
+  ];
+
+  let TypeDeclarations = [
+    SizeT,
+  ];
+
+  let Macros = [
+    NullMacro,
+  ];
+}
+
+def StdIO : PublicAPI<"stdio.h"> {
+  let TypeDeclarations = [
+    SizeT,
+  ];
+
+  let Functions = [
+    FunctionDecl<
+        "snprintf",
+
+        // The function attribute below is only valid with clang and gcc.
+        // It is included here as an example.
+        "__attribute__((format(__printf__(3, 4))))",
+
+        // The annotations below are only for illustration purpose and are valid
+        // under msvc I think.
+        // Since there are three argument annotations listed here, the function
+        // should take atleast 3 formal arguments including the varargs.
+        // The header generator will check the arg count mismatch if any.
+        [ArgAnn<["_Out_writes_opt_(_BufferCount)", "_Always_(_Post_z_)"], "_Buffer">,
+         ArgAnn<["_In_"], "_BufferCount">,
+         ArgAnn<["_In_z_", "_Printf_format_string_"], "_Format">]
+    >,
+  ];
+}

libc/config/linux/headers.txt

+
+add_libc_headers(
+  llvmlibc_headers
+  math_h
+  string_h
+)

libc/config/linux/library_targets.txt

 
 add_entrypoint_library(
   llvmlibc
   DEPENDS
     ### C standard library entrypoints
     # string.h entrypoints
     strcpy
     strcat
 )

libc/config/public_api.td

+class Standards {
+  list<string> Standards;
+}
+
+class TypeDecl<string name> {
+  string Name = name;
+  string Decl = "";
+}
+
+class MacroDef<string name> {
+  string Name = name;
+  string Defn = "";
+}
+
+class ArgAnn<list<string> annotations, string name = ""> {
+  list<string> Annotations = annotations;
+  string Name = name;
+}
+
+class FunctionDecl<string name, string attr = "", list<ArgAnn> argAnnotations = []> {
+  string Name = name;
+  string Attr = attr;
+  list<ArgAnn> ArgAnnotations = argAnnotations;
+}
+
+class PublicAPI<string header> {
+  list<FunctionDecl> Functions;
+  list<TypeDecl> TypeDeclarations;
+  list<MacroDef> Macros;
+  list<string> Structs;
+}

libc/docs/build_system.rst

 --------------------------------
 
 Standards like POSIX require that a libc provide certain library files like
 ``libc.a``, ``libm.a``, etc. The targets for such library files are listed in
 the ``lib`` directory as ``add_entrypoint_library`` targets. An
 ``add_entrypoint_library`` target  takes a list of ``add_entrypoint_object``
 targets and produces a static library containing the object files corresponding
 to the ``add_entrypoint_targets``.
+
+Rules for header files
+======================
+
+There are two rules for header files. These rules can be used for internal as
+well as public header files. They are explained in detail below:
+
+1. ``add_hdr`` - This rule should be used for public header files whose content
+   is guaranteed to be platform independent and available on all platforms that
+   LLVM libc supports.
+
+2. ``gen_hdr`` - This rule should be used to generate header files from
+   ``.h.def`` files. The format of the ``h.def`` files is explained in the
+   header generation document.
+

libc/docs/ground_truth_specification.rst

+The ground truth of standards
+=============================
+
+Like any modern libc, LLVM libc also supports a wide number of standards and
+extensions. To avoid developing headers, wrappers and sources in a disjointed
+fashion, LLVM libc employs ground truth files. These files live under the
+``spec`` directory and list ground truth corresponding the ISO C standanrd, the
+POSIX extension standard, etc. For example, the path to the ground truth file
+for the ISO C standard is ``spec/stdc.td``. Tools like the header generator
+(described in the header generation document), docs generator, etc. use the
+ground truth files to generate headers, docs etc.

libc/docs/header_generation.rst

 Arguments
 
   None.
 
 Action
 
   The header generator will only include content starting from the line after the
   line on which this command is listed.
+
+``public_api``
+~~~~~~~~~~~~~~
+
+This is a replacement command which should be listed in an input ``.h.def``
+file. The header file generator will replace this command with the public API of
+the target platform. See the build system document for more information on the
+relevant build rules. Also, see "Mechanics of public_api" to learn the mechanics
+of how the header generator replaces this command with the public API.
+
+Arguments
+
+  None.
+
+Action
+
+  The header generator will replace this command with the public API to be exposed
+  from the generated header file.

libc/docs/mechanics_of_public_api.rst

+The mechanics of the ``public_api`` command
+===========================================
+
+The build system, in combination with the header generation mechanism,
+facilitates the fine grained ability to pick and choose the public API one wants
+to expose on their platform. The public header files are always generated from
+the corresponding ``.h.def`` files. A header generation command ``%%public_api``
+is listed in these files. In the generated header file, the header generator
+replaces this command with the public API relevant for the target platform.
+
+Under the hood
+--------------
+
+When the header generator sees the ``%%public_api`` command, it looks up the
+header config file for the platform in the path
+``config/<platform>/header_config.td``. The header config file lists two kinds
+of items:
+
+1. The list of standards from which the public entities available on the platform
+   are derived from.
+2. For each header file exposed on the platfrom, the list of public members
+   provided in that header file.
+
+Note that, the header generator only learns the names of the public entities
+from the header config file (the 2nd item from above.) The exact manner in which
+the entities are to be declared is got from the standards (the 1st item from
+above.)
+
+See the ground truth document for more information on how the standards are
+formally listed in LLVM libc using LLVM table-gen files.

libc/lib/CMakeLists.txt

+# DO NOT list targets directly in this file unless it is guaranteed to be
+# platform independent.
 
-add_entrypoint_library(
-  llvmlibc
-  DEPENDS
-    ### C standard library entrypoints
-    # string.h entrypoints
-    strcpy
-    strcat
-)
+include($LIBC_SOURCE_DIR/config/${LIBC_TARGET_OS}/library_targets.txt)

libc/spec/spec.td

+class Type {}
+
+class NamedType<string name> : Type {
+  string Name = name;
+}
+
+class Field<string name, Type type> {
+  string Name = name;
+  Type FieldType = type;
+}
+
+// Class to describe concrete structs specified in the standard.
+class Struct<string name> : NamedType<name> {
+  list<Field> Fields;
+}
+
+class PtrType<Type type> : Type {
+  Type PointeeType = type;
+}
+
+class ConstType<Type type> : Type {
+  Type UnqualifiedType = type;
+}
+
+class RestrictedPtrType<Type type> : Type {
+  Type PointeeType = type;
+}
+
+// Builtin types.
+def VarArgType : Type {}
+def VoidType : NamedType<"void">;
+def IntType : NamedType<"int">;
+def FloatType : NamedType<"float">;
+def DoubleType : NamedType<"double">;
+def CharType : NamedType<"char">;
+
+class Macro<string name> {
+  string Name = name;
+}
+
+class Function<string name, Type return_type, list<Type> args> {
+  string Name = name;
+  Type ReturnType = return_type;
+  list<Type> Args = args;
+}
+
+class Header<string name> {
+  string Name = name;
+  list<Function> Functions;
+  list<Type> Types;
+  list<Macro> Macros;
+}

libc/spec/stdc.td

+include "spec/spec.td"
+
+def Math : Header<"math.h"> {
+  let Functions = [
+    Function<"acos", DoubleType, [DoubleType]>
+  ];
+
+  let Types = [
+    NamedType<"float_t">,
+    NamedType<"double_t">,
+  ];
+}
+
+def VoidPtr : PtrType<VoidType>;
+def VoidRestrictedPtr : RestrictedPtrType<VoidType>;
+def ConstVoidRestrictedPtr : ConstType<VoidRestrictedPtr>;
+
+def CharPtr : PtrType<CharType>;
+def CharRestrictedPtr : RestrictedPtrType<CharType>;
+def ConstCharRestrictedPtr : ConstType<CharRestrictedPtr>;
+
+def SizeTType : NamedType<"size_t">;
+
+def String : Header<"string.h"> {
+  let Types = [
+    SizeTType,
+  ];
+
+  let Macros = [
+    Macro<"NULL">,
+  ];
+
+  let Functions = [
+    Function<"strcpy", CharPtr, [CharRestrictedPtr, ConstCharRestrictedPtr]>,
+    Function<"memcpy", VoidPtr, [VoidRestrictedPtr, ConstVoidRestrictedPtr, SizeTType]>
+  ];
+}
+
+def StdIO : Header<"stdio.h"> {
+  let Types = [
+    SizeTType,
+  ];
+
+  let Functions = [
+    Function<"snprintf", IntType, [CharPtr, SizeTType, ConstCharRestrictedPtr, VarArgType]>,
+  ];
+}