[libc][hdr-gen] Add special offloading handling for the GPU target

[jhuber6]

Summary

The previous patch in D153794 provided us with the necessary hooks to
wrap special handling required for GPU headers. As discussed in
https://discourse.llvm.org/t/rfc-implementing-gpu-headers-in-the-llvm-c-library,
we need special handling for the GPU headers if we want them to be
included from a standard offloading lanuage such as CUDA, HIP, or
OpenMP. This is important if we with to actually export the GPU libc
as there are the expected users of the interface. The internal
implementation uses a freestanding C++ mode chosen to keep the interface
common.

The requirements summarized for these offloading headers are as follows:

1. The same header must be included by both the host and device as they must agree on definitions
2. We must outline which definitions are provided by the LLVM GPU libc

This means that we need to include the system headers when compiling for
the GPU. Many issues with that are worked around with offloading
language semantics.

It is worth noting that this header mess will not munge the internal
implementation. This is merely a hack required to export these functions
in a way that's compatible with existing offloading languages. The
current use-case for libc is a freestanding mode which can use the
standard headers generated by libc. If we do not want these headers I
have added an option to disable this.

Depends on D153794

[jhuber6]

The extent with which we need to work against system headers that are hostile to being included is unfortunate, but I firmly believe this is the only workable solution for exporting these interfaces to the expected users. The only alternative and sound solution is to provide a complete libc for the host system and GPU and use a unified header that works on both flawlessly. This is a nice goal, but is most likely many, many years away from being possible and would require the user to change their system configuration. This at least has the effect of generating headers that can then replace the ad-hoc ones that exist in clang and can be used as a base for further development.

There are a few ugly points that have not been addressed by this patch, namely

1. If the system header is not compatible with the implemented ABI: E.g. if div_t is not defined in the order quot; rem. For these cases we can statically assert to at least prevent miscompilations, but I think this will be rare
2. System definitions we will need to manually override on the GPU: Something like errno is a macro that's defined by the system headers. If we with to provide this on the GPU we will need a special section to undef it and reset it to ours only for the GPU compilation.
3. A handful of entrypoints use a type that is not C-standard. E.g. the atexit function uses __atexit_handler_t which is defined internally. We will need to replace these, or define it as a workaround for the GPU.

[JonChesterfield]

Including system headers which know nothing about GPU architectures on the GPU causes a lot of grief in the offloading languages.

How necessary is it to put both our headers and the system one in the same translation unit, given that libc is meant to abstract over architectures? If a fundamental goal is bugward compat with glibc, we're probably better off including the glibc headers in whatever hacky way seems fit and providing an implementation shim that makes this work. E.g. for the div_t example, we swap the fields on the way into and out of the libc implementation.

This setup would be
Glibc headers
Header hacks
Llvm libc translation shim
Llvm libc

While the pure, non-glibc-hybrid thing would just use the libc headers directly on the host and the GPU.

[jhuber6]

In D153897#4453408, @JonChesterfield wrote:

Including system headers which know nothing about GPU architectures on the GPU causes a lot of grief in the offloading languages.

Yes, but not including system headers causes an equivalent amount of grief. The main problem is that we don't have nice system headers to use, but that's a far bigger ask than making a shim into what already exists.

How necessary is it to put both our headers and the system one in the same translation unit, given that libc is meant to abstract over architectures? If a fundamental goal is bugward compat with glibc, we're probably better off including the glibc headers in whatever hacky way seems fit and providing an implementation shim that makes this work. E.g. for the div_t example, we swap the fields on the way into and out of the libc implementation.

This setup would be
Glibc headers
Header hacks
Llvm libc translation shim
Llvm libc

While the pure, non-glibc-hybrid thing would just use the libc headers directly on the host and the GPU.

I don't think we could have any other ordering. If we include the libc headers unmodified they will always result in a conflict with the system types, yet we still need them to tell the host / device which values are on the GPU. The approach taken here is to let the system headers define the types and then post-facto declare things on the GPU.

[jhuber6]

Should probably show what one of these generated headers looks like,

#ifndef LLVM_LIBC_STDIO_H
#define LLVM_LIBC_STDIO_H

#include <llvm-libc-macros/gpu-macros.h>

#ifdef __OFFLOADING
#include_next <stdio.h>
#else
#include <__llvm-libc-common.h>
#include <llvm-libc-macros/file-seek-macros.h>
#include <llvm-libc-macros/stdio-macros.h>
#endif

#ifndef __OFFLOADING
#define EOF -1

#define _IONBF 2

#define _IOLBF 1

#define _IOFBF 0

#include <llvm-libc-types/FILE.h>
#include <llvm-libc-types/size_t.h>

#endif

__BEGIN_OPENMP_DECLS
__BEGIN_C_DECLS

int puts(const char *__restrict) __NOEXCEPT __DEVICE;

int fputs(const char *__restrict, FILE *__restrict) __NOEXCEPT __DEVICE;

extern FILE * stdin __DEVICE;
extern FILE * stdout __DEVICE;
extern FILE * stderr __DEVICE;

__END_C_DECLS
__END_OPENMP_DECLS

#endif // LLVM_LIBC_STDIO_H

[jhuber6]

So, these headers need the ability to do some potentially ugly hacks. I have
added an extra field to the PublicAPI called ExtraDefn, which is currently a
simple string that we can use to inject some special-case handling for the
wrapper headers. In this case, we needed to define the __atexithander_t type
for the atexit function. In general this can be used as a general purpose shim
to address edge cases.

It's not pretty, but as I've stated I think this is the only sound solution to
this problem short of fully implementing the LLVM C library for both the host
and device.

The generated headers still pass the test suite as internally nothing has
changed. Externally, I can now safely include these headers inside OpenMP
without the prior errors.

[sivachandra]

In D153897#4454578, @jhuber6 wrote:

So, these headers need the ability to do some potentially ugly hacks. I have
added an extra field to the PublicAPI called ExtraDefn, which is currently a
simple string that we can use to inject some special-case handling for the
wrapper headers. In this case, we needed to define the __atexithander_t type
for the atexit function. In general this can be used as a general purpose shim
to address edge cases.

It's not pretty, but as I've stated I think this is the only sound solution to
this problem short of fully implementing the LLVM C library for both the host
and device.

I am still not convinced about the "soundness" of the solution. I am on vacation but will try to spend more time on this tomorrow (late evening/night US west coast time) so that we can move forward on this in whatever way. I think the RFC is missing a comparison of alternatives. If you can describe the potential alternative approaches and their pros and cons, it will become easier for me (and others I am sure) to understand (and may be even concur). One of the alternatives that I had mentioned previously is separating out the offloading language use case and standalone GPU use case. IIRC, you wanted to combine them as a matter of convenience - I think you should describe that in the RFC for others benefit (and at the least for the sake of record).

[jhuber6]

In D153897#4454762, @sivachandra wrote:

I am still not convinced about the "soundness" of the solution. I am on vacation but will try to spend more time on this tomorrow (late evening/night US west coast time) so that we can move forward on this in whatever way. I think the RFC is missing a comparison of alternatives. If you can describe the potential alternative approaches and their pros and cons, it will become easier for me (and others I am sure) to understand (and may be even concur). One of the alternatives that I had mentioned previously is separating out the offloading language use case and standalone GPU use case. IIRC, you wanted to combine them as a matter of convenience - I think you should describe that in the RFC for others benefit (and at the least for the sake of record).

Thanks a lot for looking into this admittedly weird and unfortunate quirk of offloading languages and sorry to disturb your vacation. Yes, this presents a lot of unique difficulties because we are trying to provide a library for something like a kind-of-hosted-but-not-really interface. I don't think separating the headers is a big distinction, it's the same approach in the end. Whether we have a single header that ifdefs it off, or two headers that we conditionally include, it'll be the same to the preprocessor at the end of the day. I went with a single header because it is easier to implement and use since the existing workflow only creates one header and I didn't see a compelling reason to separate them given the extra work.

For alternative approaches, I've mentioned before that if we had a unified environment all of this can go away save for the device declarations. That's a solution but is a much, much bigger ask at this point in time. Another alternative would be to only export the functions via the LLVM-libc and include that in a separate wrapper header, maybe in clang/lib/Headers/__, e.g. like the following

#include_next <stdlib.h>

#define __DEVICE __attribute__((device))

#pragma omp begin declare target
#include <__llvm_libc_declarations/stdlib.h>
#pragma omp end declare target

Where that file could look like

#ifndef __DEVICE
#define __DEVICE
#endif

int abs(int) __DEVICE;
...

This would have the benefit of keeping CUDA / OpenMP weirdness out of libc and we'd be more or less free to hack our way to victory without anyone in libc smelling it. The downside is that we would still need a few workarounds, e.g. for something like errno we'd need to copy the declaration from libc, or with atexit we would need to manually define the __atexit_handler_t somewhere. Doing this would basically need a special operation to libc-hdrgen that only emits the entrypoints and gets rid of the other stuff, so we might end up still needing the %%begin and %%end mess unless we want to parse it more completely.

[JonChesterfield]

The offloading languages do some things with libc which seem broadly reasonable but might not be optimal. The reasoning goes roughly - the host part will probably use glibc, so the gpu part should use the same ABI as glibc, so the gpu should include the glibc headers. Unfortunately the glibc headers don't know anything about the GPU so there's a cascade of hacks to make that work which have been hard won.

I don't think this is totally sensible. For one thing, using the glibc ABI on the gpu has rather limited scope since we can't make normal function calls across that barrier by one arch emitting a jump or call. I think all that matters is agreeing on struct layout in case they get copied across. A defining characteristic of libc is that we know what is supposed to be in it, and of glibc in particular is the ABI is stable, versioned and totally documented. So the GPU could use the same struct definitions as glibc without including glibc headers. It seems unimportant to me that the two sides have exact agreement on which functions are constexpr or not.

Nevertheless, openmp/cuda/hip presently include glibc headers directly on the gpu and patch around it. The dev teams involved seem committed to this strategy so I suggest we leave it in place.

I'd therefore like us to localise whatever header related hacks the offloading languages currently use where they are at present, i.e. in the overlay stuff in clang/lib/headers, and have absolutely none of it anywhere in libc. That allows the offloading languages to add or remove complexity, diverge from one another or converge as they see fit. This totally isolates libc from any adverse behaviour that results from those changes.

[jdoerfert]

In D153897#4456096, @JonChesterfield wrote:

Nevertheless, openmp/cuda/hip presently include glibc headers directly on the gpu and patch around it. The dev teams involved seem committed to this strategy so I suggest we leave it in place.

The history of this is long. If we want to avoid system headers we run into various problems, including but not limited:
Missing extensions, missing/different macros, different function pointer value, missing includes (as a header on the path provides a function or an include that it doesn't have to according to the standard).
During the time we provided math functions w/o system headers we had a lot of complaints, lots of people found it pretty unusable.
We might be able to work around those issue with our libc headers, but it is uncharted territory and I foresee problems. I'd be happy to try, but rather have a working fallback fist.

I'd therefore like us to localise whatever header related hacks the offloading languages currently use where they are at present, i.e. in the overlay stuff in clang/lib/headers, and have absolutely none of it anywhere in libc. That allows the offloading languages to add or remove complexity, diverge from one another or converge as they see fit. This totally isolates libc from any adverse behaviour that results from those changes.

Localizing the hacks seems reasonable. Putting it in clang/lib/headers might not be the best idea though but I won't fight it. I was hoping to have it self contained in libc, but again, that is debatable.

In D153897#4453451, @jhuber6 wrote:

In D153897#4453408, @JonChesterfield wrote:

While the pure, non-glibc-hybrid thing would just use the libc headers directly on the host and the GPU.

I don't think we could have any other ordering. If we include the libc headers unmodified they will always result in a conflict with the system types, yet we still need them to tell the host / device which values are on the GPU. The approach taken here is to let the system headers define the types and then post-facto declare things on the GPU.

I agree, just libc headers will clash. That is why our current headers are annotated properly to make sure they will work well with host headers.
It is the curse of offloading that you need to keep the user program running the way it was, and make it work in the same build configuration, or people will take away it does not work.

[jhuber6]

In D153897#4456220, @jdoerfert wrote:

Localizing the hacks seems reasonable. Putting it in clang/lib/headers might not be the best idea though but I won't fight it. I was hoping to have it self contained in libc, but again, that is debatable.

I'll mirror a comment I made on the original RFC what the alternate approach will look like.

#ifndef __ATTRS
#define __ATTRS
#endif

int fputs(const char *__restrict, FILE *__restrict) __ATTRS;
extern FILE * stderr __ATTRS;

This would then be included by a wrapper header in clang/ilb/Headers/__libc_wrappers/stdio.h which looks like the following,

#ifndef __CLANG_LIBC_WRAPPERS_STDIO_H__
#define __CLANG_LIBC_WRAPPERS_STDIO_H__
#include_next <stdio.h>

#if defined(__CUDA__) || defined(__HIP__)
#define __ATTRS __attribiute__((device))
#endif

#pragma omp begin declare target
#include <llvm-gpu-none/llvm-libc-declarations/stdlib.h>
#pragma omp end declare target

#endif

The downside is that we don't get these nice unified headers, but I think with the extent of hacks for each language it may be better to shuttle that next to where clang is. This would pretty much be a separate mode for libc-hdrgen that just spits out the entrypoints, so it's much less intrusive to libc and it gives us a hook that we can add whatever workarounds we'll require for the offloading case. I think I'm favoring this approach having thought through it, because in my head I can already imagine a large number of #undefs we'll need to do, and I don't think we should mash those into the libc implementation itself.

I'll try to make a new patch to illustrate this after lunch that's hopefully agreeable with everyone involved. Getting this done would be a major milestone in being able to ship the libc so I'm eager to make progress. So thanks for everyone's continued involvement.

libc/cmake/modules/LLVMLibCHeaderRules.cmake

function(add_gen_header target_name)
   list(TRANSFORM ENTRYPOINT_NAME_LIST_ARG PREPEND "--e=")
 
   if(LIBC_HDRGEN_EXE)
     set(hdrgen_exe ${LIBC_HDRGEN_EXE})
   else()
     set(hdrgen_exe ${LIBC_TABLEGEN_EXE})
     set(hdrgen_deps "${LIBC_TABLEGEN_EXE};${LIBC_TABLEGEN_TARGET}")
   endif()
+  option(LIBC_GPU_OFFLOAD_HEADERS "Create offloading headers for the GPU" ON)
+  if(LIBC_TARGET_ARCHITECTURE_IS_GPU AND LIBC_GPU_OFFLOAD_HEADERS)
+    set(offloading "--offloading")
+  else()
+    set(offloading "")
+  endif()
   add_custom_command(
     OUTPUT ${out_file}
     COMMAND ${hdrgen_exe} -o ${out_file} --header ${ADD_GEN_HDR_GEN_HDR}
             --def ${in_file} ${replacement_params} -I ${LIBC_SOURCE_DIR}
-           ${ENTRYPOINT_NAME_LIST_ARG}
+           ${ENTRYPOINT_NAME_LIST_ARG} ${offloading}
            ${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/api.td
 
     WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
     DEPENDS ${in_file} ${fq_data_files} ${td_includes}
             ${LIBC_SOURCE_DIR}/config/${LIBC_TARGET_OS}/api.td
             ${hdrgen_deps}
   )
 

libc/config/gpu/api.td

 include "config/public_api.td"
 
 include "spec/stdc.td"
 include "spec/gpu_ext.td"
 
 def StringAPI : PublicAPI<"string.h"> {
   let Types = ["size_t"];
 }
 
 def StdlibAPI : PublicAPI<"stdlib.h"> {
   let Types = [
     "size_t",
     "__atexithandler_t",
   ];
+
+  // The offloading wrapper headers need this type defined.
+  let ExtraDefn = [{
+    #include <llvm-libc-types/__atexithandler_t.h>
+  }];
 }
 
 def FenvAPI: PublicAPI<"fenv.h"> {
   let Types = ["fenv_t"];
 }
 
 def StdIOAPI : PublicAPI<"stdio.h"> {
   let Macros = [
     SimpleMacroDef<"_IOFBF", "0">,
     SimpleMacroDef<"_IOLBF", "1">,
     SimpleMacroDef<"_IONBF", "2">,
     SimpleMacroDef<"EOF", "-1">,
   ];
   let Types = ["size_t", "FILE"];
 }

libc/config/public_api.td

 class PublicAPI<string name> {
   string HeaderName = name;
   list<MacroDef> Macros = [];
   list<string> Types = [];
   list<string> Enumerations = [];
   list<string> Structs = [];
   list<string> Functions = [];
   list<string> Objects = [];
+  string ExtraDefn = "";
 }

libc/include/CMakeLists.txt

if(LIBC_TARGET_ARCHITECTURE_IS_GPU)
 
   add_gen_header(
     gpu_rpc
     DEF_FILE gpu/rpc.h.def
     GEN_HDR gpu/rpc.h
     DEPENDS
       .llvm_libc_common_h
       .llvm-libc-types.rpc_opcodes_t
+      .llvm-libc-macros.gpu_macros
   )
 endif()
 
 if(NOT LLVM_LIBC_FULL_BUILD)
   # We don't install headers in non-fullbuild mode.
   return()
 endif()
 

libc/include/llvm-libc-macros/CMakeLists.txt

DEPENDS
     .linux.unistd_macros
 )
 
 add_header(
   wchar_macros
   HDR
     wchar-macros.h
 )
+
+add_header(
+  gpu_macros
+  HDR
+    gpu-macros.h
+)

libc/include/llvm-libc-macros/gpu-macros.h

+//===-- GPU definitions for LLVM-libc public offloading header files ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_MACROS_GPU_MACROS_H
+#define LLVM_LIBC_MACROS_GPU_MACROS_H
+
+#undef __OFFLOADING
+#if defined(_OPENMP) || defined(__HIP__) || defined(__CUDA__)
+#define __OFFLOADING
+#endif
+
+#undef __BEGIN_OPENMP_DECLS
+#ifdef _OPENMP
+#define __BEGIN_OPENMP_DECLS _Pragma("omp begin declare target");
+#else
+#define __BEGIN_OPENMP_DECLS
+#endif
+
+#undef __END_OPENMP_DECLS
+#ifdef _OPENMP
+#define __END_OPENMP_DECLS _Pragma("omp end declare target");
+#else
+#define __END_OPENMP_DECLS
+#endif
+
+#undef __DEVICE
+#if defined(__HIP__) || defined(__CUDA__)
+#define __DEVICE __attribute__((device))
+#else
+#define __DEVICE
+#endif
+
+#ifdef __OFFLOADING
+#undef __NOEXCEPT
+#define __NOEXCEPT
+#endif
+
+#ifdef __cplusplus
+
+#undef __BEGIN_C_DECLS
+#define __BEGIN_C_DECLS extern "C" {
+
+#undef __END_C_DECLS
+#define __END_C_DECLS }
+
+#else
+
+#undef __BEGIN_C_DECLS
+#define __BEGIN_C_DECLS
+
+#undef __END_C_DECLS
+#define __END_C_DECLS
+
+#endif
+
+#endif // LLVM_LIBC_MACROS_GPU_MACROS_H

libc/utils/HdrGen/BeginCommand.h

 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 
 #include <string>
 
 namespace llvm_libc {
 
 class BeginCommand : public Command {
+  const bool Offloading;
+
 public:
   static const char Name[];
 
+  BeginCommand(bool Offloading) : Offloading(Offloading) {}
+
   void run(llvm::raw_ostream &OS, const ArgVector &Args,
            llvm::StringRef StdHeader, llvm::RecordKeeper &Records,
            const Command::ErrorReporter &Reporter) const override;
 };
 
 } // namespace llvm_libc
 
 #endif // LLVM_LIBC_UTILS_HDRGEN_BEGIN_COMMAND_H

libc/utils/HdrGen/BeginCommand.cpp

 
 const char BeginCommand::Name[] = "begin";
 
 void BeginCommand::run(llvm::raw_ostream &OS, const ArgVector &Args,
                        llvm::StringRef StdHeader, llvm::RecordKeeper &Records,
                        const Command::ErrorReporter &Reporter) const {
   if (Args.size() != 0)
     Reporter.printFatalError("%%begin command does not take any arguments.");
+
+  if (!Offloading)
+    return;
+
+  // Offloading languages must have the same headers included from both the host
+  // and device compilations. For this reason, if the user has an offloading
+  // language enabled we need to defer to the system header.
+  OS << "#include <llvm-libc-macros/gpu-macros.h>\n\n"
+     << "#ifdef __OFFLOADING\n"
+     << "#include_next <" << StdHeader << ">\n"
+     << "#else\n";
 }
 
 } // namespace llvm_libc

libc/utils/HdrGen/EndCommand.h

 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 
 #include <string>
 
 namespace llvm_libc {
 
 class EndCommand : public Command {
+  const bool Offloading;
+
 public:
   static const char Name[];
 
+  EndCommand(bool Offloading) : Offloading(Offloading) {}
+
   void run(llvm::raw_ostream &OS, const ArgVector &Args,
            llvm::StringRef StdHeader, llvm::RecordKeeper &Records,
            const Command::ErrorReporter &Reporter) const override;
 };
 
 } // namespace llvm_libc
 
 #endif // LLVM_LIBC_UTILS_HDRGEN_END_COMMAND_H

libc/utils/HdrGen/EndCommand.cpp

 
 const char EndCommand::Name[] = "end";
 
 void EndCommand::run(llvm::raw_ostream &OS, const ArgVector &Args,
                      llvm::StringRef StdHeader, llvm::RecordKeeper &Records,
                      const Command::ErrorReporter &Reporter) const {
   if (Args.size() != 0)
     Reporter.printFatalError("%%end command does not take any arguments.");
+
+  if (!Offloading)
+    return;
+
+  OS << "#endif\n";
 }
 
 } // namespace llvm_libc

libc/utils/HdrGen/Generator.h

 namespace llvm_libc {
 
 class Command;
 
 class Generator {
   llvm::StringRef HeaderDefFile;
   const std::vector<std::string> &EntrypointNameList;
   llvm::StringRef StdHeader;
+  const bool Offloading;
   std::unordered_map<std::string, std::string> &ArgMap;
 
   std::unique_ptr<Command> BeginCmd;
   std::unique_ptr<Command> EndCmd;
   std::unique_ptr<Command> IncludeFileCmd;
   std::unique_ptr<Command> PublicAPICmd;
 
   Command *getCommandHandler(llvm::StringRef CommandName);
 
   void parseCommandArgs(llvm::StringRef ArgStr, ArgVector &Args);
 
   void printError(llvm::StringRef Msg);
 
 public:
   Generator(const std::string &DefFile, const std::vector<std::string> &EN,
-            const std::string &Header,
+            const std::string &Header, const bool Offloading,
             std::unordered_map<std::string, std::string> &Map)
       : HeaderDefFile(DefFile), EntrypointNameList(EN), StdHeader(Header),
-        ArgMap(Map) {}
+        Offloading(Offloading), ArgMap(Map) {}
 
   void generate(llvm::raw_ostream &OS, llvm::RecordKeeper &Records);
 };
 
 } // namespace llvm_libc
 
 #endif // LLVM_LIBC_UTILS_HDRGEN_GENERATOR_H

libc/utils/HdrGen/Generator.cpp

 static const size_t ParamNameSuffixSize =
     llvm::StringRef(ParamNameSuffix).size();
 
 namespace llvm_libc {
 
 Command *Generator::getCommandHandler(llvm::StringRef CommandName) {
   if (CommandName == BeginCommand::Name) {
     if (!BeginCmd)
-      BeginCmd = std::make_unique<BeginCommand>();
+      BeginCmd = std::make_unique<BeginCommand>(Offloading);
     return BeginCmd.get();
   } else if (CommandName == EndCommand::Name) {
     if (!EndCmd)
-      EndCmd = std::make_unique<EndCommand>();
+      EndCmd = std::make_unique<EndCommand>(Offloading);
     return EndCmd.get();
   } else if (CommandName == IncludeFileCommand::Name) {
     if (!IncludeFileCmd)
       IncludeFileCmd = std::make_unique<IncludeFileCommand>();
     return IncludeFileCmd.get();
   } else if (CommandName == PublicAPICommand::Name) {
     if (!PublicAPICmd)
-      PublicAPICmd = std::make_unique<PublicAPICommand>(EntrypointNameList);
+      PublicAPICmd =
+          std::make_unique<PublicAPICommand>(EntrypointNameList, Offloading);
     return PublicAPICmd.get();
   } else {
     return nullptr;
   }
 }
 
 void Generator::parseCommandArgs(llvm::StringRef ArgStr, ArgVector &Args) {
   if (!ArgStr.contains(',') && ArgStr.trim(' ').trim('\t').size() == 0) {

libc/utils/HdrGen/Main.cpp

 llvm::cl::list<std::string> EntrypointNamesOption(
     "e", llvm::cl::value_desc("<list of entrypoints>"),
     llvm::cl::desc(
         "Each --e is one entrypoint (generated from entrypoints.txt)"),
     llvm::cl::OneOrMore);
 llvm::cl::list<std::string> ReplacementValues(
     "args", llvm::cl::desc("Command separated <argument name>=<value> pairs."),
     llvm::cl::value_desc("<name=value>[,name=value]"));
+llvm::cl::opt<bool> Offloading(
+    "offloading",
+    llvm::cl::desc("Create headers suitable for offloading languages"));
 
 void ParseArgValuePairs(std::unordered_map<std::string, std::string> &Map) {
   for (std::string &R : ReplacementValues) {
     auto Pair = llvm::StringRef(R).split('=');
     Map[std::string(Pair.first)] = std::string(Pair.second);
   }
 }
 
 } // anonymous namespace
 
 namespace llvm_libc {
 
 bool HeaderGeneratorMain(llvm::raw_ostream &OS, llvm::RecordKeeper &Records) {
   std::unordered_map<std::string, std::string> ArgMap;
   ParseArgValuePairs(ArgMap);
-  Generator G(HeaderDefFile, EntrypointNamesOption, StandardHeader, ArgMap);
+  Generator G(HeaderDefFile, EntrypointNamesOption, StandardHeader, Offloading,
+              ArgMap);
   G.generate(OS, Records);
 
   return false;
 }
 
 } // namespace llvm_libc
 
 int main(int argc, char *argv[]) {
   llvm::cl::ParseCommandLineOptions(argc, argv);
   return TableGenMain(argv[0], &llvm_libc::HeaderGeneratorMain);
 }

libc/utils/HdrGen/PublicAPICommand.h

 } // namespace llvm
 
 namespace llvm_libc {
 
 class PublicAPICommand : public Command {
 private:
   const std::vector<std::string> &EntrypointNameList;
 
+  const bool Offloading;
+
 public:
   static const char Name[];
 
-  PublicAPICommand(const std::vector<std::string> &EntrypointNames)
-      : EntrypointNameList(EntrypointNames) {}
+  PublicAPICommand(const std::vector<std::string> &EntrypointNames,
+                   const bool Offloading)
+      : EntrypointNameList(EntrypointNames), Offloading(Offloading) {}
 
   void run(llvm::raw_ostream &OS, const ArgVector &Args,
            llvm::StringRef StdHeader, llvm::RecordKeeper &Records,
            const Command::ErrorReporter &Reporter) const override;
 };
 
 } // namespace llvm_libc
 
 #endif // LLVM_LIBC_UTILS_HDRGEN_PUBLICAPICOMMAND_H

libc/utils/HdrGen/PublicAPICommand.cpp

static std::string getTypeHdrName(const std::string &Name) {
   llvm::SmallVector<llvm::StringRef> Parts;
   llvm::SplitString(llvm::StringRef(Name), Parts);
   return llvm::join(Parts.begin(), Parts.end(), "_");
 }
 
 namespace llvm_libc {
 
 void writeAPIFromIndex(APIIndexer &G,
-                       std::vector<std::string> EntrypointNameList,
-                       llvm::raw_ostream &OS) {
+                       const std::vector<std::string> &EntrypointNameList,
+                       const bool Offloading, llvm::raw_ostream &OS) {
+  if (Offloading)
+    OS << "#ifndef __OFFLOADING\n";
+
   for (auto &Pair : G.MacroDefsMap) {
     const std::string &Name = Pair.first;
     if (G.MacroSpecMap.find(Name) == G.MacroSpecMap.end())
       llvm::PrintFatalError(Name + " not found in any standard spec.\n");
 
     llvm::Record *MacroDef = Pair.second;
     dedentAndWrite(MacroDef->getValueAsString("Defn"), OS);
 
if (Value == "__default__") {
       OS << ",\n";
     } else {
       OS << " = " << Value << ",\n";
     }
   }
   if (G.Enumerations.size() != 0)
     OS << "};\n\n";
 
+  if (Offloading) {
+    if (!G.ExtraDefn.empty()) {
+      OS << "#else\n";
+      dedentAndWrite(G.ExtraDefn, OS);
+    }
+    OS << "#endif\n\n";
+  }
+
+  if (Offloading)
+    OS << "__BEGIN_OPENMP_DECLS\n";
   OS << "__BEGIN_C_DECLS\n\n";
   for (auto &Name : EntrypointNameList) {
     if (G.FunctionSpecMap.find(Name) == G.FunctionSpecMap.end()) {
       continue; // Functions that aren't in this header file are skipped as
                 // opposed to erroring out because the list of functions being
                 // iterated over is the complete list of functions with
                 // entrypoints. Thus this is filtering out the functions that
                 // don't go to this header file, whereas the other, similar
for (auto &Name : EntrypointNameList) {
     auto ArgsList = FunctionSpec->getValueAsListOfDefs("Args");
     for (size_t i = 0; i < ArgsList.size(); ++i) {
       llvm::Record *ArgType = ArgsList[i]->getValueAsDef("ArgType");
       OS << G.getTypeAsString(ArgType);
       if (i < ArgsList.size() - 1)
         OS << ", ";
     }
 
+    if (Offloading)
+      OS << ") __NOEXCEPT __DEVICE;\n\n";
+    else
       OS << ") __NOEXCEPT;\n\n";
   }
 
   // Make another pass over entrypoints to emit object declarations.
   for (const auto &Name : EntrypointNameList) {
     if (G.ObjectSpecMap.find(Name) == G.ObjectSpecMap.end())
       continue;
     llvm::Record *ObjectSpec = G.ObjectSpecMap[Name];
     auto Type = ObjectSpec->getValueAsString("Type");
+    if (Offloading)
+      OS << "extern " << Type << " " << Name << " __DEVICE;\n";
+    else
       OS << "extern " << Type << " " << Name << ";\n";
   }
   OS << "__END_C_DECLS\n";
+  if (Offloading)
+    OS << "__END_OPENMP_DECLS\n";
 }
 
 void writePublicAPI(llvm::raw_ostream &OS, llvm::RecordKeeper &Records) {}
 
 const char PublicAPICommand::Name[] = "public_api";
 
 void PublicAPICommand::run(llvm::raw_ostream &OS, const ArgVector &Args,
                            llvm::StringRef StdHeader,
                            llvm::RecordKeeper &Records,
                            const Command::ErrorReporter &Reporter) const {
   if (Args.size() != 0) {
     Reporter.printFatalError("public_api command does not take any arguments.");
   }
 
   APIIndexer G(StdHeader, Records);
-  writeAPIFromIndex(G, EntrypointNameList, OS);
+  writeAPIFromIndex(G, EntrypointNameList, Offloading, OS);
 }
 
 } // namespace llvm_libc

libc/utils/LibcTableGenUtil/APIIndexer.h

public:
   std::unordered_map<std::string, std::string> ObjectToHeaderMap;
 
   NameSet RequiredTypes;
   NameSet Structs;
   NameSet Enumerations;
   NameSet Functions;
   NameSet Objects;
   NameSet PublicHeaders;
+  llvm::StringRef ExtraDefn;
 
   std::string getTypeAsString(llvm::Record *TypeRecord);
 };
 
 } // namespace llvm_libc
 
 #endif // LLVM_LIBC_UTILS_LIBC_TABLE_GEN_UTILS_API_INDEXER_H

libc/utils/LibcTableGenUtil/APIIndexer.cpp

void APIIndexer::indexPublicAPIDef(llvm::Record *PublicAPI) {
 
   auto EnumerationList = PublicAPI->getValueAsListOfStrings("Enumerations");
   for (llvm::StringRef EnumerationName : EnumerationList)
     Enumerations.insert(std::string(EnumerationName));
 
   auto ObjectList = PublicAPI->getValueAsListOfStrings("Objects");
   for (llvm::StringRef ObjectName : ObjectList)
     Objects.insert(std::string(ObjectName));
+
+  ExtraDefn = PublicAPI->getValueAsString("ExtraDefn");
 }
 
 void APIIndexer::index(llvm::RecordKeeper &Records) {
   NamedTypeClass = Records.getClass(NamedTypeClassName);
   PtrTypeClass = Records.getClass(PtrTypeClassName);
   RestrictedPtrTypeClass = Records.getClass(RestrictedPtrTypeClassName);
   StructClass = Records.getClass(StructTypeClassName);
   ConstTypeClass = Records.getClass(ConstTypeClassName);