[libc] Add GPU support for `printf` and `fprintf`

[jhuber6]

Summary

This patch adds support for calling the printf and fprintf functions
from the GPU. The overall goal is to provide an implementation of
printf that is light-weight enough to be used on the GPU without
straining resources or being too slow.

The RPC interface allows for streaming arbitrary bytes from the client to
the server. To implement printf, we first stream the format string to
the server. Then, both sides use an extremely minimal parser to identify
the sizes and number of arguments that will be exchanged. The client will
then use this information to traverse through the variadic list with the
appropriate sizes and send the argument to the server. The server uses
this to build an array of all the argument to printf.

Once the server has these arguments, we can re-use the LLVM C library
handling of printf. With a custom argument list we can instead
traverse this array of pointers instead of a varadic list.

This patch depends on correct codegen and implementation of variadics on
the GPU, which is being handled by @JonChesterfield.

Depends on D158246

[jplehr]

libc/src/__support/arg_list.h

73

Should the argument be const & here and below?

libc/utils/gpu/server/rpc_server.cpp

94

Allocating char here and deleting uint8_t later on.
Is there a particular reason for the type difference?

[arsenm]

libc/src/stdio/gpu/parser.h

60

Why all the ()s

libc/utils/gpu/server/rpc_server.cpp

111

use unique_ptr<>?

[arsenm]

libc/test/integration/startup/gpu/rpc_printf_test.cpp

18

ASSERT_EQ

[arsenm]

libc/src/stdio/gpu/parser.h

91

swap these cases?

[jhuber6]

Addressing comments

[jhuber6]

Improve test

[jhuber6]

Some small updates. I verified the implementation on an Nvidia sm_70 and noted
functionality. The test as written does not work because for some reason it
seems that the Nvidia loader cannot call RPC functions inside of the
constructor. Unsure why, I'll either work around it or fix it.

[jhuber6]

Fix a bug that caused deadlocks if a different number of arguments were passed.
It's very important that the *same* warp / wave mask that opened the port
participates in the sending. So this changes it to always send a size of 0 if
it is not active.

[jhuber6]

Fix printing pointers not working. This required having the libc parser
explicitly use char * when it gets the next argument so we can detect if we
need to treat the array argument as a pointer to a string or a pointer to a
pointer.

[michaelrj]

libc/src/stdio/gpu/parser.h

135

I realize that this was probably for size reasons, but this should probably add back something for parsing flags, widths, and precisions. The current design will accept all valid format specifiers, but won't reject all invalid format specifiers (e.g. %?d).

169

this should be a bit_cast

[jhuber6]

libc/src/stdio/gpu/parser.h

135

I originally had code for that but it added an absurd amount of resource usage that I don't think we really need. The parsing here will definitely pick this up as an argument and copy the argument to the server. We use a real parser there so the actual printing will pick it up, but I guess the number of arguments might be off?

I'm guessing the standard solution is to just ignore it?

[michaelrj]

libc/src/stdio/gpu/parser.h

135

in the case of an invalid format specifier (like %?d) the behavior is undefined. The way we generally handle it is by treating the format specifier (in this case %?) as a raw string. The current micro-parser would mistakenly skip past the ? and assume the format name is d, which is just a letter d in this case. This would cause argument misalignment since there would be an extra int argument where there shouldn't be one.

[jhuber6]

Address comments and enable the printf handling to handle '*' flags and more gracefully handle malformed input.

[jhuber6]

Update to handle malformed inputs better, we now catch things like %1.1.1f.
This does not have a solution to severely malformed input like %****d, but I'm
unsure if we should sacrifice more complexity to catch this case.

[michaelrj]

libc/src/stdio/gpu/file.h

2 ↗

(On Diff #555944)

nit: formatting

libc/src/stdio/gpu/parser.h

104

This should be

if(format[cur_pos] == '.'){
  ++cur_pos;
  while (format[cur_pos] != '\0' && isdigit(format[cur_pos]))
    ++cur_pos;
}

A solo . is an acceptable precision, but ..1 is not.

143

instead of having all of the flag/width/precision parsing in parse_length_modifier wouldn't it be better to put those out here? That would also let you skip those parsing steps when unfinished is set.

[jhuber6]

Address comments

[michaelrj]

libc/src/stdio/gpu/parser.h

132

to handle the *s you could check if format[cur_pos]=='*' after this loop and if true set unfinished and return a specifier with the int.

If you make unfinished an enum with three states - finished, width, and precision - then you can set it to width here and precision after precision. This lets you skip all the loops before the point you're resuming to, which also makes it more accurate by avoiding consuming flags in invalid positions (e.g. the current parser would accept %-*-d since it loops through the flags again after each *).

[jhuber6]

libc/src/stdio/gpu/parser.h

132

I was messing around with something like that, but a lot of complicated control flow really bloats the register count. The first version of this used like 22 and we're already up to 26 so we're well into the bounds of non-trivial. I'm somewhat apprehensive to increase it further, but I can try to see what it costs since I'm not happy with the big FIXME up there.

[michaelrj]

libc/src/stdio/gpu/parser.h

132

is register usage proportional to the number of variables or to the number of branches? I don't know a lot about how GPU code works.

[jhuber6]

libc/src/stdio/gpu/parser.h

135

So it's similar to any other architecture, you have registers that store intermediate values. The way the GPU differs is that we have different kind of registers, and a lot of them. So, because the GPU uses a SIMD architecture at its core, we have a separation between scalar registers and vector registers. These conditions are heavy on the vector registers because they're values that can be divergent across the SIMD group. E.g. we have 32 threads in a "warp" all executing in lock-step. A vector register thus contains 32-values. If we're doing a printf where the format string is different between threads (uncommon but possible) then we'll hit the vector registers harder.

Unrelated, @arsenm, do we have a way to assert that some of these values will be uniform if the format string is uniform? It would probably save us a good amount of vector registers if we knew that this internal state would all be the same.

[jhuber6]

libc/src/stdio/gpu/parser.h

132

I replied to the wrong comment.

Also, would the above description handle a case like %*.**? I think we'd skip past the parsing and then just get the next *.

[michaelrj]

libc/src/stdio/gpu/parser.h

132

the way I'm describing would, I believe, parse %*.** as follows:
% - start the format specifier
* - width is a variable (need to send an int)
. - precision exists
* - precision is a variable (need to send an int)
* - format name is *.

The parser treats the first character that doesn't fit into another category as the format name. Since this should be past the width and precision members the only category left that isn't the format name is length modifier, and * is not valid as a length modifier. Since * isn't valid as a format name this whole format specifier gets marked as raw and is printed literally, though the two variables will be consumed. Needless to say this is very undefined and I wouldn't recommend actually doing it.

As a sidenote, I'd be fascinated to know what NVidia's printf does for printf("%*.** %d %d\n",1,2,3,4);. I've gotten a different answer from every libc I've tried it on.

[jhuber6]

libc/src/stdio/gpu/parser.h

132

I'll sate your curiousity, Nvidia's printf on CUDA 12.2 returns the following: %1* 0 2. My glibc version 2.38 prints %1.2* 3 4. I have no clue what ** is even supposed to do since it doesn't seem to be ignored directly.

[michaelrj]

libc/src/stdio/gpu/parser.h

132

** should be treated like a * followed by any other invalid format name, equivalent to *?. Evidently not all libc implementations handle it cleanly though, I tried it with MacOS's libc and it returns %d 4. I have no idea how CUDA manages to find a 0 in there. From previous experience I believe that glibc prints a format specifier with all of the arguments included when it fails to parse, meaning it will deduplicate flags and convert integer arguments to strings. It's actually pretty cool, if a bit overkill for such an unusual situation. Our libc returns %*.** 3 4 which, is in my opinion, reasonable.

[jhuber6]

libc/src/stdio/gpu/parser.h

132

The Nvidia implementation of printf is a complete mystery to me. It's proprietary so you can't see how it's implemented, but process of elimination suggests that it's a ring buffer that's flushed by the host runtime at predetermined points. They must do some form of parsing on the GPU similar to here, because it handles non-constant strings and divergent values correctly as far as I'm aware. Pretty much the way it works is if your program has an undefined reference to something called printf it will automatically link in a magic implementation that lives somewhere. What's confusing to me is the resource usage. I'm pretty sure that this printf support is some opaque binary blob that isn't taken into account when checking things like register usage. This is because I attempted to compare register usage with my implementation and theirs. This implementation uses something like 64 (which used to be the entire register budget back in the sm_20 days, but we have a lot more now) but checking the Nvidia implementation it states it's something like 23 register, which just so happens to be the exact same results you'd get if you called an empty external function with the same signature. That's about all I've been able to glean from it, it's basically magic. I think they've improved it a lot since I last checked, they used to ignore a lot of flags.

[jhuber6]

Addressing malformed star inputs. This used like five more registers in the average case, but they tended to be scalar so it's a bit less of an issue.

[jhuber6]

libc/src/stdio/gpu/parser.h

132

With this most recent patch I get %*.** 3 4 as the output for your provided most-vexing printf parse.

[michaelrj]

libc/src/stdio/gpu/parser.h

132

That's very strange. We know they have to parse the format string on-device or else the %s conversion wouldn't work. Maybe they have some proprietary hardware trickery.

I'm okay with this parser having some strange behavior on aggressively undefined inputs, though I would recommend adding tests for the parser specifically.

libc/test/integration/startup/gpu/rpc_printf_test.cpp

18

it's likely not important here since this is a very specific test, but we do have a utility (libc_make_test_file_path) for finding the correct folder to write test data to. It's mostly important for bazel since it complains when files write to unexpected places.

73

is it possible to re-open the file in read mode at the end to check that the output strings are what is expected?

[jhuber6]

libc/test/integration/startup/gpu/rpc_printf_test.cpp

73

It's a little annoying here because I need to test this stuff with multiple threads. Meaning that everything's in a random order.

[michaelrj]

libc/src/stdio/gpu/parser.h

132

excellent! The printf parser side looks good to me.

137–141

this check needs to go before the isdigit loop.

[michaelrj]

libc/test/integration/startup/gpu/rpc_printf_test.cpp

73

Would it be possible to have each thread open a file with a different name (e.g. thread 1 opens test_data1.txt)?

[jhuber6]

Moving '*' check above width check.

[jhuber6]

libc/test/integration/startup/gpu/rpc_printf_test.cpp

73

Might be possible in theory. I deliberately chose string sizes that would hopefully be obviously wrong if it wasn't parsed correctly to avoid the extra trouble however. I can try to do it if it's necessary, but I'm wondering if that would be handed by another unit test. The importance of this "integration test" is that the integration tests are the only ones I can run with many threads to make sure it doesn't deadlock or anything.

[michaelrj]

LGTM from the printf side, I don't know enough about GPU coding to review the RPC stuff

libc/test/integration/startup/gpu/rpc_printf_test.cpp

73

ah, fair enough.

[jhuber6]

In D158774#4641360, @michaelrj wrote:

LGTM from the printf side, I don't know enough about GPU coding to review the RPC stuff

I'm actually having second thoughts about this approach. There were basically two options, we either parse it on the CPU and tell the GPU what to do, or we parse it on the GPU directly. I went with the latter because the former requires more cross-device communication (which is the really slow part). But looking at it now, it's basically doubling the amount of time it takes to process an argument (~5us to ~10us) but it does save us a lot of time. It would basically require yet another mock argument parser that goes through the list and encodes some enum for what type to use for next_var() in the va_list. I did a mock-up of this and it's much lighter weight. I would allow us to get rid of the ad-hoc parser I've created though, so you might like that. I could maybe hide some of the latency better... It's a tough choice. I might play around with it some more tomorrow.

[jhuber6]

In D158774#4641394, @jhuber6 wrote:

I'm actually having second thoughts about this approach. There were basically two options, we either parse it on the CPU and tell the GPU what to do, or we parse it on the GPU directly. I went with the latter because the former requires more cross-device communication (which is the really slow part). But looking at it now, it's basically doubling the amount of time it takes to process an argument (~5us to ~10us) but it does save us a lot of time. It would basically require yet another mock argument parser that goes through the list and encodes some enum for what type to use for next_var() in the va_list. I did a mock-up of this and it's much lighter weight. I would allow us to get rid of the ad-hoc parser I've created though, so you might like that. I could maybe hide some of the latency better... It's a tough choice. I might play around with it some more tomorrow.

Just for reference, tried this and didn't get nearly enough of a decrease in resource usage to make it worthwhile.

[arsenm]

libc/src/stdio/gpu/parser.h

55

no else after return

65

no else after return

[jhuber6]

Rebase

[michaelrj]

LGTM

libc/config/gpu/entrypoints.txt

set(TARGET_LIBC_ENTRYPOINTS
 
     # errno.h entrypoints
     libc.src.errno.errno
 
     # stdio.h entrypoints
     libc.src.stdio.feof
     libc.src.stdio.ferror
     libc.src.stdio.clearerr
+    libc.src.stdio.printf
+    libc.src.stdio.fprintf
     libc.src.stdio.snprintf
     libc.src.stdio.vsnprintf
     libc.src.stdio.puts
     libc.src.stdio.fopen
     libc.src.stdio.fclose
     libc.src.stdio.fread
     libc.src.stdio.fputs
     libc.src.stdio.fwrite

libc/include/llvm-libc-types/rpc_opcodes_t.h

typedef enum : unsigned short {
   RPC_CLOSE_FILE = 7,
   RPC_MALLOC = 8,
   RPC_FREE = 9,
   RPC_HOST_CALL = 10,
   RPC_ABORT = 11,
   RPC_FEOF = 12,
   RPC_FERROR = 13,
   RPC_CLEARERR = 14,
+  RPC_PRINTF_TO_STDOUT = 15,
+  RPC_PRINTF_TO_STDERR = 16,
+  RPC_PRINTF_TO_STREAM = 17
 } rpc_opcode_t;
 
 #endif // __LLVM_LIBC_TYPES_RPC_OPCODE_H__

libc/src/__support/arg_list.h

 //===-- Holder Class for manipulating va_lists ------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIBC_SRC___SUPPORT_ARG_LIST_H
 #define LLVM_LIBC_SRC___SUPPORT_ARG_LIST_H
 
+#include "src/__support/CPP/type_traits.h"
 #include "src/__support/common.h"
 
 #include <stdarg.h>
 #include <stddef.h>
 
 namespace __llvm_libc {
 namespace internal {
 
public:
   template <class T> LIBC_INLINE T next_var() {
     ++arg_counter;
     return T(arg_counter);
   }
 
   size_t read_count() const { return arg_counter; }
 };
 
+// Used for the GPU implementation of `printf`. This models a variadic list as a
+// simple array of pointers that are built manually by the implementation.
+class ArrayArgList {
+  size_t idx;
+  void **list;
+
+public:
+  LIBC_INLINE ArrayArgList(void **list) : idx(0), list(list) {}
+  LIBC_INLINE ArrayArgList(va_list) {}
+  LIBC_INLINE ArrayArgList(const ArrayArgList &other) {
+    idx = other.idx;
+    list = other.list;
+  }
+  LIBC_INLINE ~ArrayArgList() = default;
+
+  LIBC_INLINE ArrayArgList &operator=(const ArrayArgList &rhs) {
+    idx = rhs.idx;
+    list = rhs.list;
+    return *this;
+  }
+
+  template <class T> LIBC_INLINE T next_var() {
+    if constexpr (cpp::is_same_v<T, char *>)
+      return reinterpret_cast<T>(list[idx++]);
+    else
+      return *reinterpret_cast<T *>(list[idx++]);
+  }
+}; // namespace __llvm_libc
+
 } // namespace internal
 } // namespace __llvm_libc
 
 #endif // LLVM_LIBC_SRC___SUPPORT_ARG_LIST_H

libc/src/stdio/CMakeLists.txt

 if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${LIBC_TARGET_OS})
   add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/${LIBC_TARGET_OS})
 endif()
 
 if(NOT LIBC_TARGET_ARCHITECTURE_IS_GPU)
   add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/generic)
 endif()
 
+add_subdirectory(printf_core)
+add_subdirectory(scanf_core)
+
 add_entrypoint_object(
   fflush
   SRCS
     fflush.cpp
   HDRS
     fflush.h
   DEPENDS
     libc.src.errno.errno
SRCS
     snprintf.cpp
   HDRS
     snprintf.h
   DEPENDS
     libc.src.stdio.printf_core.printf_main
     libc.src.stdio.printf_core.writer
 )
 
-list(APPEND printf_deps
-      libc.src.__support.arg_list
-      libc.src.stdio.printf_core.vfprintf_internal
-)
-
-if(LIBC_CONF_PRINTF_DISABLE_FLOAT)
-  list(APPEND printf_copts "-DLIBC_COPT_PRINTF_DISABLE_FLOAT")
-endif()
-if(LIBC_CONF_PRINTF_DISABLE_INDEX_MODE)
-  list(APPEND printf_copts "-DLIBC_COPT_PRINTF_DISABLE_INDEX_MODE")
-endif()
-if(LIBC_CONF_PRINTF_DISABLE_WRITE_INT)
-  list(APPEND printf_copts "-DLIBC_COPT_PRINTF_DISABLE_WRITE_INT")
-endif()
-if(LIBC_CONF_PRINTF_FLOAT_TO_STR_USE_MEGA_LONG_DOUBLE_TABLE)
-  list(APPEND printf_copts "-DLIBC_COPT_FLOAT_TO_STR_USE_MEGA_LONG_DOUBLE_TABLE")
-endif()
-
-if(LLVM_LIBC_FULL_BUILD)
-  list(APPEND printf_deps
-      libc.src.__support.File.file
-      libc.src.__support.File.platform_file
-      libc.src.__support.File.platform_stdout
-  )
-else()
-  list(APPEND printf_copts "-DLIBC_COPT_PRINTF_USE_SYSTEM_FILE")
-endif()
-
-add_entrypoint_object(
-  printf
-  SRCS
-    printf.cpp
-  HDRS
-    printf.h
-  DEPENDS
-    ${printf_deps}
-  COMPILE_OPTIONS
-    ${printf_copts}
-)
-
-add_entrypoint_object(
-  fprintf
-  SRCS
-    fprintf.cpp
-  HDRS
-    fprintf.h
-  DEPENDS
-    libc.src.__support.arg_list
-    libc.src.stdio.printf_core.vfprintf_internal
-  COMPILE_OPTIONS
-    ${printf_copts}
-)
-
 add_entrypoint_object(
   vsprintf
   SRCS
     vsprintf.cpp
   HDRS
     vsprintf.h
   DEPENDS
     libc.src.stdio.printf_core.printf_main
     libc.src.stdio.printf_core.writer
 )
 
 add_entrypoint_object(
   vsnprintf
   SRCS
     vsnprintf.cpp
   HDRS
     vsnprintf.h
   DEPENDS
     libc.src.stdio.printf_core.printf_main
     libc.src.stdio.printf_core.writer
 )
 
 add_entrypoint_object(
-  vprintf
-  SRCS
-    vprintf.cpp
-  HDRS
-    vprintf.h
-  DEPENDS
-    ${printf_deps}
-  COMPILE_OPTIONS
-    ${printf_copts}
-)
-
-add_entrypoint_object(
-  vfprintf
-  SRCS
-    vfprintf.cpp
-  HDRS
-    vfprintf.h
-  DEPENDS
-    libc.src.__support.arg_list
-    libc.src.stdio.printf_core.vfprintf_internal
-  COMPILE_OPTIONS
-    ${printf_copts}
-)
-
-add_subdirectory(printf_core)
-add_subdirectory(scanf_core)
-
-add_entrypoint_object(
   ftell
   SRCS
     ftell.cpp
   HDRS
     ftell.h
   DEPENDS
     libc.src.errno.errno
     libc.include.stdio

 add_stdio_entrypoint_object(getc)
 add_stdio_entrypoint_object(getc_unlocked)
 add_stdio_entrypoint_object(getchar)
 add_stdio_entrypoint_object(getchar_unlocked)
 add_stdio_entrypoint_object(fgets)
 add_stdio_entrypoint_object(stdin)
 add_stdio_entrypoint_object(stdout)
 add_stdio_entrypoint_object(stderr)
+add_stdio_entrypoint_object(printf)
+add_stdio_entrypoint_object(fprintf)
+add_stdio_entrypoint_object(vprintf)
+add_stdio_entrypoint_object(vfprintf)

libc/src/stdio/generic/CMakeLists.txt

SRCS
     stderr.cpp
   HDRS
     ../stderr.h
   DEPENDS
     libc.include.stdio
     libc.src.__support.File.file
     libc.src.__support.File.platform_stderr
 )
+
+list(APPEND printf_deps
+      libc.src.__support.arg_list
+      libc.src.stdio.printf_core.vfprintf_internal
+)
+
+if(LIBC_CONF_PRINTF_DISABLE_FLOAT)
+  list(APPEND printf_copts "-DLIBC_COPT_PRINTF_DISABLE_FLOAT")
+endif()
+if(LIBC_CONF_PRINTF_DISABLE_INDEX_MODE)
+  list(APPEND printf_copts "-DLIBC_COPT_PRINTF_DISABLE_INDEX_MODE")
+endif()
+if(LIBC_CONF_PRINTF_DISABLE_WRITE_INT)
+  list(APPEND printf_copts "-DLIBC_COPT_PRINTF_DISABLE_WRITE_INT")
+endif()
+if(LIBC_CONF_PRINTF_FLOAT_TO_STR_USE_MEGA_LONG_DOUBLE_TABLE)
+  list(APPEND printf_copts "-DLIBC_COPT_FLOAT_TO_STR_USE_MEGA_LONG_DOUBLE_TABLE")
+endif()
+
+if(LLVM_LIBC_FULL_BUILD)
+  list(APPEND printf_deps
+      libc.src.__support.File.file
+      libc.src.__support.File.platform_file
+      libc.src.__support.File.platform_stdout
+  )
+else()
+  list(APPEND printf_copts "-DLIBC_COPT_PRINTF_USE_SYSTEM_FILE")
+endif()
+
+add_entrypoint_object(
+  printf
+  SRCS
+    printf.cpp
+  HDRS
+    ../printf.h
+  DEPENDS
+    ${printf_deps}
+  COMPILE_OPTIONS
+    ${printf_copts}
+)
+
+add_entrypoint_object(
+  fprintf
+  SRCS
+    fprintf.cpp
+  HDRS
+    ../fprintf.h
+  DEPENDS
+    libc.src.__support.arg_list
+    libc.src.stdio.printf_core.vfprintf_internal
+  COMPILE_OPTIONS
+    ${printf_copts}
+)
+
+add_entrypoint_object(
+  vprintf
+  SRCS
+    vprintf.cpp
+  HDRS
+    ../vprintf.h
+  DEPENDS
+    ${printf_deps}
+  COMPILE_OPTIONS
+    ${printf_copts}
+)
+
+add_entrypoint_object(
+  vfprintf
+  SRCS
+    vfprintf.cpp
+  HDRS
+    ../vfprintf.h
+  DEPENDS
+    libc.src.__support.arg_list
+    libc.src.stdio.printf_core.vfprintf_internal
+  COMPILE_OPTIONS
+    ${printf_copts}
+)

libc/src/stdio/generic/fprintf.cpp

 //===-- Implementation of fprintf -------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "src/stdio/fprintf.h"
 
 #include "src/__support/File/file.h"
 #include "src/__support/arg_list.h"
 #include "src/stdio/printf_core/vfprintf_internal.h"
 
 #include <stdarg.h>
 #include <stdio.h>
 
 namespace __llvm_libc {
 
 LLVM_LIBC_FUNCTION(int, fprintf,
                    (::FILE *__restrict stream, const char *__restrict format,
                     ...)) {
   va_list vlist;
   va_start(vlist, format);
   internal::ArgList args(vlist); // This holder class allows for easier copying
                                  // and pointer semantics, as well as handling
                                  // destruction automatically.
   va_end(vlist);
   int ret_val = printf_core::vfprintf_internal(stream, format, args);
   return ret_val;
 }
 
 } // namespace __llvm_libc

libc/src/stdio/generic/printf.cpp

 //===-- Implementation of printf --------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "src/stdio/printf.h"
 
 #include "src/__support/File/file.h"
 #include "src/__support/arg_list.h"
 #include "src/stdio/printf_core/vfprintf_internal.h"
 
 #include <stdarg.h>
 #include <stdio.h>
 
 #ifndef LIBC_COPT_PRINTF_USE_SYSTEM_FILE
 #define PRINTF_STDOUT __llvm_libc::stdout
 #else // LIBC_COPT_PRINTF_USE_SYSTEM_FILE
 #define PRINTF_STDOUT ::stdout
 #endif // LIBC_COPT_PRINTF_USE_SYSTEM_FILE
 
 namespace __llvm_libc {
 
 LLVM_LIBC_FUNCTION(int, printf, (const char *__restrict format, ...)) {
   va_list vlist;
   va_start(vlist, format);
   internal::ArgList args(vlist); // This holder class allows for easier copying
                                  // and pointer semantics, as well as handling
                                  // destruction automatically.
   va_end(vlist);
   int ret_val = printf_core::vfprintf_internal(
       reinterpret_cast<::FILE *>(PRINTF_STDOUT), format, args);
   return ret_val;
 }
 
 } // namespace __llvm_libc

libc/src/stdio/generic/vfprintf.cpp

 //===-- Implementation of vfprintf ------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "src/stdio/vfprintf.h"
 
 #include "src/__support/File/file.h"
 #include "src/__support/arg_list.h"
 #include "src/stdio/printf_core/vfprintf_internal.h"
 
 #include <stdarg.h>
 #include <stdio.h>
 
 namespace __llvm_libc {
 
 LLVM_LIBC_FUNCTION(int, vfprintf,
                    (::FILE *__restrict stream, const char *__restrict format,
                     va_list vlist)) {
   internal::ArgList args(vlist); // This holder class allows for easier copying
                                  // and pointer semantics, as well as handling
                                  // destruction automatically.
   int ret_val = printf_core::vfprintf_internal(stream, format, args);
   return ret_val;
 }
 
 } // namespace __llvm_libc

libc/src/stdio/generic/vprintf.cpp

 //===-- Implementation of vprintf -------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "src/stdio/vprintf.h"
 
 #include "src/__support/File/file.h"
 #include "src/__support/arg_list.h"
 #include "src/stdio/printf_core/vfprintf_internal.h"
 
 #include <stdarg.h>
 #include <stdio.h>
 
 #ifndef LIBC_COPT_PRINTF_USE_SYSTEM_FILE
 #define PRINTF_STDOUT __llvm_libc::stdout
 #else // LIBC_COPT_PRINTF_USE_SYSTEM_FILE
 #define PRINTF_STDOUT ::stdout
 #endif // LIBC_COPT_PRINTF_USE_SYSTEM_FILE
 
 namespace __llvm_libc {
 
 LLVM_LIBC_FUNCTION(int, vprintf,
                    (const char *__restrict format, va_list vlist)) {
   internal::ArgList args(vlist); // This holder class allows for easier copying
                                  // and pointer semantics, as well as handling
                                  // destruction automatically.
   int ret_val = printf_core::vfprintf_internal(
       reinterpret_cast<::FILE *>(PRINTF_STDOUT), format, args);
   return ret_val;
 }
 
 } // namespace __llvm_libc

libc/src/stdio/gpu/CMakeLists.txt

+add_entrypoint_object(
+  stdin
+  SRCS
+    stdin.cpp
+  HDRS
+    ../stdin.h
+  DEPENDS
+    libc.include.stdio
+)
+
+add_entrypoint_object(
+  stdout
+  SRCS
+    stdout.cpp
+  HDRS
+    ../stdout.h
+  DEPENDS
+    libc.include.stdio
+)
+
+add_entrypoint_object(
+  stderr
+  SRCS
+    stderr.cpp
+  HDRS
+    ../stderr.h
+  DEPENDS
+    libc.include.stdio
+)
+
 add_header_library(
   gpu_file
   HDRS
     file.h
   DEPENDS
     libc.src.__support.RPC.rpc_client
     libc.src.__support.common
     .stdin
     .stdout
     .stderr
 )
 
+add_header_library(
+  gpu_parser
+  HDRS
+    parser.h
+  DEPENDS
+    libc.src.__support.arg_list
+)
+
+add_header_library(
+  gpu_printf_impl
+  HDRS
+    print_impl.h
+  DEPENDS
+    libc.src.__support.arg_list
+    .parser
+    .gpu_file
+)
+
 add_entrypoint_object(
   feof
   SRCS
     feof.cpp
   HDRS
     ../feof.h
   DEPENDS
     libc.include.stdio
     libc.src.__support.RPC.rpc_client
+    .stdin
+    .stdout
+    .stderr
 )
 
 add_entrypoint_object(
   ferror
   SRCS
     ferror.cpp
   HDRS
     ../ferror.h
   DEPENDS
     libc.include.stdio
     libc.src.__support.RPC.rpc_client
+    .stdin
+    .stdout
+    .stderr
 )
 
 add_entrypoint_object(
   clearerr
   SRCS
     clearerr.cpp
   HDRS
     ../clearerr.h
   DEPENDS
     libc.include.stdio
     libc.src.__support.RPC.rpc_client
+    .stdin
+    .stdout
+    .stderr
 )
 
 add_entrypoint_object(
   fopen
   SRCS
     fopen.cpp
   HDRS
     ../fopen.h
add_entrypoint_object(
   DEPENDS
     libc.include.stdio
     .gpu_file
     .feof
     .ferror
 )
 
 add_entrypoint_object(
-  stdin
-  SRCS
-    stdin.cpp
-  HDRS
-    ../stdin.h
-  DEPENDS
-    libc.include.stdio
-)
-
-add_entrypoint_object(
-  stdout
+  printf
   SRCS
-    stdout.cpp
+    printf.cpp
   HDRS
-    ../stdout.h
+    ../printf.h
   DEPENDS
     libc.include.stdio
 )
 
 add_entrypoint_object(
-  stderr
+  fprintf
   SRCS
-    stderr.cpp
+    fprintf.cpp
   HDRS
-    ../stderr.h
+    ../fprintf.h
   DEPENDS
     libc.include.stdio
+    libc.src.__support.arg_list
+    libc.src.stdio.printf_core.parser
 )

libc/src/stdio/gpu/fprintf.cpp

-//===-- Implementation of fprintf -------------------------------*- C++ -*-===//
+//===-- GPU Implementation of fprintf -------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "src/stdio/fprintf.h"
+#include "src/stdio/gpu/printf_impl.h"
 
-#include "src/__support/File/file.h"
-#include "src/__support/arg_list.h"
-#include "src/stdio/printf_core/vfprintf_internal.h"
-
-#include <stdarg.h>
 #include <stdio.h>
 
 namespace __llvm_libc {
 
 LLVM_LIBC_FUNCTION(int, fprintf,
                    (::FILE *__restrict stream, const char *__restrict format,
                     ...)) {
   va_list vlist;
   va_start(vlist, format);
-  internal::ArgList args(vlist); // This holder class allows for easier copying
-                                 // and pointer semantics, as well as handling
-                                 // destruction automatically.
+  internal::ArgList args(vlist);
   va_end(vlist);
-  int ret_val = printf_core::vfprintf_internal(stream, format, args);
-  return ret_val;
+
+  return static_cast<int>(printf_common(stream, format, args));
 }
 
 } // namespace __llvm_libc

libc/src/stdio/gpu/parser.h

+//===--------------- Printf format parsing for the GPU --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/arg_list.h"
+#include "src/__support/ctype_utils.h"
+#include "src/string/string_utils.h"
+
+namespace __llvm_libc {
+
+namespace gpu {
+
+// These sizes need to be compatible to simplify parsing the lengths.
+static_assert(sizeof(uintptr_t) == sizeof(long) &&
+                  sizeof(uintptr_t) == sizeof(long long) &&
+                  sizeof(uintptr_t) == sizeof(intmax_t) &&
+                  sizeof(uintptr_t) == sizeof(size_t) &&
+                  sizeof(uintptr_t) == sizeof(ptrdiff_t),
+              "Invalid lengths for target");
+
+enum class LengthModifier { none = 0, l = 1 };
+
+enum class SizeArgument { finished = 0, width = 1, precision = 2 };
+
+struct Specifier {
+  uintptr_t raw_value;
+  bool is_string;
+  bool is_empty;
+};
+
+template <typename ArgProvider> struct MicroParser {
+  LIBC_INLINE MicroParser(const char *format, ArgProvider args)
+      : format(format), args(args) {}
+
+  LIBC_INLINE static constexpr bool is_flag(char c) {
+    switch (c) {
+    case ' ':
+    case '-':
+    case '+':
+    case '#':
+    case '0':
+      return true;
+    default:
+      return false;
+    }
+  }
+
+  LIBC_INLINE Specifier get_next_specifier() {
+    Specifier specifier{};
+    // Skip any characters until we reach a control character or the end.
+    while (format[cur_pos] != '\0' && format[cur_pos] != '%' &&
+           size_pos == SizeArgument::finished)
+      ++cur_pos;
+
+    if (format[cur_pos] != '\0')
+      cur_pos++;
+
+    // Skip all characters that aren't related to the length or type.
+    if (size_pos == SizeArgument::finished) {
+      while (format[cur_pos] != '\0' && is_flag(format[cur_pos]))
+        ++cur_pos;
+
+      if (format[cur_pos] == '*') {
+        specifier.raw_value =
+            static_cast<uintptr_t>(args.template next_var<uint32_t>());
+        size_pos = SizeArgument::width;
+        return specifier;
+      }
+
+      while (format[cur_pos] != '\0' && internal::isdigit(format[cur_pos]))
+        ++cur_pos;
+    }
+
+    if (format[cur_pos] == '.' && size_pos != SizeArgument::precision) {
+      ++cur_pos;
+
+      if (format[cur_pos] == '*') {
+        specifier.raw_value =
+            static_cast<uintptr_t>(args.template next_var<uint32_t>());
+        size_pos = SizeArgument::precision;
+        return specifier;
+      }
+
+      while (format[cur_pos] != '\0' && internal::isdigit(format[cur_pos]))
+        ++cur_pos;
+    }
+
+    LengthModifier lm = parse_length_modifier();
+
+    // We use the type and length modifier to access the variadic argument
+    // appropriately. All arguments are promoted to a simple integer.
+    switch (format[cur_pos]) {
+    case 'c':
+      specifier.raw_value =
+          static_cast<uintptr_t>(args.template next_var<uint32_t>());
+      break;
+    case 'd':
+    case 'i':
+    case 'o':
+    case 'x':
+    case 'X':
+    case 'u':
+      if (lm == LengthModifier::none)
+        specifier.raw_value =
+            static_cast<uintptr_t>(args.template next_var<uint32_t>());
+      else
+        specifier.raw_value =
+            static_cast<uintptr_t>(args.template next_var<uint64_t>());
+      break;
+    case 'f':
+    case 'F':
+    case 'e':
+    case 'E':
+    case 'a':
+    case 'A':
+    case 'g':
+    case 'G': {
+      specifier.raw_value =
+          cpp::bit_cast<uintptr_t>(args.template next_var<double>());
+      break;
+    }
+    case 'p':
+      specifier.raw_value =
+          reinterpret_cast<uintptr_t>(args.template next_var<void *>());
+      break;
+    case 's':
+      // Strings require special handling as they cannot simply be promoted.
+      specifier.raw_value =
+          reinterpret_cast<uintptr_t>(args.template next_var<void *>());
+      specifier.is_string = true;
+      break;
+    default:
+      // This was a malformed input or a '%' literal.
+      specifier.is_empty = true;
+      break;
+    }
+
+    size_pos = SizeArgument::finished;
+    return specifier;
+  }
+
+  LIBC_INLINE bool end(const Specifier &cur) const {
+    return format[cur_pos] == '\0' && cur.is_empty;
+  }
+
+  LIBC_INLINE size_t get_size(const Specifier &cur) const {
+    if (cur.is_empty)
+      return 0;
+    else if (cur.is_string)
+      return internal::string_length(reinterpret_cast<char *>(cur.raw_value)) +
+             1;
+    else
+      return sizeof(uintptr_t);
+  }
+
+  LIBC_INLINE const void *get_pointer(const Specifier &cur) const {
+    if (cur.is_string)
+      return reinterpret_cast<void *>(cur.raw_value);
+    else
+      return &cur.raw_value;
+  }
+
+private:
+  LIBC_INLINE LengthModifier parse_length_modifier() {
+    // We are only concerned with whether or not the length specifier is larger
+    // than a regular integer.
+    switch (format[cur_pos]) {
+    case 'l': {
+      if (format[cur_pos + 1] == 'l')
+        ++cur_pos;
+      [[fallthrough]];
+    case 't':
+    case 'j':
+    case 'z':
+      ++cur_pos;
+      return LengthModifier::l;
+    }
+    case 'h': {
+      if (format[cur_pos + 1] == 'h')
+        ++cur_pos;
+      [[fallthrough]];
+    case 'q':
+    case 'L':
+      ++cur_pos;
+      return LengthModifier::none;
+    }
+    default:
+      return LengthModifier::none;
+    };
+    return LengthModifier::none;
+  }
+
+  const char *__restrict const format;
+  ArgProvider args;
+  uint32_t cur_pos = 0;
+  SizeArgument size_pos = SizeArgument::finished;
+};
+
+} // namespace gpu
+
+} // namespace __llvm_libc

libc/src/stdio/gpu/printf.cpp

+//===-- GPU Implementation of printf --------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "src/stdio/printf.h"
+#include "src/stdio/gpu/printf_impl.h"
+
+#include <stdio.h>
+
+namespace __llvm_libc {
+
+LLVM_LIBC_FUNCTION(int, printf, (const char *__restrict format, ...)) {
+  va_list vlist;
+  va_start(vlist, format);
+  internal::ArgList args(vlist);
+  va_end(vlist);
+
+  return static_cast<int>(printf_common(stdout, format, args));
+}
+
+} // namespace __llvm_libc

libc/src/stdio/gpu/printf_impl.h

+//===--------------- Printf format parsing for the GPU --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/RPC/rpc_client.h"
+#include "src/__support/arg_list.h"
+#include "src/stdio/gpu/parser.h"
+
+#include <stdio.h>
+
+namespace __llvm_libc {
+
+template <unsigned opcode>
+LIBC_INLINE uint64_t printf_impl(::FILE *__restrict stream,
+                                 const char *__restrict format,
+                                 internal::ArgList &args) {
+  rpc::Client::Port port = rpc::client.open<opcode>();
+  if constexpr (opcode == RPC_PRINTF_TO_STREAM)
+    port.send([&](rpc::Buffer *buffer) {
+      buffer->data[0] = reinterpret_cast<uintptr_t>(stream);
+    });
+
+  port.send_n(format, internal::string_length(format) + 1);
+
+  uint64_t mask = gpu::get_lane_mask();
+  gpu::MicroParser<internal::ArgList> parser(format, args);
+  for (gpu::Specifier cur = parser.get_next_specifier();
+       gpu::ballot(mask, !parser.end(cur)); cur = parser.get_next_specifier())
+    port.send_n(parser.get_pointer(cur), parser.get_size(cur));
+
+  uint64_t ret = 0;
+  port.recv([&](rpc::Buffer *buffer) {
+    ret = reinterpret_cast<uint64_t *>(buffer->data)[0];
+  });
+  port.close();
+
+  return ret;
+}
+
+LIBC_INLINE uint64_t printf_common(::FILE *__restrict stream,
+                                   const char *__restrict format,
+                                   internal::ArgList &args) {
+  if (stream == stdout)
+    return printf_impl<RPC_PRINTF_TO_STDOUT>(stdout, format, args);
+  else if (stream == stderr)
+    return printf_impl<RPC_PRINTF_TO_STDERR>(stderr, format, args);
+  else
+    return printf_impl<RPC_PRINTF_TO_STREAM>(stream, format, args);
+}
+
+} // namespace __llvm_libc

libc/test/integration/startup/gpu/CMakeLists.txt

 add_custom_target(libc-startup-tests)
 add_dependencies(libc-integration-tests libc-startup-tests)
 
+# Create an output directory for any temporary test files.
+file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/testdata)
+
 add_integration_test(
   startup_args_test
   SUITE libc-startup-tests
   SRCS
     args_test.cpp
   ARGS
     1 2 3
   ENV
add_integration_test(
   startup_rpc_stream_test
   SUITE libc-startup-tests
   SRCS
    rpc_stream_test.cpp
   LOADER_ARGS
    --threads 32
    --blocks 8
 )
+
+add_integration_test(
+  startup_rpc_printf_test
+  SUITE libc-startup-tests
+  SRCS
+   rpc_printf_test.cpp
+  DEPENDS
+    libc.include.stdio
+    libc.src.__support.RPC.rpc_client
+    libc.src.__support.GPU.utils
+    libc.src.stdio.stdout
+    libc.src.stdio.stderr
+    libc.src.stdio.fprintf
+    libc.src.stdio.fopen
+    libc.src.stdio.fclose
+  LOADER_ARGS
+   --threads 32
+   --blocks 4
+)

libc/test/integration/startup/gpu/rpc_printf_test.cpp

+//===-- RPC test to check args to printf ----------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/GPU/utils.h"
+#include "src/stdio/fopen.h"
+#include "src/stdio/fprintf.h"
+#include "test/IntegrationTest/test.h"
+#include <stdint.h>
+
+using namespace __llvm_libc;
+
+FILE *file = __llvm_libc::fopen("testdata/test_data.txt", "w");
+
+// NVPTX requires that the constructor contains the full signature.
+[[gnu::constructor]] void init(int, char **, char **) {
+  file = __llvm_libc::fopen("testdata/test_data.txt", "w");
+}
+
+TEST_MAIN(int argc, char **argv, char **envp) {
+  ASSERT_TRUE(file && "failed to open file");
+
+  int written = 0;
+  written = __llvm_libc::fprintf(file, "A simple string\n");
+  ASSERT_EQ(written, 16);
+
+  written = __llvm_libc::fprintf(file, "%s", "A simple string\n");
+  ASSERT_EQ(written, 16);
+
+  // Check printing a different value with each thread.
+  written = __llvm_libc::fprintf(file, "%8ld\n", gpu::get_thread_id());
+  ASSERT_EQ(written, 9);
+
+  // Check a literal '%' with a bunch of stuff inbetween.
+  written = __llvm_libc::fprintf(file, "%00000%%c", 'c');
+  ASSERT_EQ(written, 2);
+
+  // Check floating point precision and printing.
+  written = __llvm_libc::fprintf(file, "%d%c%.1f\n", 1, 'c', 1.0);
+  ASSERT_EQ(written, 6);
+
+  // Check various length modifiers.
+  written = __llvm_libc::fprintf(file, "%hhd%hd%d%ld%lld%jd%zd%td%.f", '\x1', 1,
+                                 1, 1l, 1ll, 1ll, 1ll, 1ll, 1.1);
+  ASSERT_EQ(written, 9);
+
+  // Check that the server properly handles a divergent number of arguments.
+  const char *str = gpu::get_thread_id() % 2 ? "%s" : "%20ld\n";
+  written = __llvm_libc::fprintf(file, str, "string\n");
+  ASSERT_EQ(written, gpu::get_thread_id() % 2 ? 7 : 21);
+
+  const char *arg = gpu::get_thread_id() % 2 ? "string\n" : "%s";
+  written = __llvm_libc::fprintf(file, arg, "string\n");
+  ASSERT_EQ(written, 7);
+
+  // Check that we correctly ignore some malformed input.
+  written = __llvm_libc::fprintf(file, "%()d %d %d %d", 1, 2, 3);
+  ASSERT_EQ(written, 10);
+  written = __llvm_libc::fprintf(file, "%1.1.d %d", 1);
+  ASSERT_EQ(written, 8);
+
+  // Check that we handle variable widths correctly.
+  written = __llvm_libc::fprintf(file, "%*d%*.*f", 5, 100, 10, 5);
+  ASSERT_EQ(written, 15);
+
+  // Check the values of some sign modifiers.
+  written = __llvm_libc::fprintf(file, "%-10f%++10f% 10f", 1.0, 1.0, 1.0);
+  ASSERT_EQ(written, 30);
+
+  // Check for extremely abused variable width arguments
+  written = __llvm_libc::fprintf(file, "%**d", 1, 2);
+  ASSERT_EQ(written, 4);
+  written = __llvm_libc::fprintf(file, "%**d%6d", 1, 1);
+  ASSERT_EQ(written, 10);
+  written = __llvm_libc::fprintf(file, "%**.**f", 1, 1, 1.0);
+  ASSERT_EQ(written, 7);
+
+  return 0;
+}

libc/test/src/__support/arg_list_test.cpp

long int check_struct_type(int first, ...) {
   __llvm_libc::internal::ArgList args(vlist);
   va_end(vlist);
 
   S s = args.next_var<S>();
   int last = args.next_var<int>();
   return s.c + s.s + s.i + s.l + s.f + s.d + last;
 }
 
+// FIXME: The NVPTX backend doesn't handle by-val struct types correctly.
+#ifndef LIBC_TARGET_ARCH_IS_NVPTX
+
 TEST(LlvmLibcArgListTest, TestStructTypes) {
   S s{'\x1', 2, 3, 4l, 5.0f, 6.0};
   ASSERT_EQ(check_struct_type(0, s, 1), 22l);
 }
 
+#endif
+
 // Test vector extensions from clang.
 #if LIBC_HAS_ATTRIBUTE(ext_vector_type)
 
 using int1 = int __attribute__((ext_vector_type(1)));
 using int2 = int __attribute__((ext_vector_type(2)));
 using int3 = int __attribute__((ext_vector_type(3)));
 using int4 = int __attribute__((ext_vector_type(4)));
 

libc/test/src/stdio/CMakeLists.txt

if(LLVM_LIBC_FULL_BUILD)
   # This is to be used for tests which write to libc's platform streams
   # under full build but write to system-lib's streams otherwise.
   set(hermetic_test_only HERMETIC_TEST_ONLY)
 else()
 # Else in overlay mode they use the system's FILE.
  set(fprintf_test_copts "-DLIBC_COPT_PRINTF_USE_SYSTEM_FILE")
 endif()
 
-add_libc_unittest(
+add_libc_test(
   fprintf_test
   SUITE
     libc_stdio_unittests
   SRCS
     fprintf_test.cpp
   DEPENDS
     libc.src.stdio.fprintf
     ${fprintf_test_deps}
   COMPILE_OPTIONS
     ${fprintf_test_copts}
 )
 
 add_libc_test(
   printf_test
   ${hermetic_test_only}
   SUITE
     libc_stdio_unittests
   SRCS
     printf_test.cpp
   DEPENDS
     libc.src.stdio.printf
+    libc.src.stdio.stdout
+    libc.src.stdio.stderr
 )
 
 add_fp_unittest(
   vsprintf_test
   UNIT_TEST_ONLY
   SUITE
     libc_stdio_unittests
   SRCS

libc/utils/gpu/server/CMakeLists.txt

-add_library(llvmlibc_rpc_server STATIC rpc_server.cpp)
+add_library(llvmlibc_rpc_server
+  STATIC
+  ${LIBC_SOURCE_DIR}/src/stdio/printf_core/writer.cpp
+  ${LIBC_SOURCE_DIR}/src/stdio/printf_core/converter.cpp
+  rpc_server.cpp)
 
 # Include the RPC implemenation from libc.
 target_include_directories(llvmlibc_rpc_server PRIVATE ${LIBC_SOURCE_DIR})
 target_include_directories(llvmlibc_rpc_server PUBLIC ${LIBC_SOURCE_DIR}/include)
 target_include_directories(llvmlibc_rpc_server PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
 
 # Ignore unsupported clang attributes if we're using GCC.
 target_compile_options(llvmlibc_rpc_server PUBLIC
                        $<$<CXX_COMPILER_ID:GNU>:-Wno-attributes>)
 target_compile_definitions(llvmlibc_rpc_server PUBLIC
                            LIBC_NAMESPACE=${LIBC_NAMESPACE})
 
+target_compile_definitions(llvmlibc_rpc_server
+                           PRIVATE
+                           LIBC_COPT_PRINTF_DISABLE_WRITE_INT
+                           LIBC_COPT_PRINTF_DISABLE_INDEX_MODE)
+
 # Install the server and associated header.
 install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/rpc_server.h
         DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/gpu-none-llvm/
         COMPONENT libc)
 install(TARGETS llvmlibc_rpc_server
         ARCHIVE DESTINATION "lib${LLVM_LIBDIR_SUFFIX}"
         COMPONENT libc)

libc/utils/gpu/server/rpc_server.cpp

 //===-- Shared memory RPC server instantiation ------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "rpc_server.h"
 
+#include "src/__support/arg_list.h"
+#include "src/stdio/gpu/parser.h"
+#include "src/stdio/printf_core/converter.h"
+#include "src/stdio/printf_core/parser.h"
+#include "src/stdio/printf_core/writer.h"
+
 #include "src/__support/RPC/rpc.h"
 #include "src/stdio/gpu/file.h"
 #include <atomic>
 #include <cstdio>
 #include <cstring>
 #include <memory>
 #include <mutex>
 #include <unordered_map>
 #include <variant>
 #include <vector>
 
 using namespace __llvm_libc;
 
 static_assert(sizeof(rpc_buffer_t) == sizeof(rpc::Buffer),
               "Buffer size mismatch");
 
 static_assert(RPC_MAXIMUM_PORT_COUNT == rpc::MAX_PORT_COUNT,
               "Incorrect maximum port count");
 
+template <uint32_t lane_size>
+static void handle_printf(typename rpc::Server<lane_size>::Port &port) {
+  uint64_t total_sizes[lane_size] = {0};
+  uint64_t sizes[lane_size] = {0};
+
+  void *format[lane_size] = {nullptr};
+  FILE *files[lane_size] = {nullptr};
+
+  // Get the appropriate output stream to use.
+  if (port.get_opcode() == RPC_PRINTF_TO_STREAM)
+    port.recv([&](rpc::Buffer *buffer, uint32_t id) {
+      files[id] = reinterpret_cast<FILE *>(buffer->data[0]);
+    });
+  else if (port.get_opcode() == RPC_PRINTF_TO_STDOUT)
+    std::fill(files, files + lane_size, stdout);
+  else
+    std::fill(files, files + lane_size, stderr);
+
+  // Recieve the format string from the client.
+  port.recv_n(format, sizes, [&](uint64_t size) { return new char[size]; });
+
+  for (uint32_t lane = 0; lane < lane_size; ++lane)
+    total_sizes[lane] += rpc::align_up(sizes[lane], sizeof(uintptr_t));
+
+  // Parse the formatting string using the same parser the client uses. This
+  // tells us exactly how many packets we need to be recieving from the client.
+  uint32_t num_specifiers = 0;
+  for (uint32_t lane = 0; lane < lane_size; ++lane) {
+    if (!format[lane])
+      continue;
+
+    internal::MockArgList args;
+    uint32_t count = 0;
+    gpu::MicroParser<internal::MockArgList> parser(
+        reinterpret_cast<const char *>(format[lane]), args);
+    for (gpu::Specifier cur = parser.get_next_specifier(); !parser.end(cur);
+         cur = parser.get_next_specifier())
+      count++;
+    num_specifiers = std::max(num_specifiers, count);
+  }
+
+  // Recieve all the arguments from the client and allocate storage for them.
+  std::vector<void *> all_args[lane_size];
+  for (uint32_t i = 0; i < num_specifiers; ++i) {
+    void *args[lane_size] = {nullptr};
+    port.recv_n(args, sizes, [&](uint64_t size) { return new char[size]; });
+
+    for (uint32_t lane = 0; lane < lane_size; ++lane) {
+      if (sizes[lane] > 0)
+        all_args[lane].push_back(args[lane]);
+      total_sizes[lane] += rpc::align_up(sizes[lane], sizeof(uintptr_t));
+    }
+  }
+
+  int results[lane_size] = {0};
+  for (uint32_t lane = 0; lane < lane_size; ++lane) {
+    if (!format[lane])
+      continue;
+
+    // We assume that twice the input length will be enough to fit the string.
+    uint64_t buffer_size = std::max(256ul, 2 * total_sizes[lane]);
+
+    std::unique_ptr<char[]> buffer(new char[buffer_size]);
+    printf_core::WriteBuffer wb(buffer.get(), buffer_size);
+    printf_core::Writer writer(&wb);
+
+    internal::ArrayArgList args(all_args[lane].data());
+    printf_core::Parser<internal::ArrayArgList> parser(
+        reinterpret_cast<const char *>(format[lane]), args);
+
+    // Parse and print the format string using the arguments we copied from the
+    // client.
+    for (printf_core::FormatSection cur_section = parser.get_next_section();
+         !cur_section.raw_string.empty();
+         cur_section = parser.get_next_section()) {
+      if (cur_section.has_conv) {
+        convert(&writer, cur_section);
+      } else {
+        writer.write(cur_section.raw_string);
+      }
+    }
+    results[lane] =
+        fwrite(buffer.get(), 1, writer.get_chars_written(), files[lane]);
+    if (results[lane] != writer.get_chars_written())
+      results[lane] = -1;
+  }
+
+  port.send(
+      [&](rpc::Buffer *buffer, uint32_t id) { buffer->data[0] = results[id]; });
+
+  for (uint32_t lane = 0; lane < lane_size; ++lane) {
+    if (!format[lane])
+      continue;
+
+    delete[] reinterpret_cast<char *>(format[lane]);
+    for (void *ptr : all_args[lane])
+      delete[] reinterpret_cast<uintptr_t *>(ptr);
+  }
+}
+
 // The client needs to support different lane sizes for the SIMT model. Because
 // of this we need to select between the possible sizes that the client can use.
 struct Server {
   template <uint32_t lane_size>
   Server(std::unique_ptr<rpc::Server<lane_size>> &&server)
       : server(std::move(server)) {}
 
   rpc_status_t handle_server(
case RPC_FERROR: {
       break;
     }
     case RPC_CLEARERR: {
       port->recv_and_send([](rpc::Buffer *buffer) {
         clearerr(file::to_stream(buffer->data[0]));
       });
       break;
     }
-    case RPC_NOOP: {
+    case RPC_PRINTF_TO_STDOUT:
+    case RPC_PRINTF_TO_STDERR:
+    case RPC_PRINTF_TO_STREAM:
+      handle_printf<lane_size>(*port);
+      break;
+    case RPC_NOOP:
       port->recv([](rpc::Buffer *) {});
       break;
-    }
     default: {
       auto handler =
           callbacks.find(static_cast<rpc_opcode_t>(port->get_opcode()));
 
       // We error out on an unhandled opcode.
       if (handler == callbacks.end())
         return RPC_STATUS_UNHANDLED_OPCODE;