[compiler-rt][profiling] Add an incremental buffer writing mode to libprofile

[paquette]

Summary

In environments with tight memory constraints, users may not be able to actually allocate a buffer large enough to fit all of their program's profiling data.

In this case, a workflow like this would be helpful:

while (!__llvm_profile_incremental_writer_done()) {
  char Buffer[BUFFER_SIZE];
  if (__llvm_profile_write_buffer_incremental(Buffer, BUFFER_SIZE) == -1)
    return -1;
  if (putBufferSomewhere(Buffer, BUFFER_SIZE))
    return -1;
}

That is, make a smaller buffer, fill it with BUFFER_SIZE bytes of data, transfer that somewhere else, and repeat the process until all of the profiling data has been transferred out.

This patch adds the state and APIs necessary to make the above workflow possible.

The added APIs are as follows:

• int __llvm_profile_init_incremental_writer_state(void) - Initializes internal structures to remember where we are in the process of writing profiling data
• __llvm_profile_write_buffer_incremental(char *Buffer, uint64_t WriteSizeInBytes) - Writes a certain number of bytes into a buffer
• int __llvm_profile_incremental_writer_done(void) - Returns whether or not all profiling bytes have been written

The added state is a struct (IncrementalProfileWriterState) that keeps track of

• How many sources we need to write data from
• What those sources are
• Which source we're writing from, and where in it we should write from

The struct reuses the existing ProfDataIOVec structure used throughout the rest of libprofile.

Currently, binary ID writing is not supported, since it seems very platform-specific. This will emit a warning on platforms that enable binary ID writing.

This patch also adds a couple refactors/error message improvements:

• Factor out some of the header construction code in lprofWriteDataImpl
• Add a PROF_ERR_INTERNAL to communicate when an error is likely not a user error, but rather a bug of some sort

[paquette]

• Rebase
• Don't assume dst buffer is 0 filled; use memset for padding
• Make __llvm_profile_write_buffer_incremental return the number of bytes actually written on success. -1 is still the error.

[paquette]

ping

[aemerson]

compiler-rt/lib/profile/InstrProfilingBuffer.c

191

I think this can just take a pointer to ProfDataIOVec? It's a small struct but seems unnecessary to copy since the call site is already dealing with a ptr.

[paquette]

Make isPadding take a pointer.

[paquette]

ping

[aemerson]

I'm not sure I'm the best person to approve but so far LGTM from my shallow understanding.

[gulfem]

I did not have a change to finish reviewing this patch, but it breaks binary-id.c and binary-id-lookup.c tests that you can see in the pre-merge checks.

LLVM Profile Error: Failed to write file "/var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/profile/Profile-x86_64/Linux/Output/binary-id.c.tmp.profraw": File exists
error: /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/profile/Profile-x86_64/Linux/Output/binary-id.c.tmp.profraw: invalid instrumentation profile data (file header is corrupt)

https://reviews.llvm.org/harbormaster/unit/view/6779087/.

[tellenbach]

compiler-rt/lib/profile/InstrProfilingWriter.c

365

Seems like this function is missing a return in case the branch is not taken. This is UB.

[jroelofs]

compiler-rt/lib/profile/InstrProfilingBuffer.c

237

Should this be a uint64_t, since the struct's NumBytesWritten has that type? Likewise, shouldn't the return type be changed to match?

compiler-rt/lib/profile/InstrProfilingWriter.c

442

I think the comment should elaborate a little on why this isn't supported yet, what makes it tricky or a lot of work or whatever.

[phosek]

The runtime memory overhead of profile instrumentation comes from the different allocated sections: __llvm_prf_cnts, __llvm_prf_data, __llvm_prf_vnds, __llvm_prf_names; let's call all of these together A. When using the buffer mode, we'll need a buffer to hold all the profile data, let's call it B. The total memory overhead is therefore A+B. This change replaces B with a fixed size buffer k, so the memory overhead is going to decrease to A+k.

My concern is that this change adds a lot of complexity to the profile runtime, which already suffers from poor code quality and is in a dire need of a substantial rewrite/refactoring, but the impact is somewhat limited because this change does nothing to reduce the size of A.

Have you considered alternatives such as using instrumentation function groups? For example, using -fprofile-function-groups=4 would reduce the memory overhead to (A+B)/4 which should be strictly better than A+k. Would that work for your use case?

Another direction we discussed for reducing the memory overhead is making __llvm_prf_data, __llvm_prf_vnds, __llvm_prf_names non-allocated. Our observation is that the only thing we really need in the profile are the header and the counters, everything else can be recovered from the binary (which becomes easier if the profile has binary ID and you can use debuginfod to retrieve the binary). That's direction we'd like to pursue to support profiling in embedded environments.

compiler-rt/lib/profile/InstrProfilingBuffer.c

19

This struct is fairly large and would be always allocated, even if we aren't using the incremental mode unless it can be GC'ed by the linker (which requires compiling with GC enabled which isn't enabled by default on all platforms).

Could we move all the symbols related to the incremental mode into a separate TU so if the incremental mode isn't being used, those symbols won't be pulled into the link in in the first place?

278–281

I think we might need to rethink the error reporting. The callees will already print their own error message, but so will the callers, so we'll end up with reporting the error multiple times.

I think we should either omit the error message here, and rely on callees to do the error reporting, or change the callees to return a return code and do all the error reporting here (but not both).

compiler-rt/lib/profile/InstrProfilingWriter.c

314

Would it be possible to split this change and land it separately?

336

Since this is C and not C++, do you know if all compilers supported by compiler-rt perform RVO in C? It might be safer to use an output parameter here to avoid a copy.

372

Could this be an assert?

377

Could this be an assert?

411

Even though Binary ID is only supported for ELF, we have a plan for how to support it for COFF and Mach-O as well, it just hasn't been implemented yet.

Do you have an idea for how we could support Binary ID with incremental mode? I'm a little wary about introducing features that are mutually incompatible without some long term plan for resolving those incompatibilities.

clang/docs/SourceBasedCodeCoverage.rst

 
 * Optionally, forward-declare ``void __llvm_profile_reset_counters(void)`` and
   call it to reset the counters before entering a specific section to be
   profiled. This is only useful if there is some setup that should be excluded
   from the profile.
 
 In C++ files, declare these as ``extern "C"``.
 
+Using filesystemless (buffer) APIs incrementally
+------------------------------------------------
+
+Users working in environments with strict memory constraints may be unable to
+allocate a buffer large enough to fit all profiling data.
+
+These users may want to incrementally construct their profiling data in a
+smaller buffer.
+
+The profiling runtime provides the following APIs for this purpose:
+
+* ``int __llvm_profile_init_incremental_writer_state(void)``
+* ``int __llvm_profile_write_buffer_incremental(char *DstBuffer, uint64_t NumBytesToWrite)``
+* ``int __llvm_profile_incremental_writer_done(void)``
+
+An example of how these APIs work in user-code is below:
+
+.. code-block:: C
+  // Disable static initializers.
+  int __llvm_profile_runtime = 0;
+  // Initialize internal profiling structures.
+  int __llvm_profile_init_incremental_writer_state(void);
+  // Writes at most NumBytesToWrite bytes to the destination buffer.
+  int __llvm_profile_write_buffer_incremental(char *DstBuffer,
+                                              uint64_t NumBytesToWrite);
+  // Returns 1 if all profiling data has been written.
+  int __llvm_profile_incremental_writer_done(void);
+
+  /* ... Let's pretend there is a bunch of code here... */
+
+  // Let's say that this takes the buffer and puts it somewhere else.
+  void transferBuffer(char* Buffer) {
+    // ... Do something with the buffer ...
+  }
+
+  // Write in 64-bit chunks.
+  #define BUFFER_SIZE 8
+  int placeWhereIWantToOutputProfilingData(void) {
+    if (__llvm_profile_init_incremental_writer_state())
+      return -1;
+    while (!__llvm_profile_incremental_writer_done()) {
+      char Buffer[BUFFER_SIZE];
+      if (__llvm_profile_write_buffer_incremental(Buffer, BUFFER_SIZE))
+         return -1;
+      transferBuffer(Buffer);
+    }
+  }
+
 Collecting coverage reports for the llvm project
 ================================================
 
 To prepare a coverage report for llvm (and any of its sub-projects), add
 ``-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On`` to the cmake configuration. Raw
 profiles will be written to ``$BUILD_DIR/profiles/``. To prepare an html
 report, run ``llvm/utils/prepare-code-coverage-artifact.py``.
 

compiler-rt/lib/profile/InstrProfiling.h

 /*!
  * \brief Write instrumentation data to the given buffer.
  *
  * \pre \c Buffer is the start of a buffer at least as big as \a
  * __llvm_profile_get_size_for_buffer().
  */
 int __llvm_profile_write_buffer(char *Buffer);
 
+/// Incrementally copy profiling data into a user-provided buffer.
+/// \p DstBuffer - The buffer to write to.
+/// \p NumBytesToWrite - A nonzero number of bytes to write into the buffer.
+/// \note DstBuffer must be large enough to fit \p NumBytesToWrite bytes.
+int __llvm_profile_write_buffer_incremental(char *DstBuffer,
+                                            uint64_t NumBytesToWrite);
+
+/// Initialize writer state for incremental writes. Must be called before
+/// __llvm_profile_write_buffer_incremental.
+int __llvm_profile_init_incremental_writer_state(void);
+
+/// \returns 1 if the incremental writer is finished writing, and 0 otherwise.
+int __llvm_profile_incremental_writer_done(void);
+
 const __llvm_profile_data *__llvm_profile_begin_data(void);
 const __llvm_profile_data *__llvm_profile_end_data(void);
 const char *__llvm_profile_begin_names(void);
 const char *__llvm_profile_end_names(void);
 char *__llvm_profile_begin_counters(void);
 char *__llvm_profile_end_counters(void);
 ValueProfNode *__llvm_profile_begin_vnodes();
 ValueProfNode *__llvm_profile_end_vnodes();

compiler-rt/lib/profile/InstrProfilingBuffer.c

 /*===- InstrProfilingBuffer.c - Write instrumentation to a memory buffer --===*\
 |*
 |* Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 |* See https://llvm.org/LICENSE.txt for license information.
 |* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 |*
 \*===----------------------------------------------------------------------===*/
 
 // Note: This is linked into the Darwin kernel, and must remain compatible
 // with freestanding compilation. See `darwin_add_builtin_libraries`.
 
 #include "InstrProfiling.h"
 #include "InstrProfilingInternal.h"
 #include "InstrProfilingPort.h"
+#include <stdint.h>
+#include <string.h>
+
+/// Global state for writing incrementally.
+static IncrementalProfileWriterState GlobalIncrementalProfileWriterState;
+
+/// Set to a nonzero value if the IncrementalProfileWriterState has been
+/// initialized.
+static char GlobalWriterStateWasInitialized = 0;
 
 /* When continuous mode is enabled (%c), this parameter is set to 1.
  *
  * This parameter is defined here in InstrProfilingBuffer.o, instead of in
  * InstrProfilingFile.o, to sequester all libc-dependent code in
  * InstrProfilingFile.o. The test `instrprof-without-libc` will break if this
  * layering is violated. */
 static int ContinuouslySyncProfile = 0;
COMPILER_RT_VISIBILITY int __llvm_profile_write_buffer_internal(
     char *Buffer, const __llvm_profile_data *DataBegin,
     const __llvm_profile_data *DataEnd, const char *CountersBegin,
     const char *CountersEnd, const char *NamesBegin, const char *NamesEnd) {
   ProfDataWriter BufferWriter;
   initBufferWriter(&BufferWriter, Buffer);
   return lprofWriteDataImpl(&BufferWriter, DataBegin, DataEnd, CountersBegin,
                             CountersEnd, 0, NamesBegin, NamesEnd, 0);
 }
+
+COMPILER_RT_VISIBILITY char
+doneCopyingAllEltsFromSrcBufferAtIdx(IncrementalProfileWriterState WriterState,
+                                     uint64_t SrcBufferIdx) {
+  return WriterState.WriteIdxIntoSrcBuffer[SrcBufferIdx] >=
+         WriterState.SrcBuffers[SrcBufferIdx].NumElm;
+}
+
+COMPILER_RT_VISIBILITY char
+isIncrementalProfileWriterDone(IncrementalProfileWriterState WriterState) {
+  return WriterState.CurrSrcBufIdx == WriterState.LastSrcBufIdx;
+}
+
+/// \returns whether or not \p Vec is padding or actual profiling data.
+static char isPadding(ProfDataIOVec Vec) {
+  return Vec.UseZeroPadding;
+}
+
+COMPILER_RT_VISIBILITY BytesWrittenAndBytesLeftover
+tryWriteOneChunkAndUpdateWriterState(IncrementalProfileWriterState *WriterState,
+                                     char *DstBuffer,
+                                     uint64_t WriteSizeInBytes) {
+  // Find the source buffer that we are going to perform the write from.
+  uint64_t Idx = WriterState->CurrSrcBufIdx;
+  ProfDataIOVec *WriteSrc = &WriterState->SrcBuffers[Idx];
+  uint64_t UnwrittenBytesInBuffer = WriterState->SrcBuffers[Idx].NumElm -
+                                    WriterState->WriteIdxIntoSrcBuffer[Idx];
+  // Number of bytes that must be carried into a subsequent call to this
+  // function.
+  uint64_t Leftover = 0;
+  // Does the size of this write exceed the number of unwritten bytes in this
+  // buffer?
+  if (WriteSizeInBytes > UnwrittenBytesInBuffer) {
+    // Size exceeded. Write the entire remainder of this buffer. Return the
+    // amount that was not written in OutLeftover.
+    Leftover = WriteSizeInBytes - UnwrittenBytesInBuffer;
+    WriteSizeInBytes = UnwrittenBytesInBuffer;
+  }
+  // Like in the non-incremental case, we'll assume the destination buffer is
+  // already zeroed-out.
+  if (!isPadding(*WriteSrc)) {
+    memcpy(DstBuffer,
+           (uint8_t *)WriteSrc->Data + WriterState->WriteIdxIntoSrcBuffer[Idx],
+           WriteSizeInBytes);
+  }
+  // Update the point of the next write.
+  WriterState->WriteIdxIntoSrcBuffer[Idx] += WriteSizeInBytes;
+  if (doneCopyingAllEltsFromSrcBufferAtIdx(*WriterState, Idx))
+    WriterState->CurrSrcBufIdx++;
+  return (BytesWrittenAndBytesLeftover){WriteSizeInBytes, Leftover};
+}
+
+COMPILER_RT_VISIBILITY int
+writeProfileDataInChunks(IncrementalProfileWriterState *WriterState,
+                         char *DstBuffer, uint64_t WriteSizeInBytes) {
+  if (!WriteSizeInBytes || DstBuffer == NULL || WriterState == NULL)
+    return -1;
+  // Incrementally write from each of the source buffers in WriterState until
+  // we either complete the full write of WriteSizeInBytes, or we run out of
+  // source data to write entirely.
+  BytesWrittenAndBytesLeftover WrittenAndLeftover;
+  do {
+    WrittenAndLeftover = tryWriteOneChunkAndUpdateWriterState(
+        WriterState, DstBuffer, WriteSizeInBytes);
+    DstBuffer += WrittenAndLeftover.NumBytesWritten;
+    WriteSizeInBytes = WrittenAndLeftover.NumBytesLeftover;
+    // We finished writing all bytes in the buffer during this write.
+    if (isIncrementalProfileWriterDone(*WriterState))
+      return 0;
+  } while (WriteSizeInBytes);
+  return 0;
+}
+
+COMPILER_RT_VISIBILITY int
+__llvm_profile_write_buffer_incremental(char *Buffer,
+                                        uint64_t WriteSizeInBytes) {
+  if (!WriteSizeInBytes) {
+    PROF_ERR("%s\n", "WriteSizeInBytes must be at least 1.\n");
+    return -1;
+  }
+
+  // User must call __llvm_profile_init_incremental_writer_state.
+  if (!GlobalWriterStateWasInitialized) {
+    PROF_ERR("%s\n", "User must call "
+                     "__llvm_profile_init_incremental_writer_state() before "
+                     "performing an incremental write\n");
+    return -1;
+  }
+
+  if (isIncrementalProfileWriterDone(GlobalIncrementalProfileWriterState)) {
+    PROF_ERR("%s\n",
+             "All profiling data has already been written; please check "
+             "__llvm_profile_incremental_writer_done() in user code.");
+    return -1;
+  }
+  return writeProfileDataInChunks(&GlobalIncrementalProfileWriterState, Buffer,
+                                  WriteSizeInBytes);
+}
+
+COMPILER_RT_VISIBILITY int __llvm_profile_init_incremental_writer_state(void) {
+  if (initIncrementalProfileWriterState(&GlobalIncrementalProfileWriterState)) {
+    PROF_ERR_INTERNAL("%s\n", "Failed to initialize global writer state.");
+    return 1;
+  }
+  GlobalWriterStateWasInitialized = 1;
+  return 0;
+}
+
+COMPILER_RT_VISIBILITY int __llvm_profile_incremental_writer_done(void) {
+  if (!isIncrementalProfileWriterDone(GlobalIncrementalProfileWriterState))
+    return 0;
+  // Verify that all other buffers were written and emit an error message if
+  // any were somehow not written.
+  for (size_t I = 0; I < NUM_PROFILING_SECTIONS_PLUS_ONE; ++I) {
+    if (!doneCopyingAllEltsFromSrcBufferAtIdx(
+            GlobalIncrementalProfileWriterState, I))
+      PROF_ERR_INTERNAL("Missing profile data from source buffer %zu; "
+                        "please file a bug.\n",
+                        I);
+    return -1;
+  }
+  return 1;
+}

compiler-rt/lib/profile/InstrProfilingInternal.h

 /*===- InstrProfiling.h- Support library for PGO instrumentation ----------===*\
 |*
 |* Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 |* See https://llvm.org/LICENSE.txt for license information.
 |* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 |*
 \*===----------------------------------------------------------------------===*/
 
 #ifndef PROFILE_INSTRPROFILING_INTERNALH_
 #define PROFILE_INSTRPROFILING_INTERNALH_
 
 #include <stddef.h>
+#include <stdint.h>
 
 #include "InstrProfiling.h"
 
+/// Creates a __llvm_profile_header for the instrumented binary and returns it.
+__llvm_profile_header createHeader();
+
 /*!
  * \brief Write instrumentation data to the given buffer, given explicit
  * pointers to the live data in memory.  This function is probably not what you
  * want.  Use __llvm_profile_get_size_for_buffer instead.  Use this function if
  * your program has a custom memory layout.
  */
 uint64_t __llvm_profile_get_size_for_buffer_internal(
     const __llvm_profile_data *DataBegin, const __llvm_profile_data *DataEnd,

 extern void (*VPMergeHook)(struct ValueProfData *, __llvm_profile_data *);
 
 /*
  * Write binary ids into profiles if writer is given.
  * Return -1 if an error occurs, otherwise, return total size of binary ids.
  */
 int __llvm_write_binary_ids(ProfDataWriter *Writer);
 
+/// Number of sections that need to be written into a buffer, plus one for the
+/// header.
+#define NUM_PROFILING_SECTIONS_PLUS_ONE 7
+
+/// Used for streaming writes.
+///
+/// A structure which stores all data that must be written to a raw profile
+/// and what is currently being written.
+typedef struct IncrementalProfileWriterState {
+  /// Index of current source buffer being written from.
+  uint64_t CurrSrcBufIdx;
+  /// The last source buffer to write from.
+  uint64_t LastSrcBufIdx;
+  /// Source buffers containing data to copy into the destination buffer.
+  ProfDataIOVec SrcBuffers[NUM_PROFILING_SECTIONS_PLUS_ONE];
+  /// Given a source buffer at an index, the index into that source buffer that
+  /// should be written from next.
+  uint64_t WriteIdxIntoSrcBuffer[NUM_PROFILING_SECTIONS_PLUS_ONE];
+  /// Copy of a constructed header. Used to ensure that SrcBuffers[0] actually
+  /// has some valid data.
+  uint8_t ProfileHeaderAsByteArray[sizeof(__llvm_profile_header)];
+} IncrementalProfileWriterState;
+
+/// Populate the source buffer at a given index in \p WriterState.
+/// \p SrcBufferData - Pointer to data, or NULL in the case of writing padding.
+/// \p NumBytes - Number of bytes in buffer, or number of padding bytes.
+/// \p SrcBufferIdxToPopulate - The source buffer index in \p WriterState to
+/// populate.
+/// \p IsPadding - Nonzero if the data is padding.
+/// \returns 0 on success, and a nonzero value otherwise.
+char addProfDataIOVecToWriterState(IncrementalProfileWriterState *WriterState,
+                                   const void *SrcBufferData, size_t NumBytes,
+                                   size_t SrcBufferIdxToPopulate,
+                                   int IsPadding);
+/// Add a source buffer to \p WriterState which contains padding bytes.
+/// \p NumBytes - Number of padding bytes.
+/// \p SrcBufferIdxToPopulate - The source buffer index in \p WriterState to
+/// populate.
+/// \returns 0 on success, and a nonzero value otherwise.
+char addPaddingBytesToWriterState(IncrementalProfileWriterState *WriterState,
+                                  size_t NumBytes,
+                                  size_t SrcBufferIdxToPopulate);
+
+/// Add a source buffer to \p WriterState which contains non-padding data.
+/// \p SrcBufferData - Pointer to profiling data. Must not be NULL.
+/// \p NumBytes - Number of data bytes in \p SrcBufferData
+/// \p SrcBufferIdxToPopulate - The source buffer index in \p WriterState to
+/// populate.
+/// \returns 0 on success, and a nonzero value otherwise.
+char addDataBytesToWriterState(IncrementalProfileWriterState *WriterState,
+                               const void *SrcBufferData, size_t NumBytes,
+                               size_t SrcBufferIdxToPopulate);
+
+/// \returns 1 if the source buffer at index \p SrcBufferIdx in \p WriterState
+/// has been completely exhausted, and 0 otherwise.
+char doneCopyingAllEltsFromSrcBufferAtIdx(
+    IncrementalProfileWriterState WriterState, uint64_t SrcBufferIdx);
+
+/// \returns 1 if all source buffers in \p WriterState have been exhausted, and
+/// 0 otherwise.
+char isIncrementalProfileWriterDone(IncrementalProfileWriterState WriterState);
+
+/// Helper struct for tryWriteOneChunkAndUpdateWriterState.
+///
+/// Tracks the number of bytes which were actually written in the call, and the
+/// number that need to be written from the next source buffer.
+typedef struct BytesWrittenAndBytesLeftover {
+  /// Number of bytes written from the current source buffer into the
+  /// destination buffer.
+  uint64_t NumBytesWritten;
+  /// Number of bytes that were not written in the call to writeOneChunk.
+  /// These must be written in a subsequent call to writeOneChunk.
+  uint64_t NumBytesLeftover;
+} BytesWrittenAndBytesLeftover;
+
+/// Helper function for writeProfileDataInChunks.
+///
+/// Writes bytes from the current source buffer in \p WriterState into
+/// \p DstBuffer. Writes at most \p WriteSizeInBytes bytes.
+///
+/// \returns a BytesWrittenAndBytes struct containing the number of bytes
+/// written from the current source buffer, and the number of bytes which must
+/// be written in a subsequent call to writeOneChunk to complete the full write.
+///
+/// If bytes are leftover, this means that the current source buffer being
+/// written from in \p WriterState did not have enough remaining data to
+/// complete the full write.
+///
+/// Updates \p WriterState's current source buffer if a whole write could not
+/// be completed.
+BytesWrittenAndBytesLeftover
+tryWriteOneChunkAndUpdateWriterState(IncrementalProfileWriterState *WriterState,
+                                     char *DstBuffer,
+                                     uint64_t WriteSizeInBytes);
+
+/// Write \p WriteSizeInBytes bytes from the source buffers in \p WriterState
+/// into \p DstBuffer.
+/// \returns -1 on error. 0 on success if there are still contents to write from
+/// \p WriterState, and 1 on success if all bytes have been written.
+int writeProfileDataInChunks(IncrementalProfileWriterState *WriterState,
+                             char *DstBuffer, uint64_t WriteSizeInBytes);
+
+/// Initialize \p WriterState for incremental writes. Populates all source
+/// buffers within the structure.
+/// \returns 0 on success, and a nonzero value on error.
+char initIncrementalProfileWriterState(
+    IncrementalProfileWriterState *WriterState);
+
 #endif

compiler-rt/lib/profile/InstrProfilingPort.h

#define PROF_ERR(Format, ...)                                                  \
   fprintf(stderr, "LLVM Profile Error: " Format, __VA_ARGS__);
 
 #define PROF_WARN(Format, ...)                                                 \
   fprintf(stderr, "LLVM Profile Warning: " Format, __VA_ARGS__);
 
 #define PROF_NOTE(Format, ...)                                                 \
   fprintf(stderr, "LLVM Profile Note: " Format, __VA_ARGS__);
 
+#define PROF_ERR_INTERNAL(Format, ...)                                         \
+  fprintf(stderr, "LLVM Profile Error: INTERNAL: " Format, __VA_ARGS__);
+
 #ifndef MAP_FILE
 #define MAP_FILE 0
 #endif
 
 #ifndef O_BINARY
 #define O_BINARY 0
 #endif
 

compiler-rt/lib/profile/InstrProfilingWriter.c

 /*===- InstrProfilingWriter.c - Write instrumentation to a file or buffer -===*\
 |*
 |* Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 |* See https://llvm.org/LICENSE.txt for license information.
 |* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 |*
 \*===----------------------------------------------------------------------===*/
 
 // Note: This is linked into the Darwin kernel, and must remain compatible
 // with freestanding compilation. See `darwin_add_builtin_libraries`.
 
+#include <stdint.h>
 #ifdef _MSC_VER
 /* For _alloca */
 #include <malloc.h>
 #endif
 #include <string.h>
 
 #include "InstrProfiling.h"
 #include "InstrProfilingInternal.h"
COMPILER_RT_VISIBILITY int lprofWriteData(ProfDataWriter *Writer,
   const char *CountersEnd = __llvm_profile_end_counters();
   const char *NamesBegin = __llvm_profile_begin_names();
   const char *NamesEnd = __llvm_profile_end_names();
   return lprofWriteDataImpl(Writer, DataBegin, DataEnd, CountersBegin,
                             CountersEnd, VPDataReader, NamesBegin, NamesEnd,
                             SkipNameDataWrite);
 }
 
-COMPILER_RT_VISIBILITY int
-lprofWriteDataImpl(ProfDataWriter *Writer, const __llvm_profile_data *DataBegin,
-                   const __llvm_profile_data *DataEnd,
-                   const char *CountersBegin, const char *CountersEnd,
-                   VPDataReaderType *VPDataReader, const char *NamesBegin,
-                   const char *NamesEnd, int SkipNameDataWrite) {
+COMPILER_RT_VISIBILITY __llvm_profile_header createHeader() {
+  __llvm_profile_header Header;
+  const __llvm_profile_data *DataBegin = __llvm_profile_begin_data();
+  const __llvm_profile_data *DataEnd = __llvm_profile_end_data();
+  const char *CountersBegin = __llvm_profile_begin_counters();
+  const char *CountersEnd = __llvm_profile_end_counters();
+  const char *NamesBegin = __llvm_profile_begin_names();
+  const char *NamesEnd = __llvm_profile_end_names();
   int DebugInfoCorrelate =
       (__llvm_profile_get_version() & VARIANT_MASK_DBG_CORRELATE) != 0ULL;
-
   /* Calculate size of sections. */
   const uint64_t DataSectionSize =
       DebugInfoCorrelate ? 0 : __llvm_profile_get_data_size(DataBegin, DataEnd);
   const uint64_t NumData =
       DebugInfoCorrelate ? 0 : __llvm_profile_get_num_data(DataBegin, DataEnd);
   const uint64_t CountersSectionSize =
       __llvm_profile_get_counters_size(CountersBegin, CountersEnd);
   const uint64_t NumCounters =
       __llvm_profile_get_num_counters(CountersBegin, CountersEnd);
   const uint64_t NamesSize = DebugInfoCorrelate ? 0 : NamesEnd - NamesBegin;
 
-  /* Create the header. */
-  __llvm_profile_header Header;
-
   /* Determine how much padding is needed before/after the counters and after
    * the names. */
   uint64_t PaddingBytesBeforeCounters, PaddingBytesAfterCounters,
       PaddingBytesAfterNames;
   __llvm_profile_get_padding_sizes_for_counters(
       DataSectionSize, CountersSectionSize, NamesSize,
       &PaddingBytesBeforeCounters, &PaddingBytesAfterCounters,
       &PaddingBytesAfterNames);

 #endif
 
   /* The data and names sections are omitted in lightweight mode. */
   if (DebugInfoCorrelate) {
     Header.CountersDelta = 0;
     Header.NamesDelta = 0;
   }
 
+  return Header;
+}
+
+COMPILER_RT_VISIBILITY int
+lprofWriteDataImpl(ProfDataWriter *Writer, const __llvm_profile_data *DataBegin,
+                   const __llvm_profile_data *DataEnd,
+                   const char *CountersBegin, const char *CountersEnd,
+                   VPDataReaderType *VPDataReader, const char *NamesBegin,
+                   const char *NamesEnd, int SkipNameDataWrite) {
+  int DebugInfoCorrelate =
+      (__llvm_profile_get_version() & VARIANT_MASK_DBG_CORRELATE) != 0ULL;
+
+  /* Calculate size of sections. */
+  const uint64_t DataSectionSize =
+      DebugInfoCorrelate ? 0 : __llvm_profile_get_data_size(DataBegin, DataEnd);
+  const uint64_t CountersSectionSize =
+      __llvm_profile_get_counters_size(CountersBegin, CountersEnd);
+  const uint64_t NamesSize = DebugInfoCorrelate ? 0 : NamesEnd - NamesBegin;
+  uint64_t PaddingBytesBeforeCounters, PaddingBytesAfterCounters,
+      PaddingBytesAfterNames;
+  __llvm_profile_get_padding_sizes_for_counters(
+      DataSectionSize, CountersSectionSize, NamesSize,
+      &PaddingBytesBeforeCounters, &PaddingBytesAfterCounters,
+      &PaddingBytesAfterNames);
+
+  /* Create the header. */
+  __llvm_profile_header Header = createHeader();
+
   /* Write the profile header. */
   ProfDataIOVec IOVec[] = {{&Header, sizeof(__llvm_profile_header), 1, 0}};
   if (Writer->Write(Writer, IOVec, sizeof(IOVec) / sizeof(*IOVec)))
     return -1;
 
   /* Write the binary id lengths and data. */
   if (__llvm_write_binary_ids(Writer) == -1)
     return -1;
ProfDataIOVec IOVecData[] = {
        sizeof(uint8_t), NamesSize, 0},
       {NULL, sizeof(uint8_t), PaddingBytesAfterNames, 1}};
   if (Writer->Write(Writer, IOVecData, sizeof(IOVecData) / sizeof(*IOVecData)))
     return -1;
 
   /* Value profiling is not yet supported in continuous mode. */
   if (__llvm_profile_is_continuous_mode_enabled())
     return 0;
-
   return writeValueProfData(Writer, VPDataReader, DataBegin, DataEnd);
 }
+
+COMPILER_RT_VISIBILITY char
+addProfDataIOVecToWriterState(IncrementalProfileWriterState *WriterState,
+                              const void *SrcBufferData, size_t NumElm,
+                              size_t SrcBufferIdxToPopulate, int IsPadding) {
+  // Certain fields are optional (e.g. padding), so allow 0 elements.
+  if (!NumElm)
+    return 0;
+  if (SrcBufferIdxToPopulate >= NUM_PROFILING_SECTIONS_PLUS_ONE) {
+    PROF_ERR_INTERNAL("%s\n", "WriterState has too many source buffers; "
+                              "please file a bug");
+    return 1;
+  }
+  if (!SrcBufferData && !IsPadding) {
+    PROF_ERR_INTERNAL("%s\n", "Non-padding source buffer data; "
+                     "please file a bug");
+    return 1;
+  }
+  WriterState->SrcBuffers[SrcBufferIdxToPopulate] =
+      (ProfDataIOVec){SrcBufferData, sizeof(uint8_t), NumElm, IsPadding};
+  return 0;
+}
+
+COMPILER_RT_VISIBILITY char
+addPaddingBytesToWriterState(IncrementalProfileWriterState *WriterState,
+                             size_t NumElm, size_t SrcBufferIdxToPopulate) {
+  return addProfDataIOVecToWriterState(WriterState, NULL, NumElm,
+                                       SrcBufferIdxToPopulate,
+                                       /*IsPadding = */ 1);
+}
+
+COMPILER_RT_VISIBILITY char
+addDataBytesToWriterState(IncrementalProfileWriterState *WriterState,
+                          const void *SrcBufferData, size_t NumElm,
+                          size_t SrcBufferIdxToPopulate) {
+  return addProfDataIOVecToWriterState(WriterState, SrcBufferData, NumElm,
+                                       SrcBufferIdxToPopulate,
+                                       /*IsPadding = */ 0);
+}
+
+COMPILER_RT_VISIBILITY char
+initIncrementalProfileWriterState(IncrementalProfileWriterState *WriterState) {
+  int DebugInfoCorrelate =
+      (__llvm_profile_get_version() & VARIANT_MASK_DBG_CORRELATE) != 0ULL;
+  if (__llvm_write_binary_ids(NULL)) {
+    PROF_ERR(
+        "%s\n",
+        "Binary ID writing is not yet supported in incremental writing mode");
+    return 1;
+  }
+
+  // Ensure that there is no data left over in WriterState before we try to
+  // fill it with profiling data.
+  WriterState->CurrSrcBufIdx = 0;
+  for (size_t I = 0; I < NUM_PROFILING_SECTIONS_PLUS_ONE; ++I) {
+    WriterState->WriteIdxIntoSrcBuffer[I] = 0;
+    WriterState->SrcBuffers[I] = (ProfDataIOVec){0, 0, 0, 0};
+  }
+  const __llvm_profile_data *DataBegin = __llvm_profile_begin_data();
+  const __llvm_profile_data *DataEnd = __llvm_profile_end_data();
+  const char *CountersBegin = __llvm_profile_begin_counters();
+  const char *CountersEnd = __llvm_profile_end_counters();
+  const char *NamesBegin = __llvm_profile_begin_names();
+  const char *NamesEnd = __llvm_profile_end_names();
+  const uint64_t DataSectionSize =
+      DebugInfoCorrelate ? 0 : __llvm_profile_get_data_size(DataBegin, DataEnd);
+  const uint64_t CountersSectionSize =
+      __llvm_profile_get_counters_size(CountersBegin, CountersEnd);
+  const uint64_t NamesSize = DebugInfoCorrelate ? 0 : NamesEnd - NamesBegin;
+  uint64_t PaddingBytesBeforeCounters, PaddingBytesAfterCounters,
+      PaddingBytesAfterNames;
+  __llvm_profile_get_padding_sizes_for_counters(
+      DataSectionSize, CountersSectionSize, NamesSize,
+      &PaddingBytesBeforeCounters, &PaddingBytesAfterCounters,
+      &PaddingBytesAfterNames);
+
+  __llvm_profile_header Header = createHeader();
+
+  // Don't support writing binary IDs in this mode right now.
+  if (Header.BinaryIdsSize) {
+    Header.BinaryIdsSize = 0;
+    PROF_WARN("%s\n", "Binary IDs will not be written in incremental mode.")
+  }
+
+  // Store a copy of the header in the writer state so that we can incrementally
+  // copy from it later. Store it as a byte array so that we can incrementally
+  // write in single byte chunks if necessary.
+  memcpy(WriterState->ProfileHeaderAsByteArray, &Header,
+         sizeof(__llvm_profile_header));
+
+  size_t SrcBufferIdxToPopulate = 0;
+  // Set up the incremental profile writer state.
+  if (addDataBytesToWriterState(
+          WriterState, WriterState->ProfileHeaderAsByteArray,
+          /*NumElm =*/sizeof(__llvm_profile_header), SrcBufferIdxToPopulate++))
+    return 1;
+  // Each thing below has a pointer into a section in the binary. Create
+  // ProfDataIOVecs for each thing, and then store them in the writer state.
+  if (addDataBytesToWriterState(WriterState, DataBegin, DataSectionSize,
+                                SrcBufferIdxToPopulate++))
+    return 1;
+  if (addPaddingBytesToWriterState(WriterState, PaddingBytesBeforeCounters,
+                                   SrcBufferIdxToPopulate++))
+    return 1;
+  if (addDataBytesToWriterState(WriterState, CountersBegin, CountersSectionSize,
+                                SrcBufferIdxToPopulate++))
+    return 1;
+  if (addPaddingBytesToWriterState(WriterState, PaddingBytesAfterCounters,
+                                   SrcBufferIdxToPopulate++))
+    return 1;
+  if (addDataBytesToWriterState(WriterState, NamesBegin, NamesSize,
+                                SrcBufferIdxToPopulate++))
+    return 1;
+  if (addPaddingBytesToWriterState(WriterState, PaddingBytesAfterNames,
+                                   SrcBufferIdxToPopulate++))
+    return 1;
+  WriterState->LastSrcBufIdx = SrcBufferIdxToPopulate;
+  return 0;
+}

compiler-rt/test/profile/IncrementalWriteMode/Inputs/buffer-to-file.c

+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#define MAXSIZE 10000llu
+uint64_t __llvm_profile_get_size_for_buffer(void);
+int __llvm_profile_runtime = 0;
+int __llvm_profile_incremental_writer_done(void);
+int __llvm_profile_init_incremental_writer_state(void);
+int __llvm_profile_write_buffer_incremental(char *, uint64_t);
+
+int g = 1;
+int h = 10;
+void bar(char c) {
+  for (int i = 0; i < h; ++i)
+    g = (g + 1) * i;
+}
+
+void baz(char c) {
+  if (c) {
+    g++;
+  }
+  h++;
+}
+
+int __attribute__((no_profile_instrument_function))
+appendToFile(const char *FileN, const char *Buffer, uint64_t Size) {
+  FILE *File = fopen(FileN, "a");
+  if (!File)
+    printf("Failed to open file\n");
+  if (fwrite(Buffer, 1, Size, File) != Size)
+    return -1;
+  return fclose(File);
+}
+
+// Incrementally fill a file.
+int __attribute__((no_profile_instrument_function))
+constructBuffer(const char *FileName) {
+  if (__llvm_profile_get_size_for_buffer() > MAXSIZE)
+    return 1;
+  if (__llvm_profile_init_incremental_writer_state())
+    return -1;
+
+  // Clear file contents to make it easier to re-run the test.
+  fclose(fopen(FileName, "w"));
+  while (!__llvm_profile_incremental_writer_done()) {
+    char Buffer[BUFFER_SIZE];
+    if (__llvm_profile_write_buffer_incremental(Buffer, BUFFER_SIZE))
+      return -1;
+    if (appendToFile(FileName, Buffer, BUFFER_SIZE))
+      return -1;
+  }
+  return 0;
+}
+
+int __attribute__((no_profile_instrument_function))
+main(int argc, const char *argv[]) {
+  if (argc != 2)
+    return 1;
+  bar(0);
+  baz(1);
+  printf("argv[1] = %s\n", argv[1]);
+  return constructBuffer(argv[1]);
+}

compiler-rt/test/profile/IncrementalWriteMode/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c

+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+// CHECK-NOT: LLVM Profile Error
+
+#define MAXSIZE 10000llu
+int __llvm_profile_runtime = 0;
+uint64_t __llvm_profile_get_size_for_buffer(void);
+int __llvm_profile_write_buffer(char *);
+int __llvm_profile_incremental_writer_done(void);
+int __llvm_profile_init_incremental_writer_state(void);
+int __llvm_profile_write_buffer_incremental(char *, uint64_t);
+
+/// Some code to profile...
+int gg = 0;
+void baz(char c) {
+  if (c > 1)
+    gg--;
+  else
+    gg++;
+}
+
+void bar(char c) {
+  if (c == '1')
+    gg++;
+  else
+    gg--;
+  baz(c + 1);
+}
+// ... End code to profile
+
+// Check for mismatches between the two buffers
+// CHECK-NOT: MISMATCH
+int __attribute__((no_profile_instrument_function))
+diffBuffers(char *BufferA, char *BufferB, uint64_t BufferSize) {
+  for (size_t Idx = 0; Idx < BufferSize; ++Idx) {
+    if (BufferA[Idx] != BufferB[Idx]) {
+      printf(
+          "MISMATCH: NUM_BYTES=%d BufferA[%zu](%02x) != BufferB[%zu](%02x)\n",
+          NUM_BYTES, Idx, BufferA[Idx], Idx, BufferB[Idx]);
+      return -1;
+    }
+  }
+  printf("... Buffers match for NUM_BYTES = %d\n", NUM_BYTES);
+  return 0;
+}
+
+// Helper to output buffer contents + make things easier to debug if there is
+// a mismatch.
+void __attribute__((no_profile_instrument_function))
+dumpBuffer(char *Buffer, uint64_t BufferSize) {
+  unsigned Col = 0;
+  for (size_t I = 0; I < BufferSize; ++I) {
+    unsigned char c = Buffer[I];
+    printf("%02x ", c);
+    if (++Col == 8) {
+      Col = 0;
+      printf("\n");
+    }
+  }
+  printf("\n");
+}
+
+int __attribute__((no_profile_instrument_function)) fillBuffersAndCompare() {
+  uint64_t Size = __llvm_profile_get_size_for_buffer();
+  if (Size > MAXSIZE)
+    return 1;
+  static char BufferA[MAXSIZE];
+  static char BufferB[MAXSIZE];
+  __llvm_profile_write_buffer(BufferA);
+  printf("\n");
+  __llvm_profile_init_incremental_writer_state();
+  for (uint64_t CurrOffset = 0; CurrOffset < Size; CurrOffset += NUM_BYTES) {
+    if (__llvm_profile_incremental_writer_done())
+      break;
+    if (__llvm_profile_write_buffer_incremental(BufferB + CurrOffset,
+                                                NUM_BYTES))
+      return 1;
+  }
+  printf("Baseline buffer:\n");
+  dumpBuffer(BufferA, Size);
+  printf("Incremental buffer (chunk size = %d):\n", NUM_BYTES);
+  dumpBuffer(BufferB, Size);
+  diffBuffers(BufferA, BufferB, Size);
+  return 0;
+}
+
+int main(void) {
+  bar(0);
+  return fillBuffersAndCompare();
+}

compiler-rt/test/profile/IncrementalWriteMode/compare-against-whole-buffer-write.c

+// FIXME: This requires can be removed when incremental writing supports binary
+// IDs.
+// REQUIRES: system-darwin
+// RUN: %clang_profgen %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c -DNUM_BYTES=1 -o %t_1_byte
+// RUN: %run %t_1_byte | FileCheck %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c
+// RUN: %clang_profgen %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c -DNUM_BYTES=2 -o %t_2_byte
+// RUN: %run %t_2_byte | FileCheck %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c
+// RUN: %clang_profgen %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c -DNUM_BYTES=3 -o %t_3_byte
+// RUN: %run %t_3_byte | FileCheck %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c
+// RUN: %clang_profgen %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c -DNUM_BYTES=4 -o %t_4_byte
+// RUN: %run %t_4_byte | FileCheck %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c
+// RUN: %clang_profgen %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c -DNUM_BYTES=8 -o %t_8_byte
+// RUN: %run %t_8_byte | FileCheck %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c
+// RUN: %clang_profgen %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c -DNUM_BYTES=65 -o %t_65_byte
+// RUN: %run %t_65_byte | FileCheck %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c
+// RUN: %clang_profgen %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c -DNUM_BYTES=2147483647 -o %t_intmax_byte
+// RUN: %run %t_intmax_byte | FileCheck %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c

compiler-rt/test/profile/IncrementalWriteMode/construct-buffer.c

+// RUN: %clang_profgen -DBUFFER_SIZE=1 %S/Inputs/buffer-to-file.c -o %t_1_byte
+// RUN: %run %t_1_byte %t_1_byte.profraw
+// RUN: llvm-profdata show --all-functions --counts %t_1_byte.profraw | FileCheck %s
+// RUN: %clang_profgen -DBUFFER_SIZE=2 %S/Inputs/buffer-to-file.c -o %t_2_byte
+// RUN: %run %t_2_byte %t_2_byte.profraw
+// RUN: llvm-profdata show --all-functions --counts %t_2_byte.profraw | FileCheck %s
+// RUN: %clang_profgen -DBUFFER_SIZE=8 %S/Inputs/buffer-to-file.c -o %t_8_byte
+// RUN: %run %t_8_byte %t_8_byte.profraw
+// RUN: llvm-profdata show --all-functions --counts %t_8_byte.profraw | FileCheck %s
+
+// CHECK: Counters:
+// CHECK-NEXT:  bar:
+// CHECK-NEXT:    Hash:
+// CHECK-NEXT:    Counters: 2
+// CHECK-NEXT:    Function count: 1
+// CHECK-NEXT:    Block counts: [10]
+// CHECK-NEXT:  baz:
+// CHECK-NEXT:    Hash:
+// CHECK-NEXT:    Counters: 2
+// CHECK-NEXT:    Function count: 1
+// CHECK-NEXT:    Block counts: [1]

compiler-rt/test/profile/IncrementalWriteMode/debug-info-correlate.c

+// FIXME: This requires can be removed when incremental writing supports binary
+// IDs.
+// REQUIRES: system-darwin
+// RUN: %clang_profgen %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c -g -mllvm --debug-info-correlate -DNUM_BYTES=1 -o %t
+// RUN: %run %t | FileCheck %S/Inputs/write-n-bytes-and-compare-against-whole-buffer-write.c

compiler-rt/test/profile/IncrementalWriteMode/error-0-byte-write.c

+// RUN: %clang_profgen %s -o %t
+// RUN: %run %t 2>&1 | FileCheck %s
+
+// Test that we get an error message when we try to write 0 bytes, and that
+// we get the expected return value from
+// __llvm_profile_write_buffer_incremental.
+
+// CHECK-NOT: LLVM Profile Error: INTERNAL:
+
+#include <stdint.h>
+#include <stdio.h>
+
+#define MAX_BUFFER_SIZE 10000llu
+int __llvm_profile_runtime = 0;
+int __llvm_profile_init_incremental_writer_state(void);
+int __llvm_profile_write_buffer_incremental(char *, uint64_t);
+
+int main(void) {
+  static char Buffer[MAX_BUFFER_SIZE];
+  if (__llvm_profile_init_incremental_writer_state())
+    return -1;
+  // CHECK: LLVM Profile Error: WriteSizeInBytes must be at least 1
+  // CHECK: Ret = -1
+  int Ret = __llvm_profile_write_buffer_incremental(Buffer, 0);
+  printf("Ret = %d\n", Ret);
+  return 0;
+}

compiler-rt/test/profile/IncrementalWriteMode/error-exhausted-buffer.c

+// RUN: %clang_profgen %s -o %t
+// RUN: %run %t 2>&1 | FileCheck %s
+
+// Test that we
+// - Get an error message when we try to write without initializing the writer
+//   state.
+// - Get the expected return code.
+
+// CHECK-NOT: LLVM Profile Error: INTERNAL:
+// CHECK-NOT: TEST_FAIL
+
+#include <stdint.h>
+#include <stdio.h>
+
+#define MAX_BUFFER_SIZE 10000llu
+int __llvm_profile_runtime = 0;
+uint64_t __llvm_profile_get_size_for_buffer(void);
+int __llvm_profile_init_incremental_writer_state(void);
+int __llvm_profile_write_buffer_incremental(char *, uint64_t);
+int __llvm_profile_incremental_writer_done(void);
+
+int main(void) {
+  static char Buffer[MAX_BUFFER_SIZE];
+  uint64_t Size = __llvm_profile_get_size_for_buffer();
+  if (Size > MAX_BUFFER_SIZE) {
+    printf("TEST_FAIL: Buffer too large! (%llu > %llu\n", Size,
+           MAX_BUFFER_SIZE);
+    return 1;
+  }
+  if (__llvm_profile_init_incremental_writer_state())
+    return 1;
+
+  // Fill the buffer entirely.
+  __llvm_profile_write_buffer_incremental(Buffer, Size);
+
+  // Since we wrote all Size bytes, the buffer should be full.
+  if (!__llvm_profile_incremental_writer_done()) {
+    printf("TEST_FAIL: __llvm_profile_incremental_writer_done() failed!");
+    return 1;
+  }
+
+  // CHECK: LLVM Profile Error: All profiling data has already been written; please check __llvm_profile_incremental_writer_done() in user code.
+  __llvm_profile_write_buffer_incremental(Buffer, 1);
+
+  return 0;
+}

compiler-rt/test/profile/IncrementalWriteMode/error-uninitialized-state.c

+// RUN: %clang_profgen %s -o %t
+// RUN: %run %t 2>&1 | FileCheck %s
+
+// Test that we
+// - Get an error message when we try to write without initializing the writer
+//   state.
+// - Get the expected return code.
+
+// CHECK-NOT: LLVM Profile Error: INTERNAL:
+
+#include <stdint.h>
+#include <stdio.h>
+
+#define MAX_BUFFER_SIZE 10000llu
+int __llvm_profile_runtime = 0;
+int __llvm_profile_write_buffer_incremental(char *, uint64_t);
+
+int main(void) {
+  static char Buffer[MAX_BUFFER_SIZE];
+  // CHECK: LLVM Profile Error: User must call __llvm_profile_init_incremental_writer_state() before performing an incremental write
+  // CHECK: Ret = -1
+  int Ret = __llvm_profile_write_buffer_incremental(Buffer, 8);
+  printf("Ret = %d\n", Ret);
+  return 0;
+}

compiler-rt/test/profile/IncrementalWriteMode/reinitialize-state.c

+// RUN: %clang_profgen %s -o %t
+// RUN: %run %t 2>&1 | FileCheck %s
+
+// Test that we can re-initialize the writer state after exhausting the writer.
+// CHECK-NOT: TEST_FAIL
+// CHECK-NOT: LLVM Profile Error:
+
+#include <stdint.h>
+#include <stdio.h>
+
+#define MAX_BUFFER_SIZE 10000llu
+int __llvm_profile_runtime = 0;
+uint64_t __llvm_profile_get_size_for_buffer(void);
+int __llvm_profile_init_incremental_writer_state(void);
+int __llvm_profile_write_buffer_incremental(char *, uint64_t);
+int __llvm_profile_incremental_writer_done(void);
+
+int main(void) {
+  static char Buffer[MAX_BUFFER_SIZE];
+  uint64_t Size = __llvm_profile_get_size_for_buffer();
+  if (Size > MAX_BUFFER_SIZE) {
+    printf("TEST_FAIL: Buffer too large! (%llu > %llu\n", Size,
+           MAX_BUFFER_SIZE);
+    return 1;
+  }
+
+  if (__llvm_profile_init_incremental_writer_state())
+    return 1;
+
+  // Fill the buffer entirely.
+  __llvm_profile_write_buffer_incremental(Buffer, Size);
+
+  // Since we wrote all Size bytes, the buffer should be full.
+  if (!__llvm_profile_incremental_writer_done()) {
+    printf("TEST_FAIL: __llvm_profile_incremental_writer_done() failed!");
+    return 1;
+  }
+
+  // We should be able to clear the state by initializing it again.
+  if (__llvm_profile_init_incremental_writer_state())
+    return 1;
+
+  int Ret = __llvm_profile_write_buffer_incremental(Buffer, 1);
+  // CHECK: Ret = 0
+  printf("Ret = %d\n", Ret);
+  return 0;
+}