[fuzzer] Use puts() rather than printf() in CopyFileToErr()

[rsundahl]

Summary

CopyFileToErr() uses Printf("%s", ...) which fails with a negative size on
files >2Gb (Its path is through var-args wrappers to an unnecessary "%s"
expansion and subject to int overflows) Using puts() in place of printf()
bypasses this path and writes the string directly to stderr. This avoids the
present loss of data when a crashed worker has generated >2Gb of output.

rdar://99384640

[rsundahl]

Clean up some wording and the title.

[rsundahl]

Removed extraneous blank line.

[yln]

LGTM, looks harmless enough!

when a crashed worker has generated >2Gb of output.

Are we fine with printing >2GB of output? That sounds like it could lead to trouble elsewhere. Since the output is coming from a file I would hope that there is a way to get to it (even for remote CI?)
Do you think it would be worth investigating (as a separate follow-up) to truncate the output?

compiler-rt/test/fuzzer/BigFileCopy.cpp

20

Bonus points for the creative test! ;)

[rsundahl]

Linting

[rsundahl]

Thanks @yln!

In D146189#4200276, @yln wrote:

when a crashed worker has generated >2Gb of output.

Are we fine with printing >2GB of output? That sounds like it could lead to trouble elsewhere. Since the output is coming from a file I would hope that there is a way to get to it (even for remote CI?)

The use case here is that a worker that has accumulated >2Gb of output (over what may be days or weeks) crashes before it can append its log to stderr and in crashing, doesn't do a clean exit so more data is potentially lost. I agree though, we should convey that log file to the parent some other way and I think you're right in suspecting that this large amount of output may bite us further downstream simply because we now succeed at generating it.

Do you think it would be worth investigating (as a separate follow-up) to truncate the output?

That's a good idea. Truncate the output or just don't use stderr as a pipe for large files. (Although it's Unix, right? K&R would be probably approve of a fix that removes a limit to the amount of data going through a pipe. lol)

[dyung]

@rsundahl the test you added big-file-copy.test is failing on a buildbot, can you take a look?

https://lab.llvm.org/buildbot/#/builders/247/builds/2653

/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25
/tmp/lit-tmp-w57ukdpc/BigFileCopy-ab4731.o: in function `main':
BigFileCopy.cpp:(.text.main[main]+0x0): multiple definition of `main'; /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer.a(FuzzerMain.cpp.o):/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19: first defined here
/usr/bin/ld: /home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/build/lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer.a(FuzzerMain.cpp.o): in function `main':
/home/buildbot/buildbot-root/llvm-clang-x86_64-gcc-ubuntu/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20: undefined reference to `LLVMFuzzerTestOneInput'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[rsundahl]

In D146189#4202880, @dyung wrote:

@rsundahl the test you added big-file-copy.test is failing on a buildbot, can you take a look?

https://lab.llvm.org/buildbot/#/builders/247/builds/2653

Oh, interesting... Second copy of main() on a different platform. Thanks for the heads up.

[rsundahl]

Reopening to push changes for Debian test error (multiple main()'s).

[rsundahl]

Don't use main() for testing body (error on Debian).

[rsundahl]

cc: @dyung and @ormris - I have modified the test to use the canonical fuzzer entry point

int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size)

rather than

int main(int argc, char *argv[])

which was causing the duplicated symbol on Debian. It's interesting that it worked for us but not for Debian but this is the correct form in any case. Thanks for your input and for the revert. (I'm going to look into how I can get a build on the failed bot before I land this time.)

[rsundahl]

I've tested this on linux and it now links (and passes). I am going to land it again and watch it closely.

[rsundahl]

Rebased

[rsundahl]

Looks like I'll have to enable this test only on Darwin.

[rsundahl]

Opening a new revision to limit the scope of the test to Darwin only. (https://reviews.llvm.org/D147094)

compiler-rt/lib/fuzzer/FuzzerIO.h

 void DupAndCloseStderr();
 
 void CloseStdout();
 
 // For testing.
 FILE *GetOutputFile();
 void SetOutputFile(FILE *NewOutputFile);
 
+void Puts(const char *Str);
 void Printf(const char *Fmt, ...);
 void VPrintf(bool Verbose, const char *Fmt, ...);
 
 // Print using raw syscalls, useful when printing at early init stages.
 void RawPrint(const char *Str);
 
 // Platform specific functions:
 bool IsFile(const std::string &Path);

compiler-rt/lib/fuzzer/FuzzerIO.cpp

 
 std::string FileToString(const std::string &Path) {
   std::ifstream T(Path, std::ios::binary);
   return std::string((std::istreambuf_iterator<char>(T)),
                      std::istreambuf_iterator<char>());
 }
 
 void CopyFileToErr(const std::string &Path) {
-  Printf("%s", FileToString(Path).c_str());
+  Puts(FileToString(Path).c_str());
 }
 
 void WriteToFile(const Unit &U, const std::string &Path) {
   WriteToFile(U.data(), U.size(), Path);
 }
 
 void WriteToFile(const std::string &Data, const std::string &Path) {
   WriteToFile(reinterpret_cast<const uint8_t *>(Data.c_str()), Data.size(),
if (OutputFd >= 0) {
     }
   }
 }
 
 void CloseStdout() {
   DiscardOutput(1);
 }
 
+void Puts(const char *Str) {
+  fputs(Str, OutputFile);
+  fflush(OutputFile);
+}
+
 void Printf(const char *Fmt, ...) {
   va_list ap;
   va_start(ap, Fmt);
   vfprintf(OutputFile, Fmt, ap);
   va_end(ap);
   fflush(OutputFile);
 }
 

compiler-rt/test/fuzzer/BigFileCopy.cpp

+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+#include <cstddef>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+
+#include "FuzzerIO.h"
+
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
+  const char *FileName = "big-file.txt";
+  FILE *f = fopen(FileName, "w");
+
+  // This is the biggest file possible unless CopyFileToErr() uses Puts()
+  fprintf(f, "%2147483646s", "2Gb-2");
+
+  // This makes the file too big if CopyFileToErr() uses fprintf("%s", <file>)
+  fprintf(f, "THIS LINE RESPONSIBLE FOR EXCEEDING 2Gb FILE SIZE\n");
+  fclose(f);
+
+  // Should now because CopyFileToErr() now uses Puts()
+  fuzzer::CopyFileToErr(FileName);
+
+  // File is >2Gb so clean up
+  remove(FileName);
+
+  return 0;
+}

compiler-rt/test/fuzzer/big-file-copy.test

+RUN: %cpp_compiler %S/BigFileCopy.cpp -o %t
+RUN: %run %t -runs=1 -rss_limit_mb=4096 2>big-file-out.txt; result=$?
+RUN: %run rm -f big-file.txt big-file-out.txt
+RUN: %run (exit $result)