[SystemZ][z/OS] Distinguish between text and binary files on z/OS

[abhina.sreeskantharajan]

Summary

This patch consists of the initial changes to help distinguish between text and binary content correctly on z/OS. I would like to get feedback from Windows users on setting OF_None for all ToolOutputFiles. This seems to have been done as an optimization to prevent CRLF translation on Windows in the past.

[abhina.sreeskantharajan]

Remove some changes that cause lit failures.

[rnk]

Yes, we took steps to suppress CRLF generation. Part of that was to avoid Windows-only test failures from trailing CR whitespace interacting with tools like diff.

[amccarth]

From a Windows point of view, this LGTM.

clang/lib/Frontend/FrontendActions.cpp

841

The preceding block that detects the type of line separator seems ripe for factoring out into a separate function. It's a lot of low-level detail that visually outweighs the primary intent of this method.

But I'm fine with the change as it doesn't impact Windows and--as far as I understand--Posix platforms don't distinguish between text and binary mode.

[abhina.sreeskantharajan]

clang/lib/Frontend/FrontendActions.cpp

841

Right, my intention was not to change the current behaviour, but only limit it to Windows which is the only affected platform.

[abhina.sreeskantharajan]

Fix formatting from lint.

[abhina.sreeskantharajan]

If there are no concerns about setting OF_None flag for all ToolOutputFiles on Windows only, I would also like to get your reviews for this patch

[zibi]

LGTM, I just wonder if we can make an extra parameter to be default. I notice some places that is a default parameter but not in all instances. With default parameter some of the calls might be simplified if there is no need to override it.

clang/lib/Frontend/FrontendActions.cpp

812

If we don't need Triple anywhere else except here we can remove the new for local variable and do crate it as part of the test.

if (HostTriple(LLVM_HOST_TRIPLE).isOSWindows()) {
...

or even

bool BinaryMode = (HostTriple(LLVM_HOST_TRIPLE).isOSWindows())  ? true : false;

llvm/lib/Support/ToolOutputFile.cpp

50

Can we avoid creating HostTriple here?

[abhina.sreeskantharajan]

Removing llvm::Triple variables

[abhina.sreeskantharajan]

In D97785#2626427, @zibi wrote:

LGTM, I just wonder if we can make an extra parameter to be default. I notice some places that is a default parameter but not in all instances. With default parameter some of the calls might be simplified if there is no need to override it.

Thanks for reviewing. Do you have an example of which function is missing a default? The functions I modified in the .h files all have defaults to OF_None or binary. I can make another patch for additional changes.

[anirudhp]

LGTM.

[zibi]

clang/lib/Frontend/CompilerInstance.cpp

771

The llvm::sys::fs::all_read | llvm::sys::fs::all_write seems to be a default so if we make that parameter last we won't need to pass it and worry about the mode parameter which would be the second last default parameter. Switch parameters only when you determine that indeed tag is more frequent parameter which need to be set comparing to mode parameter.

[abhina.sreeskantharajan]

Sorry Zibi, I misunderstood your comment. I updated the patch to switch around the Flag and Mode defaults in createUniqueFile because the flags was set explicitly more often than mode. I also had to reorder the args in createUniqueEntity to match the ordering.

[abhina.sreeskantharajan]

clang/lib/Frontend/CompilerInstance.cpp

771

Thanks, I've reordered the args as you suggested.

[zibi]

LTGM, thx for an extra mile, Abhina.

[abhina.sreeskantharajan]

NFC edit: Fix formatting/lint

[MaskRay]

The recommended named parameter form is probably /*Param=*/something. It is liked by both https://clang.llvm.org/extra/clang-tidy/checks/bugprone-argument-comment.html and clang-format.
Another form look to me as well, if both the two tools like it.

[abhina.sreeskantharajan]

In D97785#2638824, @MaskRay wrote:

The recommended named parameter form is probably /*Param=*/something. It is liked by both https://clang.llvm.org/extra/clang-tidy/checks/bugprone-argument-comment.html and clang-format.
Another form look to me as well, if both the two tools like it.

Thanks, I've created the following patch https://reviews.llvm.org/D99072 to fix the parameter formatting.

[yroux]

Hi Abhina,

I think this patch broke Windows on ARM bots, the first failure observed 6
days ago in this build:

https://lab.llvm.org/buildbot/#/builders/65/builds/1245

I don't really get what the problem is, but the issue is related to
tablegen generated file AbstractTypeReader.inc and your patch is the only
one in the set which is touching that area.

Does it ring a bell for you ?

Cheers,
Yvan

[abhina.sreeskantharajan]

In D97785#2650824, @yroux wrote:

Hi Abhina,

I think this patch broke Windows on ARM bots, the first failure observed 6
days ago in this build:

https://lab.llvm.org/buildbot/#/builders/65/builds/1245

I don't really get what the problem is, but the issue is related to
tablegen generated file AbstractTypeReader.inc and your patch is the only
one in the set which is touching that area.

Does it ring a bell for you ?

Cheers,
Yvan

Hi, thanks for bringing this to my attention. Since only Windows seems to have this error, it may be caused by my patch. I suspect that setting OF_None flag/Binary mode for Windows in getFile() should fix it if that is the case. I'll work on a patch to hopefully fix the issue.

[amccarth]

When changing an IO stream's mode from binary to text breaks only Windows, it's likely due to the fact that Windows uses CR+LF as the newline indicator. Not only are the CRs sometimes unexpected, they can also throw off code that tries to seek to a computed file position.

[abhina.sreeskantharajan]

In D97785#2651008, @amccarth wrote:

When changing an IO stream's mode from binary to text breaks only Windows, it's likely due to the fact that Windows uses CR+LF as the newline indicator. Not only are the CRs sometimes unexpected, they can also throw off code that tries to seek to a computed file position.

Yes, this is probably the cause. I created a patch to set Binary mode on Windows only https://reviews.llvm.org/D99363 which should workaround the issue and unblock the bot. I do plan to investigate a better fix.

clang/lib/Frontend/CompilerInstance.cpp

if (UseTemporary) {
     // artifacts, also append .tmp.
     StringRef OutputExtension = llvm::sys::path::extension(OutputPath);
     SmallString<128> TempPath =
         StringRef(OutputPath).drop_back(OutputExtension.size());
     TempPath += "-%%%%%%%%";
     TempPath += OutputExtension;
     TempPath += ".tmp";
     int fd;
-    std::error_code EC =
-        llvm::sys::fs::createUniqueFile(TempPath, fd, TempPath);
+    std::error_code EC = llvm::sys::fs::createUniqueFile(
+        TempPath, fd, TempPath,
+        llvm::sys::fs::all_read | llvm::sys::fs::all_write,
+        Binary ? llvm::sys::fs::OF_None : llvm::sys::fs::OF_Text);
 
     if (CreateMissingDirectories &&
         EC == llvm::errc::no_such_file_or_directory) {
       StringRef Parent = llvm::sys::path::parent_path(OutputPath);
       EC = llvm::sys::fs::create_directories(Parent);
       if (!EC) {
-        EC = llvm::sys::fs::createUniqueFile(TempPath, fd, TempPath);
+        EC = llvm::sys::fs::createUniqueFile(
+            TempPath, fd, TempPath,
+            llvm::sys::fs::all_read | llvm::sys::fs::all_write,
+            Binary ? llvm::sys::fs::OF_None : llvm::sys::fs::OF_Text);
       }
     }
 
     if (!EC) {
       OS.reset(new llvm::raw_fd_ostream(fd, /*shouldClose=*/true));
       OSFile = TempFile = std::string(TempPath.str());
     }
     // If we failed to create the temporary, fallback to writing to the file

clang/lib/Frontend/FrontendActions.cpp

void PreprocessOnlyAction::ExecuteAction() {
   do {
     PP.Lex(Tok);
   } while (Tok.isNot(tok::eof));
 }
 
 void PrintPreprocessedAction::ExecuteAction() {
   CompilerInstance &CI = getCompilerInstance();
   // Output file may need to be set to 'Binary', to avoid converting Unix style
-  // line feeds (<LF>) to Microsoft style line feeds (<CR><LF>).
+  // line feeds (<LF>) to Microsoft style line feeds (<CR><LF>) on Windows.
   //
   // Look to see what type of line endings the file uses. If there's a
   // CRLF, then we won't open the file up in binary mode. If there is
   // just an LF or CR, then we will open the file up in binary mode.
   // In this fashion, the output format should match the input format, unless
   // the input format has inconsistent line endings.
   //
   // This should be a relatively fast operation since most files won't have
   // all of their source code on a single line. However, that is still a
   // concern, so if we scan for too long, we'll just assume the file should
   // be opened in binary mode.
-  bool BinaryMode = true;
-  const SourceManager& SM = CI.getSourceManager();
+
+  bool BinaryMode = false;
+  llvm::Triple HostTriple(LLVM_HOST_TRIPLE);
+  if (HostTriple.isOSWindows()) {
+    BinaryMode = true;
+    const SourceManager &SM = CI.getSourceManager();
     if (llvm::Optional<llvm::MemoryBufferRef> Buffer =
             SM.getBufferOrNone(SM.getMainFileID())) {
       const char *cur = Buffer->getBufferStart();
       const char *end = Buffer->getBufferEnd();
       const char *next = (cur != end) ? cur + 1 : end;
 
       // Limit ourselves to only scanning 256 characters into the source
       // file.  This is mostly a sanity check in case the file has no
       // newlines whatsoever.
-    if (end - cur > 256) end = cur + 256;
+      if (end - cur > 256)
+        end = cur + 256;
 
       while (next < end) {
         if (*cur == 0x0D) {  // CR
-        if (*next == 0x0A)  // CRLF
+          if (*next == 0x0A) // CRLF
             BinaryMode = false;
 
           break;
-      } else if (*cur == 0x0A)  // LF
+        } else if (*cur == 0x0A) // LF
           break;
 
         ++cur;
         ++next;
       }
     }
+  }
 
   std::unique_ptr<raw_ostream> OS =
       CI.createDefaultOutputFile(BinaryMode, getCurrentFileOrBufferName());
   if (!OS) return;
 
   // If we're preprocessing a module map, start by dumping the contents of the
   // module itself before switching to the input buffer.
   auto &Input = getCurrentInput();

clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp

void HTMLDiagnostics::ReportDiag(const PathDiagnostic& D,
   if (!DiagOpts.ShouldWriteStableReportFilename) {
       llvm::sys::path::append(Model, Directory, "report-%%%%%%.html");
       if (std::error_code EC =
           llvm::sys::fs::make_absolute(Model)) {
           llvm::errs() << "warning: could not make '" << Model
                        << "' absolute: " << EC.message() << '\n';
         return;
       }
-      if (std::error_code EC =
-          llvm::sys::fs::createUniqueFile(Model, FD, ResultPath)) {
-          llvm::errs() << "warning: could not create file in '" << Directory
-                       << "': " << EC.message() << '\n';
-          return;
+      if (std::error_code EC = llvm::sys::fs::createUniqueFile(
+              Model, FD, ResultPath,
+              /* Mode */ llvm::sys::fs::all_read | llvm::sys::fs::all_write,
+              /* Flags */ llvm::sys::fs::OF_Text)) {
       }
   } else {
       int i = 1;
       std::error_code EC;
       do {
           // Find a filename which is not already used
           const FileEntry* Entry = SMgr.getFileEntryForID(ReportFile);
           std::stringstream filename;

clang/tools/arcmt-test/arcmt-test.cpp

static bool performTransformations(StringRef resourcesPath,
   // FIXME: TestResultForARC
 
   return false;
 }
 
 static bool filesCompareEqual(StringRef fname1, StringRef fname2) {
   using namespace llvm;
 
-  ErrorOr<std::unique_ptr<MemoryBuffer>> file1 = MemoryBuffer::getFile(fname1);
+  ErrorOr<std::unique_ptr<MemoryBuffer>> file1 = MemoryBuffer::getFile(
+      fname1, /*FileSize*/ -1, /*RequiresNullTerminator*/ true,
+      /*IsVolatile*/ false, /*IsText*/ true);
   if (!file1)
     return false;
 
-  ErrorOr<std::unique_ptr<MemoryBuffer>> file2 = MemoryBuffer::getFile(fname2);
+  ErrorOr<std::unique_ptr<MemoryBuffer>> file2 = MemoryBuffer::getFile(
+      fname2, /*FileSize*/ -1, /*RequiresNullTerminator*/ true,
+      /*IsVolatile*/ false, /*IsText*/ true);
   if (!file2)
     return false;
 
   return file1.get()->getBuffer() == file2.get()->getBuffer();
 }
 
 static bool verifyTransformedFiles(ArrayRef<std::string> resultFiles) {
   using namespace llvm;
for (ArrayRef<std::string>::iterator
     }
     resultMap[sys::path::stem(fname)] = fname;
   }
 
   ErrorOr<std::unique_ptr<MemoryBuffer>> inputBuf = std::error_code();
   if (RemappingsFile.empty())
     inputBuf = MemoryBuffer::getSTDIN();
   else
-    inputBuf = MemoryBuffer::getFile(RemappingsFile);
+    inputBuf = MemoryBuffer::getFile(RemappingsFile, /*FileSize*/ -1,
+                                     /*RequiresNullTerminator*/ true,
+                                     /*IsVolatile*/ false, /*IsText*/ true);
   if (!inputBuf) {
     errs() << "error: could not read remappings input\n";
     return true;
   }
 
   SmallVector<StringRef, 8> strs;
   inputBuf.get()->getBuffer().split(strs, "\n", /*MaxSplit=*/-1,
                                     /*KeepEmpty=*/false);

llvm/include/llvm/Support/FileSystem.h

 ///
 /// The intended use is for files that are to be kept, possibly after
 /// renaming them. For example, when running 'clang -c foo.o', the file can
 /// be first created as foo-abc123.o and then renamed.
 ///
 /// @param Model Name to base unique path off of.
 /// @param ResultFD Set to the opened file's file descriptor.
 /// @param ResultPath Set to the opened file's absolute path.
+/// @param Mode Set to the opened file's permissions.
+/// @param Flags Set to the opened file's flags.
 /// @returns errc::success if Result{FD,Path} have been successfully set,
 ///          otherwise a platform-specific error_code.
 std::error_code createUniqueFile(const Twine &Model, int &ResultFD,
                                  SmallVectorImpl<char> &ResultPath,
-                                 unsigned Mode = all_read | all_write);
+                                 unsigned Mode = all_read | all_write,
+                                 OpenFlags Flags = OF_None);
 
 /// Simpler version for clients that don't want an open file. An empty
 /// file will still be created.
 std::error_code createUniqueFile(const Twine &Model,
                                  SmallVectorImpl<char> &ResultPath,
                                  unsigned Mode = all_read | all_write);
 
 /// Represents a temporary file.

 /// The filename is of the form prefix-random_chars.suffix. Since the directory
 /// is not know to the caller, Prefix and Suffix cannot have path separators.
 /// The files are created with mode 0600.
 ///
 /// This should be used for things like a temporary .s that is removed after
 /// running the assembler.
 std::error_code createTemporaryFile(const Twine &Prefix, StringRef Suffix,
                                     int &ResultFD,
-                                    SmallVectorImpl<char> &ResultPath);
+                                    SmallVectorImpl<char> &ResultPath,
+                                    OpenFlags Flags = OF_None);
 
 /// Simpler version for clients that don't want an open file. An empty
 /// file will still be created.
 std::error_code createTemporaryFile(const Twine &Prefix, StringRef Suffix,
-                                    SmallVectorImpl<char> &ResultPath);
+                                    SmallVectorImpl<char> &ResultPath,
+                                    OpenFlags Flags = OF_None);
 
 std::error_code createUniqueDirectory(const Twine &Prefix,
                                       SmallVectorImpl<char> &ResultPath);
 
 /// Get a unique name, not currently exisiting in the filesystem. Subject
 /// to race conditions, prefer to use createUniqueFile instead.
 ///
 /// Similar to createUniqueFile, but instead of creating a file only

llvm/include/llvm/Support/MemoryBuffer.h

public:
   /// Open the specified file as a MemoryBuffer, returning a new MemoryBuffer
   /// if successful, otherwise returning null. If FileSize is specified, this
   /// means that the client knows that the file exists and that it has the
   /// specified size.
   ///
   /// \param IsVolatile Set to true to indicate that the contents of the file
   /// can change outside the user's control, e.g. when libclang tries to parse
   /// while the user is editing/updating the file or if the file is on an NFS.
+  ///
+  /// \param IsText Set to true to indicate that the file should be read in
+  /// text mode.
   static ErrorOr<std::unique_ptr<MemoryBuffer>>
   getFile(const Twine &Filename, int64_t FileSize = -1,
-          bool RequiresNullTerminator = true, bool IsVolatile = false);
+          bool RequiresNullTerminator = true, bool IsVolatile = false,
+          bool IsText = false);
 
   /// Read all of the specified file into a MemoryBuffer as a stream
   /// (i.e. until EOF reached). This is useful for special files that
   /// look like a regular file but have 0 size (e.g. /proc/cpuinfo on Linux).
   static ErrorOr<std::unique_ptr<MemoryBuffer>>
   getFileAsStream(const Twine &Filename);
 
   /// Given an already-open file descriptor, map some slice of it into a
public:
 
   /// Read all of stdin into a file buffer, and return it.
   static ErrorOr<std::unique_ptr<MemoryBuffer>> getSTDIN();
 
   /// Open the specified file as a MemoryBuffer, or open stdin if the Filename
   /// is "-".
   static ErrorOr<std::unique_ptr<MemoryBuffer>>
   getFileOrSTDIN(const Twine &Filename, int64_t FileSize = -1,
-                 bool RequiresNullTerminator = true);
+                 bool RequiresNullTerminator = true, bool IsText = false);
 
   /// Map a subrange of the specified file as a MemoryBuffer.
   static ErrorOr<std::unique_ptr<MemoryBuffer>>
   getFileSlice(const Twine &Filename, uint64_t MapSize, uint64_t Offset,
                bool IsVolatile = false);
 
   //===--------------------------------------------------------------------===//
   // Provided for performance analysis.

llvm/lib/IRReader/IRReader.cpp

std::unique_ptr<Module> llvm::parseIR(MemoryBufferRef Buffer, SMDiagnostic &Err,
 
   return parseAssembly(Buffer, Err, Context, nullptr, DataLayoutCallback);
 }
 
 std::unique_ptr<Module>
 llvm::parseIRFile(StringRef Filename, SMDiagnostic &Err, LLVMContext &Context,
                   DataLayoutCallbackTy DataLayoutCallback) {
   ErrorOr<std::unique_ptr<MemoryBuffer>> FileOrErr =
-      MemoryBuffer::getFileOrSTDIN(Filename);
+      MemoryBuffer::getFileOrSTDIN(Filename, /*FileSize*/ -1,
+                                   /*RequiresNullTerminator*/ true,
+                                   /*IsText*/ true);
   if (std::error_code EC = FileOrErr.getError()) {
     Err = SMDiagnostic(Filename, SourceMgr::DK_Error,
                        "Could not open input file: " + EC.message());
     return nullptr;
   }
 
   return parseIR(FileOrErr.get()->getMemBufferRef(), Err, Context,
                  DataLayoutCallback);

llvm/lib/Support/MemoryBuffer.cpp

MemoryBuffer::BufferKind getBufferKind() const override {
     return MemoryBuffer::MemoryBuffer_Malloc;
   }
 };
 } // namespace
 
 template <typename MB>
 static ErrorOr<std::unique_ptr<MB>>
 getFileAux(const Twine &Filename, int64_t FileSize, uint64_t MapSize,
-           uint64_t Offset, bool RequiresNullTerminator, bool IsVolatile);
+           uint64_t Offset, bool RequiresNullTerminator, bool IsVolatile,
+           bool IsText);
 
 std::unique_ptr<MemoryBuffer>
 MemoryBuffer::getMemBuffer(StringRef InputData, StringRef BufferName,
                            bool RequiresNullTerminator) {
   auto *Ret = new (NamedBufferAlloc(BufferName))
       MemoryBufferMem<MemoryBuffer>(InputData, RequiresNullTerminator);
   return std::unique_ptr<MemoryBuffer>(Ret);
 }
MemoryBuffer::getMemBufferCopy(StringRef InputData, const Twine &BufferName) {
   auto Buf = getMemBufferCopyImpl(InputData, BufferName);
   if (Buf)
     return std::move(*Buf);
   return nullptr;
 }
 
 ErrorOr<std::unique_ptr<MemoryBuffer>>
 MemoryBuffer::getFileOrSTDIN(const Twine &Filename, int64_t FileSize,
-                             bool RequiresNullTerminator) {
+                             bool RequiresNullTerminator, bool IsText) {
   SmallString<256> NameBuf;
   StringRef NameRef = Filename.toStringRef(NameBuf);
 
   if (NameRef == "-")
     return getSTDIN();
-  return getFile(Filename, FileSize, RequiresNullTerminator);
+  return getFile(Filename, FileSize, RequiresNullTerminator, false, IsText);
 }
 
 ErrorOr<std::unique_ptr<MemoryBuffer>>
 MemoryBuffer::getFileSlice(const Twine &FilePath, uint64_t MapSize,
                            uint64_t Offset, bool IsVolatile) {
   return getFileAux<MemoryBuffer>(FilePath, -1, MapSize, Offset, false,
-                                  IsVolatile);
+                                  IsVolatile, false);
 }
 
 //===----------------------------------------------------------------------===//
 // MemoryBuffer::getFile implementation.
 //===----------------------------------------------------------------------===//
 
 namespace {
 
for (;;) {
     if (*ReadBytes == 0)
       break;
     Buffer.set_size(Buffer.size() + *ReadBytes);
   }
 
   return getMemBufferCopyImpl(Buffer, BufferName);
 }
 
 
 ErrorOr<std::unique_ptr<MemoryBuffer>>
 MemoryBuffer::getFile(const Twine &Filename, int64_t FileSize,
-                      bool RequiresNullTerminator, bool IsVolatile) {
+                      bool RequiresNullTerminator, bool IsVolatile,
+                      bool IsText) {
   return getFileAux<MemoryBuffer>(Filename, FileSize, FileSize, 0,
-                                  RequiresNullTerminator, IsVolatile);
+                                  RequiresNullTerminator, IsVolatile, IsText);
 }
 
 template <typename MB>
 static ErrorOr<std::unique_ptr<MB>>
 getOpenFileImpl(sys::fs::file_t FD, const Twine &Filename, uint64_t FileSize,
                 uint64_t MapSize, int64_t Offset, bool RequiresNullTerminator,
                 bool IsVolatile);
 
 template <typename MB>
 static ErrorOr<std::unique_ptr<MB>>
 getFileAux(const Twine &Filename, int64_t FileSize, uint64_t MapSize,
-           uint64_t Offset, bool RequiresNullTerminator, bool IsVolatile) {
+           uint64_t Offset, bool RequiresNullTerminator, bool IsVolatile,
+           bool IsText) {
   Expected<sys::fs::file_t> FDOrErr =
-      sys::fs::openNativeFileForRead(Filename, sys::fs::OF_None);
+      sys::fs::openNativeFileForRead(Filename, IsText ? sys::fs::OF_Text
+                                                      : sys::fs::OF_None);
   if (!FDOrErr)
     return errorToErrorCode(FDOrErr.takeError());
   sys::fs::file_t FD = *FDOrErr;
   auto Ret = getOpenFileImpl<MB>(FD, Filename, FileSize, MapSize, Offset,
                                  RequiresNullTerminator, IsVolatile);
   sys::fs::closeFile(FD);
   return Ret;
 }
 
 ErrorOr<std::unique_ptr<WritableMemoryBuffer>>
 WritableMemoryBuffer::getFile(const Twine &Filename, int64_t FileSize,
                               bool IsVolatile) {
   return getFileAux<WritableMemoryBuffer>(Filename, FileSize, FileSize, 0,
                                           /*RequiresNullTerminator*/ false,
-                                          IsVolatile);
+                                          IsVolatile, false);
 }
 
 ErrorOr<std::unique_ptr<WritableMemoryBuffer>>
 WritableMemoryBuffer::getFileSlice(const Twine &Filename, uint64_t MapSize,
                                    uint64_t Offset, bool IsVolatile) {
   return getFileAux<WritableMemoryBuffer>(Filename, -1, MapSize, Offset, false,
-                                          IsVolatile);
+                                          IsVolatile, false);
 }
 
 std::unique_ptr<WritableMemoryBuffer>
 WritableMemoryBuffer::getNewUninitMemBuffer(size_t Size, const Twine &BufferName) {
   using MemBuffer = MemoryBufferMem<WritableMemoryBuffer>;
   // Allocate space for the MemoryBuffer, the data and the name. It is important
   // that MemoryBuffer and data are aligned so PointerIntPair works with them.
   // TODO: Is 16-byte alignment enough?  We copy small object files with large

llvm/lib/Support/Path.cpp

void createUniquePath(const Twine &Model, SmallVectorImpl<char> &ResultPath,
   for (unsigned i = 0, e = ModelStorage.size(); i != e; ++i) {
     if (ModelStorage[i] == '%')
       ResultPath[i] = "0123456789abcdef"[sys::Process::GetRandomNumber() & 15];
   }
 }
 
 std::error_code createUniqueFile(const Twine &Model, int &ResultFd,
                                  SmallVectorImpl<char> &ResultPath,
-                                 unsigned Mode) {
-  return createUniqueEntity(Model, ResultFd, ResultPath, false, Mode, FS_File);
-}
-
-static std::error_code createUniqueFile(const Twine &Model, int &ResultFd,
-                                        SmallVectorImpl<char> &ResultPath,
-                                        unsigned Mode, OpenFlags Flags) {
   return createUniqueEntity(Model, ResultFd, ResultPath, false, Mode, FS_File,
                             Flags);
 }
 
 std::error_code createUniqueFile(const Twine &Model,
                                  SmallVectorImpl<char> &ResultPath,
                                  unsigned Mode) {
   int FD;
   auto EC = createUniqueFile(Model, FD, ResultPath, Mode);
   if (EC)
     return EC;
   // FD is only needed to avoid race conditions. Close it right away.
   close(FD);
   return EC;
 }
 
 static std::error_code
 createTemporaryFile(const Twine &Model, int &ResultFD,
-                    llvm::SmallVectorImpl<char> &ResultPath, FSEntity Type) {
+                    llvm::SmallVectorImpl<char> &ResultPath, FSEntity Type,
+                    sys::fs::OpenFlags Flags = sys::fs::OF_None) {
   SmallString<128> Storage;
   StringRef P = Model.toNullTerminatedStringRef(Storage);
   assert(P.find_first_of(separators(Style::native)) == StringRef::npos &&
          "Model must be a simple filename.");
   // Use P.begin() so that createUniqueEntity doesn't need to recreate Storage.
   return createUniqueEntity(P.begin(), ResultFD, ResultPath, true,
-                            owner_read | owner_write, Type);
+                            owner_read | owner_write, Type, Flags);
 }
 
 static std::error_code
 createTemporaryFile(const Twine &Prefix, StringRef Suffix, int &ResultFD,
-                    llvm::SmallVectorImpl<char> &ResultPath, FSEntity Type) {
+                    llvm::SmallVectorImpl<char> &ResultPath, FSEntity Type,
+                    sys::fs::OpenFlags Flags = sys::fs::OF_None) {
   const char *Middle = Suffix.empty() ? "-%%%%%%" : "-%%%%%%.";
   return createTemporaryFile(Prefix + Middle + Suffix, ResultFD, ResultPath,
-                             Type);
+                             Type, Flags);
 }
 
 std::error_code createTemporaryFile(const Twine &Prefix, StringRef Suffix,
                                     int &ResultFD,
-                                    SmallVectorImpl<char> &ResultPath) {
-  return createTemporaryFile(Prefix, Suffix, ResultFD, ResultPath, FS_File);
+                                    SmallVectorImpl<char> &ResultPath,
+                                    sys::fs::OpenFlags Flags) {
+  return createTemporaryFile(Prefix, Suffix, ResultFD, ResultPath, FS_File,
+                             Flags);
 }
 
 std::error_code createTemporaryFile(const Twine &Prefix, StringRef Suffix,
-                                    SmallVectorImpl<char> &ResultPath) {
+                                    SmallVectorImpl<char> &ResultPath,
+                                    sys::fs::OpenFlags Flags) {
   int FD;
-  auto EC = createTemporaryFile(Prefix, Suffix, FD, ResultPath);
+  auto EC = createTemporaryFile(Prefix, Suffix, FD, ResultPath, Flags);
   if (EC)
     return EC;
   // FD is only needed to avoid race conditions. Close it right away.
   close(FD);
   return EC;
 }
 
 
 // This is a mkdtemp with a different pattern. We use createUniqueEntity mostly
 // for consistency. We should try using mkdtemp.
 std::error_code createUniqueDirectory(const Twine &Prefix,
                                       SmallVectorImpl<char> &ResultPath) {
   int Dummy;
   return createUniqueEntity(Prefix + "-%%%%%%", Dummy, ResultPath, true, 0,
                             FS_Dir);
 }

llvm/lib/Support/ToolOutputFile.cpp

 //===--- ToolOutputFile.cpp - Implement the ToolOutputFile class --------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This implements the ToolOutputFile class.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Support/ToolOutputFile.h"
+#include "llvm/ADT/Triple.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Signals.h"
 using namespace llvm;
 
 static bool isStdout(StringRef Filename) { return Filename == "-"; }
 
 ToolOutputFile::CleanupInstaller::CleanupInstaller(StringRef Filename)
     : Filename(std::string(Filename)), Keep(false) {

 ToolOutputFile::ToolOutputFile(StringRef Filename, std::error_code &EC,
                                sys::fs::OpenFlags Flags)
     : Installer(Filename) {
   if (isStdout(Filename)) {
     OS = &outs();
     EC = std::error_code();
     return;
   }
-  OSHolder.emplace(Filename, EC, Flags);
+
+  llvm::Triple HostTriple(LLVM_HOST_TRIPLE);
+  // On Windows, we set the OF_None flag even for text files to avoid
+  // CRLF translation.
+  OSHolder.emplace(Filename, EC,
+                   HostTriple.isOSWindows() ? sys::fs::OF_None : Flags);
   OS = OSHolder.getPointer();
   // If open fails, no cleanup is needed.
   if (EC)
     Installer.Keep = true;
 }
 
 ToolOutputFile::ToolOutputFile(StringRef Filename, int FD)
     : Installer(Filename) {
   OSHolder.emplace(FD, true);
   OS = OSHolder.getPointer();
 }

llvm/lib/TableGen/Main.cpp

 ///
 /// This functionality is really only for the benefit of the build system.
 /// It is similar to GCC's `-M*` family of options.
 static int createDependencyFile(const TGParser &Parser, const char *argv0) {
   if (OutputFilename == "-")
     return reportError(argv0, "the option -d must be used together with -o\n");
 
   std::error_code EC;
-  ToolOutputFile DepOut(DependFilename, EC, sys::fs::OF_None);
+  ToolOutputFile DepOut(DependFilename, EC, sys::fs::OF_Text);
   if (EC)
     return reportError(argv0, "error opening " + DependFilename + ":" +
                                   EC.message() + "\n");
   DepOut.os() << OutputFilename << ":";
   for (const auto &Dep : Parser.getDependencies()) {
     DepOut.os() << ' ' << Dep;
   }
   DepOut.os() << "\n";
   DepOut.keep();
   return 0;
 }
 
 int llvm::TableGenMain(const char *argv0, TableGenMainFn *MainFn) {
   RecordKeeper Records;
 
   if (TimePhases)
     Records.startPhaseTiming();
 
   // Parse the input file.
 
   Records.startTimer("Parse, build records");
   ErrorOr<std::unique_ptr<MemoryBuffer>> FileOrErr =
-      MemoryBuffer::getFileOrSTDIN(InputFilename);
+      MemoryBuffer::getFileOrSTDIN(InputFilename, -1, true, true);
   if (std::error_code EC = FileOrErr.getError())
     return reportError(argv0, "Could not open input file '" + InputFilename +
                                   "': " + EC.message() + "\n");
 
   Records.saveInputFilename(InputFilename);
 
   // Tell SrcMgr about this buffer, which is what TGParser will pick up.
   SrcMgr.AddNewSourceBuffer(std::move(*FileOrErr), SMLoc());
int llvm::TableGenMain(const char *argv0, TableGenMainFn *MainFn) {
   }
 
   Records.startTimer("Write output");
   bool WriteFile = true;
   if (WriteIfChanged) {
     // Only updates the real output file if there are any differences.
     // This prevents recompilation of all the files depending on it if there
     // aren't any.
-    if (auto ExistingOrErr = MemoryBuffer::getFile(OutputFilename))
+    if (auto ExistingOrErr = MemoryBuffer::getFile(OutputFilename, -1,
+                                                   true, false, true))
       if (std::move(ExistingOrErr.get())->getBuffer() == Out.str())
         WriteFile = false;
   }
   if (WriteFile) {
     std::error_code EC;
-    ToolOutputFile OutFile(OutputFilename, EC, sys::fs::OF_None);
+    ToolOutputFile OutFile(OutputFilename, EC, sys::fs::OF_Text);
     if (EC)
       return reportError(argv0, "error opening " + OutputFilename + ": " +
                                     EC.message() + "\n");
     OutFile.os() << Out.str();
     if (ErrorsPrinted == 0)
       OutFile.keep();
   }
   
   Records.stopTimer();
   Records.stopPhaseTiming();
 
   if (ErrorsPrinted > 0)
     return reportError(argv0, Twine(ErrorsPrinted) + " errors.\n");
   return 0;
 }

llvm/utils/FileCheck/FileCheck.cpp

errs() << "Unable to combine check-prefix strings into a prefix regular "
            << REError << "\n";
     return 2;
   }
 
   SourceMgr SM;
 
   // Read the expected strings from the check file.
   ErrorOr<std::unique_ptr<MemoryBuffer>> CheckFileOrErr =
-      MemoryBuffer::getFileOrSTDIN(CheckFilename);
+      MemoryBuffer::getFileOrSTDIN(CheckFilename, /*FileSize*/ -1,
+                                   /*RequiresNullTerminator*/ true,
+                                   /*IsText*/ true);
   if (std::error_code EC = CheckFileOrErr.getError()) {
     errs() << "Could not open check file '" << CheckFilename
            << "': " << EC.message() << '\n';
     return 2;
   }
   MemoryBuffer &CheckFile = *CheckFileOrErr.get();
 
   SmallString<4096> CheckFileBuffer;
   StringRef CheckFileText = FC.CanonicalizeFile(CheckFile, CheckFileBuffer);
 
   unsigned CheckFileBufferID =
       SM.AddNewSourceBuffer(MemoryBuffer::getMemBuffer(
                                 CheckFileText, CheckFile.getBufferIdentifier()),
                             SMLoc());
 
   std::pair<unsigned, unsigned> ImpPatBufferIDRange;
   if (FC.readCheckFile(SM, CheckFileText, PrefixRE, &ImpPatBufferIDRange))
     return 2;
 
   // Open the file to check and add it to SourceMgr.
   ErrorOr<std::unique_ptr<MemoryBuffer>> InputFileOrErr =
-      MemoryBuffer::getFileOrSTDIN(InputFilename);
+      MemoryBuffer::getFileOrSTDIN(InputFilename, /*FileSize*/ -1,
+                                   /*RequiresNullTerminator*/ true,
+                                   /*IsText*/ true);
   if (InputFilename == "-")
     InputFilename = "<stdin>"; // Overwrite for improved diagnostic messages
   if (std::error_code EC = InputFileOrErr.getError()) {
     errs() << "Could not open input file '" << InputFilename
            << "': " << EC.message() << '\n';
     return 2;
   }
   MemoryBuffer &InputFile = *InputFileOrErr.get();