[Windows] Convert from UTF-8 to UTF-16 when writing to a Windows console

[rnk]

Summary

Calling WriteConsoleW is the most reliable way to print Unicode
characters to a Windows console.

If binary data gets printed to the console, attempting to re-encode it
shouldn't be a problem, since garbage in can produce garbage out.

This breaks printing strings in the local codepage, which WriteConsoleA
knows how to handle. For example, this can happen when user source code
is encoded with the local codepage, and an LLVM tool quotes it while
emitting a caret diagnostic. This is unfortunate, but well-behaved tools
should validate that their input is UTF-8 and escape non-UTF-8
characters before sending them to raw_fd_ostream. Clang already does
this, but not all LLVM tools do this.

One drawback to the current implementation is printing a string a byte
at a time doesn't work. Consider this LLVM code:

for (char C : MyStr) outs() << C;

Because outs() is now unbuffered, we wil try to convert each byte to
UTF-16, which will fail. However, this already didn't work, so I think
we may as well update callers that do that as we find them to print
complete portions of strings. You can see a real example of this in my
patch to SourceMgr.cpp

Fixes PR38669 and PR36267.

[zturner]

llvm/lib/Support/raw_ostream.cpp

572 ↗

(On Diff #163593)

You can use llvm::sys::Process::FileDescriptorIsDisplayed(fd) here. Also, no need to put it behind the #ifdef, the member variable can be set correctly regardless of platform.

632 ↗

(On Diff #163593)

StringRef Data?

755–780 ↗

(On Diff #163593)

Given that we already have this IsConsole variable, combined with my suggestion above to set the variable to a real value for every platform, can we re-write this entire function:

if (IsConsole)
  return 0;

#if !defined(__minix)
  struct stat statbuf;
  if (fstat(FD, &statbuf) != 0)
    return 0;
  return statbuf.st_blksize.
#else
  return raw_ostream::preferred_buffer_size();
#endif

[rnk]

llvm/lib/Support/raw_ostream.cpp

572 ↗

(On Diff #163593)

We need to differentiate between the case where the output is being displayed in mintty and a Windows console. FileDescriptorIsDisplayed() could reasonably return true for mintty, which uses pipes, and then we would try to use WriteConsoleW on a pipe. It's true that today FileDescriptorIsDisplayed will return false inside mintty, but someone might come along and fix that bug at some point.

I really want to be explicit that this is just for Windows console devices, and not some general cross-platform "is displayed" boolean, which is why the variable is IsConsole and not IsATTy. WDYT?

632 ↗

(On Diff #163593)

Sure, but it just moves it to the caller.

755–780 ↗

(On Diff #163593)

If we do cache tty-ness, we could do that, but that's not what IsConsole is.

[rnk]

• stringref

[zturner]

llvm/lib/Support/raw_ostream.cpp

650–652 ↗

(On Diff #163595)

Can you test this with the following?

outs() << "Test\nTest2";

I'm wondering if the LF will be handled properly. I think part of what ::write does is to transcode LF to CRLF. So if we're directly calling WriteConsole, we might not get this for free anymore.

572 ↗

(On Diff #163593)

In that case call the variable IsWindowsConsole perhaps.

632 ↗

(On Diff #163593)

Well sure, but my next question would be "why doesn't write_impl take a StringRef?" Seems like something that should keep pushing up the call stack as high as possible (Not that I'm asking you to make that change, but it just seems like the right api, despite the fact that the caller has what is, IMO, a buggy API).

[rsmith]

llvm/lib/Support/raw_ostream.cpp

636 ↗

(On Diff #163595)

Just out of curiosity: what's the difference between this and llvm::convertUTF8ToUTF16String?

650 ↗

(On Diff #163595)

I think you want &WideText[WCharWritten] here.

[zturner]

llvm/lib/Support/raw_ostream.cpp

636 ↗

(On Diff #163595)

The main difference is that sys::windows::UTF8ToUTF16 calls a native Windows API to do the conversion. Documentation for the Windows API claims to produce conformant UTF-16 in all cases, but historically people have still tried to use the Windows API whenever possible because there have been issues in the past where Windows was *almost* UTF-16 but not quite. I think those days are gone, but people still do it.

The other difference is that convertUTF8ToUTF16String returns a vector<unsigned short>, whereas all of the windows APIs take const wchar_t*. So we'd need a reintepret_cast before calling the actual Windows API, which is a little annoying. (Incidentally, even *that* function should probably be changed to return a vector<char16_t>.

[rnk]

llvm/lib/Support/raw_ostream.cpp

650 ↗

(On Diff #163595)

Thanks. Unfortunately, this isn't testable without faking up some kind of console.

650–652 ↗

(On Diff #163595)

I think the Windows console doesn't care if it gets \r\n or \n at this point. I don't think there's a way I can observe a difference in behavior now, since I can't WriteConsole to a pipe. =/

572 ↗

(On Diff #163593)

Done, thanks, that was something I thought about but didn't write down.

[rnk]

• rename variable

[zturner]

llvm/lib/Support/raw_ostream.cpp

650–652 ↗

(On Diff #163595)

Just put some code in main that calls the above with your change and run the program. See what it outputs?

[rnk]

llvm/lib/Support/raw_ostream.cpp

650–652 ↗

(On Diff #163595)

Right, I compiled and ran a program that does outs() << "Test\nTest2"; and the newlines show up. The Windows console doesn't need CRLF, it treats LF as implicitly returning the cursor to the beginning of the next line.

llvm/trunk/include/llvm/Support/raw_ostream.h

 //===----------------------------------------------------------------------===//
 
 /// A raw_ostream that writes to a file descriptor.
 ///
 class raw_fd_ostream : public raw_pwrite_stream {
   int FD;
   bool ShouldClose;
 
+  bool SupportsSeeking;
+
+  /// True if this fd refers to a Windows console device. Mintty and other
+  /// terminal emulators are TTYs, but they are not consoles.
+  bool IsWindowsConsole = false;
+
   std::error_code EC;
 
   uint64_t pos;
 
-  bool SupportsSeeking;
-
   /// See raw_ostream::write_impl.
   void write_impl(const char *Ptr, size_t Size) override;
 
   void pwrite_impl(const char *Ptr, size_t Size, uint64_t Offset) override;
 
   /// Return the current position within the stream, not counting the bytes
   /// currently in the buffer.
   uint64_t current_pos() const override { return pos; }

llvm/trunk/lib/Support/Locale.cpp

 #include "llvm/Support/Locale.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Unicode.h"
 
 namespace llvm {
 namespace sys {
 namespace locale {
 
 int columnWidth(StringRef Text) {
-#ifdef _WIN32
-  return Text.size();
-#else
   return llvm::sys::unicode::columnWidthUTF8(Text);
-#endif
 }
 
 bool isPrint(int UCS) {
-#ifdef _WIN32
-  // Restrict characters that we'll try to print to the lower part of ASCII
-  // except for the control characters (0x20 - 0x7E). In general one can not
-  // reliably output code points U+0080 and higher using narrow character C/C++
-  // output functions in Windows, because the meaning of the upper 128 codes is
-  // determined by the active code page in the console.
-  return ' ' <= UCS && UCS <= '~';
-#else
   return llvm::sys::unicode::isPrintable(UCS);
-#endif
 }
 
 } // namespace locale
 } // namespace sys
 } // namespace llvm

llvm/trunk/lib/Support/SourceMgr.cpp

for (ArrayRef<SMFixIt>::iterator I = FixIts.begin(), E = FixIts.end();
 
     std::fill(&CaretLine[FirstCol], &CaretLine[LastCol], '~');
   }
 }
 
 static void printSourceLine(raw_ostream &S, StringRef LineContents) {
   // Print out the source line one character at a time, so we can expand tabs.
   for (unsigned i = 0, e = LineContents.size(), OutCol = 0; i != e; ++i) {
-    if (LineContents[i] != '\t') {
-      S << LineContents[i];
-      ++OutCol;
-      continue;
+    size_t NextTab = LineContents.find('\t', i);
+    // If there were no tabs left, print the rest, we are done.
+    if (NextTab == StringRef::npos) {
+      S << LineContents.drop_front(i);
+      break;
     }
 
+    // Otherwise, print from i to NextTab.
+    S << LineContents.slice(i, NextTab);
+    OutCol += NextTab - i;
+    i = NextTab;
+
     // If we have a tab, emit at least one space, then round up to 8 columns.
     do {
       S << ' ';
       ++OutCol;
     } while ((OutCol % TabStop) != 0);
   }
   S << '\n';
 }

llvm/trunk/lib/Support/raw_ostream.cpp

 # define STDOUT_FILENO 1
 #endif
 #ifndef STDERR_FILENO
 # define STDERR_FILENO 2
 #endif
 #endif
 
 #ifdef _WIN32
+#include "llvm/Support/ConvertUTF.h"
 #include "Windows/WindowsSupport.h"
 #endif
 
 using namespace llvm;
 
 raw_ostream::~raw_ostream() {
   // raw_ostream's subclasses should take care to flush the buffer
   // in their destructors.
raw_fd_ostream::raw_fd_ostream(int fd, bool shouldClose, bool unbuffered)
   // property that tools that support writing file to stdout should not also
   // write informational output to stdout, but in practice we were never able to
   // maintain this invariant. Many features have been added to LLVM and clang
   // (-fdump-record-layouts, optimization remarks, etc) that print to stdout, so
   // users must simply be aware that mixed output and remarks is a possibility.
   if (FD <= STDERR_FILENO)
     ShouldClose = false;
 
+#ifdef _WIN32
+  // Check if this is a console device. This is not equivalent to isatty.
+  IsWindowsConsole =
+      ::GetFileType((HANDLE)::_get_osfhandle(fd)) == FILE_TYPE_CHAR;
+#endif
+
   // Get the starting position.
   off_t loc = ::lseek(FD, 0, SEEK_CUR);
 #ifdef _WIN32
   // MSVCRT's _lseek(SEEK_CUR) doesn't return -1 for pipes.
   sys::fs::file_status Status;
   std::error_code EC = status(FD, Status);
   SupportsSeeking = !EC && Status.type() == sys::fs::file_type::regular_file;
 #else
#endif
   // to avoid report_fatal_error calls should check for errors with
   // has_error() and clear the error flag with clear_error() before
   // destructing raw_ostream objects which may have errors.
   if (has_error())
     report_fatal_error("IO failure on output stream: " + error().message(),
                        /*GenCrashDiag=*/false);
 }
 
+#if defined(_WIN32)
+// The most reliable way to print unicode in a Windows console is with
+// WriteConsoleW. To use that, first transcode from UTF-8 to UTF-16. This
+// assumes that LLVM programs always print valid UTF-8 to the console. The data
+// might not be UTF-8 for two major reasons:
+// 1. The program is printing binary (-filetype=obj -o -), in which case it
+// would have been gibberish anyway.
+// 2. The program is printing text in a semi-ascii compatible codepage like
+// shift-jis or cp1252.
+//
+// Most LLVM programs don't produce non-ascii text unless they are quoting
+// user source input. A well-behaved LLVM program should either validate that
+// the input is UTF-8 or transcode from the local codepage to UTF-8 before
+// quoting it. If they don't, this may mess up the encoding, but this is still
+// probably the best compromise we can make.
+static bool write_console_impl(int FD, StringRef Data) {
+  SmallVector<wchar_t, 256> WideText;
+
+  // Fall back to ::write if it wasn't valid UTF-8.
+  if (auto EC = sys::windows::UTF8ToUTF16(Data, WideText))
+    return false;
+
+  // On Windows 7 and earlier, WriteConsoleW has a low maximum amount of data
+  // that can be written to the console at a time.
+  size_t MaxWriteSize = WideText.size();
+  if (!RunningWindows8OrGreater())
+    MaxWriteSize = 32767;
+
+  size_t WCharsWritten = 0;
+  do {
+    size_t WCharsToWrite =
+        std::min(MaxWriteSize, WideText.size() - WCharsWritten);
+    DWORD ActuallyWritten;
+    bool Success =
+        ::WriteConsoleW((HANDLE)::_get_osfhandle(FD), &WideText[WCharsWritten],
+                        WCharsToWrite, &ActuallyWritten,
+                        /*Reserved=*/nullptr);
+
+    // The most likely reason for WriteConsoleW to fail is that FD no longer
+    // points to a console. Fall back to ::write. If this isn't the first loop
+    // iteration, something is truly wrong.
+    if (!Success)
+      return false;
+
+    WCharsWritten += ActuallyWritten;
+  } while (WCharsWritten != WideText.size());
+  return true;
+}
+#endif
+
 void raw_fd_ostream::write_impl(const char *Ptr, size_t Size) {
   assert(FD >= 0 && "File already closed.");
   pos += Size;
 
+#if defined(_WIN32)
+  // If this is a Windows console device, try re-encoding from UTF-8 to UTF-16
+  // and using WriteConsoleW. If that fails, fall back to plain write().
+  if (IsWindowsConsole)
+    if (write_console_impl(FD, StringRef(Ptr, Size)))
+      return;
+#endif
+
   // The maximum write size is limited to INT32_MAX. A write
   // greater than SSIZE_MAX is implementation-defined in POSIX,
   // and Windows _write requires 32 bit input.
   size_t MaxWriteSize = INT32_MAX;
 
 #if defined(__linux__)
   // It is observed that Linux returns EINVAL for a very large write (>2G).
   // Make it a reasonably small value.
   MaxWriteSize = 1024 * 1024 * 1024;
-#elif defined(_WIN32)
-  // Writing a large size of output to Windows console returns ENOMEM. It seems
-  // that, prior to Windows 8, WriteFile() is redirecting to WriteConsole(), and
-  // the latter has a size limit (66000 bytes or less, depending on heap usage).
-  if (::_isatty(FD) && !RunningWindows8OrGreater())
-    MaxWriteSize = 32767;
 #endif
 
   do {
     size_t ChunkSize = std::min(Size, MaxWriteSize);
     ssize_t ret = ::write(FD, Ptr, ChunkSize);
 
     if (ret < 0) {
       // If it's a recoverable error, swallow it and retry the write.
void raw_fd_ostream::pwrite_impl(const char *Ptr, size_t Size,
                                  uint64_t Offset) {
   uint64_t Pos = tell();
   seek(Offset);
   write(Ptr, Size);
   seek(Pos);
 }
 
 size_t raw_fd_ostream::preferred_buffer_size() const {
-#if !defined(_MSC_VER) && !defined(__MINGW32__) && !defined(__minix)
-  // Windows and Minix have no st_blksize.
+#if defined(_WIN32)
+  // Disable buffering for console devices. Console output is re-encoded from
+  // UTF-8 to UTF-16 on Windows, and buffering it would require us to split the
+  // buffer on a valid UTF-8 codepoint boundary. Terminal buffering is disabled
+  // below on most other OSs, so do the same thing on Windows and avoid that
+  // complexity.
+  if (IsWindowsConsole)
+    return 0;
+  return raw_ostream::preferred_buffer_size();
+#elif !defined(__minix)
+  // Minix has no st_blksize.
   assert(FD >= 0 && "File not yet open!");
   struct stat statbuf;
   if (fstat(FD, &statbuf) != 0)
     return 0;
 
   // If this is a terminal, don't use buffering. Line buffering
   // would be a more traditional thing to do, but it's not worth
   // the complexity.