Set stdout/stdin to binary mode on Windows

[lanza]

Summary

A file opened in text mode on Windows will have \n automatically changed to 13,10 while Darwin and Linux leave it as 10.

Set the file to binary mode to avoid this automatic conversion so that Darwin, Linux and Windows have equivalent treatment of \r.

[clayborg]

We might need to open the file in binary mode to avoid \n from being converted to two characters in the JSON payload itself. So maybe this fix should be that we open the file in binary mode?

[lanza]

swap to just changing stdout & stdin

[clayborg]

The VSCode.txt part looks good without the ifdef to set binary mode. Are you gonna make another patch with just that fix?

tools/lldb-vscode/VSCode.cpp

47–50

No need for ifdef here. Just do it for all platforms.

[lanza]

rebase properly

[lanza]

No need for ifdef here. Just do it for all platforms.

So the BSD/Darwin setmode is not the same as the Windows _setmode (e.g. only one argument vs two on Windows). Linux doesn't even have a setmode.

The VSCode.txt part looks good without the ifdef to set binary mode. Are you gonna make another patch with just that fix?

I'm not sure what you mean by this. I messed up the rebase on the previous revision, maybe you're talking about that?

[clayborg]

Be sure to add a comment specifying why we are doing this only for windows. We might be able to remove the #if and do it for all platforms.

tools/lldb-vscode/VSCode.cpp

47

Add a comment specifying why we are doing this.

[lanza]

add a comment

tools/lldb-vscode/VSCode.cpp

 //===-- VSCode.cpp ----------------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include <stdarg.h>
 #include <fstream>
 #include <mutex>
 
 #include "VSCode.h"
 #include "LLDBUtils.h"
 
+#if defined(_WIN32)
+#include <io.h>
+#include <fcntl.h>
+#endif
+
 using namespace lldb_vscode;
 
 namespace {
   inline bool IsEmptyLine(llvm::StringRef S) {
     return S.ltrim().empty();
   }
 } // namespace
 
: in(stdin), out(stdout), launch_info(nullptr), variables(),
                   {"cpp_throw", "C++ Throw", lldb::eLanguageTypeC_plus_plus},
                   {"objc_catch", "Objective C Catch", lldb::eLanguageTypeObjC},
                   {"objc_throw", "Objective C Throw", lldb::eLanguageTypeObjC},
                   {"swift_catch", "Swift Catch", lldb::eLanguageTypeSwift},
                   {"swift_throw", "Swift Throw", lldb::eLanguageTypeSwift}}),
       focus_tid(LLDB_INVALID_THREAD_ID), sent_terminated_event(false),
       stop_at_entry(false) {
   const char *log_file_path = getenv("LLDBVSCODE_LOG");
+#if defined(_WIN32)
+// Windows opens stdout and stdin in text mode which converts \n to 13,10
+// while the value is just 10 on Darwin/Linux. Setting the file mode to binary
+// fixes this.
+  assert(_setmode(fileno(stdout), _O_BINARY));
+  assert(_setmode(fileno(stdin), _O_BINARY));
+#endif
   if (log_file_path)
     log.reset(new std::ofstream(log_file_path));
 }
 
 VSCode::~VSCode() {
   CloseInputStream();
   CloseOutputStream();
 }