Set console mode when -fansi-escape-codes is enabled

[xbolva00]

Summary

Windows console now supports supports ANSI escape codes, but we need to enable it using SetConsoleMode with ENABLE_VIRTUAL_TERMINAL_PROCESSING flag.

Fixes https://bugs.llvm.org/show_bug.cgi?id=38817

Tested on Windows 10, screenshot:
https://i.imgur.com/bqYq0Uy.png

[xbolva00]

• Addressed comments from PR38817

[zturner]

lib/Support/Windows/Process.inc

331–337

I was going to comment how this looks all wrong because it only operates on stdout (e.g. not stderr) and doesn't seem to notice the case when stdout is not a terminal. But I looked the rest of the code in this file and it's all broken as well. Anyway, I would write this as:

#if defined(ENABLE_VIRTUAL_TERMINAL_PROCESSING)
  if (enable) {
    HANDLE hConsole = GetStdHandle(STD_OUTPUT_HANDLE);
    DWORD consoleMode;
    GetConsoleMode(hConsole, &consoleMode);
    consoleMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
    SetConsoleMode(hConsole, consoleMode);
  }
#endif

If they aren't using the appropriate Windows SDK, they don't get the feature.

lib/Support/Windows/WindowsSupport.h

37–39

If someone wants a copy of LLDB that supports features specific to Windows 10, the supported workflow is that they should be building using the Windows SDK that has those features. So I'd rather remove this, and guard the code that uses this flag with an #ifdef.

[zturner]

lib/Support/Windows/Process.inc

332

We don't use hungarian notation. Also all variables begin with capital letters.

HANDLE Console = GetStdHandle(STD_OUTPUT_HANDLE);
DWORD Mode;
GetConsoleMode(Console, &Mode);
Mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
SetConsoleMode(Console, Mode).

[xbolva00]

• Addressed comment by @zturner

lib/Support/Windows/Process.inc

 }
 
 bool Process::StandardErrHasColors() {
   return FileDescriptorHasColors(2);
 }
 
 static bool UseANSI = false;
 void Process::UseANSIEscapeCodes(bool enable) {
-  UseANSI = enable;
+  if (enable) {
+    HANDLE hConsole = GetStdHandle(STD_OUTPUT_HANDLE);
+    DWORD consoleMode;
+    GetConsoleMode(hConsole, &consoleMode);
+    consoleMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
+    UseANSI = SetConsoleMode(hConsole, consoleMode);
+  } else {
+    UseANSI = false;
+  }
 }
 
 namespace {
 class DefaultColors
 {
   private:
     WORD defaultColor;
   public:

lib/Support/Windows/WindowsSupport.h

 // Require at least Windows 7 API.
 #define _WIN32_WINNT 0x0601
 #define _WIN32_IE    0x0800 // MinGW at it again. FIXME: verify if still needed.
 #define WIN32_LEAN_AND_MEAN
 #ifndef NOMINMAX
 #define NOMINMAX
 #endif
 
+#ifndef ENABLE_VIRTUAL_TERMINAL_PROCESSING
+#define ENABLE_VIRTUAL_TERMINAL_PROCESSING 0x0004
+#endif
+
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Config/config.h" // Get build system configuration settings
 #include "llvm/Support/Chrono.h"
 #include "llvm/Support/Compiler.h"
 #include <cassert>