Disable use-color if the output stream is not a terminal with color support.

[teemperor]

Summary

LLDB currently only checks the output terminal for color support by looking
at the TERM environment variable and comparing it to "dumb". This causes that
when running LLDB on a CI node, the syntax highlighter will not be deactivated by
LLDB and the output log is filled with color codes (unless the terminal emulator
actually exposes itself as dumb).

This patch now relies on the LLVM code for detecting color support which is more
reliable. We now also correctly actually initialize the m_supports_colors variable in File.
m_supports_colors was so far uninitialized, but the code path that uses m_supports_colors
was also dead so the sanitizers didn't sound an alarm.

The old check that compares TERM is not removed by this patch as the new LLVM code
doesn't seem to handle this case (and it's a good thing to check for "dumb" terminals).

[aprantl]

source/Core/Debugger.cpp

805 ↗

(On Diff #162510)

Shouldn't this check be obsolete now?

805 ↗

(On Diff #162510)

You can probably test that by running env TERM=dumb lldb

[teemperor]

source/Core/Debugger.cpp

805 ↗

(On Diff #162510)

LLVM doesn't seem to check for this environment variable from what I see, so the new code doesn't include this functionality. And it's technically a good thing to check for this, so I think we should let it stay until LLVM also adds this feature.

[teemperor]

source/Core/Debugger.cpp

805 ↗

(On Diff #162510)

Just tested it, and the LLVM code indeed ignored TERM=dumb. I'll fix this in LLVM and then we see what people think about this (and if it goes in, we can remove it here).

lldb/trunk/include/lldb/Host/File.h

public:
   };
 
   static mode_t ConvertOpenOptionsForPOSIXOpen(uint32_t open_options);
 
   File()
       : IOObject(eFDTypeFile, false), m_descriptor(kInvalidDescriptor),
         m_stream(kInvalidStream), m_options(0), m_own_stream(false),
         m_is_interactive(eLazyBoolCalculate),
-        m_is_real_terminal(eLazyBoolCalculate) {}
+        m_is_real_terminal(eLazyBoolCalculate),
+        m_supports_colors(eLazyBoolCalculate) {}
 
   File(FILE *fh, bool transfer_ownership)
       : IOObject(eFDTypeFile, false), m_descriptor(kInvalidDescriptor),
         m_stream(fh), m_options(0), m_own_stream(transfer_ownership),
         m_is_interactive(eLazyBoolCalculate),
-        m_is_real_terminal(eLazyBoolCalculate) {}
+        m_is_real_terminal(eLazyBoolCalculate),
+        m_supports_colors(eLazyBoolCalculate) {}
 
   //------------------------------------------------------------------
   /// Constructor with path.
   ///
   /// Takes a path to a file which can be just a filename, or a full path. If
   /// \a path is not nullptr or empty, this function will call File::Open
   /// (const char *path, uint32_t options, uint32_t permissions).
   ///

lldb/trunk/lit/Settings/TestDisableColor.test

+# RUN: %lldb -x -b -s %s | FileCheck %s
+settings show use-color
+q
+# This tests that LLDB turns off use-color if the output file is not an
+# interactive terminal. In this example, use-color should be off because LLDB
+# is run just by the non-interactive lit test runner.
+# CHECK: use-color (boolean) = false

lldb/trunk/lit/Settings/lit.local.cfg

+config.suffixes = ['.test']

lldb/trunk/source/Core/Debugger.cpp

OptionValueSInt64 *term_width =
           nullptr, ePropertyTerminalWidth);
   term_width->SetMinimumValue(10);
   term_width->SetMaximumValue(1024);
 
   // Turn off use-color if this is a dumb terminal.
   const char *term = getenv("TERM");
   if (term && !strcmp(term, "dumb"))
     SetUseColor(false);
+  // Turn off use-color if we don't write to a terminal with color support.
+  if (!m_output_file_sp->GetFile().GetIsTerminalWithColors())
+    SetUseColor(false);
 }
 
 Debugger::~Debugger() { Clear(); }
 
 void Debugger::Clear() {
   //----------------------------------------------------------------------
   // Make sure we call this function only once. With the C++ global destructor
   // chain having a list of debuggers and with code that can be running on