Implement workaround for issue that shows between libedit and low-level terminal routines on Linux

[augusto2112]

Summary

On some Linux distributions, after setting up the EditLine object, the eventual call to set_curterm, which happens when calculating terminal properties, breaks the libedit configuration, causing characters that have functions bound to them not to show up. As a workaround, we calculate these terminal properties eagerly, before initializing the EditLine object.

[augusto2112]

This issue was found on the Swift REPL. However, I tested @teemperor's C REPL patch (https://reviews.llvm.org/D87281) and the same problem shows up.

[labath]

Can you be more specific about what is the nature and cause of this breakage?

Could it maybe be the fact that the set_curterm(nullptr) call (inside terminalHasColors(fd)) leaves terminfo with no "current" terminal, (whereas before the call it contained the "current" terminal, as setup by libedit? If so then maybe the right fix is to change terminalHasColors to restore the current terminal to whatever it was at the start of the function.

[augusto2112]

@labath, you were absolutely correct! It was simply a matter of saving and restoring the terminal struct on the terminalHasColors function in the Process.inc file (I really should've tried that before). I'm currently recompiling and will re-run the tests locally, and will push the changes after that. I do worry this could potentially impact macOS though (I don't know if these low-level terminal functions work differently between differently OSes), so how do we ensure this doesn't break anything there?

Also, since the patch will be completely different, should I open a new patch or push to this one and just change the title/description accordingly?

[labath]

In D95165#2515232, @augusto2112 wrote:

@labath, you were absolutely correct! It was simply a matter of saving and restoring the terminal struct on the terminalHasColors function in the Process.inc file (I really should've tried that before). I'm currently recompiling and will re-run the tests locally, and will push the changes after that. I do worry this could potentially impact macOS though (I don't know if these low-level terminal functions work differently between differently OSes), so how do we ensure this doesn't break anything there?

We should pick reviewers which have some knowledge of this -- the original author of that code + whoever reviewed that patch is a good starting point. If this breaks anyone's use case after it gets reviewed&committed, that person will surely let us know (and then we can figure out what to do). However, I don't think this will be a particularly risky change. The more interesting question is whether there is any reasonable way of testing this....

Also, since the patch will be completely different, should I open a new patch or push to this one and just change the title/description accordingly?

A new patch would probably be better in this case. However, it might be a good idea to add a reference to this discussion for context.

lldb/include/lldb/Host/File.h

 //===-- File.h --------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLDB_HOST_FILE_H
 #define LLDB_HOST_FILE_H
 
 #include "lldb/Host/PosixApi.h"
 #include "lldb/Utility/IOObject.h"
 #include "lldb/Utility/Status.h"
 #include "lldb/lldb-private.h"
 #include "llvm/ADT/BitmaskEnum.h"
 
 #include <mutex>
 #include <stdarg.h>
 #include <stdio.h>
public:
 
   /// Get the permissions for a this file.
   ///
   /// \return
   ///     Bits logical OR'ed together from the permission bits defined
   ///     in lldb_private::File::Permissions.
   uint32_t GetPermissions(Status &error) const;
 
+  /// Ensures that the properties which represent if this file is interactive,
+  /// a real terminal, and a terminal with colors are calculated eagerly.
+  void EagerlyCalculateInteractiveAndTerminalProperties();
+
+
   /// Return true if this file is interactive.
   ///
   /// \return
   ///     True if this file is a terminal (tty or pty), false
   ///     otherwise.
   bool GetIsInteractive();
 
   /// Return true if this file from a real terminal.

lldb/source/Core/IOHandler.cpp

 //===-- IOHandler.cpp -----------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Core/IOHandler.h"
 
 #if defined(__APPLE__)
 #include <deque>
 #endif
 #include <string>
 
 #include "lldb/Core/Debugger.h"
 #include "lldb/Core/StreamFile.h"

 
 #if LLDB_ENABLE_LIBEDIT
   bool use_editline = false;
 
   use_editline = GetInputFILE() && GetOutputFILE() && GetErrorFILE() &&
                  m_input_sp && m_input_sp->GetIsRealTerminal();
 
   if (use_editline) {
+#if defined(__linux__)
+    // On some Linux distributions, after setting up the EditLine object,
+    // the eventual call to set_curterm,
+    // which happens when calculating terminal properties,
+    // breaks the libedit configuration, causing characters that have
+    // functions bound to them not to show up. As a workaround, we calculate
+    // these terminal properties eagerly, before initializing the EditLine
+    // object.
+    m_output_sp->GetFile().EagerlyCalculateInteractiveAndTerminalProperties();
+    m_error_sp->GetFile().EagerlyCalculateInteractiveAndTerminalProperties();
+#endif
+
     m_editline_up = std::make_unique<Editline>(editline_name, GetInputFILE(),
                                                GetOutputFILE(), GetErrorFILE(),
                                                m_color_prompts);
     m_editline_up->SetIsInputCompleteCallback(IsInputCompleteCallback, this);
     m_editline_up->SetAutoCompleteCallback(AutoCompleteCallback, this);
     if (debugger.GetUseAutosuggestion() && debugger.GetUseColor())
       m_editline_up->SetSuggestionCallback(SuggestionCallback, this);
     // See if the delegate supports fixing indentation

lldb/source/Host/common/File.cpp

 //===-- File.cpp ----------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Host/File.h"
 
 #include <errno.h>
 #include <fcntl.h>
 #include <limits.h>
 #include <stdarg.h>
 #include <stdio.h>
 
 #ifdef _WIN32
if (::ioctl(fd, TIOCGWINSZ, &window_size) == 0) {
         if (llvm::sys::Process::FileDescriptorHasColors(fd))
           m_supports_colors = eLazyBoolYes;
       }
     }
   }
 #endif
 }
 
+void File::EagerlyCalculateInteractiveAndTerminalProperties() {
+  if (m_supports_colors == eLazyBoolCalculate)
+    CalculateInteractiveAndTerminal();
+}
+
 bool File::GetIsInteractive() {
   if (m_is_interactive == eLazyBoolCalculate)
     CalculateInteractiveAndTerminal();
   return m_is_interactive == eLazyBoolYes;
 }
 
 bool File::GetIsRealTerminal() {
   if (m_is_real_terminal == eLazyBoolCalculate)