Implement verbose mode in FlieCheck and Lit to capture whole stdin.

[chapuni]

Summary

This changes are aiming to capture and dump whole stdin to FileCheck.
In automated builders, we are often missing actual input, to wonder why the failure was.

• Implement -verbose in FileCheck to dump stdin to stderr.
• Introduce LLVM_FILECHECK to override default value of -verbose.
• Introduce LIT_PASS_VARS in Lit, to customize variables available in the test.

How to activate this changes;

LIT_PASS_VARS=LLVM_FILECHECK LLVM_FILECHECK=-verbose bin/llvm-lit path/to/...

This patch is W-I-P. FIXME:

• Give the description for -verbose with cl::desc().
• Generalize LLVM_FILECHECK to accept additional options.
• Improve the output of -verbose.
• Add tests for -verbose itself.

[filcab]

This functionality would be really good for those random bugs we only see happening in buildbots.
But I think we should only dump the buffer if there was a problem with it.

llvm/utils/FileCheck/FileCheck.cpp

1356

I'd prefer to only print out the whole buffer in case we fail.

[filcab]

Forgot a comment.

llvm/utils/FileCheck/FileCheck.cpp

1310

This should be LLVM_FILECHECK_OPTIONS or LLVM_FILECHECK_EXTRA_ARGS or something.

llvm/utils/lit/lit/TestingConfig.py

32

When you propose this, please split this to another commit.

[jroelofs]

Add tests for -verbose itself.

I don't see any -verbose=true tests, and the default is false. Did you forget to git add them?

[chapuni]

Thanks @filcab.

llvm/utils/FileCheck/FileCheck.cpp

1310

I prefer LLVM_FILECHECK_OPTIONS. Thanks.

1356

I think we could catch successful cases, for example, to compare logs between +Asserts and -Asserts.

Shall we split verbose level, or add another option?

llvm/utils/lit/lit/TestingConfig.py

32

I see. I'll propose this individually.
It'd make "LIT_PRESERVES_TMP" generalized.

[chapuni]

In D20105#429592, @jroelofs wrote:

Add tests for -verbose itself.

I don't see any -verbose=true tests, and the default is false. Did you forget to git add them?

I just tweaked some tests in llvm/test/FileCheck not to be affected by LLVM_FILECHECK_OPTIONS.
I'll add cases for -verbose=true in the next time.

[dvyukov]

In automated builders, we are often missing actual input, to wonder why the failure was.

We are always missing it. On builders and otherwise. I don't understand why printing input is not the default.

llvm/test/FileCheck/check-label-dag.txt

-; RUN: not FileCheck -input-file %s %s 2>&1 | FileCheck --check-prefix=ERROR %s
+; RUN: not FileCheck -verbose=false -input-file %s %s 2>&1 | FileCheck --check-prefix=ERROR %s
 
 bar
 zed
 
 CHECK-LABEL: {{^}}bar
 CHECK-DAG: {{^}}foo
 CHECK-LABEL: {{^}}zed
 
 ERROR: error: expected string not found in input
 ERROR-NEXT: CHECK-DAG: {{.....}}foo

llvm/test/FileCheck/check-label.txt

 ; RUN: FileCheck -input-file %s %s -check-prefix=CHECKOK
-; RUN: not FileCheck -input-file %s %s -check-prefix=CHECKFAIL 2>&1 | FileCheck %s -check-prefix=CHECKERROR
+; RUN: not FileCheck -verbose=false -input-file %s %s -check-prefix=CHECKFAIL 2>&1 | FileCheck %s -check-prefix=CHECKERROR
 
 label0:
 a
 b
 
 label1:
 b
 c

llvm/test/FileCheck/check-multiple-prefixes-nomatch-2.txt

-; RUN: not FileCheck -input-file %s %s -check-prefix=FOO -check-prefix=BAR 2>&1 | FileCheck %s
+; RUN: not FileCheck -verbose=false -input-file %s %s -check-prefix=FOO -check-prefix=BAR 2>&1 | FileCheck %s
 
 fog
 bar
 ; _FOO not a valid check-line
 ; FOO: fo{{o}}
 ; BAR: ba{{r}}
 
 ; CHECK: {{error: expected string not found in input}}
 ; CHECK-NEXT: {{F}}OO: fo{{[{][{]o[}][}]}}

llvm/test/FileCheck/check-multiple-prefixes-nomatch.txt

-; RUN: not FileCheck -input-file %s %s -check-prefix=FOO -check-prefix=BAR 2>&1 | FileCheck %s
+; RUN: not FileCheck -verbose=false -input-file %s %s -check-prefix=FOO -check-prefix=BAR 2>&1 | FileCheck %s
 
 BAR
 bar
 foo
 ; BAR: ba{{z}}
 ; FOO: fo{{o}}
 
 ; CHECK: {{error: expected string not found in input}}
 ; CHECK-NEXT: {{B}}AR: ba{{[{][{]z[}][}]}}

llvm/test/FileCheck/check-not-diaginfo.txt

-; RUN: not FileCheck -input-file %s %s 2>&1 | FileCheck -check-prefix DIAG %s
+; RUN: not FileCheck -verbose=false -input-file %s %s 2>&1 | FileCheck -check-prefix DIAG %s
 
 CHECK-NOT: test
 
 DIAG:         CHECK-NOT: pattern specified here
 DIAG-NEXT:    CHECK-NOT: test
 DIAG-NEXT: {{^           \^}}

llvm/utils/FileCheck/FileCheck.cpp

cl::desc("Allow the input file to be empty. This is useful when making\n"
              "checks that some error message does not occur, for example."));
 
 static cl::opt<bool> MatchFullLines(
     "match-full-lines", cl::init(false),
     cl::desc("Require all positive matches to cover an entire input line.\n"
              "Allows leading and trailing whitespace if --strict-whitespace\n"
              "is not also passed."));
 
+static cl::opt<bool> Verbose(
+    "verbose", cl::init(false),
+    cl::desc("(T.B.D.)"));
+
 typedef cl::list<std::string>::const_iterator prefix_iterator;
 
 //===----------------------------------------------------------------------===//
 // Pattern Handling Code.
 //===----------------------------------------------------------------------===//
 
 namespace Check {
   enum CheckType {

 static void AddCheckPrefixIfNeeded() {
   if (CheckPrefixes.empty())
     CheckPrefixes.push_back("CHECK");
 }
 
 int main(int argc, char **argv) {
   sys::PrintStackTraceOnErrorSignal();
   PrettyStackTraceProgram X(argc, argv);
+
+  // Override default value of Verbose
+  if (auto p = std::getenv("LLVM_FILECHECK")) {
+    if (StringRef(p) == "-verbose")
+      Verbose = true;
+  }
+
   cl::ParseCommandLineOptions(argc, argv);
 
   if (!ValidateCheckPrefixes()) {
     errs() << "Supplied check-prefix is invalid! Prefixes must be unique and "
               "start with a letter and contain only alphanumeric characters, "
               "hyphens and underscores\n";
     return 2;
   }
int main(int argc, char **argv) {
   // Remove duplicate spaces in the input file if requested.
   // Remove DOS style line endings.
   std::unique_ptr<MemoryBuffer> F =
       CanonicalizeInputFile(std::move(File), NoCanonicalizeWhiteSpace);
 
   // Check that we have all of the expected strings, in order, in the input
   // file.
   StringRef Buffer = F->getBuffer();
+  if (Verbose) {
+    errs() << "================\nCommand line:";
+    for (int i = 0; i < argc; ++i)
+      errs() << " " << argv[i];
+    errs() << "\n================\n"
+           << Buffer
+           << "\n================\n";
+  }
+
 
   SM.AddNewSourceBuffer(std::move(F), SMLoc());
 
   /// VariableTable - This holds all the current filecheck variables.
   StringMap<StringRef> VariableTable;
 
   bool hasError = false;
 

llvm/utils/lit/lit/TestingConfig.py

def fromdefaults(litConfig):
             'PATH' : os.pathsep.join(litConfig.path +
                                      [os.environ.get('PATH','')]),
             'LLVM_DISABLE_CRASH_REPORT' : '1',
             }
 
         pass_vars = ['LIBRARY_PATH', 'LD_LIBRARY_PATH', 'SYSTEMROOT', 'TERM',
                      'LD_PRELOAD', 'ASAN_OPTIONS', 'UBSAN_OPTIONS',
                      'LSAN_OPTIONS', 'ADB', 'ANDROID_SERIAL']
+
+        # Passed vars can be defined in LIT_PASS_VARS, like "TEMP:TMP:TMPDIR"
+        extra_pass_vars = os.environ.get("LIT_PASS_VARS")
+        if extra_pass_vars:
+            pass_vars.extend(extra_pass_vars.split(':'))
+
         for var in pass_vars:
             val = os.environ.get(var, '')
             # Check for empty string as some variables such as LD_PRELOAD cannot be empty
             # ('') for OS's such as OpenBSD.
             if val:
                 environment[var] = val
 
         if sys.platform == 'win32':