[lldb-cmake-matrix] Fix environment leak across runs

[fdeazeve]

Summary

Any `export` statement in a pipeline description will leak across
different compiler stages or, worse, between two runs of the entire
pipeline. This was seen recently in two ways:

1. The Dwarf 2 / 4 runs were using a Clang 13.0 built on a previous
pipeline run, and they were failing because some tests require ToT
clang.
2. The Dwarf 2 / 4 runs were trying to use Clang 13.0, but the workspace
was cleaned after the previous pipeline run, resulting in a "cannot
find clang" error.

This commit addresses the issue by no longer using environment variables
to communicate which Clang to use when building LLDB tests. Instead, we
now use parameters to the `test_monorepo_build.py` script. More
generally though, there are still a lot of `export` statements in the
pipeline, and these should be examined in isolation to determine how to
best remove them.

Jenkins provides ways to set environment variables more carefully [1];
however, for the purposes of the LLDB test compiler variable, this is
dangerous: we don't want such a flag to last any longer than the CMake
configure step, so this commit uses the script argument approach.

[1]: https://www.jenkins.io/doc/book/pipeline/syntax/#environment

[fdeazeve]

Quoting what I said in D132004:

Looking at the logs, the Dwarf 2 and Dwarf 4 jobs do not use ToT, but rather Clang 13. For example, for the Dwarf 2 job, we see this:

"/Users/buildslave/jenkins/workspace/lldb-cmake-matrix/clang_1300_build/bin/clang" -std=c++11 -gdwarf-2 -isysroot

Which is weird, given how we don't build Clang 13 until much later on the pipeline.
I think the bot has a polluted environment somehow across runs. For example, build 5042:

https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake-matrix/5042/execution/node/52/log/?consoleFull

ERROR:root:/Users/buildslave/jenkins/workspace/lldb-cmake-matrix/clang_1300_build/bin/clang is not a valid compiler executable; aborting...

Sometimes the Clang13 builds get cleaned up (see how sometimes the Clang 13 jobs take 30s to run, and sometimes 40min).
I'm guessing that this error shows up when the previous build got cleaned up.

Edit: I'm assuming the intent of the Dwarf 2 / 4 tests is to use ToT Clang

[Michael137]

nice! lgtm

[kastiglione]

zorg/jenkins/monorepo_build.py

1070

the default dest name would be lldb_test_compiler, is there a reason to override with squished case?

1071

what is the benefit of specifying type=pathlib.Path? Its value is used as a string.

[fdeazeve]

zorg/jenkins/monorepo_build.py

1070

I thought I'd be saving diff lines by keeping the old name, but now that you mention it, I'm saving a single line in favour of a poor name. Will change!

1071

I thought there would be some extra type checking by using that, but everywhere else we're using strings, so I'll change this too

[fdeazeve]

Addressed review comments

[kastiglione]

zorg/jenkins/monorepo_build.py

1070

tip: ArgumentParser will automatically use lldb_test_compiler as the dest name for --lldb-test-compiler, so if you'd like you can leave the dest unspecified.

[JDevlieghere]

zorg/jenkins/jobs/jobs/lldb-cmake-matrix

203–204

If the export "leaks" across stages, can we avoid the duplication of this line and instead do this once at the beginning with a stage that sets up the environment?

[fdeazeve]

Removed default name for the variable

zorg/jenkins/jobs/jobs/lldb-cmake-matrix

203–204

Yes, I think so, this is something I'd like to look at next. I was hoping to do this in isolation though, too many things going on with these bots right now :)

test/jenkins/test_monorepo_build.py

 # RUN: python %{src_root}/zorg/jenkins/monorepo_build.py cmake all --timeout=900 > %t-timeout.log
 # RUN: FileCheck --check-prefix CHECK-TIMEOUT < %t-timeout.log %s
 # CHECK-TIMEOUT: --timeout=900
 
 # Test to check if default timeout is being set to 600.
 # RUN: python %{src_root}/zorg/jenkins/monorepo_build.py cmake all > %t-timeout-default.log
 # RUN: FileCheck --check-prefix CHECK-TIMEOUT-DEFAULT < %t-timeout-default.log %s
 # CHECK-TIMEOUT-DEFAULT: --timeout=600
+
+# RUN: python %{src_root}/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
+# RUN:   --lldb-test-compiler="MY_LLDB_TEST_COMPILER" > %t-lldb-configure.log
+# RUN: FileCheck --check-prefix CHECK-LLDB-CONFIG < %t-lldb-configure.log %s
+
+# CHECK-LLDB-CONFIG: -DLLDB_TEST_COMPILER=MY_LLDB_TEST_COMPILER

zorg/jenkins/jobs/jobs/lldb-cmake-matrix

stages {
                     '''
                 }
             }
         }
         stage('Test Clang 5.0.2') {
             steps {
                 timeout(60) {
                     sh '''
                     export PATH=$PATH:/usr/bin:/usr/local/bin
-                    export LLDB_TEST_COMPILER="$WORKSPACE/clang_502_build/bin/clang"
                     python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
                       --assertions \
+                      --lldb-test-compiler="$WORKSPACE/clang_502_build/bin/clang" \
                       --projects="clang;lldb"  \
                       --runtimes="libcxx;libcxxabi"  \
                       --cmake-type=Release \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="gmodules" \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="watchpoint" \
                 --dotest-flag="--skip-category" \
stages {
                 }
             }
         }
         stage('Test Clang 7.0.1') {
             steps {
                 timeout(60) {
                     sh '''
                     export PATH=$PATH:/usr/bin:/usr/local/bin
-                    export LLDB_TEST_COMPILER="$WORKSPACE/clang_701_build/bin/clang"
                     python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
                       --assertions \
+                      --lldb-test-compiler="$WORKSPACE/clang_701_build/bin/clang" \
                       --projects="clang;lldb"  \
                       --runtimes="libcxx;libcxxabi"  \
                       --cmake-type=Release \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="gmodules" \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="watchpoint" \
                       --dotest-flag="--skip-category" \
stages {
                 }
             }
         }
         stage('Test Clang 9.0.0') {
             steps {
                 timeout(60) {
                     sh '''
                     export PATH=$PATH:/usr/bin:/usr/local/bin
-                    export LLDB_TEST_COMPILER="$WORKSPACE/clang_900_build/bin/clang"
                     python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
                       --assertions \
+                      --lldb-test-compiler="$WORKSPACE/clang_900_build/bin/clang" \
                       --projects="clang;lldb"  \
                       --runtimes="libcxx;libcxxabi"  \
                       --cmake-type=Release \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="gmodules" \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="watchpoint" \
                 --dotest-flag="--skip-category" \
stages {
                 }
             }
         }
         stage('Test Clang 11.0.1') {
             steps {
                 timeout(60) {
                     sh '''
                     export PATH=$PATH:/usr/bin:/usr/local/bin
-                    export LLDB_TEST_COMPILER="$WORKSPACE/clang_1101_build/bin/clang"
                     python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
                       --assertions \
+                      --lldb-test-compiler="$WORKSPACE/clang_1101_build/bin/clang" \
                       --projects="clang;lldb"  \
                       --runtimes="libcxx;libcxxabi"  \
                       --cmake-type=Release \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="gmodules" \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="watchpoint" \
           --dotest-flag="--skip-category" \
stages {
                 }
             }
         }
         stage('Test Clang 13.0.0') {
             steps {
                 timeout(60) {
                     sh '''
                     export PATH=$PATH:/usr/bin:/usr/local/bin
-                    export LLDB_TEST_COMPILER="$WORKSPACE/clang_1300_build/bin/clang"
                     python llvm-zorg/zorg/jenkins/monorepo_build.py lldb-cmake-matrix configure \
                       --assertions \
+                      --lldb-test-compiler="$WORKSPACE/clang_1300_build/bin/clang" \
                       --projects="clang;lldb"  \
                       --runtimes="libcxx;libcxxabi"  \
                       --cmake-type=Release \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="gmodules" \
                       --dotest-flag="--skip-category" \
                       --dotest-flag="watchpoint" \
             --dotest-flag="--skip-category" \

zorg/jenkins/monorepo_build.py

 """Build and test clangs."""
 
 from __future__ import print_function
 import sys
 import logging
 import os
 import subprocess
 import datetime
 import time
 import argparse
 import shutil
 import math
 import re
 import xml.etree.ElementTree as ET
+import pathlib
 from contextlib import contextmanager
 from six.moves import urllib
 
 SERVER = "labmaster2.lab.llvm.org"
 
 NINJA = "/usr/local/bin/ninja"
 
 MODULE_CACHE_REGEX = re.compile(r'-fmodules-cache-path=([^\"]+)')
def __init__(self, args):
         self._llvm_src_dir = os.environ.get('LLVM_SRC_DIR', 'llvm')
         self._lldb_src_dir = os.environ.get('LLDB_SRC_DIR', 'lldb')
         self._build_dir = os.environ.get('BUILD_DIR', 'clang-build')
         self._lldb_build_dir = os.environ.get('LLDB_BUILD_DIR', 'lldb-build')
         self._lldb_standalone_build_dir = os.environ.get('LLDB_STANDALONE_BUILD_DIR', 'lldb-standalone-build')
         self._lldb_standalone_type = os.environ.get('LLDB_STANDALONE_TYPE', 'build-tree')
         self._lldb_xcode_build_dir = os.environ.get('LLDB_XCODE_BUILD_DIR', 'lldb-xcode-build')
         self._lldb_install_dir = os.environ.get('LLDB_INSTALL_DIR', 'lldb-install')
-        self._lldb_test_compiler = os.environ.get('LLDB_TEST_COMPILER', '')
         self._install_dir = os.environ.get('INSTALL_DIR', 'clang-install')
         self.j_level = os.environ.get('J_LEVEL', None)
         self.max_parallel_tests = os.environ.get('MAX_PARALLEL_TESTS', None)
         self.max_parallel_links = os.environ.get('MAX_PARALLEL_LINKS', None)
         self.host_compiler_url = os.environ.get('HOST_URL',
                                                 'http://labmaster2.local/artifacts/')
         self.artifact_url = os.environ.get('ARTIFACT', 'NONE')
         self.job_name = os.environ.get('JOB_NAME', 'NONE')
class Configuration(object):
     def lldbsrcdir(self):
         """The derived source directory for this lldb build."""
         return os.path.join(self.workspace, self._src_dir, self._lldb_src_dir)
 
     def lldbinstalldir(self):
         """The install directory for the lldb build."""
         return os.path.join(self.workspace, self._lldb_install_dir)
 
-    def lldbtestcompiler(self):
-        """The compiler used to build LLDB tests."""
-        return self._lldb_test_compiler
-
     def installdir(self):
         """The install directory for the compile."""
         return os.path.join(self.workspace, self._install_dir)
 
     def lldbstandalonetype(self):
         """The type of standalone build: against the build or install tree."""
         return self._lldb_standalone_type;
 
if variant == 'sanitized':
             ])
 
     if conf.compiler_flags:
         cmake_cmd.extend([
             "-DCMAKE_C_FLAGS={}".format(' '.join(conf.compiler_flags)),
             "-DCMAKE_CXX_FLAGS={}".format(' '.join(conf.compiler_flags))
         ])
 
-    if conf.lldbtestcompiler():
+    if conf.lldbtestcompiler is not None:
         cmake_cmd.extend([
-            '-DLLDB_TEST_COMPILER=' + conf.lldbtestcompiler(),
+            '-DLLDB_TEST_COMPILER=' + str(conf.lldbtestcompiler),
         ])
 
     cmake_cmd.extend(conf.cmake_flags)
 
     if conf.CC():
         cmake_cmd.extend([
             '-DCMAKE_C_COMPILER=' + conf.CC(),
             '-DCMAKE_CXX_COMPILER=' + conf.CC() + "++"
def parse_args():
     parser.add_argument('--projects', dest='llvm_enable_projects',
                         default="clang;clang-tools-extra;compiler-rt",
                         help="Semicolon seperated list of projects to enable.")
     parser.add_argument('--runtimes', dest='llvm_enable_runtimes',
                         default="",
                         help="Semicolon seperated list of runtimes to enable.")
     parser.add_argument('--timeout', dest='timeout', type=int, default='600',
                         help='Individual test timeout in seconds.')
+    parser.add_argument('--lldb-test-compiler', dest='lldbtestcompiler',
+                        type=pathlib.Path, default=None,
+                        help='The compiler used to build LLDB tests.')
     args = parser.parse_args()
     if args.thinlto:
         args.lto = True
     return args
 
 
 def main():
     """Run a build based on command line args and ENV."""