[gn build] Add check-lld target and make it work

[thakis]

Summary

Also add a build file for llvm-lit, which in turn needs llvm/tools/llvm-config.

With this, check-lld runs and passes all of lld's lit tests. It doesn't run any of its unit tests yet.

Running just ninja -C out/gn will build all prerequisites needed to run tests, but it won't run the tests (so that the build becomes clean after one build). Running ninja -C out/gn check-lld will build prerequisites if needed and run the tests. The check-lld target never becomes clean and runs tests every time.

llvm-config's build file is a bit gnarly: Everything not needed to run tests is basically stubbed out. Also, to generate LibraryDependencies.inc we shell out to llvm-build at build-time. It would be much nicer to get the library dependencies by using the dependency data the GN build contains (http://llvm-cs.pcc.me.uk/gen/tools/llvm-config/LibraryDependencies.inc#1).

[thakis]

phosek: ping :-)

[phosek]

llvm/utils/gn/secondary/lld/test/BUILD.gn

8 ↗

(On Diff #178715)

This is likely going to be useful elsewhere, do you plan to move into a .gni file later?

13 ↗

(On Diff #178715)

Is this formatted with gn format? I'm surprised it doesn't put this onto a single line.

26 ↗

(On Diff #178715)

Do we care about builds being relocatable? If yes then we should invoke rebase_path with root_build_dir as a second argument.

84 ↗

(On Diff #178715)

Does lld really depend on these symlinks (the same below)? I don't see any uses of e.g. llvm-ranlib, llvm-strip in lld's test suite.

llvm/utils/gn/secondary/llvm/tools/llvm-config/BUILD.gn

49 ↗

(On Diff #178715)

You can use not_needed for that purpose.

[thakis]

not_needed

[thakis]

Thanks!

llvm/utils/gn/secondary/lld/test/BUILD.gn

8 ↗

(On Diff #178715)

Right now I have one of these per {lld,llvm,clang}/test/BUILD.gn locally. The main thing in the template is the list of defines and that's somewhat different everywhere. (I'm also using a separate python script in my local branch, so I'm not 100% sure how upstream clang/test/BUILD.gn will look like. Let's revisit this when the clang test build file arrives.

13 ↗

(On Diff #178715)

Yes, I run git ls-files '*.gn' '*.gni' | xargs -n 1 gn format every now and then (and just did again, no changes). I think gn puts one-element lists on one line after += but not after =.

26 ↗

(On Diff #178715)

That's a great comment! lit currently wants absolute directories. I explored having relocatable build dirs in https://github.com/nico/llvm-project-20170507/tree/gn_swarmingdemo (the top commit has all relevant bits) and it's possible but needs some lit changes. So for now this needs to be as-is, sadly.

84 ↗

(On Diff #178715)

I figured for targets having symlinks, it's simpler to always depend on the symlink target, since that's easy to remember. Don't feel strongly about it, though.

llvm/utils/gn/secondary/llvm/tools/llvm-config/BUILD.gn

49 ↗

(On Diff #178715)

Oh, that's nicer. Done.

[phosek]

LGTM

llvm/trunk/utils/gn/build/write_cmake_config.py

for in_line in in_lines:
                 in_line = '#define %s\n' % var
             if not values[var]:
                 in_line = '/* #undef %s */\n' % var
             unused_values.discard(var)
         out_lines.append(in_line)
 
     if unused_values:
         print('unused values args:', file=sys.stderr)
-        print('    ', '\n    '.join(unused_values), file=sys.stderr)
+        print('    ' + '\n    '.join(unused_values), file=sys.stderr)
         return 1
 
     output = ''.join(out_lines)
 
     leftovers = var_re.findall(output)
     if leftovers:
         print('unprocessed values:\n', '\n'.join(leftovers), file=sys.stderr)
         return 1
 
     if not os.path.exists(args.output) or open(args.output).read() != output:
         open(args.output, 'w').write(output)
 
 
 if __name__ == '__main__':
     sys.exit(main())

llvm/trunk/utils/gn/secondary/BUILD.gn

 group("default") {
   deps = [
-    ":lld",
     "//clang/lib/Basic",
-    "//llvm/tools/llc",
-    "//llvm/tools/llvm-ar:symlinks",
-    "//llvm/tools/llvm-as",
-    "//llvm/tools/llvm-bcanalyzer",
-    "//llvm/tools/llvm-dis",
-    "//llvm/tools/llvm-dwarfdump",
-    "//llvm/tools/llvm-mc",
-    "//llvm/tools/llvm-nm:symlinks",
-    "//llvm/tools/llvm-objcopy:symlinks",
-    "//llvm/tools/llvm-objdump:symlinks",
-    "//llvm/tools/llvm-pdbutil",
-    "//llvm/tools/llvm-readobj:symlinks",
+    "//lld/test",
     "//llvm/tools/llvm-undname",
-    "//llvm/tools/obj2yaml",
-    "//llvm/tools/opt",
-    "//llvm/tools/yaml2obj",
-    "//llvm/utils/FileCheck",
-    "//llvm/utils/count",
-    "//llvm/utils/not",
   ]
+  testonly = true
 }
 
 # Symlink handling.
 # On POSIX, symlinks to the target can be created before the target exist,
 # and the target can depend on the symlink targets, so that building the
 # target ensures the symlinks exist.
 # However, symlinks didn't exist on Windows until recently, so there the
 # binary needs to be copied -- which requires it to exist. So the symlink step

llvm/trunk/utils/gn/secondary/lld/test/BUILD.gn

+import("//llvm/lib/DebugInfo/PDB/enable_dia.gni")
+import("//llvm/triples.gni")
+import("//llvm/utils/gn/build/libs/xml/enable.gni")
+import("//llvm/utils/gn/build/libs/zlib/enable.gni")
+import("lld_lit_site_cfg_files.gni")
+
+# The bits common to writing lit.site.cfg.py.in and Unit/lit.site.cfg.py.in.
+template("write_lit_cfg") {
+  action(target_name) {
+    script = "//llvm/utils/gn/build/write_cmake_config.py"
+
+    sources = [
+      invoker.input,
+    ]
+    outputs = [
+      invoker.output,
+    ]
+
+    args = [
+      "-o",
+      rebase_path(outputs[0], root_out_dir),
+      rebase_path(sources[0], root_out_dir),
+
+      "LIT_SITE_CFG_IN_HEADER=## Autogenerated from ${sources[0]}, do not edit",
+      "LLD_BINARY_DIR=" +
+          rebase_path(get_label_info("//lld", "target_out_dir")),
+      "LLD_SOURCE_DIR=" + rebase_path("//lld"),
+      "LLVM_BINARY_DIR=" +
+          rebase_path(get_label_info("//llvm", "target_out_dir")),
+      "LLVM_LIBRARY_OUTPUT_INTDIR=",  # FIXME: for shared builds only (?)
+      "LLVM_LIBS_DIR=",  # needed only for shared builds
+      "LLVM_LIT_TOOLS_DIR=",  # Intentionally empty, matches cmake build.
+      "LLVM_RUNTIME_OUTPUT_INTDIR=" + rebase_path("$root_out_dir/bin"),
+      "LLVM_SOURCE_DIR=" + rebase_path("//llvm"),
+      "LLVM_TOOLS_DIR=" + rebase_path("$root_out_dir/bin"),
+      "PYTHON_EXECUTABLE=$python_path",
+      "TARGET_TRIPLE=$llvm_target_triple",
+    ]
+    args += invoker.extra_args
+  }
+}
+
+write_lit_cfg("lit_site_cfg") {
+  # Fully-qualified instead of relative for LIT_SITE_CFG_IN_HEADER.
+  input = "//lld/test/lit.site.cfg.py.in"
+  output = lld_lit_site_cfg_file
+
+  extra_args = []
+  if (llvm_enable_dia_sdk) {
+    extra_args += [ "LLVM_ENABLE_DIA_SDK=1" ]
+  } else {
+    extra_args += [ "LLVM_ENABLE_DIA_SDK=0" ]  # Must be 0.
+  }
+
+  if (llvm_enable_libxml2) {
+    extra_args += [ "LLVM_LIBXML2_ENABLED=1" ]
+  } else {
+    extra_args += [ "LLVM_LIBXML2_ENABLED=" ]  # Must be empty.
+  }
+
+  if (llvm_enable_zlib) {
+    extra_args += [ "HAVE_LIBZ=1" ]
+  } else {
+    extra_args += [ "HAVE_LIBZ=0" ]  # Must be 0.
+  }
+}
+
+write_lit_cfg("lit_unit_site_cfg") {
+  # Fully-qualified instead of relative for LIT_SITE_CFG_IN_HEADER.
+  input = "//lld/test/Unit/lit.site.cfg.py.in"
+  output = lld_lit_unit_site_cfg_file
+  extra_args = [ "LLVM_BUILD_MODE=." ]
+}
+
+# This target should contain all dependencies of check-lld.
+# //:default depends on it, so that ninja's default target builds all
+# prerequisites for check-lld but doesn't run check-lld itself.
+group("test") {
+  deps = [
+    ":lit_site_cfg",
+    ":lit_unit_site_cfg",
+    "//lld/tools/lld:symlinks",
+    "//llvm/tools/llc",
+    "//llvm/tools/llvm-ar:symlinks",
+    "//llvm/tools/llvm-as",
+    "//llvm/tools/llvm-bcanalyzer",
+    "//llvm/tools/llvm-dis",
+    "//llvm/tools/llvm-dwarfdump",
+    "//llvm/tools/llvm-mc",
+    "//llvm/tools/llvm-nm:symlinks",
+    "//llvm/tools/llvm-objcopy:symlinks",
+    "//llvm/tools/llvm-objdump:symlinks",
+    "//llvm/tools/llvm-pdbutil",
+    "//llvm/tools/llvm-readobj:symlinks",
+    "//llvm/tools/obj2yaml",
+    "//llvm/tools/opt",
+    "//llvm/tools/yaml2obj",
+    "//llvm/utils/FileCheck",
+    "//llvm/utils/count",
+    "//llvm/utils/llvm-lit",
+    "//llvm/utils/not",
+  ]
+  testonly = true
+
+  # FIXME: Add dep on "//lld/unittests" target once it exists.
+}
+
+# This is the action that runs all of lld's tests, check-lld.
+action("check-lld") {
+  script = "$root_out_dir/bin/llvm-lit"
+  if (host_os == "win") {
+    script += ".py"
+  }
+  args = [
+    "-sv",
+    "--param",
+    "lld_site_config=" + rebase_path(lld_lit_site_cfg_file, root_out_dir),
+    "--param",
+    "lld_unit_site_config=" +
+        rebase_path(lld_lit_unit_site_cfg_file, root_out_dir),
+    rebase_path(".", root_out_dir),
+  ]
+  outputs = [
+    "$target_gen_dir/run-lit",  # Non-existing, so that ninja runs it each time.
+  ]
+
+  # Since check-lld is always dirty, //:default doesn't depend on it so that
+  # it's not part of the default ninja target.  Hence, check-lld shouldn't
+  # have any deps except :test, so that the default target is sure to build
+  # all the deps.
+  deps = [
+    ":test",
+  ]
+  testonly = true
+
+  pool = "//:console"
+}

llvm/trunk/utils/gn/secondary/lld/test/lld_lit_site_cfg_files.gni

+lld_lit_site_cfg_file = "$root_gen_dir/lld/test/lit.site.cfg.py"
+lld_lit_unit_site_cfg_file = "$root_gen_dir/lld/test/Unit/lit.site.cfg.py"

llvm/trunk/utils/gn/secondary/llvm/tools/llvm-config/BUILD.gn

+import("//llvm/lib/Target/targets.gni")
+import("//llvm/utils/gn/build/buildflags.gni")
+import("//llvm/utils/gn/build/libs/pthread/enable.gni")
+import("//llvm/utils/gn/build/libs/terminfo/enable.gni")
+import("//llvm/utils/gn/build/libs/xml/enable.gni")
+import("//llvm/utils/gn/build/libs/zlib/enable.gni")
+import("//llvm/version.gni")
+
+enable_targets = ""
+foreach(target, llvm_targets_to_build) {
+  if (enable_targets != "") {
+    enable_targets += " "
+  }
+  enable_targets += target
+}
+
+action("BuildVariables.inc") {
+  script = "//llvm/utils/gn/build/write_cmake_config.py"
+
+  sources = [
+    "BuildVariables.inc.in",
+  ]
+  outputs = [
+    "$target_gen_dir/BuildVariables.inc",
+  ]
+
+  if (is_debug) {
+    build_mode = "debug"
+  } else {
+    build_mode = "release"
+  }
+
+  # FIXME: Why doesn't llvm-config do this, why is this done in
+  # llvm-config/CMakeLists.txt?
+  if (host_os == "win") {
+    l = ""
+    lib = ".lib"
+  } else {
+    l = "-l"
+    lib = ""
+  }
+  # Windows doesn't use any of libxml2,terminfo, zlib by default.
+  # Make GN not warn about these variables being unused.
+  not_needed(["l", "lib"])
+
+  system_libs = ""
+  if (host_os == "win") {
+    # libuuid required for FOLDERID_Profile usage in
+    # lib/Support/Windows/Path.inc.
+    # advapi32 required for CryptAcquireContextW in
+    # lib/Support/Windows/Path.inc
+    system_libs = "psapi.lib shell32.lib ole32.lib uuid.lib advapi32"
+  } else {
+    system_libs += "-lm"
+    if (host_os == "linux") {
+      system_libs += " -lrt -ldl"
+    }
+    if (llvm_enable_threads) {
+      system_libs += " -llibpthreads"
+      if (host_os == "linux") {
+        system_libs += " -latomic"
+      }
+    }
+  }
+  if (llvm_enable_libxml2) {
+    system_libs += " ${l}xml2${lib}"
+  }
+  if (llvm_enable_terminfo) {
+    system_libs += " ${l}ncurses${lib}"
+  }
+  if (llvm_enable_zlib) {
+    system_libs += " ${l}z${lib}"
+  }
+
+  args = [
+    "-o",
+    rebase_path(outputs[0], root_out_dir),
+    rebase_path(sources[0], root_out_dir),
+
+    "LLVM_SRC_ROOT=" + rebase_path("//llvm"),
+    "LLVM_OBJ_ROOT=" + rebase_path(root_out_dir),
+
+    # FIXME: Only the bits needed to run LLVM's test are implemented.
+    "LLVM_CPPFLAGS=.",  # FIXME
+    "LLVM_CFLAGS=.",  # FIXME
+    "LLVM_LDFLAGS=.",  # FIXME
+    "LLVM_CXXFLAGS=.",  # FIXME
+    "LLVM_BUILDMODE=$build_mode",
+    "LLVM_LIBDIR_SUFFIX=",
+    "LLVM_TARGETS_BUILT=$enable_targets",
+    "LLVM_SYSTEM_LIBS=$system_libs",
+    "LLVM_BUILD_SYSTEM=gn",
+    "LLVM_HAS_RTTI=0",
+    "LLVM_BUILD_LLVM_DYLIB=0",
+    "LLVM_LINK_LLVM_DYLIB=0",
+    "BUILD_SHARED_LIBS=0",
+    "LLVM_DYLIB_COMPONENTS=all",
+    "LLVM_DYLIB_VERSION=${llvm_version_major}svn",
+    "LLVM_HAS_GLOBAL_ISEL=1",
+    "LLVM_TOOLS_INSTALL_DIR=",
+  ]
+}
+
+# FIXME: It'd be nice to not depend on llvm-build on this, Depending on all the
+# LLVMBuild.txt files just for this seems a bit overkill.  `gn desc` should
+# have all this information too and could be called at build time.
+action("LibraryDependencies.inc") {
+  script = "//llvm/utils/llvm-build/llvm-build"
+  output = "$target_gen_dir/LibraryDependencies.inc"
+  args = [
+    "--native-target=$native_target",
+    "--enable-targets=$enable_targets",
+    "--write-library-table=" + rebase_path(output, root_out_dir),
+  ]
+  outputs = [
+    output,
+  ]
+}
+
+executable("llvm-config") {
+  deps = [
+    ":BuildVariables.inc",
+    ":LibraryDependencies.inc",
+    "//llvm/include/llvm/Config:config",
+    "//llvm/include/llvm/Config:llvm-config",
+    "//llvm/lib/Support",
+  ]
+
+  # To pick up the generated inc files.
+  include_dirs = [ "$target_gen_dir" ]
+  sources = [
+    "llvm-config.cpp",
+  ]
+}

llvm/trunk/utils/gn/secondary/llvm/utils/llvm-lit/BUILD.gn

+import("//lld/test/lld_lit_site_cfg_files.gni")
+
+action("llvm-lit") {
+  script = "//llvm/utils/gn/build/write_cmake_config.py"
+
+  sources = [
+    "llvm-lit.in",
+  ]
+  outputs = [
+    "$root_out_dir/bin/llvm-lit",
+  ]
+
+  if (host_os == "win") {
+    # llvm-lit needs suffix.py for multiprocess to find a main module.
+    outputs[0] = "${outputs[0]}.py"
+  }
+
+  # lit's lit/llvm/config.py shells out to llvm-config.
+  deps = [
+    "//llvm/tools/llvm-config",
+  ]
+
+  # Generate LLVM_LIT_CONFIG_MAP parameter.
+  # llvm-lit contains a mapping from each lit.cfg.py file to the corresponding
+  # generated llvm.site.cfg.py file, so llvm-lit depends on all the targets to
+  # generate the site.cfg.py file.
+  config_map = ""
+
+  deps += [
+    "//lld/test:lit_site_cfg",
+    "//lld/test:lit_unit_site_cfg",
+  ]
+
+  # Note: \n is converted into a newline by write_cmake_config.py, not by gn.
+  config_map += "map_config('" + rebase_path("//lld/test/lit.cfg.py") + "', '" +
+                rebase_path(lld_lit_site_cfg_file) + "')\n"
+  config_map += "map_config('" + rebase_path("//lld/test/Unit/lit.cfg.py") +
+                "', '" + rebase_path(lld_lit_unit_site_cfg_file) + "')\n"
+
+  args = [
+    "-o",
+    rebase_path(outputs[0], root_out_dir),
+    rebase_path(sources[0], root_out_dir),
+
+    "LLVM_SOURCE_DIR=" + rebase_path("//llvm"),
+    "LLVM_BINARY_DIR=" +
+        rebase_path(get_label_info("//llvm", "target_out_dir")),
+    "BUILD_MODE=.",
+    "LLVM_LIT_CONFIG_MAP=" + config_map,
+  ]
+}