[gn build] Add infrastructure to create symlinks and use it to create lld's symlinks

[thakis]

Summary

This is slightly involved, see the comments in the code.

The GN build now builds a functional lld!

[thakis]

llvm/utils/gn/secondary/lld/tools/lld/BUILD.gn

15 ↗

(On Diff #177879)

Not sure if root_out_dir is the right thing here once we do cross builds. Is there something like "root of current toolchain"?

[thakis]

llvm/utils/gn/secondary/lld/tools/lld/BUILD.gn

15 ↗

(On Diff #177879)

Oh, root_out_dir is per-toolchain. Then this is good as-is I think.

[smeenai]

llvm/utils/gn/build/symlink_or_copy.py

3 ↗

(On Diff #177879)

Starting with Windows 10 RS2 (version 1703, released April 2017), you can create symlinks without needing to elevate to administrator if you have developer mode configured. I would imagine that's a reasonably common configuration for LLVM devs, and the symlinks end up saving a bunch of space. Do you think it's worth considering (not necessarily as a part of this patch; it could be done later) attempting to create a symlink even on Windows, and falling back to a copy if you get a permission denied error? I've been meaning to implement something similar on the CMake side.

[thakis]

llvm/utils/gn/build/symlink_or_copy.py

3 ↗

(On Diff #177879)

I think that'd be great. We probably wouldn't want this for releases (since users might have older Windows versions), but since the gn build is primarily for day-to-day development it makes a lot of sense to me.

I'd try that in a follow-up so I can test it on my Windows box if that's alright. (Or you could try making the change, it should be a < 10 line change in just this script.)

[thakis]

llvm/utils/gn/build/symlink_or_copy.py

3 ↗

(On Diff #177879)

(ps: Once we no longer need to worry about old Windows versions that don't have symlinks, we can use the LHS on https://github.com/nico/llvm-project-20170507/commit/f24e0f0152d22eecad27f63b58149942c4b9bc22 instead)

[smeenai]

llvm/utils/gn/build/symlink_or_copy.py

3 ↗

(On Diff #177879)

I'll play around with this ... I've been meaning to experiment with the gn build for a while :)

I actually think it would be fine to use symlinks for releases as well. Symlinks have been around since Windows Vista, but you had to elevate to administrator to be able to create or modify them, so they were rarely used. Once created, you could use them without elevation though. Windows 10 RS2 removed the elevation requirement when you were in developer mode, making them much more convenient. See https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/ for more details (which also reminds me, we'd have to make sure Python was actually passing SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE to its CreateSymbolicLink calls for this to work...)

[thakis]

llvm/utils/gn/build/symlink_or_copy.py

3 ↗

(On Diff #177879)

Ah, I didn't know about using symlinks not requiring privileges! That would be super nice for distribution then, actually.

I'd expect this would have to go through ctypes.windll.kernel32.CreateSymbolicLinkW instead of os.symlink; py2.7 probably precedes SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE.

[smeenai]

llvm/utils/gn/build/symlink_or_copy.py

3 ↗

(On Diff #177879)

Yup, Python 2.7 doesn't expose os.symlink at all on Windows, and even 3.7.1 doesn't use SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE (which is disappointing).

[smeenai]

I started playing with this:

gn gen --dotfile=llvm/utils/gn/.gn --root=. out/gn
ERROR at //llvm/utils/gn/build/BUILD.gn:46:5: Undefined identifier.
    defines += [
    ^------
See //llvm/utils/gn/build/BUILDCONFIG.gn:6:3: which caused the file to be included.
  "//llvm/utils/gn/build:compiler_defaults",
  ^----------------------------------------

The following fixed it, though idk if it's the right fix:

diff --git a/llvm/utils/gn/build/BUILD.gn b/llvm/utils/gn/build/BUILD.gn
index 0f369ec7395..37aa4311651 100644
--- a/llvm/utils/gn/build/BUILD.gn
+++ b/llvm/utils/gn/build/BUILD.gn
@@ -3,9 +3,11 @@ import("//llvm/utils/gn/build/mac_sdk.gni")
 import("//llvm/utils/gn/build/toolchain/compiler.gni")

 config("compiler_defaults") {
+  defines = []
+
   # FIXME: Don't define this globally here.
   if (host_os != "win") {
-    defines = [ "LLVM_ON_UNIX" ]
+    defines += [ "LLVM_ON_UNIX" ]
   }

   if (!llvm_enable_assertions) {

[thakis]

In D55606#1328694, @smeenai wrote:

I started playing with this:

gn gen --dotfile=llvm/utils/gn/.gn --root=. out/gn
ERROR at //llvm/utils/gn/build/BUILD.gn:46:5: Undefined identifier.
    defines += [
    ^------
See //llvm/utils/gn/build/BUILDCONFIG.gn:6:3: which caused the file to be included.
  "//llvm/utils/gn/build:compiler_defaults",
  ^----------------------------------------

The following fixed it, though idk if it's the right fix:

diff --git a/llvm/utils/gn/build/BUILD.gn b/llvm/utils/gn/build/BUILD.gn
index 0f369ec7395..37aa4311651 100644
--- a/llvm/utils/gn/build/BUILD.gn
+++ b/llvm/utils/gn/build/BUILD.gn
@@ -3,9 +3,11 @@ import("//llvm/utils/gn/build/mac_sdk.gni")
 import("//llvm/utils/gn/build/toolchain/compiler.gni")

 config("compiler_defaults") {
+  defines = []
+
   # FIXME: Don't define this globally here.
   if (host_os != "win") {
-    defines = [ "LLVM_ON_UNIX" ]
+    defines += [ "LLVM_ON_UNIX" ]
   }

   if (!llvm_enable_assertions) {

That looks like the right fix. Looks like I haven't trie d this on Windows in a while :-/ Want to send a patch?

[smeenai]

https://reviews.llvm.org/D55617

[smeenai]

llvm/utils/gn/build/symlink_or_copy.py

3 ↗

(On Diff #177879)

https://github.com/python/cpython/pull/3652 will fix os.symlink, but directly calling ctypes.windll.kernel32.CreateSymbolicLinkW in the meantime will work. I'll put up a patch.

[thakis]

phosek: 24h ping :-)

[phosek]

One aspect of this change I'm not super excited about is the fact that it duplicates information from the toolchain, specifically the file extension. What's worse, people can override the output_extension on individual targets. symlink_or_copy would break in those cases (or if we change the default extension e.g. for a new toolchain). Unfortunately, handling this in a proper way is tricky. Changing the link tool to also create symlinks as you mentioned is the only way I could think of that avoids that duplication, but that's pretty ugly. Ideally, we would be able to use a copy target, but even that doesn't work since you still have to specify the source which has the same issue. So this change is LGTM as is, but I think this is something we should maybe raise on gn-dev as this is not the first time it came up.

[thakis]

In D55606#1330404, @phosek wrote:

One aspect of this change I'm not super excited about is the fact that it duplicates information from the toolchain, specifically the file extension. What's worse, people can override the output_extension on individual targets. symlink_or_copy would break in those cases (or if we change the default extension e.g. for a new toolchain). Unfortunately, handling this in a proper way is tricky. Changing the link tool to also create symlinks as you mentioned is the only way I could think of that avoids that duplication, but that's pretty ugly. Ideally, we would be able to use a copy target, but even that doesn't work since you still have to specify the source which has the same issue. So this change is LGTM as is, but I think this is something we should maybe raise on gn-dev as this is not the first time it came up.

That's a great point, thanks. I added a FIXME to the py script saying that this shouldn't look at the host os but the target os.

[larawilliam]

Norton is an Antivirus, Anti Malware and internet security software that is designed and developed by Symantec. It uses techniques like signature and heuristics to identify the potential virus. In addition, it has more features like email spam filtering and anti-phishing. Norton is next-gen antivirus security software that offers powerful protection for all your PC, Mac, tablets and smartphones.
Norton login

[larawilliam]

Avast Account is a portal designed to help you manage all your Avast licenses for different products and devices that are registered to one email address.
Avast Login

[larawilliam]

Manually Update Avast Virus Definitions – Avast is a very powerful security service provider whose services are available for all kinds of devices including Windows PC, Mac, and Android Smart devices. You can visit My.Avast.com to check all the services available. Also, Avast also offers a subscription for both personal use and business purposes.
avast login

[larawilliam]

my.avast.com - Get Avast Login to My Avast Account at my.avast.com or id.avast.com and Access Avast Services like Avast Login, Sign in, Sign up Here."
my.avast.com

[larawilliam]

Kaspersky Total Security Stay Safe Online – Surfing online and using the internet has become a daily need. We use the web for multiple things such as general surfing, education, banking, e-commerce and many more.
Activation.kaspersky.com

[logan111]

amazing work done really helped a lot in installing office
https://settup-office.com/setup

llvm/trunk/utils/gn/build/symlink_or_copy.gni

+# Creates a symlink (or, on Windows, copies).
+# Args:
+#   source: Path to link to.
+#   output: Where to create the symlink.
+template("symlink_or_copy") {
+  action(target_name) {
+    forward_variables_from(invoker, [ "deps" ])
+
+    # Make a stamp file the output to work around
+    # https://github.com/ninja-build/ninja/issues/1186
+    stamp =
+        "$target_gen_dir/" + get_path_info(invoker.output, "file") + ".stamp"
+    outputs = [
+      stamp,
+    ]
+    script = "//llvm/utils/gn/build/symlink_or_copy.py"
+    args = [
+      "--stamp",
+      rebase_path(stamp, root_out_dir),
+      invoker.source,
+      rebase_path(invoker.output, root_out_dir),
+    ]
+  }
+}

llvm/trunk/utils/gn/build/symlink_or_copy.py

+#!/usr/bin/env python
+
+"""Symlinks, or on Windows copies, an existing file to a second location.
+
+Overwrites the target location if it exists.
+Updates the mtime on a stamp file when done."""
+
+import argparse
+import errno
+import os
+import sys
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--stamp', required=True,
+                        help='name of a file whose mtime is updated on run')
+    parser.add_argument('source')
+    parser.add_argument('output')
+    args = parser.parse_args()
+
+    # FIXME: This should not check the host platform but the target platform
+    # (which needs to be passed in as an arg), for cross builds.
+    if sys.platform != 'win32':
+        try:
+            os.symlink(args.source, args.output)
+        except OSError as e:
+            if e.errno == errno.EEXIST:
+                os.remove(args.output)
+                os.symlink(args.source, args.output)
+            else:
+                raise
+    else:
+        import shutil
+        output = args.output + ".exe"
+        source = args.source + ".exe"
+        shutil.copyfile(os.path.join(os.path.dirname(output), source), output)
+
+    open(args.stamp, 'w') # Update mtime on stamp file.
+
+
+if __name__ == '__main__':
+    sys.exit(main())

llvm/trunk/utils/gn/secondary/BUILD.gn

 group("default") {
   deps = [
-    "//lld/tools/lld",
+    ":lld",
     "//llvm/tools/llc",
     "//llvm/tools/llvm-undname",
   ]
 }
 
+# Symlink handling.
+# On POSIX, symlinks to the target can be created before the target exist,
+# and the target can depend on the symlink targets, so that building the
+# target ensures the symlinks exist.
+# However, symlinks didn't exist on Windows until recently, so there the
+# binary needs to be copied -- which requires it to exist. So the symlink step
+# needs to run after the target that creates the binary.
+# In the cmake build, this is done via a "postbuild" on the target, which just
+# tacks on "&& copy out.exe out2.exe" to the link command.
+# GN doesn't have a way to express postbuild commands.  It could probably be
+# emulated by having the link command in the toolchain be a wrapper script that
+# reads a ".symlinks" file next to the target, and have an action write that
+# and make the target depend on that, but then every single link has to use the
+# wrapper (unless we do further acrobatics to use a different toolchain for
+# targets that need symlinks) even though most links don't need symlinks.
+# Instead, have a top-level target for each target that needs symlinks, and
+# make that depend on the symlinks. Then the symlinks can depend on the
+# executable.  This has the effect that `ninja lld` builds lld and then creates
+# symlinks (via this target), while `ninja bin/lld` only builds lld and doesn't
+# update symlinks (in particular, on Windows it doesn't copy the new lld to its
+# new locations); also `ninja lld-link` will build lld and copy it (on Windows)
+# to lld-link, but it won't copy it to ld.lld.
+# That seems simpler, more explicit, and good enough.
+group("lld") {
+  deps = [
+    "//lld/tools/lld:symlinks",
+  ]
+}
+
 # A pool called "console" in the root BUILD.gn is magic and represents ninja's
 # built-in console pool. (Requires a GN with `gn --version` >= 552353.)
 pool("console") {
   depth = 1
 }

llvm/trunk/utils/gn/secondary/lld/tools/lld/BUILD.gn

-# FIXME: Set up symlinks pointing to lld.
+import("//llvm/utils/gn/build/symlink_or_copy.gni")
+
+symlinks = [
+  "lld-link",
+  "ld.lld",
+  "ld64.lld",
+  "wasm-ld",
+]
+foreach(target, symlinks) {
+  symlink_or_copy(target) {
+    deps = [
+      ":lld",
+    ]
+    source = "lld"
+    output = "$root_out_dir/bin/$target"
+  }
+}
+
+# //:lld depends on this symlink target, see comment in //BUILD.gn.
+group("symlinks") {
+  deps = []
+  foreach(target, symlinks) {
+    deps += [ ":$target" ]
+  }
+}
 
 executable("lld") {
   configs += [ "//llvm/utils/gn/build:lld_code" ]
   deps = [
     "//lld/COFF",
     "//lld/ELF",
     "//lld/MinGW",
     "//lld/lib/Driver",
     "//lld/wasm",
     "//llvm/lib/Support",
   ]
   sources = [
     "lld.cpp",
   ]
 }