gn build: Add get.py script to download prebuilt gn, make gn.py run downloaded gn if gn is not on PATH

[thakis]

Summary

Prebuilts are available for Linux, macOS, Windows. The script always pulls the latest GN version.

[thakis]

also recommend get.py if depot_tools broken gn forwarder is on PATH

[hans]

llvm/utils/gn/get.py

32

Fancy :-)

56

the url doesn't end in .zip or something?

58

if it's a zip file, i'm surprised this is necessary

[thakis]

comments

[thakis]

llvm/utils/gn/get.py

32

Done :-P

56

No.

58

That's actually a good point. Huh, I wonder why I have this.

Oh, I think maybe I used to not have the unzip code, and this tried to run the zip file directly, and then that failed so then I added the chmod, only to realize that this is a zip file, and then I didn't remove the chmod again.

Things seem to be fine without this, so I removed it again. Thanks for catching this!

[thakis]

restore chmod, check arch

[thakis]

Hm, turns out I do need the chmod() after all. Looks like zipfile strips x bits: https://stackoverflow.com/questions/39296101/python-zipfile-removes-execute-permissions-from-binaries

I also added a check that the host system is arm64.

[thakis]

also mention x86 in readme

llvm/utils/gn/.gitignore

+bin

llvm/utils/gn/README.rst

 
 .. _Quick start:
 
 Quick start
 ===========
 
 GN only works in the monorepo layout.
 
-#. Obtain a `gn binary <https://gn.googlesource.com/gn/#getting-started>`_.
+#. Obtain a gn binary. If gn is not already on your PATH, run
+   `llvm/utils/gn/get.py` to download a prebuilt gn binary if you're on
+   Linux, macOS, or Windows, or `build gn yourself
+   <https://gn.googlesource.com/gn/#getting-started>`_ if you're on a different
+   platform or don't want to trust prebuilt binaries.
 
 #. In the root of the monorepo, run `llvm/utils/gn/gn.py gen out/gn`.
    `out/gn` is the build directory, it can have any name, and you can have as
    many as you want, each with different build settings.  (The `gn.py` script
    adds `--dotfile=llvm/utils/gn/.gn --root=.` and just runs regular `gn`;
    you can manually pass these parameters and not use the wrapper if you
    prefer.)
 

llvm/utils/gn/get.py

+#!/usr/bin/env python
+"""Downloads a prebuilt gn binary to a place where gn.py can find it."""
+
+from __future__ import print_function
+
+import os
+import urllib2
+import sys
+import tempfile
+import zipfile
+
+
+def download_url(url, output_file):
+    """Download url into output_file."""
+    sys.stdout.write('downloading %s ...' % url)
+    sys.stdout.flush()
+    output_file.write(urllib2.urlopen(url).read())
+    print(' done')
+
+
+def download_and_unpack(url, output_dir, gn):
+    """Download an archive from url and extract gn from it into output_dir."""
+    with tempfile.TemporaryFile() as f:
+        download_url(url, f)
+        f.seek(0)
+        zipfile.ZipFile(f).extract(gn, path=output_dir)
+
+
+def set_executable_bit(path):
+    mode = os.stat(path).st_mode
+    mode |= (mode & 0o444) >> 2 # Copy R bits to X.
+    os.chmod(path, mode) # No-op on Windows.
+
+
+def get_platform():
+    if sys.platform.startswith('linux'):
+        return 'linux'
+    elif sys.platform == 'darwin':
+        return 'mac'
+    elif sys.platform == 'win32':
+        return 'windows'
+
+
+def main():
+    platform = get_platform()
+    if not platform:
+        print('no prebuilt binary for', sys.platform)
+        return 1
+
+    dirname = os.path.join(os.path.dirname(__file__), 'bin', platform)
+    if not os.path.exists(dirname):
+        os.makedirs(dirname)
+
+    url = 'https://chrome-infra-packages.appspot.com/dl/gn/gn/%s-amd64/+/latest'
+    gn = 'gn' + ('.exe' if sys.platform == 'win32' else '')
+    download_and_unpack(url % platform, dirname, gn)
+    set_executable_bit(os.path.join(dirname, gn))
+
+
+if __name__ == '__main__':
+    sys.exit(main())

llvm/utils/gn/gn.py

 #!/usr/bin/env python
 """Calls `gn` with the right --dotfile= and --root= arguments for LLVM."""
 
 # GN normally expects a file called '.gn' at the root of the repository.
 # Since LLVM's GN build isn't supported, putting that file at the root
 # is deemed inappropriate, which requires passing --dotfile= and -root= to GN.
 # Since that gets old fast, this script automatically passes these arguments.
 
+import distutils.spawn
 import os
 import subprocess
 import sys
 
 
 THIS_DIR = os.path.dirname(__file__)
 ROOT_DIR = os.path.join(THIS_DIR, '..', '..', '..')
 
 
+def get_platform():
+    if sys.platform.startswith('linux'):
+        return 'linux'
+    elif sys.platform == 'darwin':
+        return 'mac'
+    elif sys.platform == 'win32':
+        return 'windows'
+
+
+def print_no_gn(mention_get):
+    print('gn binary not found in PATH')
+    if mention_get:
+        print('run llvm/utils/gn/get.py to download a binary and try again, or')
+    print('follow https://gn.googlesource.com/gn/#getting-started')
+    return 1
+
+
 def main():
-    # Find real gn executable. For now, just assume it's on PATH.
-    # FIXME: Probably need to append '.exe' on Windows.
+    # Find real gn executable.
     gn = 'gn'
+    if not distutils.spawn.find_executable('gn'):
+        # Not on path. See if get.py downloaded a prebuilt binary and run that
+        # if it's there, or suggest to run get.py if it isn't.
+        platform = get_platform()
+        if not platform:
+            return print_no_gn(mention_get=False)
+        gn = os.path.join(os.path.dirname(__file__), 'bin', platform, 'gn')
+        if not os.path.exists(gn + ('.exe' if platform == 'windows' else '')):
+            return print_no_gn(mention_get=True)
 
     # Compute --dotfile= and --root= args to add.
     extra_args = []
     gn_main_arg = next((x for x in sys.argv[1:] if not x.startswith('-')), None)
     if gn_main_arg != 'help':  # `gn help` gets confused by the switches.
         cwd = os.getcwd()
         dotfile = os.path.relpath(os.path.join(THIS_DIR, '.gn'), cwd)
         root = os.path.relpath(ROOT_DIR, cwd)