[gn build] Add a script to check if source lists in BUILD.gn files and CMakeLists.txt files match.

[thakis]

Summary

Also fix a missing file in lib/Support/BUILD.gn found by the script.

The script is very stupid and assumes that CMakeLists.txt follow the standard LLVM CMakeLists.txt formatting with one cpp source file per line. Despite its simplicity, it works well in practice.

It would be nice if it also checked deps and maybe automatically applied its suggestions.

[phosek]

llvm/utils/gn/build/sync_source_lists_from_cmake.py

46

Move this inside the if statement below.

49

You could do just:

added = cmake_cpp - gn_cpp
if added:
  print('added:\n%s' % ',\n'.join(added))

53

ditto

[thakis]

address comments

llvm/utils/gn/build/sync_source_lists_from_cmake.py

49

Done.

To explain a bit why I did what I had: I'm indenting the "add
lines to have the right amount of space in front and with quoting for copying the file right into a gn file, and with your approach I need to duplicated the indent string. But that's better than duplicating the set difference, and your approach is better without drawbacks in the remove case.

[phosek]

LGTM

llvm/utils/gn/README.rst

 
 GN has an autoformatter: `git ls-files '*.gn' '*.gni' | xargs -n 1 gn format`
 after making GN build changes is your friend.
 
 To not put `BUILD.gn` into the main tree, they are all below
 `utils/gn/secondary`.  For example, the build file for `llvm/lib/Support` is in
 `utils/gn/secondary/llvm/lib/Support`.
 
+.. _Syncing GN files from CMake files:
+
+Syncing GN files from CMake files
+=================================
+
+Sometimes after pulling in the latest changes, the GN build doesn't work.
+Most of the time this is due to someone adding a file to CMakeLists.txt file.
+Run `llvm/utils/gn/build/sync_source_lists_from_cmake.py` to print a report
+of which files need to be added to or removed from `BUILD.gn` files to
+match the corresponding `CMakeLists.txt`. You have to manually read the output
+of the script and implement its suggestions.
+
+If new `CMakeLists.txt` files have been added, you have to manually create
+a new corresponding `BUILD.gn` file below `llvm/utils/gn/secondary/`.
+
+If the dependencies in a `CMakeLists.txt` file have been changed, you have to
+manually analyze and fix.
+
 .. _Philosophy:
 
 Philosophy
 ==========
 
 GN believes in using GN arguments to configure the build explicitly, instead
 of implicitly figuring out what to do based on what's available on the current
 system.

llvm/utils/gn/build/sync_source_lists_from_cmake.py

+#!/usr/bin/env python
+
+"""Helps to keep BUILD.gn files in sync with the corresponding CMakeLists.txt.
+
+For each BUILD.gn file in the tree, checks if the list of cpp files in
+it is identical to the list of cpp files in the corresponding CMakeLists.txt
+file, and prints the difference if not."""
+
+from __future__ import print_function
+
+import os
+import re
+import subprocess
+
+def main():
+    gn_files = subprocess.check_output(
+            ['git', 'ls-files', '*BUILD.gn']).splitlines()
+
+    # Matches e.g. |   "foo.cpp",|.
+    gn_cpp_re = re.compile(r'^\s*"([^"]+\.(?:cpp|h))",$', re.MULTILINE)
+    # Matches e.g. |   "foo.cpp"|.
+    cmake_cpp_re = re.compile(r'^\s*([A-Za-z_0-9/-]+\.(?:cpp|h))$',
+                              re.MULTILINE)
+
+    for gn_file in gn_files:
+        # The CMakeLists.txt for llvm/utils/gn/secondary/foo/BUILD.gn is
+        # directly at foo/CMakeLists.txt.
+        strip_prefix = 'llvm/utils/gn/secondary/'
+        if not gn_file.startswith(strip_prefix):
+            continue
+        cmake_file = os.path.join(
+                os.path.dirname(gn_file[len(strip_prefix):]), 'CMakeLists.txt')
+        if not os.path.exists(cmake_file):
+            continue
+
+        def get_sources(source_re, text):
+            return set([m.group(1) for m in source_re.finditer(text)])
+        gn_cpp = get_sources(gn_cpp_re, open(gn_file).read())
+        cmake_cpp = get_sources(cmake_cpp_re, open(cmake_file).read())
+
+        if gn_cpp == cmake_cpp:
+            continue
+
+        print(gn_file)
+        add = cmake_cpp - gn_cpp
+        if add:
+            print('add:\n' + '\n'.join('    "%s",' % a for a in add))
+        remove = gn_cpp - cmake_cpp
+        if remove:
+            print('remove:\n' + '\n'.join(remove))
+        print()
+
+if __name__ == '__main__':
+    main()

llvm/utils/gn/secondary/llvm/lib/Support/BUILD.gn

sources = [
     "ARMWinEH.cpp",
     "Allocator.cpp",
     "BinaryStreamError.cpp",
     "BinaryStreamReader.cpp",
     "BinaryStreamRef.cpp",
     "BinaryStreamWriter.cpp",
     "BlockFrequency.cpp",
     "BranchProbability.cpp",
+    "BuryPointer.cpp",
     "COM.cpp",
     "CachePruning.cpp",
     "Chrono.cpp",
     "CodeGenCoverage.cpp",
     "CommandLine.cpp",
     "Compression.cpp",
     "ConvertUTF.cpp",
     "ConvertUTFWrapper.cpp",