[Release] Add bump-version script.

[thieta]

Summary

There are many files that needs to be updated when you
bump the version of LLVM. This script tries to automate
that in order to make the release managers job easier.

[tschuett]

There is no room for a VERSION file at the root of the repository and everybody reads the file if necessary?

[thieta]

In D133923#3791675, @tschuett wrote:

There is no room for a VERSION file at the root of the repository and everybody reads the file if necessary?

I suggested this at some point - but it never got traction. Maybe showing the complexity of this script can be a catalyst of changing this (especially for the bazel build system).

I have used this script for 15.x branch just to make sure I don't forget anything, since it's many files that needs to be updated.

[kwk]

In D133923#3791676, @thieta wrote:

In D133923#3791675, @tschuett wrote:

There is no room for a VERSION file at the root of the repository and everybody reads the file if necessary?

I suggested this at some point - but it never got traction. Maybe showing the complexity of this script can be a catalyst of changing this (especially for the bazel build system).

I have used this script for 15.x branch just to make sure I don't forget anything, since it's many files that needs to be updated.

So far I've consumed only the llvm/CMakeLists.txt to get the version but that's of course hard-coded into the llvm subproject. I didn't know that there were so many places. I highly suggest to create a discourse post and point to this script. IMHO there's no point in doing this in so many places but rather have everything in one place and let the subprojects consume from it. In order to gain acceptance of such a mundane task, I suggest to create another patch that shows how to consume this version.

[tschuett]

For Swift I found a PR:
https://github.com/apple/swift/pull/58931/files
The version is in a CMake file, but there were not that many files updated.

For Rust. I found a version file:
https://github.com/rust-lang/rust/blob/master/src/version

Most projects store the current version in ChangeLog files.

[kwk]

In D133923#3791825, @tschuett wrote:

For Swift I found a PR:
https://github.com/apple/swift/pull/58931/files
The version is in a CMake file, but there were not that many files updated.

For Rust. I found a version file:
https://github.com/rust-lang/rust/blob/master/src/version

Most projects store the current version in ChangeLog files.

Just my 2 cnts I prefer the plain rust way of storing the version.

[thieta]

Yeah I like a single file like VERSION.txt or .version or something. But we need buy-in. I'll start a thread on discourse to discuss first.

[nikic]

Subprojects already consume the version from llvm/CMakeLists.txt. That's the only version that matters for most purposes. All the duplication is due to alternative build systems. Would bazel even be able to read this from a file?

[thieta]

There is also the lit configuration.

Actually there are many more places in the documentation - especially the release notes that we should update with this script as well later.

[thieta]

Posted a RFC here: https://discourse.llvm.org/t/rfc-centralized-location-for-version-information/65295

[tschuett]

Erlang has a OTP_VERSION file at the root of the repository:

https://github.com/erlang/otp/blob/master/OTP_VERSION

[hans]

I think libcxx/include/__config also needs updating.

[thieta]

Refactor script to share more code

Process libcxx/include/__config as well

[thieta]

I added processing of libcxx/include/__config plus that I refactored the script to be subclasses instead.

Can you guys review the code? We might circle back and centralize version information later on - but for now I would like to have this script.

[thieta]

Handle libcxx __config header better after discussion with @ldionne

[ldionne]

llvm/utils/release/bump-version.py

175

This should be more future-proof in case e.g. clang-format causes this to change in the future. Then you can also use t = match.group(1) instead of t = line[line.rfind(' ')+1:].strip(), if you prefer that.

177–179

I suggest that you only check-in the version that handles XXYZZ, and I can modify this script in D134187.

[ldionne]

llvm/utils/release/bump-version.py

177–179

Ugh, sorry, I mean in the follow-up to D134187 that will change the scheme from XXYZZ to XXYYZZ (and update the docs, etc.).

[ldionne]

llvm/utils/release/bump-version.py

177–179

The follow-up is D134209. Sorry for the numerous comments, but basically what I suggest is that you only handle XXYZZ in this script, and then I can modify this script again in D134209 to handle XXYYZZ (and only that).

[thieta]

Relax libc++ version check and address some feedback

[thieta]

Can anyone have a look at this one? Would like to merge it if no-one objects.

[hans]

I put some comments, but overall lgtm as in the end it's going to be you and Tom who run it.

llvm/utils/release/bump-version.py

195

s/i.e./e.g./

197

Maybe add an example, like /foo/llvm-project/ ? Could this default to the current working directory?

Oh I see later on that it refers to the source root of this file. Maybe the help text could mention that somehow.

216

This is very sparse, especially with all the newlines. Could it be just

[
("llvm/utils/gn/secondary/llvm/version.gni", GNIProcessor())}
...
]

[kwk]

With respect to the future solution of making the version more centralized (see https://discourse.llvm.org/t/rfc-centralized-location-for-version-information/65295) I think this looks good. Great job!

llvm/utils/release/bump-version.py

97

[ldionne]

LGTM, but please note that the XXYZZ => XXYYZZ change in libc++ has been done. My suggested changes should cover that.

llvm/utils/release/bump-version.py

175–180

[thieta]

Adressed some comments

llvm/utils/release/bump-version.py

216

I made it a bit more compact. The formatting comes from black which is pretty much pep8 - so I like to keep that even if I don't always agree with it. But it looks better now I think.

[thieta]

Thanks everyone for the comments! I landed this now and we can always follow-up with fixes if something doesn't look right.

llvm/utils/release/bump-version.py

+#!/usr/bin/env python3
+
+# This script bumps the version of LLVM in *all* the different places where
+# it needs to be defined. Which is quite a few.
+
+import sys
+import argparse
+import packaging.version
+from pathlib import Path
+import re
+from typing import Optional
+
+
+class Processor:
+    def process_line(self, line: str) -> str:
+        raise NotImplementedError()
+
+    def process_file(self, fpath: Path, version: packaging.version.Version) -> None:
+        self.version = version
+        self.major, self.minor, self.patch, self.suffix = (
+            version.major,
+            version.minor,
+            version.micro,
+            version.pre,
+        )
+        data = fpath.read_text()
+        new_data = []
+
+        for line in data.splitlines(True):
+            nline = self.process_line(line)
+
+            # Print the failing line just to inform the user.
+            if nline != line:
+                print(f"{fpath.name}: {line.strip()} -> {nline.strip()}")
+
+            new_data.append(nline)
+
+        fpath.write_text("".join(new_data), newline="\n")
+
+    # Return a string from the version class
+    # optionally include the suffix (-rcX)
+    def version_str(
+        self,
+        version: Optional[packaging.version.Version] = None,
+        include_suffix: bool = True,
+    ) -> str:
+        if version is None:
+            version = self.version
+
+        ver = f"{version.major}.{version.minor}.{version.micro}"
+        if include_suffix and version.pre:
+            ver += f"-{version.pre[0]}{version.pre[1]}"
+        return ver
+
+
+# llvm/CMakeLists.txt
+class CMakeProcessor(Processor):
+    def process_line(self, line: str) -> str:
+        nline = line
+
+        # LLVM_VERSION_SUFFIX should be set to -rcX or be blank if we are
+        # building a final version.
+        if "set(LLVM_VERSION_SUFFIX" in line:
+            if self.suffix:
+                nline = re.sub(
+                    r"set\(LLVM_VERSION_SUFFIX(.*)\)",
+                    f"set(LLVM_VERSION_SUFFIX -{self.suffix[0]}{self.suffix[1]})",
+                    line,
+                )
+            else:
+                nline = re.sub(r"set\(LLVM_VERSION_SUFFIX(.*)\)", f"set(LLVM_VERSION_SUFFIX)", line)
+
+        # Check the rest of the LLVM_VERSION_ lines.
+        elif "set(LLVM_VERSION_" in line:
+            for c, cver in (
+                ("MAJOR", self.major),
+                ("MINOR", self.minor),
+                ("PATCH", self.patch),
+            ):
+                nline = re.sub(
+                    fr"set\(LLVM_VERSION_{c} (\d+)",
+                    fr"set(LLVM_VERSION_{c} {cver}",
+                    line,
+                )
+                if nline != line:
+                    break
+
+        return nline
+
+
+# Process the many bazel files.
+class BazelProcessor(Processor):
+    def process_line(self, line: str) -> str:
+        # This matches the CLANG_VERSION line of clang/Config/config.h
+        nline = line
+        if "CLANG_VERSION " in line:
+            nline = re.sub(
+                r"#define CLANG_VERSION (.*)'",
+                f"#define CLANG_VERSION {self.version_str(include_suffix=False)}'",
+                line,
+            )
+        # Match version strings of LLVM, Clang and LLD overlay headers
+        elif "LLVM_VERSION_STRING" in line or "CLANG_VERSION_STRING" in line or "LLD_VERSION_STRING" in line:
+            nline = re.sub(
+                r"#define (LLVM|CLANG|LLD)_VERSION_STRING ([\\\"]+)[0-9\.rcgit-]+([\\\"]+)",
+                rf"#define \g<1>_VERSION_STRING \g<2>{self.version_str()}\g<3>",
+                line,
+            )
+        # Match the split out MAJOR/MINOR/PATCH versions of LLVM and Clang overlay headers
+        # in LLVM the define is called _PATCH and in clang it's called _PATCHLEVEL
+        elif "LLVM_VERSION_" in line or "CLANG_VERSION_" in line:
+            for c, cver in (
+                ("(MAJOR)", self.major),
+                ("(MINOR)", self.minor),
+                ("(PATCH|PATCHLEVEL)", self.patch),
+            ):
+                nline = re.sub(
+                    fr"(LLVM|CLANG)_VERSION_{c} \d+",
+                    rf"\g<1>_VERSION_\g<2> {cver}",
+                    line,
+                )
+                if nline != line:
+                    break
+        # Match the BACKEND_PACKAGE_STRING in clang/config.h
+        elif "BACKEND_PACKAGE_STRING" in line:
+            nline = re.sub(
+                r'#define BACKEND_PACKAGE_STRING "LLVM ([0-9\.rcgit-]+)"',
+                f'#define BACKEND_PACKAGE_STRING "LLVM {self.version_str()}"',
+                line,
+            )
+
+        return nline
+
+
+# GN build system
+class GNIProcessor(Processor):
+    def process_line(self, line: str) -> str:
+        if "llvm_version_" in line:
+            for c, cver in (
+                ("major", self.major),
+                ("minor", self.minor),
+                ("patch", self.patch),
+            ):
+                nline = re.sub(fr"llvm_version_{c} = \d+", f"llvm_version_{c} = {cver}", line)
+                if nline != line:
+                    return nline
+
+        return line
+
+
+# LIT python file, a simple tuple
+class LitProcessor(Processor):
+    def process_line(self, line: str) -> str:
+        if "__versioninfo__" in line:
+            nline = re.sub(
+                fr"__versioninfo__(.*)\((\d+), (\d+), (\d+)\)",
+                f"__versioninfo__\\1({self.major}, {self.minor}, {self.patch})",
+                line,
+            )
+            return nline
+        return line
+
+
+# Handle libc++ config header
+class LibCXXProcessor(Processor):
+    def process_line(self, line: str) -> str:
+        # match #define _LIBCPP_VERSION 160000 in a relaxed way
+        match = re.match(r".*\s_LIBCPP_VERSION\s+(\d{6})$", line)
+        if match:
+            verstr = f"{str(self.major).zfill(2)}{str(self.minor).zfill(2)}{str(self.patch).zfill(2)}"
+
+            nline = re.sub(
+                fr"_LIBCPP_VERSION (\d+)",
+                f"_LIBCPP_VERSION {verstr}",
+                line,
+            )
+            return nline
+        return line
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        usage="Call this script with a version and it will bump the version for you"
+    )
+    parser.add_argument("version", help="Version to bump to, e.g. 15.0.1", default=None)
+    parser.add_argument("--rc", default=None, type=int, help="RC version")
+    parser.add_argument(
+        "-s",
+        "--source-root",
+        default=None,
+        help="LLVM source root (/path/llvm-project). Defaults to the llvm-project the script is located in.",
+    )
+
+    args = parser.parse_args()
+
+    verstr = args.version
+    if args.rc:
+        verstr += f"-rc{args.rc}"
+
+    # parse the version string with distutils.
+    # note that -rc will end up as version.pre here
+    # since it's a prerelease
+    version = packaging.version.parse(verstr)
+
+    # Find llvm-project root
+    source_root = Path(__file__).resolve().parents[3]
+
+    if args.source_root:
+        source_root = Path(args.source_root).resolve()
+
+    files_to_update = (
+        # Main CMakeLists.
+        (source_root / "llvm" / "CMakeLists.txt", CMakeProcessor()),
+        # Lit configuration
+        (
+            "llvm/utils/lit/lit/__init__.py",
+            LitProcessor(),
+        ),
+        # GN build system
+        (
+            "llvm/utils/gn/secondary/llvm/version.gni",
+            GNIProcessor(),
+        ),
+        # Bazel build system
+        (
+            "utils/bazel/llvm-project-overlay/llvm/include/llvm/Config/llvm-config.h",
+            BazelProcessor(),
+        ),
+        (
+            "utils/bazel/llvm-project-overlay/clang/BUILD.bazel",
+            BazelProcessor(),
+        ),
+        (
+            "utils/bazel/llvm-project-overlay/clang/include/clang/Config/config.h",
+            BazelProcessor(),
+        ),
+        (
+            "utils/bazel/llvm-project-overlay/lld/BUILD.bazel",
+            BazelProcessor(),
+        ),
+        (
+            "libcxx/include/__config",
+            LibCXXProcessor(),
+        ),
+    )
+
+    for f, processor in files_to_update:
+        processor.process_file(source_root / Path(f), version)