Use tooling (pip-compile) to manage dependencies

[kwk]

Summary

NOTE: This patch updates the incremental package to version 21.3.0. It's worth checking that this works.

This patch is supposed to materialize any dependent python package dependencies into a requirements.txt file. For that to work and to make this patch more readable I've splitted the work in three commits that are worth looking at individually to make more sense. This is because this patch introduces a fully generated file called requirements.txt that used to be maintained by hand. This is why the overall resulting change looks very fuzzy.

1. Renamed requirements.txt to requirements.txt.in

This is the first step in providing a fuller requirements.txt file with
all transient dependencies in it and a simpler human manageable
requirements.txt.in.

To bring back the original requirements.txt file we'll have to run:

$ pip-compile -o requirements.txt  requirements.txt.in

This will be done in a foolowup step to make this an incremental change.

The incremental==17.5.0 wasn't resolvable because of incompatible
versions:

$ pip-compile -o requirements.txt  requirements.txt.in
Could not find a version that matches incremental==17.5.0,>=21.3.0 from https://files.pythonhosted.org/packages/f5/1d/c98a587dc06e107115cf4a58b49de20b19222c83d75335a192052af4c4b7/incremental-17.5.0-py2.py3-none-any.whl#sha256=717e12246dddf231a349175f48d74d93e2897244939173b01974ab6661406b9f (from -r requirements.txt.in (line 12))
Tried: 15.0.0, 15.1.0, 15.2.0, 15.3.0, 16.9.0, 16.9.1, 16.10.0, 16.10.0, 16.10.1, 16.10.1, 17.5.0, 17.5.0, 21.3.0, 21.3.0
There are incompatible versions in the resolved dependencies:
  incremental==17.5.0 from https://files.pythonhosted.org/packages/f5/1d/c98a587dc06e107115cf4a58b49de20b19222c83d75335a192052af4c4b7/incremental-17.5.0-py2.py3-none-any.whl#sha256=717e12246dddf231a349175f48d74d93e2897244939173b01974ab6661406b9f (from -r requirements.txt.in (line 12))
  incremental>=21.3.0 (from Twisted==22.1.0->-r requirements.txt.in (line 33))
  incremental (from treq==22.1.0->-r requirements.txt.in (line 34))

In the next step we'll see what happens when we generate the
requirements.txt file. Then the dependend packages will agree on a
version of the incremenal package and hopefully it is useful to any
direct dependency in the code.

2. Bump incremental package from 17.5.0 to 21.3.0

After running the following command without incremental present in the
requirements.txt.in file we determined a new version that is 21.3.0:

$ pip-compile -o requirements.txt  requirements.txt.in
$ grep "incremental" requirements.txt
incremental==21.3.0

NOTE: This is totally driven by a tool and not by checking if things still work.

In the next step we're going to bring back the requirements.txt file.

3. Bring back generated requirements.txt file.

To see the difference that I've made, do this:

Get the old requirements.txt file:

$ git show upstream/main:requirements.txt > /tmp/old

Compare the old requirements.txt against the new requirements.txt.in:

$ diff -u /tmp/old requirements.txt.in
--- /tmp/old	2022-02-22 13:56:08.199799503 +0100
+++ requirements.txt.in	2022-02-21 13:34:12.061910134 +0100
@@ -1,10 +1,15 @@
+# Convert this file into a requirements.txt file by running:
+#
+# pip install pip-tools
+# pip-compile -o requirements.txt  requirements.txt.in
+
 attrs==20.3.0
 six==1.15.0
 Automat==20.2.0
 constantly==15.1.0
 idna==2.10
 hyperlink==21.0.0
-incremental==17.5.0
+incremental==21.3.0
 Jinja2==2.11.3
 PyHamcrest==2.0.2
 Tempita==0.5.2

[kwk]

@gkistanova can you have a look at this non-functional "cleanup" change please?

[gkistanova]

Thanks, Konrad!

I was planning to start using the pip-tools, but you bet me to it.

I think the list of the direct dependencies needs some polishing, for example, twisted is a direct dependency. But this is not critical at this point.

Let me stage this requirements.txt for a few days or a week to see if anything unexpected would come from this version of incremental, and we will move from there.

[gkistanova]

It was staged long enough without any discovered issues.
Thanks, Konrad!
Please feel free to commit. I'll follow up with some changes to clean the dependencies.

requirements.txt

+#
+# This file is autogenerated by pip-compile with python 3.10
+# To update, run:
+#
+#    pip-compile --output-file=requirements.txt requirements.txt.in
+#
 attrs==20.3.0
-six==1.15.0
-Automat==20.2.0
+    # via
+    #   -r requirements.txt.in
+    #   automat
+    #   service-identity
+    #   treq
+    #   twisted
+automat==20.2.0
+    # via
+    #   -r requirements.txt.in
+    #   twisted
+certifi==2020.12.5
+    # via
+    #   -r requirements.txt.in
+    #   requests
+cffi==1.14.5
+    # via
+    #   -r requirements.txt.in
+    #   cryptography
+chardet==4.0.0
+    # via
+    #   -r requirements.txt.in
+    #   requests
 constantly==15.1.0
-idna==2.10
-hyperlink==21.0.0
-incremental==17.5.0
-Jinja2==2.11.3
-PyHamcrest==2.0.2
-Tempita==0.5.2
+    # via
+    #   -r requirements.txt.in
+    #   twisted
+cryptography==3.4.5
+    # via
+    #   -r requirements.txt.in
+    #   pyopenssl
+    #   service-identity
 decorator==4.4.2
+    # via
+    #   -r requirements.txt.in
+    #   sqlalchemy-migrate
+hyperlink==21.0.0
+    # via
+    #   -r requirements.txt.in
+    #   treq
+    #   twisted
+idna==2.10
+    # via
+    #   -r requirements.txt.in
+    #   hyperlink
+    #   requests
+    #   twisted
+incremental==21.3.0
+    # via
+    #   -r requirements.txt.in
+    #   treq
+    #   twisted
+jinja2==2.11.3
+    # via -r requirements.txt.in
+markupsafe==2.1.0
+    # via jinja2
 pbr==5.5.1
-sqlparse==0.4.2
-SQLAlchemy==1.3.23
-sqlalchemy-migrate==0.13.0
-certifi==2020.12.5
-chardet==4.0.0
-urllib3==1.26.8
-requests==2.25.1
-zope.interface==5.2.0
-pycparser==2.20
-pyOpenSSL==20.0.1
+    # via
+    #   -r requirements.txt.in
+    #   sqlalchemy-migrate
 pyasn1==0.4.8
+    # via
+    #   -r requirements.txt.in
+    #   pyasn1-modules
+    #   service-identity
 pyasn1-modules==0.2.8
+    # via
+    #   -r requirements.txt.in
+    #   service-identity
+pycparser==2.20
+    # via
+    #   -r requirements.txt.in
+    #   cffi
+pyhamcrest==2.0.2
+    # via -r requirements.txt.in
+pyopenssl==20.0.1
+    # via
+    #   -r requirements.txt.in
+    #   twisted
+requests==2.25.1
+    # via
+    #   -r requirements.txt.in
+    #   treq
 service-identity==18.1.0
-cryptography==3.4.5
-cffi==1.14.5
-Twisted==22.1.0
+    # via
+    #   -r requirements.txt.in
+    #   twisted
+six==1.15.0
+    # via
+    #   -r requirements.txt.in
+    #   automat
+    #   pyopenssl
+    #   sqlalchemy-migrate
+sqlalchemy==1.3.23
+    # via
+    #   -r requirements.txt.in
+    #   sqlalchemy-migrate
+sqlalchemy-migrate==0.13.0
+    # via -r requirements.txt.in
+sqlparse==0.4.2
+    # via
+    #   -r requirements.txt.in
+    #   sqlalchemy-migrate
+tempita==0.5.2
+    # via
+    #   -r requirements.txt.in
+    #   sqlalchemy-migrate
 treq==22.1.0
+    # via -r requirements.txt.in
+twisted[tls]==22.1.0
+    # via
+    #   -r requirements.txt.in
+    #   treq
+typing-extensions==4.1.1
+    # via twisted
+urllib3==1.26.8
+    # via
+    #   -r requirements.txt.in
+    #   requests
+zope-interface==5.2.0
+    # via
+    #   -r requirements.txt.in
+    #   twisted
+
+# The following packages are considered to be unsafe in a requirements file:
+# setuptools

requirements.txt.in

+# Convert this file into a requirements.txt file by running:
+#
+# pip install pip-tools
+# pip-compile -o requirements.txt  requirements.txt.in
+
 attrs==20.3.0
 six==1.15.0
 Automat==20.2.0
 constantly==15.1.0
 idna==2.10
 hyperlink==21.0.0
-incremental==17.5.0
+incremental==21.3.0
 Jinja2==2.11.3
 PyHamcrest==2.0.2
 Tempita==0.5.2
 decorator==4.4.2
 pbr==5.5.1
 sqlparse==0.4.2
 SQLAlchemy==1.3.23
 sqlalchemy-migrate==0.13.0