[LNT] Remove Python2 support

[thopre]

Summary

This patch removes all the Python 3 compability imports, thus removing
Python 2 support.

Python 2 is no longer maintained since Jan 2020, a year and a half already.
Perhaps it is time to remove Python 2 support in LNT.

[cmatthews]

FWIW, we are still using python2 everywhere. I think the python3 version needs to stew for a bit so we can find any bugs that might have been introduced.

[thopre]

In D69056#1711373, @cmatthews wrote:

FWIW, we are still using python2 everywhere. I think the python3 version needs to stew for a bit so we can find any bugs that might have been introduced.

Oh yes absolutely. I don't think tests cover all issues so we'll find more. I'll be using LNT in Python 3 mode as soon as all the patches have landed so hopefully I'll uncover a big portion of those. In terms of timeframe I was thinking end of 2020. While Red Hat will have supported versions with Python 2 for a long time yet I think that's a too long time to wait. Does that sound reasonable?

[thopre]

Rebase

[thopre]

Rebase

[tnfchris]

Ohh it's happening!

This looks good but you've missed the cPerf stuff, https://github.com/llvm/llvm-lnt/blob/f48d431f44610e339d00a33d57564c6029c4ff43/lnt/testing/profile/cPerf.cpp#L827 and lnt/testing/profile/__init__.py

[thopre]

In D69056#2981853, @tnfchris wrote:

Ohh it's happening!

This looks good but you've missed the cPerf stuff, https://github.com/llvm/llvm-lnt/blob/f48d431f44610e339d00a33d57564c6029c4ff43/lnt/testing/profile/cPerf.cpp#L827 and lnt/testing/profile/__init__.py

What's Python2 specific in lnt/testing/profile/__init__.py?

[thopre]

Remove cPerf Python2 code

[tnfchris]

In D69056#2981864, @thopre wrote:

In D69056#2981853, @tnfchris wrote:

Ohh it's happening!

This looks good but you've missed the cPerf stuff, https://github.com/llvm/llvm-lnt/blob/f48d431f44610e339d00a33d57564c6029c4ff43/lnt/testing/profile/cPerf.cpp#L827 and lnt/testing/profile/__init__.py

What's Python2 specific in lnt/testing/profile/__init__.py?

The futures import at the top https://github.com/llvm/llvm-lnt/blob/f48d431f44610e339d00a33d57564c6029c4ff43/lnt/testing/profile/__init__.py#L4

[thopre]

In D69056#2981873, @tnfchris wrote:

In D69056#2981864, @thopre wrote:

In D69056#2981853, @tnfchris wrote:

Ohh it's happening!

This looks good but you've missed the cPerf stuff, https://github.com/llvm/llvm-lnt/blob/f48d431f44610e339d00a33d57564c6029c4ff43/lnt/testing/profile/cPerf.cpp#L827 and lnt/testing/profile/__init__.py

What's Python2 specific in lnt/testing/profile/__init__.py?

The futures import at the top https://github.com/llvm/llvm-lnt/blob/f48d431f44610e339d00a33d57564c6029c4ff43/lnt/testing/profile/__init__.py#L4

It's already there: https://reviews.llvm.org/D69056#change-H7ZXp1LFQQlR

[tnfchris]

In D69056#2981892, @thopre wrote:

In D69056#2981873, @tnfchris wrote:

In D69056#2981864, @thopre wrote:

In D69056#2981853, @tnfchris wrote:

Ohh it's happening!

This looks good but you've missed the cPerf stuff, https://github.com/llvm/llvm-lnt/blob/f48d431f44610e339d00a33d57564c6029c4ff43/lnt/testing/profile/cPerf.cpp#L827 and lnt/testing/profile/__init__.py

What's Python2 specific in lnt/testing/profile/__init__.py?

The futures import at the top https://github.com/llvm/llvm-lnt/blob/f48d431f44610e339d00a33d57564c6029c4ff43/lnt/testing/profile/__init__.py#L4

It's already there: https://reviews.llvm.org/D69056#change-H7ZXp1LFQQlR

Oh I had missed it :D

[tnfchris]

Thanks! this looks complete now.

I guess the only question is, should we bump up the version number? to make a clear line in the sand? otherwise it's hard to tell people which version supports python 2 (in case they need to run an older one for whatever reason)? or more importantly, tell which one is python 3.

[thopre]

In D69056#2981910, @tnfchris wrote:

Thanks! this looks complete now.

I guess the only question is, should we bump up the version number? to make a clear line in the sand? otherwise it's hard to tell people which version supports python 2 (in case they need to run an older one for whatever reason)? or more importantly, tell which one is python 3.

Good point. We are currently in 0.4.2dev0 so perhaps we should tag a 0.4.2 first and then pass to 0.4.3dev0?

[tnfchris]

In D69056#2981915, @thopre wrote:

In D69056#2981910, @tnfchris wrote:

Thanks! this looks complete now.

I guess the only question is, should we bump up the version number? to make a clear line in the sand? otherwise it's hard to tell people which version supports python 2 (in case they need to run an older one for whatever reason)? or more importantly, tell which one is python 3.

Good point. We are currently in 0.4.2dev0 so perhaps we should tag a 0.4.2 first and then pass to 0.4.3dev0?

Perhaps 0.5 instead? that would give someone wanting to continue Python 2 support for whatever reason a way to continue in the 0.4.x ranges?

[thopre]

In D69056#2981941, @tnfchris wrote:

In D69056#2981915, @thopre wrote:

In D69056#2981910, @tnfchris wrote:

Thanks! this looks complete now.

I guess the only question is, should we bump up the version number? to make a clear line in the sand? otherwise it's hard to tell people which version supports python 2 (in case they need to run an older one for whatever reason)? or more importantly, tell which one is python 3.

Good point. We are currently in 0.4.2dev0 so perhaps we should tag a 0.4.2 first and then pass to 0.4.3dev0?

Perhaps 0.5 instead? that would give someone wanting to continue Python 2 support for whatever reason a way to continue in the 0.4.x ranges?

Good point, release 0.4.2 and then change version to 0.5.0dev0 on the main branch. If we realize we want to do more releases with Python2 we can create a python2 branch from the release but I think we should not do it preemptively. Thought @cmatthews ?

[thopre]

In D69056#3064888, @thopre wrote:

In D69056#2981941, @tnfchris wrote:

In D69056#2981915, @thopre wrote:

In D69056#2981910, @tnfchris wrote:

Thanks! this looks complete now.

I guess the only question is, should we bump up the version number? to make a clear line in the sand? otherwise it's hard to tell people which version supports python 2 (in case they need to run an older one for whatever reason)? or more importantly, tell which one is python 3.

Good point. We are currently in 0.4.2dev0 so perhaps we should tag a 0.4.2 first and then pass to 0.4.3dev0?

Perhaps 0.5 instead? that would give someone wanting to continue Python 2 support for whatever reason a way to continue in the 0.4.x ranges?

Good point, release 0.4.2 and then change version to 0.5.0dev0 on the main branch. If we realize we want to do more releases with Python2 we can create a python2 branch from the release but I think we should not do it preemptively. Thought @cmatthews ?

Ping @cmatthews ?

[thopre]

Rebase

[cmatthews]

I think this would be safe to land now. Are you still interested in landing it?

[thopre]

In D69056#3380914, @cmatthews wrote:

I think this would be safe to land now. Are you still interested in landing it?

Yes, let me rebase and see if testsuite is still passing.

[thopre]

Rebase

[cmatthews]

LGTM

docs/conf.py

 # -*- coding: utf-8 -*-
 #
 # LNT documentation build configuration file, created by
 # sphinx-quickstart on Fri Dec 25 10:01:58 2009.
 #
 # This file is execfile()d with the current directory set to its containing dir.
 #
 # Note that not all possible configuration values are present in this
 # autogenerated file.
 #
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-from __future__ import print_function
 import datetime
 import sys
 import os
 
 # When we build in readthedocs, we can't have this package.
 use_bootstrap = True
 try:
     import sphinx_bootstrap_theme

examples/functions.py

 #!/usr/bin/env python
 """
 Simple example of a test generator which just produces data on some
 mathematical functions, keyed off of the current time.
 """
-from __future__ import print_function
 from lnt.testing import Machine, Run, TestSamples, Report
 import math
 import random
 import sys
 import time
 
 
 def main():

examples/run_to_csv.py

 """Convert the JSON run file passed on stdin to a csv"""
-from __future__ import print_function
 import json
-
 import sys
 
 
 data = json.load(sys.stdin)
 
 titles = list(next(iter(data['tests'].values())).keys())
 titles.insert(0, titles.pop(titles.index("name")))
 
 print(", ".join(titles))
 
 for i, result in data['tests'].items():
 
     for title in titles:
         print(result[title], end=' ')
         print(", ", end=' ')
     print()

lnt/external/stats/pstat.py

 ## 12/05/98 ... updated doc-strings
 ##              added features to collapse() function
 ##              added flat() function for lists
 ##              fixed a broken asortrows() 
 ## 11/16/98 ... fixed minor bug in aput for 1D arrays
 ##
 ## 11/08/98 ... fixed aput to output large arrays correctly
 
-from __future__ import absolute_import
-from __future__ import print_function
 import string, copy
 from types import *
 
 __version__ = 0.4
 
 ###===========================  LIST FUNCTIONS  ==========================
 ###
 ### Here are the list functions, DEFINED FOR ALL SYSTEMS.

lnt/external/stats/stats.py

 ## 12/01/98 ... added anova() function (requires NumPy)
 ##              incorporated Dispatch class
 ## 11/12/98 ... added functionality to amean, aharmonicmean, ageometricmean
 ##              added 'asum' function (added functionality to N.add.reduce)
 ##              fixed both moment and amoment (two errors)
 ##              changed name of skewness and askewness to skew and askew
 ##              fixed (a)histogram (which sometimes counted points <lowerlimit)
 
-from __future__ import absolute_import
-from __future__ import print_function
-from builtins import input
 from . import pstat        # required 3rd party module
 import math, string, copy  # required python modules
 from types import *
 
 __version__ = 0.6
 
 ############# DISPATCH CODE ##############
 

lnt/formats/__init__.py

 """
 Utilities for converting to LNT's test format.
 
 LNT formats are described by dictionaries with 'name', 'read', and 'write'
 fields. Only the 'name' field is required. The 'read' field should be a
 callable taking a path_or_file object, the 'write' function should be a
 callable taking a Python object to write, and the path_or_file to write to.
 """
 
-from __future__ import absolute_import
 from typing import List, Dict
 from .PlistFormat import format as plist
 from .JSONFormat import format as json
 
 formats = [plist, json]  # type: List[Dict]
 formats_by_name = dict((f['name'], f) for f in formats)  # type: Dict[str, Dict]
 format_names = list(formats_by_name.keys())  # type: List[str]
 

lnt/lnttool/__init__.py

-from __future__ import absolute_import
 from .main import main

lnt/lnttool/admin.py

 #!/usr/bin/env python
-from future import standard_library
-standard_library.install_aliases()
 import click
 from .common import submit_options
 
 
 def _load_dependencies():
     global yaml, sys, requests, json, os, http
     yaml = __import__('yaml', globals(), locals())
     sys = __import__('sys', globals(), locals())

lnt/lnttool/create.py

-from __future__ import print_function
 import click
 import platform
 
 
 kConfigVersion = (0, 1, 0)
 kConfigTemplate = """\
 # LNT configuration file
 #

lnt/lnttool/main.py

 """Implement the command line 'lnt' tool."""
-from __future__ import print_function
 from .common import init_logger
 from .common import submit_options
 from .convert import action_convert
 from .create import action_create
 from .import_data import action_import
 from .import_report import action_importreport
 from .updatedb import action_updatedb
 from .viewcomparison import action_view_comparison

lnt/lnttool/viewcomparison.py

-from future import standard_library
-standard_library.install_aliases()
 import click
 import time
 import urllib.request
 import webbrowser
 import sys
 
 
 def _start_browser(url, debug=False):

lnt/server/db/testsuite.py

 """
 Database models for the TestSuites abstraction.
 """
 
-from __future__ import absolute_import
 import json
 import lnt
 from . import util
 
 import sqlalchemy
 import sqlalchemy.ext.declarative
 import sqlalchemy.orm
 from sqlalchemy import Column, Integer, ForeignKey, String, Binary, Float

lnt/server/db/testsuitedb.py

 """
 Database models for the TestSuite databases themselves.
 
 These are a bit magical because the models themselves are driven by the test
 suite metadata, so we only create the classes at runtime.
 """
 
-from __future__ import absolute_import
 import datetime
 import json
 import os
 import itertools
 
 import aniso8601
 import sqlalchemy
 import flask

lnt/server/reporting/dailyreport.py

-from future import standard_library
-standard_library.install_aliases()
 from lnt.server.reporting.analysis import REGRESSED, UNCHANGED_FAIL
 from lnt.server.reporting.report import RunResult, RunResults, report_css_styles, pairs, OrderAndHistory
 from lnt.util import multidict
 import datetime
 import lnt.server.reporting.analysis
 import lnt.server.ui.app
 import re
 import sqlalchemy.sql

lnt/server/ui/app.py

-from __future__ import print_function
-from future import standard_library
-standard_library.install_aliases()
 import io
 import logging
 import logging.handlers
 import sys
 import time
 import traceback
 from logging import Formatter
 

lnt/server/ui/filters.py

-from future import standard_library
-standard_library.install_aliases()
 import io
 import datetime
 import pprint
 import urllib.parse
 import time
 
 from lnt.server.ui import util
 

lnt/server/ui/views.py

-from future import standard_library
-standard_library.install_aliases()
 import datetime
 import json
 import os
 import re
 import time
 from collections import namedtuple, defaultdict
 from urllib.parse import urlparse, urljoin
 

lnt/testing/profile/__init__.py

 # This is the profile implementation registry. Register new profile
 # implementations here.
 
-from __future__ import absolute_import
 from .profilev1impl import ProfileV1
 from .profilev2impl import ProfileV2
 from .perf import LinuxPerfProfile
 IMPLEMENTATIONS = {0: LinuxPerfProfile, 1: ProfileV1, 2: ProfileV2}

lnt/testing/profile/perf.py

-from __future__ import absolute_import
 from lnt.util import logger
 from .profile import ProfileImpl
 from .profilev1impl import ProfileV1
 
 import os
 import traceback
 
 try:

lnt/testing/profile/profilev1impl.py

-from future import standard_library
-standard_library.install_aliases()
 from lnt.testing.profile.profile import ProfileImpl
 import pickle
 import zlib
 
 
 class ProfileV1(ProfileImpl):
     """
 ProfileV1 files not clever in any way. They are simple Python objects with

lnt/testing/profile/profilev2impl.py

-from __future__ import absolute_import
-from builtins import range
-from future import standard_library
-standard_library.install_aliases()
 from .profile import ProfileImpl
 import bz2
 import copy
 import io
 import os
 import struct
 
 """

lnt/testing/util/rcs.py

-from future import standard_library
-standard_library.install_aliases()
 import re
 import os
 from lnt.util import logger
 from lnt.testing.util import commands
 
 _git_svn_id_re = re.compile("^    git-svn-id: [^@]*@([0-9]+) .*$")
 
 

lnt/tests/builtintest.py

 """
 Base class for builtin-in tests.
 """
 
-from __future__ import print_function
 import sys
 import os
 
 from lnt.testing.util.misc import timestamp
 
 import lnt.util.ServerUtil as ServerUtil
 import lnt.util.ImportData as ImportData
 

lnt/tests/compile.py

 """Single file compile-time performance testing"""
-from __future__ import print_function
-from future import standard_library
-standard_library.install_aliases()
 import errno
 import hashlib
 import json
 import os
 import platform
 import pprint
 import re
 import shlex

lnt/tests/nt.py

 """LLVM test-suite compile and execution tests"""
-from __future__ import absolute_import
-from __future__ import print_function
-from future import standard_library
-standard_library.install_aliases()
 import csv
 import os
 import platform
 import re
 import shutil
 import subprocess
 import sys
 import glob

lnt/tests/test_suite.py

 """LLVM test-suite"""
-from __future__ import print_function
 import subprocess
 import tempfile
 import json
 import os
 import shlex
 import platform
 import pipes
 import sys

lnt/util/ImportData.py

-from __future__ import print_function
 from lnt.util import NTEmailReport
 from contextlib import closing
 from lnt.util import logger
 import collections
 import datetime
 import lnt.formats
 import lnt.server.reporting.analysis
 import lnt.testing

lnt/util/ServerUtil.py

 """
 Utility for submitting files to a web server over HTTP.
 """
-from __future__ import print_function
-from future import standard_library
-standard_library.install_aliases()
 import sys
 import urllib.request
 import urllib.parse
 import urllib.error
 import contextlib
 import json
 
 import lnt.server.instance

lnt/util/stats.py

-from __future__ import division
 import math
 from lnt.external.stats.stats import mannwhitneyu as mannwhitneyu_large
 from functools import reduce
 
 
 def safe_min(values):
     """Calculate min, but if given an empty list return None."""
     values = list(values)  # In case this is a complex type, get a simple list.

lnt/util/wsgi_restart.py

 # This code lifted from the mod_wsgi docs.
-from __future__ import print_function
-from future import standard_library
-standard_library.install_aliases()
 import os
 import sys
 import signal
 import threading
 import atexit
 import queue
 
 _interval = 1.0

requirements.client.txt

 python-gnupg==0.3.7
 pytz==2016.10
 WTForms==2.0.2
 Flask-WTF==0.12
 typing
 click==6.7
 pyyaml==3.13
 requests
-future
 -e svn+https://llvm.org/svn/llvm-project/llvm/trunk/utils/lit/#egg=lit

runtime.txt

-python-2.7.10

setup.py

""",
 
     classifiers=[
         'Development Status :: 4 - Beta',
         'Environment :: Web Environment',
         'Intended Audience :: Developers',
         'License :: OSI Approved :: Apache-2.0 with LLVM exception',
         'Natural Language :: English',
         'Operating System :: OS Independent',
-        'Programming Language :: Python',
+        'Programming Language :: Python :: 3',
         'Topic :: Software Development :: Quality Assurance',
         'Topic :: Software Development :: Testing',
     ],
 
     zip_safe=False,
 
     # Additional resource extensions we use.
     package_data={'lnt.server.ui': ['static/*.ico',
entry_points={
         'console_scripts': [
             'lnt = lnt.lnttool:main',
         ],
     },
     install_requires=reqs,
 
     ext_modules=[cPerf],
 
-    python_requires='>=2.7',
+    python_requires='>=3',
 )

tests/SharedInputs/FakeCompilers/fakecompiler.py

 #!/usr/bin/env python
 
 """
 Utilities for "faking" a compiler response.
 """
-from __future__ import print_function
-
 import inspect
 import os
 import sys
 
 g_program = None
 
 
 class FakeCompiler(object):

tests/SharedInputs/create_temp_instance.py

-from __future__ import print_function
 import sys
 import shutil
 import os.path
 import os
 import subprocess
 import hashlib
 import tempfile
 import re

tests/server/db/ImportV4TestSuiteInstance.py

-from __future__ import print_function
 # Check the import process into a v4 test suite DB.
 #
 # We first construct a temporary LNT instance.
 # RUN: rm -rf %t.install
 # RUN: lnt create %t.install
 
 # Import the first test set.
 # RUN: lnt import %t.install %{shared_inputs}/sample-a-small.plist \

tests/server/db/Migrations.py

 # Check that we can perform migration of all of the test instances.
 #
 # RUN: python %s %t
 
-from __future__ import print_function
 import logging
 import os
 import re
 import shutil
 import sys
 import glob
 
 import lnt.server.db.migrate

tests/server/ui/V4Pages.py

-from __future__ import print_function
 # Perform basic sanity checking of the V4 UI pages.
 #
 # create temporary instance
 # Cleanup temporary directory in case one remained from a previous run - also
 # see PR9904.
 # RUN: rm -rf %t.instance
 # RUN: python %{shared_inputs}/create_temp_instance.py \
 # RUN:   %s %{shared_inputs}/SmallInstance %t.instance \
 # RUN:   %S/Inputs/V4Pages_extra_records.sql
 
 # Import a profile
 # RUN: lnt import %t.instance %{shared_inputs}/profile-report.json
 #
 # RUN: lnt import %t.instance %S/Inputs/sample-failed-report1.json
 # RUN: lnt import %t.instance %S/Inputs/sample-failed-report2.json
 #
 # RUN: python %s %t.instance %{tidylib}
 
-from future import standard_library
-standard_library.install_aliases()
 import logging
 import re
 import sys
 import xml.etree.ElementTree as ET
 from html.entities import name2codepoint
 from flask import session
 import lnt.server.db.migrate
 import lnt.server.ui.app

tests/testing/CalltreeDataLoading.py

-from __future__ import print_function
 # Check the model bindings for test suite instances.
 #
 # RUN: rm -f %t.db
 # RUN: python %s %S/Inputs/test-input-01.out
 
 import sys
 
 from lnt.testing.util import valgrind
 
 data = valgrind.CalltreeData.frompath(sys.argv[1])
 print(data)
 
 assert data.command == 'true'
 assert tuple(data.events) == ('Ir', 'I1mr', 'ILmr', 'Dr', 'D1mr', 'DLmr',
                               'Dw', 'D1mw', 'DLmw',)

tests/utils/blast.py

 """
 Smash a LNT instance with random submissions.
 
 This should only be used for (load) testing.  It will add a bunch
 of bad data to your instance, so don't use it on a production instance.
 """
-from __future__ import print_function
 ## Just to make sure there are no syntax errors in this.  This does not
 ## actually run a blast.
 # RUN: python %{src_root}/tests/utils/blast.py
-from builtins import range
 import time
 import subprocess
 import os
 import lnt.testing
 import tempfile
 import random
 import sys
 

tox.ini

 [base]
 
 # Files we'd like to keep lint clean. Lets hope we can expand this list.
 whitelist_files = ./setup.py
     ./deployment/
     ./lnt/formats/
     ./lnt/util/
                   ./lnt/tests/
                   ./lnt/testing/
     ./tests/server/ui/
 
 [tox]
-envlist = py27
-          py3
+envlist = py3
           mypy
           flake8
 
 [testenv]
 whitelist_externals = rm
 
 deps = -rrequirements.txt
 
 commands =
     rm -rf ./test_run_tmp
     lit -sv ./tests/
 
 [testenv:mypy]
 basepython = python3
 skip_install = true
 deps =
     mypy
 commands =
     # No where close to passing yet, but nice to have.
-    - mypy --junit-xml=junit-{envname}.xml --py2 --ignore-missing-imports lnt
     - mypy --junit-xml=junit-{envname}.xml --ignore-missing-imports lnt
 
 [testenv:flake8]
 skip_install = true
 deps =
     flake8
 
 commands =
     # Still lots of problems.
     - flake8 --statistics --exclude=./lnt/external/  ./lnt/ ./tests/ ./setup.py ./deployment/
     # But lets keep the clean files clean.
     flake8 {[base]whitelist_files}