[flang][test] Amend test_errors.py to test warnings

[ekieri]

Summary

Presently, test_errors.py only checks errors. If Flang fails to emit an
expected error at a particular line, misphrases it, or emits an
unexpected error, test_errors.py makes the test fail. It does however
not take warnings into account. This patch makes test_errors.py check
warnings the same way as errors.

Warnings are accounted separately from errors, but warnings of
different severity are lumped together. That is, if an expected error
comes out as a warning the test fails, but test_errors.py will not care
if the warning is of "warning" or "portability" severity. To expect a
warning, the 'WARNING:' directive is used.

RFC: https://discourse.llvm.org/t/62577

Depends on D126176
Depends on D126459
Depends on D127337
Depends on D131987
Depends on D132403

[klausler]

See flang/test/Evaluate/test_folding.py, which uses "WARN:" and will fail if a warning is not issued or if a new warning appears. Tests for warnings should treat discrepancies from expected warnings as severely as discrepancies from expected errors.

[awarzynski]

Thanks for suggesting this!

In D125804#3519739, @klausler wrote:

Tests for warnings should treat discrepancies from expected warnings as severely as discrepancies from expected errors.

+1 This would match the behavior in Clang as well (see e.g. https://github.com/llvm/llvm-project/blob/main/clang/test/Sema/attr-alias-elf.c)

[ekieri]

Sure. Would you be OK with proceeding with this one as a first step on the way there? Please see the discussion in the RFC as well.

[ekieri]

In D125804#3523291, @ekieri wrote:

Sure. Would you be OK with proceeding with this one as a first step on the way there? Please see the discussion in the RFC as well.

Nevermind.

[ekieri]

Disallow unexpected warnings, rebase.

[ekieri]

Add WARNING directive to common-blocks.f90 and data06.f90, whose emitted warnings appear correct.

[ekieri]

Add a WARNING directive to resolve35.f90. The warning makes sense and appears to be emitted intentionally.

[ekieri]

Break out updates to tests to D131987

[PeteSteinfeld]

When I run "check-flang" after making these changes, I get 21 failures:

Failed Tests (21):
  Flang :: Semantics/associated.f90
  Flang :: Semantics/bindings01.f90
  Flang :: Semantics/block-data01.f90
  Flang :: Semantics/blockconstruct02.f90
  Flang :: Semantics/call03.f90
  Flang :: Semantics/case01.f90
  Flang :: Semantics/common-blocks.f90
  Flang :: Semantics/data06.f90
  Flang :: Semantics/dosemantics02.f90
  Flang :: Semantics/dosemantics12.f90
  Flang :: Semantics/forall01.f90
  Flang :: Semantics/int-literals.f90
  Flang :: Semantics/resolve108.f90
  Flang :: Semantics/resolve11.f90
  Flang :: Semantics/resolve31.f90
  Flang :: Semantics/resolve35.f90
  Flang :: Semantics/resolve37.f90
  Flang :: Semantics/resolve42.f90
  Flang :: Semantics/resolve59.f90
  Flang :: Semantics/resolve60.f90
  Flang :: Semantics/resolve85.f90

[ekieri]

In D125804#3740598, @PeteSteinfeld wrote:

When I run "check-flang" after making these changes, I get 21 failures:

Yes, I know. The dependencies intend to make those tests conform to the stricter testing of this patch. I will rebase when the last two dependencies are merged. (The base commit is a little stale. Currently, four tests would still fail if this patch was applied on top of main.)

[PeteSteinfeld]

In D125804#3740713, @ekieri wrote:

In D125804#3740598, @PeteSteinfeld wrote:

When I run "check-flang" after making these changes, I get 21 failures:

Yes, I know. The dependencies intend to make those tests conform to the stricter testing of this patch. I will rebase when the last two dependencies are merged. (The base commit is a little stale. Currently, four tests would still fail if this patch was applied on top of main.)

Well the changes look good, but I don't want to approve something that breaks the build.

[ekieri]

Fixed in D136479.

flang/test/Semantics/test_errors.py

 #!/usr/bin/env python3
 
-"""Compiles a source file and checks errors against those listed in the file.
+"""Compiles a source file and checks errors and warnings against those
+   listed in the file.
 
 Parameters:
     sys.argv[1]: a source file with contains the input and expected output
     sys.argv[2]: the Flang frontend driver
     sys.argv[3:]: Optional arguments to the Flang frontend driver"""
 
 import sys
 import re
 import tempfile
 import subprocess
 import common as cm
 
 from difflib import unified_diff
 
 cm.check_args(sys.argv)
 srcdir = cm.set_source(sys.argv[1])
 with open(srcdir, 'r') as f:
     src = f.readlines()
-actual = ""
-expect = ""
+err_actual = ""
+err_expect = ""
+warn_actual = ""
+warn_expect = ""
 diffs = ""
 log = ""
 
 flang_fc1 = cm.set_executable(sys.argv[2])
 flang_fc1_args = sys.argv[3:]
 flang_fc1_options = "-fsyntax-only"
 
 # Compiles, and reads in the output from the compilation process
 cmd = [flang_fc1, *flang_fc1_args, flang_fc1_options, str(srcdir)]
 with tempfile.TemporaryDirectory() as tmpdir:
     try:
         proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                               check=True, universal_newlines=True, cwd=tmpdir)
+        log = proc.stderr
     except subprocess.CalledProcessError as e:
         log = e.stderr
         if e.returncode >= 128:
             print(f"{log}")
             sys.exit(1)
 
 # Cleans up the output from the compilation process to be easier to process
 for line in log.split('\n'):
     m = re.search(r"[^:]*:(\d+:).*(?:error:)(.*)", line)
     if m:
-        actual += m.expand(r"\1\2\n")
+        err_actual += m.expand(r"\1\2\n")
+for line in log.split('\n'):
+    m = re.search(r"[^:]*:(\d+:).*(?:warning:|portability:)(.*)", line)
+    if m:
+        warn_actual += m.expand(r"\1\2\n")
 
-# Gets the expected errors and their line number
+# Gets the expected diagnostics and their line number
 errors = []
+warnings = []
 for i, line in enumerate(src, 1):
-    m = re.search(r"(?:^\s*!\s*ERROR: )(.*)", line)
-    if m:
-        errors.append(m.group(1))
+    me = re.search(r"(?:^\s*!\s*ERROR: )(.*)", line)
+    mw = re.search(r"(?:^\s*!\s*WARNING: )(.*)", line)
+    if me:
+        errors.append(me.group(1))
+        continue
+    if mw:
+        warnings.append(mw.group(1))
         continue
     if errors:
         for x in errors:
-            expect += f"{i}: {x}\n"
+            err_expect += f"{i}: {x}\n"
         errors = []
+    if warnings:
+        for x in warnings:
+            warn_expect += f"{i}: {x}\n"
+        warnings = []
 
-# Compares the expected errors with the compiler errors
-for line in unified_diff(actual.split("\n"), expect.split("\n"), n=0):
+# Compares the expected errors and warnings with the compiler diagnostics
+for line in unified_diff(err_actual.split("\n"), err_expect.split("\n"), n=0):
     line = re.sub(r"(^\-)(\d+:)", r"\nactual at \g<2>", line)
     line = re.sub(r"(^\+)(\d+:)", r"\nexpect at \g<2>", line)
     diffs += line
+for line in unified_diff(warn_actual.split("\n"), warn_expect.split("\n"), n=0):
+    line = re.sub(r"(^\-)(\d+:)", r"\nactual warning at \g<2>", line)
+    line = re.sub(r"(^\+)(\d+:)", r"\nexpected warning at \g<2>", line)
+    diffs += line
 
 if diffs != "":
     print(diffs)
     print()
     print("FAIL")
     sys.exit(1)
 else:
     print()
     print("PASS")