[flang][test] Amend test_errors.py to test warnings

[ekieri]

Summary

Presently, test_errors.py only checks errors. If Flang fails to emit an
expected error at a particular line, misphrases it, or emits an
unexpected error, test_errors.py makes the test fail. It does however
not take warnings into account. This patch makes test_errors.py check
warnings the same way as errors.

Warnings are accounted separately from errors, but warnings of
different severity are lumped together. That is, if an expected error
comes out as a warning the test fails, but test_errors.py will not care
if the warning is of "warning" or "portability" severity. To expect a
warning, the 'WARNING:' directive is used.

RFC: https://discourse.llvm.org/t/62577

Depends on D126176
Depends on D126459
Depends on D127337
Depends on D131987
Depends on D132403

[klausler]

See flang/test/Evaluate/test_folding.py, which uses "WARN:" and will fail if a warning is not issued or if a new warning appears. Tests for warnings should treat discrepancies from expected warnings as severely as discrepancies from expected errors.

[awarzynski]

Thanks for suggesting this!

In D125804#3519739, @klausler wrote:

Tests for warnings should treat discrepancies from expected warnings as severely as discrepancies from expected errors.

+1 This would match the behavior in Clang as well (see e.g. https://github.com/llvm/llvm-project/blob/main/clang/test/Sema/attr-alias-elf.c)

[ekieri]

Sure. Would you be OK with proceeding with this one as a first step on the way there? Please see the discussion in the RFC as well.

[ekieri]

In D125804#3523291, @ekieri wrote:

Sure. Would you be OK with proceeding with this one as a first step on the way there? Please see the discussion in the RFC as well.

Nevermind.

[ekieri]

Disallow unexpected warnings, rebase.

[ekieri]

Add WARNING directive to common-blocks.f90 and data06.f90, whose emitted warnings appear correct.

[ekieri]

Add a WARNING directive to resolve35.f90. The warning makes sense and appears to be emitted intentionally.

[ekieri]

Break out updates to tests to D131987

[PeteSteinfeld]

When I run "check-flang" after making these changes, I get 21 failures:

Failed Tests (21):
  Flang :: Semantics/associated.f90
  Flang :: Semantics/bindings01.f90
  Flang :: Semantics/block-data01.f90
  Flang :: Semantics/blockconstruct02.f90
  Flang :: Semantics/call03.f90
  Flang :: Semantics/case01.f90
  Flang :: Semantics/common-blocks.f90
  Flang :: Semantics/data06.f90
  Flang :: Semantics/dosemantics02.f90
  Flang :: Semantics/dosemantics12.f90
  Flang :: Semantics/forall01.f90
  Flang :: Semantics/int-literals.f90
  Flang :: Semantics/resolve108.f90
  Flang :: Semantics/resolve11.f90
  Flang :: Semantics/resolve31.f90
  Flang :: Semantics/resolve35.f90
  Flang :: Semantics/resolve37.f90
  Flang :: Semantics/resolve42.f90
  Flang :: Semantics/resolve59.f90
  Flang :: Semantics/resolve60.f90
  Flang :: Semantics/resolve85.f90

[ekieri]

In D125804#3740598, @PeteSteinfeld wrote:

When I run "check-flang" after making these changes, I get 21 failures:

Yes, I know. The dependencies intend to make those tests conform to the stricter testing of this patch. I will rebase when the last two dependencies are merged. (The base commit is a little stale. Currently, four tests would still fail if this patch was applied on top of main.)

[PeteSteinfeld]

In D125804#3740713, @ekieri wrote:

In D125804#3740598, @PeteSteinfeld wrote:

When I run "check-flang" after making these changes, I get 21 failures:

Yes, I know. The dependencies intend to make those tests conform to the stricter testing of this patch. I will rebase when the last two dependencies are merged. (The base commit is a little stale. Currently, four tests would still fail if this patch was applied on top of main.)

Well the changes look good, but I don't want to approve something that breaks the build.

[ekieri]

Fixed in D136479.

flang/test/Semantics/call03.f90

subroutine test06 ! 15.5.2.4(4)
     type(pdt(3)) :: var3
     type(pdt(4)) :: var4
     type(pdtWithDefault) :: defaultVardefault
     type(pdtWithDefault(3)) :: defaultVar3
     type(pdtWithDefault(4)) :: defaultVar4
     character :: ch1
     !ERROR: Actual argument variable length '1' is less than expected length '2'
     call ch2(ch1)
-    !WARN: Actual argument expression length '0' is less than expected length '2'
+    !WARNING: Actual argument expression length '0' is less than expected length '2'
     call ch2("")
     call pdtdefault(vardefault)
     call pdtdefault(var3)
     call pdtdefault(var4) ! error
     call pdt3(vardefault) ! error
     !ERROR: Actual argument type 'pdt(n=3_4)' is not compatible with dummy argument type 'pdt(n=4_4)'
     call pdt3(var3) ! error
     call pdt3(var4)

flang/test/Semantics/case01.f90

select case (grade3)
     case (.false.)
      !ERROR: CASE (.true._1) conflicts with previous cases
      case (.true.)
     !ERROR: CASE (.false._1) conflicts with previous cases
      case (grade5)
   end select
 
   select case (grade2)
-     case (51:50) ! warning
+     !WARNING: CASE has lower bound greater than upper bound
+     case (51:50)
      case (100:)
      case (:30)
      case (40)
      case (90)
      case (91:99)
      !ERROR: CASE (81_4:90_4) conflicts with previous cases
      case (81:90)
      !ERROR: CASE (:80_4) conflicts with previous cases
select case(i)
     case(:0)
   end select
 end
 
 subroutine test_overflow
   integer :: j
   select case(1_1)
   case (127)
-  !WARN: CASE value (128_4) overflows type (INTEGER(1)) of SELECT CASE expression
+  !WARNING: CASE value (128_4) overflows type (INTEGER(1)) of SELECT CASE expression
   case (128)
-  !WARN: CASE value (129_4) overflows type (INTEGER(1)) of SELECT CASE expression
-  !WARN: CASE value (130_4) overflows type (INTEGER(1)) of SELECT CASE expression
+  !WARNING: CASE value (129_4) overflows type (INTEGER(1)) of SELECT CASE expression
+  !WARNING: CASE value (130_4) overflows type (INTEGER(1)) of SELECT CASE expression
   case (129:130)
-  !WARN: CASE value (-130_4) overflows type (INTEGER(1)) of SELECT CASE expression
-  !WARN: CASE value (-129_4) overflows type (INTEGER(1)) of SELECT CASE expression
+  !WARNING: CASE value (-130_4) overflows type (INTEGER(1)) of SELECT CASE expression
+  !WARNING: CASE value (-129_4) overflows type (INTEGER(1)) of SELECT CASE expression
   case (-130:-129)
   case (-128)
   !ERROR: Must be a scalar value, but is a rank-1 array
   case ([1, 2])
   !ERROR: Must be a constant value
   case (j)
   case default
   end select
 end

flang/test/Semantics/common-blocks.f90

 ! RUN: %python %S/test_errors.py %s %flang_fc1
 
 ! Test check that enforce that a common block is initialized
 ! only once in a file.
 
 subroutine init_1
   common x, y
   common /a/ xa, ya
   common /b/ xb, yb
-  !CHECK: portability: Blank COMMON object 'x' in a DATA statement is not standard
+  !WARNING: Blank COMMON object 'x' in a DATA statement is not standard
   data x /42./, xa /42./, yb/42./
 end subroutine
 
 subroutine init_conflict
   !ERROR: Multiple initialization of COMMON block //
   common x, y
   !ERROR: Multiple initialization of COMMON block /a/
   common /a/ xa, ya
   common /b/ xb, yb
   equivalence (yb, yb_eq)
   !ERROR: Multiple initialization of COMMON block /b/
   data x /66./, xa /66./, yb_eq /66./
 end subroutine

flang/test/Semantics/dosemantics02.f90

IMPURE FUNCTION random() RESULT(i)
       i = 35
     END FUNCTION random
 END SUBROUTINE do_concurrent_c1121
 
 SUBROUTINE s1()
   INTEGER, PARAMETER :: constInt = 0
 
   ! Warn on this one for backwards compatibility
+  !WARNING: DO step expression should not be zero
   DO 10 I = 1, 10, 0
   10 CONTINUE
 
   ! Warn on this one for backwards compatibility
+  !WARNING: DO step expression should not be zero
   DO 20 I = 1, 10, 5 - 5
   20 CONTINUE
 
   ! Error, no compatibility requirement for DO CONCURRENT
   !ERROR: DO CONCURRENT step expression may not be zero
   DO CONCURRENT (I = 1 : 10 : 0)
   END DO
 
   ! Error, this time with an integer constant
   !ERROR: DO CONCURRENT step expression may not be zero
   DO CONCURRENT (I = 1 : 10 : constInt)
   END DO
 end subroutine s1

flang/test/Semantics/dosemantics12.f90

subroutine s12()
   call intentOutSub(jvar, ivar)
   do ivar = 1,10
 !ERROR: Cannot redefine DO variable 'ivar'
     call intentOutSub(jvar, ivar)
   end do
 
   call intentInOutSub(jvar, ivar)
   do ivar = 1,10
+    !WARNING: Possible redefinition of DO variable 'ivar'
     call intentInOutSub(jvar, ivar)
   end do
 
 contains
   subroutine intentInSub(arg1, arg2)
     integer, intent(in) :: arg1
     integer, intent(in) :: arg2
   end subroutine intentInSub
!ERROR: Cannot redefine DO variable 'ivar'
 
   ! Error for passing a DO variable to an INTENT(OUT) dummy, more complex 
   ! expression
   do ivar = 1, 10
 !ERROR: Cannot redefine DO variable 'ivar'
     jvar = 83 + intentInFunc(intentOutFunc(ivar))
   end do
 
-  ! Warning for passing a DO variable to an INTENT(INOUT) dummy
   do ivar = 1, 10
+    !WARNING: Possible redefinition of DO variable 'ivar'
     jvar = intentInOutFunc(ivar)
   end do
 
 contains
   function intentInFunc(dummyArg)
     integer, intent(in) :: dummyArg
     integer  :: intentInFunc
 

flang/test/Semantics/forall01.f90

subroutine forall3
   real :: x
   forall(i=1:10)
     !ERROR: Cannot redefine FORALL variable 'i'
     i = 1
   end forall
   forall(i=1:10)
     forall(j=1:10)
       !ERROR: Cannot redefine FORALL variable 'i'
+      !WARNING: FORALL index variable 'j' not used on left-hand side of assignment
       i = 1
     end forall
   end forall
   !ERROR: Cannot redefine FORALL variable 'i'
   forall(i=1:10) i = 1
 end
 
 subroutine forall4

 
 ! Note: this gets warnings but not errors
 subroutine forall5
   real, target :: x(10), y(10)
   forall(i=1:10)
     x(i) = y(i)
   end forall
   forall(i=1:10)
-    x = y  ! warning: i not used on LHS
+    !WARNING: FORALL index variable 'i' not used on left-hand side of assignment
+    x = y
     forall(j=1:10)
-      x(i) = y(i)  ! warning: j not used on LHS
-      x(j) = y(j)  ! warning: i not used on LHS
+      !WARNING: FORALL index variable 'j' not used on left-hand side of assignment
+      x(i) = y(i)
+      !WARNING: FORALL index variable 'i' not used on left-hand side of assignment
+      x(j) = y(j)
     endforall
   endforall
   do concurrent(i=1:10)
     x = y
+    !WARNING: FORALL index variable 'i' not used on left-hand side of assignment
     forall(i=1:10) x = y
   end do
 end
 
 subroutine forall6
   type t
     real, pointer :: p
   end type
   type(t) :: a(10)
   real, target :: b(10)
   forall(i=1:10)
     a(i)%p => b(i)
-    a(1)%p => b(i)  ! warning: i not used on LHS
+    !WARNING: FORALL index variable 'i' not used on left-hand side of assignment
+    a(1)%p => b(i)
   end forall
 end

flang/test/Semantics/int-literals.f90

-! RUN: %python %S/test_errors.py %s %flang_fc1
+! RUN: %python %S/test_errors.py %s %flang_fc1 -pedantic
 ! Fortran syntax considers signed int literals in complex literals
 ! to be a distinct production, not an application of unary +/- to
 ! an unsigned int literal, so they're used here to test overflow
 ! on signed int literal constants.  The literals are tested here
 ! as part of expressions that name resolution must analyze.
 
 complex, parameter :: okj1 = 127_1, okz1 = (+127_1, -128_1)
 !ERROR: Integer literal is too large for INTEGER(KIND=1)

flang/test/Semantics/resolve11.f90

 ! RUN: %python %S/test_errors.py %s %flang_fc1
 module m
   public i
   integer, private :: j
   !ERROR: The accessibility of 'i' has already been specified as PUBLIC
   private i
-  !The accessibility of 'j' has already been specified as PRIVATE
+  !WARNING: The accessibility of 'j' has already been specified as PRIVATE
   private j
 end
 
 module m2
   interface operator(.foo.)
     module procedure ifoo
   end interface
   public :: operator(.foo.)

flang/test/Semantics/resolve31.f90

-! RUN: %python %S/test_errors.py %s %flang_fc1
+! RUN: %python %S/test_errors.py %s %flang_fc1 -pedantic
 ! C735 If EXTENDS appears, SEQUENCE shall not appear.
 ! C738 The same private-or-sequence shall not appear more than once in a
 ! given derived-type-def .
 !
 ! C740 If SEQUENCE appears,
 !  the type shall have at least one component,
 !  each data component shall be declared to be of an intrinsic type or of a sequence type,
 !  the derived type shall not have any type parameter,
subroutine s2
   type, extends(t0) :: t1
   end type
 end subroutine
 
 module m4
   type :: t1
     private
     sequence
-    private  ! not a fatal error
-    sequence ! not a fatal error
+    !WARNING: PRIVATE may not appear more than once in derived type components
+    private
+    !WARNING: SEQUENCE may not appear more than once in derived type components
+    sequence
     real :: t1Field
   end type
   type :: t1a
   end type
   !ERROR: A sequence type may not have the EXTENDS attribute
   type, extends(t1a) :: t2
     sequence
     integer i
type :: sequenceType
     real :: sequenceField
   end type sequenceType
   type :: testType
     sequence
     !ERROR: A sequence type data component must either be of an intrinsic type or a derived sequence type
     class(*), allocatable :: typeStarField
     !ERROR: A sequence type data component must either be of an intrinsic type or a derived sequence type
     type(plainType) :: testField1
-    !Pointers are ok as an extension
+    !WARNING: A sequence type data component that is a pointer to a non-sequence type is not standard
     type(plainType), pointer :: testField1p
     type(sequenceType) :: testField2
     procedure(real), pointer, nopass :: procField
   end type testType
   !ERROR: A sequence type may not have type parameters
   type :: paramType(param)
     integer, kind :: param
     sequence

flang/test/Semantics/resolve59.f90

function f5(x)
     procedure(rfunc), pointer :: f5
     f5 => rfunc
     ! OK call to f5 pointer
     x = acos(f5(x+1))
     !ERROR: Actual argument for 'x=' may not be a procedure
     x = acos(f5)
   end function
   ! Sanity test: f18 handles C1560 violation by ignoring RESULT
-  function f6() result(f6) !OKI (warning)
+  !WARNING: The function name should not appear in RESULT, references to 'f6' inside the function will be considered as references to the result only
+  function f6() result(f6)
   end function
-  function f7() result(f7) !OKI (warning)
+  !WARNING: The function name should not appear in RESULT, references to 'f7' inside the function will be considered as references to the result only
+  function f7() result(f7)
     real :: x, f7
     !ERROR: Recursive call to 'f7' requires a distinct RESULT in its declaration
     x = acos(f7())
     f7 = x
     x = acos(f7) !OK
   end function
 end module
 

flang/test/Semantics/test_errors.py

 #!/usr/bin/env python3
 
-"""Compiles a source file and checks errors against those listed in the file.
+"""Compiles a source file and checks errors and warnings against those
+   listed in the file.
 
 Parameters:
     sys.argv[1]: a source file with contains the input and expected output
     sys.argv[2]: the Flang frontend driver
     sys.argv[3:]: Optional arguments to the Flang frontend driver"""
 
 import sys
 import re
 import tempfile
 import subprocess
 import common as cm
 
 from difflib import unified_diff
 
 cm.check_args(sys.argv)
 srcdir = cm.set_source(sys.argv[1])
 with open(srcdir, 'r') as f:
     src = f.readlines()
-actual = ""
-expect = ""
+err_actual = ""
+err_expect = ""
+warn_actual = ""
+warn_expect = ""
 diffs = ""
 log = ""
 
 flang_fc1 = cm.set_executable(sys.argv[2])
 flang_fc1_args = sys.argv[3:]
 flang_fc1_options = "-fsyntax-only"
 
 # Compiles, and reads in the output from the compilation process
 cmd = [flang_fc1, *flang_fc1_args, flang_fc1_options, str(srcdir)]
 with tempfile.TemporaryDirectory() as tmpdir:
     try:
         proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                               check=True, universal_newlines=True, cwd=tmpdir)
+        log = proc.stderr
     except subprocess.CalledProcessError as e:
         log = e.stderr
         if e.returncode >= 128:
             print(f"{log}")
             sys.exit(1)
 
 # Cleans up the output from the compilation process to be easier to process
 for line in log.split('\n'):
     m = re.search(r"[^:]*:(\d+:).*(?:error:)(.*)", line)
     if m:
-        actual += m.expand(r"\1\2\n")
+        err_actual += m.expand(r"\1\2\n")
+for line in log.split('\n'):
+    m = re.search(r"[^:]*:(\d+:).*(?:warning:|portability:)(.*)", line)
+    if m:
+        warn_actual += m.expand(r"\1\2\n")
 
-# Gets the expected errors and their line number
+# Gets the expected diagnostics and their line number
 errors = []
+warnings = []
 for i, line in enumerate(src, 1):
-    m = re.search(r"(?:^\s*!ERROR: )(.*)", line)
-    if m:
-        errors.append(m.group(1))
+    me = re.search(r"(?:^\s*!ERROR: )(.*)", line)
+    mw = re.search(r"(?:^\s*!WARNING: )(.*)", line)
+    if me:
+        errors.append(me.group(1))
+        continue
+    if mw:
+        warnings.append(mw.group(1))
         continue
     if errors:
         for x in errors:
-            expect += f"{i}: {x}\n"
+            err_expect += f"{i}: {x}\n"
         errors = []
+    if warnings:
+        for x in warnings:
+            warn_expect += f"{i}: {x}\n"
+        warnings = []
 
-# Compares the expected errors with the compiler errors
-for line in unified_diff(actual.split("\n"), expect.split("\n"), n=0):
+# Compares the expected errors and warnings with the compiler diagnostics
+for line in unified_diff(err_actual.split("\n"), err_expect.split("\n"), n=0):
     line = re.sub(r"(^\-)(\d+:)", r"\nactual at \g<2>", line)
     line = re.sub(r"(^\+)(\d+:)", r"\nexpect at \g<2>", line)
     diffs += line
+for line in unified_diff(warn_actual.split("\n"), warn_expect.split("\n"), n=0):
+    if re.match(r"(^\+)(\d+:)", line):
+        line = re.sub(r"(^\+)(\d+:)", r"\nexpected warning at \g<2>", line)
+        diffs += line
 
 if diffs != "":
     print(diffs)
     print()
     print("FAIL")
     sys.exit(1)
 else:
     print()
     print("PASS")