[TLI checker] Add more tests

[probinson]

Summary

D114478 identified testing gaps; this patch fills them.

Also, the response-file testing moved to the new test that does all the other multi-file tests; it made more sense there.

[jhenderson]

Nice use of numeric variables by the way!

llvm/test/tools/llvm-tli-checker/Inputs/empty-coff.yaml

1 ↗

(On Diff #391145)

Any particular reason this and empty-elf.yaml aren't inlined directly in the corresponding tests that use them?

llvm/test/tools/llvm-tli-checker/error-cases.test

18

Any particular reason this line is separated from the other checks?

20–23

I'd be inclined to include the full message, i.e. something like error: [[FILE]]: not found (or equivalent).

24

Nit: looks like you've got an extra blank line at EOF?

llvm/test/tools/llvm-tli-checker/multi-file.yaml

35

I read / in this context as an "or" not "and" (and I could imagine others might initially read it as a directory separator).

119

Nit: looks like there's an additional blank line here?

llvm/test/tools/llvm-tli-checker/ps4-tli-check.yaml

46–47

That's unfortunate. I don't suppose there's a way of doing some sort of fancy regex is there? (I can't immediately think of any).

[probinson]

llvm/test/tools/llvm-tli-checker/Inputs/empty-coff.yaml

1 ↗

(On Diff #391145)

I was originally thinking I'd include them in the archive test, but then I didn't, so yes they should be inlined into the tests that use them.

llvm/test/tools/llvm-tli-checker/error-cases.test

18

Only that it isn't combined with other checks, and all the others are in combinations.

20–23

Right, I can use -D on the FileCheck command line to do that.

llvm/test/tools/llvm-tli-checker/ps4-tli-check.yaml

46–47

A regex can have an exact count, but I don't think we can substitute a FileCheck numeric expression inside a regex, and that's what it would require.

[probinson]

Address review comments.

[jhenderson]

LGTM!

llvm/test/tools/llvm-tli-checker/coff-reject.test

+## Currently COFF files aren't supported.
+# RUN: yaml2obj %s -o %t.obj
+# RUN: llvm-tli-checker %t.obj 2>&1 | FileCheck %s
+#
+# CHECK: only ELF-format files are supported
+# CHECK: no symbols found
+# CHECK: NO symbols found!
+
+## Minimal COFF input.
+--- !COFF
+OptionalHeader:  
+  AddressOfEntryPoint: 4144
+  ImageBase:       1073741824
+  SectionAlignment: 4096
+  FileAlignment:   512
+  MajorOperatingSystemVersion: 6
+  MinorOperatingSystemVersion: 0
+  MajorImageVersion: 0
+  MinorImageVersion: 0
+  MajorSubsystemVersion: 6
+  MinorSubsystemVersion: 0
+  Subsystem:       IMAGE_SUBSYSTEM_WINDOWS_CUI
+  DLLCharacteristics: [  ]
+  SizeOfStackReserve: 1048576
+  SizeOfStackCommit: 4096
+  SizeOfHeapReserve: 1048576
+  SizeOfHeapCommit: 4096
+header:          
+  Machine:         IMAGE_FILE_MACHINE_AMD64
+  Characteristics: [  ]
+sections:        
+  - Name:            .text
+    Characteristics: [ IMAGE_SCN_CNT_CODE ]
+    VirtualAddress:  4096
+    VirtualSize:     4
+    SectionData:     C3C3C3C3
+symbols:         
+  - Name:            main
+    Value:           2
+    SectionNumber:   1
+    SimpleType:      IMAGE_SYM_TYPE_NULL
+    ComplexType:     IMAGE_SYM_DTYPE_FUNCTION
+    StorageClass:    IMAGE_SYM_CLASS_EXTERNAL

llvm/test/tools/llvm-tli-checker/error-cases.test

+## No input files.
+# RUN: not llvm-tli-checker 2>&1 | FileCheck %s --check-prefix=NOFILES
+#
+## File not found.
+# RUN: rm -f %t0.txt
+# RUN: llvm-tli-checker %t0.txt 2>&1 | FileCheck %s -DFILE=%t0.txt --check-prefixes=FILE-NOT-FOUND,NOSYMBOLS-AT-ALL
+#
+## Reject a non-binary file.
+# RUN: echo nonsense > %t1.o
+# RUN: llvm-tli-checker %t1.o 2>&1 | FileCheck %s --check-prefixes=INVALID-FILE,NOSYMBOLS-AT-ALL
+#
+## Reject an empty file.
+# RUN: yaml2obj %s -o %t2.o
+# RUN: llvm-tli-checker %t2.o 2>&1 | FileCheck %s -DFILE=%t2.o --check-prefixes=NOSYMBOLS-IN-FILE,NOSYMBOLS-AT-ALL
+#
+# NOFILES: error: no input files
+# FILE-NOT-FOUND-DAG: warning: [[FILE]]: not found
+# INVALID-FILE-DAG: warning: The file was not recognized as a valid object file
+# NOSYMBOLS-AT-ALL-DAG: error: NO symbols found!
+# NOSYMBOLS-IN-FILE-DAG: warning: [[FILE]]: no symbols found
+
+## Minimal ELF with no symbols.
+--- !ELF
+FileHeader:
+    Class:           ELFCLASS64
+    Data:            ELFDATA2LSB
+    OSABI:           ELFOSABI_FREEBSD
+    Type:            ET_DYN
+    Machine:         EM_X86_64
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS

llvm/test/tools/llvm-tli-checker/multi-file.yaml

+## Test that we can pass multiple files. There are several ways to do this:
+## list them all individually, use --libdir to shorten the list, or use a
+## response file (with or without --libdir).
+##
+# REQUIRES: x86-registered-target
+#
+# RUN: mkdir -p %t.dir
+# RUN: yaml2obj --docnum=1 %s -o %t.dir/file1
+# RUN: yaml2obj --docnum=2 %s -o %t.dir/file2
+# RUN: yaml2obj --docnum=3 %s -o %t.dir/file3
+
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 --report=full \
+# RUN:     %t.dir/file1 %t.dir/file2 %t.dir/file3 | \
+# RUN:   FileCheck %s
+#
+## Same result using --libdir.
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 --report=full \
+# RUN:     --libdir=%t.dir file1 file2 file3 | \
+# RUN:   FileCheck %s
+#
+## And with a response file.
+# RUN: echo -e "file1\nfile2\nfile3\n" > %t.dir/resp1.txt
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 --report=full \
+# RUN:     --libdir=%t.dir @%t.dir/resp1.txt | \
+# RUN:   FileCheck %s
+#
+# RUN: echo %t.dir/file1  > %t.dir/resp2.txt
+# RUN: echo %t.dir/file2 >> %t.dir/resp2.txt
+# RUN: echo %t.dir/file3 >> %t.dir/resp2.txt
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 --report=full \
+# RUN:     @%t.dir/resp2.txt | \
+# RUN:   FileCheck %s
+#
+## Also check that an archive works. Put file1 and file2 into an archive,
+## and pass the archive name plus file3 on the command line.
+# RUN: rm -f %t.dir/archive.a
+# RUN: llvm-ar rc %t.dir/archive.a %t.dir/file1 %t.dir/file2
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 --report=full \
+# RUN:     --libdir=%t.dir archive.a file3 | \
+# RUN:   FileCheck %s
+#
+# CHECK: TLI knows {{[0-9]+}} symbols, [[#AVAILABLE:]] available
+# CHECK: >> TLI no  SDK yes: '_ZdaPvj'
+# CHECK: == TLI yes SDK yes: 'abs'
+# CHECK: == TLI yes SDK yes: 'calloc'
+## The yes/no and yes/yes should add up to the total available.
+## The no/yes aren't included in the total available.
+# CHECK: << Total TLI yes SDK no: [[#AVAILABLE-2]]
+# CHECK: >> Total TLI no  SDK yes: 1{{$}}
+# CHECK: == Total TLI yes SDK yes: 2{{$}}
+
+## --separate should give one summary per input file (as listed on the
+# command line; it doesn't break things down by archive member).
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 --separate \
+# RUN:     --libdir=%t.dir archive.a file3 | \
+# RUN:   FileCheck %s --check-prefix=SEPARATE
+#
+# SEPARATE: TLI knows {{[0-9]+}} symbols, [[#AVAILABLE:]] available
+# SEPARATE: Found 2 global function symbols in '{{.*}}archive.a'
+# SEPARATE: << Total TLI yes SDK no:  [[#AVAILABLE-1]]
+# SEPARATE: >> Total TLI no  SDK yes: 1{{$}}
+# SEPARATE: == Total TLI yes SDK yes: 1{{$}}
+# SEPARATE: Found 1 global function symbols in '{{.*}}file3'
+# SEPARATE: << Total TLI yes SDK no:  [[#AVAILABLE-1]]
+# SEPARATE: >> Total TLI no  SDK yes: 0{{$}}
+# SEPARATE: == Total TLI yes SDK yes: 1{{$}}
+
+# file1
+--- !ELF
+FileHeader:
+    Class:           ELFCLASS64
+    Data:            ELFDATA2LSB
+    OSABI:           ELFOSABI_FREEBSD
+    Type:            ET_DYN
+    Machine:         EM_X86_64
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS
+DynamicSymbols:
+  - Name:            abs
+    Type:            STT_FUNC
+    Section:         .text
+    Binding:         STB_GLOBAL
+
+# file2
+--- !ELF
+FileHeader:
+    Class:           ELFCLASS64
+    Data:            ELFDATA2LSB
+    OSABI:           ELFOSABI_FREEBSD
+    Type:            ET_DYN
+    Machine:         EM_X86_64
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS
+DynamicSymbols:
+  - Name:            _ZdaPvj
+    Type:            STT_FUNC
+    Section:         .text
+    Binding:         STB_GLOBAL
+
+# file3
+--- !ELF
+FileHeader:
+    Class:           ELFCLASS64
+    Data:            ELFDATA2LSB
+    OSABI:           ELFOSABI_FREEBSD
+    Type:            ET_DYN
+    Machine:         EM_X86_64
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS
+DynamicSymbols:
+  - Name:            calloc
+    Type:            STT_FUNC
+    Section:         .text
+    Binding:         STB_GLOBAL

llvm/test/tools/llvm-tli-checker/ps4-tli-check.yaml

 # REQUIRES: x86-registered-target
 #
 ## This produces the object that matches expectations for PS4.
 # RUN: yaml2obj %s -DZDAPV=_ZdaPv -o=%t1
 # RUN: llvm-tli-checker --triple=x86_64-scei-ps4 %t1 | FileCheck %s
 #
 ## This produces an object that has _ZdaPvj instead of _ZdaPv.
 # RUN: yaml2obj %s -DZDAPV=_ZdaPvj -o=%t2
-# RUN: echo %t2 > %t2.txt
-# RUN: llvm-tli-checker --triple x86_64-scei-ps4 @%t2.txt | \
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 %t2 | \
 # RUN:     FileCheck %s --check-prefixes=WRONG_SUMMARY,WRONG_DETAIL \
 # RUN:    --implicit-check-not="==" --implicit-check-not="<<" --implicit-check-not=">>"
 #
-# RUN: llvm-tli-checker --triple x86_64-scei-ps4 @%t2.txt --report=summary | \
+## --report=discrepancy is the default, check we get the same output.
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 %t2 --report=discrepancy | \
+# RUN:     FileCheck %s --check-prefixes=WRONG_SUMMARY,WRONG_DETAIL \
+# RUN:    --implicit-check-not="==" --implicit-check-not="<<" --implicit-check-not=">>"
+#
+## --report=summary should not print the details (checked by the
+## implicit-check-not strings).
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 %t.dir/bad --report=summary | \
 # RUN:     FileCheck %s --check-prefix=WRONG_SUMMARY \
 # RUN:     --implicit-check-not="==" --implicit-check-not="<<" --implicit-check-not=">>"
+#
 ## --separate implies --report=summary.
-## Also use a response file to show we support that.
-# RUN: llvm-tli-checker --triple x86_64-scei-ps4 @%t2.txt --separate | \
+# RUN: llvm-tli-checker --triple x86_64-scei-ps4 %t2 --separate | \
 # RUN:     FileCheck %s --check-prefix=WRONG_SUMMARY \
 # RUN:     --implicit-check-not="==" --implicit-check-not="<<" --implicit-check-not=">>"
 #
 ## Verify --dump-tli reports the full set.
 # RUN: llvm-tli-checker --triple x86_64-scei-ps4 --dump-tli > %t3.txt
 # RUN: FileCheck %s --check-prefix=AVAIL --input-file %t3.txt
 # RUN: FileCheck %s --check-prefix=UNAVAIL --input-file %t3.txt
 #
 # CHECK: << Total TLI yes SDK no:  0
 # CHECK: >> Total TLI no  SDK yes: 0
 # CHECK: == Total TLI yes SDK yes: 235
 #
 # WRONG_DETAIL: << TLI yes SDK no : '_ZdaPv' aka operator delete[](void*)
 # WRONG_DETAIL: >> TLI no  SDK yes: '_ZdaPvj' aka operator delete[](void*, unsigned int)
 # WRONG_SUMMARY: << Total TLI yes SDK no:  1{{$}}
 # WRONG_SUMMARY: >> Total TLI no  SDK yes: 1{{$}}
 # WRONG_SUMMARY: == Total TLI yes SDK yes: 234
 #
 ## The -COUNT suffix doesn't care if there are too many matches, so check
 ## the exact count first; the two directives should add up to that.
+## Yes, this means additions to TLI will fail this test, but the argument
+## to -COUNT can't be an expression.
 # AVAIL: TLI knows 468 symbols, 235 available
 # AVAIL-COUNT-235: {{^}} available
-# UNAVAIL-COUNT-231: not available
+# UNAVAIL-COUNT-233: not available
 
+## This is a large file so it's worth telling lit to stop here.
+# END.
 
 --- !ELF
 FileHeader:
     Class:           ELFCLASS64
     Data:            ELFDATA2LSB
     OSABI:           ELFOSABI_FREEBSD
     Type:            ET_DYN
     Machine:         EM_X86_64