[llvm-ar][test] Add tests failing on Darwin

[gbreynoo]

Summary

The following reviews contained tests that failed on Darwin but passed on other machines. This is due to the default archive format differing on a Darwin machine, and what looks to be bugs in the output of this format. I can not investigate these issue further so the tests are considered expected failures on Darwin.

Old diffs:
D63197
D63935
D64330

From my limited investigation I believe I found 1 bug:
https://bugs.llvm.org/show_bug.cgi?id=42562

[MaskRay]

test/tools/llvm-ar/extract.test

1 ↗

(On Diff #210108)

Can you check if the file uses the \r\n ending? Some other tests you added did that. I just fixed them in r366324.

9 ↗

(On Diff #210108)

| count 0

Sorry I suggested this pattern before.. I recall a better one now.

[gbreynoo]

test/tools/llvm-ar/extract.test

1 ↗

(On Diff #210108)

Thanks, I'll have a look before I commit.

[JamesNagurne]

@gbreynoo

Our team maintains a downstream embedded ARM cross compiler, and these tests are failing on our Mac validations.
The key problem here I believe is the 'XFAIL: darwin' doesn't seem to work how you expect it to, and I'm not aware of a good alternative that I can suggest. Hopefully you or someone else has one.

XFAIL: darwin triggers if either the 'available_features' has darwin in it, or if the target triple contains 'darwin'.
Available features is not populated with the current host platform, and the target triple in our case is for our embedded ARM. Thus, the XFAIL is not active, and the failures are treated as such.

Do you know of any good solution for this? In summary:

1. Our tools (both clang and llvm-ar) are built on and run on Mac
2. The compiler (just clang) generates embedded ARM code
3. This test fails for llvm-ar built on and run on on mac
4. These lit tests seem to assume 2 implies 3, when in actuality, 1 implies 3

[gbreynoo]

Hi James,

From my understanding these has been resolving correctly as XFAIL's on upstream Darwin test machines and XFAIL: darwin is used in a few other lit tests. I do not know what cross compiling is tested on the build bots however.

Firstly, does use of XFAIL: system-darwin instead of XFAIL: darwin fix this issue?

Second can you run any of the llvm tests below on your machine and get the expected result? They all use XFAIL: darwin:

• test\DebugInfo\Generic\empty.ll
• test\DebugInfo\Generic\gmlt.test
• test\ExecutionEngine\MCJIT\test-global-ctors.ll
• test\ExecutionEngine\OrcMCJIT\test-global-ctors.ll

Finally I believe lit just uses the standard python commands to determine the platform, do the commands below give unexpected output?

import sys
import platform
print(platform.system())
print(sys.platform)

Hopefully this will make finding the cause of the issue easier. If anyone else has suggestions I would be happy to hear them.

Thanks

[JamesNagurne]

In D64802#1602758, @gbreynoo wrote:

Hi James,

From my understanding these has been resolving correctly as XFAIL's on upstream Darwin test machines and XFAIL: darwin is used in a few other lit tests. I do not know what cross compiling is tested on the build bots however.

Firstly, does use of XFAIL: system-darwin instead of XFAIL: darwin fix this issue?

Second can you run any of the llvm tests below on your machine and get the expected result? They all use XFAIL: darwin:

• test\DebugInfo\Generic\empty.ll
• test\DebugInfo\Generic\gmlt.test
• test\ExecutionEngine\MCJIT\test-global-ctors.ll
• test\ExecutionEngine\OrcMCJIT\test-global-ctors.ll

Finally I believe lit just uses the standard python commands to determine the platform, do the commands below give unexpected output?

import sys
import platform
print(platform.system())
print(sys.platform)

Hopefully this will make finding the cause of the issue easier. If anyone else has suggestions I would be happy to hear them.

Thanks

In D64802#1602758, @gbreynoo wrote:

Hi James,

From my understanding these has been resolving correctly as XFAIL's on upstream Darwin test machines and XFAIL: darwin is used in a few other lit tests. I do not know what cross compiling is tested on the build bots however.

Firstly, does use of XFAIL: system-darwin instead of XFAIL: darwin fix this issue?

Second can you run any of the llvm tests below on your machine and get the expected result? They all use XFAIL: darwin:

• test\DebugInfo\Generic\empty.ll
• test\DebugInfo\Generic\gmlt.test
• test\ExecutionEngine\MCJIT\test-global-ctors.ll
• test\ExecutionEngine\OrcMCJIT\test-global-ctors.ll

Finally I believe lit just uses the standard python commands to determine the platform, do the commands below give unexpected output?

import sys
import platform
print(platform.system())
print(sys.platform)

Hopefully this will make finding the cause of the issue easier. If anyone else has suggestions I would be happy to hear them.

Thanks

I had not considered system-darwin, and hadn't found it in my cursory look through the tests, though I now see where it gets added by the main LLVM lit config.py file.

XFAIL: system-darwin does fix the issue and marks the unexpected failure.

I think the main question here is if compiling llvm-ar on darwin causes the errors, or if compiling for darwin causes the errors.
I'm not sure of the functional differences between llvm-ar between x86_64-apple-darwin* and our arm-*-none-eabi. I'm running under the assumption that the archive format is agnostic of the target triple. If that's the case, then I believe the problem lies when llvm-ar is built and executed as a darwin-hosted tool, and the XFAIL should be system-darwin. Otherwise, it would be the case that our Arm implementation is exhibiting similar bugs to the darwin implementation, and our downstream copy should have "|| arm-*-none-eabi" in addition to "darwin".

Thanks for the response. I think we'll continue forward with the system-darwin approach unless more work is done on this front.

JB

llvm/trunk/test/tools/llvm-ar/extract.test

+## Test extract operation.
+# XFAIL: darwin
+
+RUN: rm -rf %t && mkdir -p %t/extracted/
+
+# Extracting from an empty archive should not warn or error:
+RUN: llvm-ar cr %t/empty.a
+RUN: llvm-ar x %t/empty.a 2>&1 | count 0
+
+RUN: echo filea > %t/a.txt
+RUN: echo fileb > %t/b.txt
+RUN: llvm-ar rc %t/archive.a %t/a.txt %t/b.txt
+
+# Single member:
+RUN: cd %t/extracted && llvm-ar x %t/archive.a a.txt
+RUN: diff %t/a.txt %t/extracted/a.txt 
+
+# All members:
+RUN: rm %t/extracted/a.txt
+RUN: cd %t/extracted && llvm-ar x %t/archive.a
+RUN: diff %t/a.txt %t/extracted/a.txt 
+RUN: diff %t/b.txt %t/extracted/b.txt 

llvm/trunk/test/tools/llvm-ar/mri-utf8.test

+# Test non-ascii archive members
+# XFAIL: darwin
+
+RUN: rm -rf %t && mkdir -p %t/extracted
+
+RUN: echo "contents" > %t/£.txt
+
+RUN: echo "CREATE %t/mri.ar" > %t/script.mri
+RUN: echo "ADDMOD %t/£.txt" >> %t/script.mri
+RUN: echo "SAVE" >> %t/script.mri
+
+RUN: llvm-ar -M < %t/script.mri
+RUN: cd %t/extracted && llvm-ar x %t/mri.ar
+
+# This works around problems launching processess that
+# include arguments with non-ascii characters.
+# Python on Linux defaults to ASCII encoding unless the
+# environment specifies otherwise, so it is explicitly set.
+RUN: env LANG=en_US.UTF-8 %python -c "assert open(u'\U000000A3.txt', 'rb').read() == b'contents\n'"

llvm/trunk/test/tools/llvm-ar/print.test

+## Test Print output
+# XFAIL: darwin
+
+RUN: rm -rf %t && mkdir -p %t
+RUN: echo file1 > %t/1.txt
+RUN: echo file2 > %t/2.txt
+RUN: echo file3 > %t/3.txt
+
+RUN: llvm-ar -rc %t/archive.a %t/1.txt %t/2.txt %t/3.txt
+
+# Print without member:
+RUN: llvm-ar p %t/archive.a \
+RUN:   | FileCheck %s --check-prefix=WITHOUT --match-full-lines --implicit-check-not {{.}}
+
+WITHOUT:      file1
+WITHOUT-NEXT: file2
+WITHOUT-NEXT: file3
+
+RUN: llvm-ar pv %t/archive.a \
+RUN:   | FileCheck %s --check-prefix=WITHOUT-VERBOSE --match-full-lines --implicit-check-not {{.}}
+
+WITHOUT-VERBOSE:      Printing 1.txt
+WITHOUT-VERBOSE-NEXT: file1
+WITHOUT-VERBOSE-NEXT: Printing 2.txt
+WITHOUT-VERBOSE-NEXT: file2
+WITHOUT-VERBOSE-NEXT: Printing 3.txt
+WITHOUT-VERBOSE-NEXT: file3
+
+# Print single member:
+RUN: llvm-ar p %t/archive.a %t/2.txt  \
+RUN:   | FileCheck %s --check-prefix=SINGLE --match-full-lines --implicit-check-not {{.}}
+
+SINGLE: file2
+
+RUN: llvm-ar pv %t/archive.a %t/2.txt  \
+RUN:   | FileCheck %s --check-prefix=SINGLE-VERBOSE --match-full-lines --implicit-check-not {{.}}
+
+SINGLE-VERBOSE:      Printing 2.txt
+SINGLE-VERBOSE-NEXT: file2
+
+# Print multiple members:
+RUN: llvm-ar p %t/archive.a %t/2.txt %t/1.txt \
+RUN:   | FileCheck %s --check-prefix=MULTIPLE --match-full-lines --implicit-check-not {{.}}
+
+MULTIPLE:      file1
+MULTIPLE-NEXT: file2
+
+RUN: llvm-ar pv %t/archive.a %t/2.txt %t/1.txt \
+RUN:   | FileCheck %s --check-prefix=MULTIPLE-VERBOSE --match-full-lines --implicit-check-not {{.}}
+
+MULTIPLE-VERBOSE:      Printing 1.txt
+MULTIPLE-VERBOSE-NEXT: file1
+MULTIPLE-VERBOSE-NEXT: Printing 2.txt
+MULTIPLE-VERBOSE-NEXT: file2
+
+# Print same member:
+RUN: not llvm-ar p %t/archive.a %t/2.txt %t/2.txt 2>&1 \
+RUN:   | FileCheck %s --check-prefix=SAME -DFILE=%t/2.txt
+
+SAME-DAG: file2
+SAME-DAG: error: '[[FILE]]' was not found
+
+# Print same member when containing multiple members with shared name:
+llvm-ar -q %t/archive.a %t/2.txt
+RUN: not llvm-ar p %t/archive.a %t/2.txt %t/2.txt 2>&1 \
+RUN:   | FileCheck %s --check-prefix=SAME -DFILE=%t/2.txt
+
+# No archive:
+RUN: not llvm-ar p 2>&1 \
+RUN:   | FileCheck %s --check-prefix=NO-ARCHIVE
+
+NO-ARCHIVE: error: An archive name must be specified.
+
+# Archive does not exist:
+RUN: not llvm-ar p %t/missing.a 2>&1 \
+RUN:   | FileCheck %s --check-prefix=MISSING-ARCHIVE -DARCHIVE=%t/missing.a
+
+MISSING-ARCHIVE: error: error loading '[[ARCHIVE]]': {{[nN]}}o such file or directory.
+
+# Member does not exist:
+RUN: not llvm-ar p %t/archive.a %t-missing.txt 2>&1 \
+RUN:   | FileCheck %s --check-prefix=MISSING-FILE -DFILE=%t-missing.txt
+
+MISSING-FILE: error: '[[FILE]]' was not found