[test] Use host platform specific error message substitution in lit tests

[abhina.sreeskantharajan]

Summary

On z/OS, the following error message is not matched correctly in lit tests.

EDC5129I No such file or directory.

This patch uses a lit config substitution to check for platform specific error messages.

[muiez]

LGTM

[jhenderson]

Sorry, could you revert this please. I don't think this is a good fix, as you've reduced coverage within the test, and some changes are completly redundant/add extra noise. I've commented inline with examples. Skimming D94239 suggests that has the same issue.

Could you also please explain the messages your system is actually producing so we can provide a better solution than this fix.

I'm also concerned that you are doing this to fix this test for a system, yet there are no build bots that report this error. A future contributor is likely to break them in the same way/add new tests with the same issue. If your system is a system that is supposed to be supported by LLVM, there needs to be a build bot. If it isn't supported, you should bring this up on llvm-dev (if you haven't already) to get buy-in for support.

clang/test/Driver/clang-offload-bundler.c

73–77

Here and everywhere you've added the {{.*}}, the test will no longer pick up changes in the message that occur between the end of the literal match and the start of the "no such file..." part. For example, this will no longer fail if the message reported became "error: 'afile': fsdufbsdufbno such file or directory"

Even were adding .* desirable, it should be part of the following regex, i.e. {{.*N|n}} to reduce the noise of the {{ and }}.

llvm/test/Object/archive-extract-dir.test

11

{{.*}} at the end of a FileCheck line has literally no benefit - it is implicit unless --match-full-lines is specified to FileCheck.

[grimar]

As far I understand, looking on the description of D94239, the message on z/OS looks like "EDC5129I No such file or directory.".
I guess the EDC5129I is a stable error code? So why not to check for a possible EDC5129I prefix word instead of .*?
(The same applies for other possible errors)

[abhina.sreeskantharajan]

In D95246#2518989, @jhenderson wrote:

Sorry, could you revert this please. I don't think this is a good fix, as you've reduced coverage within the test, and some changes are completly redundant/add extra noise. I've commented inline with examples. Skimming D94239 suggests that has the same issue.

Could you also please explain the messages your system is actually producing so we can provide a better solution than this fix.

I'm also concerned that you are doing this to fix this test for a system, yet there are no build bots that report this error. A future contributor is likely to break them in the same way/add new tests with the same issue. If your system is a system that is supposed to be supported by LLVM, there needs to be a build bot. If it isn't supported, you should bring this up on llvm-dev (if you haven't already) to get buy-in for support.

Thanks for the feedback. I've reverted my changes from these two patches. We have indicated that we wish to add support for the z/OS platform and we are working on getting a buildbot set up.

In D95246#2519086, @grimar wrote:

As far I understand, looking on the description of D94239, the message on z/OS looks like "EDC5129I No such file or directory.".
I guess the EDC5129I is a stable error code? So why not to check for a possible EDC5129I prefix word instead of .*?
(The same applies for other possible errors)

As grimar noted, this is indeed the correct error message. "EDC5129I No such file or directory." (Note the extra period at the end)
Based on your feedback, these are the better alternatives that were suggested:

'{{.*N|n}}o such file or directory'

{{EDC5129I N|N|n}}o such file or directory'

Some testcases fail because of the extra period at the end. For those testcases, this is a possible alternative.

{{.*N|n}}o such file or directory{{\.?}}

Please let me know if there are better alternatives I could look into.

[jhenderson]

In D95246#2519642, @abhina.sreeskantharajan wrote:

In D95246#2518989, @jhenderson wrote:

Sorry, could you revert this please. I don't think this is a good fix, as you've reduced coverage within the test, and some changes are completly redundant/add extra noise. I've commented inline with examples. Skimming D94239 suggests that has the same issue.

Could you also please explain the messages your system is actually producing so we can provide a better solution than this fix.

I'm also concerned that you are doing this to fix this test for a system, yet there are no build bots that report this error. A future contributor is likely to break them in the same way/add new tests with the same issue. If your system is a system that is supposed to be supported by LLVM, there needs to be a build bot. If it isn't supported, you should bring this up on llvm-dev (if you haven't already) to get buy-in for support.

Thanks for the feedback. I've reverted my changes from these two patches. We have indicated that we wish to add support for the z/OS platform but we have not set up a buildbot yet.

In D95246#2519086, @grimar wrote:

As far I understand, looking on the description of D94239, the message on z/OS looks like "EDC5129I No such file or directory.".
I guess the EDC5129I is a stable error code? So why not to check for a possible EDC5129I prefix word instead of .*?
(The same applies for other possible errors)

As grimar noted, this is indeed the correct error message. "EDC5129I No such file or directory." (Note the extra period at the end)
Based on your feedback, these are the better alternatives that were suggested:

Slightly off-the-wall idea: I'm assuming you don't control your system in such a way that you can change the error message to omit the error code?

'{{.*N|n}}o such file or directory'

{{EDC5129I N|N|n}}o such file or directory'

Some testcases fail because of the extra period at the end. For those testcases, this is a possible alternative.

{{.*N|n}}o such file or directory{{\.?}}

Please let me know if there are better alternatives I could look into.

I think you can just omit the trailing full stop in those cases. If the test isn't using --match-full-lines, it should be fine. If it is, adding {{.?}} seems reasonable.

Having the error code explicitly in the pattern looks like the right solution for now, but a thought on that - it seems like tests will still have the fragility problem for when someone else writes a new test that checks the message due to the error code not being present on most systems. Is the error code different for each system error message (I'm guessing it is)? I wonder if we would be better off adding some sort of lit substitution or similar that expands to the right string for the given host OS, which could in turn be fed to FileCheck. It might look a bit like this in practice:

# RUN: not do-a-thing %t 2>&1 | FileCheck %s -DMSG=%enoent -DFILE=%t

# CHECK: error: '[[FILE]]': [[MSG]]

What do you think?

[abhina.sreeskantharajan]

In D95246#2519729, @jhenderson wrote:

In D95246#2519642, @abhina.sreeskantharajan wrote:

In D95246#2518989, @jhenderson wrote:

Sorry, could you revert this please. I don't think this is a good fix, as you've reduced coverage within the test, and some changes are completly redundant/add extra noise. I've commented inline with examples. Skimming D94239 suggests that has the same issue.

Could you also please explain the messages your system is actually producing so we can provide a better solution than this fix.

I'm also concerned that you are doing this to fix this test for a system, yet there are no build bots that report this error. A future contributor is likely to break them in the same way/add new tests with the same issue. If your system is a system that is supposed to be supported by LLVM, there needs to be a build bot. If it isn't supported, you should bring this up on llvm-dev (if you haven't already) to get buy-in for support.

Thanks for the feedback. I've reverted my changes from these two patches. We have indicated that we wish to add support for the z/OS platform but we have not set up a buildbot yet.

In D95246#2519086, @grimar wrote:

As far I understand, looking on the description of D94239, the message on z/OS looks like "EDC5129I No such file or directory.".
I guess the EDC5129I is a stable error code? So why not to check for a possible EDC5129I prefix word instead of .*?
(The same applies for other possible errors)

As grimar noted, this is indeed the correct error message. "EDC5129I No such file or directory." (Note the extra period at the end)
Based on your feedback, these are the better alternatives that were suggested:

Slightly off-the-wall idea: I'm assuming you don't control your system in such a way that you can change the error message to omit the error code?

Right, I'm not able to change the error message.

'{{.*N|n}}o such file or directory'

{{EDC5129I N|N|n}}o such file or directory'

Some testcases fail because of the extra period at the end. For those testcases, this is a possible alternative.

{{.*N|n}}o such file or directory{{\.?}}

Please let me know if there are better alternatives I could look into.

I think you can just omit the trailing full stop in those cases. If the test isn't using --match-full-lines, it should be fine. If it is, adding {{.?}} seems reasonable.

Having the error code explicitly in the pattern looks like the right solution for now, but a thought on that - it seems like tests will still have the fragility problem for when someone else writes a new test that checks the message due to the error code not being present on most systems. Is the error code different for each system error message (I'm guessing it is)? I wonder if we would be better off adding some sort of lit substitution or similar that expands to the right string for the given host OS, which could in turn be fed to FileCheck. It might look a bit like this in practice:

# RUN: not do-a-thing %t 2>&1 | FileCheck %s -DMSG=%enoent -DFILE=%t

# CHECK: error: '[[FILE]]': [[MSG]]

What do you think?

I like the lit substitution solution, it will be a lot cleaner compared to a complicated regex. I've noticed there are already different regex for the same error message so this will help the error messages be uniform as well. I can look into implementing this.

[abhina.sreeskantharajan]

I've implemented the initial solution suggested by James. I haven't updated all the testcases yet because I wanted to get feedback on the implementation first.

[abhina.sreeskantharajan]

FIx CI, check if host_triple exists.

[grimar]

I like this approach.

[jhenderson]

I did briefly consider one alternative, which was to build these directly into FileCheck, but it doesn't quite feel like the right approach for FileCheck to need to know system level details. In practice, I think the lit substitution may be the best way forward overall. In the future, it should be easy to expand with more error messages.

You should add the new substitution to the lit command guide, so that it's documented. It might also be worth a heads up on llvm-dev to get feedback from those not directly on this review, to see if there are other ideas.

llvm/utils/lit/lit/llvm/config.py

349–354

These lines are quite long, so probably want reflowing.

I wonder if %errc_... might be a better name? That way, it ties to the std::errc values these match up with.

372–373

Under what conditions can there not be a host_triple? In those cases, what happens to the tests that use the new substitution?

375

Nit: too many blank lines (compared to what was there before)

[abhina.sreeskantharajan]

This patch makes the following changes:

• Define LLVM_HOST_TRIPLE for lld tests. (This was the project that didn't have host defined.)
• Change %err_no_such_file_or_directory to %errc_ENOENT as suggested by jhenderson

I've also created a post on llvm-dev about this patch: https://lists.llvm.org/pipermail/llvm-dev/2021-January/148089.html

[abhina.sreeskantharajan]

llvm/utils/lit/lit/llvm/config.py

349–354

Thanks, I've changed the error messages to your suggestion.

372–373

This was not defined for lld. I added changes to define this for lld and removed the check. I think this is defined in all the other projects.

[abhina.sreeskantharajan]

Fix CI: Other projects like flang also do not specify LLVM_HOST_TRIPLE. If this is not specified, set the triple to an empty string.

I also added one more error code %errc_EISDIR and updated the TestingGuide.
Please let me know if there are other guides I will need to update that I'm not aware of.

[jhenderson]

In D95246#2522913, @abhina.sreeskantharajan wrote:

Please let me know if there are other guides I will need to update that I'm not aware of.

There's also the lit CommandGuide located at llvm/docs/CommandGuide/lit.rst.

clang/test/Driver/clang-offload-bundler.c

73–78

Nit: These are probably good candidates to split up over multiple lines, as in the inline edit.

lld/CMakeLists.txt

115 ↗

(On Diff #319309)

The target and the host may be two completely different things. This variable setting doesn't look right to me. In a situation where someone is building LLD to run on Windows but targeting Linux, the host triple value will end up being set to a Linux triple, and the tests would fail.

The variable setting can also be moved inside the earlier if about 5 lines up, because that block is hit when LLD_BUILD_STANDALONE is set.

llvm/docs/TestingGuide.rst

545–547

This is a slightly different format to the other examples above. I think it should be like the inline edit. this also helps distinguish the difference between the two.

Note: I can't remember which way around the two versions are, so you might need to swap them!

llvm/utils/lit/lit/llvm/config.py

349–351

I'm concerned that someone might start using these substitutions in a project for the first time, and get confused why they don't work on non-windows platforms. Maybe the solution is simply to require LLVM_HOST_TRIPLE to be set in all projects, i.e. go back to what you were doing before, and letting python fail if it isn't set.

Happy to hear other ideas too.

[abhina.sreeskantharajan]

I've changed the host check to use python's sys.platform to determine the host as an alternative to using LLVM_HOST_TRIPLE. It will save us the effort of making sure it's defined in all projects as well.

I noticed that llvm/docs/CommandGuide/lit.rst only lists substitutions that are done in TestRunner.py and refers to llvm/docs/TestingGuide.rst for information on the remaining substitutions. I think if we move the error substitution code into TestRunner.py, then it would be appropriate to update the lit.rst. Where do you think is the appropriate place to define the error substitutions, TestRunner.py or config.py?

[abhina.sreeskantharajan]

llvm/utils/lit/lit/llvm/config.py

349–351

I think using sys.platform or platform.system() could be a better alternative. What do you think?

[abhina.sreeskantharajan]

Fix CI: Rebase

[jhenderson]

clang/test/Driver/clang-offload-bundler.c

74

Add one more space - it helps the indentation stand out more and therefore make it clearer this line is a continuation.

76

llvm/docs/TestingGuide.rst

545–547

We probably need to list what error codes are currently supported somewhere.

llvm/utils/lit/lit/llvm/config.py

349–351

Makes sense to me. Slight issue: cygwin on Windows uses cygwin. What error message does it produce though?

[abhina.sreeskantharajan]

llvm/utils/lit/lit/llvm/config.py

349–351

I tested this out. On a cygwin terminal on Windows it emits the same error message as Linux so it should take the else path.

[abhina.sreeskantharajan]

Address syntax comments, and add a list of supported error code substitutions in the guide.

If these changes are ok, I would like to start making changes to all affected testcases.

[jhenderson]

One nit, but otherwise looks good to me, thanks! Please go ahead with the other test updates. Do you plan on doing them in this patch?

llvm/docs/TestingGuide.rst

545

Adding "currently" implies to me that someone could add to the list easily.

[abhina.sreeskantharajan]

In D95246#2527961, @jhenderson wrote:

One nit, but otherwise looks good to me, thanks! Please go ahead with the other test updates. Do you plan on doing them in this patch?

This was my initial thought. But if it's preferred to create a second patch to make the remaining changes to affected testcases, I have no issue with doing that.

[abhina.sreeskantharajan]

Add currently in TestingGuide.

[jhenderson]

In D95246#2527999, @abhina.sreeskantharajan wrote:

In D95246#2527961, @jhenderson wrote:

One nit, but otherwise looks good to me, thanks! Please go ahead with the other test updates. Do you plan on doing them in this patch?

This was my initial thought. But if it's preferred to create a second patch to make the remaining changes to affected testcases, I have no issue with doing that.

I'm happy either way, whichever you prefer.

[abhina.sreeskantharajan]

Update all testcases to use -DMSG.
I've also updated the summary to match the changes in the patch.

[jhenderson]

Great work! LGTM.

[ASDenysPetrov]

This patch causes fails in a bunch of test files. When I roll it back, it passes.
Below you can see fail output of ubsan-blacklist-vfs.c and basic-block-sections.c.

fail_output.txt17 KBDownload

Please, pay attention to this.
I'm using build on Win10 + GCC config.

P.S. The similar trouble is in D95808. I've commented there as well.

[ASDenysPetrov]

One more mention. There are a lot of popups like this which stops the process untill press OK.

That's very annoying. Please revert or fix this patch.

[abhina.sreeskantharajan]

In D95246#2564137, @ASDenysPetrov wrote:

This patch causes fails in a bunch of test files. When I roll it back, it passes.
Below you can see fail output of ubsan-blacklist-vfs.c and basic-block-sections.c.

fail_output.txt17 KBDownload

Please, pay attention to this.
I'm using build on Win10 + GCC config.

P.S. The similar trouble is in D95808. I've commented there as well.

Hi Denys, thanks for bringing this to my attention, I'll look into this failure. I do know that the CI test for this patch and the other patch ran on Windows and did not report any errors. https://buildkite.com/llvm-project/premerge-checks/builds/23727#63570ad0-0e74-4825-b1dc-8f53a4d4dad0
Do you know what is different between your environments which is causing the spelling to be capitalized? This might help me determine how to fix the current host platform check in llvm/utils/lit/lit/llvm/config.py.

[jhenderson]

In D95246#2565351, @abhina.sreeskantharajan wrote:

Do you know what is different between your environments which is causing the spelling to be capitalized? This might help me determine how to fix the current host platform check in llvm/utils/lit/lit/llvm/config.py.

Could it be something to do with a different standard library being used?

[ASDenysPetrov]

In D95246#2565351, @abhina.sreeskantharajan wrote:

Do you know what is different between your environments which is causing the spelling to be capitalized? This might help me determine how to fix the current host platform check in llvm/utils/lit/lit/llvm/config.py.

I'd love to help you. I'm using ordinary Win10. Here is my python output:

>>> os.name
'nt'
>>> sys.platform
'win32'
>>> platform.system()
'Windows'
>>> platform.release()
'10'
>>>

If you would say what exact info you need I would execute and send you.
BTW my python3 uses Sentence case capitalization in errors as you can see:

>>> open('some')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'some'
>>>

[abhina.sreeskantharajan]

In D95246#2566029, @ASDenysPetrov wrote:

In D95246#2565351, @abhina.sreeskantharajan wrote:

Do you know what is different between your environments which is causing the spelling to be capitalized? This might help me determine how to fix the current host platform check in llvm/utils/lit/lit/llvm/config.py.

I'd love to help you. I'm using ordinary Win10. Here is my python output:

>>> os.name
'nt'
>>> sys.platform
'win32'
>>> platform.system()
'Windows'
>>> platform.release()
'10'
>>>

If you would say what exact info you need I would execute and send you.
BTW my python3 uses Sentence case capitalization in errors as you can see:

>>> open('some')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'some'
>>>

Thanks! The following Windows bot (clang-x64-windows-msvc) is also passing http://lab.llvm.org:8011/#/builders/123, I expect it would have the same output in python as your machine. James mentioned that this issue might be caused by a difference in standard library.

I was curious if all of these modified testcases are now failing or only the three you mentioned. I also don't know how this change can cause a pop-up to occur, is there more information you can provide?
And lastly, if you have the time, does this testcase pass for you?

llvm/test/tools/llvm-elfabi/fail-file-write-windows.test

I did not modify this but it requires system-windows and has a lower-case error message. I expect if my change causes failures for you, then this testcase should also fail.

[ASDenysPetrov]

In D95246#2566417, @abhina.sreeskantharajan wrote:

I was curious if all of these modified testcases are now failing or only the three you mentioned.

Here is the list of the tests which fail after the patch (I haven't build LLD so don't have tests as well):

Clang :: CodeGen/basic-block-sections.c
Clang :: CodeGen/ubsan-blacklist-vfs.c
Clang :: Frontend/output-paths.c
Clang :: Frontend/stats-file.c  
LLVM :: DebugInfo/symbolize-missing-file.test
LLVM :: Object/archive-extract-dir.test
LLVM :: Object/directory.ll
LLVM :: tools/llvm-ar/error-opening-directory.test
LLVM :: tools/llvm-ar/missing-thin-archive-member.test
LLVM :: tools/llvm-ar/move.test
LLVM :: tools/llvm-ar/print.test
LLVM :: tools/llvm-ar/quick-append.test
LLVM :: tools/llvm-ar/replace.test
LLVM :: tools/llvm-ar/response.test
LLVM :: tools/llvm-cxxdump/trivial.test
LLVM :: tools/llvm-libtool-darwin/filelist.test
LLVM :: tools/llvm-libtool-darwin/invalid-input-output-args.test
LLVM :: tools/llvm-lipo/create-arch.test
LLVM :: tools/llvm-lipo/replace-invalid-input.test
LLVM :: tools/llvm-lto/error.ll
LLVM :: tools/llvm-lto2/X86/stats-file-option.ll
LLVM :: tools/llvm-mc/basic.test
LLVM :: tools/llvm-mca/invalid_input_file_name.test
LLVM :: tools/llvm-ml/basic.test
LLVM :: tools/llvm-objcopy/COFF/add-section.test
LLVM :: tools/llvm-objcopy/ELF/add-section.test
LLVM :: tools/llvm-objcopy/ELF/error-format.test
LLVM :: tools/llvm-objcopy/MachO/add-section-error.test
LLVM :: tools/llvm-objcopy/redefine-symbols.test
LLVM :: tools/llvm-objcopy/wasm/dump-section.test
LLVM :: tools/llvm-profdata/weight-instr.test
LLVM :: tools/llvm-profdata/weight-sample.test
LLVM :: tools/llvm-readobj/ELF/thin-archive-paths.test
LLVM :: tools/llvm-readobj/basic.test
LLVM :: tools/llvm-readobj/thin-archive.test
LLVM :: tools/llvm-size/no-input.test
LLVM :: tools/llvm-xray/X86/no-such-file.txt
LLVM :: tools/obj2yaml/invalid_input_file.test
LLVM :: tools/yaml2obj/output-file.yaml

I also don't know how this change can cause a pop-up to occur, is there more information you can provide?

I got this. The popups is not the fault of this patch. They happened earlier. I'll investigate it later.

llvm/test/tools/llvm-elfabi/fail-file-write-windows.test

I did not modify this but it requires system-windows and has a lower-case error message. I expect if my change causes failures for you, then this testcase should also fail.

Yes, this fails for me too. Didn't check it earlier.

[jhenderson]

@ASDenysPetrov, could you provide your link command-line for one of the tools that produces the failing message, please, e.g. llvm-ar? That may give us a hint about whether different libraries are being used. It's possible we need some additional cmake-time configuration steps to detect this difference. Also are you using cygwin at all?

Not that I'm saying we shouldn't try to fix your use-case, but if you aren't using Cygwin, it's worth noting that https://llvm.org/docs/GettingStarted.html#hardware states that only Visual Studio builds (whether via the IDE or some other tool like ninja) are known to work for Windows.

[ASDenysPetrov]

@jhenderson

@ASDenysPetrov, could you provide your link command-line for one of the tools that produces the failing message, please, e.g. llvm-ar?

Here is output of running tests in llvm-ar folder:

fail_output_llvm_ar.txt93 KBDownload

Not that I'm saying we shouldn't try to fix your use-case, but if you aren't using Cygwin, it's worth noting that https://llvm.org/docs/GettingStarted.html#hardware states that only Visual Studio builds (whether via the IDE or some other tool like ninja) are known to work for Windows.

I'm using MSYS2 instead of Cygwin. I found it as a better alternative. I'm using GCC+MSYS2+Ninja.

[jhenderson]

In D95246#2568084, @ASDenysPetrov wrote:

@jhenderson

@ASDenysPetrov, could you provide your link command-line for one of the tools that produces the failing message, please, e.g. llvm-ar?

Here is output of running tests in llvm-ar folder:

fail_output_llvm_ar.txt93 KBDownload

Sorry, what I meant was, what is the command used to build the llvm-ar executable, not what does it print when used. The command used to build llvm-ar should tell us what libraries are being used, which could impact this behaviour.

[ASDenysPetrov]

@jhenderson
I'm using
cmake -GNinja ../llvm -DLLVM_LIT_ARGS=-sv -DLLVM_ENABLE_PROJECTS=clang -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
for setup and ninja for build. Here is my CMakeCache.txt config file.

CMakeCache.txt76 KBDownload

[abhina.sreeskantharajan]

In D95246#2568084, @ASDenysPetrov wrote:

@jhenderson

@ASDenysPetrov, could you provide your link command-line for one of the tools that produces the failing message, please, e.g. llvm-ar?

Here is output of running tests in llvm-ar folder:

fail_output_llvm_ar.txt93 KBDownload

Not that I'm saying we shouldn't try to fix your use-case, but if you aren't using Cygwin, it's worth noting that https://llvm.org/docs/GettingStarted.html#hardware states that only Visual Studio builds (whether via the IDE or some other tool like ninja) are known to work for Windows.

I'm using MSYS2 instead of Cygwin. I found it as a better alternative. I'm using GCC+MSYS2+Ninja.

MSYS2 is based on a modified version of Cygwin according to its description, maybe if we can detect that in config.py this will solve your issue.
Can you share the output of

sysconfig.get_platform()

[ASDenysPetrov]

@abhina.sreeskantharajan

Can you share the output of

sysconfig.get_platform()

Sure,

>>> sysconfig.get_platform()
'win32'
>>>

[abhina.sreeskantharajan]

@ASDenysPetrov are you running python inside MSYS bash? that is installed by

pacman -S --needed base-devel mingw-w64-x86_64-toolchain

I learned that the python should give different output

>>> sys.platform
msys
>>> platform.system()
MSYS_NT-10.0-19041
>>> sysconfig.get_platform()
msys-3.1.7-x86_64

After taking a look at the CMakeCache.txt, I think a potential workaround would be to check if config.host_cxx contains MSYS2 which is set to CMAKE_CXX_COMPILER:FILEPATH=D:/WORK/Utilities/MSYS2/mingw64/bin/c++.exe if we can't determine MSYS through python.

[ASDenysPetrov]

@abhina.sreeskantharajan wrote:

@ASDenysPetrov are you running python inside MSYS bash? that is installed by

No, I installed Python as usual via Windows msi got from official site.

Here is what I've got from Python:

>>> sys.platform
'win32'
>>> platform.system()
'Windows'
>>> sysconfig.get_platform()
'win32'
>>>

[jhenderson]

@ASDenysPetrov, if you run "ninja -v llvm-ar", what is the output of the line that actually builds the llvm-ar executable? That'll tell us which libraries are in use, which should hopefully point to the difference to our own environments. For example, the last line of output for this on my Linux machine is:

[811/811] : && /usr/bin/c++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -Wl,-rpath-link,/home/binutils/llvm/llvm-project/build/./lib  -Wl,-O3 -Wl,--gc-sections tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.o -o bin/llvm-ar  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAMDGPUAsmParser.a  lib/libLLVMARMAsmParser.a  lib/libLLVMAVRAsmParser.a  lib/libLLVMBPFAsmParser.a  lib/libLLVMHexagonAsmParser.a  lib/libLLVMLanaiAsmParser.a  lib/libLLVMMipsAsmParser.a  lib/libLLVMMSP430AsmParser.a  lib/libLLVMPowerPCAsmParser.a  lib/libLLVMRISCVAsmParser.a  lib/libLLVMSparcAsmParser.a  lib/libLLVMSystemZAsmParser.a  lib/libLLVMWebAssemblyAsmParser.a  lib/libLLVMX86AsmParser.a  lib/libLLVMAArch64Desc.a  lib/libLLVMAMDGPUDesc.a  lib/libLLVMARMDesc.a  lib/libLLVMAVRDesc.a  lib/libLLVMBPFDesc.a  lib/libLLVMHexagonDesc.a  lib/libLLVMLanaiDesc.a  lib/libLLVMMipsDesc.a  lib/libLLVMMSP430Desc.a  lib/libLLVMNVPTXDesc.a  lib/libLLVMPowerPCDesc.a  lib/libLLVMRISCVDesc.a  lib/libLLVMSparcDesc.a  lib/libLLVMSystemZDesc.a  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMX86Desc.a  lib/libLLVMXCoreDesc.a  lib/libLLVMAArch64Info.a  lib/libLLVMAMDGPUInfo.a  lib/libLLVMARMInfo.a  lib/libLLVMAVRInfo.a  lib/libLLVMBPFInfo.a  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiInfo.a  lib/libLLVMMipsInfo.a  lib/libLLVMMSP430Info.a  lib/libLLVMNVPTXInfo.a  lib/libLLVMPowerPCInfo.a  lib/libLLVMRISCVInfo.a  lib/libLLVMSparcInfo.a  lib/libLLVMSystemZInfo.a  lib/libLLVMWebAssemblyInfo.a  lib/libLLVMX86Info.a  lib/libLLVMXCoreInfo.a  lib/libLLVMBinaryFormat.a  lib/libLLVMCore.a  lib/libLLVMDlltoolDriver.a  lib/libLLVMLibDriver.a  lib/libLLVMObject.a  lib/libLLVMSupport.a  -lpthread  lib/libLLVMAArch64Utils.a  lib/libLLVMAMDGPUUtils.a  lib/libLLVMARMUtils.a  lib/libLLVMMCDisassembler.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMTextAPI.a  lib/libLLVMOption.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMBinaryFormat.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lrt  -ldl  -lpthread  -lm  /usr/lib/x86_64-linux-gnu/libz.so  /usr/lib/x86_64-linux-gnu/libtinfo.so  lib/libLLVMDemangle.a && :

This shows, for example, that the libz.so my build uses is located in /usr/lib/x86_64-linux-gnu. By having your equivalent path, we can hopefully confirm that the issue is caused by a different set of system libraries being used.

[ASDenysPetrov]

This shows, for example, that the libz.so my build uses is located in /usr/lib/x86_64-linux-gnu. By having your equivalent path, we can hopefully confirm that the issue is caused by a different set of system libraries being used.

Hi, @jhenderson
I changed llvm-ar.cpp then ran ninja -v llvm-ar and got this:

[1/2] D:\WORK\Utilities\MSYS2\mingw64\bin\c++.exe -DGTEST_HAS_RTTI=0 -D_DEBUG -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/llvm-ar -ID:/WORK/LLVM/llvm-project/llvm/tools/llvm-ar -Iinclude -ID:/WORK/LLVM/llvm-project/llvm/include -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment  -O2   -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -MF tools\llvm-ar\CMakeFiles\llvm-ar.dir\llvm-ar.cpp.obj.d -o tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -c D:/WORK/LLVM/llvm-project/llvm/tools/llvm-ar/llvm-ar.cpp
[2/2] cmd.exe /C "cd . && D:\WORK\Utilities\MSYS2\mingw64\bin\c++.exe -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment  -O2 -Wl,--stack,16777216 tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -o bin\llvm-ar.exe -Wl,--out-implib,lib\libllvm-ar.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a  lib/libLLVMX86Info.a  lib/libLLVMBinaryFormat.a  lib/libLLVMCore.a  lib/libLLVMDlltoolDriver.a  lib/libLLVMLibDriver.a  lib/libLLVMObject.a  lib/libLLVMSupport.a  lib/libLLVMMCDisassembler.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMTextAPI.a  lib/libLLVMOption.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMBinaryFormat.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lpsapi  -lshell32  -lole32  -luuid  -ladvapi32  D:/WORK/Utilities/MSYS2/mingw64/lib/libz.dll.a  lib/libLLVMDemangle.a  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."

As you can libz is here D:/WORK/Utilities/MSYS2/mingw64/lib/libz.dll.a

[jhenderson]

In D95246#2587123, @ASDenysPetrov wrote:

This shows, for example, that the libz.so my build uses is located in /usr/lib/x86_64-linux-gnu. By having your equivalent path, we can hopefully confirm that the issue is caused by a different set of system libraries being used.

Hi, @jhenderson
I changed llvm-ar.cpp then ran ninja -v llvm-ar and got this:

[1/2] D:\WORK\Utilities\MSYS2\mingw64\bin\c++.exe -DGTEST_HAS_RTTI=0 -D_DEBUG -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/llvm-ar -ID:/WORK/LLVM/llvm-project/llvm/tools/llvm-ar -Iinclude -ID:/WORK/LLVM/llvm-project/llvm/include -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment  -O2   -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -MF tools\llvm-ar\CMakeFiles\llvm-ar.dir\llvm-ar.cpp.obj.d -o tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -c D:/WORK/LLVM/llvm-project/llvm/tools/llvm-ar/llvm-ar.cpp
[2/2] cmd.exe /C "cd . && D:\WORK\Utilities\MSYS2\mingw64\bin\c++.exe -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment  -O2 -Wl,--stack,16777216 tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -o bin\llvm-ar.exe -Wl,--out-implib,lib\libllvm-ar.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a  lib/libLLVMX86Info.a  lib/libLLVMBinaryFormat.a  lib/libLLVMCore.a  lib/libLLVMDlltoolDriver.a  lib/libLLVMLibDriver.a  lib/libLLVMObject.a  lib/libLLVMSupport.a  lib/libLLVMMCDisassembler.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMTextAPI.a  lib/libLLVMOption.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMBinaryFormat.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lpsapi  -lshell32  -lole32  -luuid  -ladvapi32  D:/WORK/Utilities/MSYS2/mingw64/lib/libz.dll.a  lib/libLLVMDemangle.a  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."

As you can libz is here D:/WORK/Utilities/MSYS2/mingw64/lib/libz.dll.a

Thanks (for clarity, libz isn't what I'm looking for, it was just an example of how this information might be useful). As it is, I think we might need more information still - that commandline hides which files are actually used slightly. Could you try explicitly running the c++ command that is being requested, but with the -Wl,--trace option added too. That will tell the linker to print what files it opens when it runs the link, and should give us an hint as to which system library is different for your usage than ours.

[ASDenysPetrov]

@jhenderson

Thanks (for clarity, libz isn't what I'm looking for, it was just an example of how this information might be useful). As it is, I think we might need more information still - that commandline hides which files are actually used slightly. Could you try explicitly running the c++ command that is being requested, but with the -Wl,--trace option added too. That will tell the linker to print what files it opens when it runs the link, and should give us an hint as to which system library is different for your usage than ours.

I added -W --trace to the end of cmdline. The output is in the file:

trace_output.txt56 KBDownload

[jhenderson]

In D95246#2587167, @ASDenysPetrov wrote:

@jhenderson

Thanks (for clarity, libz isn't what I'm looking for, it was just an example of how this information might be useful). As it is, I think we might need more information still - that commandline hides which files are actually used slightly. Could you try explicitly running the c++ command that is being requested, but with the -Wl,--trace option added too. That will tell the linker to print what files it opens when it runs the link, and should give us an hint as to which system library is different for your usage than ours.

I added -W --trace to the end of cmdline. The output is in the file:

trace_output.txt56 KBDownload

Sorry, but -W --trace is not the same as -Wl,--trace. The former is a pair of options used by the compiler (one of which describes the files used by the compiler). The latter is an option passed to the linker, and is what we need. Please try again!

[ASDenysPetrov]

@jhenderson

Sorry, but -W --trace is not the same as -Wl,--trace. The former is a pair of options used by the compiler (one of which describes the files used by the compiler). The latter is an option passed to the linker, and is what we need. Please try again!

I used -W, because my c++ doesn't know -Wl option.

C:\Users\Denis>c++ -Wl
c++: error: unrecognized command-line option '-Wl'; did you mean '-W'?
c++: fatal error: no input files
compilation terminated.

Could you provide exact cmd line me to use if I'm doing smth wrong?

[jhenderson]

In D95246#2587277, @ASDenysPetrov wrote:

@jhenderson

Sorry, but -W --trace is not the same as -Wl,--trace. The former is a pair of options used by the compiler (one of which describes the files used by the compiler). The latter is an option passed to the linker, and is what we need. Please try again!

I used -W, because my c++ doesn't know -Wl option.

C:\Users\Denis>c++ -Wl
c++: error: unrecognized command-line option '-Wl'; did you mean '-W'?
c++: fatal error: no input files
compilation terminated.

Could you provide exact cmd line me to use if I'm doing smth wrong?

-Wl on its own isn't an option. It is used as a prefix to pass an option to the linker, so the exact string you pass to the compiler is -Wl,--trace (no spaces, comma required). The compiler sees the -Wl, prefix, removes it and then passes the remainder straight to the linker. You can see examples of this in your existing commandline with things like -Wl,--stack,16777216.

The cmdline should be:

D:\WORK\Utilities\MSYS2\mingw64\bin\c++.exe -DGTEST_HAS_RTTI=0 -D_DEBUG -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/llvm-ar -ID:/WORK/LLVM/llvm-project/llvm/tools/llvm-ar -Iinclude -ID:/WORK/LLVM/llvm-project/llvm/include -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment  -O2   -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -MF tools\llvm-ar\CMakeFiles\llvm-ar.dir\llvm-ar.cpp.obj.d -o tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -c D:/WORK/LLVM/llvm-project/llvm/tools/llvm-ar/llvm-ar.cpp
[2/2] cmd.exe /C "cd . && D:\WORK\Utilities\MSYS2\mingw64\bin\c++.exe -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment  -O2 -Wl,--stack,16777216 tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.obj -o bin\llvm-ar.exe -Wl,--out-implib,lib\libllvm-ar.dll.a -Wl,--major-image-version,0,--minor-image-version,0  lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a  lib/libLLVMX86Info.a  lib/libLLVMBinaryFormat.a  lib/libLLVMCore.a  lib/libLLVMDlltoolDriver.a  lib/libLLVMLibDriver.a  lib/libLLVMObject.a  lib/libLLVMSupport.a  lib/libLLVMMCDisassembler.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMTextAPI.a  lib/libLLVMOption.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMBinaryFormat.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lpsapi  -lshell32  -lole32  -luuid  -ladvapi32  D:/WORK/Utilities/MSYS2/mingw64/lib/libz.dll.a  lib/libLLVMDemangle.a  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--trace

[ASDenysPetrov]

@jhenderson
I think I'm done.
Here is output:

linker_trace_output.txt7 KBDownload

Is it OK now?

[jhenderson]

In D95246#2587410, @ASDenysPetrov wrote:

@jhenderson
I think I'm done.
Here is output:

linker_trace_output.txt7 KBDownload

Is it OK now?

Thanks, yes that does the trick. As I expected, you can see that all the libraries that are used are contained withing the msys2/mingw subdirectory, and therefore are not the standard Windows SDK files that you would typically use for a Windows build using Visual Studio. One of those will contain the definitions we are talking about.

@abhina.sreeskantharajan - I think you could this issue in one of two ways. One would be to detect when the tools/libraries used are within the msys2/mingw installation. Another way might be to actually build and run a miniature program at CMake time that prints out the messages, in such a way that CMake can inspect the output and use it to populate the lit configuration. Note: I am not a CMake expert, so don't know if this is strictly possible. The advantage of this approach is that you can remove the reliance on the system.platform call in python, and just always use the output of this simple program.

[abhina.sreeskantharajan]

In D95246#2587452, @jhenderson wrote:

In D95246#2587410, @ASDenysPetrov wrote:

@jhenderson
I think I'm done.
Here is output:

linker_trace_output.txt7 KBDownload

Is it OK now?

Thanks, yes that does the trick. As I expected, you can see that all the libraries that are used are contained withing the msys2/mingw subdirectory, and therefore are not the standard Windows SDK files that you would typically use for a Windows build using Visual Studio. One of those will contain the definitions we are talking about.

@abhina.sreeskantharajan - I think you could this issue in one of two ways. One would be to detect when the tools/libraries used are within the msys2/mingw installation. Another way might be to actually build and run a miniature program at CMake time that prints out the messages, in such a way that CMake can inspect the output and use it to populate the lit configuration. Note: I am not a CMake expert, so don't know if this is strictly possible. The advantage of this approach is that you can remove the reliance on the system.platform call in python, and just always use the output of this simple program.

Thanks, I'm also not sure if the second way is possible. I had another idea but I will need @ASDenysPetrov to test it out.
Does the following python program print out the correct error string for you?

import errno
import os
print ( os.strerror(errno.ENOENT))

If so, we can use this as the error message instead of hardcoding it per platform.

[abhina.sreeskantharajan]

I created the following review https://reviews.llvm.org/D97472 which I think should fix the error on all platforms. Let me know if this fixes the issue on MSYS2.

[abhina.sreeskantharajan]

This issue has been fixed in https://reviews.llvm.org/D97472.
@ASDenysPetrov I'm unable to close this revision and the other one https://reviews.llvm.org/D95808. Do you know what the appropriate action is here? To abandon this, or accept it again and close it?

[ASDenysPetrov]

In D95246#2606343, @abhina.sreeskantharajan wrote:

This issue has been fixed in https://reviews.llvm.org/D97472.
@ASDenysPetrov I'm unable to close this revision and the other one https://reviews.llvm.org/D95808. Do you know what the appropriate action is here? To abandon this, or accept it again and close it?

Done. Sorry for the delay.

clang/test/CodeGen/basic-block-sections.c

 // REQUIRES: x86-registered-target
 
 // RUN: %clang_cc1 -triple x86_64 -S -o - < %s | FileCheck %s --check-prefix=PLAIN
 // RUN: %clang_cc1 -triple x86_64 -S -fbasic-block-sections=all -fbasic-block-sections=none -o - < %s | FileCheck %s --check-prefix=PLAIN
 
 // RUN: %clang_cc1 -triple x86_64 -S -fbasic-block-sections=all -o - < %s | FileCheck %s --check-prefix=BB_WORLD --check-prefix=BB_ALL
 // RUN: %clang_cc1 -triple x86_64 -S -fbasic-block-sections=list=%S/Inputs/basic-block-sections.funcnames -o - < %s | FileCheck %s --check-prefix=BB_WORLD --check-prefix=BB_LIST
 // RUN: %clang_cc1 -triple x86_64 -S -fbasic-block-sections=all -funique-basic-block-section-names -o - < %s | FileCheck %s --check-prefix=UNIQUE
 // RUN: rm -f %t
-// RUN: not %clang_cc1 -fbasic-block-sections=list= -emit-obj -o %t %s 2>&1 | FileCheck %s --check-prefix=ERROR
+// RUN: not %clang_cc1 -fbasic-block-sections=list= -emit-obj -o %t %s 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=ERROR
 // RUN: not ls %t
 
 int world(int a) {
   if (a > 10)
     return 10;
   else if (a > 5)
     return 5;
   else

 // BB_ALL: .section .text.another,"ax",@progbits,unique
 // BB_ALL: another.__part.1:
 // BB_LIST-NOT: .section .text.another,"ax",@progbits,unique
 // BB_LIST: another:
 // BB_LIST-NOT: another.__part.1:
 //
 // UNIQUE: .section .text.world.world.__part.1,
 // UNIQUE: .section .text.another.another.__part.1,
-// ERROR: error:  unable to load basic block sections function list: '{{[Nn]}}o such file or directory'
+// ERROR: error:  unable to load basic block sections function list: '[[MSG]]'

clang/test/CodeGen/ubsan-blacklist-vfs.c

 // Verify ubsan doesn't emit checks for blacklisted functions and files
 // RUN: echo "fun:hash" > %t-func.blacklist
 // RUN: echo "src:%s" | sed -e 's/\\/\\\\/g' > %t-file.blacklist
 
 // RUN: rm -f %t-vfsoverlay.yaml
 // RUN: rm -f %t-nonexistent.blacklist
 // RUN: sed -e "s|@DIR@|%/T|g" %S/Inputs/sanitizer-blacklist-vfsoverlay.yaml | sed -e "s|@REAL_FILE@|%/t-func.blacklist|g" | sed -e "s|@NONEXISTENT_FILE@|%/t-nonexistent.blacklist|g" > %t-vfsoverlay.yaml
 // RUN: %clang_cc1 -fsanitize=unsigned-integer-overflow -ivfsoverlay %t-vfsoverlay.yaml -fsanitize-blacklist=%/T/only-virtual-file.blacklist -emit-llvm %s -o - | FileCheck %s --check-prefix=FUNC
 
-// RUN: not %clang_cc1 -fsanitize=unsigned-integer-overflow -ivfsoverlay %t-vfsoverlay.yaml -fsanitize-blacklist=%/T/invalid-virtual-file.blacklist -emit-llvm %s -o - 2>&1 | FileCheck %s --check-prefix=INVALID-MAPPED-FILE
-// INVALID-MAPPED-FILE: invalid-virtual-file.blacklist': {{[Nn]}}o such file or directory
+// RUN: not %clang_cc1 -fsanitize=unsigned-integer-overflow -ivfsoverlay %t-vfsoverlay.yaml -fsanitize-blacklist=%/T/invalid-virtual-file.blacklist -emit-llvm %s -o - 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=INVALID-MAPPED-FILE
+// INVALID-MAPPED-FILE: invalid-virtual-file.blacklist': [[MSG]]
 
-// RUN: not %clang_cc1 -fsanitize=unsigned-integer-overflow -ivfsoverlay %t-vfsoverlay.yaml -fsanitize-blacklist=%t-nonexistent.blacklist -emit-llvm %s -o - 2>&1 | FileCheck %s --check-prefix=INVALID
-// INVALID: nonexistent.blacklist': {{[Nn]}}o such file or directory
+// RUN: not %clang_cc1 -fsanitize=unsigned-integer-overflow -ivfsoverlay %t-vfsoverlay.yaml -fsanitize-blacklist=%t-nonexistent.blacklist -emit-llvm %s -o - 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=INVALID
+// INVALID: nonexistent.blacklist': [[MSG]]
 
 unsigned i;
 
 // DEFAULT: @hash
 // FUNC: @hash
 // FILE: @hash
 unsigned hash() {
 // DEFAULT: call {{.*}}void @__ubsan

clang/test/Driver/clang-offload-bundler.c

 
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.i,%t.tgt1,%t.tgt2 -inputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR3
 // CK-ERR3: error: only one output file supported in bundling mode
 
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu -outputs=%t.i,%t.tgt1,%t.tgt2 -inputs=%t.bundle.i -unbundle 2>&1 | FileCheck %s --check-prefix CK-ERR4
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.i,%t.tgt1 -inputs=%t.bundle.i -unbundle 2>&1 | FileCheck %s --check-prefix CK-ERR4
 // CK-ERR4: error: number of output files and targets should match in unbundling mode
 
-// RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2.notexist -outputs=%t.bundle.i 2>&1 | FileCheck %s -DFILE=%t.tgt2.notexist --check-prefix CK-ERR5
-// RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.i,%t.tgt1,%t.tgt2 -inputs=%t.bundle.i.notexist -unbundle 2>&1 | FileCheck %s -DFILE=%t.bundle.i.notexist --check-prefix CK-ERR5
-// CK-ERR5: error: '[[FILE]]': {{N|n}}o such file or directory
+// RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2.notexist -outputs=%t.bundle.i 2>&1 | \
+// RUN:   FileCheck %s -DFILE=%t.tgt2.notexist -DMSG=%errc_ENOENT --check-prefix CK-ERR5
+// RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.i,%t.tgt1,%t.tgt2 -inputs=%t.bundle.i.notexist -unbundle 2>&1 | \
+// RUN:   FileCheck %s -DFILE=%t.bundle.i.notexist -DMSG=%errc_ENOENT --check-prefix CK-ERR5
+// CK-ERR5: error: '[[FILE]]': [[MSG]]
 
 // RUN: not clang-offload-bundler -type=invalid -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s -DTYPE=invalid --check-prefix CK-ERR6
 // CK-ERR6: error: '[[TYPE]]': invalid file type specified
 
 // RUN: not clang-offload-bundler 2>&1 | FileCheck %s --check-prefix CK-ERR7
 // CK-ERR7: clang-offload-bundler: for the --type option: must be specified at least once!
 
 // RUN: not clang-offload-bundler -type=i -inputs=%t.i,%t.tgt1,%t.tgt2 2>&1 | FileCheck %s -check-prefix=CK-ERR7A
 // CK-ERR7A: error: for the --outputs option: must be specified at least once!

clang/test/Frontend/output-paths.c

 // RUN: not %clang_cc1 -emit-llvm -o %t.doesnotexist/somename %s 2> %t
-// RUN: FileCheck -check-prefix=OUTPUTFAIL -input-file=%t %s
+// RUN: FileCheck -check-prefix=OUTPUTFAIL -DMSG=%errc_ENOENT -input-file=%t %s
 
-// OUTPUTFAIL: error: unable to open output file '{{.*}}doesnotexist{{.}}somename': '{{[nN]}}o such file or directory'
+// OUTPUTFAIL: error: unable to open output file '{{.*}}doesnotexist{{.}}somename': '[[MSG]]'

clang/test/Frontend/stats-file.c

 // RUN: %clang_cc1 -emit-llvm -o /dev/null -stats-file=%t %s
 // RUN: FileCheck -input-file=%t %s
 // CHECK: {
 //  ... here come some json values ...
 // CHECK: }
 
-// RUN: %clang_cc1 -emit-llvm -o %t -stats-file=%t.doesnotexist/bla %s 2>&1 | FileCheck -check-prefix=OUTPUTFAIL %s
-// OUTPUTFAIL: warning: unable to open statistics output file '{{.*}}doesnotexist{{.}}bla': '{{[Nn]}}o such file or directory'
+// RUN: %clang_cc1 -emit-llvm -o %t -stats-file=%t.doesnotexist/bla %s 2>&1 | FileCheck -DMSG=%errc_ENOENT -check-prefix=OUTPUTFAIL %s
+// OUTPUTFAIL: warning: unable to open statistics output file '{{.*}}doesnotexist{{.}}bla': '[[MSG]]'

lld/test/COFF/driver.test

 # RUN: not lld-link nosuchfile.obj >& %t.log
-# RUN: FileCheck -check-prefix=MISSING %s < %t.log
-MISSING: 'nosuchfile.obj': {{[Nn]}}o such file or directory
+# RUN: FileCheck -DMSG=%errc_ENOENT -check-prefix=MISSING %s < %t.log
+MISSING: 'nosuchfile.obj': [[MSG]]
 
 # RUN: lld-link --version | FileCheck -check-prefix=VERSION %s
 VERSION: {{LLD [0-9]+\.[0-9]+}}
 
 # RUN: yaml2obj %p/Inputs/export.yaml -o %t.obj
 # RUN: lld-link /out:%t.dll /dll %t.obj
 # RUN: not lld-link /out:%t.exe %t.dll 2>&1 | FileCheck -check-prefix=BADFILE %s
 BADFILE: bad file type. Did you specify a DLL instead of an import library?

lld/test/COFF/manifestinput-error.test

 # UNSUPPORTED: manifest_tool
 # UNSUPPORTED: libxml2
 
 # RUN: yaml2obj %p/Inputs/ret42.yaml -o %t.obj
 # RUN: env LLD_IN_TEST=1 not lld-link /out:%t.exe /entry:main \
 # RUN:   /manifest:embed \
 # RUN:   /manifestuac:"level='requireAdministrator'" \
-# RUN:   /manifestinput:%p/Inputs/manifestinput.test %t.obj 2>&1 | FileCheck %s
+# RUN:   /manifestinput:%p/Inputs/manifestinput.test %t.obj 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
 
-# CHECK: error: unable to find mt.exe in PATH: {{[Nn]}}o such file or directory
+# CHECK: error: unable to find mt.exe in PATH: [[MSG]]

lld/test/COFF/nodefaultlib.test

 # RUN: cp %p/Inputs/hello64.obj %T
 # RUN: cp %p/Inputs/std64.lib %T
 
 # RUN: not lld-link /out:%t.exe /entry:main /subsystem:console \
 # RUN:   hello64.obj /defaultlib:std64.lib >& %t.log
-# RUN: FileCheck -check-prefix=CHECK1 %s < %t.log
+# RUN: FileCheck -DMSG=%errc_ENOENT -check-prefix=CHECK1 %s < %t.log
 
 # RUN: not lld-link /out:%t.exe /entry:main /subsystem:console \
 # RUN:   hello64 /defaultlib:std64.lib >& %t.log
-# RUN: FileCheck -check-prefix=CHECK2 %s < %t.log
+# RUN: FileCheck -DMSG=%errc_ENOENT -check-prefix=CHECK2 %s < %t.log
 
 # RUN: lld-link /libpath:%T /out:%t.exe /entry:main \
 # RUN:   /subsystem:console hello64.obj /defaultlib:std64.lib \
 # RUN:   /nodefaultlib:std64.lib >& %t.log || true
 # RUN: FileCheck -check-prefix=CHECK3 %s < %t.log
 
 # RUN: lld-link /libpath:%T /out:%t.exe /entry:main \
 # RUN:   /subsystem:console hello64.obj /defaultlib:std64 \
 # RUN:   /nodefaultlib:std64.lib >& %t.log || true
 # RUN: FileCheck -check-prefix=CHECK3 %s < %t.log
 
-CHECK1: error: could not open 'hello64.obj': {{[Nn]}}o such file or directory
-CHECK2: error: could not open 'hello64': {{[Nn]}}o such file or directory
+CHECK1: error: could not open 'hello64.obj': [[MSG]]
+CHECK2: error: could not open 'hello64': [[MSG]]
 CHECK3: error: undefined symbol: MessageBoxA
 CHECK3-NEXT: >>> referenced by {{.*}}hello64.obj:(main)
 
 # RUN: lld-link /libpath:%T /out:%t.exe /entry:main \
 # RUN:   /subsystem:console hello64.obj /defaultlib:std64.lib
 
 # RUN: env LIB=%T lld-link /out:%t.exe /entry:main \
 # RUN:   /subsystem:console hello64.obj /defaultlib:std64.lib
 
 MSVC stamps uppercase references in OBJ directives, thus ensure that passing lowercase 'libcmt' and 'oldnames' to /nodefaultlib works.
 # RUN: lld-link %S/Inputs/precomp.obj %S/Inputs/precomp-a.obj %S/Inputs/precomp-b.obj /nodefaultlib:libcmt /nodefaultlib:oldnames /entry:main /debug /pdb:%t.pdb /out:%t.exe /opt:ref /opt:icf
 # RUN: llvm-pdbutil dump -modules %t.pdb | FileCheck %s -check-prefix UPPERCASE
 
 UPPERCASE-NOT: OLDNAMES
 UPPERCASE-NOT: LIBCMT

lld/test/COFF/pdb-type-server-invalid-signature.yaml

 # RUN: cd %S
 
 # VALID-SIGNATURE-NOT: warning: Cannot use debug info for '{{.*}}.obj'
 # VALID-SIGNATURE-NOT: The signature does not match; the file(s) might be out of date
 
 # Test an invalid path reference to a PDB type server; as a fallback LLD should try to load the PDB in the same path as the OBJ
 # RUN: yaml2obj %S/Inputs/pdb-type-server-invalid-path.yaml -o %t3.obj
 # RUN: cp %S/Inputs/pdb-diff-cl.pdb %T
-# RUN: lld-link %t3.obj -out:%t3.exe -debug -pdb:%t3.pdb -nodefaultlib -entry:main 2>&1 | FileCheck %s -check-prefix=INVALID-PATH -allow-empty
+# RUN: lld-link %t3.obj -out:%t3.exe -debug -pdb:%t3.pdb -nodefaultlib -entry:main 2>&1 | FileCheck -DMSG=%errc_ENOENT %s -check-prefix=INVALID-PATH -allow-empty
 
 # INVALID-PATH-NOT: warning: Cannot use debug info for '{{.*}}3.obj' [LNK4099]
-# INVALID-PATH-NOT: failed to load reference 'c:\some_invalid_path_AABB98765\pdb-diff-cl.pdb': {{[Nn]}}o such file or directory
+# INVALID-PATH-NOT: failed to load reference 'c:\some_invalid_path_AABB98765\pdb-diff-cl.pdb': [[MSG]]
 
 --- !COFF
 header:
   Machine:         IMAGE_FILE_MACHINE_AMD64
   Characteristics: [  ]
 sections:
   - Name:            '.debug$S'
     Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]

lld/test/COFF/pdb-type-server-missing.yaml

 # This is an object compiled with /Zi (see the LF_TYPESERVER2 record) without an
 # adjacent type server PDB. Test that LLD fails gracefully on it.
 # Also try linking another OBJ with a reference to the same PDB
 
 # RUN: yaml2obj %s -o %t1.obj
 # RUN: yaml2obj %p/Inputs/pdb-type-server-missing-2.yaml -o %t2.obj
-# RUN: lld-link %t1.obj %t2.obj -out:%t.exe -debug -pdb:%t.pdb -nodefaultlib -entry:main 2>&1 | FileCheck %s -check-prefix=WARN
-# RUN: lld-link %t1.obj %t2.obj -out:%t.exe -debug:ghash -pdb:%t.pdb -nodefaultlib -entry:main 2>&1 | FileCheck %s -check-prefix=WARN
-# RUN: lld-link %t1.obj %t2.obj -out:%t.exe -debug -pdb:%t.pdb -nodefaultlib -entry:main /ignore:4099 2>&1 | FileCheck %s -check-prefix=IGNORE -allow-empty
-# RUN: not lld-link %t1.obj %t2.obj -out:%t.exe -debug -pdb:%t.pdb -nodefaultlib -entry:main /WX 2>&1 | FileCheck %s -check-prefix=ERR
-# RUN: lld-link %t1.obj %t2.obj -out:%t.exe -debug -pdb:%t.pdb -nodefaultlib -entry:main /ignore:4099 /WX 2>&1 | FileCheck %s -check-prefix=IGNORE-ERR -allow-empty
+# RUN: lld-link %t1.obj %t2.obj -out:%t.exe -debug -pdb:%t.pdb -nodefaultlib -entry:main 2>&1 | FileCheck -DMSG=%errc_ENOENT %s -check-prefix=WARN
+# RUN: lld-link %t1.obj %t2.obj -out:%t.exe -debug:ghash -pdb:%t.pdb -nodefaultlib -entry:main 2>&1 | FileCheck -DMSG=%errc_ENOENT %s -check-prefix=WARN
+# RUN: lld-link %t1.obj %t2.obj -out:%t.exe -debug -pdb:%t.pdb -nodefaultlib -entry:main /ignore:4099 2>&1 | FileCheck -DMSG=%errc_ENOENT %s -check-prefix=IGNORE -allow-empty
+# RUN: not lld-link %t1.obj %t2.obj -out:%t.exe -debug -pdb:%t.pdb -nodefaultlib -entry:main /WX 2>&1 | FileCheck -DMSG=%errc_ENOENT %s -check-prefix=ERR
+# RUN: lld-link %t1.obj %t2.obj -out:%t.exe -debug -pdb:%t.pdb -nodefaultlib -entry:main /ignore:4099 /WX 2>&1 | FileCheck -DMSG=%errc_ENOENT %s -check-prefix=IGNORE-ERR -allow-empty
 
 # WARN: warning: Cannot use debug info for '{{.*}}.obj' [LNK4099]
-# WARN-NEXT: {{N|n}}o such file or directory
+# WARN-NEXT: [[MSG]]
 
 # IGNORE-NOT: warning: Cannot use debug info for '{{.*}}.obj' [LNK4099]
-# IGNORE-NOT: {{N|n}}o such file or directory
+# IGNORE-NOT: [[MSG]]
 
 # ERR: error: Cannot use debug info for '{{.*}}.obj' [LNK4099]
-# ERR-NEXT: {{N|n}}o such file or directory
+# ERR-NEXT: [[MSG]]
 
 # IGNORE-ERR-NOT: error: Cannot use debug info for '{{.*}}.obj' [LNK4099]
-# IGNORE-ERR-NOT: {{N|n}}o such file or directory
+# IGNORE-ERR-NOT: [[MSG]]
 
 --- !COFF
 header:
   Machine:         IMAGE_FILE_MACHINE_AMD64
   Characteristics: [  ]
 sections:
   - Name:            '.debug$S'
     Characteristics: [ IMAGE_SCN_CNT_INITIALIZED_DATA, IMAGE_SCN_MEM_DISCARDABLE, IMAGE_SCN_MEM_READ ]

lld/test/ELF/archive-thin-missing-member.s

 # REQUIRES: x86
 
 # RUN: rm -f %t-no-syms.a
 # RUN: rm -f %t-syms.a
 # RUN: llvm-mc -filetype=obj -triple=x86_64-pc-linux %s -o %t.o
 # RUN: llvm-ar rcTS %t-no-syms.a %t.o
 # RUN: llvm-ar rcT %t-syms.a %t.o
 # RUN: rm %t.o
 
 # Test error when loading symbols from missing thin archive member.
-# RUN: not ld.lld --entry=_Z1fi %t-no-syms.a -o /dev/null 2>&1 | FileCheck %s --check-prefix=ERR1
-# ERR1: {{.*}}-no-syms.a: could not get the buffer for a child of the archive: '{{.*}}.o': {{[Nn]}}o such file or directory
+# RUN: not ld.lld --entry=_Z1fi %t-no-syms.a -o /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=ERR1
+# ERR1: {{.*}}-no-syms.a: could not get the buffer for a child of the archive: '{{.*}}.o': [[MSG]]
 
 # Test error when thin archive has symbol table but member is missing.
-# RUN: not ld.lld --entry=_Z1fi -m elf_amd64_fbsd %t-syms.a -o /dev/null 2>&1 | FileCheck %s --check-prefix=ERR2
-# ERR2: {{.*}}-syms.a: could not get the buffer for the member defining symbol f(int): '{{.*}}.o': {{[Nn]}}o such file or directory
-# RUN: not ld.lld --entry=_Z1fi --no-demangle -m elf_amd64_fbsd %t-syms.a -o /dev/null 2>&1 | FileCheck %s --check-prefix=ERR2MANGLE
-# ERR2MANGLE: {{.*}}-syms.a: could not get the buffer for the member defining symbol _Z1fi: '{{.*}}.o': {{[Nn]}}o such file or directory
+# RUN: not ld.lld --entry=_Z1fi -m elf_amd64_fbsd %t-syms.a -o /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=ERR2
+# ERR2: {{.*}}-syms.a: could not get the buffer for the member defining symbol f(int): '{{.*}}.o': [[MSG]]
+# RUN: not ld.lld --entry=_Z1fi --no-demangle -m elf_amd64_fbsd %t-syms.a -o /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=ERR2MANGLE
+# ERR2MANGLE: {{.*}}-syms.a: could not get the buffer for the member defining symbol _Z1fi: '{{.*}}.o': [[MSG]]
 
 # Test error when thin archive is linked using --whole-archive but member is missing.
-# RUN: not ld.lld --entry=_Z1fi --whole-archive %t-syms.a -o /dev/null 2>&1 | FileCheck %s --check-prefix=ERR3
-# ERR3: {{.*}}-syms.a: could not get the buffer for a child of the archive: '{{.*}}.o': {{[Nn]}}o such file or directory
+# RUN: not ld.lld --entry=_Z1fi --whole-archive %t-syms.a -o /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=ERR3
+# ERR3: {{.*}}-syms.a: could not get the buffer for a child of the archive: '{{.*}}.o': [[MSG]]
 
 .global _Z1fi
 _Z1fi:
     nop

lld/test/ELF/basic.s

 # WINRSP: cannot open c:\blah\foo
 
 # Test for the response file (invalid quoting style)
 # RUN: not ld.lld --rsp-quoting=patatino %t 2>&1 | FileCheck %s \
 # RUN:   --check-prefix=INVRSP
 # INVRSP: invalid response file quoting: patatino
 
 # RUN: not ld.lld %t.foo -o /dev/null 2>&1 | \
-# RUN:  FileCheck --check-prefix=MISSING %s
-# MISSING: cannot open {{.*}}.foo: {{[Nn]}}o such file or directory
+# RUN:  FileCheck -DMSG=%errc_ENOENT --check-prefix=MISSING %s
+# MISSING: cannot open {{.*}}.foo: [[MSG]]
 
 # RUN: not ld.lld -o /dev/null 2>&1 | \
 # RUN:  FileCheck --check-prefix=NO_INPUT %s
 # NO_INPUT: ld.lld{{.*}}: no input files
 
 # RUN: not ld.lld %t.no.such.file -o /dev/null 2>&1 | \
-# RUN:  FileCheck --check-prefix=CANNOT_OPEN %s
-# CANNOT_OPEN: cannot open {{.*}}.no.such.file: {{[Nn]}}o such file or directory
+# RUN:  FileCheck -DMSG=%errc_ENOENT --check-prefix=CANNOT_OPEN %s
+# CANNOT_OPEN: cannot open {{.*}}.no.such.file: [[MSG]]
 
 # RUN: not ld.lld --foo 2>&1 | FileCheck --check-prefix=UNKNOWN %s
 # UNKNOWN: unknown argument '--foo'
 
 # RUN: not ld.lld %t -o /dev/null -m wrong_emul_fbsd 2>&1 | FileCheck --check-prefix=UNKNOWN_EMUL %s
 # UNKNOWN_EMUL: unknown emulation: wrong_emul_fbsd

lld/test/ELF/reproduce-error.s

 # RUN: rm -rf %t.dir
 # RUN: mkdir -p %t.dir
 # RUN: cd %t.dir
 
-# RUN: not ld.lld --reproduce repro.tar abc -o t 2>&1 | FileCheck %s
-# CHECK: cannot open abc: {{N|n}}o such file or directory
+# RUN: not ld.lld --reproduce repro.tar abc -o t 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
+# CHECK: cannot open abc: [[MSG]]
 
 # RUN: tar xOf repro.tar repro/response.txt | FileCheck --check-prefix=RSP %s
 # RSP: abc
 # RSP: -o t

lld/test/ELF/symbol-ordering-file.s

 
 # AFTER:      Contents of section .foo:
 # AFTER-NEXT:  44337755 662211
 # AFTER:      Contents of section .init:
 # AFTER-NEXT:  1122
 
 ## Show that a nonexistent symbol ordering file causes an error.
 # RUN: not ld.lld --symbol-ordering-file=%t.nonexistent %t.o -o %t3.out 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=ERR -DFILE=%t.nonexistent
+# RUN:   FileCheck -DMSG=%errc_ENOENT %s --check-prefix=ERR -DFILE=%t.nonexistent
 
-# ERR: error: cannot open [[FILE]]: {{[Nn]}}o such file or directory
+# ERR: error: cannot open [[FILE]]: [[MSG]]
 
 ## Show that an empty ordering file can be handled (symbols remain in their
 ## normal order).
 # RUN: touch %t.empty
 # RUN: ld.lld --symbol-ordering-file=%t.empty %t.o -o %t4.out
 # RUN: llvm-objdump -s %t4.out | FileCheck %s --check-prefix=BEFORE
 
 .section .foo,"ax",@progbits,unique,1

lld/test/MachO/invalid/no-filelist.s

 # REQUIRES: x86
 # RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin %s -o %t.o
-# RUN: not %lld -filelist nonexistent %t.o -o %t 2>&1 | FileCheck %s
-# CHECK: cannot open nonexistent: {{N|n}}o such file or directory
+# RUN: not %lld -filelist nonexistent %t.o -o %t 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
+# CHECK: cannot open nonexistent: [[MSG]]
 
 .globl _main
 
 _main:
   ret

llvm/docs/TestingGuide.rst

``%:s, %:S, %:t, %:T:``
   Act like the corresponding substitution above but remove colons at
   the beginning of Windows paths. This is useful to allow concatenation
   of absolute paths on Windows to produce a legal path.
 
    Example: ``%s:  C:\Desktop Files\foo_test.s.tmp``
 
    Example: ``%:s: C\Desktop Files\foo_test.s.tmp``
 
+``%errc_<ERRCODE>``
+
+ Some error messages may be substituted to allow different spellings
+ based on the host platform.
+
+   The following error codes are currently supported: ENOENT, EISDIR.
+
+   Example: ``Linux %errc_ENOENT: No such file or directory``
+
+   Example: ``Windows %errc_ENOENT: no such file or directory``
 
 **LLVM-specific substitutions:**
 
 ``%shlibext``
    The suffix for the host platforms shared library files. This includes the
    period as the first character.
 
    Example: ``.so`` (Linux), ``.dylib`` (macOS), ``.dll`` (Windows)

llvm/test/DebugInfo/symbolize-missing-file.test

-RUN: llvm-symbolizer --obj=unexisting-file 0x1234 2>&1 | FileCheck %s
+RUN: llvm-symbolizer --obj=unexisting-file 0x1234 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
 
-CHECK: LLVMSymbolizer: error reading file: {{[Nn]}}o such file or directory
+CHECK: LLVMSymbolizer: error reading file: [[MSG]]

llvm/test/MC/Hexagon/not_found.s

-# RUN: not llvm-mc -arch=hexagon -filetype=asm junk123.s 2>%t ; FileCheck %s < %t
+# RUN: not llvm-mc -arch=hexagon -filetype=asm junk123.s 2>%t ; FileCheck -DMSG=%errc_ENOENT %s < %t
 #
 
-# CHECK: junk123.s: {{[N|n]}}o such file or directory
+# CHECK: junk123.s: [[MSG]]

llvm/test/Object/archive-extract-dir.test

 RUN: mkdir -p %t
 RUN: cd %t
 RUN: rm -rf foo
 RUN: echo foo > foo
 RUN: rm -f test.a
 RUN: llvm-ar rc          test.a foo
 RUN: rm foo
 RUN: mkdir foo
-RUN: not llvm-ar x test.a foo 2>&1 | FileCheck %s
+RUN: not llvm-ar x test.a foo 2>&1 | FileCheck -DMSG=%errc_EISDIR %s
 
-CHECK: foo: {{[Ii]}}s a directory
+CHECK: foo: [[MSG]]

llvm/test/Object/archive-extract.test

 NOTFOUND: error: 'foo.o' was not found
 
 RUN: not llvm-ar x %p/Inputs/thin.a foo.o 2>&1 | FileCheck %s --check-prefix=THINEXTRACT
 THINEXTRACT: extracting from a thin archive is not supported
 
 RUN: llvm-ar p %p/Inputs/thin.a evenlen | FileCheck %s --check-prefix=EVENLEN
 EVENLEN: evenlen
 
-RUN: not llvm-ar p %p/Inputs/thin-path.a t/test2.o 2>&1 | FileCheck %s --check-prefix=MISSING
-MISSING: error: {{N|n}}o such file or directory
+RUN: not llvm-ar p %p/Inputs/thin-path.a t/test2.o 2>&1 | FileCheck %s --DMSG=%errc_ENOENT --check-prefix=MISSING
+MISSING: error: [[MSG]]

llvm/test/Object/directory.ll

 ;RUN: rm -rf %t && mkdir -p %t
-;RUN: not llvm-ar r %t/test.a . 2>&1 | FileCheck %s
-;CHECK: .: {{I|i}}s a directory
+;RUN: not llvm-ar r %t/test.a . 2>&1 | FileCheck -DMSG=%errc_EISDIR %s
+;CHECK: .: [[MSG]]
 
 ;RUN: rm -f %t/test.a
 ;RUN: touch %t/a-very-long-file-name
 ;RUN: llvm-ar r %t/test.a %s %t/a-very-long-file-name
 ;RUN: llvm-ar r %t/test.a %t/a-very-long-file-name
 ;RUN: llvm-ar t %t/test.a | FileCheck -check-prefix=MEMBERS %s
 ;MEMBERS-NOT: /
 ;MEMBERS: directory.ll
 ;MEMBERS: a-very-long-file-name
 ;MEMBERS-NOT: a-very-long-file-name

llvm/test/tools/dsymutil/X86/papertrail-warnings.test

-RUN: env RC_DEBUG_OPTIONS=1 dsymutil -f %p/../Inputs/basic.macho.x86_64 -o - | llvm-dwarfdump -v - | FileCheck %s
+RUN: env RC_DEBUG_OPTIONS=1 dsymutil -f %p/../Inputs/basic.macho.x86_64 -o - | llvm-dwarfdump -v - | FileCheck -DMSG=%errc_ENOENT %s
 
 CHECK: .debug_info contents:
 CHECK:  Compile Unit:
 CHECK:  DW_TAG_compile_unit [1] *
 CHECK:    DW_AT_producer {{.*}}"dsymutil
 CHECK:    DW_AT_name {{.*}}"/Inputs/basic1.macho.x86_64.o"
 CHECK:    DW_TAG_constant [2]
 CHECK:      DW_AT_name {{.*}}"dsymutil_warning"
 CHECK:      DW_AT_artificial [DW_FORM_flag] (0x01)
-CHECK:      DW_AT_const_value {{.*}}"unable to open object file: {{[Nn]}}o such file or directory"
+CHECK:      DW_AT_const_value {{.*}}"unable to open object file: [[MSG]]"
 CHECK:    NULL
 CHECK:  Compile Unit:
 CHECK:  DW_TAG_compile_unit [1] *
 CHECK:    DW_AT_producer {{.*}}"dsymutil
 CHECK:    DW_AT_name {{.*}}"/Inputs/basic2.macho.x86_64.o"
 CHECK:    DW_TAG_constant [2]
 CHECK:      DW_AT_name {{.*}}"dsymutil_warning"
 CHECK:      DW_AT_artificial [DW_FORM_flag] (0x01)
-CHECK:      DW_AT_const_value {{.*}}"unable to open object file: {{[Nn]}}o such file or directory"
+CHECK:      DW_AT_const_value {{.*}}"unable to open object file: [[MSG]]"
 CHECK:    NULL
 CHECK:  Compile Unit:
 CHECK:  DW_TAG_compile_unit [1] *
 CHECK:    DW_AT_producer {{.*}}"dsymutil
 CHECK:    DW_AT_name {{.*}}"/Inputs/basic3.macho.x86_64.o"
 CHECK:    DW_TAG_constant [2]
 CHECK:      DW_AT_name {{.*}}"dsymutil_warning"
 CHECK:      DW_AT_artificial [DW_FORM_flag] (0x01)
-CHECK:      DW_AT_const_value {{.*}}"unable to open object file: {{[Nn]}}o such file or directory"
+CHECK:      DW_AT_const_value {{.*}}"unable to open object file: [[MSG]]"
 CHECK:    NULL

llvm/test/tools/dsymutil/archive-timestamp.test

-# RUN: dsymutil -no-output -oso-prepend-path=%p -y %s 2>&1 | FileCheck %s
+# RUN: dsymutil -no-output -oso-prepend-path=%p -y %s 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
 
 # This is the archive member part of basic-archive.macho.x86_64 debug map with corrupted timestamps.
 
-# CHECK: warning: {{.*}}libbasic.a(basic2.macho.x86_64.o): {{[Nn]o}} such file
-# CHECK: warning: {{.*}}libbasic.a(basic3.macho.x86_64.o): {{[Nn]o}} such file
+# CHECK: warning: {{.*}}libbasic.a(basic2.macho.x86_64.o): [[MSG]]
+# CHECK: warning: {{.*}}libbasic.a(basic3.macho.x86_64.o): [[MSG]]
 
 ---
 triple:          'x86_64-apple-darwin'
 objects:
   - filename:        '/Inputs/libbasic.a(basic2.macho.x86_64.o)'
     timestamp:       141869239
     symbols:
       - { sym: _foo, objAddr: 0x0000000000000020, binAddr: 0x0000000100000ED0, size: 0x00000050 }

llvm/test/tools/dsymutil/debug-map-parsing.test

 RUN: dsymutil -dump-debug-map -oso-prepend-path=%p %p/Inputs/basic.macho.x86_64 | FileCheck %s
 RUN: dsymutil -dump-debug-map -oso-prepend-path=%p %p/Inputs/basic-lto.macho.x86_64 | FileCheck %s --check-prefix=CHECK-LTO
 RUN: dsymutil -verbose -dump-debug-map -oso-prepend-path=%p %p/Inputs/basic-archive.macho.x86_64 2>&1 | FileCheck %s --check-prefix=CHECK-ARCHIVE
-RUN: dsymutil -dump-debug-map %p/Inputs/basic.macho.x86_64 2>&1 | FileCheck %s --check-prefix=NOT-FOUND
-RUN: not dsymutil -dump-debug-map %p/Inputs/inexistant 2>&1 | FileCheck %s --check-prefix=NO-EXECUTABLE
+RUN: dsymutil -dump-debug-map %p/Inputs/basic.macho.x86_64 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=NOT-FOUND
+RUN: not dsymutil -dump-debug-map %p/Inputs/inexistant 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=NO-EXECUTABLE
 
 
 Check that We can parse the debug map of the basic executable.
 
 CHECK-NOT: error
 CHECK: ---
 CHECK: triple: 'x86_64-apple-darwin'
 CHECK: binary-path:{{.*}}/Inputs/basic.macho.x86_64

 CHECK-ARCHIVE-DAG:  sym: _inc, objAddr: 0x70, binAddr: 0x100000F90, size: 0x19
 CHECK-ARCHIVE-DAG:  sym: _val, binAddr: 0x100001008, size: 0x0
 CHECK-ARCHIVE: ...
 
 Check that we warn about missing object files (this presumes that the files aren't
 present in the machine's /Inputs/ folder, which should be a pretty safe bet).
 
 warning: (x86_64) /Inputs/basic1.macho.x86_64.o unable to open object file:
-NOT-FOUND: warning: (x86_64) {{.*}}/Inputs/basic1.macho.x86_64.o unable to open object file: {{[Nn]o}} such file
-NOT-FOUND: warning: (x86_64) {{.*}}/Inputs/basic2.macho.x86_64.o unable to open object file: {{[Nn]o}} such file
-NOT-FOUND: warning: (x86_64) {{.*}}/Inputs/basic3.macho.x86_64.o unable to open object file: {{[Nn]o}} such file
+NOT-FOUND: warning: (x86_64) {{.*}}/Inputs/basic1.macho.x86_64.o unable to open object file: [[MSG]]
+NOT-FOUND: warning: (x86_64) {{.*}}/Inputs/basic2.macho.x86_64.o unable to open object file: [[MSG]]
+NOT-FOUND: warning: (x86_64) {{.*}}/Inputs/basic3.macho.x86_64.o unable to open object file: [[MSG]]
 NOT-FOUND: ---
 NOT-FOUND-NEXT: triple: 'x86_64-apple-darwin'
 NOT-FOUND-NEXT: binary-path:{{.*}}/Inputs/basic.macho.x86_64
 NOT-FOUND-NEXT: ...
 
 Check that we correctly error out on invalid executable.
 
-NO-EXECUTABLE: cannot parse{{.*}}/inexistant': {{[Nn]o}} such file
+NO-EXECUTABLE: cannot parse{{.*}}/inexistant': [[MSG]]
 NO-EXECUTABLE-NOT: ---

llvm/test/tools/llvm-ar/error-opening-directory.test

 ## Unsupported on AIX and FreeBSD as AIX and FreeBSD 12 and earlier allow
 ## reading directories by default.
 # UNSUPPORTED: system-freebsd, system-aix
 
 # RUN: rm -rf %t && mkdir -p %t
 
 ## Passing in a directory:
 # RUN: mkdir -p %t/tmpDir
 # RUN: not llvm-ar p %t/tmpDir 2>&1 | \
-# RUN:    FileCheck %s --check-prefix=IS-DIR -DARCHIVE=%t/tmpDir
+# RUN:    FileCheck %s --check-prefix=IS-DIR -DARCHIVE=%t/tmpDir -DMSG=%errc_EISDIR
 
-# IS-DIR: error: unable to open '[[ARCHIVE]]': {{[iI]}}s a directory
+# IS-DIR: error: unable to open '[[ARCHIVE]]': [[MSG]]

llvm/test/tools/llvm-ar/missing-thin-archive-member.test

 ## Test llvm-ar errors when thin archive members are missing.
 
 # RUN: rm -rf %t-archive.a
 # RUN: echo contents > %t-temp.txt
 
 ## File is not in archive.
 # RUN: llvm-ar qT %t-archive.a
 # RUN: not llvm-ar p %t-archive.a %t-temp.txt 2>&1 | FileCheck %s -DPATH=%t-temp.txt --check-prefix=MISSING
 
 # MISSING: error: '[[PATH]]' was not found
 
 ## File has been deleted.
 # RUN: llvm-ar qT %t-archive.a %t-temp.txt
 # RUN: rm %t-temp.txt
-# RUN: not llvm-ar p %t-archive.a 2>&1 | FileCheck %s --check-prefix=DELETED
+# RUN: not llvm-ar p %t-archive.a 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=DELETED
 
-# DELETED: error: {{[Nn]}}o such file or directory
+# DELETED: error: [[MSG]]

llvm/test/tools/llvm-ar/move.test

 # RUN: not llvm-ar m 2>&1 \
 # RUN:   | FileCheck %s --check-prefix=NO-ARCHIVE
 
 # NO-ARCHIVE: error: an archive name must be specified
 
 ## Member does not exist:
 # RUN: llvm-ar rc %t/missing.a %t/1.o %t/2.o %t/3.o
 # RUN: not llvm-ar m %t/missing.a %t/missing.txt 2>&1 \
-# RUN:   | FileCheck %s --check-prefix=MISSING-FILE -DFILE=%t/missing.txt
+# RUN:   | FileCheck %s --check-prefix=MISSING-FILE -DFILE=%t/missing.txt -DMSG=%errc_ENOENT
 
-# MISSING-FILE: error: [[FILE]]: {{[nN]}}o such file or directory
+# MISSING-FILE: error: [[FILE]]: [[MSG]]
 
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
   Type:    ET_REL
   Machine: EM_X86_64
 Sections:

llvm/test/tools/llvm-ar/print.test

 ## No archive:
 # RUN: not llvm-ar p 2>&1 \
 # RUN:   | FileCheck %s --check-prefix=NO-ARCHIVE
 # 
 # NO-ARCHIVE: error: an archive name must be specified
 
 ## Archive does not exist:
 # RUN: not llvm-ar p %t/missing.a 2>&1 \
-# RUN:   | FileCheck %s --check-prefix=MISSING-ARCHIVE -DARCHIVE=%t/missing.a
+# RUN:   | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=MISSING-ARCHIVE -DARCHIVE=%t/missing.a
 
-# MISSING-ARCHIVE: error: unable to load '[[ARCHIVE]]': {{[nN]}}o such file or directory
+# MISSING-ARCHIVE: error: unable to load '[[ARCHIVE]]': [[MSG]]
 
 ## Member does not exist:
 # RUN: not llvm-ar p %t/archive.a %t-missing.txt 2>&1 \
 # RUN:   | FileCheck %s --check-prefix=MISSING-FILE -DFILE=%t-missing.txt
 
 # MISSING-FILE: error: '[[FILE]]' was not found
 
 ## Print thin archive:
 # RUN: llvm-ar Trc %t/thin-archive.a %t/1.txt %t/2.txt %t/3.txt
 # RUN: llvm-ar p %t/archive.a %t/2.txt \
 # RUN:   | FileCheck %s --check-prefix=SINGLE --match-full-lines --implicit-check-not {{.}}

llvm/test/tools/llvm-ar/quick-append.test

 ## No archive:
 # RUN: not llvm-ar qc 2>&1 \
 # RUN:   | FileCheck %s --check-prefix=NO-ARCHIVE
 
 # NO-ARCHIVE: error: an archive name must be specified
 
 ## Member does not exist:
 # RUN: not llvm-ar qc %t/missing.a %t/missing.txt 2>&1 \
-# RUN:   | FileCheck %s --check-prefix=MISSING-FILE -DFILE=%t/missing.txt
+# RUN:   | FileCheck %s --check-prefix=MISSING-FILE -DFILE=%t/missing.txt -DMSG=%errc_ENOENT
 
-# MISSING-FILE: error: [[FILE]]: {{[nN]}}o such file or directory
+# MISSING-FILE: error: [[FILE]]: [[MSG]]
 
 ## Create and append members to a thin archive:
 # RUN: llvm-ar qcT %t/thin-multiple.a %t/1.o
 # RUN: llvm-ar qcT %t/thin-multiple.a %t/2.o
 # RUN: llvm-ar t %t/thin-multiple.a \
 # RUN:   | FileCheck %s --check-prefix=MULTIPLE
 
 # RUN: llvm-ar qcT %t/thin-same.a %t/1.o %t/1.o

llvm/test/tools/llvm-ar/replace.test

 # RUN: not llvm-ar r 2>&1 \
 # RUN:   | FileCheck %s --check-prefix=NO-ARCHIVE
 
 # NO-ARCHIVE: error: an archive name must be specified
 
 ## Member does not exist:
 # RUN: llvm-ar rc %t/missing.a %t/1.o %t/2.o %t/3.o
 # RUN: not llvm-ar r %t/missing.a %t/missing.txt 2>&1 \
-# RUN:   | FileCheck %s --check-prefix=MISSING-FILE -DFILE=%t/missing.txt
+# RUN:   | FileCheck %s --check-prefix=MISSING-FILE -DFILE=%t/missing.txt -DMSG=%errc_ENOENT
 
-# MISSING-FILE: error: [[FILE]]: {{[Nn]}}o such file or directory
+# MISSING-FILE: error: [[FILE]]: [[MSG]]
 
 ## Create and Replace member of thin archive:
 # RUN: llvm-ar rcT %t/thin.a %t/1.o %t/2.o %t/3.o
 # RUN: yaml2obj %s -o %t/1.o --docnum=4
 # RUN: llvm-ar rT %t/thin.a %t/1.o
 # RUN: llvm-ar t %t/thin.a | FileCheck %s --check-prefix=SINGLE
 
 # RUN: llvm-nm --print-armap %t/thin.a \

llvm/test/tools/llvm-ar/response.test

 
 ## rsp-quoting
 # RUN: not llvm-ar --rsp-quoting=foobar @%t.response1.txt 2>&1 | \
 # RUN:   FileCheck  %s --check-prefix=ERROR
 # ERROR: Invalid response file quoting style foobar
 
 # RUN: echo -e 'rc %/t.a blah\\foo' > %t-rsp.txt
 # RUN: not llvm-ar --rsp-quoting=windows @%t-rsp.txt 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=WIN
-# WIN: error: blah\foo: {{[Nn]}}o such file or directory
+# RUN:   FileCheck -DMSG=%errc_ENOENT %s --check-prefix=WIN
+# WIN: error: blah\foo: [[MSG]]
 
 # RUN: not llvm-ar -rsp-quoting posix @%t-rsp.txt 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=POSIX
-# POSIX: error: blahfoo: {{[Nn]}}o such file or directory
+# RUN:   FileCheck -DMSG=%errc_ENOENT %s --check-prefix=POSIX
+# POSIX: error: blahfoo: [[MSG]]

llvm/test/tools/llvm-cxxdump/trivial.test

 
 ELF-I386:      _ZTS1A: 1A
 ELF-I386-NEXT: _ZTV1A[0]: 0
 ELF-I386-NEXT: _ZTV1A[4]: _ZTI1A
 ELF-I386-NEXT: _ZTV1A[8]: _ZN1A1fEv
 
 MIXEDARCOFF-I386:      ??_7S@@6B@[0]: ??_R4S@@6B@
 
-RUN: not llvm-cxxdump %t.blah 2>&1 | FileCheck --check-prefix=ENOENT %s
-ENOENT: {{.*}}.blah: error: {{[Nn]}}o such file or directory
+RUN: not llvm-cxxdump %t.blah 2>&1 | FileCheck -DMSG=%errc_ENOENT --check-prefix=ENOENT %s
+ENOENT: {{.*}}.blah: error: [[MSG]]

llvm/test/tools/llvm-libtool-darwin/filelist.test

 # REVERSE-SYMBOLS:      Archive map
 # REVERSE-SYMBOLS-NEXT: _symbol2 in [[PREFIX]]-input2.o
 # REVERSE-SYMBOLS-NEXT: _symbol1 in [[PREFIX]]-input1.o
 # REVERSE-SYMBOLS-EMPTY:
 
 ## Check that an error is thrown when a file in the filelist doesn't exist in the cwd and no dirname is specified:
 # RUN: echo 'no-such-file' > %t.invalid-list.txt
 # RUN: not llvm-libtool-darwin -static -o %t.lib -filelist %t.invalid-list.txt 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=FILE-ERROR -DFILE=no-such-file
+# RUN:   FileCheck %s --check-prefix=FILE-ERROR -DFILE=no-such-file -DMSG=%errc_ENOENT
 
-# FILE-ERROR: error: '[[FILE]]': {{[nN]}}o such file or directory
+# FILE-ERROR: error: '[[FILE]]': [[MSG]]
 
 ## Check that an error is thrown when the directory exists but does not contain the requested file:
 # RUN: not llvm-libtool-darwin -static -o %t.lib -filelist %t.invalid-list.txt,%t/dirname 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=DIR-ERROR -DDIR=%t/dirname -DFILE=no-such-file
+# RUN:   FileCheck %s --check-prefix=DIR-ERROR -DDIR=%t/dirname -DFILE=no-such-file -DMSG=%errc_ENOENT
 
-# DIR-ERROR: error: '[[DIR]]{{[/\\]}}[[FILE]]': {{[nN]}}o such file or directory
+# DIR-ERROR: error: '[[DIR]]{{[/\\]}}[[FILE]]': [[MSG]]
 
 ## Check that an error is thrown when a file is in the cwd but dirname is specified:
 # RUN: yaml2obj %S/Inputs/input2.yaml -o %basename_t.tmp-input2.o
 # RUN: echo %basename_t.tmp-input2.o > %t.files-cwd.txt
 # RUN: not llvm-libtool-darwin -static -o %t.lib -filelist %t.files-cwd.txt,%t/dirname 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=DIR-ERROR -DDIR=%t/dirname -DFILE=%basename_t.tmp-input2.o
+# RUN:   FileCheck %s --check-prefix=DIR-ERROR -DDIR=%t/dirname -DFILE=%basename_t.tmp-input2.o -DMSG=%errc_ENOENT
 
 ## Check that an error is thrown when the directory doesn't exist:
 # RUN: not llvm-libtool-darwin -static -o %t.lib -filelist %t.files-cwd.txt,%t/Invalid-Dir 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=DIR-ERROR -DDIR=%t/Invalid-Dir -DFILE=%basename_t.tmp-input2.o
+# RUN:   FileCheck %s --check-prefix=DIR-ERROR -DDIR=%t/Invalid-Dir -DFILE=%basename_t.tmp-input2.o -DMSG=%errc_ENOENT
 
 ## Check that an error is thrown when the filelist is empty:
 # RUN: touch %t.empty-list
 # RUN: not llvm-libtool-darwin -static -o %t.lib -filelist %t.empty-list 2>&1 | \
 # RUN:   FileCheck %s --check-prefix=EMPTY-ERROR -DFILE=%t.empty-list
 
 # EMPTY-ERROR: error: file list file: '[[FILE]]' is empty
 
 ## Check that an error is thrown when the filelist contains a blank line:
 # RUN: echo %t-input2.o > %t.blank-line.txt
 # RUN: echo '' >> %t.blank-line.txt
 # RUN: not llvm-libtool-darwin -static -o %t.lib -filelist %t.blank-line.txt 2>&1 | \
 # RUN:   FileCheck %s --check-prefix=EMPTY-FILENAME -DFILE=%t.blank-line.txt
 
 # EMPTY-FILENAME: error: file list file: '[[FILE]]': filename cannot be empty
 
 ## Check that an error is thrown when the filelist contains a line with only spaces:
 # RUN: echo %t-input2.o > %t.space-line.txt
 # RUN: echo "     " >> %t.space-line.txt
 # RUN: not llvm-libtool-darwin -static -o %t.lib -filelist %t.space-line.txt 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=FILE-ERROR -DFILE='     ' --strict-whitespace
+# RUN:   FileCheck %s --check-prefix=FILE-ERROR -DFILE='     ' -DMSG=%errc_ENOENT --strict-whitespace
 
 ## Filelist option specified more than once:
 # RUN: touch %t.list1.txt and %t.list2.txt
 # RUN: not llvm-libtool-darwin -static -o %t.lib -filelist %t.list1.txt -filelist %t.list2.txt 2>&1 | \
 # RUN:   FileCheck %s --check-prefix=DUPLICATE-ERROR
 
 # DUPLICATE-ERROR: for the --filelist option: may only occur zero or one times!

llvm/test/tools/llvm-libtool-darwin/invalid-input-output-args.test

 ## Passing in two output files:
 # RUN: not llvm-libtool-darwin -static %t.input -o %t.lib1 -o %t.lib2 2>&1 | \
 # RUN:   FileCheck %s --check-prefix=DOUBLE-OUTPUT
 
 # DOUBLE-OUTPUT: for the -o option: may only occur zero or one times!
 
 ## Input file not found:
 # RUN: not llvm-libtool-darwin -static -o %t.lib %t.missing 2>&1 | \
-# RUN:   FileCheck %s --check-prefix=NO-FILE -DFILE=%t.missing
+# RUN:   FileCheck %s --check-prefix=NO-FILE -DFILE=%t.missing -DMSG=%errc_ENOENT
 
-# NO-FILE: error: '[[FILE]]': {{[nN]}}o such file or directory
+# NO-FILE: error: '[[FILE]]': [[MSG]]
 
 ## Input file is not an object file:
 # RUN: touch %t.invalid
 # RUN: not llvm-libtool-darwin -static -o %t.lib %t.invalid 2>&1 | \
 # RUN:   FileCheck %s --check-prefix=NOT-OBJECT -DFILE=%basename_t.tmp.invalid
 
 # NOT-OBJECT: error: '[[FILE]]': The file was not recognized as a valid object file
 

llvm/test/tools/llvm-lipo/create-arch.test

 # RUN: yaml2obj %p/Inputs/i386-slice.yaml -o %t-i386.o
 # RUN: yaml2obj %p/Inputs/x86_64-slice.yaml -o %t-x86_64.o
 
 # RUN: llvm-lipo %t-i386.o %t-x86_64.o -create -output %t-universal.o
 # RUN: llvm-lipo %t-i386.o -arch x86_64 %t-x86_64.o -create -output %t-universal-1.o
 # RUN: cmp %t-universal.o %t-universal-1.o
 # RUN: llvm-lipo -arch i386 %t-i386.o -arch x86_64 %t-x86_64.o -create -output %t-universal-2.o
 # RUN: cmp %t-universal.o %t-universal-2.o
 #
 # RUN: not llvm-lipo -arch armv7 %t-i386.o -arch x86_64 %t-x86_64.o -create -output /dev/null 2>&1 | FileCheck --check-prefix=ARCH_NOT_MATCHED %s
 # ARCH_NOT_MATCHED: error: specified architecture: armv7 for file: {{.*}} does not match the file's architecture (i386)
 #
 # # RUN: not llvm-lipo -arch i3866 %t-32.o -create -o /dev/null 2>&1 | FileCheck --check-prefix=INVALID_ARCH %s
 # INVALID_ARCH: error: Invalid architecture: i3866
 #
-# RUN: not llvm-lipo -arch i386 %t-33.o -create -o /dev/null 2>&1 | FileCheck --check-prefix=INVALID_FILE %s
-# INVALID_FILE: {{[nN]}}o such file or directory
+# RUN: not llvm-lipo -arch i386 %t-33.o -create -o /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT --check-prefix=INVALID_FILE %s
+# INVALID_FILE: [[MSG]]

llvm/test/tools/llvm-lipo/replace-invalid-input.test

 # RUN: yaml2obj %p/Inputs/i386-slice.yaml -o %t-32.o
 # RUN: yaml2obj %p/Inputs/arm64-slice.yaml -o %t-arm64.o
 # RUN: yaml2obj %p/Inputs/i386-x86_64-universal.yaml -o %t-universal.o
 
 # RUN: not llvm-lipo %t-universal.o -replace %t-32.o 2>&1 | FileCheck --check-prefix=MISSING_ARG %s
 # MISSING_ARG: error: replace is missing an argument: expects -replace arch_type file_name
 
 # RUN: not llvm-lipo %t-universal.o -replace i386 %t-32.o 2>&1 | FileCheck --check-prefix=OUTPUT_FILE %s
 # OUTPUT_FILE: error: replace expects a single output file to be specified
 
 # RUN: not llvm-lipo %t-universal.o %t-universal.o -replace i386 %t-32.o -o %t.o 2>&1 | FileCheck --check-prefix=INPUT_ARGS %s
 # INPUT_ARGS: error: replace expects a single input file
 
-# RUN: not llvm-lipo %t-universal.o -replace i386 %t-33.o  -o %t.o 2>&1 | FileCheck --check-prefix=INVALID_FILE %s
-# INVALID_FILE: {{[nN]}}o such file or directory
+# RUN: not llvm-lipo %t-universal.o -replace i386 %t-33.o  -o %t.o 2>&1 | FileCheck -DMSG=%errc_ENOENT --check-prefix=INVALID_FILE %s
+# INVALID_FILE: [[MSG]]
 
 # RUN: not llvm-lipo %t-universal.o -replace i3866 %t-32.o -o %t.o 2>&1 | FileCheck --check-prefix=INVALID_ARCH %s
 # INVALID_ARCH: error: Invalid architecture: i3866
 
 # RUN: not llvm-lipo %t-universal.o -replace arm64 %t-32.o -o %t.o 2>&1 | FileCheck --check-prefix=ARCH_NOT_MATCHED %s
 # ARCH_NOT_MATCHED: error: specified architecture: arm64 for file: {{.*}} does not match the file's architecture (i386)
 
 # RUN: not llvm-lipo %t-universal.o -replace arm64 %t-arm64.o -o %t.o 2>&1 | FileCheck --check-prefix=ARCH_NOT_IN_INPUT %s
 # ARCH_NOT_IN_INPUT: error: -replace arm64 <file_name> specified but fat file: {{.*}} does not contain that architecture
 
 # RUN: not llvm-lipo %t-universal.o -replace i386 %t-32.o -replace arm64 %t-universal.o -o %t.o 2>&1 | FileCheck --check-prefix=REPLACE_UNIVERSAL %s
 # REPLACE_UNIVERSAL: replacement file: {{.*}} is a fat file (must be a thin file)
 
 # RUN: not llvm-lipo %t-universal.o -replace i386 %t-32.o -replace i386 %t-32.o -o %t.o 2>&1 | FileCheck --check-prefix=ARCH_DUPLICATE %s
 # ARCH_DUPLICATE: error: -replace i386 <file_name> specified multiple times:

llvm/test/tools/llvm-lto/error.ll

-; RUN: not llvm-lto foobar 2>&1 | FileCheck %s
-; CHECK: llvm-lto: error loading file 'foobar': {{N|n}}o such file or directory
+; RUN: not llvm-lto foobar 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
+; CHECK: llvm-lto: error loading file 'foobar': [[MSG]]
 
 ; RUN: not llvm-lto --list-symbols-only %S/Inputs/empty.bc 2>&1 | FileCheck %s --check-prefix=CHECK-LIST
 ; CHECK-LIST: llvm-lto: error loading file '{{.*}}/Inputs/empty.bc': The file was not recognized as a valid object file
 
 ; RUN: not llvm-lto --list-dependent-libraries-only %S/Inputs/empty.bc 2>&1 | FileCheck %s --check-prefix=CHECK-LIBS
 ; CHECK-LIBS: llvm-lto: {{.*}}/Inputs/empty.bc: Could not read LTO input file: The file was not recognized as a valid object file
 
 ; RUN: not llvm-lto --print-macho-cpu-only %S/Inputs/empty.bc 2>&1 | FileCheck %s --check-prefix=CHECK-MACHO
 ; CHECK-MACHO: llvm-lto: error: The file was not recognized as a valid object file
 
 ; RUN: not llvm-lto --thinlto %S/Inputs/empty.bc 2>&1 | FileCheck %s --check-prefix=CHECK-THIN
 ; CHECK-THIN: llvm-lto: error loading file '{{.*}}/Inputs/empty.bc': file too small to contain bitcode header

llvm/test/tools/llvm-lto2/X86/stats-file-option.ll

 
 ; CHECK: {
 ; CHECK: "asm-printer.EmittedInsts":
 ; CHECK: }
 
 
 ; Try to save statistics to an invalid file.
 ; RUN: not llvm-lto2 run %t1.bc -o %t.o -r %t1.bc,patatino,px \
-; RUN:     -stats-file=%t2/foo.stats 2>&1 | FileCheck --check-prefix=ERROR %s
-; ERROR: LTO::run failed: {{[Nn]}}o such file or directory
+; RUN:     -stats-file=%t2/foo.stats 2>&1 | FileCheck -DMSG=%errc_ENOENT --check-prefix=ERROR %s
+; ERROR: LTO::run failed: [[MSG]]

llvm/test/tools/llvm-mc/basic.test

-# RUN: not llvm-mc %t.blah -o /dev/null 2>&1 | FileCheck --check-prefix=ENOENT %s
+# RUN: not llvm-mc %t.blah -o /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT --check-prefix=ENOENT %s
 
-# ENOENT: {{.*}}.blah: {{[Nn]}}o such file or directory
+# ENOENT: {{.*}}.blah: [[MSG]]

llvm/test/tools/llvm-mca/invalid_input_file_name.test

-# RUN: not llvm-mca %t.blah -o /dev/null 2>&1 | FileCheck --check-prefix=ENOENT %s
+# RUN: not llvm-mca %t.blah -o /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT --check-prefix=ENOENT %s
 
-# ENOENT: {{.*}}.blah: {{[Nn]}}o such file or directory
+# ENOENT: {{.*}}.blah: [[MSG]]

llvm/test/tools/llvm-ml/basic.test

-# RUN: not llvm-ml %t.blah.asm /Fo /dev/null 2>&1 | FileCheck --check-prefix=ENOENT %s
+# RUN: not llvm-ml %t.blah.asm /Fo /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT --check-prefix=ENOENT %s
 
-# ENOENT: {{.*}}.blah.asm: {{[Nn]}}o such file or directory
+# ENOENT: {{.*}}.blah.asm: [[MSG]]

llvm/test/tools/llvm-objcopy/COFF/add-section.test

 # CHECK-EXTENDED-RELOCS-NEXT:   IMAGE_SCN_CNT_INITIALIZED_DATA
 # CHECK-EXTENDED-RELOCS-NEXT: ]
 # CHECK-EXTENDED-RELOCS:      SectionData (
 # CHECK-EXTENDED-RELOCS-NEXT:  0000: {{.+}}|DEADBEEF{{.+}}|
 # CHECK-EXTENDED-RELOCS-NEXT: )
 
 ## Test that llvm-objcopy produces an error if the file with section contents
 ## to be added does not exist.
-# RUN: not llvm-objcopy --add-section=.another.section=%t2 %t %t3 2>&1 | FileCheck -DFILE1=%t -DFILE2=%t2 %s --check-prefixes=ERR1
+# RUN: not llvm-objcopy --add-section=.another.section=%t2 %t %t3 2>&1 | FileCheck -DFILE1=%t -DFILE2=%t2 -DMSG=%errc_ENOENT %s --check-prefixes=ERR1
 
-# ERR1: error: '[[FILE1]]': '[[FILE2]]': {{[Nn]}}o such file or directory
+# ERR1: error: '[[FILE1]]': '[[FILE2]]': [[MSG]]
 
 ## Another negative test for invalid --add-sections command line argument.
 # RUN: not llvm-objcopy --add-section=.another.section %t %t3 2>&1 | FileCheck %s --check-prefixes=ERR2
 
 # ERR2: error: bad format for --add-section: missing '='
 
 ## Negative test for invalid --add-sections argument - missing file name.
 # RUN: not llvm-objcopy --add-section=.section.name= %t %t3 2>&1 | FileCheck %s --check-prefixes=ERR3

llvm/test/tools/llvm-objcopy/ELF/add-section.test

 # CHECK: Name: .shstrtab
 # CHECK: Name: .test2
 # CHECK: SectionData (
 # CHECK-NEXT:   0000: DEADBEEF
 # CHECK-NEXT: )
 
 ## Test that llvm-objcopy produces an error if the file with section contents
 ## to be added does not exist.
-# RUN: not llvm-objcopy --add-section=.section.name=%t.missing %t %t.out 2>&1 | FileCheck -DFILE1=%t -DFILE2=%t.missing %s --check-prefixes=ERR1
+# RUN: not llvm-objcopy --add-section=.section.name=%t.missing %t %t.out 2>&1 | FileCheck -DFILE1=%t -DFILE2=%t.missing -DMSG=%errc_ENOENT %s --check-prefixes=ERR1
 
-# ERR1: error: '[[FILE1]]': '[[FILE2]]': {{[Nn]}}o such file or directory
+# ERR1: error: '[[FILE1]]': '[[FILE2]]': [[MSG]]
 
 ## Negative test for invalid --add-sections argument - missing '='.
 # RUN: not llvm-objcopy --add-section=.section.name %t %t.out 2>&1 | FileCheck %s --check-prefixes=ERR2
 
 # ERR2: error: bad format for --add-section: missing '='
 
 ## Negative test for invalid --add-sections argument - missing file name.
 # RUN: not llvm-objcopy --add-section=.section.name= %t %t.out 2>&1 | FileCheck %s --check-prefixes=ERR3
 
 # ERR3: error: bad format for --add-section: missing file name

llvm/test/tools/llvm-objcopy/ELF/error-format.test

 ## This test shows that we include the tool name in error/warning messages.
 
-# RUN: not llvm-objcopy %S/non-existent 2>&1 | FileCheck --check-prefix=ERR %s -DTOOL=objcopy
-# RUN: not llvm-strip %S/non-existent 2>&1 | FileCheck --check-prefix=ERR %s -DTOOL=strip
+# RUN: not llvm-objcopy %S/non-existent 2>&1 | FileCheck --check-prefix=ERR %s -DTOOL=objcopy -DMSG=%errc_ENOENT
+# RUN: not llvm-strip %S/non-existent 2>&1 | FileCheck --check-prefix=ERR %s -DTOOL=strip -DMSG=%errc_ENOENT
 
-# ERR: llvm-[[TOOL]]{{(\.exe)?}}: error: '{{.*}}': {{[Nn]}}o such file or directory
+# ERR: llvm-[[TOOL]]{{(\.exe)?}}: error: '{{.*}}': [[MSG]]
 
 ## Currently llvm-objcopy does not issue warnings, so it is not tested.
 
 # RUN: yaml2obj %s -o %t
 # RUN: llvm-strip %t %t 2>&1 | FileCheck --check-prefix=WARN %s
 
 # WARN: llvm-strip{{(\.exe)?}}: warning: '{{.*}}' was already specified
 
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
   Type:    ET_DYN
   Machine: EM_RISCV

llvm/test/tools/llvm-objcopy/MachO/add-section-error.test

 ## Test --add-section error messages.
 
 # RUN: yaml2obj %p/Inputs/x86_64.yaml -o %t
 # RUN: echo -n abcdefg > %t.data
 
 ## Error case 1: Nonexistent input file is specified by --add-section.
 # RUN: not llvm-objcopy --add-section __TEXT,__text=%t.missing %t %t.nonexistent-file 2>&1 \
-# RUN:   | FileCheck %s -DINPUT=%t -DSECTION_DATA_FILE=%t.missing --check-prefix=NONEXSITENT-FILE
-# NONEXSITENT-FILE: error: '[[INPUT]]': '[[SECTION_DATA_FILE]]': {{[Nn]}}o such file or directory
+# RUN:   | FileCheck %s -DINPUT=%t -DSECTION_DATA_FILE=%t.missing -DMSG=%errc_ENOENT --check-prefix=NONEXSITENT-FILE
+# NONEXSITENT-FILE: error: '[[INPUT]]': '[[SECTION_DATA_FILE]]': [[MSG]]
 
 ## Error case 2: Too long segment name.
 # RUN: not llvm-objcopy --add-section __TOOOOOOOOO_LONG,__text=%t.data %t %t.too-long-seg-name 2>&1 \
 # RUN:   | FileCheck %s -DINPUT=%t --check-prefix=TOO-LONG-SEG-NAME
 # TOO-LONG-SEG-NAME: error: '[[INPUT]]': too long segment name: '__TOOOOOOOOO_LONG'

llvm/test/tools/llvm-objcopy/redefine-symbols.test

 ## Test common parsing errors general to all supported binary formats.
 
 # RUN: yaml2obj %s -o %t
 
 # RUN: not llvm-objcopy --redefine-sym bar %t /dev/null 2>&1 | FileCheck %s --check-prefix=BAD-FORMAT
 # BAD-FORMAT: bad format for --redefine-sym
 
 # RUN: not llvm-objcopy --redefine-sym foo=f1 --redefine-sym foo=f2 %t /dev/null 2>&1 | FileCheck %s --check-prefix=MULTIPLE-REDEFINITION
 # MULTIPLE-REDEFINITION: multiple redefinition of symbol 'foo'
 
 # RUN: echo '  foo   oof #rename foo  ' > %t.rename.txt
 # RUN: echo 'bar' >> %t.rename.txt
 # RUN: not llvm-objcopy --redefine-syms %t.rename.txt %t /dev/null 2>&1 | FileCheck %s --check-prefix=MISSING-SYM-NAME
 # MISSING-SYM-NAME: error: {{.*}}.rename.txt:2: missing new symbol name
 
-# RUN: not llvm-objcopy --redefine-syms %t.rename-none.txt %t /dev/null 2>&1 | FileCheck %s --check-prefix=NO-FILE
-# NO-FILE: error: '{{.*}}.rename-none.txt': {{[Nn]}}o such file or directory
+# RUN: not llvm-objcopy --redefine-syms %t.rename-none.txt %t /dev/null 2>&1 | FileCheck -DMSG=%errc_ENOENT %s --check-prefix=NO-FILE
+# NO-FILE: error: '{{.*}}.rename-none.txt': [[MSG]]
 
 !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
   Type:    ET_EXEC
   Machine: EM_X86_64
 Sections:
   - Name:  .text
     Type:  SHT_PROGBITS
     Flags: [ SHF_ALLOC, SHF_EXECINSTR ]
 Symbols:
   - Name:    foo
     Section: .text
   - Name:    bar
     Section: .text

llvm/test/tools/llvm-objcopy/wasm/dump-section.test

 ## Test the contents of a custom section dumped from a binary.
 # RUN: yaml2obj %s -o %t
 # RUN: llvm-objcopy --dump-section=producers=%t.sec %t
 # RUN: od -t x1 %t.sec | FileCheck %s
 
 # RUN: not llvm-objcopy --dump-section=nonexistent=%t.sec %t 2>&1 | FileCheck --check-prefix=NONEXISTENT %s
-# RUN: not llvm-objcopy --dump-section=producers=%t.dir/bar %t 2>&1 | FileCheck --check-prefix=DIROUT %s
+# RUN: not llvm-objcopy --dump-section=producers=%t.dir/bar %t 2>&1 | FileCheck -DMSG=%errc_ENOENT --check-prefix=DIROUT %s
 
 ## Raw contents of the producers section.
 # CHECK: 0000000 01 0c 70 72 6f 63 65 73 73 65 64 2d 62 79 01 05
 # CHECK: 0000020 63 6c 61 6e 67 05 39 2e 30 2e 30
 
 # NONEXISTENT: section 'nonexistent' not found
-# DIROUT: error: {{.*}}/bar': {{[nN]}}o such file or directory
+# DIROUT: error: {{.*}}/bar': [[MSG]]
 
 ## Check that dumping and removing a section works in the same invocation
 # RUN: llvm-objcopy --dump-section=producers=%t.sec --remove-section=producers %t %t2
 # RUN: od -t x1 %t.sec | FileCheck %s
 # RUN: obj2yaml %t2 | FileCheck --check-prefix=REMOVED %s
 
 # REMOVED-NOT: producers
 

llvm/test/tools/llvm-profdata/weight-instr.test

 RUN: not llvm-profdata merge -instr -weighted-input=3,%p/Inputs/weight-instr-bar.profdata -weighted-input=0,%p/Inputs/weight-instr-foo.profdata -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 RUN: not llvm-profdata merge -instr -weighted-input=3,%p/Inputs/weight-instr-bar.profdata -weighted-input=0.75,%p/Inputs/weight-instr-foo.profdata -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 RUN: not llvm-profdata merge -instr -weighted-input=3,%p/Inputs/weight-instr-bar.profdata -weighted-input=-5,%p/Inputs/weight-instr-foo.profdata -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 RUN: not llvm-profdata merge -instr -weighted-input=3,%p/Inputs/weight-instr-bar.profdata -weighted-input=,%p/Inputs/weight-instr-foo.profdata -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 RUN: not llvm-profdata merge -instr -weighted-input=3,%p/Inputs/weight-instr-bar.profdata -weighted-input=%p/Inputs/weight-instr-foo.profdata -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 INVALID_WEIGHT: error: Input weight must be a positive integer.
 
 4- Bad merge: input path does not exist
-RUN: not llvm-profdata merge -instr -weighted-input=3,%p/Inputs/does-not-exist.profdata -weighted-input=2,%p/Inputs/does-not-exist-either.profdata -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_INPUT
-INVALID_INPUT: {{.*}}: {{.*}}does-not-exist.profdata: {{[Nn]}}o such file or directory
+RUN: not llvm-profdata merge -instr -weighted-input=3,%p/Inputs/does-not-exist.profdata -weighted-input=2,%p/Inputs/does-not-exist-either.profdata -o %t.out 2>&1 | FileCheck -DMSG=%errc_ENOENT %s -check-prefix=INVALID_INPUT
+INVALID_INPUT: {{.*}}: {{.*}}does-not-exist.profdata: [[MSG]]
 
 5- No inputs
 RUN: not llvm-profdata merge -instr -o %t.out 2>&1 | FileCheck %s -check-prefix=NO_INPUT
 NO_INPUT: {{.*}}: No input files specified. See llvm-profdata{{(\.EXE|\.exe)?}} merge -help

llvm/test/tools/llvm-profdata/weight-sample.test

 RUN: not llvm-profdata merge -sample -weighted-input=3,%p/Inputs/weight-sample-bar.proftext -weighted-input=0,%p/Inputs/weight-sample-foo.proftext -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 RUN: not llvm-profdata merge -sample -weighted-input=3,%p/Inputs/weight-sample-bar.proftext -weighted-input=0.75,%p/Inputs/weight-sample-foo.proftext -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 RUN: not llvm-profdata merge -sample -weighted-input=3,%p/Inputs/weight-sample-bar.proftext -weighted-input=-5,%p/Inputs/weight-sample-foo.proftext -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 RUN: not llvm-profdata merge -sample -weighted-input=3,%p/Inputs/weight-sample-bar.proftext -weighted-input=,%p/Inputs/weight-sample-foo.proftext -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 RUN: not llvm-profdata merge -sample -weighted-input=3,%p/Inputs/weight-sample-bar.proftext -weighted-input=%p/Inputs/weight-sample-foo.proftext -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_WEIGHT
 INVALID_WEIGHT: error: Input weight must be a positive integer.
 
 4- Bad merge: input path does not exist
-RUN: not llvm-profdata merge -sample -weighted-input=3,%p/Inputs/does-not-exist.proftext -weighted-input=2,%p/Inputs/does-not-exist-either.proftext -o %t.out 2>&1 | FileCheck %s -check-prefix=INVALID_INPUT
-INVALID_INPUT: {{.*}}: {{.*}}does-not-exist.proftext: {{[Nn]}}o such file or directory
+RUN: not llvm-profdata merge -sample -weighted-input=3,%p/Inputs/does-not-exist.proftext -weighted-input=2,%p/Inputs/does-not-exist-either.proftext -o %t.out 2>&1 | FileCheck -DMSG=%errc_ENOENT %s -check-prefix=INVALID_INPUT
+INVALID_INPUT: {{.*}}: {{.*}}does-not-exist.proftext: [[MSG]]
 
 5- No inputs
 RUN: not llvm-profdata merge -sample -o %t.out 2>&1 | FileCheck %s -check-prefix=NO_INPUT
 NO_INPUT: {{.*}}: No input files specified. See llvm-profdata{{(\.EXE|\.exe)?}} merge -help

llvm/test/tools/llvm-readobj/ELF/thin-archive-paths.test

 # RUN: rm -rf %t
 # RUN: mkdir -p %t/a/b
 # RUN: cd %t
 # RUN: yaml2obj %s -o a/b/1.o
 # The archive is deliberately in a different directory to the cwd, to show that
 # the member paths printed are exactly what's in the archive and not relative
 # to the cwd.
 # RUN: llvm-ar rcT a/relative.a a/b/1.o
 
 # Show that relative paths in the file header printing look sensible.
 # RUN: llvm-readobj --file-headers a/relative.a | FileCheck %s -DARFILE="a/relative.a" --check-prefix=REL
 # REL: File: [[ARFILE]](b/1.o)
 
 # Show that relative paths in an error message for both archive and member look
 # sensible.
 # RUN: rm a/b/1.o
-# RUN: not llvm-readobj --file-headers a/relative.a 2>&1 | FileCheck %s --check-prefix=ERR1
-# RUN: not llvm-readelf --file-headers a/relative.a 2>&1 | FileCheck %s --check-prefix=ERR1
-# ERR1: error: 'a/relative.a': 'b/1.o': {{[Nn]}}o such file or directory
+# RUN: not llvm-readobj --file-headers a/relative.a 2>&1 | FileCheck %s --check-prefix=ERR1 -DMSG=%errc_ENOENT
+# RUN: not llvm-readelf --file-headers a/relative.a 2>&1 | FileCheck %s --check-prefix=ERR1 -DMSG=%errc_ENOENT
+# ERR1: error: 'a/relative.a': 'b/1.o': [[MSG]]
 
 # RUN: mkdir -p %t/c
 # RUN: yaml2obj %s -o a/b/1.o
 # RUN: llvm-ar rcT c/absolute.a %t/a/b/1.o
 
 # Show that absolute paths in the file header printing are correct.
 # RUN: llvm-readobj --file-headers c/absolute.a | FileCheck %s --check-prefix=ABS -DARFILE="c/absolute.a" -DDIR=%/t
 # ABS: File: [[ARFILE]]([[DIR]]/a/b/1.o)
 
 # Show that absolute paths in an error message for both archive and member are correct.
 # RUN: rm a/b/1.o
-# RUN: not llvm-readobj --file-headers %/t/c/absolute.a 2>&1 | FileCheck %s --check-prefix=ERR2 -DDIR=%/t
-# RUN: not llvm-readelf --file-headers %/t/c/absolute.a 2>&1 | FileCheck %s --check-prefix=ERR2 -DDIR=%/t
-# ERR2: error: '[[DIR]]/c/absolute.a': '[[DIR]]/a/b/1.o': {{[Nn]}}o such file or directory
+# RUN: not llvm-readobj --file-headers %/t/c/absolute.a 2>&1 | FileCheck %s --check-prefix=ERR2 -DDIR=%/t -DMSG=%errc_ENOENT
+# RUN: not llvm-readelf --file-headers %/t/c/absolute.a 2>&1 | FileCheck %s --check-prefix=ERR2 -DDIR=%/t -DMSG=%errc_ENOENT
+# ERR2: error: '[[DIR]]/c/absolute.a': '[[DIR]]/a/b/1.o': [[MSG]]
 
 --- !ELF
 FileHeader:
   Class: ELFCLASS32
   Data:  ELFDATA2LSB
   Type:  ET_DYN

llvm/test/tools/llvm-readobj/basic.test

 ## Test case where input file does not exit.
-RUN: not llvm-readobj %t.blah 2>&1 | FileCheck --check-prefix=ENOENT -DTOOL=readobj %s
-RUN: not llvm-readelf %t.blah 2>&1 | FileCheck --check-prefix=ENOENT -DTOOL=readelf %s
+RUN: not llvm-readobj %t.blah 2>&1 | FileCheck --check-prefix=ENOENT -DTOOL=readobj -DMSG=%errc_ENOENT %s
+RUN: not llvm-readelf %t.blah 2>&1 | FileCheck --check-prefix=ENOENT -DTOOL=readelf -DMSG=%errc_ENOENT %s
 
-ENOENT: llvm-[[TOOL]]{{(\.exe)?}}: error: '{{.*}}.blah': {{[Nn]}}o such file or directory
+ENOENT: llvm-[[TOOL]]{{(\.exe)?}}: error: '{{.*}}.blah': [[MSG]]
 
 # Test case with no input file.
 RUN: not llvm-readobj 2>&1 | FileCheck %s --check-prefix=NO-FILE
 RUN: not llvm-readelf 2>&1  | FileCheck %s --check-prefix=NO-FILE
 NO-FILE: error: no input files specified
 
 # Test case where input file is too small to be a recognised object file.
 RUN: touch %t.empty

llvm/test/tools/llvm-readobj/thin-archive.test

 # RESIZED: File: {{.*}}1.o
 # RESIZED: Format: elf64-x86-64
 # RESIZED: File: {{.*}}2.o
 # RESIZED: Format: elf64-x86-64
 
 # Remove the second member and show that the first can still be dumped, but that the last isn't.
 # RUN: rm %t/2.o
 # RUN: not llvm-readobj --file-headers %t.a 2> %t.err | FileCheck %s --check-prefix=MISSING
-# RUN: FileCheck %s --check-prefix=ERR --input-file=%t.err
+# RUN: FileCheck -DMSG=%errc_ENOENT %s --check-prefix=ERR --input-file=%t.err
 
 # MISSING:     File: {{.*}}1.o
 # MISSING:     Format: elf64-x86-64
 # MISSING-NOT: File: {{.*}}3.o
-# ERR: error: '{{.*}}.a': '{{.*}}2.o': {{[Nn]}}o such file or directory
+# ERR: error: '{{.*}}.a': '{{.*}}2.o': [[MSG]]
 
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
   Type:    ET_REL
   Machine: EM_X86_64
 

llvm/test/tools/llvm-size/no-input.test

 ## Show that llvm-size emits an error if passed in a non-existent file.
 
-# RUN: not llvm-size %t.blah 2>&1 | FileCheck %s -DFILE=%t.blah --check-prefix=ENOENT
-# ENOENT: {{.*}}llvm-size{{.*}}: error: '[[FILE]]': {{[Nn]}}o such file or directory
+# RUN: not llvm-size %t.blah 2>&1 | FileCheck %s -DFILE=%t.blah -DMSG=%errc_ENOENT --check-prefix=ENOENT
+# ENOENT: {{.*}}llvm-size{{.*}}: error: '[[FILE]]': [[MSG]]
 
 ## Show that llvm-size reads a.out if not passed any file.
 
 # RUN: rm -rf %t && mkdir -p %t
 # RUN: cd %t
 # RUN: yaml2obj %s -o a.out
 # RUN: llvm-size 2>&1 | FileCheck %s -DFILE=a.out
 

llvm/test/tools/llvm-symbolizer/pdb/missing_pdb.test

 RUN: llvm-symbolizer 0x401000 0x401001 -obj="%p/Inputs/missing_pdb.exe" 2>%t.err \
 RUN:    | FileCheck %s
-RUN: FileCheck --check-prefix=ERROR %s < %t.err
+RUN: FileCheck -DMSG=%errc_ENOENT --check-prefix=ERROR %s < %t.err
 
 llvm-symbolizer should print one error and two unknown line info records.
 
-ERROR: LLVMSymbolizer: error reading file: {{.*}}: {{N|n}}o such file or directory
+ERROR: LLVMSymbolizer: error reading file: {{.*}}: [[MSG]]
 ERROR-NOT: error reading file
 
 CHECK: ??
 CHECK: ??:0:0
 CHECK: ??
 CHECK: ??:0:0

llvm/test/tools/llvm-xray/X86/no-such-file.txt

-; RUN: not llvm-xray extract no-such-file 2>&1 | FileCheck %s
+; RUN: not llvm-xray extract no-such-file 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
 
 ; CHECK: llvm-xray: Cannot extract instrumentation map from 'no-such-file'.
-; CHECK-NEXT: {{[Nn]}}o such file or directory
+; CHECK-NEXT: [[MSG]]

llvm/test/tools/obj2yaml/invalid_input_file.test

-# RUN: not obj2yaml  %p/path/does/not/exist 2>&1 | FileCheck %s
+# RUN: not obj2yaml  %p/path/does/not/exist 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
 
-# CHECK: Error reading file: {{.*}}/path/does/not/exist: {{[Nn]}}o such file or directory
+# CHECK: Error reading file: {{.*}}/path/does/not/exist: [[MSG]]

llvm/test/tools/yaml2obj/output-file.yaml

 ## Test that -o sets the output file name.
 
 # RUN: rm -f %t
 # RUN: yaml2obj %s -o %t
 # RUN: ls %t
 # RUN: rm -f %t
 # RUN: yaml2obj %s -o%t
 # RUN: ls %t
 
-# RUN: not yaml2obj -o %p/path/does/not/exist 2>&1 | FileCheck %s
+# RUN: not yaml2obj -o %p/path/does/not/exist 2>&1 | FileCheck -DMSG=%errc_ENOENT %s
 
-# CHECK: yaml2obj: error: failed to open '{{.*}}/path/does/not/exist': {{[Nn]}}o such file or directory
+# CHECK: yaml2obj: error: failed to open '{{.*}}/path/does/not/exist': [[MSG]]
 
 !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2LSB
   Type:    ET_REL
   Machine: EM_NONE

llvm/utils/lit/lit/llvm/config.py

def add_tool_substitutions(self, tools, search_dirs=None):
             if not command:
                 return False
 
             substitutions.append((subst_key, tool_pipe + command))
 
         self.config.substitutions.extend(substitutions)
         return True
 
+    def add_err_msg_substitutions(self):
+        if (sys.platform == 'zos'):
+            self.config.substitutions.append(('%errc_ENOENT', '\'EDC5129I No such file or directory.\''))
+            self.config.substitutions.append(('%errc_EISDIR', '\'EDC5123I Is a directory.\''))
+        elif (sys.platform == 'win32'):
+            self.config.substitutions.append(('%errc_ENOENT', '\'no such file or directory\''))
+            self.config.substitutions.append(('%errc_EISDIR', '\'is a directory\''))
+        else:
+            self.config.substitutions.append(('%errc_ENOENT', '\'No such file or directory\''))
+            self.config.substitutions.append(('%errc_EISDIR', '\'Is a directory\''))
+
     def use_default_substitutions(self):
         tool_patterns = [
             ToolSubst('FileCheck', unresolved='fatal'),
             # Handle these specially as they are strings searched for during testing.
             ToolSubst(r'\| \bcount\b', command=FindTool(
                 'count'), verbatim=True, unresolved='fatal'),
             ToolSubst(r'\| \bnot\b', command=FindTool('not'), verbatim=True, unresolved='fatal')]
 
         self.config.substitutions.append(('%python', '"%s"' % (sys.executable)))
 
         self.add_tool_substitutions(
             tool_patterns, [self.config.llvm_tools_dir])
 
+        self.add_err_msg_substitutions()
+
     def use_llvm_tool(self, name, search_env=None, required=False, quiet=False):
         """Find the executable program 'name', optionally using the specified
         environment variable as an override before searching the
         configuration's PATH."""
         # If the override is specified in the environment, use it without
         # validation.
         if search_env:
             tool = self.config.environment.get(search_env)
             if tool:
                 return tool