Set rpath on openmp executables

[JonChesterfield]

Summary

Openmp executables need to find libomp and libomptarget at runtime.
This currently requires LD_LIBRARY_PATH or the user to specify rpath. Change
that to set the expected location of the openmp libraries in the install tree.

Whether rpath means rpath or runpath is system dependent. The attached test
shows that the Wl,--disable-new-dtags control interacts correctly with this feature.

The implicit rpath field is appended to any user specified ones which is ideal.

[JonChesterfield]

Testing manually looks good, provided there's no command line argument involved. -rpath even composes sanely, so doing this plus passing -Wl,rpath= results in multiple embedded rpaths. At a loss as to why changing Options.td is working so poorly, though it may be relevant that ninja thinks there's no work to do when the file changes. Trying things like:

def fno_openmp_add_rpath: Flag<["-"], "fno-openmp-add-rpath">, Flags<[NoArgumentUnused]>,
  HelpText<"Do not add -rpath with architecture-specific resource directory to the linker flags">;

which is a direct copy of fno_rtlib_add_path, didn't even bother changing the help text, and it's still missing from the clang executable.

[jdoerfert]

Can we test this somehow?

[JonChesterfield]

My plan is to feed executables to readelf -d. Commandline problems seem to be solvable by clean builds between changes, slow but feasible. Might upset the incremental buildbot

[JonChesterfield]

• Now with commandline argument

[JonChesterfield]

Slightly stalled on testing - I'd like to emit the object and feed it to readelf, something like:
// RUN: %clang %s -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK

which errors with cannot find -lomp. I feel there should be a linker flag to the effect of "ignore libraries you can't find", which I have failed to identify. Candidate workarounds are putting empty files called libomp.a in Inputs, changing the control flow in addOpenMPRuntime or finding said linker flag.

Toolchain style testing (like Driver/amdgpu-openmp-toolchain) is sufficient to show rpath is inserted, e.g. "-lomp" "-lomptarget" "-rpath" "*/llvm/lib", but doesn't let one check interaction with user provided rpaths. I'll pick this up over the weekend.

Poking it manually works as one would wish so the functionality looks fine. Please speak up in the meantime if you'd like the option to be named or documented differently, or if there's any other change desired for the non-testing part.

[JonChesterfield]

• set rpath after libomptarget, reads better than between the two -l flags

[JonChesterfield]

• Test rpath and runpath setting

[JonChesterfield]

• composes with user specified rpath flags

[JonChesterfield]

• Disable implicit rpath for the tests so we can be certain the tests aren't picking up unrelated libs

[JonChesterfield]

• rebase and format

[JonChesterfield]

clang/test/OpenMP/implicit_rpath.c

29

This ^ probably has path separator issues on windows, will try to find what the proper regex is for that

[jhuber6]

clang/test/OpenMP/implicit_rpath.c

29

https://llvm.org/docs/CommandGuide/lit.html#substitutions I think it's one of these.

[jhuber6]

LGTM, unless someone else has reservations.

[JonChesterfield]

Yep. I'm not totally confident what windows builds will embed in the dynamic table - might be windows style \ or it might be linux style /. Or it might be \\, which can be matched as in D109057. I think I'm going with the very conservative llvm{{*.}}lib pattern, as provided rpath/runpath has been written in the binary it probably points to the llvm dir in platform-specific fashion.

[JonChesterfield]

• More paranoid regex - not sure what windows uses as the path separator in the dynamic table

[JonChesterfield]

Apologies for the noise, turns out 'llvm/lib' and 'llvm/lib64' are not the only two options. Made it more permissive and reapplied. Also looks like ppc64le defaults to rpath as opposed to runpath, so that's worth knowing.

[JonChesterfield]

This test doesn't work on hexagon because hexagon doesn't have a linker (??), emailed @kparzysz asking how to disable tests on hexagon as I can't find a likely looking UNSUPPORTED string using grep.

[thakis]

looks like the mac linker doesn't like this test either: http://45.33.8.238/macm1/26834/step_7.txt

[JonChesterfield]

In D118493#3284617, @thakis wrote:

looks like the mac linker doesn't like this test either: http://45.33.8.238/macm1/26834/step_7.txt

Error message is ld: file too small (length=8) file '/Users/thakis/src/llvm-project/clang/test/OpenMP/Inputs/libomp.a' for architecture arm64

On the face of it I'd say that's a bug in your linker. The archive file is the magic !<arch>\n string by itself which is well formed to the extent the archive format is documented. It's definitely possible to put more text in the stub file to work around though it's hard to tell how much text, or whether your linker will do other validation like seeing whether the symbol table is present or objects actually contain machine code.

I'm going to delete the test file in the meantime.
edit: or less aggressive, is there an UNSUPPORTED flag I can add that will drop it from your CI?

[thakis]

In D118493#3284663, @JonChesterfield wrote:

In D118493#3284617, @thakis wrote:

looks like the mac linker doesn't like this test either: http://45.33.8.238/macm1/26834/step_7.txt

Error message is ld: file too small (length=8) file '/Users/thakis/src/llvm-project/clang/test/OpenMP/Inputs/libomp.a' for architecture arm64

On the face of it I'd say that's a bug in your linker.

It's just the standard Xcode linker, I didn't write it :)

The archive file is the magic !<arch>\n string by itself which is well formed to the extent the archive format is documented. It's definitely possible to put more text in the stub file to work around though it's hard to tell how much text, or whether your linker will do other validation like seeing whether the symbol table is present or objects actually contain machine code.

I'm going to delete the test file in the meantime.
edit: or less aggressive, is there an UNSUPPORTED flag I can add that will drop it from your CI?

Maybe you can pass an explicit --target triple to clang, and just check -### instead of actually linking, like driver tests usually do?

[JonChesterfield]

In D118493#3284781, @thakis wrote:

In D118493#3284663, @JonChesterfield wrote:

In D118493#3284617, @thakis wrote:

looks like the mac linker doesn't like this test either: http://45.33.8.238/macm1/26834/step_7.txt

Error message is ld: file too small (length=8) file '/Users/thakis/src/llvm-project/clang/test/OpenMP/Inputs/libomp.a' for architecture arm64

On the face of it I'd say that's a bug in your linker.

It's just the standard Xcode linker, I didn't write it :)

The archive file is the magic !<arch>\n string by itself which is well formed to the extent the archive format is documented. It's definitely possible to put more text in the stub file to work around though it's hard to tell how much text, or whether your linker will do other validation like seeing whether the symbol table is present or objects actually contain machine code.

I'm going to delete the test file in the meantime.
edit: or less aggressive, is there an UNSUPPORTED flag I can add that will drop it from your CI?

Maybe you can pass an explicit --target triple to clang, and just check -### instead of actually linking, like driver tests usually do?

The point of the test is checking that the weirdness of rpath vs runpath combined with user specified flags works out properly after going through the linker so that would lose most (possibly all?) of the coverage.

The test may be more bother than it's worth though. I'll drop it for now. d14897c7dad8c05c003bada019c0b573ace63a1a

[thakis]

The behavior change can still be observed by a -### test -- just check that -rpath gets passed to the linker.

Clang tests generally can't rely on running linkers. Linkers tend to be platform-specific. Windows link.exe can't produce elf binaries, macOS's ld64 can't produce elf binaries, etc. That's why driver tests are all -###-based. (That, and we try to do more unit testing and less integration testing.)

[tstellar]

Fedora recently banned RUNPATH, so packages built with clang and -fopenmp are rejected by the build system with this change. Can you elaborate more about the intended use case that this change will fix? Is it just meant for when libomp.so is installed outside of /usr/lib64 ?

[JonChesterfield]

It lets applications run with the libomp and libomptarget built with the toolchain. For users, they don't have to be root. For compiler devs, we test our work instead of whatever the distro had installed.

There's no info at that link. What's 'broken rpath'? If they mean runpath, then it's really on fedora to have Wl,--rpath set rpath instead of runpath. Setting the default in the linker to the one your own build tools reject will break everything, not just llvm openmp.

Edit: https://fedoraproject.org/wiki/Changes/Broken_RPATH_will_fail_rpmbuild
Doesn't mean runpath. There's a list of strings Fedora (IBM?) have decided they don't like, named 'broken' or 'invalid', and declared shall not be present. Page doesn't go into the rpath/runpath distinction, maybe they mean for the string restrictions to apply to both.

Can't tell which rule we're violating (will need to work out what the install directory resolves to on fedora), though the one about executing untrusted code seems likely, where untrusted seems to be anything not owned by root. E.g. applications people have written themselves.

We can introduce a cmake flag that changes the default on this commandline argument, set when building on Fedora (and maybe redhat, centos-stream, others). That'll annoy anyone developing openmp on, or using a local llvm build on, Fedora but that might be zero people and they could override the cmake flag and disable the build checking scripts.

Or disable building openmp on Fedora, that seems reasonable too.

[JonChesterfield]

@estewart08 Fedora are rejecting some uses of rpath (and probably runpath). I can't find a corresponding page for redhat. This may become a problem for our aomp/rocm builds.

[tstellar]

In D118493#3398808, @JonChesterfield wrote:

It lets applications run with the libomp and libomptarget built with the toolchain. For users, they don't have to be root. For compiler devs, we test our work instead of whatever the distro had installed.

There's no info at that link. What's 'broken rpath'? If they mean runpath, then it's really on fedora to have Wl,--rpath set rpath instead of runpath. Setting the default in the linker to the one your own build tools reject will break everything, not just llvm openmp.

Edit: https://fedoraproject.org/wiki/Changes/Broken_RPATH_will_fail_rpmbuild
Doesn't mean runpath. There's a list of strings Fedora (IBM?) have decided they don't like, named 'broken' or 'invalid', and declared shall not be present. Page doesn't go into the rpath/runpath distinction, maybe they mean for the string restrictions to apply to both.

I'm not sure I follow this, but there is a script run after every RPM built for fedora that checks for RUNPATH (among other things in the binary). I have been building packages for Fedora with clang14 and the script fails for packages built with clang + -fopenmp

Can't tell which rule we're violating (will need to work out what the install directory resolves to on fedora), though the one about executing untrusted code seems likely, where untrusted seems to be anything not owned by root. E.g. applications people have written themselves.

The rule that is broken is "standard RPATHs" Fedora installs libomp to /usr/lib64.

We can introduce a cmake flag that changes the default on this commandline argument, set when building on Fedora (and maybe redhat, centos-stream, others). That'll annoy anyone developing openmp on, or using a local llvm build on, Fedora but that might be zero people and they could override the cmake flag and disable the build checking scripts.

I don't really like adding CMake options for things like this. I think what we'll do in Fedora is just add -fno-openmp-implicit-rpath to the default CFLAGS so we can get the old behavior for official RPMs.

Or disable building openmp on Fedora, that seems reasonable too.

We want to be able to build packages that use OpenMP, so disabling is not really an option.

[JonChesterfield]

In D118493#3401448, @tstellar wrote:

The rule that is broken is "standard RPATHs" Fedora installs libomp to /usr/lib64.

...

I think what we'll do in Fedora is just add -fno-openmp-implicit-rpath to the default CFLAGS so we can get the old behavior for official RPMs.

Sounds good to me. I especially like the rapid unblocking.

Slightly longer term, we might want to adjust clang to notice when the rpath it is about to set coincides with a system directory. The purpose of this patch is to have clang -fopenmp -o a.out && ./a.out work without setting environment variables, but where the openmp libs in question are on the system search path the rpath is not necessary.

[MaskRay]

Enabled-by-default -fopenmp-implicit-rpath is odd. For other shared objects (e.g. libclang_rt.*.so) we don't do this.
For libraries in a well-known system directory, we just rely on the default /etc/ld.so.conf*.
For libraries in other directories or when --sysroot= is specified, the user is responsible to set up it correctly.
An opt-out behavior causes follow-up workarounds like D142174.

(I am on a trip and don't check emails frequently.)

[JonChesterfield]

That works if you have one toolchain installed globally or you are happy to cobble together a working system using environment variables. If you have multiple toolchains, they can't all sit in the global directory. If you don't have root, you can't install them there.

Previously we required people to set LD_LIBRARY_PATH to use openmp at all. That's an inherently poor UX and interacted especially poorly with module setup systems found in HPC.

We could make this opt-in, at the cost of new users seeing that openmp just doesn't work out of the box and other ones having to modify build scripts. In exchange we get - consistency with other tools that fail to work without global installation or extra user burden.

In my opinion dynamically linking language runtimes is a bad default. It wins us an awful lot of failure modes. I've also argued against being able to swap out individual components via environment variables as way too error prone for a user facing interface. This is what the community consensus went for. The combination of dynamically substitutable and requiring LD_LIBRARY_PATH was especially sharp.

[MaskRay]

In D118493#4075970, @JonChesterfield wrote:

That works if you have one toolchain installed globally or you are happy to cobble together a working system using environment variables. If you have multiple toolchains, they can't all sit in the global directory. If you don't have root, you can't install them there.

Previously we required people to set LD_LIBRARY_PATH to use openmp at all. That's an inherently poor UX and interacted especially poorly with module setup systems found in HPC.

We could make this opt-in, at the cost of new users seeing that openmp just doesn't work out of the box and other ones having to modify build scripts. In exchange we get - consistency with other tools that fail to work without global installation or extra user burden.

Well, if you have a toolchain at a non-standard location (I do this a lot myself), you can go an extra mile by specifying -Wl,-rpath=... in a Clang configuration file.
Gentoo has a nice summary of the feature https://blogs.gentoo.org/mgorny/2022/10/07/clang-in-gentoo-now-sets-default-runtimes-via-config-file/
libc++, libc++abi, libunwind, and compiler-rt don't add the extra DT_RUNPATH. Some build systems want to handle DT_RUNPATH themselves (e.g. CMAKE_INSTALL_RPATH).
I don't think it is a good idea for openmp to diverge and introduce an opt-out -fno-openmp-implicit-rpath.

In my opinion dynamically linking language runtimes is a bad default. It wins us an awful lot of failure modes. I've also argued against being able to swap out individual components via environment variables as way too error prone for a user facing interface. This is what the community consensus went for. The combination of dynamically substitutable and requiring LD_LIBRARY_PATH was especially sharp.

Well, some distributions have policies against DT_RUNPATH and this change is causing trouble. While a patch like D142174 can mitigate the problem, the hard-coded paths are ugly and may not work for another Linux distribution or another ELF-based operating system.

You don't necessarily use LD_LIBRARY_PATH. You can use -Wl,-rpath=...

[JonChesterfield]

Reporting after another round of discussion. I don't have much support from the llvm openmp working group that we should maintain the current divergence from libc++ and the like so I'm backing down. Let's revert this and regress for all users who don't install globally.

The intent is to document in the FAQ things users can do themselves to replace the functionality here, where possibly one enumerated option is to pass --fopenmp-implicit-rpath (maybe renamed) though my preference would be to delete this code path from the compiler entirely if it's going to be a minimally-tested, non-default path (as it'll break and we won't notice).

@MaskRay please revert this patch at your leisure. I'm reluctantly consenting to the capability regression but not volunteering to fix any CI fallout from removing it.

@ronlieb I think we should keep this functionality in rocm so that rocm built binaries don't start going looking for llvm libs under /usr instead and fall over

[MaskRay]

In D118493#4178250, @JonChesterfield wrote:

Reporting after another round of discussion. I don't have much support from the llvm openmp working group that we should maintain the current divergence from libc++ and the like so I'm backing down. Let's revert this and regress for all users who don't install globally.

The intent is to document in the FAQ things users can do themselves to replace the functionality here, where possibly one enumerated option is to pass --fopenmp-implicit-rpath (maybe renamed) though my preference would be to delete this code path from the compiler entirely if it's going to be a minimally-tested, non-default path (as it'll break and we won't notice).

@MaskRay please revert this patch at your leisure. I'm reluctantly consenting to the capability regression but not volunteering to fix any CI fallout from removing it.

@ronlieb I think we should keep this functionality in rocm so that rocm built binaries don't start going looking for llvm libs under /usr instead and fall over

Thank you! I am preparing a commit to remove this then...

[JonChesterfield]

Duplicating a comment from the commit thread so it's easier for me to find later.

You've applied this to the release branches going back as far as 14. It's a user facing breaking change. As in people who have a working openmp toolchain and update to the point release which looks like a semantic versioning thing indicating minor bugfixes will experience immediate cessation of function. I consider that very poor user experience and do not agree with the scope of breakage.

[tstellar]

In D118493#4182583, @JonChesterfield wrote:

Duplicating a comment from the commit thread so it's easier for me to find later.

You've applied this to the release branches going back as far as 14. It's a user facing breaking change. As in people who have a working openmp toolchain and update to the point release which looks like a semantic versioning thing indicating minor bugfixes will experience immediate cessation of function. I consider that very poor user experience and do not agree with the scope of breakage.

I don't see this revert in any of the official release branches. Am I missing something?

[MaskRay]

In D118493#4182583, @JonChesterfield wrote:

Duplicating a comment from the commit thread so it's easier for me to find later.

You've applied this to the release branches going back as far as 14. It's a user facing breaking change. As in people who have a working openmp toolchain and update to the point release which looks like a semantic versioning thing indicating minor bugfixes will experience immediate cessation of function. I consider that very poor user experience and do not agree with the scope of breakage.

555b572e3f407ac48b5f30fc06760cc4d0549977 is only in the main branch, not in release/14.x, release/15.x, or releast/16.x.

[JonChesterfield]

That is great news, phabricator's list of branches at https://reviews.llvm.org/rG9b9d08111b618d74574ba03e5cc3d752ecc56f55 has mislead me. Thanks!

clang/include/clang/Driver/Options.td

def resource_dir_EQ : Joined<["-"], "resource-dir=">, Flags<[NoXarchOption, CoreOption]>,
   Alias<resource_dir>;
 def rpath : Separate<["-"], "rpath">, Flags<[LinkerInput]>, Group<Link_Group>;
 def rtlib_EQ : Joined<["-", "--"], "rtlib=">,
   HelpText<"Compiler runtime library to use">;
 def frtlib_add_rpath: Flag<["-"], "frtlib-add-rpath">, Flags<[NoArgumentUnused]>,
   HelpText<"Add -rpath with architecture-specific resource directory to the linker flags">;
 def fno_rtlib_add_rpath: Flag<["-"], "fno-rtlib-add-rpath">, Flags<[NoArgumentUnused]>,
   HelpText<"Do not add -rpath with architecture-specific resource directory to the linker flags">;
+defm openmp_implicit_rpath: BoolFOption<"openmp-implicit-rpath",
+  LangOpts<"OpenMP">,
+  DefaultTrue,
+  PosFlag<SetTrue, [], "Set rpath on OpenMP executables">,
+  NegFlag<SetFalse>>;
 def r : Flag<["-"], "r">, Flags<[LinkerInput,NoArgumentUnused]>,
         Group<Link_Group>;
 def save_temps_EQ : Joined<["-", "--"], "save-temps=">, Flags<[CC1Option, NoXarchOption]>,
   HelpText<"Save intermediate compilation results.">;
 def save_temps : Flag<["-", "--"], "save-temps">, Flags<[NoXarchOption]>,
   Alias<save_temps_EQ>, AliasArgs<["cwd"]>,
   HelpText<"Save intermediate compilation results">;
 def save_stats_EQ : Joined<["-", "--"], "save-stats=">, Flags<[NoXarchOption]>,

clang/lib/Driver/ToolChains/CommonArgs.h

 
 unsigned ParseDebugDefaultVersion(const ToolChain &TC,
                                   const llvm::opt::ArgList &Args);
 
 void AddAssemblerKPIC(const ToolChain &ToolChain,
                       const llvm::opt::ArgList &Args,
                       llvm::opt::ArgStringList &CmdArgs);
 
+void addOpenMPRuntimeSpecificRPath(const ToolChain &TC,
+                                   const llvm::opt::ArgList &Args,
+                                   llvm::opt::ArgStringList &CmdArgs);
 void addArchSpecificRPath(const ToolChain &TC, const llvm::opt::ArgList &Args,
                           llvm::opt::ArgStringList &CmdArgs);
 /// Returns true, if an OpenMP runtime has been added.
 bool addOpenMPRuntime(llvm::opt::ArgStringList &CmdArgs, const ToolChain &TC,
                       const llvm::opt::ArgList &Args,
                       bool ForceStaticHostRuntime = false,
                       bool IsOffloadingHost = false, bool GompNeedsRT = false);
 

clang/lib/Driver/ToolChains/CommonArgs.cpp

#endif
 
   // Handle remarks hotness/threshold related options.
   renderRemarksHotnessOptions(Args, CmdArgs);
 
   addMachineOutlinerArgs(D, Args, CmdArgs, ToolChain.getEffectiveTriple(),
                          /*IsLTO=*/true);
 }
 
+void tools::addOpenMPRuntimeSpecificRPath(const ToolChain &TC,
+                                          const ArgList &Args,
+                                          ArgStringList &CmdArgs) {
+
+  if (Args.hasFlag(options::OPT_fopenmp_implicit_rpath,
+                   options::OPT_fno_openmp_implicit_rpath, true)) {
+    // Default to clang lib / lib64 folder, i.e. the same location as device
+    // runtime
+    SmallString<256> DefaultLibPath =
+        llvm::sys::path::parent_path(TC.getDriver().Dir);
+    llvm::sys::path::append(DefaultLibPath, Twine("lib") + CLANG_LIBDIR_SUFFIX);
+    CmdArgs.push_back("-rpath");
+    CmdArgs.push_back(Args.MakeArgString(DefaultLibPath));
+  }
+}
+
 void tools::addArchSpecificRPath(const ToolChain &TC, const ArgList &Args,
                                  ArgStringList &CmdArgs) {
   // Enable -frtlib-add-rpath by default for the case of VE.
   const bool IsVE = TC.getTriple().isVE();
   bool DefaultValue = IsVE;
   if (!Args.hasFlag(options::OPT_frtlib_add_rpath,
                     options::OPT_fno_rtlib_add_rpath, DefaultValue))
     return;
bool tools::addOpenMPRuntime(ArgStringList &CmdArgs, const ToolChain &TC,
   if (RTKind == Driver::OMPRT_GOMP && GompNeedsRT)
       CmdArgs.push_back("-lrt");
 
   if (IsOffloadingHost)
     CmdArgs.push_back("-lomptarget");
 
   addArchSpecificRPath(TC, Args, CmdArgs);
 
+  if (RTKind == Driver::OMPRT_OMP)
+    addOpenMPRuntimeSpecificRPath(TC, Args, CmdArgs);
+
   return true;
 }
 
 static void addSanitizerRuntime(const ToolChain &TC, const ArgList &Args,
                                 ArgStringList &CmdArgs, StringRef Sanitizer,
                                 bool IsShared, bool IsWhole) {
   // Wrap any static runtimes that must be forced into executable in
   // whole-archive.

clang/test/OpenMP/Inputs/libomp.a

+!<arch>

clang/test/OpenMP/implicit_rpath.c

+// RUN: %clang -fopenmp %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-DEFAULT
+// RUN: %clang -fopenmp -fopenmp-implicit-rpath %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-EXPLICIT
+// RUN: %clang -fopenmp -fno-openmp-implicit-rpath %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-DISABLED
+
+// RUN: %clang -fopenmp -Wl,--disable-new-dtags %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-DEFAULT-RPATH
+// RUN: %clang -fopenmp -fopenmp-implicit-rpath -Wl,--disable-new-dtags %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-EXPLICIT-RPATH
+// RUN: %clang -fopenmp -fno-openmp-implicit-rpath -Wl,--disable-new-dtags %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-DISABLED-RPATH
+
+// RUN: %clang -fopenmp -Wl,--enable-new-dtags %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-DEFAULT-RUNPATH
+// RUN: %clang -fopenmp -fopenmp-implicit-rpath -Wl,--enable-new-dtags %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-EXPLICIT-RUNPATH
+// RUN: %clang -fopenmp -fno-openmp-implicit-rpath -Wl,--enable-new-dtags %s -L%S/Inputs -o %t && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-DISABLED-RUNPATH
+
+// RUN: %clang -Wl,-rpath=early -fopenmp %s -L%S/Inputs -o %t -Wl,-rpath=late && llvm-readelf --dynamic-table %t | FileCheck %s --check-prefixes=CHECK-COMPOSABLE
+
+// CHECK-DEFAULT:      ({{R|RUN}}PATH) Library {{r|run}}path: [{{.*}}llvm{{.*}}lib{{.*}}]
+// CHECK-EXPLICIT:     ({{R|RUN}}PATH) Library {{r|run}}path: [{{.*}}llvm{{.*}}lib{{.*}}]
+// CHECK-DISABLED-NOT: ({{R|RUN}}PATH)
+
+// CHECK-DEFAULT-RPATH:      (RPATH) Library rpath: [{{.*}}llvm{{.*}}lib{{.*}}]
+// CHECK-EXPLICIT-RPATH:     (RPATH) Library rpath: [{{.*}}llvm{{.*}}lib{{.*}}]
+// CHECK-DISABLED-RPATH-NOT: (RPATH)
+
+// CHECK-DEFAULT-RUNPATH:      (RUNPATH) Library runpath: [{{.*}}llvm{{.*}}lib{{.*}}]
+// CHECK-EXPLICIT-RUNPATH:     (RUNPATH) Library runpath: [{{.*}}llvm{{.*}}lib{{.*}}]
+// CHECK-DISABLED-RUNPATH-NOT: (RUNPATH)
+
+// CHECK-COMPOSABLE: ({{R|RUN}}PATH) Library {{r|run}}path: [early:late:{{.*}}llvm{{.*}}lib{{.*}}]
+
+int main() {}

openmp/libomptarget/test/lit.cfg

except ValueError:
     # If the architecture is invalid, assume it is supported.
     supports_unified_shared_memory = True
 if config.libomptarget_current_target.startswith('amdgcn'):
     supports_unified_shared_memory = False
 if supports_unified_shared_memory:
    config.available_features.add('unified_shared_memory')
 
 # Setup environment to find dynamic library at runtime
+# Disable the implicit path to increase certainty over which library is picked up
+config.test_flags += " -fno-openmp-implicit-rpath"
 if config.operating_system == 'Windows':
     append_dynamic_library_path('PATH', config.library_dir, ";")
     append_dynamic_library_path('PATH', config.omp_host_rtl_directory, ";")
 elif config.operating_system == 'Darwin':
     append_dynamic_library_path('DYLD_LIBRARY_PATH', config.library_dir, ":")
     append_dynamic_library_path('DYLD_LIBRARY_PATH', \
         config.omp_host_rtl_directory, ";")
     config.test_flags += " -Wl,-rpath," + config.library_dir