Add -fpass-plugin option to Flang

[tarunprabhu]

Summary

This patch adds the -fpass-plugin option to flang which dynamically loads LLVM passes from the shared object passed as the argument to the flag. The behavior of the option is designed to replicate that of the same option in clang and thus has the same capabilities and limitations.

Features:

• Multiple instances of -fpass-plugin=path-to-file can be specified and each of the files will be loaded in that order.

• The flag can be passed to both flang-new and flang-new -fc1.

• The flag will be listed when the -help flag is passed to both flang-new and flang-new -fc1. It will also be listed when the --help-hidden flag is passed.

[tarunprabhu]

Updating the patch because the earlier version was missing some files.

[awarzynski]

Hello! Thanks for implementing this :)

I'm traveling this week, so only had a chance to skim through. I'm away next week, but will try to review properly before that. Btw, have you considered re-using LLVM's Hello pass for testing?

[tarunprabhu]

Btw, have you considered re-using LLVM's Hello pass for testing?

I wanted to also propose a similar option to write out-of-tree FIR (MLIR) passes. I thought that it might be instructive to have the same/similar functionality in an example plugin, an MLIR pass and an LLVM pass.

[awarzynski]

Finally had a chance to take a look. My comments below and inline. Again, thanks for working on this!

In D129156#3632398, @tarunprabhu wrote:

Btw, have you considered re-using LLVM's Hello pass for testing?

I wanted to also propose a similar option to write out-of-tree FIR (MLIR) passes. I thought that it might be instructive to have the same/similar functionality in an example plugin, an MLIR pass and an LLVM pass.

OK, but then is this patch about adding support for -fpass-plugin or is it for comparing "hello-world" passes in MLIR and LLVM? From the driver perspective, -fpass-plugin is merely a flag that tweaks the behavior of the middle-end. And the driver/Flang should only really care about whether the pass that was loaded dynamically was actually run or not. This can be tested with -fdebug-pass-manager. But for that you only need a "dummy" pass, right? Or, an example from LLVM.

And btw, at what optimisation level will this plugin be run? Also, note that what you included here is an in-tree plugin. A few other points below.

1. "flang/test/Examples/print-fns-definitions.f90" was introduced to test "PrintFunctionNames" parse-tree plugin (i.e. a Flang plugin). LLVM plugins are very different (e.g. run at a completely different stage of the compilation pipeline). IMO, we shouldn't be testing such different things in one file.
2. In your summary, could you indicate that similar option already exists in Clang and that you are (or perhaps not?) making sure that the semantics of this option in Flang are consistent with Clang. I think that it is important that we maintain compatibility with Clang and that we document any divergence from that. Also, by stating that the semantics are identical to those in Clang, you save yourself the need to document them. Otherwise, how are people going to know how to use this option?
3. Could you add a relevant entry in https://github.com/llvm/llvm-project/blob/main/flang/test/Driver/frontend-forwarding.f90?
4. Could you update the release notes and driver documentation? This could be something brief.
5. Will this work on all platforms? (e.g. Windows?)

clang/lib/Driver/ToolChains/Flang.cpp

55

This is not really something within the scope of this patch, but AddOtherOptions should be renamed as e.g. ForwardToFC1 or something similar.

flang/examples/PrintFunctionNamesLLVMPass/PrintFunctionNamesLLVMPass.cpp

11 ↗

(On Diff #442367)

"an out-of-tree LLVM pass"

This is an "in-tree" pass.

flang/include/flang/Frontend/CodeGenOptions.h

48–58

DELETEME

flang/lib/Frontend/FrontendActions.cpp

702

IMO, it's a bit weird to report an error and to continue anyway. Apparently that's what Clang does 🤔 . Do you know why?

[tarunprabhu]

In D129156#3638935, @awarzynski wrote:

OK, but then is this patch about adding support for -fpass-plugin or is it for comparing "hello-world" passes in MLIR and LLVM? From the driver perspective, -fpass-plugin is merely a flag that tweaks the behavior of the middle-end.

That's a good point. I think something like that could be useful to newcomers to help decide what is most appropriate, but it should be separate from this patch.

And btw, at what optimisation level will this plugin be run? Also, note that what you included here is an in-tree plugin. A few other points below.

1. "flang/test/Examples/print-fns-definitions.f90" was introduced to test "PrintFunctionNames" parse-tree plugin (i.e. a Flang plugin). LLVM plugins are very different (e.g. run at a completely different stage of the compilation pipeline). IMO, we shouldn't be testing such different things in one file.

2. In your summary, could you indicate that similar option already exists in Clang and that you are (or perhaps not?) 3. 4.

Ok, I'll update the patch with these changes.

5. Will this work on all platforms? (e.g. Windows?)

I don't know. Unfortunately, I don't have access to a Windows machine on which to test this. I imagine that since this largely uses LLVM's machinery, if it works with clang, it should work for us too. However, it would still need to be tested. Do you have any suggestions?

clang/lib/Driver/ToolChains/Flang.cpp

55

I don't think that I should change that here, unless you want me to. As you say, it probably belongs to a different patch.

flang/examples/PrintFunctionNamesLLVMPass/PrintFunctionNamesLLVMPass.cpp

11 ↗

(On Diff #442367)

When I update this patch, I'll remove this pass and update the tests to use LLVM's Hello pass instead. I called this an out-of-tree pass because it can be used as is out-of-tree as well (that's how I tested it before adding it to the tree).

The registerPass function will need to be changed if the pass is to run at a specific optimization level. Since that OptimizationLevel argument is ignored, it will run even on -O0.

flang/include/flang/Frontend/CodeGenOptions.h

48–58

Do you want me to delete the second "public:" label?

flang/lib/Frontend/FrontendActions.cpp

702

I think the idea is to display as many errors as reasonably possible instead of just failing on the first error that was encountered. I imagine that the Diagnostics object was intended to be used when parsing where this approach is helpful. But when using it elsewhere, you end up with more ... peculiar behavior.

[awarzynski]

In D129156#3639153, @tarunprabhu wrote:

5. Will this work on all platforms? (e.g. Windows?)

I don't know. Unfortunately, I don't have access to a Windows machine on which to test this. I imagine that since this largely uses LLVM's machinery, if it works with clang, it should work for us too. However, it would still need to be tested. Do you have any suggestions?

"Officially", plugins in LLVM are not supported on Windows :) I'm not aware of one single "source of truth" for this, but there are plenty of references on Discourse and Phabricator:

• https://reviews.llvm.org/D16761
• https://discourse.llvm.org/t/clang-wont-run-example-plugins-on-windows/57231

Going over https://reviews.llvm.org/D56935 should also help understand the implementation in Clang (and the logic behind the semantics of this option). That should help writing the documentation for Flang too.

clang/lib/Driver/ToolChains/Flang.cpp

55

I agree, this would be an "unrelated change". Hopefully one of us will remember to rename this in the future ;-)

flang/include/flang/Frontend/CodeGenOptions.h

48–58

it is not needed, is it?

flang/lib/Frontend/FrontendActions.cpp

702

So, an "error" indicates that something went fundamentally wrong and indeed, flang-new will fail (that's why you had to use not in tests). If the driver does not immediately return here, could you add a comment to hint "why"? I also went over the discussion in https://reviews.llvm.org/D56935 and the rationale in Clang is still unclear to me. How about you?

If you are simply trying to reproduce the Clang semantics here, then that would be a good comment for me.

[tarunprabhu]

flang/include/flang/Frontend/CodeGenOptions.h

48–58

Strictly speaking, no. I tend to group data members and member functions with the same access specifier and explicitly add the access specifier for each group. If the class ever grows, it just helps to keep things a bit better organized. I didn't look at the rest of the flang code to see if it is done differently elsewhere.

flang/lib/Frontend/FrontendActions.cpp

702

I think it's just deferring failure until later because that can be more helpful. It lets you inform the user about multiple potential problems at the same time. If you look at clang/lib/Frontend/CompilerInvocation.cpp, you will see a pattern where the DiagnosticsEngine::getNumErrors() is called at the start and end of each function, to indicate whether that particular functionality was completed without errors. Even there, the error is recorded, the but the failure is deferred.

But, I haven't been involved with clang development, so I am hardly an authority on the matter and I could be completely wrong about this.

Nevertheless, I'll add a comment saying that the code is intended to mimic clang.

[tarunprabhu]

It looks like LLVM's HelloWorld pass with the new pass manager is not built as a separate shared object. LLVMHello.so uses the old pass manager and cannot be loaded using -fpass-plugin.

A quick search for "-fpass-plugin" in clang/test didn't seem to yield anything so I don't know if this is actually tested there.

I would like to make a case for keeping the PrintFunctionNamesLLVMPass example for the following reasons:

• It can be used to explicitly test -fpass-plugin.

• It is a full working example of how to write an out-of-tree pass that can be loaded with -fpass-plugin (on further consideration, I disagree with your claim that it is an in-tree pass. It can be built independently of Flang/LLVM. Shouldn't that make it out-of-tree by definition? If not, even the PrintFunctionNames plugin would then become "in-tree").

The counter-arguments to this would be that the pass is not strictly related to flang, and probably ought to belong elsewhere in the LLVM tree. We could add it to flang now and move it after consultation with the broader community. As far as I have been able to tell, there is no full example of building a complete out-of-tree LLVM pass in the official documentation.

The other consideration is that there needs to be a way to ensure that the -fpass-plugin test is marked as "expected-to-fail" on Windows. I don't know how to do that but hopefully someone else knows.

[awarzynski]

In D129156#3640027, @tarunprabhu wrote:

It looks like LLVM's HelloWorld pass with the new pass manager is not built as a separate shared object. LLVMHello.so uses the old pass manager and cannot be loaded using -fpass-plugin.

1. Good point, HelloWorldPass was refactored in https://reviews.llvm.org/D95907 not to be built as a standalone shared object. Might be worth bringing this up on Discourse - perhaps it would be OK to change this? If not, we'll know that a separate example is indeed needed.
2. With the switch to the new PM, LLVM's HelloWorld pass should probably be removed. It would be good to bring this up on Discourse.
3. Have you tried the Bye pass?

A quick search for "-fpass-plugin" in clang/test didn't seem to yield anything so I don't know if this is actually tested there.

From what I can see, it is not. If you do want to test this flag with a proper pass, then it would be very desirable to extend this patch to Clang and add a test there too. This way your work would benefit both sub-projects,

I would like to make a case for keeping the PrintFunctionNamesLLVMPass example for the following reasons:

• It can be used to explicitly test -fpass-plugin.

This is a fair point, but do we need an LLVM pass in Flang for this? I don't believe we do. And if PrintFunctionNamesLLVMPass were to be included, why can't the run method be as simple as return PreservedAnalyses::all();?

• It is a full working example of how to write an out-of-tree pass that can be loaded with -fpass-plugin

But this patch is not about writing out-of-tree LLVM passes, is it? If it is, then it should be a contribution made to LLVM instead.

(on further consideration, I disagree with your claim that it is an in-tree pass. It can be built independently of Flang/LLVM. Shouldn't that make it out-of-tree by definition? If not, even the PrintFunctionNames plugin would then become "in-tree").

This is my understanding of "in-tree" vs "out-of-tree":

• "in-tree" --> located inside llvm-project
• "out-of-tree" --> located outside llvm-project

You are proposing to add PrintFunctionNamesLLVMPass inside "llvm-project", so it's an "in-tree" example. In particular, if I copy "PrintFunctionNamesLLVMPass.cpp" to a directory outside "llvm-project", then I will no longer be able to build your example without extra CMake logic.

The counter-arguments to this would be that the pass is not strictly related to flang, and probably ought to belong elsewhere in the LLVM tree.

Indeed. In particular, an example like this inside LLVM (instead of Flang) could be used by both Flang and Clang for testing -fpass-plugin.

We could add it to flang now and move it after consultation with the broader community.

Why delay this? Also, you might be able to use the Bye pass in the meantime.

As far as I have been able to tell, there is no full example of building a complete out-of-tree LLVM pass in the official documentation.

Yup. I've tried to fill this gap by creating llvm-tutor (apologies for shameless self-promotion). I'm not sure whether there would be much appetite for this sort of content "in-tree" (is "in-tree" documentation the right place to document "out-of-tree" development?). In any case, is this patch about documenting out-of-tree LLVM pass development or about adding support for -fpass-plugin in Flang?

The other consideration is that there needs to be a way to ensure that the -fpass-plugin test is marked as "expected-to-fail" on Windows. I don't know how to do that but hopefully someone else knows.

You want to use ! UNSUPPORTED: system-windows (see e.g. save-temps.f90).

flang/include/flang/Frontend/CodeGenOptions.h

48–58

To me, with so few fields in this class, it's just a noise.

flang/lib/Frontend/FrontendActions.cpp

702

If you look at clang/lib/Frontend/CompilerInvocation.cpp, you will see a pattern where the DiagnosticsEngine::getNumErrors() is called at the start and end of each function, to indicate whether that particular functionality was completed without errors.

Similar pattern has been used in Flang, see e.g. here.

Even there, the error is recorded, the but the failure is deferred.

CompilerInvocation.cpp deals with "option parsing" and "compiler configuration". FrontendActions.cpp deals with "running" the compiler. These are very different stages in the compiler executation.

[tarunprabhu]

3. Have you tried the Bye pass?

Ah, good point. I hadn't. I'll see if I can use that instead.

From what I can see, it is not. If you do want to test this flag with a proper pass, then it would be very desirable to extend this patch to Clang and add a test there too. This way your work would benefit both sub-projects,

Ok. It shouldn't be too difficult to create a patch for that. I suppose I
can add one and see what the clang community has to say about it.

Yup. I've tried to fill this gap by creating llvm-tutor (apologies for shameless self-promotion).

In the spirit of shameless self-promotion, I have done something similar for clang
plugins too :-) .

CompilerInvocation.cpp deals with "option parsing" and "compiler configuration". FrontendActions.cpp deals with "running" the compiler. These are very different stages in the compiler executation.

True, but even with actions, there may be those that would benefit from not
failing immediately. For instance, the ParseArgs() method in a
PluginASTAction may also report errors during option parsing that may
benefit from deferred failure (although I must confess that I don't know
exactly when a PluginASTAction's ParseArgs() is run during the
compilation process -- in particular, whether it runs during or after the
other arguments have been parsed. I think I did trace the path at one
point, but I don't remember now).

[EDIT: Fixed formatting]

[tarunprabhu]

In D129156#3642328, @tarunprabhu wrote:

3. Have you tried the Bye pass?

Ah, good point. I hadn't. I'll see if I can use that instead.

It seems as if the Bye pass will only print a message when the -wave-goodbye flag is passed to it. But I don't know of a way to pass that flag to the pass when loading the pass using -fpass-plugin. If not, is there a way, with the new pass manager, to print the list of passes that were executed?

[awarzynski]

In D129156#3647064, @tarunprabhu wrote:

It seems as if the Bye pass will only print a message when the -wave-goodbye flag is passed to it. But I don't know of a way to pass that flag to the pass when loading the pass using -fpass-plugin. If not, is there a way, with the new pass manager, to print the list of passes that were executed?

You can use -fdebug-pass-manager, see e.g. here.

In D129156#3642328, @tarunprabhu wrote:

Ok. It shouldn't be too difficult to create a patch for that. I suppose I
can add one and see what the clang community has to say about it.

You can do it in this patch. It's going to be a small change.

True, but even with actions, there may be those that would benefit from not failing immediately.

We haven't identified a specific case in which not failing immediately would be desirable. I'm fine with the current approach, but would appreciate if consistency with Clang was emphasized (either in the commit message or as a comment, preferably both).

[tarunprabhu]

Removed the pass to test the option. Instead, the Bye plugin provided in LLVM is used to test that the option works. It is not clear if loading passes dynamically is supported on Windows and the test is designed to fail on that platform.

Addressed comments from @awarzynski.

Added documentation to the ReleaseNotes. It was put under "Major changes" but it can be moved to compiler options if that is a more appropriate place for this.

Added some text to the frontend driver documentation.

Added a test to checked that the -fpass-plugin option is correctly passed to the fc1 driver.

[awarzynski]

Thanks for the updates!

In D129156#3807119, @tarunprabhu wrote:

It is not clear if loading passes dynamically is supported on Windows and the test is designed to fail on that platform.

It isn't in your implementation, so it is safe to document it as unsupported on Windows :) In general, plugins in LLVM are not officially supported on Windows.

[awarzynski]

flang/docs/FlangDriver.md

513–514

Pass plugins are an extension to the frontend drive that make it possible to run out-of-tree LLVM passes on the LLVM-IR.

This isn't quite true.

1. These plugins can be loaded from both the compiler and frontend drivers.
2. Both out-of-tree and in-tree passes are supported.
3. This is not an extension to any of the drivers. The drivers that implement -fpass-plugin simply provide an interface to forward the specified plugin to LLVM (i.e. the middle-end).

515

516–517

See the documentation for llvm::PassBuilder for details

Please, can you provide a link?

519–520

The capitalization is inconsistent. Please use either "Flang" and "Clang" or "flang" and "clang". Usually, "Clang"/"Flang" refers to the sub-project and "flang"/"clang" to the binaries. Note that tools in this document are surrounded with backticks (i.e. "`").

flang/include/flang/Frontend/CodeGenOptions.h

48–51

DELETEME

49

DOCUMENTME :)

flang/test/Driver/frontend-forwarding.f90

4

This will really limit this test. I'd rather have it run on all supported platforms (with plugins enabled *and* disabled) than not.

flang/test/Driver/pass-plugin.f90

8–9

[tarunprabhu]

Added a sentence in the driver documentation stating that LLVM plugins are not officially supported on Windows.

Addressed other comments and issues.

[awarzynski]

Thanks for the update! Looks like per-commit CI is failing.

flang/docs/FlangDriver.md

515

It's probably worth specifying that this is the LLVM middle-end and that these passes are run by the optimisation rather than code-gen/backend pipelines (there are two).

521

[tarunprabhu]

Made it clear that dynamically loading passes from a pass plugin would add the pass to the optimization pipeline.

Addressed other issues.

The test is not failing for me, but I will look into it.

[tarunprabhu]

Potential fix to failing test. It seems that on some platforms, the output is `{anonymous}::Bye`` while on others it is `{anonymous namespace}::Bye``. The test has been fixed to allow for both. I was not able to test this locally, so will see if the buildbot is happy with it.

[tarunprabhu]

Potential fix to failing test.

It seems that on some platforms, the output is `{anonymous}::Bye`` while on others it is `{anonymous namespace}::Bye``. Hopefully, the buildbot will be happy with this change.

[tarunprabhu]

Used wrong bracket type in previous revision.

Fixed that test and use regular expressions to match the alternatives instead.

[tarunprabhu]

Potentially fixed regex to correctly match round brackets.

[awarzynski]

clang/lib/Driver/ToolChains/Flang.cpp

58–59

CLANGFORMATME

flang/test/Driver/pass-plugin.f90

11

[tarunprabhu]

Ran clang-format.

Fixed test to allow for any namespace prefix.

[tarunprabhu]

I noticed that clang-format changed several other lines in clang/lib/Driver/ToolChains/Flang.cpp. I undid those and only left in the line relevant to this patch. Do you think that those lines should be run on that file as a separate patch?

[awarzynski]

LGTM, many thanks for addressing all my comments and for seeing this through!

In D129156#3833501, @tarunprabhu wrote:

I noticed that clang-format changed several other lines in clang/lib/Driver/ToolChains/Flang.cpp. I undid those and only left in the line relevant to this patch. Do you think that those lines should be run on that file as a separate patch?

Up to you :) In general, it's good to keep the noise to a minimum. For things like this I would just push a separate patch without a review. Such patches implement non-functional-changes, are rather straightforward and very desirable (they help to maintain the code in good health).

Let me know if you need somebody to merge this for you.

[tarunprabhu]

Thanks for the review and the comments. :-)

[clementval]

I get a failure since this commit. Could not load library By.so ... cannot open shared object file: No such file or directory. Maybe a missing dependency?

[tarunprabhu]

I think there is a typo somehow. It should be "Bye.so". Not sure how that happened. I'm looking into this.

[clementval]

In D129156#3836768, @tarunprabhu wrote:

I think there is a typo somehow. It should be "Bye.so". Not sure how that happened. I'm looking into this.

I probably didn't copy the error correctly .. this is the correct libs that is not found llvm-project/build/lib/Bye.so

[tarunprabhu]

In D129156#3836838, @clementval wrote:

I probably didn't copy the error correctly .. this is the correct libs that is not found llvm-project/build/lib/Bye.so

I need some help with this.

The failing build is building flang out-of-tree. It looks like %llvmshlibdir/ does not get set correctly in this case. Bye.so is located in %llvmshlibdir which is within the LLVM build directory.

Do you know if there is a way to tell the test-suite about %llvmshlibdir?

[MatsPetersson]

In D129156#3836895, @tarunprabhu wrote:

In D129156#3836838, @clementval wrote:

I probably didn't copy the error correctly .. this is the correct libs that is not found llvm-project/build/lib/Bye.so

I need some help with this.

The failing build is building flang out-of-tree. It looks like %llvmshlibdir/ does not get set correctly in this case. Bye.so is located in %llvmshlibdir which is within the LLVM build directory.

Do you know if there is a way to tell the test-suite about %llvmshlibdir?

I'm fairly sure it is not a "out of tree" build, I'm seeing the same problem building in tree.
As far as I can see, there's nothing creatubg a "Bye.so" plugin. Maybe I'm missing something tho'

[DavidTruby]

This is also causing phabricator's pre-commit CI builds to fail, for example this build for my patch: https://buildkite.com/llvm-project/premerge-checks/builds/115370#0183a8a5-6684-4602-952f-81bde55220ea
Could we revert this until it is fixed? I think the CI passing is quite important.

[awarzynski]

I suspect that this fails when running ninja check-flang, right?

Most likely Bye needs to be added as a dependency for Flang tests, something akin to this. Alternatively, try adding examples to ! REQUIRES: in pass-plugin.f90.

[MatsPetersson]

In D129156#3837709, @awarzynski wrote:

I suspect that this fails when running ninja check-flang, right?

Most likely Bye needs to be added as a dependency for Flang tests, something akin to this. Alternatively, try adding examples to ! REQUIRES: in pass-plugin.f90.

As this seems to be breaking the testing on the public servers, as well as locally on my machine, I've reverted the commit for now. Please feel free to submit an updated version, this is not a criticism of the work as such, just trying to keep the "build & test" green.

[clementval]

As @awarzynski mentioned, it is likely a missing dependency on the test.

[tarunprabhu]

Added examples to REQUIRES in `test/Driver/pass-plugin.f90.

I still cannot reproduce the build failure on my end. @MatsPetersson tested this patch and the tests passed. Could someone else test this on their build and let me know if it works - especially if the previous version failed for you.

[tarunprabhu]

I tried an out-of-tree flang build. check-flang passes in that case too.

[awarzynski]

In D129156#3847396, @tarunprabhu wrote:

Added examples to REQUIRES in `test/Driver/pass-plugin.f90.

Thanks for the update!

I still cannot reproduce the build failure on my end. @MatsPetersson tested this patch and the tests passed.

@MatsPetersson & @clementval , could you share you build command so that the failure can be reproduced before this re-lands?

Could someone else test this on their build and let me know if it works - especially if the previous version failed for you.

I can try as soon as folks share their build commands. In general, I'm concerned that this fix is not enough. It merely makes sure that "pass-plugin.f90" is only run when LLVM_BUILD_EXAMPLES is set (or whatever other flag that sets examples in LIT). However, that's not sufficient to make sure that libBye.so is built when running ninja check-flang. I might wrong though - have you checked this?

[clementval]

In D129156#3848692, @awarzynski wrote:

In D129156#3847396, @tarunprabhu wrote:

Added examples to REQUIRES in `test/Driver/pass-plugin.f90.

Thanks for the update!

I still cannot reproduce the build failure on my end. @MatsPetersson tested this patch and the tests passed.

@MatsPetersson & @clementval , could you share you build command so that the failure can be reproduced before this re-lands?

I shared it with @tarunprabhu and I think the only major difference is the use of Ninja vs. Unix Makefiles.

cmake \
  -G "Unix Makefiles" \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DLLVM_TARGETS_TO_BUILD=host \
  -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" \
  -DLLVM_ENABLE_RUNTIMES="compiler-rt"

[awarzynski]

In D129156#3848704, @clementval wrote:

I still cannot reproduce the build failure on my end. @MatsPetersson tested this patch and the tests passed.

@MatsPetersson & @clementval , could you share you build command so that the failure can be reproduced before this re-lands?

I shared it with @tarunprabhu and I think the only major difference is the use of Ninja vs. Unix Makefiles.

cmake \
  -G "Unix Makefiles" \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DLLVM_TARGETS_TO_BUILD=host \
  -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" \
  -DLLVM_ENABLE_RUNTIMES="compiler-rt"

Thanks Valentin! Switching between generators will definitely change the order in which libraries will built (and, AFAIK, the order is non-deterministic to begin with). I will try to experiment later today.

If anyone manages to test this in the meantime, please share here - that's part of the public review process :)

[Leporacanthicus]

@MatsPetersson & @clementval , could you share you build command so that the failure can be reproduced before this re-lands?

I can't share the same command that I used, because it's some old CMAKE command that I no longer have in my history. But I tested this patch, and it works in the same tree that I was using it last week.

I just double-checked again in my release-build directory, and it also works. So I think the latest version works...

[tarunprabhu]

In D129156#3848728, @awarzynski wrote:

Thanks Valentin! Switching between generators will definitely change the order in which libraries will built (and, AFAIK, the order is non-deterministic to begin with). I will try to experiment later today.

I ran this with Unix Makefiles as the generator and it passes there too.

What I did notice when I tried the out-of-tree build of flang is that the test did not run. The message I got was that the test was "unsupported".

I assume that is because there is no dependence on LLVM examples in an out-of-tree build. Would adding an explicit dependence on LLVM's examples cause the test to run? I imagine that we want the feature to be tested even when building flang out-of-tree.

[awarzynski]

In @clementval 's CMake invocation LLVM_BUILD_EXAMPLES is not set, which means that the examples (e.g. the Bye plugin) are not built. Adding ! REQUIRES: examples to the test should fix the failure in this case. I did verify locally.

However, the pre-commit CI that was failing did include the examples:

cmake ../llvm -D LLVM_ENABLE_PROJECTS="mlir;flang;clang;llvm" -G Ninja -D CMAKE_BUILD_TYPE=Release -D LLVM_ENABLE_ASSERTIONS=ON -D LLVM_BUILD_EXAMPLES=ON -D LLVM_LIT_ARGS="-v --xunit-xml-output test-results.xml" -D LLVM_ENABLE_LLD=ON -D CMAKE_CXX_FLAGS=-gmlt -DBOLT_CLANG_EXE=/usr/bin/clang
ninja check-flang

You can see there that libBye.so is not built. Could you try that command, pls?

In D129156#3849487, @tarunprabhu wrote:

I imagine that we want the feature to be tested even when building flang out-of-tree.

I don't built standalone LLVM Flang, so won't be needing it. Perhaps somebody else will, dunno :)

[awarzynski]

@tarunprabhu Could you confirm that with the build command above "pass-plugin.f90" is failing for you? It is for me.

In order to fix that, you will have to add this in Flang's test/CMakeLists.txt. After the change, libBye.so will be added as a dependency to Flang tests, but only when FLANG_BUILD_EXAMPLES is set. I would also add a note in "pass-plugin.f90" that it depends on the "Bye" pass from LLVM.

[tarunprabhu]

In D129156#3851843, @awarzynski wrote:

@tarunprabhu Could you confirm that with the build command above "pass-plugin.f90" is failing for you? It is for me.

@awarzynski What compiler do you use to build this? gcc doesn't seem to like -gmlt, clang-12 complains about -fuse-ld=lld (could be the clang installation on the machine on which I am building), and clang-13 causes a compile error.

[EDIT: Tag @awarzynski]

[awarzynski]

In D129156#3852553, @tarunprabhu wrote:

In D129156#3851843, @awarzynski wrote:

@tarunprabhu Could you confirm that with the build command above "pass-plugin.f90" is failing for you? It is for me.

@awarzynski What compiler do you use to build this? gcc doesn't seem to like -gmlt, clang-12 complains about -fuse-ld=lld (could be the clang installation on the machine on which I am building), and clang-13 causes a compile error.

[EDIT: Tag @awarzynski]

I doubt those particular flags matter here. You can trim that CMake command as follows:

cmake ../llvm -D LLVM_ENABLE_PROJECTS="mlir;flang;clang;llvm" -G Ninja -D CMAKE_BUILD_TYPE=Release -D LLVM_ENABLE_ASSERTIONS=ON -D LLVM_BUILD_EXAMPLES=ON 
ninja check-flang

[tarunprabhu]

In D129156#3851843, @awarzynski wrote:

@tarunprabhu Could you confirm that with the build command above "pass-plugin.f90" is failing for you? It is for me.

The tests still passed. Some relevant output from the test is below (I added the -DLLVM_LIT_ARGS="-v" to the configure line).

UNSUPPORTED: Flang :: Driver/pass-plugin.f90 (1721 of 1809)
PASS: Flang :: Driver/pass-plugin-not-found.f90 (560 of 1809)

Testing Time: 4.81s
  Unsupported      :   11
  Passed           : 1780
  Expectedly Failed:   18

I am not sure what I am doing differently. This is starting from an empty build directory.

[awarzynski]

In D129156#3853045, @tarunprabhu wrote:

The tests still passed.

No, it wasn't run. We need to understand why before proceeding.

[tarunprabhu]

Added a dependence for the flang tests on LLVM's Bye plugin. This results in the Bye plugin being built when check-flang is run, even if building examples has been explicitly disabled.

Removed the dependency on 'examples' in the pass-plugin.f90 test.

Some minor changes to function names for better consistency.

[awarzynski]

flang/test/CMakeLists.txt

66

IIUC, Bye is only needed when plugins are enabled.

[tarunprabhu]

Only add a dependence to the Bye plugin when plugins are enabled.

[tarunprabhu]

Previous revision was missing the tests.

Only add a dependence to the Bye plugin when plugins are enabled.

[awarzynski]

Please be aware of https://reviews.llvm.org/D137673 - you may need to rebase if it lands before this patch.

Please just go ahead and merge, but please keep WIN32 in the final version of "flang/test/CMakeLists.txt" (see my comment inline).

LGTM

flang/test/CMakeLists.txt

66

IIUC, WIN32 is still required. Sorry for not being clearer earlier.

[tarunprabhu]

Yes, I am aware of the other patch which also adds test/Driver/pass-plugin.f90. I will keep an eye out for it.

flang/test/CMakeLists.txt

66

Ugh! Yes, of course. I looked at llvm/CMakeLists.txt and somehow concluded that LLVM_ENABLE_PLUGINS is disabled if WIN32, but it is in fact the _default value_ of LLVM_ENABLE_PLUGINS that is set to false in that case.

[ro]

This introduced a new failure on Solaris:

FAIL: Flang :: Driver/pass-plugin-not-found.f90

Running the failing command manually shows:

error: unable to load plugin 'X.Y': 'Could not load library 'X.Y': ld.so.1: flang-new: X.Y: open failed: No such file or directory'

while the test currently expects

error: unable to load plugin 'X.Y': 'Could not load library 'X.Y': X.Y: cannot open shared object file: No such file or directory'

This expectation is unjustified, I believe: the message is produced by PassPlugin::Load -> sys::DynamicLibrary::getPermanentLibrary -> HandleSet::DLOpen -> ::dlerror. Obviously the output of the last is system-dependent; I don't think we can put any requirements on it (maybe not even the exact wording of the No such file or directory part.

[awarzynski]

In D129156#3949156, @ro wrote:

This introduced a new failure on Solaris:

FAIL: Flang :: Driver/pass-plugin-not-found.f90

Running the failing command manually shows:

error: unable to load plugin 'X.Y': 'Could not load library 'X.Y': ld.so.1: flang-new: X.Y: open failed: No such file or directory'

while the test currently expects

error: unable to load plugin 'X.Y': 'Could not load library 'X.Y': X.Y: cannot open shared object file: No such file or directory'

This expectation is unjustified, I believe: the message is produced by PassPlugin::Load -> sys::DynamicLibrary::getPermanentLibrary -> HandleSet::DLOpen -> ::dlerror. Obviously the output of the last is system-dependent; I don't think we can put any requirements on it (maybe not even the exact wording of the No such file or directory part.

Sorry that you are hitting this - things like this happen, sadly. I think that the easiest to resolve it would be to tweak the expected error so that it works on Solaris as well as other popular platforms. Given that you might be the only person able to test on Solaris, would you be able to upload something for a review? Thanks!

[ro]

In D129156#3949194, @awarzynski wrote:

Sorry that you are hitting this - things like this happen, sadly. I think that the easiest to resolve it would be to tweak the expected error so that it works on Solaris as well as other popular platforms. Given that you might be the only person able to test on Solaris, would you be able to upload something for a review? Thanks!

No worries, especially given that this seems to be the first test ever to check the Could not load library error message.

I've just submitted a patch that works on Linux and Solaris (all that I can easily test): D138663.

clang/include/clang/Driver/Options.td

defm rwpi : BoolFOption<"rwpi",
   PosFlag<SetTrue, [CC1Option], "Generate read-write position independent code (ARM only)">,
   NegFlag<SetFalse>>;
 def fplugin_EQ : Joined<["-"], "fplugin=">, Group<f_Group>, Flags<[NoXarchOption]>, MetaVarName<"<dsopath>">,
   HelpText<"Load the named plugin (dynamic shared object)">;
 def fplugin_arg : Joined<["-"], "fplugin-arg-">,
   MetaVarName<"<name>-<arg>">,
   HelpText<"Pass <arg> to plugin <name>">;
 def fpass_plugin_EQ : Joined<["-"], "fpass-plugin=">,
-  Group<f_Group>, Flags<[CC1Option]>, MetaVarName<"<dsopath>">,
+  Group<f_Group>, Flags<[CC1Option,FlangOption,FC1Option]>, MetaVarName<"<dsopath>">,
   HelpText<"Load pass plugin from a dynamic shared object file (only with new pass manager).">,
   MarshallingInfoStringVector<CodeGenOpts<"PassPlugins">>;
 defm preserve_as_comments : BoolFOption<"preserve-as-comments",
   CodeGenOpts<"PreserveAsmComments">, DefaultTrue,
   NegFlag<SetFalse, [CC1Option], "Do not preserve comments in inline assembly">,
   PosFlag<SetTrue>>;
 def framework : Separate<["-"], "framework">, Flags<[LinkerInput]>;
 def frandom_seed_EQ : Joined<["-"], "frandom-seed=">, Group<clang_ignored_f_Group>;

clang/lib/Driver/ToolChains/Flang.h

private:
   void addPreprocessingOptions(const llvm::opt::ArgList &Args,
                                llvm::opt::ArgStringList &CmdArgs) const;
 
   /// Extract PIC options from the driver arguments and add them to
   /// the command arguments.
   ///
   /// \param [in] Args The list of input driver arguments
   /// \param [out] CmdArgs The list of output command arguments
-  void AddPicOptions(const llvm::opt::ArgList &Args,
+  void addPicOptions(const llvm::opt::ArgList &Args,
                      llvm::opt::ArgStringList &CmdArgs) const;
 
-  /// This method will effectively copy options from \a Args into \a CmdArgs.
+  /// Extract other compilation options from the driver arguments and add them
+  /// to the command arguments.
   ///
   /// \param [in] Args The list of input driver arguments
   /// \param [out] CmdArgs The list of output command arguments
-  void forwardOptions(const llvm::opt::ArgList &Args,
+  void addOtherOptions(const llvm::opt::ArgList &Args,
                        llvm::opt::ArgStringList &CmdArgs) const;
 
 public:
   Flang(const ToolChain &TC);
   ~Flang() override;
 
   bool hasGoodDiagnostics() const override { return true; }
   bool hasIntegratedAssembler() const override { return true; }
   bool hasIntegratedCPP() const override { return true; }

clang/lib/Driver/ToolChains/Flang.cpp

 
 void Flang::addPreprocessingOptions(const ArgList &Args,
                                     ArgStringList &CmdArgs) const {
   Args.AddAllArgs(CmdArgs,
                   {options::OPT_P, options::OPT_D, options::OPT_U,
                    options::OPT_I, options::OPT_cpp, options::OPT_nocpp});
 }
 
-void Flang::forwardOptions(const ArgList &Args, ArgStringList &CmdArgs) const {
+void Flang::addOtherOptions(const ArgList &Args, ArgStringList &CmdArgs) const {
   Args.AddAllArgs(CmdArgs,
                   {options::OPT_module_dir, options::OPT_fdebug_module_writer,
                    options::OPT_fintrinsic_modules_path, options::OPT_pedantic,
                    options::OPT_std_EQ, options::OPT_W_Joined,
-                   options::OPT_fconvert_EQ});
+                   options::OPT_fconvert_EQ, options::OPT_fpass_plugin_EQ});
 }
 
-void Flang::AddPicOptions(const ArgList &Args, ArgStringList &CmdArgs) const {
+void Flang::addPicOptions(const ArgList &Args, ArgStringList &CmdArgs) const {
   // ParsePICArgs parses -fPIC/-fPIE and their variants and returns a tuple of
   // (RelocationModel, PICLevel, IsPIE).
   llvm::Reloc::Model RelocationModel;
   unsigned PICLevel;
   bool IsPIE;
   std::tie(RelocationModel, PICLevel, IsPIE) =
       ParsePICArgs(getToolChain(), Args);
 
void Flang::ConstructJob(Compilation &C, const JobAction &JA,
   // re-parsed to construct this job; claim any possible color diagnostic here
   // to avoid warn_drv_unused_argument.
   Args.getLastArg(options::OPT_fcolor_diagnostics,
                   options::OPT_fno_color_diagnostics);
   if (D.getDiags().getDiagnosticOptions().ShowColors)
     CmdArgs.push_back("-fcolor-diagnostics");
 
   // -fPIC and related options.
-  AddPicOptions(Args, CmdArgs);
+  addPicOptions(Args, CmdArgs);
 
   // Floating point related options
   addFloatingPointOptions(D, Args, CmdArgs);
 
-  // Handle options which are simply forwarded to -fc1.
-  forwardOptions(Args, CmdArgs);
+  // Add other compile options
+  addOtherOptions(Args, CmdArgs);
 
   // Forward -Xflang arguments to -fc1
   Args.AddAllArgValues(CmdArgs, options::OPT_Xflang);
 
   // Forward -mllvm options to the LLVM option parser. In practice, this means
   // forwarding to `-fc1` as that's where the LLVM parser is run.
   for (const Arg *A : Args.filtered(options::OPT_mllvm)) {
     A->claim();

flang/docs/FlangDriver.md

 trivial. Please take this into consideration when designing your plugin. In
 particular, creating a transformation plugin will be noticeably harder than
 analysis plugins that just consume (rather than edit) `ParseTree`.
 
 Lastly, if `ParseTree` modifications are performed, then it might be necessary
 to re-analyze expressions and modify scope or symbols. You can check
 [Semantics.md](Semantics.md) for more details on how `ParseTree` is edited
 e.g. during the semantic checks.
+
+# LLVM Pass Plugins
+
+Pass plugins are dynamic shared objects that consist of one or more LLVM IR
+passes. The `-fpass-plugin` option enables these passes to be passed to the
+middle-end where they are added to the optimization pass pipeline and run after
+lowering to LLVM IR.The exact position of the pass in the pipeline will depend
+on how it has been registered with the `llvm::PassBuilder`. See the
+documentation for
+[`llvm::PassBuilder`](https://llvm.org/doxygen/classllvm_1_1PassBuilder.html)
+for details.
+
+The framework to enable pass plugins in `flang-new` uses the exact same
+machinery as that used by `clang` and thus has the same capabilities and
+limitations.
+
+In order to use a pass plugin, the pass(es) must be compiled into a dynamic
+shared object which is then loaded using the `-fpass-plugin` option.
+
+```
+flang-new -fpass-plugin=/path/to/plugin.so <file.f90>
+```
+
+This option is available in both the compiler driver and the frontend driver.
+Note that LLVM plugins are not officially supported on Windows.

flang/docs/ReleaseNotes.md

 
 Note that if you are reading this file from a Git checkout, this
 document applies to the *next* release, not the current one. To see the
 release notes for a specific release, please see the [releases
 page](https://llvm.org/releases/).
 
 ## Major New Features
 
+* Flang now supports loading LLVM pass plugins with the `-fpass-plugin` option
+  which is also available in clang. The option mimics the behavior of the
+  corresponding option in clang and has the same capabilities and limitations.
+
 ## Bug Fixes
 
 ## Non-comprehensive list of changes in this release
 * The bash wrapper script, `flang`, is renamed as `flang-to-external-fc`.
 * In contrast to Clang, Flang will not default to using `-fpie` when linking
   executables. This is only a temporary solution and the goal is to align with
   Clang in the near future. First, however, the frontend driver needs to be
   extended so that it can generate position independent code (that requires

flang/include/flang/Frontend/CodeGenOptions.h

 #define ENUM_CODEGENOPT(Name, Type, Bits, Default) unsigned Name : Bits;
 #include "flang/Frontend/CodeGenOptions.def"
 };
 
 /// Tracks various options which control how the code is optimized and passed
 /// to the LLVM backend.
 class CodeGenOptions : public CodeGenOptionsBase {
 
 public:
+  /// The paths to the pass plugins that were registered using -fpass-plugin.
+  std::vector<std::string> LLVMPassPlugins;
+
   // Define accessors/mutators for code generation options of enumeration type.
 #define CODEGENOPT(Name, Bits, Default)
 #define ENUM_CODEGENOPT(Name, Type, Bits, Default)                             \
   Type get##Name() const { return static_cast<Type>(Name); }                   \
   void set##Name(Type Value) { Name = static_cast<unsigned>(Value); }
 #include "flang/Frontend/CodeGenOptions.def"
 
   CodeGenOptions();
 };
 
 } // end namespace Fortran::frontend
 
 #endif

flang/lib/Frontend/CompilerInvocation.cpp

static void parseCodeGenArgs(Fortran::frontend::CodeGenOptions &opts,
                              llvm::opt::ArgList &args,
                              clang::DiagnosticsEngine &diags) {
   opts.OptimizationLevel = getOptimizationLevel(args, diags);
 
   if (args.hasFlag(clang::driver::options::OPT_fdebug_pass_manager,
                    clang::driver::options::OPT_fno_debug_pass_manager, false))
     opts.DebugPassManager = 1;
 
+  for (auto *a : args.filtered(clang::driver::options::OPT_fpass_plugin_EQ))
+    opts.LLVMPassPlugins.push_back(a->getValue());
+
   // -mrelocation-model option.
   if (const llvm::opt::Arg *A =
           args.getLastArg(clang::driver::options::OPT_mrelocation_model)) {
     llvm::StringRef ModelName = A->getValue();
     auto RM = llvm::StringSwitch<llvm::Optional<llvm::Reloc::Model>>(ModelName)
                   .Case("static", llvm::Reloc::Static)
                   .Case("pic", llvm::Reloc::PIC_)
                   .Case("dynamic-no-pic", llvm::Reloc::DynamicNoPIC)

flang/lib/Frontend/FrontendActions.cpp

 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Bitcode/BitcodeWriterPass.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/IRReader/IRReader.h"
 #include "llvm/MC/TargetRegistry.h"
 #include "llvm/Passes/PassBuilder.h"
+#include "llvm/Passes/PassPlugin.h"
 #include "llvm/Passes/StandardInstrumentations.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Target/TargetMachine.h"
 #include <memory>
 
 using namespace Fortran::frontend;
 
static void generateMachineCodeOrAssemblyImpl(clang::DiagnosticsEngine &diags,
   }
 
   // Run the passes
   codeGenPasses.run(llvmModule);
 }
 
 void CodeGenAction::runOptimizationPipeline(llvm::raw_pwrite_stream &os) {
   auto opts = getInstance().getInvocation().getCodeGenOpts();
+  auto &diags = getInstance().getDiagnostics();
   llvm::OptimizationLevel level = mapToLevel(opts);
 
   // Create the analysis managers.
   llvm::LoopAnalysisManager lam;
   llvm::FunctionAnalysisManager fam;
   llvm::CGSCCAnalysisManager cgam;
   llvm::ModuleAnalysisManager mam;
 
   // Create the pass manager builder.
   llvm::PassInstrumentationCallbacks pic;
   llvm::PipelineTuningOptions pto;
   llvm::Optional<llvm::PGOOptions> pgoOpt;
   llvm::StandardInstrumentations si(opts.DebugPassManager);
   si.registerCallbacks(pic, &fam);
   llvm::PassBuilder pb(tm.get(), pto, pgoOpt, &pic);
 
+  // Attempt to load pass plugins and register their callbacks with PB.
+  for (auto &pluginFile : opts.LLVMPassPlugins) {
+    auto passPlugin = llvm::PassPlugin::Load(pluginFile);
+    if (passPlugin) {
+      passPlugin->registerPassBuilderCallbacks(pb);
+    } else {
+      diags.Report(clang::diag::err_fe_unable_to_load_plugin)
+          << pluginFile << passPlugin.takeError();
+    }
+  }
+
   // Register all the basic analyses with the managers.
   pb.registerModuleAnalyses(mam);
   pb.registerCGSCCAnalyses(cgam);
   pb.registerFunctionAnalyses(fam);
   pb.registerLoopAnalyses(lam);
   pb.crossRegisterProxies(lam, fam, cgam, mam);
 
   // Create the pass manager.

flang/test/CMakeLists.txt

set(FLANG_TEST_DEPENDS
   bbc
   llvm-dis
   llvm-objdump
   split-file
   FortranRuntime
   Fortran_main
   FortranDecimal
 )
+if (LLVM_ENABLE_PLUGINS)
+  list(APPEND FLANG_TEST_DEPENDS Bye)
+endif()
 
 if (FLANG_INCLUDE_TESTS)
   if (FLANG_GTEST_AVAIL)
     list(APPEND FLANG_TEST_DEPENDS FlangUnitTests)
   endif()
 endif()
 
 if (FLANG_BUILD_EXAMPLES)

flang/test/Driver/driver-help-hidden.f90

 ! CHECK-NEXT: -flarge-sizes          Use INTEGER(KIND=8) for the result type in size-related intrinsics
 ! CHECK-NEXT: -flogical-abbreviations Enable logical abbreviations
 ! CHECK-NEXT: -fno-automatic         Implies the SAVE attribute for non-automatic local objects in subprograms unless RECURSIVE
 ! CHECK-NEXT: -fno-color-diagnostics  Disable colors in diagnostics
 ! CHECK-NEXT: -fno-integrated-as     Disable the integrated assembler
 ! CHECK-NEXT: -fno-signed-zeros      Allow optimizations that ignore the sign of floating point zeros
 ! CHECK-NEXT: -fopenacc              Enable OpenACC
 ! CHECK-NEXT: -fopenmp               Parse OpenMP pragmas and generate parallel code.
+! CHECK-NEXT: -fpass-plugin=<dsopath> Load pass plugin from a dynamic shared object file (only with new pass manager).
 ! CHECK-NEXT: -freciprocal-math      Allow division operations to be reassociated
 ! CHECK-NEXT: -fsyntax-only          Run the preprocessor, parser and semantic analysis stages
 ! CHECK-NEXT: -fxor-operator         Enable .XOR. as a synonym of .NEQV.
 ! CHECK-NEXT: -help     Display available options
 ! CHECK-NEXT: -I <dir>               Add directory to the end of the list of include search paths
 ! CHECK-NEXT: -mllvm <value>         Additional arguments to forward to LLVM's option processing
 ! CHECK-NEXT: -mmlir <value>         Additional arguments to forward to MLIR's option processing
 ! CHECK-NEXT: -module-dir <dir>      Put MODULE files in <dir>

 ! CHECK-NEXT: --version Print version information
 ! CHECK-NEXT: -W<warning>            Enable the specified warning
 ! CHECK-NEXT: -Xflang <arg>          Pass <arg> to the flang compiler
 
 ! ERROR-FLANG: error: unknown argument '-help-hidden'; did you mean '--help-hidden'?
 
 ! Frontend driver -help-hidden is not supported
 ! ERROR-FLANG-FC1: error: unknown argument: '{{.*}}'
-

flang/test/Driver/driver-help.f90

 ! HELP-NEXT: -flarge-sizes          Use INTEGER(KIND=8) for the result type in size-related intrinsics
 ! HELP-NEXT: -flogical-abbreviations Enable logical abbreviations
 ! HELP-NEXT: -fno-automatic         Implies the SAVE attribute for non-automatic local objects in subprograms unless RECURSIVE
 ! HELP-NEXT: -fno-color-diagnostics  Disable colors in diagnostics
 ! HELP-NEXT: -fno-integrated-as      Disable the integrated assembler
 ! HELP-NEXT: -fno-signed-zeros      Allow optimizations that ignore the sign of floating point zeros
 ! HELP-NEXT: -fopenacc              Enable OpenACC
 ! HELP-NEXT: -fopenmp               Parse OpenMP pragmas and generate parallel code.
+! HELP-NEXT: -fpass-plugin=<dsopath> Load pass plugin from a dynamic shared object file (only with new pass manager).
 ! HELP-NEXT: -freciprocal-math      Allow division operations to be reassociated
 ! HELP-NEXT: -fsyntax-only          Run the preprocessor, parser and semantic analysis stages
 ! HELP-NEXT: -fxor-operator         Enable .XOR. as a synonym of .NEQV.
 ! HELP-NEXT: -help                  Display available options
 ! HELP-NEXT: -I <dir>               Add directory to the end of the list of include search paths
 ! HELP-NEXT: -mllvm <value>         Additional arguments to forward to LLVM's option processing
 ! HELP-NEXT: -mmlir <value>         Additional arguments to forward to MLIR's option processing
 ! HELP-NEXT: -module-dir <dir>      Put MODULE files in <dir>

 ! HELP-FC1-NEXT: -fno-analyzed-objects-for-unparse
 ! HELP-FC1-NEXT:                        Do not use the analyzed objects when unparsing
 ! HELP-FC1-NEXT: -fno-automatic         Implies the SAVE attribute for non-automatic local objects in subprograms unless RECURSIVE
 ! HELP-FC1-NEXT: -fno-debug-pass-manager Disables debug printing for the new pass manager
 ! HELP-FC1-NEXT: -fno-reformat          Dump the cooked character stream in -E mode
 ! HELP-FC1-NEXT: -fno-signed-zeros      Allow optimizations that ignore the sign of floating point zeros
 ! HELP-FC1-NEXT: -fopenacc              Enable OpenACC
 ! HELP-FC1-NEXT: -fopenmp               Parse OpenMP pragmas and generate parallel code.
+! HELP-FC1-NEXT: -fpass-plugin=<dsopath> Load pass plugin from a dynamic shared object file (only with new pass manager).
 ! HELP-FC1-NEXT: -freciprocal-math      Allow division operations to be reassociated
 ! HELP-FC1-NEXT: -fsyntax-only          Run the preprocessor, parser and semantic analysis stages
 ! HELP-FC1-NEXT: -fxor-operator         Enable .XOR. as a synonym of .NEQV.
 ! HELP-FC1-NEXT: -help                  Display available options
 ! HELP-FC1-NEXT: -init-only             Only execute frontend initialization
 ! HELP-FC1-NEXT: -I <dir>               Add directory to the end of the list of include search paths
 ! HELP-FC1-NEXT: -load <dsopath>        Load the named plugin (dynamic shared object)
 ! HELP-FC1-NEXT: -menable-no-infs       Allow optimization to assume there are no infinities.

flang/test/Driver/frontend-forwarding.f90

 ! Test that flang-new forwards Flang frontend
 ! options to flang-new -fc1 as expected.
 
 ! RUN: %flang -fsyntax-only -### %s -o %t 2>&1 \
 ! RUN:     -finput-charset=utf-8 \
 ! RUN:     -fdefault-double-8 \
 ! RUN:     -fdefault-integer-8 \
 ! RUN:     -fdefault-real-8 \
 ! RUN:     -flarge-sizes \
 ! RUN:     -fconvert=little-endian \
 ! RUN:     -ffp-contract=fast \
 ! RUN:     -fno-honor-infinities \
 ! RUN:     -fno-honor-nans \
 ! RUN:     -fapprox-func \
 ! RUN:     -fno-signed-zeros \
 ! RUN:     -fassociative-math \
 ! RUN:     -freciprocal-math \
+! RUN:     -fpass-plugin=Bye%pluginext \
 ! RUN:     -mllvm -print-before-all\
 ! RUN:     -P \
 ! RUN:   | FileCheck %s
 
 ! CHECK: "-P"
 ! CHECK: "-finput-charset=utf-8"
 ! CHECK: "-fdefault-double-8"
 ! CHECK: "-fdefault-integer-8"
 ! CHECK: "-fdefault-real-8"
 ! CHECK: "-flarge-sizes"
 ! CHECK: "-ffp-contract=fast"
 ! CHECK: "-menable-no-infs"
 ! CHECK: "-menable-no-nans"
 ! CHECK: "-fapprox-func"
 ! CHECK: "-fno-signed-zeros"
 ! CHECK: "-mreassociate"
 ! CHECK: "-freciprocal-math"
 ! CHECK: "-fconvert=little-endian"
-! CHECK:  "-mllvm" "-print-before-all"
+! CHECK: "-fpass-plugin=Bye
+! CHECK: "-mllvm" "-print-before-all"

flang/test/Driver/pass-plugin-not-found.f90

+! Check the correct error diagnostic is reported when a pass plugin shared object isn't found
+
+! REQUIRES: plugins, shell
+
+! RUN: not %flang -fpass-plugin=X.Y %s 2>&1 | FileCheck %s --check-prefix=ERROR
+! RUN: not %flang_fc1 -emit-llvm -o /dev/null -fpass-plugin=X.Y %s 2>&1 | FileCheck %s --check-prefix=ERROR
+
+! ERROR: error: unable to load plugin 'X.Y': 'Could not load library 'X.Y': X.Y: cannot open shared object file: No such file or directory'

flang/test/Driver/pass-plugin.f90

+! Verify that the plugin passed to -fpass-plugin is loaded and run
+
+! UNSUPPORTED: system-windows
+
+! REQUIRES: plugins, shell
+
+! RUN: %flang -S %s -fpass-plugin=%llvmshlibdir/Bye%pluginext -Xflang -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -S %s -fpass-plugin=%llvmshlibdir/Bye%pluginext -fdebug-pass-manager -o /dev/null 2>&1 | FileCheck %s
+
+! CHECK: Running pass: {{.*}}Bye on empty_
+
+subroutine empty
+end subroutine empty