Page MenuHomePhabricator

[MLIR, OpenMP] Translation of OpenMP barrier construct to LLVM IR
ClosedPublic

Authored by kiranchandramohan on Jan 17 2020, 3:34 PM.

Details

Summary

This patch adds support for translation of the OpenMP barrier construct to LLVM IR. The OpenMP IRBuilder is used for this translation. In this patch the code for translation is added to the existing LLVM dialect translation to LLVM IR.

The patch includes code changes and a testcase.

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
kiranchandramohan retitled this revision from Translation of OpenMP barrier construct to LLVM IR to [MLIR, OpenMP] Translation of OpenMP barrier construct to LLVM IR.Jan 17 2020, 3:34 PM
kiranchandramohan added a reviewer: ftynse.

Unit tests: unknown.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt

jdoerfert added inline comments.Jan 17 2020, 4:01 PM
mlir/lib/Target/LLVMIR/ConvertToOpenMPLLVMIR.cpp
62 ↗(On Diff #238903)

Need to be initialized to nullptr if you want to be allowed to call postLLVMTranslation before postLLVMModuleCreation.

Made changes suggested by the clang-format bot.

rriddle requested changes to this revision.Jan 17 2020, 4:28 PM
rriddle added inline comments.
mlir/include/mlir/Target/OpenMPLLVMIR.h
32 ↗(On Diff #238919)

I think this revision highlights a problem with the way the LLVM converter(or at least the way it is being used) is structured, mainly that it isn't extensible. I would imagine that lowering support for OpenMP would be an (optional) extension to the LLVM converter, not an entirely different one.

mlir/lib/Target/LLVMIR/ConvertToOpenMPLLVMIR.cpp
61 ↗(On Diff #238919)

I would imagine this would be a unique_ptr instead of a raw pointer.

This revision now requires changes to proceed.Jan 17 2020, 4:28 PM
kiranchandramohan edited the summary of this revision. (Show Details)

In this patch the code for translation is added to the existing LLVM dialect translation to LLVM IR. A new flag enable-openmp is used to enable the OpenMP translation.

Had to update makefiles of couple of toy examples and a tool to fix linking issues since the LLVM translator needs the OpenMP libraries.

Also changing the Summary to reflect this patch. Note that the previous version had the translation for OpenMP in a separate class and file.

kiranchandramohan marked an inline comment as done.Jan 23 2020, 11:53 AM

I have taken @rriddle's suggestion and moved the translation into the LLVM translator. Also using unique pointer for the OpenMPIRBuilder. I believe this fixes the null pointer issue pointed out by @jdoerfert.

No need to wait for me on this review but I added two more comments.

mlir/lib/Target/LLVMIR/ConvertToLLVMIR.cpp
25 ↗(On Diff #239961)

You could just check if OMPBuilder is initialized in order to keep this cmd line option private to a single file. Arguably, you could also create a method getOMPBuilder() which create one if none is present as soon as an OpenMP op is discovered. Maybe I'm missing some intended interaction of OpenMP ops and this flag set to false?

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
382

Nit: If you want it less verbose, OMPBuilder->CreateBarrier(builder.saveIP(), llvm::omp::OMPD_barrier); should work too.

mehdi_amini added inline comments.Jan 25 2020, 7:46 PM
mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h
29

cl::opt are only intended for testing, I wouldn't expect it in library code.

mlir/lib/Target/LLVMIR/ConvertToLLVMIR.cpp
25 ↗(On Diff #239961)

Lazily creating the OMPBuilder on demand seems like ideal to me? It there a downside?

What about always creating it eagerly: is it that heavy/costly?

jdoerfert added inline comments.Jan 25 2020, 7:55 PM
mlir/lib/Target/LLVMIR/ConvertToLLVMIR.cpp
25 ↗(On Diff #239961)

Lazily creating the OMPBuilder on demand seems like ideal to me? It there a downside?

I can't think of one.

What about always creating it eagerly: is it that heavy/costly?

No it's not. It create a couple of types, incl. function and struct types, but that's it.

The enable-openmp flag is now local to file mlir/lib/Target/LLVMIR/ModuleTranslation.cpp. Added a function to lazily create the OpenMPIR builder when an OpenMP dialect op is detected.

Changes to address review comments from @jdoerfert and @mehdi_amini.

kiranchandramohan marked 6 inline comments as done.Jan 27 2020, 1:52 AM
kiranchandramohan added inline comments.
mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h
29

The cl::opt flag is now only visible in mlir/lib/Target/LLVMIR/ModuleTranslation.cpp. Is that what you suggested?

ftynse requested changes to this revision.Jan 27 2020, 2:25 AM

Thanks! I have a couple of comments.

mlir/examples/toy/Ch6/CMakeLists.txt
55 ↗(On Diff #240496)

Why are these changes necessary? The code doesn't seem to be modifying the tutorial.

mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h
102

Please add a comment for this. "Original and translated module" does not describe it appropriately.

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
379

opInst.getDialect() == context->getRegisteredDialect<OpenMPDialect>() looks cleaner to me, even if it does the same comparison under the hood.

381

isa<omp::BarrierOp>

385

you cannot have non-OpenMP operation here, the outermost if checked that it belongs to the OpenMP dialect.

387

Nit: I'd provide more specific error message in case OpenMP is not enabled. Currenty, we will just fail to translate with the generic "unsupported or non-LLVM operation".

672

This should not be controlled by a command-line flag. It makes this flow unusable as a library (e.g., if the integration does not have a command-line interface). Pass the option to the constructor of ModuleTranslation (or whatever the way it is currently created) instead and default-initialize it from the command-line flag.

mlir/tools/mlir-cpu-runner/CMakeLists.txt
9 ↗(On Diff #240496)

Do we really need whole archive link for LLVMFrontendOpenMP, i.e. is there some static registration in that library?

This revision now requires changes to proceed.Jan 27 2020, 2:25 AM
jdoerfert added inline comments.Jan 27 2020, 1:32 PM
mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
672

I don't think we need the switch to begin with. If "omp" ops are supposed to be lowered we need a OMPBuilder. There is no reason not to build one and run into an assertion instead. With the lazy generation of the OMPBuilder we can really just remove the command line flag and get sane behavior w/ and w/o OpenMP ops.

mlir/tools/mlir-cpu-runner/CMakeLists.txt
9 ↗(On Diff #240496)
i.e. is there some static registration in that library?

Should not be.

kiranchandramohan marked 9 inline comments as done.Jan 28 2020, 7:42 AM

Thanks @ftynse for the comments. I will make a patch soon to address comments. The only question is regarding the necessity of the flag. Please see comment inline.

mlir/examples/toy/Ch6/CMakeLists.txt
55 ↗(On Diff #240496)

Are you asking specifically for the whole-archive link or for target_link_libraries also?

I was getting some link errors since the OpenMP symbols are needed now.

mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h
102

OK will do.

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
379

Thanks. Will do.

381

Will do.

385

Will change.

387

OK.

672

I think one of the suggestions early on was that if there is no OpenMP support (indicated via the switch/flag) then someone can try to generate code with other parallel libraries (like pthreads) or maybe run sequentially (is that possible? would it be equal to ignoring the openmp op).

For now, is it OK to not have this flag as is being suggested by @jdoerfert ?

mlir/tools/mlir-cpu-runner/CMakeLists.txt
9 ↗(On Diff #240496)

I will check and remove if not needed.

jdoerfert added inline comments.Feb 12 2020, 9:32 AM
mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
672

FWIW, running OpenMP programs "sequentially", in the sense that you *ignore* OpenMP, is only sound in *special cases*. Various OpenMP directives change the program semantics (at least potentially) such that ignoring OpenMP directives is not a strategy we should recommend (and we most certainly cannot "support" it in a sane way). You can run them "sequentially" by setting the appropriate upper bounds to 1 though.

MLIROpenMP and LLVMFrontendOpenMP libraries looks like an additional dependency for MLIR users who wants LLVM lowering and doesn't want anything to do with the OpenMP! I think Toy example cmake file changes is one of the early symptoms.

Can't OpenMP dialect have a separate LLVM dialect lowering similar to all other dialects in MLIR? I think it makes sense to lower OpenMP constructs in LLVM dialect rather than having a OpenMPIRBuilder based on LLVM IR.

What happens in case of LLM IR + OpenMP to LLVM dialect conversion? OpenMP dialect will be created too? Or else the LLVM IR to/from LLVM dialect (+ OpenMP dialect) will behave differently!

Thanks for your comments.

MLIROpenMP and LLVMFrontendOpenMP libraries looks like an additional dependency for MLIR users who wants LLVM lowering and doesn't want anything to do with the OpenMP! I think Toy example cmake file changes is one of the early symptoms.

One way to handle that would be to go back to the first version of this patch which implements translation by deriving from public LLVM::ModuleTranslation.

Can't OpenMP dialect have a separate LLVM dialect lowering similar to all other dialects in MLIR? I think it makes sense to lower OpenMP constructs in LLVM dialect rather than having a OpenMPIRBuilder based on LLVM IR.

We decided to use the OpenMPIRBuilder since that way we will be able to reuse existing code in the llvm-project (clang) for OpenMP code generation.

What happens in case of LLM IR + OpenMP to LLVM dialect conversion? OpenMP dialect will be created too? Or else the LLVM IR to/from LLVM dialect (+ OpenMP dialect) will behave differently!

I believe we can support this option under a flag. With the flag OpenMP + LLVM dialect is created. Without the flag only LLVM dialect will be created.

Thanks for your comments.

MLIROpenMP and LLVMFrontendOpenMP libraries looks like an additional dependency for MLIR users who wants LLVM lowering and doesn't want anything to do with the OpenMP! I think Toy example cmake file changes is one of the early symptoms.

One way to handle that would be to go back to the first version of this patch which implements translation by deriving from public LLVM::ModuleTranslation.

I think conversion patterns would be more appropriate. But, I think, that is dependent on the decision to emit LLVM IR instead of LLVM Dialect.

Can't OpenMP dialect have a separate LLVM dialect lowering similar to all other dialects in MLIR? I think it makes sense to lower OpenMP constructs in LLVM dialect rather than having a OpenMPIRBuilder based on LLVM IR.

We decided to use the OpenMPIRBuilder since that way we will be able to reuse existing code in the llvm-project (clang) for OpenMP code generation.

Given that MLIR has potential to support various parellel IR semantics as different dialects, I think having an LLVM dilaect lowering for OpenMP would be much cleaner solution. Also, shouldn't there be minimum difference between the LLVM Dialect and LLVM IR. If I am not wrong the goal (or atleast it definitely makes sense) is to have a tablegen based LLVM Dialect to LLVM IR conversion and viceversa. Having OpenMP dialect would affect it?

What happens in case of LLM IR + OpenMP to LLVM dialect conversion? OpenMP dialect will be created too? Or else the LLVM IR to/from LLVM dialect (+ OpenMP dialect) will behave differently!

I believe we can support this option under a flag. With the flag OpenMP + LLVM dialect is created. Without the flag only LLVM dialect will be created.

But, would it be possible to extract the OpenMP operations from the LLVM IR? I think that is a difficult problem to solve, especially when the IR is optimized. Also, I dont think it can be under a flag as conversion should be round-trippable?

Thanks for your comments.

MLIROpenMP and LLVMFrontendOpenMP libraries looks like an additional dependency for MLIR users who wants LLVM lowering and doesn't want anything to do with the OpenMP! I think Toy example cmake file changes is one of the early symptoms.

One way to handle that would be to go back to the first version of this patch which implements translation by deriving from public LLVM::ModuleTranslation.

I think conversion patterns would be more appropriate. But, I think, that is dependent on the decision to emit LLVM IR instead of LLVM Dialect.

OK. I was pointing out that if we do not want these dependencies then we have a way to achieve that by deriving from the LLVM Module translator.

Can't OpenMP dialect have a separate LLVM dialect lowering similar to all other dialects in MLIR? I think it makes sense to lower OpenMP constructs in LLVM dialect rather than having a OpenMPIRBuilder based on LLVM IR.

We decided to use the OpenMPIRBuilder since that way we will be able to reuse existing code in the llvm-project (clang) for OpenMP code generation.

Given that MLIR has potential to support various parellel IR semantics as different dialects, I think having an LLVM dialect lowering for OpenMP would be much cleaner solution. Also, shouldn't there be minimum difference between the LLVM Dialect and LLVM IR. If I am not wrong the goal (or atleast it definitely makes sense) is to have a tablegen based LLVM Dialect to LLVM IR conversion and viceversa. Having OpenMP dialect would affect it?

LLVM dialect and its conversion will not be affected by the OpenMP dialect or the changes in the translation library to lower the dialect. The OpenMP translation will only kick in if there are OpenMP dialect operations during translation.

We did consider lowering to the LLVM dialect as can bee seen in Slide 8 of the presentation (link below) and my first mail to mlir list (link below, see also replies from mlir developers). But given that there is the OpenMPIRBuilder (which provides a good interface to generate LLVM IR for OpenMP constructs), we felt it would be best to use the OpenMPIRBuilder rather than to re-implement that work in MLIR.
https://drive.google.com/file/d/1vU6LsblsUYGA35B_3y9PmBvtKOTXj1Fu/view?usp=sharing
https://groups.google.com/a/tensorflow.org/d/msg/mlir/4Aj_eawdHiw/BSpWvR6zCAAJ

If there was no OpenMP codegen present in LLVM then I would have also have preferred to lower to LLVM dialect. Maybe in future when Clang also uses MLIR, we can modify OpenMPIRBuilder to generate LLVM dialect.

What happens in case of LLM IR + OpenMP to LLVM dialect conversion? OpenMP dialect will be created too? Or else the LLVM IR to/from LLVM dialect (+ OpenMP dialect) will behave differently!

I believe we can support this option under a flag. With the flag OpenMP + LLVM dialect is created. Without the flag only LLVM dialect will be created.

But, would it be possible to extract the OpenMP operations from the LLVM IR? I think that is a difficult problem to solve, especially when the IR is optimized. Also, I dont think it can be under a flag as conversion should be round-trippable?

Yes, this will be a more difficult problem to solve.

We decided to use the OpenMPIRBuilder since that way we will be able to reuse existing code in the llvm-project (clang) for OpenMP code generation.

Given that MLIR has potential to support various parellel IR semantics as different dialects, I think having an LLVM dilaect lowering for OpenMP would be much cleaner solution.

I agree that having dialect lowering would be cleaner, but I don't get the relationship with "support various parallel IR semantics as different dialects"?

Also, shouldn't there be minimum difference between the LLVM Dialect and LLVM IR. If I am not wrong the goal (or atleast it definitely makes sense) is to have a tablegen based LLVM Dialect to LLVM IR conversion and viceversa. Having OpenMP dialect would affect it?

It wouldn't: as far as I understand every concept in LLVM can be represented in the LLVM dialect and vice-versa.

(This likely need a rebase)

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
672

For now this is fine with me to skip the flag entirely.

sri added a subscriber: sri.Feb 14 2020, 11:30 AM
schweitz added inline comments.
mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
672

There are several design choices one could try here.

Enable/disable OpenMP by:

  • some switch in the front-end (throws away the directives, very early)
  • creating/not creating the OpenMP dialect ops
  • keeping/erasing the OpenMP dialect ops (before the LLVMIR dialect)
  • translation modes of OpenMP dialect to LLVMIR dialect
  • maybe even closer to OpenMPIRBuilder (late, some sort of target support pushback?)

There is most likely trade offs here in terms of "user experience" and reporting diagnostics as to where and how to balance these approaches. Although supporting the "disabled vs. enabled OpenMP" use cases does seem like one worth providing with some mechanism.

kiranchandramohan updated this revision to Diff 245465.EditedFeb 19 2020, 10:54 AM
kiranchandramohan edited the summary of this revision. (Show Details)

This revision includes the following changes,
-> Rebased.
-> Removed the enable-openmp flag.
-> Addressed review comments from @ftynse.

Thanks @schweitz for your input. I have created a project note in https://github.com/orgs/flang-compiler/projects/9 to investigate the various possibilities that you have suggested.

kiranchandramohan marked 8 inline comments as done.Feb 19 2020, 11:00 AM
kiranchandramohan added inline comments.
mlir/examples/toy/Ch6/CMakeLists.txt
55 ↗(On Diff #240496)

As mentioned before, this was required to the dependence of MLIRTranslation on LLVMFrontendOpenMP.

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
672

Have removed the flag for now.

mlir/tools/mlir-cpu-runner/CMakeLists.txt
9 ↗(On Diff #240496)

I had to add LLVMFrontendOpenMP to FULL_LIBS used in whole_archive_link in mlir/tools/mlir-translate/CMakeLists.txt since libMLIRTranslation appears after LLVMFrontendOpenMP in the link line.

mehdi_amini added inline comments.Feb 19 2020, 9:50 PM
mlir/examples/toy/Ch6/CMakeLists.txt
55 ↗(On Diff #240496)

This seems like a missing transitive dependency: it does not seem right to add this here. The dependency on LLVMFrontendOpenMP needs to be specified on the library that is depending on it instead of pushed to every client (in this case I believe MLIRLLVMIR)

Added LLVMFrontendOpenMP and MLIROpenMP to the dependency of MLIRLLVMIR and also linking these libraries along with MLIRLLVMIR. Removed from other clients.

@mehdi_amini hope this is what you suggested. Thanks.

kiranchandramohan marked an inline comment as done.Feb 20 2020, 1:43 PM
mehdi_amini accepted this revision.Feb 20 2020, 2:35 PM

Do @rriddle and @ftynse require further changes?

ftynse accepted this revision.Feb 21 2020, 4:32 AM

Just wanted to check whether @rriddle has more comments on this patch.

rriddle accepted this revision.Mar 3 2020, 9:37 AM
rriddle added inline comments.
mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h
107

Variables names should be camelCase.

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
379

When adding more OMP operations(i.e. in a followup) can we split translating these operations into a different function? This one is already getting quite large.

380

We should really pre-compute this. Looking up a registered dialect involves grabbing a lock, and iterating O(N) over all dialects.

This revision is now accepted and ready to land.Mar 3 2020, 9:37 AM
kiranchandramohan marked 4 inline comments as done.Mar 5 2020, 3:51 AM

Thanks for the reviews. Committing with changes suggested.

mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
379

Will move to a new function for the next operation.

This revision was automatically updated to reflect the committed changes.
kiranchandramohan marked an inline comment as done.