This is an archive of the discontinued LLVM Phabricator instance.

[MLIR][OpenMP] Alterations to LLVM-IR lowering of OpenMP Dialect due to Bounds and MapEntry operations
ClosedPublic

Authored by agozillon on Aug 24 2023, 7:21 AM.

Details

Summary

This patch adjusts the lower to LLVM-IR inside of
OpenMPToLLVMIRTranslation to faciliate the changes made
to Target related operations to add the new Map related
operations. It also includes adjustments to tests to support
these changes, primarily modifying the MLIR as opposed to
the LLVM-IR, the LLVM-IR should be identical after this patch.

Depends on D158735

Diff Detail

Event Timeline

agozillon created this revision.Aug 24 2023, 7:21 AM
Herald added a project: Restricted Project. · View Herald Transcript
agozillon requested review of this revision.Aug 24 2023, 7:21 AM

Patch 4/4 of the map changes from the patch series made by: https://reviews.llvm.org/D158732, https://reviews.llvm.org/D158735, https://reviews.llvm.org/D158737 and https://reviews.llvm.org/D158734 that aims to expand the current map support of the OpenMP dialect and lower from Fortran -> MLIR -> LLVM IR, future support on expanding the lowering from the OpenMP dialect to LLVM IR for map on TargetOp (omp target) will be forthcoming for declare target and explicit map variables hopefully in the near future (implicit likely a while after that).

This patch (provided I've setup the patch dependencies correctly) should pass CI, the others in the series will not.

A small ping for some attention on this patch if a reviewer can spare some time, thank you very much!

razvanlupusoru accepted this revision.Aug 31 2023, 1:02 PM
razvanlupusoru added a subscriber: razvanlupusoru.

Looks reasonable to me based on the updated dialect design.

mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
2210

This seems reasonable to me - but where is DataBoundsOp actually being handled? Or is that not implemented just yet?

This revision is now accepted and ready to land.Aug 31 2023, 1:02 PM

Looks reasonable to me based on the updated dialect design.

Thank you very much for all your review points and input across the patch series, it's very appreciated.

Yes, this patch doesn't contain any altered lowering, it's just modifying the current status quo to support the dialect changes to map, the lowering will come in a later patch series provided this current series is successful. I thought 4 patches was already pushing it for all you poor reviewers (my apologies, and thank you all for the awesome work), as well as my own bandwidth!

mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
2210

The handling of it is currently not implemented in this patch series. However, currently in a downstream patch (which I'll seek to upstream in pieces after/if this series lands) the information will be used alongside other information stored inside of MapEntryOp when lowering target operations that hold MapEntryOps (for the moment and near future it's only TargetOp, with the other operations on the horizon).

For example, in TargetOp where we perform map processing (now MapEntryOp processing), the lowering we have in place primarily uses the DataBoundsOp information to aid with array sectioning, so we primarily use the defined extents to generate some LLVM-IR operations differently (e.g. offset the array start and only pass so many elements).

Of course, if you have any pointers, ideas or directions I'd be more than happy to take them into consideration!

agozillon updated this revision to Diff 556316.Sep 8 2023, 3:36 PM

rebase and update tests based on changes to the Map operations printing and parsing in seperate patch in patch series

razvanlupusoru accepted this revision.Sep 11 2023, 8:12 AM

@razvanlupusoru thank you very much for your review!

I will leave the patch series open until Thursday to give time for further comments/feedback from other reviewers (in particular @kiranchandramohan and @jsjodin) at that point if no further comments/change requests are received I will push it upstream!

TIFitis added inline comments.Sep 12 2023, 7:28 AM
mlir/test/Target/LLVMIR/omptarget-llvm.mlir
16

Can you please explain why the type is changing here?

agozillon added inline comments.Sep 12 2023, 8:37 AM
mlir/test/Target/LLVMIR/omptarget-llvm.mlir
16

For offloading basic arguments the OpenMP runtime appears to also need the OMP_MAP_TARGET_PARAM bits set (more complex arguments need a variety of other bits set), at least for target just the tofrom/from/to bits aren't enough, perhaps it's different for target_data? I don't think that's the case, but I am perhaps wrong, I've only focused on target!

The target_param bits are being set in the OpenMPToLLVMTranslation phase at the moment as the maps are being processed, but I'm trying to move the setting of these bits to the frontend (the initial PFT lowering of map) where they can be kept together and reasoned about a bit better.

So it's changed to better represent what the IR lowering will be, with the addition of the target_param bits! I believe, currently in upstream only convertTargetOmp has the bits set during genMapInfos (and I believe I should actually include the removal of that segment in these patches actually, rather than future ones in hindsight, so I'll do that in the next update) for the moment.

TIFitis added inline comments.Sep 12 2023, 11:06 AM
mlir/test/Target/LLVMIR/omptarget-llvm.mlir
16

As this affects correctness of generated code, I think we should be careful about making any changes that change the test output.

Can you please reason for why the map_type should be 35 here instead of 3 with examples from Clang perhaps? AFAIK the map_type should remain 3 here for this example. Similarly we might want to also look at the other tests with similar change.

agozillon added inline comments.Sep 12 2023, 5:49 PM
mlir/test/Target/LLVMIR/omptarget-llvm.mlir
16

I was very incorrect! Thank you very much for pointing this out. I focused a little too much on the target case. It appears for target_data the map_types remain 3 (or the respective base mapping) and do not get appended with target_param, so I think will restrict the application of target_param to just TargetOp related map operations as it currently is (just later in the pipeline) and it can be expanded as we increase map support if required, if that sounds like a reasonable change?

For an example of the map_type value for Clang for target directives, this small program for target, when used in conjunction with the usual command, clang++ --offload-host-device --offload-arch=gfx90a -c -fopenmp -S -emit-llvm out-ir.cpp -o out-ir.ll :

#include <iostream>

int main() {
  int i = 1;

#pragma omp target map(tofrom : i)
{
    i = 99;
}

  std::cout << i << std::endl;

  return 0;
}

Should emit a map_type of:

@.offload_maptypes = private unnamed_addr constant [1 x i64] [i64 35]

Currently, the change from 3 -> 35 (or 33/34 for to/from I believe) already occurs, but only for TargetOp's because of this change here: https://github.com/llvm/llvm-project/blob/main/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp#L1555

I'd like to move this to the frontend PFT lowering phase if possible, alongside where the rest of the map flags are currently set I believe, as I think it's likely better to keep them assigned in a similar location as best we can, so it's easier to reason about things (from the perspective of it mostly all being together and represented more clearly in the various stages of the IR). But perhaps I am wrong! :-)

It also could help to simplify this stage of the lowering a bit further, I think, for example, not all things need marked with OMP_TARGET_PARAM e.g. declare target variables don't require it, they require OMP_MAP_PTR_AND_OBJ instead, so we need to have a check that detects if something is declare target and then switch the appropriate bits. Implicit captures also need marked with OMP_IMPLICIT_MAP. Although, both of these checks can likely be done inside of the OpenMPToLLVMIRTranslation phase as well. I think it'd work perfectly fine in either location, at least from my current understanding. Although, at least some of the future added logic will still have to be inside of the OpenMPToLLVMIRTranslation phase as MEMBER_OF requires knowledge of the size of the combinedInfo arrays to set its bits (at least that appears to be how it's done in Clang).

Would you prefer us to continue setting the more specialized flags in the OpenMPToLLVMIRTranslation phase or does it sound reasonable to move what we can to the initial PFT lowering phase?

Thank you again for the great catch!

TIFitis added inline comments.Sep 13 2023, 5:47 AM
mlir/test/Target/LLVMIR/omptarget-llvm.mlir
16

They way I see it, lowering phase is supposed to just translate the information as-is from high level code to FIR/MLIR.

A lot of the flags present here that can be set are specifically for the kernel launch and I don't think any MLIR passes etc benefit from having that information, rather it might just bloat the MLIR. In fact for cases such as implicitly captured operands, use_device_ptr/addr, privatisation etc. it might be infeasible to have all the flags correctly set at the lowering phase itself.

Hence my opinion would be to only lower the flags which are explicitly present in the code, and we can add the context based flags later on when lowering from MLIR to llvm.

Also, you are right. genMapInfos isn't complete at this point. The intention was to support the bare minimum of target data and target directives. As new directives are added or support for existing directives are expanded, we should also update genMapInfos accordingly to support the new cases.

agozillon updated this revision to Diff 556780.Sep 14 2023, 6:08 AM

Update tests based on changes to the PFT lowering (no longer applies unneccesary flags, happens in later lowering in declare target patch and only for target as was happening before) and changes to the name of the operation and parsing/printing of the clauses.

TIFitis accepted this revision.Sep 18 2023, 5:15 AM

LGTM :)

mlir/test/Target/LLVMIR/omptarget-llvm.mlir
56

Nit: Can you please change the map type to from to prevent the change in test output here.

No need to push another review, feel free to commit with the change made.

LGTM :)

Thank you for the reviews :-)

mlir/test/Target/LLVMIR/omptarget-llvm.mlir
56

I will do so! I may have done an incorrect find and replace that caught this map clause at some point, thank you for the catch.

@kiranchandramohan would it please be possible to get final acceptance on your end for each of the patches (https://reviews.llvm.org/D158732, https://reviews.llvm.org/D158734, https://reviews.llvm.org/D158735 and https://reviews.llvm.org/D158737) you've not signed off on if you're happy with them landing? (I have a minor buildbot error from what I believe is a missing change that I lost in a rebase that I plan on fixing today, but other than that no larger changes are forthcoming).

I'll land the patches after the additional acceptances provided no additional comments/change requests are made!

@kiranchandramohan would it please be possible to get final acceptance on your end for each of the patches (https://reviews.llvm.org/D158732, https://reviews.llvm.org/D158734, https://reviews.llvm.org/D158735 and https://reviews.llvm.org/D158737) you've not signed off on if you're happy with them landing? (I have a minor buildbot error from what I believe is a missing change that I lost in a rebase that I plan on fixing today, but other than that no larger changes are forthcoming).

I'll land the patches after the additional acceptances provided no additional comments/change requests are made!

OK. Please go ahead.

reverting the test back to its original form in most recent update as requested.