[OpenMP][Docs] Added offloading command line reference to OpenMP FAQ

[AntonRydahl]

Summary

I have added a few things to the OpenMP FAQ which I think were missing. Feel free to suggest some changes. Are there missing options in the offloading command line reference? And what do you think about the section "Q: Why is my
build taking a long time"?

[jplehr]

Thanks for putting this together!

I had some minor comments.

openmp/docs/SupportAndFAQ.rst

84

I think this should be capability

477–478

Flip the flags or the sentences, so that the flags and their meaning are in the same order.

482

have in this and the next paragraph also "Pass an argument `<arg>` ... " as it is written further down.

506

I assume LTO is not actually running on the device, so maybe rephrase to

... optimization for device code.

527

Should be indicated which one is the default, i.e., old or new?

[AntonRydahl]

Thanks for the nice suggestions! I will fix it right away.

openmp/docs/SupportAndFAQ.rst

477–478

Thank you for pointing this out! I thought the two flags had the same meaning as they had the same description in the Clang command line reference.

clang --help | grep offload-host-device
  --offload-host-device   Only compile for the offloading host.
clang --help | grep offload-host-only
  --offload-host-only     Only compile for the offloading host.

What exactly is the difference between the two flags?

[jplehr]

openmp/docs/SupportAndFAQ.rst

477–478

To be honest, I only guessed from their name, hopefully someone else knows. My interpretation:
`--offload-host-device` -> Compile for host and device
`--offload-host-only` -> Compile for host only

Let's try summoning @jdoerfert or @jhuber6

[AntonRydahl]

openmp/docs/SupportAndFAQ.rst

527

In clang/lib/Driver/ToolChains/Clang.cpp the following means that it is turned off by default, right?

6343-  if (Args.hasFlag(options::OPT_offload_new_driver,
6344-                   options::OPT_no_offload_new_driver, false))
6345:    CmdArgs.push_back("--offload-new-driver");

[jdoerfert]

openmp/docs/SupportAndFAQ.rst

462

Add sth like: Note that this option is often not needed anymore if `--offload-arch is provided.

468

Mention auto detection (for the machine used for compiling) and mention {amdgpu,nvptx}-arch executables together with the "key" that corresponds to the sub architecture.

473

This is not what this does. It will compile only the code that goes on the device, not the code for the host. Mention this is for debug purposes mostly, or if device only runtimes are created.

504–507

Just make arg above optional and say what the default is [=<arg>] ...

513

I would not propose this to verify anything, honestly. Actually, mention that this is not to verify and mention how you should verify (with the env var set to mandatory). Instead, this is to avoid the host fallback which can help if the target contains code that cannot be compiled for the host (like unguarded device intrinsics), or if you want to save compile time.

533–535

@jhuber6, am I correct in assuming there is no "old" driver for OpenMP, and we should tell people this is only for HIP/CUDA? (even there it seems silly, this is probably not of use right now)

[jhuber6]

Thinking we should put these in a separate file and have the FAQ link to it.

openmp/docs/SupportAndFAQ.rst

89

Unrelated, but this section definitely needs to be updated either by me or @JonChesterfield.

468

Add a line showing that --offload-arch=sm_80,gfx90a does both.

472

Maybe worth mentioning that this is primarily used for checking the IR output when compiling for the device.

477–478

--offload-host-only skips the device portion, has the effect of not having the .llvm.offloading section where the embedded device code would otherwise live, check the IR and you'll see the lack of a giant binary string.

482

You can also use -Xarch_device and -Xarch_host which are worth mentioning.

527

Upstream, OpenMP only uses the new driver. This flag is for enabling it for CUDA / HIP so that it can be interoperable w/ OpenMP.

[jdoerfert]

openmp/docs/SupportAndFAQ.rst

518–521

Remove the negative option, describe the bits (or link to the docs) that you can set and that you need the env var too.

525

Not can be, *is*. So: Embed LLVM-IR for the device code in the object files rather than binary code for the respective target. At runtime the LLVM-IR is optimized again and compiled for the target device. Link to the JIT options, and mention this is good for debugging.

533–535

So, let's just talk about the positive option right now.

541

This is default for OpenMP, right (@jhuber6)?

[AntonRydahl]

This update should hopefully resolve most of your comments. Let me know if the changes do not meet your expectations.

[AntonRydahl]

I have tried my best to incorporate your feedback. If you want to see the HTML, it looks like this

Support, Getting Involved, and FAQ — LLVM_OpenMP documentation.pdf389 KBDownload

I want to move the offloading command line reference to another file, but I think that would delete all of your comments.

[AntonRydahl]

Removed section about --no-offload-new-driver

[jdoerfert]

I'm in favor of updating it, extracting into a new file, and then improving as people see things.

openmp/docs/SupportAndFAQ.rst

561

I doubt it is -O3, probably 3

565

[AntonRydahl]

openmp/docs/SupportAndFAQ.rst

561

You are right, I just misunderstood the documentation. I will fix it right away.

[AntonRydahl]

Moved command-line reference to a separate RST file.

[jhuber6]

openmp/docs/CommandLineArgumentReference.rst

13 ↗

(On Diff #545019)

I think these are supposed to use ---- for a subsection.

[AntonRydahl]

Changed subtitles to subsections and noted that --offload-arch is automatically detected if not specified.

openmp/docs/CommandLineArgumentReference.rst

13 ↗

(On Diff #545019)

Thanks, you are right as always!

[jhuber6]

openmp/docs/CommandLineArgumentReference.rst

54 ↗

(On Diff #545217)

Maybe mention CPU offloading here, like with -fopenmp-targets=x86_64-pc-linux-gnu.

65–66 ↗

(On Diff #545217)

Drop the bin. Also we infer with --offload-arch=native as well.

184 ↗

(On Diff #545217)

Should be fixed

openmp/docs/SupportAndFAQ.rst

84

Wait I think this is totally wrong, I removed that in 17. It's LIBOMPTARGET_DEVICE_ARCHITECTURES=sm_70,gfx1030 now for example.

[AntonRydahl]

Added new lines to the end of the files and included `LIBOMPTARGET_DEVICE_ARCHITECTURES`.

[AntonRydahl]

Added note on offloading to CPUS.

[jhuber6]

Thanks a lot for updating this documentation, appreciate it.

[AntonRydahl]

In D156387#4543386, @jhuber6 wrote:

Thanks a lot for updating this documentation, appreciate it.

Thanks a lot for helping me improving it! :D

[AntonRydahl]

Resetting the patch to the version of the patch that was accepted.

openmp/docs/SupportAndFAQ.rst

 <https://llvm.org/docs/Contributing.html#how-to-submit-a-patch>`_.
 
 
 .. _build_offload_capable_compiler:
 
 Q: How to build an OpenMP GPU offload capable compiler?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 To build an *effective* OpenMP offload capable compiler, only one extra CMake
-option, `LLVM_ENABLE_RUNTIMES="openmp"`, is needed when building LLVM (Generic
+option, ``LLVM_ENABLE_RUNTIMES="openmp"``, is needed when building LLVM (Generic
 information about building LLVM is available `here
-<https://llvm.org/docs/GettingStarted.html>`__.).  Make sure all backends that
-are targeted by OpenMP to be enabled. By default, Clang will be built with all
-backends enabled.  When building with `LLVM_ENABLE_RUNTIMES="openmp"` OpenMP
-should not be enabled in `LLVM_ENABLE_PROJECTS` because it is enabled by
-default.
+<https://llvm.org/docs/GettingStarted.html>`__.). Make sure all backends that
+are targeted by OpenMP are enabled. That can be done by adjusting the CMake 
+option ``LLVM_TARGETS_TO_BUILD``. The corresponding targets for offloading to AMD 
+and Nvidia GPUs are ``"AMDGPU"`` and ``"NVPTX"``, respectively. By default, 
+Clang will be built with all backends enabled. When building with 
+``LLVM_ENABLE_RUNTIMES="openmp"`` OpenMP should not be enabled in 
+``LLVM_ENABLE_PROJECTS`` because it is enabled by default.
 
 For Nvidia offload, please see :ref:`build_nvidia_offload_capable_compiler`.
 For AMDGPU offload, please see :ref:`build_amdgpu_offload_capable_compiler`.
 
 .. note::
   The compiler that generates the offload code should be the same (version) as
   the compiler that builds the OpenMP device runtimes. The OpenMP host runtime
   can be built by a different compiler.
 
 .. _advanced_builds: https://llvm.org//docs/AdvancedBuilds.html
 
 .. _build_nvidia_offload_capable_compiler:
 
-Q: How to build an OpenMP NVidia offload capable compiler?
+Q: How to build an OpenMP Nvidia offload capable compiler?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 The Cuda SDK is required on the machine that will execute the openmp application.
 
 If your build machine is not the target machine or automatic detection of the
 available GPUs failed, you should also set:
 
-- `LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=YY` where `YY` is the numeric compute capacity of your GPU, e.g., 75.
+- ``LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=YY`` where ``YY`` is the numeric compute capacity of your GPU, e.g., 75.
 
 
 .. _build_amdgpu_offload_capable_compiler:
 
 Q: How to build an OpenMP AMDGPU offload capable compiler?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 A subset of the `ROCm <https://github.com/radeonopencompute>`_ toolchain is
 required to build the LLVM toolchain and to execute the openmp application.
 Either install ROCm somewhere that cmake's find_package can locate it, or
 build the required subcomponents ROCt and ROCr from source.
 
 The two components used are ROCT-Thunk-Interface, roct, and ROCR-Runtime, rocr.
 Roct is the userspace part of the linux driver. It calls into the driver which

 targets, such as offloading to AMD and NVIDIA GPUs simultaneously, as well as
 multiple sub-architectures for the same target. Additionally, static libraries
 will only extract archive members if an architecture is used, allowing users to
 create generic libraries.
 
 The architecture can either be specified manually using ``--offload-arch=``. If
 ``--offload-arch=`` is present no ``-fopenmp-targets=`` flag is present then the
 targets will be inferred from the architectures. Conversely, if
-``--fopenmp-targets=`` is present with no ``--offload-arch``  then the target
+``--fopenmp-targets=`` is present with no ``--offload-arch`` then the target
 architecture will be set to a default value, usually the architecture supported
 by the system LLVM was built on.
 
 For example, an executable can be built that runs on AMDGPU and NVIDIA hardware
 given that the necessary build tools are installed for both.
 
 .. code-block:: shell
 

 with OpenMP.
 
 .. code-block:: shell
 
    clang++ openmp.cpp -fopenmp --offload-arch=gfx90a -lcgpu
 
 For more information on how this is implemented in LLVM/OpenMP's offloading 
 runtime, refer to the `runtime documentation <libomptarget_libc>`_.
+
+Q: What command line options can I use for OpenMP offloading?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+``-fopenmp-targets``
+""""""""""""""""""""
+Specify which OpenMP offloading targets should be supported. For example, you 
+may specify ``-fopenmp-targets=amdgcn-amd-amdhsa,nvptx-none``.
+
+``--offload-arch``
+""""""""""""""""""
+Specify the device architecture for OpenMP offloading. For instance 
+``--offload-arch=sm_80`` to target an Nvidia Tesla A100 or 
+``--offload-arch=gfx90a`` to target an AMD Instinct MI250X.
+
+``--offload-device-only``
+"""""""""""""""""""""""""
+Compile the target regions for the device only. All target regions will be 
+compiled for both host and device if not specified.
+
+``--offload-host-device`` or ``--offload-host-only``
+""""""""""""""""""""""""""""""""""""""""""""""""""""
+Compile the target regions for the host only. All target regions will be 
+compiled for both host and device if not specified.
+
+``-Xopenmp-target <arg>``
+"""""""""""""""""""""""""
+Pass an argument to the offloading toolchain, for instance 
+``-Xopenmp-target -march=sm_80``.
+
+``-Xopenmp-target=<triple> <arg>``
+""""""""""""""""""""""""""""""""""
+Pass an argument to the offloading toolchain for the triple. That is especially 
+useful when an argument must differ for each triple. For instance 
+``-Xopenmp-target=nvptx64 --offload-arch=sm_80 
+-Xopenmp-target=amdgcn --offload-arch=gfx90a`` to specify the device 
+architecture.
+
+``-Xoffload-linker<triple> <arg>``
+""""""""""""""""""""""""""""""""""
+Pass an argument ``<arg>`` to the offloading linker for the target specified in 
+``<triple>``.
+
+``-foffload-lto=<arg>``
+"""""""""""""""""""""""
+Enable device link time optimization (LTO) and select the LTO mode ``<arg>``. 
+Select either ``-foffload-lto=thin`` or ``-foffload-lto=full``. Thin LTO takes 
+less time while still achieving some performance gains.
+
+``-foffload-lto``
+"""""""""""""""""
+Enable ``full`` link time optimization on the device. This option is equivalent to 
+``-foffload-lto=full``. 
+
+``-fopenmp-offload-mandatory``
+""""""""""""""""""""""""""""""
+With this option enabled, a host fallback will not be created for a situation 
+when offloading to the device fails. An example use case of this option is to 
+verify that code is being offloaded to the device.
+
+``-fopenmp-target-debug``
+"""""""""""""""""""""""""
+Enable debugging in the device runtime library (RTL).
+
+``-fno-openmp-target-debug``
+""""""""""""""""""""""""""""
+Disable debugging in the device RTL.
+
+``-fopenmp-target-jit``
+"""""""""""""""""""""""
+Emit code that can be Just-in-Time (JIT) compiled for OpenMP offloading.
+
+``--offload-new-driver``
+""""""""""""""""""""""""
+Use the new driver for offloading compilation. OpenMP offloading can be 
+experimentally linked with CUDA and HIP files. That requires using the new 
+offloading driver.
+
+``--no-offload-new-driver``
+"""""""""""""""""""""""""""
+Do not use the new driver for offloading compilation.
+
+``--offload-link``
+""""""""""""""""""
+Use the new offloading linker to perform the link job. OpenMP offloading can be 
+experimentally linked with CUDA and HIP files. The new offloading linker must be
+used when linking with CUDA or HIP files.
+
+``-nogpulib``
+"""""""""""""
+Do not link the device library for CUDA or HIP device compilation.
+
+``-nogpuinc``
+"""""""""""""
+Do not include the default CUDA or HIP headers, and do not add CUDA or HIP
+include paths.
+
+Q: Why is my build taking a long time?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+When installing OpenMP and other LLVM components, the build time on multicore 
+systems can be significantly reduced with parallel build jobs. As suggested in 
+*LLVM Techniques, Tips, and Best Practices*, one could consider using `ninja` as the
+generator. This can be done with the CMake option `cmake -G Ninja`. Afterward, 
+use `ninja install` and specify the number of parallel jobs with `-j`. The build
+time can also be reduced by setting the build type to `Release ` with the 
+`CMAKE_BUILD_TYPE` option. Recompilation can also be sped up by caching previous
+compilations. Consider enabling `Ccache` with 
+`CMAKE_CXX_COMPILER_LAUNCHER=ccache`.
+
+Q: Did this FAQ not answer your question?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Feel free to post questions or browse old threads at 
+`LLVM Discourse <https://discourse.llvm.org/c/runtimes/openmp/>`__.
 No newline at end of file