[flang][driver] Add support for -embed-offload-object flag in flang -fc1

[jsjodin]

Summary

This patch adds support for the -embed-offload-object flag to embed offloading binaries in host code. This flag is identical to the flag with the same name in clang.

[awarzynski]

Thanks for sending this, @jsjodin ! Overall, this makes sense. But I have no experience with offloading and I would appreciate some RFC that would explain what the overall direction. What are the next steps? And what's the overall goal?

Also, could add a note in the summary that would explain whether the semantics of this option in Flang are identical to Clang?

flang/lib/Frontend/FrontendActions.cpp

740–741

Could you add a test that will exercise this diagnostic?

[jsjodin]

Added negative test and early return when error occurrs.

[jsjodin]

In D142244#4071213, @awarzynski wrote:

Thanks for sending this, @jsjodin ! Overall, this makes sense. But I have no experience with offloading and I would appreciate some RFC that would explain what the overall direction. What are the next steps? And what's the overall goal?

I can give a short description. The over all goal is to support OpenMP in flang including target offloading. The direction is basically to implement the missing features in flang that are implied from the driver, which is shared with clang. Minimizing the difference between flang and clang is desirable to be able to share code as much as possible. For code generation, some work has already been done to move OpenMP code from clang to llvm for sharing implementations. For the flang driver the main work will be to implement the various flags for the commands that are being generated, and add support for the GPU targets. For example, this patch implements -embed-offload-object, which is an option that can already be generated by the flang driver if the -fopenmp and--offload-arch flags are provided (assuming there is a gpu target available, which will have to be added soon).
There is a presentation on the entire mechanism how the new offloading driver works here:
https://www.youtube.com/watch?v=4NnzymmQe7k
Slides here:
https://docs.google.com/presentation/d/1JG3GF7sqZ6lnl5kf1MrSW0-_k8FDrzaGk8hzGaq0K8k/edit?usp=sharing

Also, could add a note in the summary that would explain whether the semantics of this option in Flang are identical to Clang?

Sure, I will update this.

[jhuber6]

In D142244#4071213, @awarzynski wrote:

Thanks for sending this, @jsjodin ! Overall, this makes sense. But I have no experience with offloading and I would appreciate some RFC that would explain what the overall direction. What are the next steps? And what's the overall goal?

Also, could add a note in the summary that would explain whether the semantics of this option in Flang are identical to Clang?

This documentation is a little out-of-date, but is mostly correct, https://clang.llvm.org/docs/OffloadingDesign.html. I understand the desire is to replicate this handling very closely in Clang.

Can we get a test to show that this works when given straight LLVM-IR as well? Those are actually handled differently in Clang so I had to make sure that they both call the embedding function.

Also please add the metadata checks to the test, these are important as we check for the module metadata to identify these when the user specifies -flto. E.g. like this https://github.com/llvm/llvm-project/blob/main/clang/test/Frontend/embed-object.ll.

[jsjodin]

Added test to embed with .bc file as input instead of source file. Fixed comment in negative test.

[jhuber6]

In D142244#4080708, @jsjodin wrote:

Added test to embed with .bc file as input instead of source file. Fixed comment in negative test.

Oh yeah, does Flang accept .ll files? I don't know the details of Flang's driver model since it uses MLIR. But if it does we should have a .ll test that shows the metadata attached to the embedded object.

[jsjodin]

Update test to check metadata.

[jhuber6]

LGTM

flang/lib/Frontend/FrontendActions.cpp

736

Nit, Is this in style w/ Flang? Probably copied from Clang.

[jsjodin]

flang/lib/Frontend/FrontendActions.cpp

736

It definitely is from Clang, I was considering putting this in the OMPIRBuilder, but given the callback needed for error handling it didn't seem worth it. I wasn't aware that flang has a different style than clang.

[jhuber6]

flang/lib/Frontend/FrontendActions.cpp

736

Just looking around it seems like Flang prefers camelCase while Clang uses PascalCase. But I don't know the specific style rules. Usually clang-tidy will yell at you.

[jsjodin]

flang/lib/Frontend/FrontendActions.cpp

736

Yes, I will update the patch to do camel case.

[jsjodin]

Update variable names to camel case.

[jhuber6]

Do you know what's left on the Flang driver side to enable offloading?

[jsjodin]

@awarzynski do you have any other concerns about the patch?

[jsjodin]

In D142244#4081187, @jhuber6 wrote:

Do you know what's left on the Flang driver side to enable offloading?

I don't know exactly what is left. The Flang driver already uses the two stage compilation, so as far as I could tell generating llvm-ir there are no difference in the phases between clang and flang. If doing a full compilation with linking, then there seemed to be some issues with the various commands and there were some differences in the phases as well. My guess is that there is not a huge amount of work.

[awarzynski]

In D142244#4082934, @jsjodin wrote:

@awarzynski do you have any other concerns about the patch?

Sorry for the delay, I was traveling. This is very exciting, thanks for implementing it!

LGTM

In D142244#4074906, @jsjodin wrote:

There is a presentation on the entire mechanism how the new offloading driver works here:
https://www.youtube.com/watch?v=4NnzymmQe7k
Slides here:
https://docs.google.com/presentation/d/1JG3GF7sqZ6lnl5kf1MrSW0-_k8FDrzaGk8hzGaq0K8k/edit?usp=sharing

You could add one of these links in the driver docs. Or perhaps a link to https://clang.llvm.org/docs/OffloadingDesign.html?

flang/lib/Frontend/FrontendActions.cpp

736

The coding style in Flang is tricky business ;) The driver strives to follow MLIR: https://github.com/llvm/llvm-project/blob/main/flang/lib/Frontend/FrontendActions.cpp#L9 :) This is just for future reference.

flang/test/Driver/embed-error.f90

3–5

[nit] I used to add these comments, but then people complained and removed some of them. Feel free to keep them or to remove.

[jsjodin]

Merge with trunk so patch applies.

[clementval]

The new test embed.f90 and embed-error.f90 are failing on my side. Do they assume a specific configuration?

[jsjodin]

In D142244#4096492, @clementval wrote:

The new test embed.f90 and embed-error.f90 are failing on my side. Do they assume a specific configuration?

No, it should work for any configuration as far as I know. How are you running the test?

[clementval]

In D142244#4096512, @jsjodin wrote:

No, it should work for any configuration as far as I know. How are you running the test?

Just with check-flang

[jsjodin]

In D142244#4096538, @clementval wrote:

In D142244#4096512, @jsjodin wrote:

No, it should work for any configuration as far as I know. How are you running the test?

Just with check-flang

How does it fail? Can you show the output? Also what CPU are you running on?

[clementval]

In D142244#4096546, @jsjodin wrote:

In D142244#4096538, @clementval wrote:

In D142244#4096512, @jsjodin wrote:

No, it should work for any configuration as far as I know. How are you running the test?

Just with check-flang

How does it fail? Can you show the output? Also what CPU are you running on?

********************
FAIL: Flang :: Driver/embed.f90 (2 of 1993)
******************** TEST 'Flang :: Driver/embed.f90' FAILED ********************
Script:
--
: 'RUN: at line 6';   llvm-project/build/bin/flang-new -fc1 -emit-llvm -o - -fembed-offload-object=llvm-project/flang/test/Driver/Inputs/hello.f90 llvm-project/flang/test/Driver/embed.f90 2>&1 | llvm-project/build/bin/FileCheck llvm-project/flang/test/Driver/embed.f90
: 'RUN: at line 8';   llvm-project/build/bin/flang-new -fc1 -emit-llvm-bc -o llvm-project/build/tools/flang/test/Driver/Output/embed.f90.tmp.bc llvm-project/flang/test/Driver/embed.f90 2>&1
: 'RUN: at line 9';   llvm-project/build/bin/flang-new -fc1 -emit-llvm -o - -fembed-offload-object=llvm-project/flang/test/Driver/Inputs/hello.f90 llvm-project/build/tools/flang/test/Driver/Output/embed.f90.tmp.bc 2>&1 | llvm-project/build/bin/FileCheck llvm-project/flang/test/Driver/embed.f90
--
Exit Code: 1

Command Output (stderr):
--
llvm-project/flang/test/Driver/embed.f90:11:10: error: CHECK: expected string not found in input
! CHECK: @[[OBJECT_1:.+]] = private constant [61 x i8] c"program hello\0A write(*,*), \22Hello world!\22\0Aend program hello\0A", section ".llvm.offloading", align 8, !exclude !0
         ^
<stdin>:1:1: note: scanning from here
; ModuleID = 'FIRModule'
^

Input file: <stdin>
Check file: llvm-project/flang/test/Driver/embed.f90

-dump-input=help explains the following input dump.

Input was:
<<<<<<
          1: ; ModuleID = 'FIRModule' 
check:11     X~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
          2: source_filename = "FIRModule" 
check:11     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          3: target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128" 
check:11     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          4: target triple = "x86_64-unknown-linux-gnu" 
check:11     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          5:  
check:11     ~
          6: @_QFECi = internal constant i32 1 
check:11     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          .
          .
          .
>>>>>>

`
FAIL: Flang :: Driver/embed-error.f90 (1 of 1993)
******************** TEST 'Flang :: Driver/embed-error.f90' FAILED ********************
Script:
--
: 'RUN: at line 6';   not llvm-project/build/bin/flang-new -fc1 -emit-llvm -o - -fembed-offload-object=llvm-project/flang/test/Driver/Inputs/missing.f90 llvm-project/flang/test/Driver/embed-error.f90 2>&1 | llvm-project/build/bin/FileCheck llvm-project/flang/test/Driver/embed-error.f90 --check-prefix=ERROR
--
Exit Code: 1

Command Output (stderr):
--
llvm-project/flang/test/Driver/embed-error.f90:8:10: error: ERROR: expected string not found in input
! ERROR: error: could not open
         ^
<stdin>:1:1: note: scanning from here
; ModuleID = 'FIRModule'
^
<stdin>:6:14: note: possible intended match here
@_QFECi = internal constant i32 1
             ^

Input file: <stdin>
Check file: llvm-project/flang/test/Driver/embed-error.f90

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           1: ; ModuleID = 'FIRModule' 
check:8'0     X~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
           2: source_filename = "FIRModule" 
check:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           3: target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128" 
check:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           4: target triple = "x86_64-unknown-linux-gnu" 
check:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           5:  
check:8'0     ~
           6: @_QFECi = internal constant i32 1 
check:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:8'1                  ?                     possible intended match
           7: @_QQEnvironmentDefaults = constant ptr null 
check:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           8:  
check:8'0     ~
           9: declare ptr @malloc(i64) 
check:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~
          10:  
check:8'0     ~
          11: declare void @free(ptr) 
check:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~
           .
           .
           .

I run on x86_64

[jsjodin]

In D142244#4096579, @clementval wrote:

In D142244#4096546, @jsjodin wrote:

In D142244#4096538, @clementval wrote:

In D142244#4096512, @jsjodin wrote:

No, it should work for any configuration as far as I know. How are you running the test?

Just with check-flang

How does it fail? Can you show the output? Also what CPU are you running on?

I run on `x86_64`

How did you configure your build? I'd like to reproduce your build and see if I can make it fail on my machine. Right now the test passes.

[clementval]

In D142244#4096648, @jsjodin wrote:

In D142244#4096579, @clementval wrote:

In D142244#4096546, @jsjodin wrote:

In D142244#4096538, @clementval wrote:

In D142244#4096512, @jsjodin wrote:

No, it should work for any configuration as far as I know. How are you running the test?

Just with check-flang

How does it fail? Can you show the output? Also what CPU are you running on?

I run on `x86_64`

How did you configure your build? I'd like to reproduce your build and see if I can make it fail on my machine. Right now the test passes.

My bad .. my CompilerInvocation.cpp was overwritten and missed part of this patch. It works fine. Sorry to have bothered you with that.

[jsjodin]

In D142244#4096777, @clementval wrote:

In D142244#4096648, @jsjodin wrote:

In D142244#4096579, @clementval wrote:

In D142244#4096546, @jsjodin wrote:

In D142244#4096538, @clementval wrote:

In D142244#4096512, @jsjodin wrote:

No, it should work for any configuration as far as I know. How are you running the test?

Just with check-flang

How does it fail? Can you show the output? Also what CPU are you running on?

I run on `x86_64`

How did you configure your build? I'd like to reproduce your build and see if I can make it fail on my machine. Right now the test passes.

My bad .. my CompilerInvocation.cpp was overwritten and missed part of this patch. It works fine. Sorry to have bothered you with that.

No problem, glad it works for you. I'm working on another patch that is configuration dependent so I want to make sure there are no errors.

clang/include/clang/Driver/Options.td

defm experimental_library : BoolFOption<"experimental-library",
   PosFlag<SetTrue, [CC1Option, CoreOption], "Control whether unstable and experimental library features are enabled. "
           "This option enables various library features that are either experimental (also known as TSes), or have been "
           "but are not stable yet in the selected Standard Library implementation. It is not recommended to use this option "
           "in production code, since neither ABI nor API stability are guaranteed. This is intended to provide a preview "
           "of features that will ship in the future for experimentation purposes">,
   NegFlag<SetFalse>>;
 
 def fembed_offload_object_EQ : Joined<["-"], "fembed-offload-object=">,
-  Group<f_Group>, Flags<[NoXarchOption, CC1Option]>,
+  Group<f_Group>, Flags<[NoXarchOption, CC1Option, FC1Option]>,
   HelpText<"Embed Offloading device-side binary into host object file as a section.">,
   MarshallingInfoStringVector<CodeGenOpts<"OffloadObjects">>;
 def fembed_bitcode_EQ : Joined<["-"], "fembed-bitcode=">,
     Group<f_Group>, Flags<[NoXarchOption, CC1Option, CC1AsOption]>, MetaVarName<"<option>">,
     HelpText<"Embed LLVM bitcode">,
     Values<"off,all,bitcode,marker">, NormalizedValuesScope<"CodeGenOptions">,
     NormalizedValues<["Embed_Off", "Embed_All", "Embed_Bitcode", "Embed_Marker"]>,
     MarshallingInfoEnum<CodeGenOpts<"EmbedBitcode">, "Embed_Off">;

flang/include/flang/Frontend/CodeGenOptions.h

 /// Tracks various options which control how the code is optimized and passed
 /// to the LLVM backend.
 class CodeGenOptions : public CodeGenOptionsBase {
 
 public:
   /// The paths to the pass plugins that were registered using -fpass-plugin.
   std::vector<std::string> LLVMPassPlugins;
 
+  /// List of filenames passed in using the -fembed-offload-object option. These
+  /// are offloading binaries containing device images and metadata.
+  std::vector<std::string> OffloadObjects;
+
   // Define accessors/mutators for code generation options of enumeration type.
 #define CODEGENOPT(Name, Bits, Default)
 #define ENUM_CODEGENOPT(Name, Type, Bits, Default)                             \
   Type get##Name() const { return static_cast<Type>(Name); }                   \
   void set##Name(Type Value) { Name = static_cast<unsigned>(Value); }
 #include "flang/Frontend/CodeGenOptions.def"
 
   CodeGenOptions();
 };
 
 } // end namespace Fortran::frontend
 
 #endif

flang/include/flang/Frontend/FrontendActions.h

protected:
   std::unique_ptr<mlir::ModuleOp> mlirModule;
   std::unique_ptr<mlir::MLIRContext> mlirCtx;
   /// }
 
   /// @name LLVM IR
   std::unique_ptr<llvm::LLVMContext> llvmCtx;
   std::unique_ptr<llvm::Module> llvmModule;
 
+  /// Embeds offload objects given with specified with -fembed-offload-object
+  void embedOffloadObjects();
+
   /// Generates an LLVM IR module from CodeGenAction::mlirModule and saves it
   /// in CodeGenAction::llvmModule.
   void generateLLVMIR();
 
   BackendActionTy action;
 
   std::unique_ptr<llvm::TargetMachine> tm;
   /// }

flang/lib/Frontend/CompilerInvocation.cpp

static void parseCodeGenArgs(Fortran::frontend::CodeGenOptions &opts,
 
   if (args.hasFlag(clang::driver::options::OPT_fdebug_pass_manager,
                    clang::driver::options::OPT_fno_debug_pass_manager, false))
     opts.DebugPassManager = 1;
 
   for (auto *a : args.filtered(clang::driver::options::OPT_fpass_plugin_EQ))
     opts.LLVMPassPlugins.push_back(a->getValue());
 
+  // -fembed-offload-object option
+  for (auto *a :
+       args.filtered(clang::driver::options::OPT_fembed_offload_object_EQ))
+    opts.OffloadObjects.push_back(a->getValue());
+
   // -mrelocation-model option.
   if (const llvm::opt::Arg *A =
           args.getLastArg(clang::driver::options::OPT_mrelocation_model)) {
     llvm::StringRef ModelName = A->getValue();
     auto RM = llvm::StringSwitch<std::optional<llvm::Reloc::Model>>(ModelName)
                   .Case("static", llvm::Reloc::Static)
                   .Case("pic", llvm::Reloc::PIC_)
                   .Case("dynamic-no-pic", llvm::Reloc::DynamicNoPIC)

flang/lib/Frontend/FrontendActions.cpp

 #include "llvm/ADT/StringRef.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Bitcode/BitcodeWriterPass.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/IRReader/IRReader.h"
 #include "llvm/MC/TargetRegistry.h"
+#include "llvm/Object/OffloadBinary.h"
 #include "llvm/Passes/PassBuilder.h"
 #include "llvm/Passes/PassPlugin.h"
 #include "llvm/Passes/StandardInstrumentations.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Target/TargetMachine.h"
+#include "llvm/Transforms/Utils/ModuleUtils.h"
 #include <memory>
 
 using namespace Fortran::frontend;
 
 // Declare plugin extension function declarations.
 #define HANDLE_EXTENSION(Ext)                                                  \
   llvm::PassPluginLibraryInfo get##Ext##PluginInfo();
 #include "llvm/Support/Extension.def"
#include "llvm/Support/Extension.def"
 
   if (action == BackendActionTy::Backend_EmitBC)
     mpm.addPass(llvm::BitcodeWriterPass(os));
 
   // Run the passes.
   mpm.run(*llvmModule, mam);
 }
 
+void CodeGenAction::embedOffloadObjects() {
+  CompilerInstance &ci = this->getInstance();
+  const auto &cgOpts = ci.getInvocation().getCodeGenOpts();
+
+  for (llvm::StringRef offloadObject : cgOpts.OffloadObjects) {
+    llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> objectOrErr =
+        llvm::MemoryBuffer::getFileOrSTDIN(offloadObject);
+    if (std::error_code ec = objectOrErr.getError()) {
+      auto diagID = ci.getDiagnostics().getCustomDiagID(
+          clang::DiagnosticsEngine::Error, "could not open '%0' for embedding");
+      ci.getDiagnostics().Report(diagID) << offloadObject;
+      return;
+    }
+    llvm::embedBufferInModule(
+        *llvmModule, **objectOrErr, ".llvm.offloading",
+        llvm::Align(llvm::object::OffloadBinary::getAlignment()));
+  }
+}
+
 void CodeGenAction::executeAction() {
   CompilerInstance &ci = this->getInstance();
 
   // If the output stream is a file, generate it and define the corresponding
   // output stream. If a pre-defined output stream is available, we will use
   // that instead.
   //
   // NOTE: `os` is a smart pointer that will be destroyed at the end of this
void CodeGenAction::executeAction() {
   // given on the command-line).
   setUpTargetMachine();
   const std::string &theTriple = tm->getTargetTriple().str();
 
   if (llvmModule->getTargetTriple() != theTriple) {
     ci.getDiagnostics().Report(clang::diag::warn_fe_override_module)
         << theTriple;
   }
+
   // Always set the triple and data layout, to make sure they match and are set.
   // Note that this overwrites any datalayout stored in the LLVM-IR. This avoids
   // an assert for incompatible data layout when the code-generation happens.
   llvmModule->setTargetTriple(theTriple);
   llvmModule->setDataLayout(tm->createDataLayout());
 
+  // Embed offload objects specified with -fembed-offload-object
+  if (!ci.getInvocation().getCodeGenOpts().OffloadObjects.empty())
+    embedOffloadObjects();
+
   // Run LLVM's middle-end (i.e. the optimizer).
   runOptimizationPipeline(ci.isOutputStreamNull() ? *os : ci.getOutputStream());
 
   if (action == BackendActionTy::Backend_EmitLL) {
     llvmModule->print(ci.isOutputStreamNull() ? *os : ci.getOutputStream(),
                       /*AssemblyAnnotationWriter=*/nullptr);
     return;
   }

flang/test/Driver/driver-help.f90

 ! HELP-FC1-NEXT: -fdebug-pre-fir-tree    Dump the pre-FIR tree
 ! HELP-FC1-NEXT: -fdebug-unparse-no-sema Unparse and stop (skips the semantic checks)
 ! HELP-FC1-NEXT: -fdebug-unparse-with-symbols
 ! HELP-FC1-NEXT:                        Unparse and stop.
 ! HELP-FC1-NEXT: -fdebug-unparse        Unparse and stop.
 ! HELP-FC1-NEXT: -fdefault-double-8  Set the default double precision kind to an 8 byte wide type
 ! HELP-FC1-NEXT: -fdefault-integer-8 Set the default integer kind to an 8 byte wide type
 ! HELP-FC1-NEXT: -fdefault-real-8    Set the default real kind to an 8 byte wide type
+! HELP-FC1-NEXT: -fembed-offload-object=<value>
+! HELP-FC1-NEXT:                        Embed Offloading device-side binary into host object file as a section.
 ! HELP-FC1-NEXT: -ffast-math            Allow aggressive, lossy floating-point optimizations
 ! HELP-FC1-NEXT: -ffixed-form           Process source files in fixed form
 ! HELP-FC1-NEXT: -ffixed-line-length=<value>
 ! HELP-FC1-NEXT: Use <value> as character line width in fixed mode
 ! HELP-FC1-NEXT: -ffp-contract=<value> Form fused FP ops (e.g. FMAs)
 ! HELP-FC1-NEXT: -ffree-form            Process source files in free form
 ! HELP-FC1-NEXT: -fget-definition <value> <value> <value>
 ! HELP-FC1-NEXT:                        Get the symbol definition from <line> <start-column> <end-column>

flang/test/Driver/embed-error.f90

+
+!----------
+! RUN lines
+!----------
+! Try to embed missing file
+! RUN: not %flang_fc1 -emit-llvm -o - -fembed-offload-object=%S/Inputs/missing.f90 %s 2>&1 | FileCheck %s --check-prefix=ERROR
+
+! ERROR: error: could not open
+
+parameter(i=1)
+integer :: j
+end program

flang/test/Driver/embed.f90

+
+!----------
+! RUN lines
+!----------
+! Embed something that can be easily checked
+! RUN: %flang_fc1 -emit-llvm -o - -fembed-offload-object=%S/Inputs/hello.f90 %s 2>&1 | FileCheck %s
+
+! RUN: %flang_fc1 -emit-llvm-bc -o %t.bc %s 2>&1
+! RUN: %flang_fc1 -emit-llvm -o - -fembed-offload-object=%S/Inputs/hello.f90 %t.bc 2>&1 | FileCheck %s
+
+! CHECK: @[[OBJECT_1:.+]] = private constant [61 x i8] c"program hello\0A  write(*,*), \22Hello world!\22\0Aend program hello\0A", section ".llvm.offloading", align 8, !exclude !0
+! CHECK: @llvm.compiler.used = appending global [1 x ptr] [ptr @[[OBJECT_1]]], section "llvm.metadata"
+
+
+! CHECK: !llvm.embedded.objects = !{![[METADATA_1:[0-9]+]]}
+! CHECK: ![[METADATA_1]] = !{ptr @[[OBJECT_1]], !".llvm.offloading"}
+
+parameter(i=1)
+integer :: j
+end program