[clang-offload-bundler] Add option -list

[yaxunl]

Summary

clang-offload-bundler is not only used by clang driver
to bundle/unbundle files for offloading toolchains,
but also used by out of tree tools to unbundle
fat binaries generated by clang. It is important
to be able to list the bundle IDs in a bundled
file so that the bundles can be extracted.

This patch adds an option -list to list bundle
ID's in a bundled file. Each bundle ID is separated
by new line. If the file is not a bundled file
nothing is output and returns 0.

[tra]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

781

Having to create a temporary file in order to *list* content of the bundle strikes me as rather odd.
It looks like in order to list the content we actually do bundle unpacking, printing bundled content in the process, and discard the results afterwards. Is that so?

Perhaps it would be better to refactor the code a bit and separate iteration over the bundle from what each iteration does.
E.g. make a function forEachBundledFile(input, lambda) and then pass a function that writes things out for normal operations and a function which just prints the info in case of --list. It may simplify the code a bit as right now you have to copy/paste the loops iterating over ReadBundleStart.

[yaxunl]

Revised by Artem's comments: removing unnecessary output to temporary file, extract forEachBundle.

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

781

done. thanks.

[yaxunl]

Remove unnecessary formatting changes.

[tra]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

199

Now, if we could save the triple in a BundleInfo when it's parsed, and pass BundleInfo to Func() that would make the iterator more useful.

351–352

I'm still not quite happy with the fact that the listing is interleaving with reading the bundle.
I think the code would benefit from further refactoring that would separate bundle reading from what we do with the bundles we've read.

662

Once BundleInfo carries the triple, ListBundleIDs could then be changed to print the bundle info and moved into the FileHandler class:

if (Error Err = ReadHeader(Input))
    return Err;

return forEachBundle(Input, [](BundleInfo bundle) { llvm::outs() << bundle.triple << '\n'});

All other printouts of the triple should no longer be necessary.

[yaxunl]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

199

done

351–352

good point. this can be moved to the lambda without incurring significant overhead.

662

done

[yaxunl]

Revised by Artem's comments.

[yaxunl]

ping

[yaxunl]

ping

[tra]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

183

Now that listBundleIDs is only used by ListBundleIDsInFile(), perhaps it could all be simplified and moved out of the class.

889

I'd pass InputFileNames as an argument. Makes it easier to tell what the function needs.

1071–1075

Does it make sense to continue once we know that CLI options are wrong?
If we just early-exit with an error that may help simplifying the code below a bit.

1087

Perhaps we can separate list option processing from bundle/unbundle?

I think if we could do something like this it would be more readable:

if (ListBundleIDs) {
  if (Unbundle) {
    error...
    exit.
  }
  ... other list-specific checks...
  ListBundleIDsInFile(InputFileNames)
  exit 0;
}

// complicated bundle/unbundle logic can proceed without having to bother about `list` option.

[yaxunl]

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

183

listBundleIDsCallback needs to be a virtual function and it is called by listBundleIDs. Keep listBundleIDs as a member function together with listBundleIDsCallback shows their relation and is more readable.

889

done

1071–1075

done

1087

done

[yaxunl]

revised by Artem's comments

clang/test/Driver/clang-offload-bundler.c

 // RUN: clang-offload-bundler --help | FileCheck %s --check-prefix CK-HELP
 // CK-HELP: {{.*}}OVERVIEW: A tool to bundle several input files of the specified type <type>
 // CK-HELP: {{.*}}referring to the same source file but different targets into a single
 // CK-HELP: {{.*}}one. The resulting file can also be unbundled into different files by
 // CK-HELP: {{.*}}this tool if -unbundle is provided.
 // CK-HELP: {{.*}}USAGE: clang-offload-bundler [options]
 // CK-HELP: {{.*}}-allow-missing-bundles {{.*}}- Create empty files if bundles are missing when unbundling
 // CK-HELP: {{.*}}-inputs=<string>  - [<input file>,...]
+// CK-HELP: {{.*}}-list {{.*}}- List bundle IDs in the bundled file.
 // CK-HELP: {{.*}}-outputs=<string> - [<output file>,...]
 // CK-HELP: {{.*}}-targets=<string> - [<offload kind>-<target triple>,...]
 // CK-HELP: {{.*}}-type=<string>    - Type of the files to be bundled/unbundled.
 // CK-HELP: {{.*}}Current supported types are:
 // CK-HELP: {{.*}}i {{.*}}- cpp-output
 // CK-HELP: {{.*}}ii {{.*}}- c++-cpp-output
 // CK-HELP: {{.*}}ll {{.*}}- llvm
 // CK-HELP: {{.*}}bc {{.*}}- llvm-bc
 // CK-HELP: {{.*}}s {{.*}}- assembler
 // CK-HELP: {{.*}}o {{.*}}- object
 // CK-HELP: {{.*}}gch {{.*}}- precompiled-header
 // CK-HELP: {{.*}}ast {{.*}}- clang AST file
 // CK-HELP: {{.*}}-unbundle {{.*}}- Unbundle bundled file into several output files.
 
 //
 // Check errors.
 //
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i -unbundle 2>&1 | FileCheck %s --check-prefix CK-ERR1
 // CK-ERR1: error: only one input file supported in unbundling mode
-// CK-ERR1: error: number of output files and targets should match in unbundling mode
+
+// RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i -outputs=%t.bundle.i -unbundle 2>&1 | FileCheck %s --check-prefix CK-ERR1A
+// CK-ERR1A: error: number of output files and targets should match in unbundling mode
 
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR2
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1 -outputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR2
 // CK-ERR2: error: number of input files and targets should match in bundling mode
 
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.i,%t.tgt1,%t.tgt2 -inputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR3
 // CK-ERR3: error: only one output file supported in bundling mode
-// CK-ERR3: error: number of input files and targets should match in bundling mode
 
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu -outputs=%t.i,%t.tgt1,%t.tgt2 -inputs=%t.bundle.i -unbundle 2>&1 | FileCheck %s --check-prefix CK-ERR4
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.i,%t.tgt1 -inputs=%t.bundle.i -unbundle 2>&1 | FileCheck %s --check-prefix CK-ERR4
 // CK-ERR4: error: number of output files and targets should match in unbundling mode
 
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2.notexist -outputs=%t.bundle.i 2>&1 | FileCheck %s -DFILE=%t.tgt2.notexist --check-prefix CK-ERR5
 // RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.i,%t.tgt1,%t.tgt2 -inputs=%t.bundle.i.notexist -unbundle 2>&1 | FileCheck %s -DFILE=%t.bundle.i.notexist --check-prefix CK-ERR5
 // CK-ERR5: error: '[[FILE]]': {{N|n}}o such file or directory
 
 // RUN: not clang-offload-bundler -type=invalid -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s -DTYPE=invalid --check-prefix CK-ERR6
 // CK-ERR6: error: '[[TYPE]]': invalid file type specified
 
 // RUN: not clang-offload-bundler 2>&1 | FileCheck %s --check-prefix CK-ERR7
-// CK-ERR7-DAG: clang-offload-bundler: for the --type option: must be specified at least once!
-// CK-ERR7-DAG: clang-offload-bundler: for the --inputs option: must be specified at least once!
-// CK-ERR7-DAG: clang-offload-bundler: for the --outputs option: must be specified at least once!
-// CK-ERR7-DAG: clang-offload-bundler: for the --targets option: must be specified at least once!
+// CK-ERR7: clang-offload-bundler: for the --type option: must be specified at least once!
+
+// RUN: not clang-offload-bundler -type=i -inputs=%t.i,%t.tgt1,%t.tgt2 2>&1 | FileCheck %s -check-prefix=CK-ERR7A
+// CK-ERR7A: error: for the --outputs option: must be specified at least once!
+
+// RUN: not clang-offload-bundler -type=i -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s -check-prefix=CK-ERR7B
+// CK-ERR7B: error: for the --targets option: must be specified at least once!
 
 // RUN: not clang-offload-bundler -type=i -targets=hxst-powerpcxxle-ibm-linux-gnu,openxp-pxxerpc64le-ibm-linux-gnu,xpenmp-x86_xx-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR8
 // CK-ERR8: error: invalid target 'hxst-powerpcxxle-ibm-linux-gnu', unknown offloading kind 'hxst', unknown target triple 'powerpcxxle-ibm-linux-gnu'
-// CK-ERR8: error: invalid target 'openxp-pxxerpc64le-ibm-linux-gnu', unknown offloading kind 'openxp', unknown target triple 'pxxerpc64le-ibm-linux-gnu'
-// CK-ERR8: error: invalid target 'xpenmp-x86_xx-pc-linux-gnu', unknown offloading kind 'xpenmp', unknown target triple 'x86_xx-pc-linux-gnu'
+
+// RUN: not clang-offload-bundler -type=i -targets=host-powerpc64le-ibm-linux-gnu,openxp-pxxerpc64le-ibm-linux-gnu,xpenmp-x86_xx-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR8A
+// CK-ERR8A: error: invalid target 'openxp-pxxerpc64le-ibm-linux-gnu', unknown offloading kind 'openxp', unknown target triple 'pxxerpc64le-ibm-linux-gnu'
+
+// RUN: not clang-offload-bundler -type=i -targets=host-powerpc64le-ibm-linux-gnu,openmp-powerpc64le-ibm-linux-gnu,xpenmp-x86_xx-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR8B
+// CK-ERR8B: error: invalid target 'xpenmp-x86_xx-pc-linux-gnu', unknown offloading kind 'xpenmp', unknown target triple 'x86_xx-pc-linux-gnu'
 
 // RUN: not clang-offload-bundler -type=i -targets=openmp-powerpc64le-linux,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR9A
-// RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR9B
 // CK-ERR9A: error: expecting exactly one host target but got 0
+
+// RUN: not clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle.i 2>&1 | FileCheck %s --check-prefix CK-ERR9B
 // CK-ERR9B: error: Duplicate targets are not allowed
 
 //
 // Check text bundle. This is a readable format, so we check for the format we expect to find.
 //
 // RUN: clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.i,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.i
 // RUN: clang-offload-bundler -type=ii -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.ii,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.ii
 // RUN: clang-offload-bundler -type=ll -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.ll,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.ll

 // CK-TEXTS-UNORDERED: # __CLANG_OFFLOAD_BUNDLE____END__ host-[[HOST]]
 // CK-TEXTS-UNORDERED: # __CLANG_OFFLOAD_BUNDLE____START__ openmp-x86_64-pc-linux-gnu
 // CK-TEXTS-UNORDERED: Content of device file 2
 // CK-TEXTS-UNORDERED: # __CLANG_OFFLOAD_BUNDLE____END__ openmp-x86_64-pc-linux-gnu
 
 //
 // Check text unbundle. Check if we get the exact same content that we bundled before for each file.
 //
+// RUN: clang-offload-bundler -type=i -inputs=%t.bundle3.i -list | FileCheck -check-prefix=CKLST %s
 // RUN: clang-offload-bundler -type=i -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.i,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.i -unbundle
 // RUN: diff %t.i %t.res.i
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=i -targets=openmp-powerpc64le-ibm-linux-gnu -outputs=%t.res.tgt1 -inputs=%t.bundle3.i -unbundle
 // RUN: diff %t.tgt1 %t.res.tgt1
+// RUN: clang-offload-bundler -type=ii -inputs=%t.bundle3.ii -list | FileCheck -check-prefix=CKLST %s
 // RUN: clang-offload-bundler -type=ii -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.ii,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.ii -unbundle
 // RUN: diff %t.ii %t.res.ii
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=ii -targets=openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt2 -inputs=%t.bundle3.ii -unbundle
 // RUN: diff %t.tgt2 %t.res.tgt2
+// RUN: clang-offload-bundler -type=ll -inputs=%t.bundle3.ll -list | FileCheck -check-prefix=CKLST %s
 // RUN: clang-offload-bundler -type=ll -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.ll,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.ll -unbundle
 // RUN: diff %t.ll %t.res.ll
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=ll -targets=openmp-powerpc64le-ibm-linux-gnu -outputs=%t.res.tgt1 -inputs=%t.bundle3.ll -unbundle
 // RUN: diff %t.tgt1 %t.res.tgt1
+// RUN: clang-offload-bundler -type=s -inputs=%t.bundle3.s -list | FileCheck -check-prefix=CKLST %s
 // RUN: clang-offload-bundler -type=s -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.s,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.s -unbundle
 // RUN: diff %t.s %t.res.s
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=s -targets=openmp-powerpc64le-ibm-linux-gnu,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt1,%t.res.s,%t.res.tgt2 -inputs=%t.bundle3.s -unbundle
 // RUN: diff %t.s %t.res.s
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=s -targets=openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt2 -inputs=%t.bundle3.s -unbundle
 // RUN: diff %t.tgt2 %t.res.tgt2
 
 // Check if we can unbundle a file with no magic strings.
+// RUN: clang-offload-bundler -type=s -inputs=%t.s -list | FileCheck -check-prefix=CKLST2 --allow-empty %s
 // RUN: clang-offload-bundler -type=s -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.s,%t.res.tgt1,%t.res.tgt2 -inputs=%t.s -unbundle -allow-missing-bundles
 // RUN: diff %t.s %t.res.s
 // RUN: diff %t.empty %t.res.tgt1
 // RUN: diff %t.empty %t.res.tgt2
 // RUN: clang-offload-bundler -type=s -targets=openmp-powerpc64le-ibm-linux-gnu,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt1,%t.res.s,%t.res.tgt2 -inputs=%t.s -unbundle -allow-missing-bundles
 // RUN: diff %t.s %t.res.s
 // RUN: diff %t.empty %t.res.tgt1
 // RUN: diff %t.empty %t.res.tgt2
 
 // Check that bindler prints an error if given host bundle does not exist in the fat binary.
 // RUN: not clang-offload-bundler -type=s -targets=host-x86_64-xxx-linux-gnu,openmp-powerpc64le-ibm-linux-gnu -outputs=%t.res.s,%t.res.tgt1 -inputs=%t.bundle3.s -unbundle -allow-missing-bundles 2>&1 | FileCheck %s --check-prefix CK-NO-HOST-BUNDLE
 // CK-NO-HOST-BUNDLE: error: Can't find bundle for the host target
 
 //
 // Check binary bundle/unbundle. The content that we have before bundling must be the same we have after unbundling.
 //
 // RUN: clang-offload-bundler -type=bc -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.bc,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.bc
 // RUN: clang-offload-bundler -type=gch -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.ast,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.gch
 // RUN: clang-offload-bundler -type=ast -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.ast,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.ast
 // RUN: clang-offload-bundler -type=ast -targets=openmp-powerpc64le-ibm-linux-gnu,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -inputs=%t.tgt1,%t.ast,%t.tgt2 -outputs=%t.bundle3.unordered.ast
+// RUN: clang-offload-bundler -type=bc -inputs=%t.bundle3.bc -list | FileCheck -check-prefix=CKLST %s
 // RUN: clang-offload-bundler -type=bc -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.bc,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.bc -unbundle
 // RUN: diff %t.bc %t.res.bc
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=bc -targets=openmp-powerpc64le-ibm-linux-gnu -outputs=%t.res.tgt1 -inputs=%t.bundle3.bc -unbundle
 // RUN: diff %t.tgt1 %t.res.tgt1
+// RUN: clang-offload-bundler -type=gch -inputs=%t.bundle3.gch -list | FileCheck -check-prefix=CKLST %s
 // RUN: clang-offload-bundler -type=gch -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.gch,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.gch -unbundle
 // RUN: diff %t.ast %t.res.gch
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=gch -targets=openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt2 -inputs=%t.bundle3.gch -unbundle
 // RUN: diff %t.tgt2 %t.res.tgt2
+// RUN: clang-offload-bundler -type=ast -inputs=%t.bundle3.ast -list | FileCheck -check-prefix=CKLST %s
 // RUN: clang-offload-bundler -type=ast -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.ast,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.ast -unbundle
 // RUN: diff %t.ast %t.res.ast
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=ast -targets=openmp-powerpc64le-ibm-linux-gnu,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt1,%t.res.ast,%t.res.tgt2 -inputs=%t.bundle3.ast -unbundle
 // RUN: diff %t.ast %t.res.ast
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2

 //
 
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.o,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.o -### 2>&1 \
 // RUN: | FileCheck %s -DHOST=%itanium_abi_triple -DINOBJ1=%t.o -DINOBJ2=%t.tgt1 -DINOBJ3=%t.tgt2 -DOUTOBJ=%t.bundle3.o --check-prefix CK-OBJ-CMD
 // CK-OBJ-CMD: llvm-objcopy{{(.exe)?}}" "--add-section=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]={{.*}}" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu=[[INOBJ2]]" "--add-section=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu=[[INOBJ3]]" "[[INOBJ1]]" "[[TEMPOBJ:.*]]"
 // CK-OBJ-CMD: llvm-objcopy{{(.exe)?}}" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__host-[[HOST]]=readonly,exclude" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-powerpc64le-ibm-linux-gnu=readonly,exclude" "--set-section-flags=__CLANG_OFFLOAD_BUNDLE__openmp-x86_64-pc-linux-gnu=readonly,exclude" "[[TEMPOBJ]]" "[[OUTOBJ]]"
 
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.o,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.o
+// RUN: clang-offload-bundler -type=o -inputs=%t.bundle3.o -list | FileCheck -check-prefix=CKLST %s
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.o,%t.res.tgt1,%t.res.tgt2 -inputs=%t.bundle3.o -unbundle
 // RUN: diff %t.bundle3.o %t.res.o
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=o -targets=openmp-powerpc64le-ibm-linux-gnu,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt1,%t.res.o,%t.res.tgt2 -inputs=%t.bundle3.o -unbundle
 // RUN: diff %t.bundle3.o %t.res.o
 // RUN: diff %t.tgt1 %t.res.tgt1
 // RUN: diff %t.tgt2 %t.res.tgt2
 // RUN: clang-offload-bundler -type=o -targets=openmp-powerpc64le-ibm-linux-gnu -outputs=%t.res.tgt1 -inputs=%t.bundle3.o -unbundle
 // RUN: diff %t.tgt1 %t.res.tgt1
 
 // Check if we can unbundle a file with no magic strings.
+// RUN: clang-offload-bundler -type=o -inputs=%t.o -list | FileCheck -check-prefix=CKLST2 --allow-empty %s
 // RUN: clang-offload-bundler -type=o -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -outputs=%t.res.o,%t.res.tgt1,%t.res.tgt2 -inputs=%t.o -unbundle -allow-missing-bundles
 // RUN: diff %t.o %t.res.o
 // RUN: diff %t.empty %t.res.tgt1
 // RUN: diff %t.empty %t.res.tgt2
 // RUN: clang-offload-bundler -type=o -targets=openmp-powerpc64le-ibm-linux-gnu,host-%itanium_abi_triple,openmp-x86_64-pc-linux-gnu -outputs=%t.res.tgt1,%t.res.o,%t.res.tgt2 -inputs=%t.o -unbundle -allow-missing-bundles
 // RUN: diff %t.o %t.res.o
 // RUN: diff %t.empty %t.res.tgt1
 // RUN: diff %t.empty %t.res.tgt2

 // Check error due to duplicate targets
 //
 // RUN: not clang-offload-bundler -type=bc -targets=host-%itanium_abi_triple,hip-amdgcn-amd-amdhsa-gfx900,hip-amdgcn-amd-amdhsa-gfx900 \
 // RUN:   -inputs=%t.bc,%t.tgt1,%t.tgt1 -outputs=%t.hip.bundle.bc 2>&1 | FileCheck -check-prefix=DUP %s
 // RUN: not clang-offload-bundler -type=bc -inputs=%t.hip.bundle.bc -outputs=%t.tmp.bc,%t.tmp2.bc -unbundle \
 // RUN:   -targets=hip-amdgcn-amd-amdhsa-gfx906,hip-amdgcn-amd-amdhsa-gfx906 \
 // RUN:   2>&1 | FileCheck -check-prefix=DUP %s
 // DUP: error: Duplicate targets are not allowed
+//
+// Check -list option
+//
+
+// RUN: clang-offload-bundler -bundle-align=4096 -type=bc -targets=host-%itanium_abi_triple,openmp-powerpc64le-ibm-linux-gnu,openmp-x86_64-pc-linux-gnu -inputs=%t.bc,%t.tgt1,%t.tgt2 -outputs=%t.bundle3.bc
+// RUN: not clang-offload-bundler -type=bc -inputs=%t.bundle3.bc -unbundle -list 2>&1 | FileCheck -check-prefix=CKLST-ERR %s
+// CKLST-ERR: error: -unbundle and -list cannot be used together
+// RUN: not clang-offload-bundler -type=bc -inputs=%t.bundle3.bc -targets=host-%itanium_abi_triple -list 2>&1 | FileCheck -check-prefix=CKLST-ERR2 %s
+// CKLST-ERR2: error: -targets option is invalid for -list
+// RUN: not clang-offload-bundler -type=bc -inputs=%t.bundle3.bc -outputs=out.txt -list 2>&1 | FileCheck -check-prefix=CKLST-ERR3 %s
+// CKLST-ERR3: error: -outputs option is invalid for -list
+// RUN: not clang-offload-bundler -type=bc -inputs=%t.bundle3.bc,%t.bc -list 2>&1 | FileCheck -check-prefix=CKLST-ERR4 %s
+// CKLST-ERR4: error: only one input file supported for -list
+
+// CKLST-DAG: host-
+// CKLST-DAG: openmp-powerpc64le-ibm-linux-gnu
+// CKLST-DAG: openmp-x86_64-pc-linux-gnu
+
+// CKLST2-NOT: host-
+// CKLST2-NOT: openmp-powerpc64le-ibm-linux-gnu
+// CKLST2-NOT: openmp-x86_64-pc-linux-gnu
 
 // Some code so that we can create a binary out of this file.
 int A = 0;
 void test_func(void) {
   ++A;
 }

clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

 static cl::OptionCategory
     ClangOffloadBundlerCategory("clang-offload-bundler options");
 
 static cl::list<std::string>
     InputFileNames("inputs", cl::CommaSeparated, cl::OneOrMore,
                    cl::desc("[<input file>,...]"),
                    cl::cat(ClangOffloadBundlerCategory));
 static cl::list<std::string>
-    OutputFileNames("outputs", cl::CommaSeparated, cl::OneOrMore,
+    OutputFileNames("outputs", cl::CommaSeparated,
                     cl::desc("[<output file>,...]"),
                     cl::cat(ClangOffloadBundlerCategory));
 static cl::list<std::string>
-    TargetNames("targets", cl::CommaSeparated, cl::OneOrMore,
+    TargetNames("targets", cl::CommaSeparated,
                 cl::desc("[<offload kind>-<target triple>,...]"),
                 cl::cat(ClangOffloadBundlerCategory));
 static cl::opt<std::string>
     FilesType("type", cl::Required,
               cl::desc("Type of the files to be bundled/unbundled.\n"
                        "Current supported types are:\n"
                        "  i   - cpp-output\n"
                        "  ii  - c++-cpp-output\n"
                        "  cui - cuda/hip-output\n"
                        "  d   - dependency\n"
                        "  ll  - llvm\n"
                        "  bc  - llvm-bc\n"
                        "  s   - assembler\n"
                        "  o   - object\n"
                        "  gch - precompiled-header\n"
                        "  ast - clang AST file"),
               cl::cat(ClangOffloadBundlerCategory));
 static cl::opt<bool>
     Unbundle("unbundle",
              cl::desc("Unbundle bundled file into several output files.\n"),
              cl::init(false), cl::cat(ClangOffloadBundlerCategory));
 
+static cl::opt<bool>
+    ListBundleIDs("list", cl::desc("List bundle IDs in the bundled file.\n"),
+                  cl::init(false), cl::cat(ClangOffloadBundlerCategory));
+
 static cl::opt<bool> PrintExternalCommands(
     "###",
     cl::desc("Print any external commands that are to be executed "
              "instead of actually executing them - for testing purposes.\n"),
     cl::init(false), cl::cat(ClangOffloadBundlerCategory));
 
 static cl::opt<bool>
     AllowMissingBundles("allow-missing-bundles",
static bool hasHostKind(StringRef Target) {
   StringRef Triple;
   getOffloadKindAndTriple(Target, OffloadKind, Triple);
   return OffloadKind == "host";
 }
 
 /// Generic file handler interface.
 class FileHandler {
 public:
+  struct BundleInfo {
+    StringRef BundleID;
+  };
+
   FileHandler() {}
 
   virtual ~FileHandler() {}
 
   /// Update the file handler with information from the header of the bundled
   /// file.
   virtual Error ReadHeader(MemoryBuffer &Input) = 0;
 
virtual Error WriteBundleStart(raw_fd_ostream &OS,
                                  StringRef TargetTriple) = 0;
 
   /// Write the marker that closes a bundle for the triple \a TargetTriple to \a
   /// OS.
   virtual Error WriteBundleEnd(raw_fd_ostream &OS, StringRef TargetTriple) = 0;
 
   /// Write the bundle from \a Input into \a OS.
   virtual Error WriteBundle(raw_fd_ostream &OS, MemoryBuffer &Input) = 0;
+
+  /// List bundle IDs in \a Input.
+  virtual Error listBundleIDs(MemoryBuffer &Input) {
+    if (Error Err = ReadHeader(Input))
+      return Err;
+
+    return forEachBundle(Input, [&](const BundleInfo &Info) -> Error {
+      llvm::outs() << Info.BundleID << '\n';
+      Error Err = listBundleIDsCallback(Input, Info);
+      if (Err)
+        return Err;
+      return Error::success();
+    });
+  }
+
+  /// For each bundle in \a Input, do \a Func.
+  Error forEachBundle(MemoryBuffer &Input,
+                      std::function<Error(const BundleInfo &)> Func) {
+    while (true) {
+      Expected<Optional<StringRef>> CurTripleOrErr = ReadBundleStart(Input);
+      if (!CurTripleOrErr)
+        return CurTripleOrErr.takeError();
+
+      // No more bundles.
+      if (!*CurTripleOrErr)
+        break;
+
+      StringRef CurTriple = **CurTripleOrErr;
+      assert(!CurTriple.empty());
+
+      BundleInfo Info{CurTriple};
+      if (Error Err = Func(Info))
+        return Err;
+    }
+    return Error::success();
+  }
+
+protected:
+  virtual Error listBundleIDsCallback(MemoryBuffer &Input,
+                                      const BundleInfo &Info) {
+    return Error::success();
+  }
 };
 
 /// Handler for binary files. The bundled file will have the following format
 /// (all integers are stored in little-endian format):
 ///
 /// "OFFLOAD_BUNDLER_MAGIC_STR" (ASCII encoding of the string)
 ///
 /// NumberOfOffloadBundles (8-byte integer)
for (unsigned i = 0; i < 8; ++i) {
     char Char = (char)(Val & 0xffu);
     OS.write(&Char, 1);
     Val >>= 8;
   }
 }
 
 class BinaryFileHandler final : public FileHandler {
   /// Information about the bundles extracted from the header.
-  struct BundleInfo final {
+  struct BinaryBundleInfo final : public BundleInfo {
     /// Size of the bundle.
     uint64_t Size = 0u;
     /// Offset at which the bundle starts in the bundled file.
     uint64_t Offset = 0u;
 
-    BundleInfo() {}
-    BundleInfo(uint64_t Size, uint64_t Offset) : Size(Size), Offset(Offset) {}
+    BinaryBundleInfo() {}
+    BinaryBundleInfo(uint64_t Size, uint64_t Offset)
+        : Size(Size), Offset(Offset) {}
   };
 
   /// Map between a triple and the corresponding bundle information.
-  StringMap<BundleInfo> BundlesInfo;
+  StringMap<BinaryBundleInfo> BundlesInfo;
 
   /// Iterator for the bundle information that is being read.
-  StringMap<BundleInfo>::iterator CurBundleInfo;
-  StringMap<BundleInfo>::iterator NextBundleInfo;
+  StringMap<BinaryBundleInfo>::iterator CurBundleInfo;
+  StringMap<BinaryBundleInfo>::iterator NextBundleInfo;
 
   /// Current bundle target to be written.
   std::string CurWriteBundleTarget;
 
 public:
   BinaryFileHandler() : FileHandler() {}
 
   ~BinaryFileHandler() final {}
for (uint64_t i = 0; i < NumberOfBundles; ++i) {
       ReadChars += 8;
 
       // Read triple.
       if (ReadChars + TripleSize > FC.size())
         return Error::success();
 
       StringRef Triple(&FC.data()[ReadChars], TripleSize);
       ReadChars += TripleSize;
 
       // Check if the offset and size make sense.
       if (!Offset || Offset + Size > FC.size())
         return Error::success();
 
       assert(BundlesInfo.find(Triple) == BundlesInfo.end() &&
              "Triple is duplicated??");
-      BundlesInfo[Triple] = BundleInfo(Size, Offset);
+      BundlesInfo[Triple] = BinaryBundleInfo(Size, Offset);
     }
     // Set the iterator to where we will start to read.
     CurBundleInfo = BundlesInfo.end();
     NextBundleInfo = BundlesInfo.begin();
     return Error::success();
   }
 
   Expected<Optional<StringRef>> ReadBundleStart(MemoryBuffer &Input) final {
Error WriteHeader(raw_fd_ostream &OS,
     unsigned Idx = 0;
     for (auto &T : TargetNames) {
       MemoryBuffer &MB = *Inputs[Idx++];
       HeaderSize = alignTo(HeaderSize, BundleAlignment);
       // Bundle offset.
       Write8byteIntegerToBuffer(OS, HeaderSize);
       // Size of the bundle (adds to the next bundle's offset)
       Write8byteIntegerToBuffer(OS, MB.getBufferSize());
-      BundlesInfo[T] = BundleInfo(MB.getBufferSize(), HeaderSize);
+      BundlesInfo[T] = BinaryBundleInfo(MB.getBufferSize(), HeaderSize);
       HeaderSize += MB.getBufferSize();
       // Size of the triple
       Write8byteIntegerToBuffer(OS, T.size());
       // Triple
       OS << T;
     }
     return Error::success();
   }
public:
   }
 
 private:
   static Error executeObjcopy(StringRef Objcopy, ArrayRef<StringRef> Args) {
     // If the user asked for the commands to be printed out, we do that
     // instead of executing it.
     if (PrintExternalCommands) {
       errs() << "\"" << Objcopy << "\"";
       for (StringRef Arg : drop_begin(Args, 1))
         errs() << " \"" << Arg << "\"";
       errs() << "\n";
     } else {
       if (sys::ExecuteAndWait(Objcopy, Args))
         return createStringError(inconvertibleErrorCode(),
                                  "'llvm-objcopy' tool failed");
     }
     return Error::success();

 public:
   TextFileHandler(StringRef Comment)
       : FileHandler(), Comment(Comment), ReadChars(0) {
     BundleStartString =
         "\n" + Comment.str() + " " OFFLOAD_BUNDLER_MAGIC_STR "__START__ ";
     BundleEndString =
         "\n" + Comment.str() + " " OFFLOAD_BUNDLER_MAGIC_STR "__END__ ";
   }
+
+  Error listBundleIDsCallback(MemoryBuffer &Input,
+                              const BundleInfo &Info) final {
+    // TODO: To list bundle IDs in a bundled text file we need to go through
+    // all bundles. The format of bundled text file may need to include a
+    // header if the performance of listing bundle IDs of bundled text file is
+    // important.
+    ReadChars = Input.getBuffer().find(BundleEndString, ReadChars);
+    if (Error Err = ReadBundleEnd(Input))
+      return Err;
+    return Error::success();
+  }
 };
 
 /// Return an appropriate object file handler. We use the specific object
 /// handler if we know how to deal with that format, otherwise we use a default
 /// binary file handler.
 static std::unique_ptr<FileHandler>
 CreateObjectFileHandler(MemoryBuffer &FirstInput) {
   // Check if the input file format is one that we know how to deal with.
if (Error Err = FH->WriteBundle(OutputFile, **Input))
       return Err;
     if (Error Err = FH->WriteBundleEnd(OutputFile, Triple))
       return Err;
     ++Input;
   }
   return Error::success();
 }
 
+// List bundle IDs. Return true if an error was found.
+static Error ListBundleIDsInFile(StringRef InputFileName) {
+  // Open Input file.
+  ErrorOr<std::unique_ptr<MemoryBuffer>> CodeOrErr =
+      MemoryBuffer::getFileOrSTDIN(InputFileName);
+  if (std::error_code EC = CodeOrErr.getError())
+    return createFileError(InputFileName, EC);
+
+  MemoryBuffer &Input = **CodeOrErr;
+
+  // Select the right files handler.
+  Expected<std::unique_ptr<FileHandler>> FileHandlerOrErr =
+      CreateFileHandler(Input);
+  if (!FileHandlerOrErr)
+    return FileHandlerOrErr.takeError();
+
+  std::unique_ptr<FileHandler> &FH = *FileHandlerOrErr;
+  assert(FH);
+  return FH->listBundleIDs(Input);
+}
+
 // Unbundle the files. Return true if an error was found.
 static Error UnbundleFiles() {
   // Open Input file.
   ErrorOr<std::unique_ptr<MemoryBuffer>> CodeOrErr =
       MemoryBuffer::getFileOrSTDIN(InputFileNames.front());
   if (std::error_code EC = CodeOrErr.getError())
     return createFileError(InputFileNames.front(), EC);
 
int main(int argc, const char **argv) {
 
   if (Help) {
     cl::PrintHelpMessage();
     return 0;
   }
 
   auto reportError = [argv](Error E) {
     logAllUnhandledErrors(std::move(E), WithColor::error(errs(), argv[0]));
+    exit(1);
+  };
+
+  auto doWork = [&](std::function<llvm::Error()> Work) {
+    // Save the current executable directory as it will be useful to find other
+    // tools.
+    BundlerExecutable = argv[0];
+    if (!llvm::sys::fs::exists(BundlerExecutable))
+      BundlerExecutable =
+          sys::fs::getMainExecutable(argv[0], &BundlerExecutable);
+
+    if (llvm::Error Err = Work()) {
+      reportError(std::move(Err));
+    }
   };
 
-  bool Error = false;
+  if (ListBundleIDs) {
+    if (Unbundle) {
+      reportError(
+          createStringError(errc::invalid_argument,
+                            "-unbundle and -list cannot be used together"));
+    }
+    if (InputFileNames.size() != 1) {
+      reportError(createStringError(errc::invalid_argument,
+                                    "only one input file supported for -list"));
+    }
+    if (OutputFileNames.size()) {
+      reportError(createStringError(errc::invalid_argument,
+                                    "-outputs option is invalid for -list"));
+    }
+    if (TargetNames.size()) {
+      reportError(createStringError(errc::invalid_argument,
+                                    "-targets option is invalid for -list"));
+    }
+
+    doWork([]() { return ListBundleIDsInFile(InputFileNames.front()); });
+    return 0;
+  }
+
+  if (OutputFileNames.getNumOccurrences() == 0) {
+    reportError(createStringError(
+        errc::invalid_argument,
+        "for the --outputs option: must be specified at least once!"));
+  }
+  if (TargetNames.getNumOccurrences() == 0) {
+    reportError(createStringError(
+        errc::invalid_argument,
+        "for the --targets option: must be specified at least once!"));
+  }
   if (Unbundle) {
     if (InputFileNames.size() != 1) {
-      Error = true;
       reportError(createStringError(
           errc::invalid_argument,
           "only one input file supported in unbundling mode"));
     }
     if (OutputFileNames.size() != TargetNames.size()) {
-      Error = true;
       reportError(createStringError(errc::invalid_argument,
                                     "number of output files and targets should "
                                     "match in unbundling mode"));
     }
   } else {
     if (OutputFileNames.size() != 1) {
-      Error = true;
       reportError(createStringError(
           errc::invalid_argument,
           "only one output file supported in bundling mode"));
     }
     if (InputFileNames.size() != TargetNames.size()) {
-      Error = true;
       reportError(createStringError(
           errc::invalid_argument,
           "number of input files and targets should match in bundling mode"));
     }
   }
 
   // Verify that the offload kinds and triples are known. We also check that we
   // have exactly one host target.
   unsigned Index = 0u;
   unsigned HostTargetNum = 0u;
   llvm::DenseSet<StringRef> ParsedTargets;
   for (StringRef Target : TargetNames) {
     if (ParsedTargets.contains(Target)) {
       reportError(createStringError(errc::invalid_argument,
                                     "Duplicate targets are not allowed"));
-      return 1;
     }
     ParsedTargets.insert(Target);
 
     StringRef Kind;
     StringRef Triple;
     getOffloadKindAndTriple(Target, Kind, Triple);
 
     bool KindIsValid = !Kind.empty();
     KindIsValid = KindIsValid && StringSwitch<bool>(Kind)
                                      .Case("host", true)
                                      .Case("openmp", true)
                                      .Case("hip", true)
                                      .Default(false);
 
     bool TripleIsValid = !Triple.empty();
     llvm::Triple T(Triple);
     TripleIsValid &= T.getArch() != Triple::UnknownArch;
 
     if (!KindIsValid || !TripleIsValid) {
-      Error = true;
-
       SmallVector<char, 128u> Buf;
       raw_svector_ostream Msg(Buf);
       Msg << "invalid target '" << Target << "'";
       if (!KindIsValid)
         Msg << ", unknown offloading kind '" << Kind << "'";
       if (!TripleIsValid)
         Msg << ", unknown target triple '" << Triple << "'";
       reportError(createStringError(errc::invalid_argument, Msg.str()));
     }
 
     if (KindIsValid && Kind == "host") {
       ++HostTargetNum;
       // Save the index of the input that refers to the host.
       HostInputIndex = Index;
     }
 
     ++Index;
   }
 
   // Host triple is not really needed for unbundling operation, so do not
   // treat missing host triple as error if we do unbundling.
   if ((Unbundle && HostTargetNum > 1) || (!Unbundle && HostTargetNum != 1)) {
-    Error = true;
     reportError(createStringError(errc::invalid_argument,
                                   "expecting exactly one host target but got " +
                                       Twine(HostTargetNum)));
   }
 
-  if (Error)
-    return 1;
-
-  // Save the current executable directory as it will be useful to find other
-  // tools.
-  BundlerExecutable = argv[0];
-  if (!llvm::sys::fs::exists(BundlerExecutable))
-    BundlerExecutable = sys::fs::getMainExecutable(argv[0], &BundlerExecutable);
-
-  if (llvm::Error Err = Unbundle ? UnbundleFiles() : BundleFiles()) {
-    reportError(std::move(Err));
-    return 1;
-  }
+  doWork([]() { return Unbundle ? UnbundleFiles() : BundleFiles(); });
   return 0;
 }