[OpenMPOpt] ICV macro definitions

[sstefan1]

Summary

This defines some basic information about ICVs in OMPKinds.def.
We also emit remarks with initial values for each function (which are default for now)
as a way to test this.

[sstefan1]

llvm/test/Transforms/OpenMP/icv_tracking.ll

13 ↗

(On Diff #272076)

I wasn't sure how to add filenames here like in other remarks tests. Not sure we need them, but if someone knows how, please let me know.

[jhuber6]

llvm/test/Transforms/OpenMP/icv_tracking.ll

13 ↗

(On Diff #272076)

It reads the debug information for the filename and line numbers. Something like this should let you set the filename.

!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 10.0.0 ")
!1 = !DIFile(filename: "filename.c", directory: "/tmp")

Adding line numbers manually is a bit more complicated since it requires scoping information. There's usually a line like this

!21 = !DILocation(line: 5, column: 10, scope: !14)

And a corresponding usage assigning it to one of the instructions

%1 = tail call i32 @__kmpc_global_thread_num(%struct.ident_t* nonnull @0), !dbg !21

[sstefan1]

llvm/test/Transforms/OpenMP/icv_tracking.ll

13 ↗

(On Diff #272076)

Thanks for answering. I guess I'm on my own if this is not completely generated by clang?

[jdoerfert]

llvm/test/Transforms/OpenMP/icv_tracking.ll

13 ↗

(On Diff #272076)

run clang with -g. Don't generate these on your own.

[sstefan1]

llvm/test/Transforms/OpenMP/icv_tracking.ll

13 ↗

(On Diff #272076)

One more question. Does it make sense to have a separate file, eg. icv_remarks.ll (or use one of existing remarks tests) with only one simple function to test this? Debug info adds a lot of noise here. For exampe, for these limited number of tests we get 450+ additional lines. Furthermore, this test file will only grow in the future with the addition of other ICVs. This will probably also affect reviewing test changes. WDYT?

[jdoerfert]

llvm/test/Transforms/OpenMP/icv_tracking.ll

13 ↗

(On Diff #272076)

New file(s) is fine.

[sstefan1]

new test file for icv remarks

[sstefan1]

leftover check lines

[sstefan1]

llvm/test/Transforms/OpenMP/icv_remarks.ll

1

In the new PM remarks are emitted in reverse order. I wasn't sure how to handle this nicely.

[jdoerfert]

Once the test is adjusted for both PMs, LGTM.

llvm/test/Transforms/OpenMP/icv_remarks.ll

1

Either:

1. Make all check lines CHECK-DAG:, or
2. Use two different prefixes.

[fhahn]

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

327

Is it possible to use a different name than FALSE here, because it will fail to compile if the system headers contain a FALSE define? Maybe something like ICV_FALSE to avoid conflicts.

[sstefan1]

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

327

Sure! I hope we will move away from macro magic soon anyway.

@jdoerfert sounds good?

[daltenty]

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

327

We currently have exactly such a build break from this change on AIX due to the system headers defining FALSE

[jdoerfert]

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

327

ICV_.. for all of them seems sensible. Thanks for catching this!

[sstefan1]

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

327

Sorry about that. Should be fixed with rG951e43f357ec3ee0ffc570aea9cbf19871696c42

[fhahn]

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

327

great thanks!

[daltenty]

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

327

Thanks, this seems to have resolved our build issue.

[hoyFB]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

Hello,

Looks like the Int32 type with the full name llvm::omp::types::Int32 is a global variable. Creating a value using it may cause a race condition in thinLTO mode. Although Int32 is initialized with a module specific context, but it may or never get re-created per module. Please correct me if I misunderstand anything. Thanks.

Below is the my failing call stack in thinLTO mode:

PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0. Running pass 'CallGraph Pass Manager' on module ...'.
#0 0x00000000014a3a8f llvm::sys::PrintStackTrace(llvm::raw_ostream&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:564:13
#1 0x00000000014a1c30 llvm::sys::RunSignalHandlers() /home/hoy/local/server-llvm/llvm-project/llvm/lib/Support/Signals.cpp:69:18
#2 0x00000000014a402c SignalHandler(int) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:396:3
#3 0x00007f00baf0b630 __restore_rt (/lib64/libpthread.so.0+0xf630)
#4 0x00000000030afd08 llvm::ConstantInt::get(llvm::LLVMContext&, llvm::APInt const&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/IR/Constants.cpp:776:40
#5 0x00000000030b2a9b llvm::ConstantInt::get(llvm::IntegerType*, unsigned long, bool) /home/hoy/local/server-llvm/llvm-project/llvm/lib/IR/Constants.cpp:797:10
#6 0x00000000024ccc56 (anonymous namespace)::OMPInformationCache::initializeInternalControlVars() /home/hoy/local/server-llvm/llvm-project/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def:326:1
#7 0x00000000024ccc56 (anonymous namespace)::OMPInformationCache::OMPInformationCache(llvm::Module&, llvm::AnalysisGetter&, llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>&, llvm::SetVector<llvm::Function*, std::vector<llvm::Function*, std::allocator<llvm::Function*> >, llvm::DenseSet<llvm::Function*, llvm::DenseMapInfo<llvm::Function*> > >*, llvm::SmallPtrSetImpl<llvm::Function*>&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Transforms/IPO/OpenMPOpt.cpp:69:0
#8 0x00000000024f10ff llvm::SmallVectorTemplateCommon<llvm::Function*, void>::begin() /home/hoy/local/server-llvm/llvm-project/llvm/include/llvm/ADT/SmallVector.h:152:45
#9 0x00000000024f10ff (anonymous namespace)::OpenMPOpt::OpenMPOpt(llvm::SmallVectorImpl<llvm::Function*>&, llvm::CallGraphUpdater&, llvm::function_ref<llvm::OptimizationRemarkEmitter& (llvm::Function*)>, (anonymous namespace)::OMPInformationCache&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Transforms/IPO/OpenMPOpt.cpp:336:0
#10 0x00000000024f10ff (anonymous namespace)::OpenMPOptLegacyPass::runOnSCC(llvm::CallGraphSCC&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Transforms/IPO/OpenMPOpt.cpp:834:0

[sstefan1]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

Not really sure, but turns out types were initialized twice, maybe that was the problem. I guess I could also get module context explicitly , instead of using Int32->getContext().

Anyway, I'll try to build with thinLTO and report back.

[jdoerfert]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

Not really sure, but turns out types were initialized twice

That is a problem. Can you fix that first. I would not be surprised if the LTO error goes away.

[sstefan1]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

@hoyFB I wasn't able to reproduce this locally, but I committed a potential fix: rG61238d2690a6ebdf3c4f3f68f39101fac30263a7

Can you check again if the problem is still there?

[christylee]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

@sstefan1 Thanks! I'll check and report back.

[hoyFB]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

@sstefan1 Thanks for the quick response. I applied the fix and the problem is still there:

#0 0x0000000001474c2f llvm::sys::PrintStackTrace(llvm::raw_ostream&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:564:13
 #1 0x0000000001472dd0 llvm::sys::RunSignalHandlers() /home/hoy/local/server-llvm/llvm-project/llvm/lib/Support/Signals.cpp:69:18
 #2 0x00000000014751cc SignalHandler(int) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Support/Unix/Signals.inc:396:3
 #3 0x00007fb6ac7cc630 __restore_rt (/lib64/libpthread.so.0+0xf630)
 #4 0x0000000003107065 llvm::Type::getInt32Ty(llvm::LLVMContext&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/IR/Type.cpp:186:66
 #5 0x0000000002462754 (anonymous namespace)::OMPInformationCache::initializeInternalControlVars() /home/hoy/local/server-llvm/llvm-project/llvm/include/llvm/Frontend/OpenMP/OMPKinds.def:326:1
 #6 0x0000000002462754 (anonymous namespace)::OMPInformationCache::OMPInformationCache(llvm::Module&, llvm::AnalysisGetter&, llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>&, llvm::SetVector<llvm::Function*, std::vector<llvm::Function*, std::allocator<llvm::Function*> >, llvm::DenseSet<llvm::Function*, llvm::DenseMapInfo<llvm::Function*> > >*, llvm::SmallPtrSetImpl<llvm::Function*>&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Transforms/IPO/OpenMPOpt.cpp:69:5
 #7 0x0000000002484e0f llvm::SmallVectorTemplateCommon<llvm::Function*, void>::begin() /home/hoy/local/server-llvm/llvm-project/llvm/include/llvm/ADT/SmallVector.h:152:45
 #8 0x0000000002484e0f (anonymous namespace)::OpenMPOpt::OpenMPOpt(llvm::SmallVectorImpl<llvm::Function*>&, llvm::CallGraphUpdater&, llvm::function_ref<llvm::OptimizationRemarkEmitter& (llvm::Function*)>, (anonymous namespace)::OMPInformationCache&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Transforms/IPO/OpenMPOpt.cpp:334:18
 #9 0x0000000002484e0f (anonymous namespace)::OpenMPOptLegacyPass::runOnSCC(llvm::CallGraphSCC&) /home/hoy/local/server-llvm/llvm-project/llvm/lib/Transforms/IPO/OpenMPOpt.cpp:832:15

It seems we still have a race condition. The Int32 type can be initialized by a module and then consumed by another module simultaneously. Your early proposal to use the explicit module context to create a constant value instead of using Int32.getContext() which can be from whatever module that first initializes the type sounds good to me. Thanks.

[sstefan1]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

Can you provide some details on how to reproduce this? I wasn't able to. I would like to be able to test this locally.

[jdoerfert]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

I think I understand now. Two processes spin up OMPIRBuilder or OpenMPOpt passes at the same time, both of those initialize global types, we created a race on these globals and all bets are off. I assume (thin)LTO has a single llvm::Context so we "just" need to prevent two threads from initializing at the same time. LLVM has a RAII mutex in llvm/include/llvm/Support/Mutex.h (SmartScopedLock) that we should place in a (new) initializer function (which calls the initializer functions we call now in the constructor) to prevent the race.

[hoyFB]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

The issue happened to one of our internal large projects with many files having OpenMP usage. The project is built with 40 threads in thinLTO mode. I guess it doesn't reproduce with smaller projects.

You may want to print out the Int32.getContext() object as well as the Module object for each ConstantInt::get(Type::getInt32Ty(Int32->getContext()), 0); call and see if they match. Two modules creating constants in the same context may cause a problem.

[hoyFB]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

Yeah, that's the problem.

The global OMP types are initialized without lock protection with a module local context:

https://github.com/llvm/llvm-project/blob/1a70077b5a64189d9c04d1a2d7ea6ff0e49744d6/llvm/lib/Frontend/OpenMP/OMPConstants.cpp#L73

Even if they are lock protected, the module local context passed to them can still be used to create a constant by another module in parallel. Module constants should be created in sequence.

[jdoerfert]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

Even if they are lock protected, the module local context passed to them can still be used to create a constant by another module in parallel. Module constants should be created in sequence.

This is not up to the pass. The race in question here is home made but llvm::Context has to be thread safe itself if you use it in parallel, right? This is not the only pass creating constants and types. If, however, thinLTO uses different contexts per module, we have a different problem here. If the latter is the case we need to keep the Types as part of the OpenMPIRBuilder object which is "local" to one lvm::Context. @sstefan1 is going to look into the race tomorrow. I think it might be the best thing to move the types into the OpenMPIRBuilder. That will require some rewrite but make it race free and working with multiple contexts in parallel.

[hoyFB]

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

230

If I understand correctly, functions of one module are always compiled sequentially, even in thinLTO mode. Different modules can be compiled in parallel. A module should have a unique context. Constants and types created during the compilation of one module are appended to the module's context in sequence. With the global Int32 type owning a context of an arbitrary module, creating multiple constants in that context from different threads concurrently result in a race.

I think moving the types into OpenMPIRBuilder objects should fix the issue. If you look at this code: https://github.com/llvm/llvm-project/blob/15440191b57237eafb18600ac653b9a0023db391/llvm/lib/IR/Constants.cpp#L776 , the constants are added to a DenseMap object with no lock protection. As long as they are added sequentially, it should be fine.

The problem should be thinLTO-specific. In full LTO mode, a dummy module is created to cover all functions which are then compiled sequentially. In thinLTO mode, an input file is one module owning its own context, and they are compiled in parallel.

llvm/include/llvm/Frontend/OpenMP/OMPConstants.h

 /// Make the enum values available in the llvm::omp namespace. This allows us to
 /// write something like OMPD_parallel if we have a `using namespace omp`. At
 /// the same time we do not loose the strong type guarantees of the enum class,
 /// that is we cannot pass an unsigned as Directive without an explicit cast.
 #define OMP_DIRECTIVE(Enum, ...) constexpr auto Enum = omp::Directive::Enum;
 #define OMP_CLAUSE(Enum, ...) constexpr auto Enum = omp::Clause::Enum;
 #include "llvm/Frontend/OpenMP/OMPKinds.def"
 
+/// IDs for all Internal Control Variables (ICVs).
+enum class InternalControlVar {
+#define ICV_DATA_ENV(Enum, ...) Enum,
+#include "llvm/Frontend/OpenMP/OMPKinds.def"
+};
+
+#define ICV_DATA_ENV(Enum, ...)                                                \
+  constexpr auto Enum = omp::InternalControlVar::Enum;
+#include "llvm/Frontend/OpenMP/OMPKinds.def"
+
+enum class ICVInitValue {
+#define ICV_DATA_ENV(Enum, Name, EnvVar, Init) Init,
+#include "llvm/Frontend/OpenMP/OMPKinds.def"
+};
+
+#define ICV_DATA_ENV(Enum, Name, EnvVar, Init)                                 \
+  constexpr auto Init = omp::ICVInitValue::Init;
+#include "llvm/Frontend/OpenMP/OMPKinds.def"
+
 /// IDs for all omp runtime library (RTL) functions.
 enum class RuntimeFunction {
 #define OMP_RTL(Enum, ...) Enum,
 #include "llvm/Frontend/OpenMP/OMPKinds.def"
 };
 
 #define OMP_RTL(Enum, ...) constexpr auto Enum = omp::RuntimeFunction::Enum;
 #include "llvm/Frontend/OpenMP/OMPKinds.def"

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

 
 #ifndef OMP_TYPE
 #define OMP_TYPE(VarName, InitValue)
 #endif
 
 #define __OMP_TYPE(VarName) OMP_TYPE(VarName, Type::get##VarName##Ty(Ctx))
 
 __OMP_TYPE(Void)
+__OMP_TYPE(Int1)
 __OMP_TYPE(Int8)
 __OMP_TYPE(Int32)
 __OMP_TYPE(Int64)
 __OMP_TYPE(Int8Ptr)
 __OMP_TYPE(Int32Ptr)
 __OMP_TYPE(Int64Ptr)
 
 OMP_TYPE(SizeTy, M.getDataLayout().getIntPtrType(Ctx))

 __OMP_FUNCTION_TYPE(TaskRoutineEntry, false, Int32, Int32,
                     /* kmp_task_t */ VoidPtr)
 
 #undef __OMP_FUNCTION_TYPE
 #undef OMP_FUNCTION_TYPE
 
 ///}
 
+/// Internal Control Variables information
+///
+///{
+
+#ifndef ICV_DATA_ENV
+#define ICV_DATA_ENV(Enum, Name, EnvVarName, Init)
+#endif
+
+#define __ICV_DATA_ENV(Name, EnvVarName, Init)                                 \
+  ICV_DATA_ENV(ICV_##Name, #Name, #EnvVarName, Init)
+
+__ICV_DATA_ENV(nthreads, OMP_NUM_THREADS, IMPLEMENTATION_DEFINED)
+__ICV_DATA_ENV(active_levels, NONE, ZERO)
+__ICV_DATA_ENV(cancel, OMP_CANCELLATION, FALSE)
+__ICV_DATA_ENV(__last, last, LAST)
+
+#undef __ICV_DATA_ENV
+#undef ICV_DATA_ENV
+
+#ifndef ICV_RT_SET
+#define ICV_RT_SET(Name, RTL)
+#endif
+
+#define __ICV_RT_SET(Name, RTL) ICV_RT_SET(ICV_##Name, OMPRTL_##RTL)
+
+__ICV_RT_SET(nthreads, omp_set_num_threads)
+
+#undef __ICV_RT_SET
+#undef ICV_RT_SET
+
+#ifndef ICV_RT_GET
+#define ICV_RT_GET(Name, RTL)
+#endif
+
+#define __ICV_RT_GET(Name, RTL) ICV_RT_GET(ICV_##Name, OMPRTL_##RTL)
+
+__ICV_RT_GET(nthreads, omp_get_max_threads)
+__ICV_RT_GET(active_levels, omp_get_active_level)
+__ICV_RT_GET(cancel, omp_get_cancellation)
+
+#undef __ICV_RT_GET
+#undef ICV_RT_GET
+
+///}
+
 /// Runtime library function (and their attributes)
 ///
 ///{
 
 #ifndef OMP_RTL
 #define OMP_RTL(Enum, Str, IsVarArg, ReturnType, ...)
 #endif
 

llvm/lib/Transforms/IPO/OpenMPOpt.cpp

 
 #define DEBUG_TYPE "openmp-opt"
 
 static cl::opt<bool> DisableOpenMPOptimizations(
     "openmp-opt-disable", cl::ZeroOrMore,
     cl::desc("Disable OpenMP specific optimizations."), cl::Hidden,
     cl::init(false));
 
+static cl::opt<bool> PrintICVValues("openmp-print-icv-values", cl::init(false),
+                                    cl::Hidden);
+
 STATISTIC(NumOpenMPRuntimeCallsDeduplicated,
           "Number of OpenMP runtime calls deduplicated");
 STATISTIC(NumOpenMPParallelRegionsDeleted,
           "Number of OpenMP parallel regions deleted");
 STATISTIC(NumOpenMPRuntimeFunctionsIdentified,
           "Number of OpenMP runtime functions identified");
 STATISTIC(NumOpenMPRuntimeFunctionUsesIdentified,
           "Number of OpenMP runtime function uses identified");

 struct OMPInformationCache : public InformationCache {
   OMPInformationCache(Module &M, AnalysisGetter &AG,
                       BumpPtrAllocator &Allocator, SetVector<Function *> *CGSCC,
                       SmallPtrSetImpl<Function *> &ModuleSlice)
       : InformationCache(M, AG, Allocator, CGSCC), ModuleSlice(ModuleSlice),
         OMPBuilder(M) {
     initializeTypes(M);
     initializeRuntimeFunctions();
+    initializeInternalControlVars();
 
     OMPBuilder.initialize();
   }
 
+  /// Generic information that describes an internal control variable.
+  struct InternalControlVarInfo {
+    /// The kind, as described by InternalControlVar enum.
+    InternalControlVar Kind;
+
+    /// The name of the ICV.
+    StringRef Name;
+
+    /// Environment variable associated with this ICV.
+    StringRef EnvVarName;
+
+    /// Initial value kind.
+    ICVInitValue InitKind;
+
+    /// Initial value.
+    ConstantInt *InitValue;
+
+    /// Setter RTL function associated with this ICV.
+    RuntimeFunction Setter;
+
+    /// Getter RTL function associated with this ICV.
+    RuntimeFunction Getter;
+
+    /// RTL Function corresponding to the override clause of this ICV
+    RuntimeFunction Clause;
+  };
+
   /// Generic information that describes a runtime function
   struct RuntimeFunctionInfo {
 
     /// The kind, as described by the RuntimeFunction enum.
     RuntimeFunction Kind;
 
     /// The name of the function.
     StringRef Name;
struct OMPInformationCache : public InformationCache {
   /// An OpenMP-IR-Builder instance
   OpenMPIRBuilder OMPBuilder;
 
   /// Map from runtime function kind to the runtime function description.
   EnumeratedArray<RuntimeFunctionInfo, RuntimeFunction,
                   RuntimeFunction::OMPRTL___last>
       RFIs;
 
+  /// Map from ICV kind to the ICV description.
+  EnumeratedArray<InternalControlVarInfo, InternalControlVar,
+                  InternalControlVar::ICV___last>
+      ICVs;
+
+  /// Helper to initialize all internal control variable information for those
+  /// defined in OMPKinds.def.
+  void initializeInternalControlVars() {
+#define ICV_RT_SET(_Name, RTL)                                                 \
+  {                                                                            \
+    auto &ICV = ICVs[_Name];                                                   \
+    ICV.Setter = RTL;                                                          \
+  }
+#define ICV_RT_GET(Name, RTL)                                                  \
+  {                                                                            \
+    auto &ICV = ICVs[Name];                                                    \
+    ICV.Getter = RTL;                                                          \
+  }
+#define ICV_DATA_ENV(Enum, _Name, _EnvVarName, Init)                           \
+  {                                                                            \
+    auto &ICV = ICVs[Enum];                                                    \
+    ICV.Name = _Name;                                                          \
+    ICV.Kind = Enum;                                                           \
+    ICV.InitKind = Init;                                                       \
+    ICV.EnvVarName = _EnvVarName;                                              \
+    switch (ICV.InitKind) {                                                    \
+    case IMPLEMENTATION_DEFINED:                                               \
+      ICV.InitValue = nullptr;                                                 \
+      break;                                                                   \
+    case ZERO:                                                                 \
+      ICV.InitValue =                                                          \
+          ConstantInt::get(Type::getInt32Ty(Int32->getContext()), 0);          \
+      break;                                                                   \
+    case FALSE:                                                                \
+      ICV.InitValue = ConstantInt::getFalse(Int1->getContext());               \
+      break;                                                                   \
+    case LAST:                                                                 \
+      break;                                                                   \
+    }                                                                          \
+  }
+#include "llvm/Frontend/OpenMP/OMPKinds.def"
+  }
+
   /// Returns true if the function declaration \p F matches the runtime
   /// function types, that is, return type \p RTFRetType, and argument types
   /// \p RTFArgTypes.
   static bool declMatchesRTFTypes(Function *F, Type *RTFRetType,
                                   SmallVector<Type *, 8> &RTFArgTypes) {
     // TODO: We should output information to the user (under debug output
     //       and via remarks).
 
struct OpenMPOpt {
   /// Run all OpenMP optimizations on the underlying SCC/ModuleSlice.
   bool run() {
     bool Changed = false;
 
     LLVM_DEBUG(dbgs() << TAG << "Run on SCC with " << SCC.size()
                       << " functions in a slice with " << ModuleSlice.size()
                       << " functions\n");
 
+    /// Print initial ICV values for testing.
+    /// FIXME: This should be done from the Attributor once it is added.
+    if (PrintICVValues) {
+      InternalControlVar ICVs[] = {ICV_nthreads, ICV_active_levels, ICV_cancel};
+
+      for (Function *F : ModuleSlice) {
+        for (auto ICV : ICVs) {
+          auto ICVInfo = OMPInfoCache.ICVs[ICV];
+          auto Remark = [&](OptimizationRemark OR) {
+            return OR << "OpenMP ICV " << ore::NV("OpenMPICV", ICVInfo.Name)
+                      << " Value: "
+                      << (ICVInfo.InitValue
+                              ? ICVInfo.InitValue->getValue().toString(10, true)
+                              : "IMPLEMENTATION_DEFINED");
+          };
+
+          emitRemarkOnFunction(F, "OpenMPICVTracker", Remark);
+        }
+      }
+    }
+
     Changed |= deduplicateRuntimeCalls();
     Changed |= deleteParallelRegions();
 
     return Changed;
   }
 
   /// Return the call if \p U is a callee use in a regular call. If \p RFI is
   /// given it has to be the callee or a nullptr is returned.
void emitRemark(Instruction *Inst, StringRef RemarkName,
                   RemarkCallBack &&RemarkCB) {
     Function *F = Inst->getParent()->getParent();
     auto &ORE = OREGetter(F);
 
     ORE.emit(
         [&]() { return RemarkCB(RemarkKind(DEBUG_TYPE, RemarkName, Inst)); });
   }
 
+  /// Emit a remark on a function. Since only OptimizationRemark is supporting
+  /// this, it can't be made generic.
+  void emitRemarkOnFunction(
+      Function *F, StringRef RemarkName,
+      function_ref<OptimizationRemark(OptimizationRemark &&)> &&RemarkCB) {
+    auto &ORE = OREGetter(F);
+
+    ORE.emit([&]() {
+      return RemarkCB(OptimizationRemark(DEBUG_TYPE, RemarkName, F));
+    });
+  }
+
   /// The underyling module.
   Module &M;
 
   /// The SCC we are operating on.
   SmallVectorImpl<Function *> &SCC;
 
   /// The slice of the module we are allowed to look at.
   SmallPtrSetImpl<Function *> &ModuleSlice;

llvm/test/Transforms/OpenMP/icv_remarks.ll

+; opt -passes=openmpopt -pass-remarks=openmp-opt -openmp-print-icv-values -disable-output < %s 2>&1 | FileCheck %s
+; RUN: opt -openmpopt -pass-remarks=openmp-opt -openmp-print-icv-values -disable-output < %s 2>&1 | FileCheck %s
+
+; ModuleID = 'icv_remarks.c'
+source_filename = "icv_remarks.c"
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+%struct.ident_t = type { i32, i32, i32, i32, i8* }
+
+@.str = private unnamed_addr constant [23 x i8] c";unknown;unknown;0;0;;\00", align 1
+@0 = private unnamed_addr constant %struct.ident_t { i32 0, i32 2, i32 0, i32 0, i8* getelementptr inbounds ([23 x i8], [23 x i8]* @.str, i32 0, i32 0) }, align 8
+@1 = private unnamed_addr constant [26 x i8] c";icv_remarks.c;foo;18;1;;\00", align 1
+
+; CHECK: remark: icv_remarks.c:12:0: OpenMP ICV nthreads Value: IMPLEMENTATION_DEFINED
+; CHECK: remark: icv_remarks.c:12:0: OpenMP ICV active_levels Value: 0
+; CHECK: remark: icv_remarks.c:12:0: OpenMP ICV cancel Value: 0
+define dso_local void @foo(i32 %a) local_unnamed_addr #0 !dbg !17 {
+entry:
+  %.kmpc_loc.addr = alloca %struct.ident_t, align 8
+  %0 = bitcast %struct.ident_t* %.kmpc_loc.addr to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* nonnull align 8 dereferenceable(24) %0, i8* nonnull align 8 dereferenceable(24) bitcast (%struct.ident_t* @0 to i8*), i64 16, i1 false)
+  call void @llvm.dbg.value(metadata i32 %a, metadata !19, metadata !DIExpression()), !dbg !21
+  tail call void @omp_set_num_threads(i32 %a) #1, !dbg !22
+  %call = tail call i32 @omp_get_max_threads() #1, !dbg !23
+  call void @llvm.dbg.value(metadata i32 %call, metadata !20, metadata !DIExpression()), !dbg !21
+  tail call void @use(i32 %call) #1, !dbg !24
+  %1 = getelementptr inbounds %struct.ident_t, %struct.ident_t* %.kmpc_loc.addr, i64 0, i32 4, !dbg !25
+  store i8* getelementptr inbounds ([26 x i8], [26 x i8]* @1, i64 0, i64 0), i8** %1, align 8, !dbg !25, !tbaa !26
+  call void (%struct.ident_t*, i32, void (i32*, i32*, ...)*, ...) @__kmpc_fork_call(%struct.ident_t* nonnull %.kmpc_loc.addr, i32 0, void (i32*, i32*, ...)* bitcast (void (i32*, i32*)* @.omp_outlined. to void (i32*, i32*, ...)*)) #1, !dbg !25
+  ret void, !dbg !32
+}
+
+declare !dbg !4 dso_local void @omp_set_num_threads(i32) local_unnamed_addr #1
+
+declare !dbg !9 dso_local i32 @omp_get_max_threads() local_unnamed_addr #1
+
+declare !dbg !12 dso_local void @use(i32) local_unnamed_addr #2
+
+; CHECK: remark: icv_remarks.c:18:0: OpenMP ICV nthreads Value: IMPLEMENTATION_DEFINED
+; CHECK: remark: icv_remarks.c:18:0: OpenMP ICV active_levels Value: 0
+; CHECK: remark: icv_remarks.c:18:0: OpenMP ICV cancel Value: 0
+define internal void @.omp_outlined.(i32* noalias nocapture readnone %.global_tid., i32* noalias nocapture readnone %.bound_tid.) #3 !dbg !33 {
+entry:
+  call void @llvm.dbg.value(metadata i32* %.global_tid., metadata !41, metadata !DIExpression()), !dbg !43
+  call void @llvm.dbg.value(metadata i32* %.bound_tid., metadata !42, metadata !DIExpression()), !dbg !43
+  call void @llvm.dbg.value(metadata i32* undef, metadata !44, metadata !DIExpression()) #1, !dbg !50
+  call void @llvm.dbg.value(metadata i32* undef, metadata !47, metadata !DIExpression()) #1, !dbg !50
+  tail call void @omp_set_num_threads(i32 10) #1, !dbg !52
+  %call.i = tail call i32 @omp_get_max_threads() #1, !dbg !53
+  call void @llvm.dbg.value(metadata i32 %call.i, metadata !48, metadata !DIExpression()) #1, !dbg !54
+  tail call void @use(i32 %call.i) #1, !dbg !55
+  ret void, !dbg !56
+}
+
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64, i1 immarg) #4
+
+declare !callback !57 dso_local void @__kmpc_fork_call(%struct.ident_t*, i32, void (i32*, i32*, ...)*, ...) local_unnamed_addr #1
+
+declare void @llvm.dbg.value(metadata, metadata, metadata) #5
+
+attributes #0 = { nounwind uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="none" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #1 = { nounwind }
+attributes #2 = { "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="none" "less-precise-fpmad"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #3 = { norecurse nounwind uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="none" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #4 = { argmemonly nounwind willreturn }
+attributes #5 = { nounwind readnone speculatable willreturn }
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!13, !14, !15}
+!llvm.ident = !{!16}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 73cea83a6f5ab521edf3cccfc603534776d691ec)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, splitDebugInlining: false, nameTableKind: None)
+!1 = !DIFile(filename: "icv_remarks.c", directory: "/tmp")
+!2 = !{}
+!3 = !{!4, !9, !12}
+!4 = !DISubprogram(name: "omp_set_num_threads", scope: !5, file: !5, line: 57, type: !6, flags: DIFlagPrototyped, spFlags: DISPFlagOptimized, retainedNodes: !2)
+!5 = !DIFile(filename: "/usr/local/lib/clang/11.0.0/include/omp.h", directory: "")
+!6 = !DISubroutineType(types: !7)
+!7 = !{null, !8}
+!8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!9 = !DISubprogram(name: "omp_get_max_threads", scope: !5, file: !5, line: 67, type: !10, flags: DIFlagPrototyped, spFlags: DISPFlagOptimized, retainedNodes: !2)
+!10 = !DISubroutineType(types: !11)
+!11 = !{!8}
+!12 = !DISubprogram(name: "use", scope: !1, file: !1, line: 10, type: !6, flags: DIFlagPrototyped, spFlags: DISPFlagOptimized, retainedNodes: !2)
+!13 = !{i32 7, !"Dwarf Version", i32 4}
+!14 = !{i32 2, !"Debug Info Version", i32 3}
+!15 = !{i32 1, !"wchar_size", i32 4}
+!16 = !{!"clang version 11.0.0 (https://github.com/llvm/llvm-project.git 73cea83a6f5ab521edf3cccfc603534776d691ec)"}
+!17 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 12, type: !6, scopeLine: 12, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !18)
+!18 = !{!19, !20}
+!19 = !DILocalVariable(name: "a", arg: 1, scope: !17, file: !1, line: 12, type: !8)
+!20 = !DILocalVariable(name: "num", scope: !17, file: !1, line: 15, type: !8)
+!21 = !DILocation(line: 0, scope: !17)
+!22 = !DILocation(line: 13, column: 3, scope: !17)
+!23 = !DILocation(line: 15, column: 13, scope: !17)
+!24 = !DILocation(line: 17, column: 3, scope: !17)
+!25 = !DILocation(line: 18, column: 1, scope: !17)
+!26 = !{!27, !31, i64 16}
+!27 = !{!"ident_t", !28, i64 0, !28, i64 4, !28, i64 8, !28, i64 12, !31, i64 16}
+!28 = !{!"int", !29, i64 0}
+!29 = !{!"omnipotent char", !30, i64 0}
+!30 = !{!"Simple C/C++ TBAA"}
+!31 = !{!"any pointer", !29, i64 0}
+!32 = !DILocation(line: 24, column: 1, scope: !17)
+!33 = distinct !DISubprogram(name: ".omp_outlined.", scope: !1, file: !1, line: 18, type: !34, scopeLine: 18, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !40)
+!34 = !DISubroutineType(types: !35)
+!35 = !{null, !36, !36}
+!36 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !37)
+!37 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !38)
+!38 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !39, size: 64)
+!39 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !8)
+!40 = !{!41, !42}
+!41 = !DILocalVariable(name: ".global_tid.", arg: 1, scope: !33, type: !36, flags: DIFlagArtificial)
+!42 = !DILocalVariable(name: ".bound_tid.", arg: 2, scope: !33, type: !36, flags: DIFlagArtificial)
+!43 = !DILocation(line: 0, scope: !33)
+!44 = !DILocalVariable(name: ".global_tid.", arg: 1, scope: !45, type: !36, flags: DIFlagArtificial)
+!45 = distinct !DISubprogram(name: ".omp_outlined._debug__", scope: !1, file: !1, line: 19, type: !34, scopeLine: 19, flags: DIFlagPrototyped | DIFlagAllCallsDescribed, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !0, retainedNodes: !46)
+!46 = !{!44, !47, !48}
+!47 = !DILocalVariable(name: ".bound_tid.", arg: 2, scope: !45, type: !36, flags: DIFlagArtificial)
+!48 = !DILocalVariable(name: "num1", scope: !49, file: !1, line: 21, type: !8)
+!49 = distinct !DILexicalBlock(scope: !45, file: !1, line: 19, column: 3)
+!50 = !DILocation(line: 0, scope: !45, inlinedAt: !51)
+!51 = distinct !DILocation(line: 18, column: 1, scope: !33)
+!52 = !DILocation(line: 20, column: 5, scope: !49, inlinedAt: !51)
+!53 = !DILocation(line: 21, column: 16, scope: !49, inlinedAt: !51)
+!54 = !DILocation(line: 0, scope: !49, inlinedAt: !51)
+!55 = !DILocation(line: 22, column: 5, scope: !49, inlinedAt: !51)
+!56 = !DILocation(line: 18, column: 1, scope: !33)
+!57 = !{!58}
+!58 = !{i64 2, i64 -1, i64 -1, i1 true}