Profiling the code generated by MCJIT engine using Intel VTune profiler

[ekovanov]

Summary

Problem:
LLVM already has a feature to profile the JIT-compiled code with VTune. This is done using Intel JIT Profiling API (https://github.com/intel/ittapi). Function information is captured by VTune as soon as the function is JIT-compiled. We tried to use the same approach to report the function information generated by the MCJIT engine – read parsing the debug information for in-memory ELF module and report it using JIT API. As the results, we figured out that it did not work properly for the following cases: inline functions, the functions located in multiple source files, the functions having several bodies (address ranges).

Solution:
To overcome limitations described above, we have introduced new APIs as a part of Intel ITT APIs to report the entire in-memory ELF module to be further processed as regular ELF binaries with debug information.

This patch

1. Switches LLVM to open source version of Intel ITT/JIT APIs ((https://github.com/intel/ittapi) to keep it always up to date
2. Adds support of profiling the code generated by MCJIT engine using Intel VTune profiler

Another separate patch will get rid of obsolete Intel ITT APIs stuff, having LLVM already switched to https://github.com/intel/ittapi

llvm/lib/ExecutionEngine/IntelJITEvents

ittnotify_config.h
ittnotify_types.h
jitprofiling.c
jitprofiling.h

[lhames]

I'm not familiar with the profiling listeners. Out of interest: Is the backwards compatibility mode necessary? Is there a plan to deprecate it in the future?

[ekovanov]

corrected indents that were reported by pre-merge checks

[ekovanov]

In D86435#2234559, @lhames wrote:

I'm not familiar with the profiling listeners. Out of interest: Is the backwards compatibility mode necessary? Is there a plan to deprecate it in the future?

@lhames yes, we plan to deprecate it in the future.
For now, by using backwards compatibility, we allow users to switch to the old flow in case of problems with the new one.

[lhames]

@lhames yes, we plan to deprecate it in the future.
For now, by using backwards compatibility, we allow users to switch to the old flow in case of problems with the new one.

Sounds good to me. Thanks Elena!

[ekovanov]

updated diff with remarks for pre-merge checks

[ekovanov]

corrections for pre-merge checks

[ekovanov]

Hi @lhames @andrew.w.kaylor

Looking at "Build Status" for the last diff, I can see 29 tests failed (windows) and 41 tests failed (linux).
do you think that tests failures are related to my commit?
Thank you in advance!

[ekovanov]

added submodule update command

[lhames]

In D86435#2256396, @ekovanov wrote:

Hi @lhames @andrew.w.kaylor

Looking at "Build Status" for the last diff, I can see 29 tests failed (windows) and 41 tests failed (linux).
do you think that tests failures are related to my commit?
Thank you in advance!

Hi Elena,

I can't be sure from the error messages -- I don't know the relevant profiling or LLDB APIs well enough. Are you able to reproduce any of them when you build locally?

[ekovanov]

In D86435#2315812, @lhames wrote:

In D86435#2256396, @ekovanov wrote:

Hi @lhames @andrew.w.kaylor

Looking at "Build Status" for the last diff, I can see 29 tests failed (windows) and 41 tests failed (linux).
do you think that tests failures are related to my commit?
Thank you in advance!

Hi Elena,

I can't be sure from the error messages -- I don't know the relevant profiling or LLDB APIs well enough. Are you able to reproduce any of them when you build locally?

Hi @lhames !

I tried to run failed tests with applied diff on my system and tests have passed (master from 9th September was used).
But it constantly fails on Phabricator.
Also, I believe that most of my changes are related to -DLLVM_USE_INTEL_JITEVENTS=ON compile option, that does not participate in build process on Phabricator.

[ekovanov]

Hi @clayborg !

There are several failed tests for both Linux and Windows in "Build Status" section of this review.
The vast majority of failed tests is related to lldb.

But I wouldn’t expect my commit to have any effect unless -DLLVM_USE_INTEL_JITEVENTS=ON is used.
(the option -DLLVM_USE_INTEL_JITEVENTS=ON does not participate in build process on Phabricator)

I tried to run failed tests with applied diff on my system and tests have passed. (master from 9th September)

Do you think that tests failures are related to my commit?

Number of failed tests is unstable(multiple revisions of this patch), but there are consistently repeating items:

40 tests on Linux:

lldb-api :: commands/expression/import-std-module/basic/TestImportStdModule.py
lldb-api :: commands/expression/import-std-module/conflicts/TestStdModuleWithConflicts.py
lldb-api :: commands/expression/import-std-module/deque-basic/TestDequeFromStdModule.py
lldb-api :: commands/expression/import-std-module/deque-dbg-info-content/TestDbgInfoContentDequeFromStdModule.py
lldb-api :: commands/expression/import-std-module/forward_list-dbg-info-content/TestDbgInfoContentForwardListFromStdModule.py
lldb-api :: commands/expression/import-std-module/forward_list/TestForwardListFromStdModule.py
lldb-api :: commands/expression/import-std-module/list-dbg-info-content/TestDbgInfoContentListFromStdModule.py
lldb-api :: commands/expression/import-std-module/list/TestListFromStdModule.py
lldb-api :: commands/expression/import-std-module/no-std-module/TestMissingStdModule.py
lldb-api :: commands/expression/import-std-module/queue/TestQueueFromStdModule.py
lldb-api :: commands/expression/import-std-module/shared_ptr-dbg-info-content/TestSharedPtrDbgInfoContentFromStdModule.py
lldb-api :: commands/expression/import-std-module/shared_ptr/TestSharedPtrFromStdModule.py
lldb-api :: commands/expression/import-std-module/stack/TestStackFromStdModule.py
lldb-api :: commands/expression/import-std-module/unique_ptr-dbg-info-content/TestUniquePtrDbgInfoContent.py
lldb-api :: commands/expression/import-std-module/unique_ptr/TestUniquePtrFromStdModule.py
lldb-api :: commands/expression/import-std-module/vector-bool/TestVectorBoolFromStdModule.py
lldb-api :: commands/expression/import-std-module/vector-dbg-info-content/TestDbgInfoContentVectorFromStdModule.py
lldb-api :: commands/expression/import-std-module/vector-of-vectors/TestVectorOfVectorsFromStdModule.py
lldb-api :: commands/expression/import-std-module/vector/TestVectorFromStdModule.py
lldb-api :: commands/expression/import-std-module/weak_ptr-dbg-info-content/TestDbgInfoContentWeakPtrFromStdModule.py
lldb-api :: commands/expression/import-std-module/weak_ptr/TestWeakPtrFromStdModule.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/bitset/TestDataFormatterLibcxxBitset.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/forward_list/TestDataFormatterLibcxxForwardList.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/function/TestLibCxxFunction.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/iterator/TestDataFormatterLibccIterator.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/list/loop/TestDataFormatterLibcxxListLoop.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/multimap/TestDataFormatterLibccMultiMap.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/multiset/TestDataFormatterLibcxxMultiSet.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/optional/TestDataFormatterLibcxxOptional.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/queue/TestDataFormatterLibcxxQueue.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/string/TestDataFormatterLibcxxString.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/tuple/TestDataFormatterLibcxxTuple.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/unique_ptr/TestDataFormatterLibcxxUniquePtr.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/unordered/TestDataFormatterUnordered.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/variant/TestDataFormatterLibcxxVariant.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/vbool/TestDataFormatterLibcxxVBool.py
lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.py
lldb-api :: lang/cpp/std-function-step-into-callable/TestStdFunctionStepIntoCallable.py

25 tests on Windows:

Profile-x86_64 :: gcc-flag-compatibility.test
Profile-x86_64 :: instrprof-override-filename.c
cfi-standalone-lld-thinlto-x86_64 :: stats.cpp
cfi-standalone-lld-x86_64 :: stats.cpp
debuginfo-tests :: dexter/feature_tests/unittests/run.test
libomp :: affinity/format/affinity_values.c
libomp :: affinity/format/fields_values.c
libomp :: tasking/bug_nested_proxy_task.c
libomp :: tasking/bug_proxy_task_dep_waiting.c
lldb-shell :: SymbolFile/NativePDB/stack_unwinding01.cpp
lldb-shell :: SymbolFile/PDB/ast-restore.test
lldb-shell :: SymbolFile/PDB/class-layout.test
lldb-shell :: SymbolFile/PDB/enums-layout.test
lldb-shell :: SymbolFile/PDB/typedefs.test
lldb-shell :: SymbolFile/PDB/variables.test
lldb-unit :: Utility/./UtilityTests.exe/PassiveReplayTest.InstrumentedBar
lldb-unit :: Utility/./UtilityTests.exe/PassiveReplayTest.InstrumentedBarPtr
lldb-unit :: Utility/./UtilityTests.exe/PassiveReplayTest.InstrumentedBarRef
lldb-unit :: Utility/./UtilityTests.exe/PassiveReplayTest.InstrumentedFoo
lldb-unit :: Utility/./UtilityTests.exe/PassiveReplayTest.InstrumentedFooInvalid
lldb-unit :: Utility/./UtilityTests.exe/RecordReplayTest.InstrumentedBar
lldb-unit :: Utility/./UtilityTests.exe/RecordReplayTest.InstrumentedBarPtr
lldb-unit :: Utility/./UtilityTests.exe/RecordReplayTest.InstrumentedBarRef
lldb-unit :: Utility/./UtilityTests.exe/RecordReplayTest.InstrumentedFoo
lldb-unit :: Utility/./UtilityTests.exe/RecordReplayTest.InstrumentedFooSameThis

Thank you in advance!

[clayborg]

I added some folks from LLDB that are more familiar with the depths of the expression parser and how the JIT are used. Jim, Fred and Jonas, do you guys have any ideas on wether this diff could affect the LLDB expression parser when using JIT?

[jingham]

I don't know how this JIT Profiling code is wired into the regular JIT mechanism. I assume it is opt-in, in which case it shouldn't have any effect on expression parsing. If it isn't opt in, we should have a way to opt out on the "don't do work whose outcome you don't care about" principle.

[lhames]

@jingham @clayborg Do either of you recognize the error that seems to be triggered? It looks like it's consistent across the tests, or at least the ones that I looked at:

FAIL: test_non_cpp_language_dwarf (TestImportStdModule.ImportStdModule)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1825, in test_method
    return attrvalue(self)
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 134, in wrapper
    return func(*args, **kwargs)
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/test/API/commands/expression/import-std-module/basic/TestImportStdModule.py", line 38, in test_non_cpp_language
    lldbutil.run_to_source_breakpoint(self,
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 953, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 875, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5

[lhames]

It's not clear to me that Elena's patch should affect this. I'm inclined to accept it and just keep an eye on the bots. If everything passes we can chalk it up to a misconfig on Harbormaster. If the bots fail we can revert, and maybe the logs will give us some more insight into what's going on.

[jingham]

In D86435#2342826, @lhames wrote:

@jingham @clayborg Do either of you recognize the error that seems to be triggered? It looks like it's consistent across the tests, or at least the ones that I looked at:

FAIL: test_non_cpp_language_dwarf (TestImportStdModule.ImportStdModule)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1825, in test_method
    return attrvalue(self)
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/packages/Python/lldbsuite/test/decorators.py", line 134, in wrapper
    return func(*args, **kwargs)
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/test/API/commands/expression/import-std-module/basic/TestImportStdModule.py", line 38, in test_non_cpp_language
    lldbutil.run_to_source_breakpoint(self,
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 953, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/mnt/disks/ssd0/agent/llvm-project/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 875, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5

That's the test failing to stop at the first breakpoint. 10 -> eStateExited, 5 -> eStateStopped. This sequence, build the binary then do:

lldbutil.run_to_source_breakpoint(self,
                                  "// Set break point at this line.",
                                  lldb.SBFileSpec("main.cpp"))

to set a breakpoint on that source pattern and run to it is how the vast majority of the Shell tests that actually run a target start up. Moreover, in this test we actually did manage to resolve the breakpoint, or the test would have failed earlier. So we must have had debug info we thought was okay.

IIRC the bots use the just-built clang as the builder for the test files, so it's possible that the presence of this patch in the clang we are using for building messes up the line table info so that the breakpoint ends up somewhere that doesn't correspond to code we actually run??? If you have a builder which still has the built products available, it would be interesting to grab one of the ones that is failing like this and try:

(lldb) break set -p "// Set break point at this line."
(lldb) run

and see if it hits the breakpoint.

[ekovanov]

In D86435#2342836, @lhames wrote:

It's not clear to me that Elena's patch should affect this. I'm inclined to accept it and just keep an eye on the bots. If everything passes we can chalk it up to a misconfig on Harbormaster. If the bots fail we can revert, and maybe the logs will give us some more insight into what's going on.

Hi Lang,

That's sounds good to me.

Btw, previously I mentioned, that could not reproduce the tests failures on my system with my patch applied(master from 9th September).
But, I tried to run tests on master revision 4ce176bed2c4f88804e7d4bb9671637d88206e78 without my patch and the same tests failures reproduced.
(this revision was chosen because it is near the time I created review on Phabricator)

[clayborg]

I agree that I can't see how this would affect things. Fine with this going in and taking our chances.

[lhames]

In D86435#2348346, @clayborg wrote:

I agree that I can't see how this would affect things. Fine with this going in and taking our chances.

Ok -- Sounds good to me. Elena -- You're welcome to land this, and thanks for being patient with the review.

[ekovanov]

Thanks a lot for your help!
Being very new here, I do not have commit access to land this, could you please land this?

[lhames]

In D86435#2349843, @ekovanov wrote:

Thanks a lot for your help!
Being very new here, I do not have commit access to land this, could you please land this?

Hi Elena,

I missed that this included a definition of a git submodule. This would be the first in the main LLVM tree and I think we'd need to ask the community about inclusion. Is it required for this patch, or could it be dropped and downloading of that module left to clients who want to enable Intel JIT events support?

[ekovanov]

In D86435#2357748, @lhames wrote:

In D86435#2349843, @ekovanov wrote:

Thanks a lot for your help!
Being very new here, I do not have commit access to land this, could you please land this?

Hi Elena,

I missed that this included a definition of a git submodule. This would be the first in the main LLVM tree and I think we'd need to ask the community about inclusion. Is it required for this patch, or could it be dropped and downloading of that module left to clients who want to enable Intel JIT events support?

Hi Lang!

At my opinion, to add submodule is a good solution for open source project to be up to date with ITT API source.
The alternative solution - just add necessary source files from ITT API to the llvm/lib/ExecutionEngine/IntelJITEvents/ directory and manually update them from time to time.
How do you think, what is better to implement in my case?

[lhames]

In D86435#2359890, @ekovanov wrote:

In D86435#2357748, @lhames wrote:

In D86435#2349843, @ekovanov wrote:

Thanks a lot for your help!
Being very new here, I do not have commit access to land this, could you please land this?

Hi Elena,

I missed that this included a definition of a git submodule. This would be the first in the main LLVM tree and I think we'd need to ask the community about inclusion. Is it required for this patch, or could it be dropped and downloading of that module left to clients who want to enable Intel JIT events support?

Hi Lang!

At my opinion, to add submodule is a good solution for open source project to be up to date with ITT API source.
The alternative solution - just add necessary source files from ITT API to the llvm/lib/ExecutionEngine/IntelJITEvents/ directory and manually update them from time to time.
How do you think, what is better to implement in my case?

I'm not opposed to submodules, but I think you would need to start a thread on LLVM dev about it: I'm not the right person to approve that change.

In the interim, if you would like to land the rest of this patch, do you think it would be reasonable to require clients using intel JIT events to manually create and checkout the submodule? I think that would only be a couple of extra lines when setting up an LLVM checkout, right?

[ekovanov]

removed git submodule, added cmake action to clone ittapi repo

[ekovanov]

In D86435#2360978, @lhames wrote:

In D86435#2359890, @ekovanov wrote:

In D86435#2357748, @lhames wrote:

In D86435#2349843, @ekovanov wrote:

Thanks a lot for your help!
Being very new here, I do not have commit access to land this, could you please land this?

Hi Elena,

I missed that this included a definition of a git submodule. This would be the first in the main LLVM tree and I think we'd need to ask the community about inclusion. Is it required for this patch, or could it be dropped and downloading of that module left to clients who want to enable Intel JIT events support?

Hi Lang!

At my opinion, to add submodule is a good solution for open source project to be up to date with ITT API source.
The alternative solution - just add necessary source files from ITT API to the llvm/lib/ExecutionEngine/IntelJITEvents/ directory and manually update them from time to time.
How do you think, what is better to implement in my case?

I'm not opposed to submodules, but I think you would need to start a thread on LLVM dev about it: I'm not the right person to approve that change.

In the interim, if you would like to land the rest of this patch, do you think it would be reasonable to require clients using intel JIT events to manually create and checkout the submodule? I think that would only be a couple of extra lines when setting up an LLVM checkout, right?

Hi Lang,

I think I got your point and I fully agree with you - it does not necessary to create submodule.
An action to clone ITT API repo (CMakeLists.txt) is enough for this case.

[ekovanov]

Hi Lang,

Could you please take a look at the latest diff?
Is there something I should fix?

[ekovanov]

friendly reminder

[lhames]

Hi Elena,

Sorry for the delayed reply -- I've been traveling and I'm still catching up. This looks good to me -- I'll land it now. :)

• Lang.

llvm/lib/ExecutionEngine/IntelJITEvents/CMakeLists.txt

 include_directories( ${CMAKE_CURRENT_SOURCE_DIR}/.. )
 
+set(GIT_REPOSITORY https://github.com/intel/ittapi.git)
+set(GIT_TAG v3.18.8)
+
+if(NOT EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/ittapi)
+    execute_process(COMMAND ${GIT_EXECUTABLE} clone ${GIT_REPOSITORY}
+                    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+                    RESULT_VARIABLE GIT_CLONE_RESULT)
+    if(NOT GIT_CLONE_RESULT EQUAL "0")
+        message(FATAL_ERROR "git clone ${GIT_REPOSITORY} failed with ${GIT_CLONE_RESULT}, please clone ${GIT_REPOSITORY}")
+    endif()
+endif()
+
+execute_process(COMMAND ${GIT_EXECUTABLE} checkout ${GIT_TAG}
+                WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/ittapi
+                RESULT_VARIABLE GIT_CHECKOUT_RESULT)
+if(NOT GIT_CHECKOUT_RESULT EQUAL "0")
+    message(FATAL_ERROR "git checkout ${GIT_TAG} failed with ${GIT_CHECKOUT_RESULT}, please checkout ${GIT_TAG} at ${CMAKE_CURRENT_SOURCE_DIR}/ittapi")
+endif()
+
+include_directories( ${CMAKE_CURRENT_SOURCE_DIR}/ittapi/include/ )
+
 if( HAVE_LIBDL )
     set(LLVM_INTEL_JIT_LIBS ${CMAKE_DL_LIBS})
 endif()
 
 set(LLVM_INTEL_JIT_LIBS ${LLVM_PTHREAD_LIB} ${LLVM_INTEL_JIT_LIBS})
 
 
 add_llvm_component_library(LLVMIntelJITEvents
   IntelJITEventListener.cpp
   jitprofiling.c
+  ittapi/src/ittnotify/ittnotify_static.c
 
   LINK_LIBS ${LLVM_INTEL_JIT_LIBS}
 
   LINK_COMPONENTS
   CodeGen
   Core
   DebugInfoDWARF
   Support
   Object
   ExecutionEngine
 )
 
 add_dependencies(LLVMIntelJITEvents LLVMCodeGen)

llvm/lib/ExecutionEngine/IntelJITEvents/IntelJITEventListener.cpp

 //===-- IntelJITEventListener.cpp - Tell Intel profiler about JITed code --===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines a JITEventListener object to tell Intel(R) VTune(TM)
 // Amplifier XE 2011 about JITted functions.
 //
 //===----------------------------------------------------------------------===//
 
 #include "IntelJITEventsWrapper.h"
+#include "ittnotify.h"
 #include "llvm-c/ExecutionEngine.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/Config/config.h"
 #include "llvm/DebugInfo/DIContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/ExecutionEngine/JITEventListener.h"
 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/ValueHandle.h"
 #include "llvm/Object/ObjectFile.h"
 #include "llvm/Object/SymbolSize.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Errno.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace llvm;
 using namespace llvm::object;
 
 #define DEBUG_TYPE "amplifier-jit-event-listener"
 
 namespace {
 
+class IntelIttnotifyInfo {
+  std::string ModuleName;
+  std::vector<std::string> SectionNamesVector;
+  std::vector<__itt_section_info> SectionInfoVector;
+  __itt_module_object *ModuleObject;
+  IntelJITEventsWrapper &WrapperRef;
+
+public:
+  IntelIttnotifyInfo(IntelJITEventsWrapper &Wrapper)
+      : ModuleObject(NULL), WrapperRef(Wrapper){};
+  ~IntelIttnotifyInfo() { delete ModuleObject; };
+
+  void setModuleName(const char *Name) { ModuleName = std::string(Name); }
+
+  const char *getModuleName() { return ModuleName.c_str(); }
+
+  void setModuleObject(__itt_module_object *ModuleObj) {
+    ModuleObject = ModuleObj;
+  }
+
+  __itt_module_object *getModuleObject() { return ModuleObject; }
+
+  __itt_section_info *getSectionInfoVectorBegin() {
+    if (SectionInfoVector.size())
+      return &SectionInfoVector[0];
+    return NULL;
+  }
+
+  void reportSection(llvm::IttEventType EventType, const char *SectionName,
+                     unsigned int SectionSize) {
+    WrapperRef.iJitIttNotifyInfo(EventType, SectionName, SectionSize);
+  }
+
+  int fillSectionInformation(const ObjectFile &Obj,
+                             const RuntimeDyld::LoadedObjectInfo &L) {
+
+    int SectionCounter = 0;
+
+    for (auto &Section : Obj.sections()) {
+      uint64_t SectionLoadAddr = L.getSectionLoadAddress(Section);
+      if (SectionLoadAddr) {
+        object::ELFSectionRef ElfSection(Section);
+
+        __itt_section_info SectionInfo;
+        memset(&SectionInfo, 0, sizeof(SectionInfo));
+        SectionInfo.start_addr = reinterpret_cast<void *>(SectionLoadAddr);
+        SectionInfo.file_offset = ElfSection.getOffset();
+        SectionInfo.flags = ElfSection.getFlags();
+
+        StringRef SectionName("");
+        auto SectionNameOrError = ElfSection.getName();
+        if (SectionNameOrError)
+          SectionName = *SectionNameOrError;
+
+        SectionNamesVector.push_back(SectionName.str());
+        SectionInfo.size = ElfSection.getSize();
+        reportSection(llvm::LoadBinarySection, SectionName.str().c_str(),
+                      SectionInfo.size);
+
+        if (ElfSection.isBSS()) {
+          SectionInfo.type = itt_section_type_bss;
+        } else if (ElfSection.isData()) {
+          SectionInfo.type = itt_section_type_data;
+        } else if (ElfSection.isText()) {
+          SectionInfo.type = itt_section_type_text;
+        }
+        SectionInfoVector.push_back(SectionInfo);
+        ++SectionCounter;
+      }
+    }
+    // Hereinafter: don't change SectionNamesVector content to avoid vector
+    // reallocation - reallocation invalidates all the references, pointers, and
+    // iterators referring to the elements in the sequence.
+    for (int I = 0; I < SectionCounter; ++I) {
+      SectionInfoVector[I].name = SectionNamesVector[I].c_str();
+    }
+    return SectionCounter;
+  }
+};
+
 class IntelJITEventListener : public JITEventListener {
   typedef DenseMap<void*, unsigned int> MethodIDMap;
 
   std::unique_ptr<IntelJITEventsWrapper> Wrapper;
   MethodIDMap MethodIDs;
 
   typedef SmallVector<const void *, 64> MethodAddressVector;
   typedef DenseMap<const void *, MethodAddressVector>  ObjectMap;
 
   ObjectMap  LoadedObjectMap;
   std::map<ObjectKey, OwningBinary<ObjectFile>> DebugObjects;
 
+  std::map<ObjectKey, std::unique_ptr<IntelIttnotifyInfo>> KeyToIttnotify;
+
 public:
   IntelJITEventListener(IntelJITEventsWrapper* libraryWrapper) {
       Wrapper.reset(libraryWrapper);
   }
 
   ~IntelJITEventListener() {
   }
 
static iJIT_Method_Load FunctionDescToIntelJITFormat(
   Result.class_file_name = NULL;
   Result.user_data = NULL;
   Result.user_data_size = 0;
   Result.env = iJDE_JittingAPI;
 
   return Result;
 }
 
+int getBackwardCompatibilityMode() {
+
+  char *BackwardCompatibilityEnv = getenv("INTEL_JIT_BACKWARD_COMPATIBILITY");
+  int BackwardCompatibilityMode = 0;
+  if (BackwardCompatibilityEnv) {
+    StringRef(BackwardCompatibilityEnv)
+        .getAsInteger(10, BackwardCompatibilityMode);
+  }
+  return BackwardCompatibilityMode;
+}
+
 void IntelJITEventListener::notifyObjectLoaded(
     ObjectKey Key, const ObjectFile &Obj,
     const RuntimeDyld::LoadedObjectInfo &L) {
 
+  int BackwardCompatibilityMode = getBackwardCompatibilityMode();
+  if (BackwardCompatibilityMode == 0) {
+    if (Obj.isELF()) {
+      std::unique_ptr<IntelIttnotifyInfo> ModuleIttnotify =
+          std::make_unique<IntelIttnotifyInfo>(*Wrapper);
+      ModuleIttnotify->setModuleName(
+          StringRef(llvm::utohexstr(
+                        MD5Hash(Obj.getMemoryBufferRef().getBuffer()), true))
+              .str()
+              .c_str());
+
+      __itt_module_object *ModuleObject = new __itt_module_object();
+      ModuleObject->module_name = ModuleIttnotify->getModuleName();
+      ModuleObject->module_size = Obj.getMemoryBufferRef().getBufferSize();
+      Wrapper->iJitIttNotifyInfo(llvm::LoadBinaryModule,
+                                 ModuleObject->module_name,
+                                 ModuleObject->module_size);
+      ModuleObject->module_type = __itt_module_type_elf;
+      ModuleObject->section_number =
+          ModuleIttnotify->fillSectionInformation(Obj, L);
+      ModuleObject->module_buffer =
+          (void *)const_cast<char *>(Obj.getMemoryBufferRef().getBufferStart());
+      ModuleObject->module_id =
+          __itt_id_make((void *)&(*ModuleObject), ModuleObject->module_size);
+      ModuleObject->section_array =
+          ModuleIttnotify->getSectionInfoVectorBegin();
+      ModuleIttnotify->setModuleObject(ModuleObject);
+
+      __itt_module_load_with_sections(ModuleObject);
+
+      KeyToIttnotify[Key] = std::move(ModuleIttnotify);
+    }
+  } else if (BackwardCompatibilityMode == 1) {
+
     OwningBinary<ObjectFile> DebugObjOwner = L.getObjectForDebug(Obj);
     const ObjectFile *DebugObj = DebugObjOwner.getBinary();
     if (!DebugObj)
       return;
 
     // Get the address of the object image for use as a unique identifier
-  const void* ObjData = DebugObj->getData().data();
+    const void *ObjData = DebugObj->getData().data();
     std::unique_ptr<DIContext> Context = DWARFContext::create(*DebugObj);
     MethodAddressVector Functions;
 
     // Use symbol info to iterate functions in the object.
-  for (const std::pair<SymbolRef, uint64_t> &P : computeSymbolSizes(*DebugObj)) {
+    for (const std::pair<SymbolRef, uint64_t> &P :
+         computeSymbolSizes(*DebugObj)) {
       SymbolRef Sym = P.first;
       std::vector<LineNumberInfo> LineInfo;
       std::string SourceFileName;
 
       Expected<SymbolRef::Type> SymTypeOrErr = Sym.getType();
       if (!SymTypeOrErr) {
         // TODO: Actually report errors helpfully.
         consumeError(SymTypeOrErr.takeError());
         continue;
       }
       SymbolRef::Type SymType = *SymTypeOrErr;
       if (SymType != SymbolRef::ST_Function)
         continue;
 
       Expected<StringRef> Name = Sym.getName();
       if (!Name) {
         // TODO: Actually report errors helpfully.
         consumeError(Name.takeError());
         continue;
       }
 
       Expected<uint64_t> AddrOrErr = Sym.getAddress();
       if (!AddrOrErr) {
         // TODO: Actually report errors helpfully.
         consumeError(AddrOrErr.takeError());
         continue;
       }
       uint64_t Addr = *AddrOrErr;
       uint64_t Size = P.second;
 
       auto SecOrErr = Sym.getSection();
       if (!SecOrErr) {
         // TODO: Actually report errors helpfully.
         consumeError(SecOrErr.takeError());
         continue;
       }
       object::section_iterator Sec = *SecOrErr;
       if (Sec == Obj.section_end())
         continue;
       uint64_t Index = Sec->getIndex();
 
       // Record this address in a local vector
-    Functions.push_back((void*)Addr);
+      Functions.push_back((void *)Addr);
 
       // Build the function loaded notification message
       iJIT_Method_Load FunctionMessage =
           FunctionDescToIntelJITFormat(*Wrapper, Name->data(), Addr, Size);
       DILineInfoTable Lines =
           Context->getLineInfoForAddressRange({Addr, Index}, Size);
       DILineInfoTable::iterator Begin = Lines.begin();
       DILineInfoTable::iterator End = Lines.end();
       for (DILineInfoTable::iterator It = Begin; It != End; ++It) {
         LineInfo.push_back(
             DILineInfoToIntelJITFormat((uintptr_t)Addr, It->first, It->second));
       }
       if (LineInfo.size() == 0) {
         FunctionMessage.source_file_name = 0;
         FunctionMessage.line_number_size = 0;
         FunctionMessage.line_number_table = 0;
       } else {
         // Source line information for the address range is provided as
         // a code offset for the start of the corresponding sub-range and
         // a source line. JIT API treats offsets in LineNumberInfo structures
         // as the end of the corresponding code region. The start of the code
         // is taken from the previous element. Need to shift the elements.
 
         LineNumberInfo last = LineInfo.back();
         last.Offset = FunctionMessage.method_size;
         LineInfo.push_back(last);
         for (size_t i = LineInfo.size() - 2; i > 0; --i)
           LineInfo[i].LineNumber = LineInfo[i - 1].LineNumber;
 
         SourceFileName = Lines.front().second.FileName;
         FunctionMessage.source_file_name =
             const_cast<char *>(SourceFileName.c_str());
         FunctionMessage.line_number_size = LineInfo.size();
         FunctionMessage.line_number_table = &*LineInfo.begin();
       }
 
       Wrapper->iJIT_NotifyEvent(iJVM_EVENT_TYPE_METHOD_LOAD_FINISHED,
                                 &FunctionMessage);
-    MethodIDs[(void*)Addr] = FunctionMessage.method_id;
+      MethodIDs[(void *)Addr] = FunctionMessage.method_id;
     }
 
     // To support object unload notification, we need to keep a list of
     // registered function addresses for each loaded object.  We will
     // use the MethodIDs map to get the registered ID for each function.
     LoadedObjectMap[ObjData] = Functions;
     DebugObjects[Key] = std::move(DebugObjOwner);
   }
+}
 
 void IntelJITEventListener::notifyFreeingObject(ObjectKey Key) {
+
+  int BackwardCompatibilityMode = getBackwardCompatibilityMode();
+  if (BackwardCompatibilityMode == 0) {
+    if (KeyToIttnotify.find(Key) == KeyToIttnotify.end())
+      return;
+    __itt_module_unload_with_sections(KeyToIttnotify[Key]->getModuleObject());
+    Wrapper->iJitIttNotifyInfo(
+        llvm::UnloadBinaryModule,
+        KeyToIttnotify[Key]->getModuleObject()->module_name,
+        KeyToIttnotify[Key]->getModuleObject()->module_size);
+    KeyToIttnotify.erase(Key);
+  } else if (BackwardCompatibilityMode == 1) {
     // This object may not have been registered with the listener. If it wasn't,
     // bail out.
     if (DebugObjects.find(Key) == DebugObjects.end())
       return;
 
     // Get the address of the object image for use as a unique identifier
     const ObjectFile &DebugObj = *DebugObjects[Key].getBinary();
-  const void* ObjData = DebugObj.getData().data();
+    const void *ObjData = DebugObj.getData().data();
 
     // Get the object's function list from LoadedObjectMap
     ObjectMap::iterator OI = LoadedObjectMap.find(ObjData);
     if (OI == LoadedObjectMap.end())
       return;
-  MethodAddressVector& Functions = OI->second;
+    MethodAddressVector &Functions = OI->second;
 
     // Walk the function list, unregistering each function
     for (MethodAddressVector::iterator FI = Functions.begin(),
                                        FE = Functions.end();
-       FI != FE;
-       ++FI) {
-    void* FnStart = const_cast<void*>(*FI);
+         FI != FE; ++FI) {
+      void *FnStart = const_cast<void *>(*FI);
       MethodIDMap::iterator MI = MethodIDs.find(FnStart);
       if (MI != MethodIDs.end()) {
         Wrapper->iJIT_NotifyEvent(iJVM_EVENT_TYPE_METHOD_UNLOAD_START,
                                   &MI->second);
         MethodIDs.erase(MI);
       }
     }
 
     // Erase the object from LoadedObjectMap
     LoadedObjectMap.erase(OI);
     DebugObjects.erase(Key);
   }
+}
 
 }  // anonymous namespace.
 
 namespace llvm {
 JITEventListener *JITEventListener::createIntelJITEventListener() {
   return new IntelJITEventListener(new IntelJITEventsWrapper);
 }
 

llvm/lib/ExecutionEngine/IntelJITEvents/IntelJITEventsWrapper.h

 
 #ifndef INTEL_JIT_EVENTS_WRAPPER_H
 #define INTEL_JIT_EVENTS_WRAPPER_H
 
 #include "jitprofiling.h"
 
 namespace llvm {
 
+typedef enum {
+  LoadBinaryModule,
+  LoadBinarySection,
+  UnloadBinaryModule,
+  UnloadBinarySection
+} IttEventType;
+
 class IntelJITEventsWrapper {
   // Function pointer types for testing implementation of Intel jitprofiling
   // library
   typedef int (*NotifyEventPtr)(iJIT_JVM_EVENT, void*);
+  typedef int (*IttnotifyInfoPtr)(IttEventType, const char *, unsigned int);
   typedef void (*RegisterCallbackExPtr)(void *, iJIT_ModeChangedEx );
   typedef iJIT_IsProfilingActiveFlags (*IsProfilingActivePtr)(void);
   typedef void (*FinalizeThreadPtr)(void);
   typedef void (*FinalizeProcessPtr)(void);
   typedef unsigned int (*GetNewMethodIDPtr)(void);
 
   NotifyEventPtr NotifyEventFunc;
+  IttnotifyInfoPtr IttnotifyInfoFunc;
   RegisterCallbackExPtr RegisterCallbackExFunc;
   IsProfilingActivePtr IsProfilingActiveFunc;
   GetNewMethodIDPtr GetNewMethodIDFunc;
 
 public:
   bool isAmplifierRunning() {
     return iJIT_IsProfilingActive() == iJIT_SAMPLING_ON;
   }
 
   IntelJITEventsWrapper()
-  : NotifyEventFunc(::iJIT_NotifyEvent),
+      : NotifyEventFunc(::iJIT_NotifyEvent), IttnotifyInfoFunc(0),
         RegisterCallbackExFunc(::iJIT_RegisterCallbackEx),
         IsProfilingActiveFunc(::iJIT_IsProfilingActive),
-    GetNewMethodIDFunc(::iJIT_GetNewMethodID) {
-  }
+        GetNewMethodIDFunc(::iJIT_GetNewMethodID) {}
 
   IntelJITEventsWrapper(NotifyEventPtr NotifyEventImpl,
+                        IttnotifyInfoPtr IttnotifyInfoImpl,
                         RegisterCallbackExPtr RegisterCallbackExImpl,
                         IsProfilingActivePtr IsProfilingActiveImpl,
                         FinalizeThreadPtr FinalizeThreadImpl,
                         FinalizeProcessPtr FinalizeProcessImpl,
                         GetNewMethodIDPtr GetNewMethodIDImpl)
-  : NotifyEventFunc(NotifyEventImpl),
+      : NotifyEventFunc(NotifyEventImpl), IttnotifyInfoFunc(IttnotifyInfoImpl),
         RegisterCallbackExFunc(RegisterCallbackExImpl),
         IsProfilingActiveFunc(IsProfilingActiveImpl),
-    GetNewMethodIDFunc(GetNewMethodIDImpl) {
-  }
+        GetNewMethodIDFunc(GetNewMethodIDImpl) {}
 
   // Sends an event announcing that a function has been emitted
   //   return values are event-specific.  See Intel documentation for details.
   int  iJIT_NotifyEvent(iJIT_JVM_EVENT EventType, void *EventSpecificData) {
     if (!NotifyEventFunc)
       return -1;
     return NotifyEventFunc(EventType, EventSpecificData);
   }
 
+  int iJitIttNotifyInfo(IttEventType EventType, const char *Name,
+                        unsigned int Size) {
+    if (!IttnotifyInfoFunc)
+      return -1;
+    return IttnotifyInfoFunc(EventType, Name, Size);
+  }
+
   // Registers a callback function to receive notice of profiling state changes
   void iJIT_RegisterCallbackEx(void *UserData,
                                iJIT_ModeChangedEx NewModeCallBackFuncEx) {
     if (RegisterCallbackExFunc)
       RegisterCallbackExFunc(UserData, NewModeCallBackFuncEx);
   }
 
   // Returns the current profiler mode

llvm/test/JitListener/lit.local.cfg

 if not config.root.llvm_use_intel_jitevents:
     config.unsupported = True
+config.environment['INTEL_JIT_BACKWARD_COMPATIBILITY'] = '1'

llvm/tools/llvm-jitlistener/llvm-jitlistener.cpp

switch (EventType) {
     }
     break;
     default:
       break;
   }
   return 0;
 }
 
+int ittNotifyInfo(IttEventType EventType, const char *Name, unsigned int Size) {
+  switch (EventType) {
+  case LoadBinaryModule: {
+    if (!Name) {
+      errs() << "Error: The IttNotify event listener did not provide a module "
+                "name.";
+      return -1;
+    }
+    outs() << "Module loaded : Name = " << Name << ", Size = " << Size << "\n";
+  } break;
+  case LoadBinarySection: {
+    if (!Name) {
+      errs() << "Error: The IttNotify event listener did not provide a section "
+                "name.";
+      return -1;
+    }
+    outs() << "Loaded section : Name = " << Name << ", Size = " << Size << "\n";
+  } break;
+  case UnloadBinaryModule: {
+    if (!Name) {
+      errs() << "Error: The IttNotify event listener did not provide a module "
+                "name.";
+      return -1;
+    }
+    outs() << "Module unloaded : Name = " << Name << ", Size = " << Size
+           << "\n";
+  } break;
+  case UnloadBinarySection: {
+    if (!Name) {
+      errs() << "Error: The IttNotify event listener did not provide a section "
+                "name.";
+      return -1;
+    }
+    outs() << "Unloaded section : Name = " << Name << ", Size = " << Size
+           << "\n";
+  } break;
+  }
+  return 0;
+}
+
 iJIT_IsProfilingActiveFlags IsProfilingActive(void) {
   // for testing, pretend we have an Intel Parallel Amplifier XE 2011
   // instance attached
   return iJIT_SAMPLING_ON;
 }
 
 unsigned int GetNewMethodID(void) {
   static unsigned int id = 0;
protected:
   std::unique_ptr<ExecutionEngine> TheJIT;
 
 public:
   void ProcessInput(const std::string &Filename) {
     InitEE(Filename);
 
     std::unique_ptr<llvm::JITEventListener> Listener(
         JITEventListener::createIntelJITEventListener(new IntelJITEventsWrapper(
-            NotifyEvent, 0, IsProfilingActive, 0, 0, GetNewMethodID)));
+            NotifyEvent, ittNotifyInfo, 0, IsProfilingActive, 0, 0,
+            GetNewMethodID)));
 
     TheJIT->RegisterJITEventListener(Listener.get());
 
     TheJIT->finalizeObject();
 
     // Destroy the JIT engine instead of unregistering to get unload events.
     DestroyEE();
   }