Using MPI for Profiling Data Reduction

[grotdunst]

Summary

Distributed-memory systems, from small clusters to supercomputers, use MPI (http://www.mpi-forum.org/) to manage inter-process communication. On large systems, a single job might consist of millions of simultaneously-running processes. Even though millions of processes are rare, tens of thousands are now common. Profiling such applications using our current infrastructure is difficult because it requires generating, and then merging, many thousands of files. On the distributed file systems often used on these machines, creating so many files often leads to performance problems (in addition to being difficult to manage). This problem can be eliminated by teaching compiler-rt how to aggregate the profiling counters so that only one process writes out the combined profiling data for all processes. This patch adds this functionality whenever compiler-rt is compiled with an available mpi.h header.

Any well-formed MPI program calls MPI_Finalize to shut down its communication resources, making them unavailable by the time atexit handlers are called. By design, all MPI functions used by MPI applications are provided as weak symbols. The strong symbols are provided using PMPI_* names. Thus, MPI_Finalize, by default, is a weak function calling PMPI_Finalize. This allows tools to easily hook the MPI functions used by applications while easily retaining the ability to access the underlying implementation. This is utilized within the new file InstrProfilingReduce.c, where each processor’s counters are aggregated onto just one processor before terminating the parallel environment.

The order in which libraries are included on Clang’s command line was leveraged to ensure compatibility across all applications and computing environments. Programs that include MPI functions will be linked with the actual MPI library, since compiler-rt libraries are linked last. However, ones that do not will simply link using the stub function implementations in InstrProfilingStub.c.

To accomplish the goal of limiting the file-writing to one process (for applications that properly call MPI_Finalize), we needed to change the way that the truncation of existing files works. Currently, we use fopen to force the truncation of any pre-existing profiling output at application startup, and then later open the file to actually write the collected profiling data. When using MPI, we only want the process identified as MPI rank 0 to create any profiling outputs. This cannot be determined at startup, and so we need to delay the initial creation of the profiling outputs until after MPI_Finalize is called. To do this, we change the current behavior (at least on POSIX systems) so that existing profiling outputs are truncated only if they already exist (i.e. the file is opened without O_CREAT). If an output does not exist, we don't create an empty file, but rather, create the file later when it is opened for writing. This seems like a generally-beneficial change for all systems.

Lastly, the testing file instrprof-reduce.c was created to test that only processes assigned to rank 0 produce a profiling file when MPI_Finalize is called. Tests were also done to ensure that profiling counts increased proportionally with the number of processes involved in a program.

[davidxl]

Nice! This is the complement of the in-process profile merging (for single
node or distributed build with network file system) I plan to add. I will
get to the actual review shortly.

thanks,

David

[davidxl]

lib/profile/InstrProfilingFile.c

29

is fcntl.h needed here?

29

This seems like a good variable to have unconditionally. Put the declaration in InstrProfiling.h file.

96–97

The behavior divergence between windows and other platforms is not desirable. How about using

_open(Filename, _O_WRONLY | _O_TRUNC)
...

107

How about just calling truncate(..) method?

lib/profile/InstrProfilingReduce.c

15 ↗

(On Diff #50199)

Define this variable unconditionally in InstrProfiling.c file.

24 ↗

(On Diff #50199)

Notice that value profile data is not 'reduced' here. It can be non-trivial to do so, but please add a comment about the limitation.

30 ↗

(On Diff #50199)

Should return be used for Rank !=0 cases too?

lib/profile/InstrProfilingStub.c

12 ↗

(On Diff #50199)

guard these with #ifdef HAVE_MPI_H?

14 ↗

(On Diff #50199)

Is the MPI library defining thse functions static or a DSO? If the former, then perhaps declare those stubs weak so that we don't rely on the link order of the runtime library

test/profile/instrprof-reduce.c

17

when HAVE_MPI_H is not defined, will this test case link? MPI_Finalize is not defined anywhere.

[silvas]

lib/profile/InstrProfilingFile.c

84

Let's split this change into a separate patch for a focused discussion on the requirements and possible solutions.

lib/profile/InstrProfilingReduce.c

1 ↗

(On Diff #50199)

Let's call this InstrProfilingMPI.c consistent with the comment.

lib/profile/InstrProfilingStub.c

14 ↗

(On Diff #50199)

Why do we need these stubs?

[hfinkel]

lib/profile/InstrProfilingReduce.c

30 ↗

(On Diff #50199)

It is used in both cases, however this is not visually obvious because the else statement has no braces.

Please add braces around the else body (for consistency with the if body).

lib/profile/InstrProfilingStub.c

14 ↗

(On Diff #50199)

I'm not sure defining these as weak solves the dependency-on-link-order problem because, except for PMPI_Finalize, they're weak in the actual MPI library too.

Also, to be more specific, we're depending on the fact that the linker won't use any symbols from this object file because nothing in this object file is required (when there is an MPI implementation that also provides these symbols). This might be a link order effect, or it could just be the fact that the MPI library is required to resolve other symbols, I'm not sure which.

[davidxl]

lib/profile/InstrProfilingReduce.c

30 ↗

(On Diff #50199)

Right -- my eyes certainly fooled me in this case.

[grotdunst]

lib/profile/InstrProfilingFile.c

29

It seems to be needed for file truncation in Windows. Therefore, this has been kept, but inside the _MSC_VER guard.

84

I decided to not separate the patches because file truncation and profiling data reduction are still logically connected. The __llvm_write_prof_data variable is used within this file, but its initialization needs to be kept in the new file that hooks to MPI_Finalize so that the new implementation and function hooks will be used by the linker.

lib/profile/InstrProfilingReduce.c

15 ↗

(On Diff #50199)

This change was not made, because otherwise the linker will not be forced to use this file, thereby bringing in the expanded implementation of MPI_Finalize and the new stub functions.

test/profile/instrprof-reduce.c

17

A stub implementation of MPI_Finalize has now been included that will be used when HAVE_MPI_H is not defined.

[grotdunst]

Addressed comments from yesterday's patch.

[grotdunst]

Errors were found when putting stubs in same file as hook to MPI_Finalize. Putting them in their own file removed the errors.

[davidxl]

lib/profile/InstrProfiling.h

16

Make the name conform to the 'name space' convention:

--> __llvm_profile_write_data

lib/profile/InstrProfilingMPI.c

43

Why not put the stub def in the test case itself?

#ifndef HAVE_MPI_H
...
#endif

lib/profile/InstrProfilingStubs.c

14

what are the issues of merging this into InstrProfilingMPI.c?

test/profile/instrprof-reduce.c

16

With the stub definition for MPI_Finalize when HAVE_MPI_H is not defined, this test case will link, but the run will still fail -- so it will won't work.

I suggest adding the stub definition of MPI_Finalize like this:

... MPI_Finalize() {

 if (ThisRank != 0) {
  __llvm_write_prof_data = 0;
 }
return 0;

}

[davidxl]

Any update on this patch?

[grotdunst]

No not in the past two weeks. I've had graduation and a job interview in
that time but I should be able to refocus on this today.

CMakeLists.txt

 # COMPILER_RT_DEBUG_PYBOOL is used by lit.common.configured.in.
 pythonize_bool(COMPILER_RT_DEBUG)
 
 #================================
 # Setup Compiler Flags
 #================================
 include(CheckIncludeFile)
 check_include_file(unwind.h HAVE_UNWIND_H)
+check_include_file(mpi.h HAVE_MPI_H)
 
 include(config-ix)
 
 if(MSVC)
   append_string_if(COMPILER_RT_HAS_W3_FLAG /W3 CMAKE_C_FLAGS CMAKE_CXX_FLAGS)
 else()
   append_string_if(COMPILER_RT_HAS_WALL_FLAG -Wall CMAKE_C_FLAGS CMAKE_CXX_FLAGS)
 endif()

lib/profile/CMakeLists.txt

set(PROFILE_SOURCES
   InstrProfilingBuffer.c
   InstrProfilingFile.c
   InstrProfilingMerge.c
   InstrProfilingMergeFile.c
   InstrProfilingWriter.c
   InstrProfilingPlatformDarwin.c
   InstrProfilingPlatformLinux.c
   InstrProfilingPlatformOther.c
+  InstrProfilingMPI.c
   InstrProfilingRuntime.cc
+  InstrProfilingStubs.c
   InstrProfilingUtil.c)
 
 if(WIN32)
     list(APPEND PROFILE_SOURCES WindowsMMap.c)
 endif()
 
 if(UNIX)
  set(EXTRA_FLAGS
      -fPIC
      -Wno-pedantic)
 endif()
 
 if(COMPILER_RT_TARGET_HAS_ATOMICS)
  set(EXTRA_FLAGS
      ${EXTRA_FLAGS}
      -DCOMPILER_RT_HAS_ATOMICS=1)
 endif() 
 
+if(HAVE_MPI_H)
+ set(EXTRA_FLAGS
+     ${EXTRA_FLAGS}
+     -DHAVE_MPI_H=1)
+endif() 
+
 if(APPLE)
   add_compiler_rt_runtime(clang_rt.profile
     STATIC
     OS ${PROFILE_SUPPORTED_OS}
     ARCHS ${PROFILE_SUPPORTED_ARCH}
     SOURCES ${PROFILE_SOURCES}
     PARENT_TARGET profile)
 else()

lib/profile/InstrProfiling.h

 /*===- InstrProfiling.h- Support library for PGO instrumentation ----------===*\
 |*
 |*                     The LLVM Compiler Infrastructure
 |*
 |* This file is distributed under the University of Illinois Open Source
 |* License. See LICENSE.TXT for details.
 |*
 \*===----------------------------------------------------------------------===*/
 
 #ifndef PROFILE_INSTRPROFILING_H_
 #define PROFILE_INSTRPROFILING_H_
 
 #include "InstrProfilingPort.h"
 #include "InstrProfData.inc"
 
+COMPILER_RT_VISIBILITY extern int __llvm_write_prof_data;
+
 enum ValueKind {
 #define VALUE_PROF_KIND(Enumerator, Value) Enumerator = Value,
 #include "InstrProfData.inc"
 };
 
 typedef void *IntPtrT;
 typedef struct COMPILER_RT_ALIGNAS(INSTR_PROF_DATA_ALIGNMENT)
     __llvm_profile_data {

lib/profile/InstrProfilingFile.c

 /*===- InstrProfilingFile.c - Write instrumentation to a file -------------===*\
 |*
 |*                     The LLVM Compiler Infrastructure
 |*
 |* This file is distributed under the University of Illinois Open Source
 |* License. See LICENSE.TXT for details.
 |*
 \*===----------------------------------------------------------------------===*/
 
 #include "InstrProfiling.h"
 #include "InstrProfilingInternal.h"
 #include "InstrProfilingUtil.h"
 #include <errno.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+
+#ifdef _MSC_VER
+#include <fcntl.h>
+#include <io.h>
+#else
+#include <unistd.h>
+#endif
 
 #define UNCONST(ptr) ((void *)(uintptr_t)(ptr))
 
 /* Return 1 if there is an error, otherwise return  0.  */
 static uint32_t fileWriter(ProfDataIOVec *IOVecs, uint32_t NumIOVecs,
                            void **WriterCtx) {
   uint32_t I;
   FILE *File = (FILE *)*WriterCtx;
   for (I = 0; I < NumIOVecs; I++) {
     if (fwrite(IOVecs[I].Data, IOVecs[I].ElmSize, IOVecs[I].NumElm, File) !=
         IOVecs[I].NumElm)
       return 1;
static int writeFileWithName(const char *OutputName) {
   return RetVal;
 }
 
 COMPILER_RT_WEAK int __llvm_profile_OwnsFilename = 0;
 COMPILER_RT_WEAK const char *__llvm_profile_CurrentFilename = NULL;
 
 static void truncateCurrentFile(void) {
   const char *Filename;
-  FILE *File;
 
   Filename = __llvm_profile_CurrentFilename;
   if (!Filename || !Filename[0])
     return;
 
   /* Create the directory holding the file, if needed. */
   if (strchr(Filename, '/')) {
     char *Copy = malloc(strlen(Filename) + 1);
     strcpy(Copy, Filename);
     __llvm_profile_recursive_mkdir(Copy);
     free(Copy);
   }
 
-  /* Truncate the file.  Later we'll reopen and append. */
-  File = fopen(Filename, "w");
-  if (!File)
+  /* truncate, or open without O_CREAT, is used to avoid creating empty files.
+   * In MPI applications, a file will only be created by one process */
+#ifdef _MSC_VER
+  int File = _open(Filename, _O_WRONLY | _O_TRUNC);
+  if (File == -1)
     return;
-  fclose(File);
+  _close(File);
+#else
+  (void) truncate(Filename, 0);
+#endif
 }
 
 static void setFilename(const char *Filename, int OwnsFilename) {
   /* Check if this is a new filename and therefore needs truncation. */
   int NewFile = !__llvm_profile_CurrentFilename ||
       (Filename && strcmp(Filename, __llvm_profile_CurrentFilename));
   if (__llvm_profile_OwnsFilename)
     free(UNCONST(__llvm_profile_CurrentFilename));
 
void __llvm_profile_override_default_filename(const char *Filename) {
   /* If the env var is set, skip setting filename from argument. */
   const char *Env_Filename = getenv("LLVM_PROFILE_FILE");
   if (Env_Filename && Env_Filename[0])
     return;
   setFilenamePossiblyWithPid(Filename);
 }
 
 COMPILER_RT_VISIBILITY
-int __llvm_profile_write_file(void) {
+int __llvm_profile_write_file(void) {  
+  if (!__llvm_write_prof_data)
+      return 0;
+
   int rc;
 
   GetEnvHook = &getenv;
   /* Check the filename. */
   if (!__llvm_profile_CurrentFilename) {
     PROF_ERR("LLVM Profile: Failed to write file : %s\n", "Filename not set");
     return -1;
   }

lib/profile/InstrProfilingMPI.c

+/*===-- InstrProfilingMPI.c - Support library for applications using MPI --===*\
+|*
+|*                     The LLVM Compiler Infrastructure
+|*
+|* This file is distributed under the University of Illinois Open Source
+|* License. See LICENSE.TXT for details.
+|*
+\*===----------------------------------------------------------------------===*/
+
+#include <stdlib.h>
+#include <InstrProfiling.h>
+#ifdef HAVE_MPI_H
+#include <mpi.h>
+#endif
+
+/* Defined here to make sure that the linker brings in the MPI_Finalize
+ * definition provided in this file */
+COMPILER_RT_VISIBILITY int __llvm_write_prof_data = 1;
+
+#ifdef HAVE_MPI_H
+COMPILER_RT_VISIBILITY int MPI_Finalize() {
+   int Rank;
+   uint64_t *CountersBegin = __llvm_profile_begin_counters();
+   uint64_t *CountersEnd = __llvm_profile_end_counters();
+   uint64_t CountersSize = CountersEnd - CountersBegin;
+
+   PMPI_Comm_rank(MPI_COMM_WORLD, &Rank);
+
+   /* Only counters are reduced here, not value profile data */
+   if (Rank) {
+      PMPI_Reduce(CountersBegin, NULL, CountersSize, MPI_UINT64_T, MPI_SUM,
+              0, MPI_COMM_WORLD);
+      __llvm_write_prof_data = 0;
+   } else {
+      PMPI_Reduce(MPI_IN_PLACE, CountersBegin, CountersSize, MPI_UINT64_T,
+                 MPI_SUM, 0, MPI_COMM_WORLD);
+   }
+   return PMPI_Finalize();
+}
+
+#else
+/* Included here so regression test has a definition to use */
+COMPILER_RT_WEAK int MPI_Finalize() { return PMPI_Finalize(); }
+#endif
+

lib/profile/InstrProfilingStubs.c

+/*===-- InstrProfilingStubs.c - Stubs for applications not using MPI ------===*\
+|*
+|*                     The LLVM Compiler Infrastructure
+|*
+|* This file is distributed under the University of Illinois Open Source
+|* License. See LICENSE.TXT for details.
+|*
+\*===----------------------------------------------------------------------===*/
+
+
+#include <stdlib.h>
+#include "InstrProfiling.h"
+
+COMPILER_RT_WEAK int PMPI_Finalize() { abort(); }
+COMPILER_RT_WEAK int PMPI_Comm_rank(void *comm, int *rank) { abort(); }
+COMPILER_RT_WEAK int PMPI_Reduce() { abort(); }

test/profile/instrprof-reduce.c

+// RUN: %clang_profgen -o %t -O3 %s
+// RUN: rm -f %t.profraw
+// RUN: env LLVM_PROFILE_FILE=%t.profraw %run %t 0
+// RUN: test -e %t.profraw
+// RUN: rm -f %t.profraw
+// RUN: env LLVM_PROFILE_FILE=%t.profraw %run %t 1
+// RUN: test ! -e %t.profraw
+// REQUIRES: shell
+
+#include <stdlib.h>
+#include <stdio.h>
+int ThisRank;
+int PMPI_Comm_rank(void *Comm, int *Rank) { *Rank = ThisRank; return 0; }
+int PMPI_Reduce() { return 0; }
+int PMPI_Finalize() { return 0; }
+int MPI_Finalize();
+
+int main(int argc, const char *argv[]) {
+  ThisRank = atoi(argv[1]);
+  MPI_Finalize();
+  return 0;
+}
+