[esan|cfrag] Add struct info registration

[zhaoqin]

Summary

Adds StructInfo to CacheFragInfo to match the LLVM's EfficiencySanitizer
structs.

Uses StructHashMap to keep track of the struct info used by the app.

Adds registerStructInfo/unregisterStructInfo to add/remove struct infos
to/from StructHashMap.

updates test struct-simple.cpp with more C structs.

[aizatsky]

lib/esan/cache_frag.cpp

39

Not sure about this, but maybe add "print" member methods to structs?

41

why (uptr) casts?

[filcab]

LGTM, with the fixes.

lib/esan/cache_frag.cpp

34

No need to have Ty in the name.

36

Maybe put the u32 at the end?
It'll end up being the same wasted space, but it might be easier to extend and keep binary compatibility, if we want (unsure we'd want that, though).

39

+1

[zhaoqin]

lib/esan/cache_frag.cpp

34

It was from my habit of using struct name_t in C.
Removed.

36

Why u32 at the end would keep binary compatibility? Won't anything append after won't change this layout? The only benefit I can see is some potential space saving in the future.

Adding ToolType back to ToolInfo/CacheFragInfo would be the most likely change in the future, and it would break any compatibility, so I won't worry too much about this.

39

I am not sure if I want to add anything to the struct created from the ESan pass.
It may increase the chance that the struct layout defined in the runtime is different from the struct layout defined in the compiler.
I would rather delete this routine as it is mainly for debugging.

41

From other examples.

lib/sanitizer_common/sanitizer_common.h:INLINE int Verbosity() {
lib/sanitizer_common/sanitizer_common.h: if ((uptr)Verbosity() >= (level)) Report(VA_ARGS); \
lib/sanitizer_common/sanitizer_common.h: if ((uptr)Verbosity() >= (level)) Printf(VA_ARGS); \

[zhaoqin]

[esan|cfrag] Add struct info registration

[filcab]

LGTM with a pass of clang-format.

lib/esan/cache_frag.cpp

24

Why u32 at the end would keep binary compatibility?

Would be easier to add fields (up to 4 bytes added) after the u32 and still fit in the same size. (Only valid because we know exactly how ABIs work on the platforms we support).
You can still add stuff using padding bytes with the u32 being where it is, so no big deal there.

Since this sanitizer is much more complicated than something like UBSan (where we pulled off changing the structures emitted once (twice after a revision in process gets accepted)), I don't think we should worry about this too much. OK as it is.

113

Please run clang-format on the patch.

[zhaoqin]

Update according to D20661

[zhaoqin]

Final

lib/esan/cache_frag.cpp

 //===-- cache_frag.cpp ----------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is a part of EfficiencySanitizer, a family of performance tuners.
 //
 // This file contains cache fragmentation-specific code.
 //===----------------------------------------------------------------------===//
 
 #include "esan.h"
+#include "sanitizer_common/sanitizer_addrhashmap.h"
+#include "sanitizer_common/sanitizer_placement_new.h"
 
 namespace __esan {
 
+//===-- Struct field access counter runtime -------------------------------===//
+
 // This should be kept consistent with LLVM's EfficiencySanitizer StructInfo.
 struct StructInfo {
   const char *StructName;
   u32 NumOfFields;
   u64 *FieldCounters;
   const char **FieldTypeNames;
 };
 
 // This should be kept consistent with LLVM's EfficiencySanitizer CacheFragInfo.
 // The tool-specific information per compilation unit (module).
 struct CacheFragInfo {
   const char *UnitName;
   u32 NumOfStructs;
   StructInfo *Structs;
 };
 
+struct StructCounter {
+  StructInfo *Struct;
+  u64 Count;    // The total access count of the struct.
+  u32 Variance; // Variance score for the struct layout access.
+};
+
+// We use StructHashMap to keep track of an unique copy of StructCounter.
+typedef AddrHashMap<StructCounter, 31051> StructHashMap;
+struct Context {
+  StructHashMap StructMap;
+  u32 NumOfStructs;
+  u64 TotalCount; // The total access count of all structs.
+};
+static Context *Ctx;
+
+static void registerStructInfo(CacheFragInfo *CacheFrag) {
+  for (u32 i = 0; i < CacheFrag->NumOfStructs; ++i) {
+    StructInfo *Struct = &CacheFrag->Structs[i];
+    StructHashMap::Handle H(&Ctx->StructMap, (uptr)Struct->FieldCounters);
+    if (H.created()) {
+      VPrintf(2, " Register %s: %u fields\n",
+              Struct->StructName, Struct->NumOfFields);
+      H->Struct = Struct;
+      ++Ctx->NumOfStructs;
+    } else {
+      VPrintf(2, " Duplicated %s: %u fields\n",
+              Struct->StructName, Struct->NumOfFields);
+    }
+  }
+}
+
+static void unregisterStructInfo(CacheFragInfo *CacheFrag) {
+  // FIXME: if the library is unloaded before finalizeCacheFrag, we should
+  // collect the result for later report.
+  for (u32 i = 0; i < CacheFrag->NumOfStructs; ++i) {
+    StructInfo *Struct = &CacheFrag->Structs[i];
+    StructHashMap::Handle H(&Ctx->StructMap, (uptr)Struct->FieldCounters, true);
+    if (H.exists()) {
+      VPrintf(2, " Unregister %s: %u fields\n",
+              Struct->StructName, Struct->NumOfFields);
+      --Ctx->NumOfStructs;
+    } else {
+      VPrintf(2, " Duplicated %s: %u fields\n",
+              Struct->StructName, Struct->NumOfFields);
+    }
+  }
+}
+
+static void reportStructSummary() {
+  // FIXME: iterate StructHashMap and generate the final report.
+  Report("%s is not finished: nothing yet to report\n", SanitizerToolName);
+}
+
 //===-- Init/exit functions -----------------------------------------------===//
 
 void processCacheFragCompilationUnitInit(void *Ptr) {
   CacheFragInfo *CacheFrag = (CacheFragInfo *)Ptr;
   VPrintf(2, "in esan::%s: %s with %u class(es)/struct(s)\n",
           __FUNCTION__, CacheFrag->UnitName, CacheFrag->NumOfStructs);
+  registerStructInfo(CacheFrag);
 }
 
 void processCacheFragCompilationUnitExit(void *Ptr) {
   CacheFragInfo *CacheFrag = (CacheFragInfo *)Ptr;
   VPrintf(2, "in esan::%s: %s with %u class(es)/struct(s)\n",
           __FUNCTION__, CacheFrag->UnitName, CacheFrag->NumOfStructs);
+  unregisterStructInfo(CacheFrag);
 }
 
 void initializeCacheFrag() {
   VPrintf(2, "in esan::%s\n", __FUNCTION__);
+  // We use placement new to initialize Ctx before C++ static initializaion.
+  // We make CtxMem 8-byte aligned for atomic operations in AddrHashMap.
+  static u64 CtxMem[sizeof(Context) / sizeof(u64) + 1];
+  Ctx = new(CtxMem) Context();
+  Ctx->NumOfStructs = 0;
 }
 
 int finalizeCacheFrag() {
   VPrintf(2, "in esan::%s\n", __FUNCTION__);
-  // FIXME: add the cache fragmentation final report.
-  Report("%s is not finished: nothing yet to report\n", SanitizerToolName);
+  reportStructSummary();
   return 0;
 }
 
 } // namespace __esan

test/esan/TestCases/struct-simple.cpp

-// RUN: %clang_esan_frag -O0 %s -DPART -c -o %t-part.o 2>&1
+// RUN: %clang_esan_frag -O0 %s -DPART1 -c -o %t-part1.o 2>&1
+// RUN: %clang_esan_frag -O0 %s -DPART2 -c -o %t-part2.o 2>&1
 // RUN: %clang_esan_frag -O0 %s -DMAIN -c -o %t-main.o 2>&1
-// RUN: %clang_esan_frag -O0 %t-part.o %t-main.o -o %t 2>&1
+// RUN: %clang_esan_frag -O0 %t-part1.o %t-part2.o %t-main.o -o %t 2>&1
 // RUN: %env_esan_opts=verbosity=2 %run %t 2>&1 | FileCheck %s
 
 // We generate two different object files from this file with different
 // macros, and then link them together. We do this to test how we handle
 // separate compilation with multiple compilation units.
 
 #include <stdio.h>
 
 extern "C" {
-  void part();
+  void part1();
+  void part2();
 }
 
-//===-- compilation unit without main function ----------------------------===//
+//===-- compilation unit part1 without main function ----------------------===//
 
-#ifdef PART
-void part()
+#ifdef PART1
+struct A {
+  int x;
+  int y;
+};
+
+struct B {
+  float m;
+  double n;
+};
+
+union U {
+  float f;
+  double d;
+};
+
+// Same struct in both main and part1.
+struct S {
+  int s1;
+  int s2;
+};
+
+// Different structs with the same name in main and part1.
+struct D {
+  int d1;
+  int d2;
+};
+
+void part1()
 {
+  struct A a;
+  struct B b;
+  union  U u;
+  struct S s;
+  struct D d;
+  for (int i = 0; i < (1 << 11); i++)
+    a.x = 0;
+  a.y = 1;
+  b.m = 2.0;
+  for (int i = 0; i < (1 << 21); i++)
+    b.n = 3.0;
+  u.f = 0.0;
+  u.d = 1.0;
+  s.s1 = 0;
+  d.d1 = 0;
 }
-#endif // PART
+#endif // PART1
+
+//===-- compilation unit part2 without main function ----------------------===//
+#ifdef PART2
+// No struct in this part.
+void part2()
+{
+  // do nothing
+}
+#endif // PART2
 
 //===-- compilation unit with main function -------------------------------===//
 
 #ifdef MAIN
+class C {
+public:
+  struct {
+    int x;
+    int y;
+  } cs;
+  union {
+    float f;
+    double d;
+  } cu;
+  char c[10];
+};
+
+// Same struct in both main and part1.
+struct S {
+  int s1;
+  int s2;
+};
+
+// Different structs with the same name in main and part1.
+struct D {
+  int d1;
+  int d2;
+  int d3;
+};
+
 int main(int argc, char **argv) {
   // CHECK:      in esan::initializeLibrary
   // CHECK:      in esan::initializeCacheFrag
   // CHECK-NEXT: in esan::processCompilationUnitInit
-  // CHECK-NEXT: in esan::processCacheFragCompilationUnitInit: {{.*}}struct-simple.cpp with 0 class(es)/struct(s)
+  // CHECK-NEXT: in esan::processCacheFragCompilationUnitInit: {{.*}}struct-simple.cpp with 5 class(es)/struct(s)
+  // CHECK-NEXT:  Register struct.A#2#11#11: 2 fields
+  // CHECK-NEXT:  Register struct.B#2#3#2:   2 fields
+  // CHECK-NEXT:  Register union.U#1#3:      1 fields
+  // CHECK-NEXT:  Register struct.S#2#11#11: 2 fields
+  // CHECK-NEXT:  Register struct.D#2#11#11: 2 fields
   // CHECK-NEXT: in esan::processCompilationUnitInit
   // CHECK-NEXT: in esan::processCacheFragCompilationUnitInit: {{.*}}struct-simple.cpp with 0 class(es)/struct(s)
-  part();
+  // CHECK-NEXT: in esan::processCompilationUnitInit
+  // CHECK-NEXT: in esan::processCacheFragCompilationUnitInit: {{.*}}struct-simple.cpp with 5 class(es)/struct(s)
+  // CHECK-NEXT:  Register class.C#3#14#13#13:  3 fields
+  // CHECK-NEXT:  Register struct.anon#2#11#11: 2 fields
+  // CHECK-NEXT:  Register union.anon#1#3:      1 fields
+  // CHECK-NEXT:  Duplicated struct.S#2#11#11:  2 fields
+  // CHECK-NEXT:  Register struct.D#3#11#11#11: 3 fields
+  struct C c[2];
+  struct S s;
+  struct D d;
+  c[0].cs.x = 0;
+  c[1].cs.y = 1;
+  c[0].cu.f = 0.0;
+  c[1].cu.d = 1.0;
+  c[0].c[2] = 0;
+  s.s1 = 0;
+  d.d1 = 0;
+  d.d2 = 0;
+  part1();
+  part2();
   return 0;
   // CHECK:      in esan::finalizeLibrary
   // CHECK-NEXT: in esan::finalizeCacheFrag
   // CHECK-NEXT: {{.*}}EfficiencySanitizer is not finished: nothing yet to report
   // CHECK-NEXT: in esan::processCompilationUnitExit
-  // CHECK-NEXT: in esan::processCacheFragCompilationUnitExit: {{.*}}struct-simple.cpp with 0 class(es)/struct(s)
+  // CHECK-NEXT: in esan::processCacheFragCompilationUnitExit: {{.*}}struct-simple.cpp with 5 class(es)/struct(s)
+  // CHECK-NEXT:  Unregister class.C#3#14#13#13:  3 fields
+  // CHECK-NEXT:  Unregister struct.anon#2#11#11: 2 fields
+  // CHECK-NEXT:  Unregister union.anon#1#3:      1 fields
+  // CHECK-NEXT:  Unregister struct.S#2#11#11:    2 fields
+  // CHECK-NEXT:  Unregister struct.D#3#11#11#11: 3 fields
   // CHECK-NEXT: in esan::processCompilationUnitExit
   // CHECK-NEXT: in esan::processCacheFragCompilationUnitExit: {{.*}}struct-simple.cpp with 0 class(es)/struct(s)
+  // CHECK-NEXT: in esan::processCompilationUnitExit
+  // CHECK-NEXT: in esan::processCacheFragCompilationUnitExit: {{.*}}struct-simple.cpp with 5 class(es)/struct(s)
+  // CHECK-NEXT:  Unregister struct.A#2#11#11:    2 fields
+  // CHECK-NEXT:  Unregister struct.B#2#3#2:      2 fields
+  // CHECK-NEXT:  Unregister union.U#1#3:         1 fields
+  // CHECK-NEXT:  Duplicated struct.S#2#11#11:    2 fields
+  // CHECK-NEXT:  Unregister struct.D#2#11#11:    2 fields
 }
 #endif // MAIN