[lldb] Add ConstString memory usage statistics

[JDevlieghere]

Summary

Add statistics about the memory usage of the string pool. I'm particularly interested in the memory used by the allocator, i.e. the number of bytes actually used by the allocator it self as well as the number of bytes allocated through the allocator.

[JDevlieghere]

lldb/include/lldb/Utility/ConstString.h

409

I purposely didn't use this function for the reasons explained in the summary. It seems like a few classes implement this, but it's not actually used anywhere. If we care, I can reimplement this in terms of the new MemoryStats. If we don't I can remove it in a follow-up patch.

[kastiglione]

nice

[clayborg]

lldb/include/lldb/Utility/ConstString.h

415–416

lldb/source/Target/Statistics.cpp

72

Maybe "bytesUnallocated" instead of "bytesWasted"?

231

"constStrings" maybe? or "stringPool"? "strings" makes it seem like we are tracking all strings in LLDB

Do we want another top level key here for "memory" that we can slowly add things to? It would be nice to know how much memory symbols tables consume, DWARF data structures like the DIE vectors, and many others things. So maybe adding a "memory" key/value pair at the top here might be a good idea?

"memory": {"strings": ..., "symtab": ..., "dwarf": .... }

[JDevlieghere]

lldb/source/Target/Statistics.cpp

231

I like that

[JDevlieghere]

Put string statistics under memory:

"memory": {
  "strings": {
    "bytesAllocated": 2056916,
    "bytesTotal": 3145728,
    "bytesUnallocated": 1088812
  }
},

[kastiglione]

I find "Unallocated" ambiguous. To the Allocator, maybe it's not, but to VM it is. What about Total/Used/Unused? Note that the docs for BytesAllocated say it can be used to calculated "wasted" space.

/// How many bytes we've allocated.
///
/// Used so that we can compute how much space was wasted.
size_t BytesAllocated = 0;

[JDevlieghere]

In D117914#3262467, @kastiglione wrote:

I find "Unallocated" ambiguous. To the Allocator, maybe it's not, but to VM it is. What about Total/Used/Unused? Note that the docs for BytesAllocated say it can be used to calculated "wasted" space.

/// How many bytes we've allocated.
///
/// Used so that we can compute how much space was wasted.
size_t BytesAllocated = 0;

I got the original terminology from the dumpStats function on the BumpPtrAllocator. I like Total/Used/Unused because there's less ambiguity.

[JDevlieghere]

"memory": {
  "strings": {
    "bytesTotal": 3145728,
    "bytesUnused": 1088812,
    "bytesUsed": 2056916
  }
},

[JDevlieghere]

@clayborg Does the latest scheme look good to you?

[clayborg]

Yep! Looks good.

As a follow up patch it would be great to ask the SymbolFile classes to break down some memory usage. I know in the SymbolFileDWARF we have each DWARFUnit that may or may not parse the unit DIE or all of the DIEs. These data structures take up memory and we lazily try to only parse all DIEs in a DWARFUnit when we need to. If we index the debug info, we will load the info and remembers which DWARFUnits had all of their DIEs parsed, and after indexing we will free the memory to keep memory usage down.

[JDevlieghere]

In D117914#3267657, @clayborg wrote:

Yep! Looks good.

As a follow up patch it would be great to ask the SymbolFile classes to break down some memory usage. I know in the SymbolFileDWARF we have each DWARFUnit that may or may not parse the unit DIE or all of the DIEs. These data structures take up memory and we lazily try to only parse all DIEs in a DWARFUnit when we need to. If we index the debug info, we will load the info and remembers which DWARFUnits had all of their DIEs parsed, and after indexing we will free the memory to keep memory usage down.

Are you referring to the m_die_array in DWARFUnit and ClearDIEsRWLocked? If so that actually looks like a good fit for the BumpPtrAllocator in which case it should be fairly easy to report.

lldb/include/lldb/Target/Statistics.h

 //===-- Statistics.h --------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLDB_TARGET_STATISTICS_H
 #define LLDB_TARGET_STATISTICS_H
 
+#include "lldb/Utility/ConstString.h"
 #include "lldb/Utility/Stream.h"
 #include "lldb/lldb-forward.h"
 #include "llvm/Support/JSON.h"
 #include <atomic>
 #include <chrono>
 #include <ratio>
 #include <string>
 #include <vector>
struct ModuleStats {
   double debug_index_time = 0.0;
   uint64_t debug_info_size = 0;
   bool symtab_loaded_from_cache = false;
   bool symtab_saved_to_cache = false;
   bool debug_info_index_loaded_from_cache = false;
   bool debug_info_index_saved_to_cache = false;
 };
 
+struct ConstStringStats {
+  llvm::json::Value ToJSON() const;
+  ConstString::MemoryStats stats = ConstString::GetMemoryStats();
+};
+
 /// A class that represents statistics for a since lldb_private::Target.
 class TargetStats {
 public:
   llvm::json::Value ToJSON(Target &target);
 
   void SetLaunchOrAttachTime();
   void SetFirstPrivateStopTime();
   void SetFirstPublicStopTime();

lldb/include/lldb/Utility/ConstString.h

public:
   /// Get the size in bytes of the current global string pool.
   ///
   /// Reports the size in bytes of all shared C string values, containers and
   /// any other values as a byte size for the entire string pool.
   ///
   /// \return
   ///     The number of bytes that the global string pool occupies
   ///     in memory.
   static size_t StaticMemorySize();
 
+  struct MemoryStats {
+    size_t GetBytesTotal() const { return bytes_total; }
+    size_t GetBytesUsed() const { return bytes_used; }
+    size_t GetBytesUnused() const { return bytes_total - bytes_used; }
+    size_t bytes_total = 0;
+    size_t bytes_used = 0;
+  };
+
+  static MemoryStats GetMemoryStats();
+
 protected:
   template <typename T, typename Enable> friend struct ::llvm::DenseMapInfo;
   /// Only used by DenseMapInfo.
   static ConstString FromStringPoolPointer(const char *ptr) {
     ConstString s;
     s.m_string = ptr;
     return s;
   };

lldb/source/Target/Statistics.cpp

json::Value ModuleStats::ToJSON() const {
   module.try_emplace("debugInfoByteSize", (int64_t)debug_info_size);
   module.try_emplace("debugInfoIndexLoadedFromCache",
                      debug_info_index_loaded_from_cache);
   module.try_emplace("debugInfoIndexSavedToCache",
                      debug_info_index_saved_to_cache);
   return module;
 }
 
+llvm::json::Value ConstStringStats::ToJSON() const {
+  json::Object obj;
+  obj.try_emplace<int64_t>("bytesTotal", stats.GetBytesTotal());
+  obj.try_emplace<int64_t>("bytesUsed", stats.GetBytesUsed());
+  obj.try_emplace<int64_t>("bytesUnused", stats.GetBytesUnused());
+  return obj;
+}
+
 json::Value TargetStats::ToJSON(Target &target) {
   CollectStats(target);
 
   json::Array json_module_uuid_array;
   for (auto module_identifier : m_module_identifiers)
     json_module_uuid_array.emplace_back(module_identifier);
 
   json::Object target_metrics_json{
for (size_t image_idx = 0; image_idx < num_modules; ++image_idx) {
     symtab_parse_time += module_stat.symtab_parse_time;
     symtab_index_time += module_stat.symtab_index_time;
     debug_parse_time += module_stat.debug_parse_time;
     debug_index_time += module_stat.debug_index_time;
     debug_info_size += module_stat.debug_info_size;
     json_modules.emplace_back(module_stat.ToJSON());
   }
 
+  ConstStringStats const_string_stats;
+  json::Object json_memory{
+      {"strings", const_string_stats.ToJSON()},
+  };
+
   json::Object global_stats{
       {"targets", std::move(json_targets)},
       {"modules", std::move(json_modules)},
+      {"memory", std::move(json_memory)},
       {"totalSymbolTableParseTime", symtab_parse_time},
       {"totalSymbolTableIndexTime", symtab_index_time},
       {"totalSymbolTablesLoadedFromCache", symtabs_loaded},
       {"totalSymbolTablesSavedToCache", symtabs_saved},
       {"totalDebugInfoParseTime", debug_parse_time},
       {"totalDebugInfoIndexTime", debug_index_time},
       {"totalDebugInfoIndexLoadedFromCache", debug_index_loaded},
       {"totalDebugInfoIndexSavedToCache", debug_index_saved},
       {"totalDebugInfoByteSize", debug_info_size},
   };
   return std::move(global_stats);
 }

lldb/source/Utility/ConstString.cpp

size_t MemorySize() const {
     for (const auto &pool : m_string_pools) {
       llvm::sys::SmartScopedReader<false> rlock(pool.m_mutex);
       for (const auto &entry : pool.m_string_map)
         mem_size += sizeof(StringPoolEntryType) + entry.getKey().size();
     }
     return mem_size;
   }
 
+  ConstString::MemoryStats GetMemoryStats() const {
+    ConstString::MemoryStats stats;
+    for (const auto &pool : m_string_pools) {
+      llvm::sys::SmartScopedReader<false> rlock(pool.m_mutex);
+      const Allocator &alloc = pool.m_string_map.getAllocator();
+      stats.bytes_total += alloc.getTotalMemory();
+      stats.bytes_used += alloc.getBytesAllocated();
+    }
+    return stats;
+  }
+
 protected:
   uint8_t hash(const llvm::StringRef &s) const {
     uint32_t h = llvm::djbHash(s);
     return ((h >> 24) ^ (h >> 16) ^ (h >> 8) ^ h) & 0xff;
   }
 
   struct PoolEntry {
     mutable llvm::sys::SmartRWMutex<false> m_mutex;
void ConstString::SetTrimmedCStringWithLength(const char *cstr,
   m_string = StringPool().GetConstTrimmedCStringWithLength(cstr, cstr_len);
 }
 
 size_t ConstString::StaticMemorySize() {
   // Get the size of the static string pool
   return StringPool().MemorySize();
 }
 
+ConstString::MemoryStats ConstString::GetMemoryStats() {
+  return StringPool().GetMemoryStats();
+}
+
 void llvm::format_provider<ConstString>::format(const ConstString &CS,
                                                 llvm::raw_ostream &OS,
                                                 llvm::StringRef Options) {
   format_provider<StringRef>::format(CS.GetStringRef(), OS, Options);
 }
 
 void llvm::yaml::ScalarTraits<ConstString>::output(const ConstString &Val,
                                                    void *, raw_ostream &Out) {

lldb/test/API/commands/statistics/basic/TestStats.py

def test_default_no_run(self):
         When we don't run the target, we expect to not see any 'firstStopTime'
         or 'launchOrAttachTime' top level keys that measure the launch or
         attach of the target.
 
         Output expected to be something like:
 
         (lldb) statistics dump
         {
+          "memory" : {...},
           "modules" : [...],
           "targets" : [
             {
                 "targetCreateTime": 0.26566899599999999,
                 "expressionEvaluation": {
                     "failures": 0,
                     "successes": 0
                 },
def test_default_no_run(self):
           "totalDebugInfoParseTime": 8.2121400240000071,
           "totalSymbolTableParseTime": 0.123,
           "totalSymbolTableIndexTime": 0.234,
         }
         """
         target = self.createTestTarget()
         debug_stats = self.get_stats()
         debug_stat_keys = [
+            'memory',
             'modules',
             'targets',
             'totalSymbolTableParseTime',
             'totalSymbolTableIndexTime',
             'totalSymbolTablesLoadedFromCache',
             'totalSymbolTablesSavedToCache',
             'totalDebugInfoByteSize',
             'totalDebugInfoIndexTime',
def test_default_with_run(self):
 
         When we run the target, we expect to see 'launchOrAttachTime' and
         'firstStopTime' top level keys.
 
         Output expected to be something like:
 
         (lldb) statistics dump
         {
+          "memory" : {...},
           "modules" : [...],
           "targets" : [
                 {
                     "firstStopTime": 0.34164492800000001,
                     "launchOrAttachTime": 0.31969605400000001,
                     "moduleIdentifiers": [...],
                     "targetCreateTime": 0.0040863039999999998
                     "expressionEvaluation": {
def test_default_with_run(self):
         }
 
         """
         target = self.createTestTarget()
         lldbutil.run_to_source_breakpoint(self, "// break here",
                                           lldb.SBFileSpec("main.c"))
         debug_stats = self.get_stats()
         debug_stat_keys = [
+            'memory',
             'modules',
             'targets',
             'totalSymbolTableParseTime',
             'totalSymbolTableIndexTime',
             'totalSymbolTablesLoadedFromCache',
             'totalSymbolTablesSavedToCache',
             'totalDebugInfoByteSize',
             'totalDebugInfoIndexTime',
def test_default_with_run(self):
             'moduleIdentifiers',
             'targetCreateTime',
         ]
         self.verify_keys(stats, '"stats"', keys_exist, None)
         self.assertGreater(stats['firstStopTime'], 0.0)
         self.assertGreater(stats['launchOrAttachTime'], 0.0)
         self.assertGreater(stats['targetCreateTime'], 0.0)
 
+    def test_memory(self):
+        """
+            Test "statistics dump" and the memory information.
+        """
+        exe = self.getBuildArtifact("a.out")
+        target = self.createTestTarget(file_path=exe)
+        debug_stats = self.get_stats()
+        debug_stat_keys = [
+            'memory',
+            'modules',
+            'targets',
+            'totalSymbolTableParseTime',
+            'totalSymbolTableIndexTime',
+            'totalSymbolTablesLoadedFromCache',
+            'totalSymbolTablesSavedToCache',
+            'totalDebugInfoParseTime',
+            'totalDebugInfoIndexTime',
+            'totalDebugInfoIndexLoadedFromCache',
+            'totalDebugInfoIndexSavedToCache',
+            'totalDebugInfoByteSize'
+        ]
+        self.verify_keys(debug_stats, '"debug_stats"', debug_stat_keys, None)
+
+        memory = debug_stats['memory']
+        memory_keys= [
+            'strings',
+        ]
+        self.verify_keys(memory, '"memory"', memory_keys, None)
+
+        strings = memory['strings']
+        strings_keys= [
+            'bytesTotal',
+            'bytesUsed',
+            'bytesUnused',
+        ]
+        self.verify_keys(strings, '"strings"', strings_keys, None)
+
+
     def find_module_in_metrics(self, path, stats):
         modules = stats['modules']
         for module in modules:
             if module['path'] == path:
                 return module
         return None
 
     def test_modules(self):
         """
             Test "statistics dump" and the module information.
         """
         exe = self.getBuildArtifact("a.out")
         target = self.createTestTarget(file_path=exe)
         debug_stats = self.get_stats()
         debug_stat_keys = [
+            'memory',
             'modules',
             'targets',
             'totalSymbolTableParseTime',
             'totalSymbolTableIndexTime',
             'totalSymbolTablesLoadedFromCache',
             'totalSymbolTablesSavedToCache',
             'totalDebugInfoParseTime',
             'totalDebugInfoIndexTime',
def test_modules(self):
         self.verify_keys(exe_module, 'module dict for "%s"' % (exe), module_keys)
 
     def test_breakpoints(self):
         """Test "statistics dump"
 
         Output expected to be something like:
 
         {
+          "memory" : {...},
           "modules" : [...],
           "targets" : [
                 {
                     "firstStopTime": 0.34164492800000001,
                     "launchOrAttachTime": 0.31969605400000001,
                     "moduleIdentifiers": [...],
                     "targetCreateTime": 0.0040863039999999998
                     "expressionEvaluation": {
def test_breakpoints(self):
         }
 
         """
         target = self.createTestTarget()
         self.runCmd("b main.cpp:7")
         self.runCmd("b a_function")
         debug_stats = self.get_stats()
         debug_stat_keys = [
+            'memory',
             'modules',
             'targets',
             'totalSymbolTableParseTime',
             'totalSymbolTableIndexTime',
             'totalSymbolTablesLoadedFromCache',
             'totalSymbolTablesSavedToCache',
             'totalDebugInfoParseTime',
             'totalDebugInfoIndexTime',