Implement GetMemoryRegions() for Linux and Mac OSX core files.

[hhellyer]

Summary

This patch fills in the implementation of GetMemoryRegions() on the Linux and Mac OS core file implementations of lldb_private::Process (ProcessElfCore::GetMemoryRegions and ProcessMachCore::GetMemoryRegions.) The GetMemoryRegions API was added under: http://reviews.llvm.org/D20565

The patch re-uses the m_core_range_infos list that was recently added to implement GetMemoryRegionInfo in both ProcessElfCore and ProcessMachCore to ensure the returned regions match the regions returned by Process::GetMemoryRegionInfo(addr_t load_addr, MemoryRegionInfo &region_info).

[hhellyer]

Squashing local commits.

[clayborg]

We shouldn't need to implement this on all process subclasses. Process::GetMemoryRegions(...) could just call the Process::GetMemoryRegionInfo() and fill the stuff in. We would need to modify lldb_private::MemoryRegionInfo to support knowing when no memory exists for a given address. Why? Because Mac programs actually have a __PAGEZERO section that no read/write/execute, but it is actually mapped into memory, but we currently can't tell the difference between something that isn't mapped, and a section that is mapped with no read/write/exe. If we fix this, then lldb_private::Process subclasses only need to override Process::GetMemoryRegionInfo(...) and we can do the rest up in Process::GetMemoryRegions().

[clayborg]

This of course would rely on the fact that each Process subclass the overrides GetMemoryRegionInfo() will follow the following rules:
1 - if no memory is mapped for a given address, set an extra "not_mapped" variable in MemoryRegionInfo and provide the next address that is mapped by setting the end address of the current MemoryRegionInfo correctly.
2 - set values correctly for mapped regions with the correct permissions
3 - For the last entry that is past all mapped entries, set the extra "not_mapped" and set the end address as LLDB_INVALID_ADDRESS

[hhellyer]

Since there’s a list of memory regions already created in the Process implementations for the core fles I thought it made sense just to use that directly, at least when reading from a core file. It meant there was no requirement to change those GetMemoryRegionInfo implementations if they didn’t behave quite the same. That said it is roughly what I planned to do for the live connections as the behaviour of the qMemoryRegionInfo query is more tightly specified and moving to that model would just push that implementation one layer higher.

I will push a revision which follows the approach you suggest, if we find that throws up it’s own problems then I can always revert. As I mentioned the remote connections should (hopefully) all work with that pattern so GetMemoryRegions implemented in fewer commits with that approach.

Is PAGEZERO actually included in Mac core dumps? I can see it as load command 0 when I run otool against an executable but when I look at a core dump from the same executable the first load command looks like load command 1 in the executable. There doesn’t seem to be a load command in the core for address 0. I might just be missing an flag on otool though.

[clayborg]

In D21751#468738, @hhellyer wrote:

Since there’s a list of memory regions already created in the Process implementations for the core fles I thought it made sense just to use that directly, at least when reading from a core file. It meant there was no requirement to change those GetMemoryRegionInfo implementations if they didn’t behave quite the same. That said it is roughly what I planned to do for the live connections as the behaviour of the qMemoryRegionInfo query is more tightly specified and moving to that model would just push that implementation one layer higher.

We should document the Process::GetMemoryRegionInfo() and what we expect of it so that all plug-ins implement it correctly.

I will push a revision which follows the approach you suggest, if we find that throws up it’s own problems then I can always revert. As I mentioned the remote connections should (hopefully) all work with that pattern so GetMemoryRegions implemented in fewer commits with that approach.

Indeed. Since we already expect a behavior from Process::GetMemoryRegionInfo(), I would say lets enforce these requirements by making Process::GetMemoryRegions() just use that API as required. The only time we should see an error from Process::GetMemoryRegionInfo() is when it isn't implemented.

Is PAGEZERO actually included in Mac core dumps? I can see it as load command 0 when I run otool against an executable but when I look at a core dump from the same executable the first load command looks like load command 1 in the executable. There doesn’t seem to be a load command in the core for address 0. I might just be missing an flag on otool though.

It might not be, but that brings up another thing I was thinking of is that we could start supplementing the Process::GetMemoryRegionInfo() info by also looking in any sections that were found in core file memory that might not be fully described by the core file itself. There is no requirements that if two sections are adjacent in a core file that they show up as different mapped regions, so you might find a mapped section of memory from 0x1000-0x8000 that is actually 0x1000-0x4000 for ".data" and 0x4000-0x8000 that is ".bss". The memory region was mapped read+write, but would it be nice to know about these different regions separately? We don't need to do this now, but this is one place where we could figure out that 0x0-0x100000000 is __PAGEZERO even though it isn't mapped into the current mach-o file's memory regions. Also there are different makers of core files: lldb, the kernel, possibly others. They don't always follow the same rules when making mach-o core files. So it is nice to be flexible.

[labath]

Please also add a test for this in TestLinuxCore.py.

There shouldn't be a need to add a new core file, you can just add an additional check to the do_test method that verifies the right memory regions (which you can pass in the same way as the pid). You can pass in bogus (empty) values for the i386 and s390x test cases, as they are not working now anyway.

[hhellyer]

In D21751#468976, @clayborg wrote:

We should document the Process::GetMemoryRegionInfo() and what we expect of it so that all plug-ins implement it correctly.

I'll add a full comment in Process.h and I'll make sure the description matches the behaviour of qMemoryRegionInfo.

I will push a revision which follows the approach you suggest, if we find that throws up it’s own problems then I can always revert. As I mentioned the remote connections should (hopefully) all work with that pattern so GetMemoryRegions implemented in fewer commits with that approach.

Indeed. Since we already expect a behavior from Process::GetMemoryRegionInfo(), I would say lets enforce these requirements by making Process::GetMemoryRegions() just use that API as required. The only time we should see an error from Process::GetMemoryRegionInfo() is when it isn't implemented.

Ok, I’ve looked at the implementation on a couple of Mac and Linux cores, at the moment it will return an error if the function is unimplemented or you request a load_addr past the last mapped memory range. After the change it will only return an error when unimplemented. As GetMemoryRegionInfo (on Mac and Linux cores) doesn't return an error if you ask for unmapped memory ranges in the middle of the address space this is probably more consistent as the MemoryRange will always be populated for any address that’s queried, there's nothing magic about addresses beyond the last mapped region it should behave the same for those. I’ll go through and check the behaviour elsewhere and update the patch. I will have to update the existing implementations to set the flag that says whether a memory region is mapped or not but that doesn't look complicated.

Is PAGEZERO actually included in Mac core dumps? I can see it as load command 0 when I run otool against an executable but when I look at a core dump from the same executable the first load command looks like load command 1 in the executable. There doesn’t seem to be a load command in the core for address 0. I might just be missing an flag on otool though.

It might not be, but that brings up another thing I was thinking of is that we could start supplementing the Process::GetMemoryRegionInfo() info by also looking in any sections that were found in core file memory that might not be fully described by the core file itself. There is no requirements that if two sections are adjacent in a core file that they show up as different mapped regions, so you might find a mapped section of memory from 0x1000-0x8000 that is actually 0x1000-0x4000 for ".data" and 0x4000-0x8000 that is ".bss". The memory region was mapped read+write, but would it be nice to know about these different regions separately? We don't need to do this now, but this is one place where we could figure out that 0x0-0x100000000 is __PAGEZERO even though it isn't mapped into the current mach-o file's memory regions. Also there are different makers of core files: lldb, the kernel, possibly others. They don't always follow the same rules when making mach-o core files. So it is nice to be flexible.

Another possibility, instead of splitting regions, is providing a way to iterate over all the sections a region contains. I'm not sure which is better. Do you want me to add a "backed" flag, possibly just for cores, that identifies whether there is any data for this region actually in the core dump. (For example a core might be truncated because of a ulimit or have omitted regions because of /proc/[pid]/coredump_filter settings on Linux.)

In D21751#469903, @labath wrote:

Please also add a test for this in TestLinuxCore.py.

There shouldn't be a need to add a new core file, you can just add an additional check to the do_test method that verifies the right memory regions (which you can pass in the same way as the pid). You can pass in bogus (empty) values for the i386 and s390x test cases, as they are not working now anyway.

Ok, I'll add that into this patch unless you prefer test updates as separate commits.

[hhellyer]

I’ve added a test in TestLinuxCore.py. It found a bug which the change in RangeMap.h resolves.

It fills in the implementation of Process::GetMemoryRegions() and makes minor changes to support the mapped attribute in MemoryRegionInfo to the Mac and Linux process implementations. I also updated the documentation for Process::GetMemoryRegionInfo.

Windows will require more changes to support the API correctly, I’m currently getting an environment setup where I can work on those.

[clayborg]

It would be nice to export the is_mapped through the SBMemoryRegionInfo in the public API for completeness.

[hhellyer]

Updated patch to export IsMapped in the C++ and Python APIs.
Updated the test case to verify all the regions returned in the list are mapped.

[clayborg]

Looks good.

include/lldb/Target/Process.h

GetMemoryRegionInfo (lldb::addr_t load_addr,
                          MemoryRegionInfo &range_info)
     {
         Error error;
         error.SetErrorString ("Process::GetMemoryRegionInfo() not supported");
         return error;
     }
 
     virtual Error
-    GetMemoryRegions (std::vector<lldb::MemoryRegionInfoSP>&)
+    GetMemoryRegions (std::vector<lldb::MemoryRegionInfoSP> &regions)
     {
         Error error;
         error.SetErrorString ("Process::GetMemoryRegions() not supported");
         return error;
     }
 
     virtual Error
     GetWatchpointSupportInfo (uint32_t &num)

source/Plugins/Process/elf-core/ProcessElfCore.h

public:
     //------------------------------------------------------------------
     size_t ReadMemory(lldb::addr_t addr, void *buf, size_t size, lldb_private::Error &error) override;
 
     size_t DoReadMemory(lldb::addr_t addr, void *buf, size_t size, lldb_private::Error &error) override;
 
     lldb_private::Error
     GetMemoryRegionInfo(lldb::addr_t load_addr, lldb_private::MemoryRegionInfo &region_info) override;
 
+    lldb_private::Error
+    GetMemoryRegions (std::vector<lldb::MemoryRegionInfoSP> &regions) override;
+
     lldb::addr_t GetImageInfoAddress() override;
 
     lldb_private::ArchSpec
     GetArchitecture();
 
     // Returns AUXV structure found in the core file
     const lldb::DataBufferSP
     GetAuxvData() override;

source/Plugins/Process/elf-core/ProcessElfCore.cpp

if (permission_entry)
             region_info.SetWritable(MemoryRegionInfo::eNo);
             region_info.SetExecutable(MemoryRegionInfo::eNo);
         }
         return Error();
     }
     return Error("invalid address");
 }
 
+Error
+ProcessElfCore::GetMemoryRegions (std::vector<lldb::MemoryRegionInfoSP> &regions) {
+    regions.clear();
+    // The list of VMRangeToPermissions is sorted when it is created.
+    for( size_t i = 0; i < m_core_range_infos.GetSize(); ++i ) {
+        const VMRangeToPermissions::Entry *permission_entry = m_core_range_infos.GetEntryAtIndex(i);
+        lldb::MemoryRegionInfoSP region_info_sp( new MemoryRegionInfo() );
+        region_info_sp->GetRange().SetRangeBase(permission_entry->GetRangeBase());
+        region_info_sp->GetRange().SetRangeEnd(permission_entry->GetRangeEnd());
+        const Flags permissions(permission_entry->data);
+        region_info_sp->SetReadable(permissions.Test(lldb::ePermissionsReadable) ? MemoryRegionInfo::eYes
+                : MemoryRegionInfo::eNo);
+        region_info_sp->SetWritable(permissions.Test(lldb::ePermissionsWritable) ? MemoryRegionInfo::eYes
+                : MemoryRegionInfo::eNo);
+        region_info_sp->SetExecutable(permissions.Test(lldb::ePermissionsExecutable) ? MemoryRegionInfo::eYes
+                : MemoryRegionInfo::eNo);
+        regions.push_back( region_info_sp );
+    }
+    return Error();
+}
+
 size_t
 ProcessElfCore::DoReadMemory (lldb::addr_t addr, void *buf, size_t size, Error &error)
 {
     ObjectFile *core_objfile = m_core_module_sp->GetObjectFile();
 
     if (core_objfile == NULL)
         return 0;
 

source/Plugins/Process/mach-core/ProcessMachCore.h

public:
     ReadMemory (lldb::addr_t addr, void *buf, size_t size, lldb_private::Error &error) override;
     
     size_t
     DoReadMemory (lldb::addr_t addr, void *buf, size_t size, lldb_private::Error &error) override;
 
     lldb_private::Error
     GetMemoryRegionInfo(lldb::addr_t load_addr, lldb_private::MemoryRegionInfo &region_info) override;
 
+    lldb_private::Error
+    GetMemoryRegions (std::vector<lldb::MemoryRegionInfoSP> &regions) override;
+
     lldb::addr_t
     GetImageInfoAddress () override;
 
 protected:
     friend class ThreadMachCore;
     
     void
     Clear ( );

source/Plugins/Process/mach-core/ProcessMachCore.cpp

if (permission_entry)
             region_info.SetExecutable(MemoryRegionInfo::eNo);
         }
         return Error();
     }
 
     return Error("invalid address");
 }
 
+Error
+ProcessMachCore::GetMemoryRegions (std::vector<lldb::MemoryRegionInfoSP> &regions) {
+    regions.clear();
+    // The list of VMRangeToPermissions is sorted when it is created.
+    for( size_t i = 0; i < m_core_range_infos.GetSize(); ++i ) {
+        const VMRangeToPermissions::Entry *permission_entry = m_core_range_infos.GetEntryAtIndex(i);
+        lldb::MemoryRegionInfoSP region_info_sp( new MemoryRegionInfo() );
+        region_info_sp->GetRange().SetRangeBase(permission_entry->GetRangeBase());
+        region_info_sp->GetRange().SetRangeEnd(permission_entry->GetRangeEnd());
+        const Flags permissions(permission_entry->data);
+        region_info_sp->SetReadable(permissions.Test(lldb::ePermissionsReadable) ? MemoryRegionInfo::eYes
+                : MemoryRegionInfo::eNo);
+        region_info_sp->SetWritable(permissions.Test(lldb::ePermissionsWritable) ? MemoryRegionInfo::eYes
+                : MemoryRegionInfo::eNo);
+        region_info_sp->SetExecutable(permissions.Test(lldb::ePermissionsExecutable) ? MemoryRegionInfo::eYes
+                : MemoryRegionInfo::eNo);
+        regions.push_back( region_info_sp );
+    }
+    return Error();
+}
+
 void
 ProcessMachCore::Clear()
 {
     m_thread_list.Clear();
 }
 
 void
 ProcessMachCore::Initialize()