lldb RFC: Exposing set/get address masks, Fix*Address methods in SBProcess

[jasonmolenda]

Summary

We've had a few people doing SB API scripting who have asked for access to the Process settings of address masks, and API to fix address+metadata down to just addressable memory.

We're mostly focused on AArch64 these days at Apple so the number of addressing bits is what everyone is really asking for, but given that Linux's native representation is an address mask which can express things number-of-bits cannot, the SB API needs to represent that way, I think.

I'm also including the "Highmem" variants here, which makes for a not great looking SB API additions, but I know someone is going to need it. I'm open to suggestions or rejections of including this. There is an environment I need to support where they have items in both high and low memory (in the same execution level) with different page table settings for both so they have different numbers of addressing bits for low and high memory. This is quite uncommon I think -- if you have code running in high and low memory simultaneously, it's usually a kernel in high memory at EL1 and userland in low memory at EL0 and on Darwin they run with the same page table setups.

For the internal Process API, I didn't mind having these separate Highmem accessors which only differed from the base ones when we're in this situation of low & high memory on AArch64 having different page table setups. The only user of this is the Apple AArch64 ABI plugin when fixing addresses, so it wasn't something most people needed to consider. But putting it in SBProcess, it locks the API down more.

Adding Fix*Address is obvious.

No tests written yet, I didn't want to get ahead of myself. Given that I can get / set the address masks with these API, it will be easy enough to mutate it, run some uint64_t's through the Fix*Address API and confirm they're modified.

[DavidSpickett]

The only part of this that's maybe tricky is the HighMem parts of it. Given the niche use case it should be ok if they turn out a bit awkward for general use.

You are missing a way to fix a high mem address, is that intentional? Or would you get the high mem mask, change the setting, fix the address, as needed by context.

The rest are passing on existing calls we know are useful within lldb.

lldb/include/lldb/API/SBProcess.h

401

may -> should?

Though for API uses that only run on simple systems, calling this would just be unnecessary overhead.

402–403

"should be set in the mask". Or in reverse "should be cleared by the mask". Just be clear whether it's the mask or the address.

416

Should we note somewhere, maybe below where the any method is, the logic behind the code/data split?

Such as it is. Overall it's that data (load store) and code (fetch) addresses may have a different arrangement of addressing bits, so choose the one that makes sense in context. If we don't know, use any. It'd be an obvious question from anyone integrating this into an IDE.

453

Here is a good place for a note about which to pick.

[jasonmolenda]

Thanks for the feedback David.

You are missing a way to fix a high mem address, is that intentional? Or would you get the high mem mask, change the setting, fix the address, as needed by context.

The FixAddress method will choose whether to use the high mem mask or the low mem mask based on the address.

lldb/include/lldb/API/SBProcess.h

401

Probably the right call. The masking is implemented in the ABI so there are systems that don't have this concept and have no mask function, which is what I was thinking of with "maybe".

It makes me wonder if doing it in the ABI is correct; if we have an address mask for the system, is it a part of the ABI? On an ARMv8.3 ptrauth using process, the ABI defines what things are signed and need to be cleared before dereferencing (where we need to call FixAddress from within lldb), but we solve this by calling FixAddress at all sites that any ABI might use; ultimately they all need to resolve to an addressable address before reading, so if one ABI signs the C++ vtable pointer and one does not, having both call FixAddress on that is fine.

Of course I'm coming at this with an AArch64 perspective. On AArch64 we need to look at b55 to decide if the bits that are masked off are set to 0 or 1 (bits 56-63 possibly being TBI non-address bits), which is not a generic behavior that would apply on other processors that might need to apply an address mask. So maybe it's an architecture specific method instead of an ABI method, I should say.

402–403

Ah very good point.

416

Yeah I mean the reality is that on all our systems where code and data addresses are the same, FixCodeAddress may take advantage of the alignment of instructions to add a couple of bits, whereas FixDataAddress has to be able to address at a byte boundary so we really have "FixCodeAddress" and "FixAnyAddress", and it's only the Linux address mask namings that got us "FixDataAddress".

Of course there exist systems where instruction addresses and data addresses may not be as interchangeable, and I could even imagine some Harvard architecture system where the number of bits used for addressing of instructions is different than data, and applying the Data address mask to an instruction could corrupt it.

But I should explain the ideal use of these methods.

[clayborg]

lldb/include/lldb/API/SBProcess.h

415–416

Will there only ever be two variants of this for code and data? If we think there might be another address mask type later it might be a good idea to use an enumeration for these two functions and all functions below? I was thinking of something like:

enum AddressMaskType {
  eTypeCode = 0,
  eTypeData
};
lldb::addr_t GetAddressMask(AddressMaskType mask_type);

This enum could be used in all subsequent calls. I am just thinking of any needed extra API changes needed in the future. If we need another address mask type in the future we just add another enum and all functions stay the same if we do it as suggested above. Else we will need to add many more API functions if we need a new type later.

430–434

Is this comment needed for the set accessors? These functions don't return anything

451–453

Should the parameter be named "load_addr" to ensure people know it is a load address?

[jasonmolenda]

lldb/include/lldb/API/SBProcess.h

415–416

That's a very good idea. The HighCode/HighData can be rolled into the same list of enums.

430–434

I wanted to include this concrete example on both the Get and Set methods because the Linux address masks are the opposite of what I would have expected if someone told me it was an address mask.

[jasonmolenda]

Also interesting to consider if there should be an "Any" define. e.g.

enum AddressMaskType {
  eTypeCode = 0,
  eTypeData,
  eTypeHighmemCode,
  eTypeHighmemData,
  eTypeAny
};
lldb::addr_t GetAddressMask(AddressMaskType mask_type);
void SetAddressMask(AddressMaskType mask_type, lldb::addr_t mask);

The patch also adds SBProcess::FixCodeAddress, SBProcess::FixDataAddress, and SBProcess::FixAddress -- FixAddress is always calling FixDataAddress right now, because Data can be at any address on all the targets we support today, whereas CodeAddress may have low bits masked off (e.g. armv7 where the low bit is sometimes used to indicate arm/thub, but it could be for AArch64 as well). So GetAddressMask(eTypeAny) would probably return the data mask, and SetAddressMask(eTypeAny, mask) would set all the address masks to the same value.

[clayborg]

In D155905#4536917, @jasonmolenda wrote:

Also interesting to consider if there should be an "Any" define. e.g.

enum AddressMaskType {
  eTypeCode = 0,
  eTypeData,
  eTypeHighmemCode,
  eTypeHighmemData,
  eTypeAny
};
lldb::addr_t GetAddressMask(AddressMaskType mask_type);
void SetAddressMask(AddressMaskType mask_type, lldb::addr_t mask);

The patch also adds SBProcess::FixCodeAddress, SBProcess::FixDataAddress, and SBProcess::FixAddress -- FixAddress is always calling FixDataAddress right now, because Data can be at any address on all the targets we support today, whereas CodeAddress may have low bits masked off (e.g. armv7 where the low bit is sometimes used to indicate arm/thub, but it could be for AArch64 as well). So GetAddressMask(eTypeAny) would probably return the data mask, and SetAddressMask(eTypeAny, mask) would set all the address masks to the same value.

I like it the above approach with more enums for the high and low code/data. Not sure if eTypeAny makes sense in the GetAddressMask(eTypeAny) scenario, but I can see the use for a eTypeAll in case you wanted to set all of the various address masks to zero though using SetAddressMask(eTypeAll, mask). We would need to document this in the enum header file if we do add a eTypeAny or eTypeAll.

[jasonmolenda]

In D155905#4537036, @clayborg wrote:

I like it the above approach with more enums for the high and low code/data. Not sure if eTypeAny makes sense in the GetAddressMask(eTypeAny) scenario, but I can see the use for a eTypeAll in case you wanted to set all of the various address masks to zero though using SetAddressMask(eTypeAll, mask). We would need to document this in the enum header file if we do add a eTypeAny or eTypeAll.

Yeah I agree SetAddressMask(eTypeAll) and GetAddressMask(eTypeAny) would be the clearest names, that was my first thought too. Maybe adding two enum names for the same value. And I'm a little worried about encouraging script writers to assume there is one mask active -- all of the targets I work on today have the same mask for code and data, but I could imagine some harvard architecture target that behaved differently (surely this is why Linux has two address masks), and scripts wouldn't work correctly then. But let's be honest, even if it's not easy they're probably going to pick one mask anyway.

[DavidSpickett]

I like it the above approach with more enums for the high and low code/data. Not sure if eTypeAny makes sense in the GetAddressMask(eTypeAny) scenario

It could be like the FixAny method, a fallback when you don't know either way. eTypeAny returns FixAnyAddress(~0) essentially.

Or it could or together all the masks but while that's good for curiosity's sake (what address bits are "safe"?) you would still need specific masks to construct an address for specific purposes.

It could fail, but then we're carrying around a Status object for one case, and that seems a shame. Unless we can think of other failure modes we haven't considered yet.

[DavidSpickett]

but I could imagine some harvard architecture target that behaved differently (surely this is why Linux has two address masks)

I'm not privy to the exact reasoning, but at least part of it comes from the architecture itself. You could have a target that enables top byte ignore and pointer authentication for data addresses, but only enables pointer authentication for code addresses. So ptrace will show you different values for the pointer authentication masks in that case. I'm not sure you can actually configure a kernel that way today, but it's viable.

For the debugger, the result is the same. When top byte ignore is off, pointer authentication just uses that free space for itself. We end up removing the same set of bits either way.

For very specific tools you might want to only remove pointer authentication bits. Making this up, but maybe you want to take pointers from a pointer authenticated ABI application and pass them to a shared library without those protections. Niche, but ptrace leaves the door open for that rather than breaking userspace later by adding it.

Finally, you are right that one day there may be some scheme that truly does need different handling for code and data and again, let's keep the door open for that up here in lldb (though such an architecture would be major surgery in lldb anyway).

[DavidSpickett]

And +1 to the enum. This mitigates the MacOS specific-ness of the high and low masks at this time, and makes expansion easier if we do manage to generalise them later.

[jasonmolenda]

In D155905#4537892, @DavidSpickett wrote:

but I could imagine some harvard architecture target that behaved differently (surely this is why Linux has two address masks)

I'm not privy to the exact reasoning, but at least part of it comes from the architecture itself. You could have a target that enables top byte ignore and pointer authentication for data addresses, but only enables pointer authentication for code addresses. So ptrace will show you different values for the pointer authentication masks in that case. I'm not sure you can actually configure a kernel that way today, but it's viable.

For the debugger, the result is the same. When top byte ignore is off, pointer authentication just uses that free space for itself. We end up removing the same set of bits either way.

For very specific tools you might want to only remove pointer authentication bits. Making this up, but maybe you want to take pointers from a pointer authenticated ABI application and pass them to a shared library without those protections. Niche, but ptrace leaves the door open for that rather than breaking userspace later by adding it.

On macOS, fwiw, we have one set of the system libraries which are built to use ptrauth ABI ("arm64e"), but non-system processes are all running a non-ptrauth ABI ("arm64"), with most of the signing keys zeroed out so a "signed" function pointer is the same as the unsigned function pointer, making it possible to call between them. iirc there's one key that is still enabled so the ptrauth-using code can sign its link register before spilling to memory, because that doesn't have ABI impact.

Like you've suggested above, the FixAddress methods are used before lldb accesses memory, the goal is to clear/set all the non-addressable bits. Whether they were TBI metadata or ptrauth signing doesn't make any difference to lldb, it needs to find the actual VA that this uint64_t is referring to. But maybe retaining that distinction will be useful for something some day.

In the AArch64 macOS ABI I have it clear/set the top byte always, even if the address mask is "all bits are used for addressing", so if the processor was not running in TBI mode and the user has a pointer variable with a smashed top byte, lldb would dereference it fine but the program would crash when it does the same. I didn't think this was a problem worth trying to handle correctly. (and we run our main application processors in TBI mode)

[jasonmolenda]

Update the patch to express the mask getter/setters in terms of a type enum, after discussions with Greg and David. I still need to write a test for this API but I'm liking where this is now.

[clayborg]

Looks much better, just a few questions in comments.

lldb/include/lldb/API/SBProcess.h

425

s/eMaskTypeall/eMaskTypeAll

(missing camel case on "All")

445

What does this function do? Does it try to auto detect code/data low/hi on its own assuming that specific regions are easily identifiable?

lldb/source/API/SBProcess.cpp

1260–1261

Any reason we actually have a case for eMaskTypeAny? What makes this useful?

[jasonmolenda]

lldb/include/lldb/API/SBProcess.h

445

We have the same method in Process. Linux has separate masks for code & data addresses, so we have FixCode and FixData methods, but sometimes you just have an addr_t and you don't know the context (e.g. a register value that you're testing to see if it points to a symbol so you can print that as additional info to the user), and so in that case we have a Process::FixAddress.

There is one armv7 abi that clears the 0th bit on FixCode address, and I could see other ABIs doing similar things for the fixed nature of code addresses. But Data addresses are the most general, having to point to any addressable byte.

Our Process/SBProcess say "FixCode if you know it's code, FixData if you know it's data. And if you don't know, FixData". But I think having a "FixAddress" method, making it clear that you don't know the origin of the value, makes the intention a little clearer. Even if it's just calling FixData inside Process.

lldb/source/API/SBProcess.cpp

1260–1261

We have 4 different masks because of Linux's design, and because I have an internal customer doing different page table layouts for high & low memory in the same process (sigh), but realistically 99% of our programs are going to have the same mask for all 4 types. Making an SB API user choose which mask they want of the four makes it harder to use, imo. The folks doing genuinely unusual things, like the internal customer I have using high & low memory with different page table setups, can jump through the hoops of getting the different masks.

[jasonmolenda]

lldb/source/API/SBProcess.cpp

1260–1261

Just to be clear: I expect any SB API user who is using these is always going to call SBProcess::SetAddressMask(lldb.eMaskTypeAll, mask) and SBProcess::GetAddressMask(lldb.eMaskTypeAny). Maybe some day we'll support a target where code & data masks are necessarily different and this simplification will be a problem for SB API users, but today I think they are fine with doing simply this.

[jasonmolenda]

Given the AddressMaskType enum, I wonder if instead of FixCodeAddress, FixDataAddress, and FixAddress, there should be one SBProcess::FixAddress(lldb::AddressMaskType type, lldb::addr_t address) and it would be used with eMaskTypeCode, eMaskTypeData, or eMaskTypeAny.

[clayborg]

Thanks for the explanation. LGTM

[DavidSpickett]

lldb/include/lldb/API/SBProcess.h

401

I don't think it being in the ABI plugin is particularly deliberate, even if it is correct.

The ABI states something like the top byte is reserved for the OS and language runtimes, lldb is dealing with a consequence of that choice. So you could argue the ABI plugin should say "yes this should be fixed" and the fixing should happen elsewhere.

But right now those two things are so close together I don't think it matters.

433

"Clear bits from an addr value that are not used for addressing" is clearer to me.

Addressable makes me think I can craft an address to point to that bit, and I've been saying "non-address" but that term is also inside baseball, so something more wordy I think is better.

445

FixAnyAddress is also a marker for the, hopefully far from now, time, that lldb has to track address provenance. That's not an API concern though.

Should these functions also be one function that takes the enum value?

It's FixAddress(addr, lldb.eMaskTypeData instead of FixDataAddress(addr), but it keeps the theme consistent between all the functions.

[jasonmolenda]

Update patch to remove the HasValue/IsValue and Clear methods from AddressableBits. Change methods that may fill in an AddressableBits object with values to return an AddressableBits object unconditionally. Change AddressableBits::SetProcessMasks so it can be called unconditionally, and it only sets mask values in Process if values were set.

I wanted to allow for someone to create an AddressableBits object with only the high memory addressing bits specified (for instance) and the low memory address mask will not be changed. I updated AddressableBits::SetProcessMasks to use the current Process mask values if it doesn't have a value specified. This will be more important when I add the SB API to SBProcess to set address masks in https://reviews.llvm.org/D155905 I expect.

[jasonmolenda]

ah dangit, I meant to update the patch in https://reviews.llvm.org/D158041 and accidentally blew away the last patch I had for this one. let's see if I can extract it from here somehow.

[jasonmolenda]

Updated back to my most recent version of this patch, thanks for the help all.

[clayborg]

Looks good

lldb/include/lldb/API/SBProcess.h

   /// valid.
   lldb::SBProcessInfo GetProcessInfo();
 
+  /// Get the current address mask that may be applied to addresses
+  /// before reading from memory.
+  ///
+  /// \param[in] type
+  ///     lldb may have different address masks for code and data
+  ///     addresses.  And it may have different masks for code
+  ///     and data in high memory, which differ from the low memory
+  ///     code/data masks.  Each of these can be requested, or
+  ///     most commonly, eMaskTypeAny can be requested, with the
+  ///     assumption that the masks are all identical.
+  ///
+  /// \return
+  ///     The address mask currently in use.  Bits which are not used
+  ///     for addressing will be set to 1 in the mask.
+  lldb::addr_t GetAddressMask(lldb::AddressMaskType type);
+
+  /// Set the current address mask that may be applied to addresses
+  /// before reading from memory.
+  ///
+  /// \param[in] type
+  ///     lldb can have different address masks for code and data
+  ///     addresses.  And it can have different masks for code
+  ///     and data in high memory, which differ from the low memory
+  ///     code/data masks.  Each of these can be set, or
+  ///     most commonly, eMaskTypeall can be set, when all masks are
+  ///     identical.
+  ///
+  /// \param[in] mask
+  ///     The address mask to set.  Bits which are not used for addressing
+  ///     should be set to 1 in the mask.
+  void SetAddressMask(lldb::AddressMaskType type, lldb::addr_t mask);
+
+  /// Clear the non-addressable bits of an \a addr value and return a
+  /// virtual address in memory.
+  ///
+  /// Bits that are not used in addressing may be used for other purposes;
+  /// pointer authentication, or metadata in the top byte, or the 0th bit
+  /// of armv7 code addresses to indicate arm/thumb are common examples.
+  ///
+  /// When the target has provided an address mask of which bits are
+  /// valid for addressing, lldb can clear (or set, for high-memory addresses)
+  /// these bits.
+  lldb::addr_t FixCodeAddress(lldb::addr_t addr);
+  lldb::addr_t FixDataAddress(lldb::addr_t addr);
+  lldb::addr_t FixAnyAddress(lldb::addr_t addr);
+
   /// Allocate memory within the process.
   ///
   /// This function will allocate memory in the process's address space.

lldb/include/lldb/lldb-enumerations.h

   eCustomCompletion = (1u << 25)
 };
 
+enum AddressMaskType {
+  eMaskTypeCode = 0,
+  eMaskTypeData,
+  eMaskTypeHighmemCode,
+  eMaskTypeHighmemData,
+  eMaskTypeAny,
+  eMaskTypeAll = eMaskTypeAny
+};
+
 } // namespace lldb
 
 #endif // LLDB_LLDB_ENUMERATIONS_H

lldb/source/API/SBProcess.cpp

   return sb_proc_info;
 }
 
+addr_t SBProcess::GetAddressMask(lldb::AddressMaskType type) {
+  LLDB_INSTRUMENT_VA(this, type);
+  addr_t default_mask = 0;
+  if (ProcessSP process_sp = GetSP()) {
+    switch (type) {
+    case eMaskTypeCode:
+      return process_sp->GetCodeAddressMask();
+    case eMaskTypeData:
+      return process_sp->GetDataAddressMask();
+    case eMaskTypeHighmemCode:
+      return process_sp->GetHighmemCodeAddressMask();
+    case eMaskTypeHighmemData:
+      return process_sp->GetHighmemDataAddressMask();
+    case eMaskTypeAny:
+      return process_sp->GetDataAddressMask();
+    }
+  }
+  return default_mask;
+}
+
+void SBProcess::SetAddressMask(lldb::AddressMaskType type, addr_t mask) {
+  LLDB_INSTRUMENT_VA(this, type, mask);
+  if (ProcessSP process_sp = GetSP()) {
+    switch (type) {
+    case eMaskTypeCode:
+      process_sp->SetCodeAddressMask(mask);
+      break;
+    case eMaskTypeData:
+      process_sp->SetDataAddressMask(mask);
+      break;
+    case eMaskTypeHighmemCode:
+      process_sp->SetHighmemCodeAddressMask(mask);
+      break;
+    case eMaskTypeHighmemData:
+      process_sp->SetHighmemDataAddressMask(mask);
+      break;
+    case eMaskTypeAll:
+      process_sp->SetCodeAddressMask(mask);
+      process_sp->SetDataAddressMask(mask);
+      process_sp->SetHighmemCodeAddressMask(mask);
+      process_sp->SetHighmemDataAddressMask(mask);
+      break;
+    }
+  }
+}
+
+addr_t SBProcess::FixCodeAddress(addr_t addr) {
+  LLDB_INSTRUMENT_VA(this, addr);
+  if (ProcessSP process_sp = GetSP())
+    return process_sp->FixCodeAddress(addr);
+  return addr;
+}
+
+addr_t SBProcess::FixDataAddress(addr_t addr) {
+  LLDB_INSTRUMENT_VA(this, addr);
+  if (ProcessSP process_sp = GetSP())
+    return process_sp->FixDataAddress(addr);
+  return addr;
+}
+
+addr_t SBProcess::FixAnyAddress(addr_t addr) {
+  LLDB_INSTRUMENT_VA(this, addr);
+  if (ProcessSP process_sp = GetSP())
+    return process_sp->FixAnyAddress(addr);
+  return addr;
+}
+
 lldb::addr_t SBProcess::AllocateMemory(size_t size, uint32_t permissions,
                                        lldb::SBError &sb_error) {
   LLDB_INSTRUMENT_VA(this, size, permissions, sb_error);