Thanks for checking. As long as ld64 allows multiple, I am good.

The N_AST symbol is used to specify a file on disk that contains the one and only Swift AST blob. This blob of data will be given back to the Swift compiler that is built into LLDB. I don't think this can be specified multiple times. Inline comments ask if

Since no one else is responding, lets try this out and see how things go.

LGTM

Nice! Do you have anymore patches that we can do to improve dsymutil now that we don't require byte-for-byte compatibility with dsymutil-classic?

I think this really should be fixed in the VS code variable view. It allows debugging of languages that support having multiple variables with the same name, so their debugger should support it.

Just need to fix the default python arguments that were: "foo={}". This doesn't do what you think it will do. See https://docs.quantifiedcode.com/python-anti-patterns/correctness/mutable_default_value_as_argument.html

In D98761#2664375, @clayborg wrote:

Anyone else have any issue with this patch?

Anyone else have any issue with this patch?

Looks good as long as we fix to DWARFUnit::GetRnglist() to not return a full copy of the "llvm::Optional<llvm::DWARFDebugRnglistTable>" each time it is called.

Sounds good. Lets gets this in then and start iterating on future patches!

Looking really good.

Remove show_inline_callsite_line_info setting from SymbolContext dumping function as it isn't needed, show_inlined_frames controls this.

I am fine with always adding MAP_NORESERVE unless anyone can think of a reason not to add it. Or we can modify this patch to add the MAP_NORESERVE only when PROT_WRITE isn't specified if that is worrying anyone.

I'd love to get this in as this affects many of our clients. Does anyone else have an issues with this?

In D99497#2656502, @labath wrote:

Isn't there a better way to ensure synchronization here? Maybe, executing some command (setting a breakpoint, for instance), that will ensure that all symbols have been parsed?

lgtm!

Thanks for looking into this!

Simplify expression for calculating the m_next_offset.

Fix issues found by JDevlieghere.

I built on mac with ASAN and ran the tests and they passed just fine. I was also able to compile on Debian linux and the tests ran ok for me. Can anyone else reproduce this failure?

In D99401#2653285, @dblaikie wrote:

(generally makes sense to me - but I'll leave it to some more lldb-focussed reviewers to do more review/approval)

Usual caveat/question: Does this take us closer or further away from unifying this code with LLVM's libDebugInfoDWARF? (or neutral)

I will take a look and apply this patch on my linux server and enable ASAN. Looks like lldb-vscode must be crashing from what the errors look like.

Don't store a debugger pointer in the lldb_private::Progress class, store a debugger ID as an optional value. Then use this value in the debugger report progress function to do the right thing.

Added the option for people to add a debugger to a lldb_private::Progress instance. This allows progress to be reported only to specific debuggers.

Removed all callback related code and now only use events.

In D97739#2636343, @jingham wrote:

Thanks for doing this! The event version looks pretty clean to me. I think we should go that way. I don't think we should have two ways, that seems confusing and still leaves us calling unknown user code in the middle of symbol updates...

Added support for getting progress events via SBEvent delivery.

If anyone wants to try this out, you can yaml2obj the yaml file and then do manual lookups. If you are on a mac, you can also debug any binary with inline function calls and stop anywhere and compare "bt" to the "image lookup --address $pc" to verify this works as expected.

Remove extra parameter call to SymbolContext::DumpStopContext() recrusive call that was left over from iterating on my patch.

I agree that we the hover happens way too often and it actually happens with any text you hover over including comments, so this is a good change.

Looks good to me, probably best to get an ok from someone else as well.

In D97739#2608510, @friss wrote:

In D97739#2607961, @clayborg wrote:

In D97739#2607869, @jingham wrote:

This way of doing progress is going to look odd in anything that uses multiple debuggers. If I'm reading the code aright, if I have two debuggers, and a target in Debugger A starts doing something that would cause progress traffic, both debuggers will show activity.

that is true, but it is a global module repository that benefits both debuggers. And I very rarely debug two things at the same time, so most of the time for most people this will be beneficial and shouldn't cause too much confusion.

Just one tidbit here. Most users are actually routinely running tens of debuggers at the same time, because tests run in parallel and they have a debugger attached by default. Now if you have a long running operation kick in in your unit tests, you might already have a different kind of issue, but I’d like to avoid a world where the IDE displays spurious and wrong information because of this.

In D97739#2608066, @jingham wrote:

In D97739#2607993, @clayborg wrote:

I agree that we should avoid SBEvent after thinking about it.

First off, doing long running operations on a thread that is the one handling the major lldb events is obviously a bad idea. The driver doesn't do it this way. Commands get run on the main thread, and events get processed on the event-handler thread.

I agree that we should avoid SBEvent after thinking about it.

One serious vote against the SBEvent way of doing things is that it might stop progress notification from appearing in the UI immediately if someone is currently calling something that causes a long running operation from the main thread that is running the SBEvent loop...

SBDebugger currently doesn't vend any events so that would need to be added.

In D97739#2607869, @jingham wrote:

This way of doing progress is going to look odd in anything that uses multiple debuggers. If I'm reading the code aright, if I have two debuggers, and a target in Debugger A starts doing something that would cause progress traffic, both debuggers will show activity.

yes, anything in the debug_aranges are safe to ignore as dsymutil will create a new debug_aranges and doesn't end up using any of the debug_aranges sections from the .o files and nothing from debug_aranges ends up in the debug map.

So we have had a vote for allowing multiple callbacks. Any other main issues with the approach in this patch? If everyone is mostly happy, please chime in and I will make unit tests and vscode tests.

Fix typo in headerdoc comment.

This version should address all comments and issue that I am aware of. Now the callbacks are registered with each debugger and the Progress class no longer has any global state. The Progress class calls a static method on the debugger which allows the Debugger class to iterate over the global debugger list and report progress to any debuggers that have callbacks. This also means that later we can modify the Progress class to allow debugger specific progress notifications by modifying the Progress class to take a debugger ID or pointer as an optional constructor argument and then that progress can be reported to only that debugger instance. I don't have any places I need this yet, so I haven't added support for it.

Update fixes:

• make progress callback and baton members of a lldb_private::Debugger object to allow each debugger to have their own callbacks.
• switch to have std::string title and clients use llvm::formatv() to create the title to avoid printf variadic args in progress constructor
• make the progress class thread safe using a mutex
• verified that lldb-vscode callback is threadsafe

I would mark as accepted but I am not the code owner for llvm/lib/Support/Unix.

From my brief reading of the man pages on linux I don't see any issue with adding this flag.

LGTM

This is fully hooked up and working in Visual Studio Code now! Works well. The Visual Studio Code shows any progress that takes more than 0.5 seconds down in the bottom toolbar for me. We can and should discuss these changes now that they are working.

Added a more granular progress when manaully indexing the DWARF where we report on unit of progress for parsing all dies in a unit, and one unit of progress for indexing each compile unit, and one unit of work when merging the IndexSets.

Forgot to mention I added a lldb-vscode implementation after someone told me they now support progress reporting in Visual Studio Code. See https://microsoft.github.io/debug-adapter-protocol/specification#Events_ProgressStart

Added a "uint64_t progress_id;" as a member variable of the lldb_private::Progress class to help users track individual progress dialogs without conflict. Prior to this the message was assumed to be unique.

In D97739#2596218, @jingham wrote:

I haven't thought about the details of this implementation closely yet. How will this handled nested progress bars? For instance, if I'm Indexing (maybe even on multiple threads) and then one of the threads results in a debug symbols fetch request (e.g. dsymForUUID). Is the consumer just supposed to check the string that comes into the progress callback to match the now two simultaneous progress elements?

Another thing to clarify: The current design of the Progress objects, when destructed, will report progress with the stored "m_total" value to indicate that the progress is complete. Even if exceptions are thrown, the destructor will still be called. So all current uses of the "Progress" objects, will currently report progress when constructed with a "completed" value of zero:

ProgressCallback(message = "Indexing DWARF for /tmp/a.out", completed = 0, total = UINT64_MAX, baton = <baton>);

Then when the Progress object is destroyed, we will report a completed value that is equal to the "Progress::m_total" to indicate things are done:

ProgressCallback(message = "Indexing DWARF for /tmp/a.out", completed = UINT64_MAX, total = UINT64_MAX, baton = <baton>);

The idea is when the code that installs the progress callback gets its callback called, and if the "completed == total" then the UI can remove the progress indicator for "<message>". The message string is a const string and will be unique and the same pointer value for the message will always be used, so the progress is keyed off of the message pointer value.

The current Progress class allows users to specify how many items are to be done when they construct a lldb_private::Progress object, but this part is not used right now for anything yet because it is often hard to know how many items of work there are. For symbol tables in ELF, we have two of them, but we might also create new symbols from EH frame or other sources when there are no symbols, so it was hard to pre-determine a full number of symbols for a given executable. Same goes for mach-o files with the normal symbol table, then new symbols are created from the LC_FUNCTION_STARTS load command that has start addresses for all functions, many of which have symbols. For DWARF, we have 3 ways to index the DWARF: Apple tables, .debug_names, and manually. I will comment inline where we could specify a valid number of work items for DWARF to improve the feedback. The other reason the progress objects don't report immediate feedback is the frequency of the progress callbacks could slow things down if they are called too often with too little work. I am happy to talk about any of these issues if anyone has any questions.

So this is a first pass on adding progress notifications to the LLDB public API, and to see how it would be used internally. If we can agree on the implementation, then I will add full tests and documentation to this diff. We used a similar patch to this in an older internal version of lldb-vscode before lldb-vscode was fully open sourced, so we know it works at a base level.

In D97644#2593228, @jasonmolenda wrote:

Another alternative to RegisterContext::BehavesLikeZerothFrame that I've thought of is RegisterContext::GetPCForSymbolication, similar to GetPC() today. I think everyone who is decrementing $pc is doing it for symbolication, so having this hint and then leaving it to everyone to decrement-or-not may not be a great choice. It may be easier for higher-levels if RegisterContext can provide an Address suitable for symbolication directly.

In D96035#2585694, @friss wrote:

In D96035#2585324, @avl wrote:

How would you guarantee that the artificial CU is always emitted in the same order? It seems like you need to stash the types somewhere, then create a deterministic ordering and then emit it, but I don't remember whether that's doable with LLVM's DIEs (It's been a while since I touched this code).

We might sort types on name basis inside that artificial CU. In this case it would always be emitted in the same order.

I don't think finding how to sort them is the complicated part. In my mind the issue is keeping them around until you can sort them as I'm not sure DIEs can exist not attached to a CU. Also, it's not just the one type, but the transitive closure of all its dependencies that you need to consider which makes things (much) more complicated. As I said before, I haven't worked on this in a long time, so feel free to shoot my concerns down!

One idea is for each type we want to unique, each CU as it is being parsed on its own thread creates a type map that maps "type compile unit path" (which is DW_AT_decl_file of that type) to a list of uniqued types with decl context info. This would allow all types with the same DW_AT_decl_file to be stored in the same type compile unit. As we emit the debug info for a normal compile unit, any type references are changed to be DW_AT_ref_addr references with values that will need to be fixed up. After all normal CUs have been emitted, combine all of the CU specific type maps together, and now emit the type compile units (carefully to include all needed children in each type to make the most complete version of the type possible and emit them) in a sorted order to allow determinism. Then we fix up all DW_AT_ref_addr references in the normal CUs to point to the one type definition from the type compile units.

In D96035#2579263, @avl wrote:

@JDevlieghere @aprantl @echristo @dblaikie @clayborg

What do you think of this patch? Is it OK in general and Is it worth to divide it into smaller patches and integrate?

In D96637#2573758, @aadsm wrote:

I don't think this does what you think it does. The $() doesn't give you the process id of anything -- it substitutes a string by the result of running that string as a shell command. So, the PID variable would get the (entire) stdout of %s.out

I'm confused here, "the PID variable would get the (entire) stdout of %s.out" is exactly what I'm expecting to happen, the stdout of the program is its pid.

I was finally able to figure out what the issue was. I thought pause() would continue once the debugger attached because it sends a signal, but that doesn't seem to be the case?

needs a test. Just make a test app with a shared library that gets dlopen'ed in the main function. At first the breakpoint in the shared library will be unresolved, then after stepping over the dlopen call, it should get resolved.

In D96817#2569595, @dblaikie wrote:

CRTP was my first implementation, however, I discarded it as more bug-prone. Virtual Clone function at the interface is so common that, I believe, everyone knows it must be overridden by a new derived class. The necessity of inheriting from base_clone_helper is not so obvious.

I would've thought it'd be pretty easy to accidentally miss either of these - I think the CRTP helper ensures consistency of implementation (harder to accidentally slice/copy the wrong type/etc. But I'm not a code owner/major contributor to lldb specifically, so probably more up to other developers who are.

In D96637#2567269, @aadsm wrote:

We should have a test for this.

how do you recommend doing this? I spent a couple of hours on this but got no where. From what I understood we should prefer lit tests, so I was thinking of creating a binary that dlopens a module. However, I wasn't able to create a binary that I can start and capture its pid address so that I can attach to. Here's what I've tried so far:

// RUN: cp %s %s.cpp
// RUN: %clang -g -O0 --target=x86_64-linux-gnu %s.cpp -o %s.out
// RUN: PID=$(%s.out)
// RUN: %lldb -p $PID -b -o 'target list' | FileCheck %s
// RUN: kill -9 $PID
// CHECK: foo

#include <stdio.h>
#include <unistd.h>

int main() {
    pid_t pid = fork();
    if (pid > 0) {
        // parent process, print child pid
        printf("%d", pid);
        return 0;
    } else if (pid < 0) {
        printf("Unable to fork\n");
        return -1;
    }
    // child process
    pause();
}

The lit test get stuck on // RUN: PID=$(%s.out). Not sure why, the parent process shouldn't wait on its children..

Can each real stack frame inject N frames in the middle of a stack frame for these asynchronous functions? Or is this like the libdispatch stuff that show more frames at the end of a normal stack trace? Can you attach some sample backtraces with some annotations?

Makes sense, LGTM.

I added Jim and Pavel to get some more feedback.

After thinking about this a bit more, it seems a mistake that we can't change the file handles _prior_ to sourcing the init files. I have proposed one solution in the inline comments to create a new SBDebugger::Create() method that takes the input output and error file handles as arguments. The other way to do this is to make it so that we _can_ run the init files later. Currently if we say "false" to "bool source_init_files", then the command interpreter marks the init files as not being able to be run again later if you call the SBCommandInterpreter::SourceInitFileInHomeDirectory(...). So we need to either be able to either specify the file handles at debugger creation time, or run them one time later. The current "run them once" means "you can only do this at debugger creation time, but never again if you specify false for source_init_files. We can change this to be "only run the init files once. so if we already run them in SBDebugger::Create(), then don't run them again if you call SBCommandInterpreter::SourceInitFileInHomeDirectory()

See inlined comment about the packet name issue and let me know what you think

Handling inter-CU references: inter-CU references are hard to process using only one pass. f.e. if CU1 references CU100 and CU100 references CU1, we could not finish handling of CU1 until we finished CU100. Thus we either need to load all CUs into the memory, either load CUs several times.