[dsymutil][DWARFlinker] implement separate multi-thread processing for compile units.

[avl]

Summary

The idea of that refactoring is to speed up DWARFLinker by utilizing parallel execution and minimizing memory usages(in case the object file contains multiple compilation units). This patch creates a new library DWARFLinkerNext, which is called from standard dsymutil using --use-dlnext command-line option. This is a second version of the patch which additionally does types merging and produces a predictable result(generated .dSYM file is the same between similar runs). The best configuration is able to produce .debug_info table of 68% less in size and works two times faster than the current upstream dsymutil.

The overall linking process looks like this:

parrallel_for_each(ObjectFile) {
  for_each (Compile Unit) {
    1. load Clang modules.
  }
    
  parrallel_for_each(Compile Unit) {
    1. load input DWARF for Compile Unit.
    2. report warnings for Clang modules.
    3. analyze live DIEs.
    4. analyze types if ODR deduplication is requested.
    5. clone DIEs(Generate output DIEs and resulting DWARF tables).
       The result is in OutDebugInfoBytes, which is ELF file containg
       DWARF tables corresponding for current compile unit.
    6. cleanup Input and Output DIEs.
  }  
  
  deallocate loaded Object file.
}

if (ODR deduplication is requested)
  generate artificial compilation unit "Type Table"(It uses partially generated dies at clone stage).

for_each (ObjectFile) {
  for_each (Compile Unit) {
    1. set offsets to Compile Units DWARF tables.
    2. sort offsets/attributes/patches to have a predictable result.
    3. patch size/offsets fields.
    4. generate index tables.
    5. move DWARF tables of compile units into the resulting file.
  }
}

i.e. every compile unit is processed separately, visited only once
(except case inter-CU references exist), and used data are freed
after the compile unit is processed. The resulting file is glued
from generated debug tables, corresponding to separate compile units.

Handling inter-CU references: inter-CU references are hard to process
using only one pass. f.e. if CU1 references CU100 and CU100 references
CU1, we could not finish handling of CU1 until we finished CU100.
Thus we either need to load all CUs into the memory, either load CUs several times.
This patch discovers dependencies during the first pass. So that depending compile
units stay loaded into the memory. And then, during the second pass it handles
inter-connected compile units.

That approach works well for cases when the number of inter-CU references is low. It allows not to load all CUs into the memory at once.

Changes from the current implementation(making DWARFLinkerNext to be binary incompatible with current DWARFLinker):

a) No common abbreviation table. Each compile unit has
   its own abbreviation table. Generating common abbreviation table slowdowns parallel execution(This is a resource that is accessed many times from many threads). Abbreviation table does not take a lot of space,
   so it looks cheap to have separate abbreviations tables.
   Later, it might be optimized a bit(by removing equal abbreviations tables).
   
b) .debug_frame. Already generated CIE records are not reused between object files

c) ODR type deduplication works using another approach than current dsymutil. All types are moved into the artificial compilation unit. References to the types are changed so that they point to types from artificial compilation unit. Declarations and partial declarations of the same type are merged into single definition/declaration located in the artificial compilation unit.

Some DWARF features are not supported(it might be supported though): DWARF64, split dwarf, DWARF 5.

Performance results for this patch for the clang binary(Darwin 24-core 64G):

1. clang binary, ODR deduplication is ON:

|----------------------------------------------------------------------
|       |           dsymutil           |     dsymutil --use-dlnext    |
|-------|------------------------------|------------------------------|
|       |exec time|  memory  | DWARF(*)|exec time|  memory  |  DWARF  |
|       |   sec   |    GB    |    MB   |   sec   |    GB    |   MB    |
|-------|------------------------------|------------------------------|
|threads|         |          |         |         |          |         |
|-------|------------------------------|------------------------------|
|   1   |   161   |   16.5   |   486   |   204   |   12.0   |   157   |
|-------|------------------------------|------------------------------|
|   2   |   101   |   17.9   |   486   |   118   |   12.0   |   157   |
|-------|------------------------------|------------------------------|
|   4   |   101   |   17.9   |   486   |    75   |   12.0   |   157   |
|-------|------------------------------|------------------------------|
|   8   |   101   |   17.9   |   486   |    51   |   12.0   |   157   |
|-------|------------------------------|------------------------------|
|  16   |   101   |   17.9   |   486   |    42   |   12.2   |   157   |
|---------------------------------------------------------------------|

(*) DWARF is the size of .debug_info section.

2. clang binary, ODR deduplication is OFF(--no-odr):

|----------------------------------------------------------------------
|       |           dsymutil           |     dsymutil --use-dlnext    |
|-------|------------------------------|------------------------------|
|       |exec time|  memory  |  DWARF  |exec time|  memory  |  DWARF  |
|       |   sec   |    GB    |    MB   |   sec   |    GB    |   MB    |
|-------|------------------------------|------------------------------|
|threads|         |          |         |         |          |         |
|-------|------------------------------|------------------------------|
|   1   |   227   |   16.3   |   1460  |   227   |   16.2   |   1470  |
|-------|------------------------------|------------------------------|
|   2   |   218   |   17.9   |   1460  |   132   |   16.2   |   1470  |
|-------|------------------------------|------------------------------|
|   4   |   218   |   17.9   |   1460  |    83   |   16.5   |   1470  |
|-------|------------------------------|------------------------------|
|   8   |   218   |   17.9   |   1460  |    58   |   16.5   |   1470  |
|-------|------------------------------|------------------------------|
|  16   |   218   |   17.9   |   1460  |    47   |   16.9   |   1470  |
|---------------------------------------------------------------------|

Effective CPU utilization ratio for that implementation is 52%.

There still exist room for improvements. The run-time performance might be speed-up to 10-20%. The run-time memory usage might be decreased up to 10%.

[clayborg]

Handling inter-CU references: inter-CU references are hard to process using only one pass. f.e. if CU1 references CU100 and CU100 references CU1, we could not finish handling of CU1 until we finished CU100. Thus we either need to load all CUs into the memory, either load CUs several times.

What kind of inter-CU references do you see? There really shouldn't be any unless they are all in the same .o file and were produced by an LTO pass. Are there any inter CU references that I am forgetting about?

a) No common abbreviation table. Each compile unit has its own abbreviation table. Generating common abbreviation table slowdowns parallel execution (This is a resource that is accessed many times from many threads). Abbreviation table does not take a lot of space, so it looks cheap to have separate abbreviations tables. Later, it might be optimized a bit(by removing equal abbreviations tables).

Can't the linking process create on large uniqued abbrev table as it links each .o file's DWARF into the final output? This info isn't that large that is true, but it'd be nice to have just one table as long as we are optimizing the DWARF.

b) Removed check for ambiguous type declarations. Existed check does not solve the problem in general (current check does not work if there are ambiguous types in different compile units).

I am assuming you didn't pull out ODR type uniquing all together right? That is very important and one of the main reasons for using dsymutil. There are many things that can make this ODR type uniquing fail to work correctly, but those are bugs that should be fixed. One of the main things is if a class or struct has a forward declaration to another class inside of it. If this is the case the llvm dsymutil doesn't try and see if everyone has the same full class definition with a forward decl on it, it just punts and emits another copy of the type. I saw this when I would have a "FILE" type from the unix fopen(...) function. If multiple files had this type in a C++ source file, they wouldn't get uniqued because one of the members had a forward declaration. The reason it doesn't get uniqued is dsymutil worries someone might have a "FILE" type info in the DWARF that _does_ have a full definition for that forward declaration. A lot of stuff the compiler emits these days also causes many of the types to not be correctly removed even though they aren't used.

So there are many C++ compiler things that seem to be keeping things alive in the DWARF output by dsymutil these days that used to not be there before. When ODR works correctly, the first instance of a type gets emitted from the first object file file that has it, and then all other .o files that have this same type remove it and have all CU references within changed to a DW_FORM_ref_addr inter CU reference. We should make sure that this isn't happening to many clang types and causing us to not unique types.

When the type uniquing works correctly it really reduces the output DWARF size. Back at Apple the WebCore.framework DWARF debug info went from 1.8GB down to 450MB.

As an example of C++ compiler type bloat, you can see this if you do "#include <cstdio>". If you compile this with a recent clang:

#include <stdio.h>

int main() {
    printf("hello\n");
    return 0;
}

The DWARF looks very small:

0x0000000b: DW_TAG_compile_unit
              DW_AT_producer	("Apple clang version 12.0.0 (clang-1200.0.32.29)")
              DW_AT_language	(DW_LANG_C_plus_plus_11)
              DW_AT_name	("main.cpp")
              DW_AT_LLVM_sysroot	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk")
              DW_AT_APPLE_sdk	("MacOSX.sdk")
              DW_AT_stmt_list	(0x00000000)
              DW_AT_comp_dir	("/Users/gclayton/Documents/src/args")
              DW_AT_low_pc	(0x0000000100003f60)
              DW_AT_high_pc	(0x0000000100003f8a)

0x00000032:   DW_TAG_subprogram
                DW_AT_low_pc	(0x0000000100003f60)
                DW_AT_high_pc	(0x0000000100003f8a)
                DW_AT_frame_base	(DW_OP_reg6 RBP)
                DW_AT_name	("main")
                DW_AT_decl_file	("/Users/gclayton/Documents/src/args/main.cpp")
                DW_AT_decl_line	(4)
                DW_AT_type	(0x000000000000004b "int")
                DW_AT_external	(true)

0x0000004b:   DW_TAG_base_type
                DW_AT_name	("int")
                DW_AT_encoding	(DW_ATE_signed)
                DW_AT_byte_size	(0x04)

0x00000052:   NULL

But if you include the "cstdio" file instead:

#include <cstdio>

int main() {
    printf("hello\n");
    return 0;
}

This ends up having tons of extra type stuff that really bloats the DWARF. The C++ header ends up with full definitions for every function in stdio.h and data types (like "FILE"!) for functions that are not used in the .o file. In this case the C++ code add DW_TAG_imported_declaration to all these types and that ends up keeping all of these extra types alive since something refers to it and ends up with way toe much stuff in the final dSYM. We can see the DW_TAG_imported_declaration below:

0x00000032:   DW_TAG_namespace
                DW_AT_name	("std")

0x00000037:     DW_TAG_namespace
                  DW_AT_name	("__1")
                  DW_AT_export_symbols	(true)

0x0000003c:       DW_TAG_imported_declaration
                    DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstdio")
                    DW_AT_decl_line	(109)
                    DW_AT_import	(0x0000000000000195)
`

And later we see:

0x00000193:       NULL

0x00000194:     NULL

0x00000195:   DW_TAG_typedef
                DW_AT_type	(0x00000000000001a0 "__sFILE")
                DW_AT_name	("FILE")
                DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h")
                DW_AT_decl_line	(157)

0x000001a0:   DW_TAG_structure_type
                DW_AT_calling_convention	(DW_CC_pass_by_value)
                DW_AT_name	("__sFILE")
                DW_AT_byte_size	(0x98)
                DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h")
                DW_AT_decl_line	(126)

0x000001a9:     DW_TAG_member
                  DW_AT_name	("_p")
                  DW_AT_type	(0x000000000000029a "unsigned char*")
                  DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h")
                  DW_AT_decl_line	(127)
                  DW_AT_data_member_location	(0x00)
....

c) .debug_frame. Already generated CIE records are not reused between object files

Can't we unique these when the .o file gets linked on the main thread?

d) In multi-thread mode there is no predictable order of how compile units/dies are processed. That leads to selecting "canonical" dies, or adding strings into the debug_str table in some unpredictable order. It is semantically correct but creates not-matching references:

DW_AT_name [DW_FORM_strp]     ( .debug_str[0x00000095] = "AnonC")   
DW_AT_name [DW_FORM_strp]     ( .debug_str[0x0000007f] = "AnonC")

DW_AT_type [DW_FORM_ref_addr] (0x000000000000014f "U")
DW_AT_type [DW_FORM_ref_addr] (0x0000000000000170 "U")

Yes this will happen and it is ok as long as "--num-threads 1" allows consistent builds. Any clients that require dSYM files to be the same can use this slower method

e) .debug_names is not supported by this patch.

Any reason why this can't be done? I believe that dsymutil just generates the .debug_names (or Apple accelerator tables) after you have written all of the DWARF out by traversing it no? If this doesn't happen, it can be added with the --update option to dsymutil, but again, this takes time.

One of the main benefits of running dsymutil is the accelerator tables that speed up debugging. If you are solely looking for a way to reduce link times, then this succeeds. But if there are no accelerator tables, each time you debug this the debugger is going to have to manually index all of the DWARF and cause long delays each time you debug something. You might as well just not run the dsymutil and leave the DWARF in the .o files as that is the workflow that is designed to be used during the edit/compile/debug workflow.

I haven't looked at the code itself yet, just wanted to get some more overall info on the approach.

[dblaikie]

Snipping a fragment here:

A lot of stuff the compiler emits these days also causes many of the types to not be correctly removed even though they aren't used.
So there are many C++ compiler things that seem to be keeping things alive in the DWARF output by dsymutil these days that used to not be there before.

Depends on your definition of "used" I guess, and I don't think it's
much related to dsymutil. at least the example you give is
specifically about imported_declarations - I implemented those (... 7
years ago https://github.com/llvm/llvm-project/commit/bd4837665b2ee1e77dc9337e11ee2aed52a00a93
) & yeah, they're not great. Specifically the compiler doesn't track
if they are used for name lookup, so we don't have a way to only emit
those that were used - and even if we did emit only the used ones,
they aren't used in the DWARF sense (if you have a "using namespace
x;" in code - references to names in namespace 'x' don't refer to that
using declaration in the DWARF (unlike a typedef, or a using
declaration "using x = y;")) so you wouldn't be able to strategically
drop them in dsymutil. (eg: if a certain using directive was only used
by one function body, but that function was optimized away or not
linked - there's no chain of liveness between that function and the
using directive - no way to know which using directives should be kept
around).

& even using declarations for functions (unlike for types) - again, no
chain of connection. So if you call a function via a using declaration
("void f1(); using f2 = f1; void f3() { f2(); }") there's nothing in
the DWARF that says f2 is used by f3 - with call_site debug info we
could describe the call-site, but it wouldn't cover all references to
f2 (indirect calls, for instance).

So, yeah, compiler improvements to track the used-ness of using
declarations and directives would be nice. Oh, but even if we knew
they were used, we'd have to know /where/ they were used from (eg: if
the were used in an inline function that was never called, that
wouldn't be much use to us/we wouldn't want to emit them in the
resulting DWARF).

But even if we did have that information, unfortunately I'm not sure
it'd help dsymutil greatly - because currently those entities don't
always form chains of ownership within the DWARF, so you just sort of
have to keep carrying them.

Though they'd still benefit from all the usual type deduplication
stuff - so this is probably orthogonal to the discussion.

• Dave

[avl]

In D96035#2543434, @clayborg wrote:

Handling inter-CU references: inter-CU references are hard to process using only one pass. f.e. if CU1 references CU100 and CU100 references CU1, we could not finish handling of CU1 until we finished CU100. Thus we either need to load all CUs into the memory, either load CUs several times.

What kind of inter-CU references do you see? There really shouldn't be any unless they are all in the same .o file and were produced by an LTO pass. Are there any inter CU references that I am forgetting about?

yes. That is the case: they are all in the same .o file and were produced by an LTO pass.

a) No common abbreviation table. Each compile unit has its own abbreviation table. Generating common abbreviation table slowdowns parallel execution (This is a resource that is accessed many times from many threads). Abbreviation table does not take a lot of space, so it looks cheap to have separate abbreviations tables. Later, it might be optimized a bit(by removing equal abbreviations tables).

Can't the linking process create on large uniqued abbrev table as it links each .o file's DWARF into the final output? This info isn't that large that is true, but it'd be nice to have just one table as long as we are optimizing the DWARF.

If we try to generate it while cloning DIEs it leads to slow execution, because threads are waiting for each other.
(All threads need access to the global abbreviation table for every die. Thus execution time spent on waiting).

If we try to implement it as some post-process step(when all compile units are processed) - then it would also require too much additional time. For each DIE we need to add abbreviations into the global abbreviation table. This might change the abbreviation code. Since the abbreviation code is ULEB128 - The size of DIE might change. That means we need to update all reference attributes. It would take a significant amount of time. Technically it might be done, but it looks like it would not be practical.

Probably, it might be easily done for "num-threads 1" case only.

b) Removed check for ambiguous type declarations. Existed check does not solve the problem in general (current check does not work if there are ambiguous types in different compile units).

I am assuming you didn't pull out ODR type uniquing all together right? That is very important and one of the main reasons for using dsymutil.

Right. This patch does mostly the same ODR deduplication as the current dsymutil.
There are small differences:

1. In multi-thread mode, some canonical dies might become not seen by some threads.
   
   If the request for setting canonical dies come from several threads at the same time - some threads might receive information that there is not canonical die(though it is actually, it would be set by another thread). As the result, some type would not be de-duplicated. But the number of such cases is small. For the clang, there is no noticeable difference in the size of the binary.

2. Ambiguous types. That is what currently done by this code: https://github.com/llvm/llvm-project/blob/main/llvm/lib/DWARFLinker/DWARFLinkerDeclContext.cpp#L19
   
   i.e. when dsymutil can not distinguish function type because of parameterized arguments. It marks the declaration context as invalid. The current workaround does not solve the problem fully. A similar approach might be implemented(with the same limitations) by this patch. But, probably, we might want to think of a better solution.

There are many things that can make this ODR type uniquing fail to work correctly, but those are bugs that should be fixed. One of the main things is if a class or struct has a forward declaration to another class inside of it. If this is the case the llvm dsymutil doesn't try and see if everyone has the same full class definition with a forward decl on it, it just punts and emits another copy of the type. I saw this when I would have a "FILE" type from the unix fopen(...) function. If multiple files had this type in a C++ source file, they wouldn't get uniqued because one of the members had a forward declaration. The reason it doesn't get uniqued is dsymutil worries someone might have a "FILE" type info in the DWARF that _does_ have a full definition for that forward declaration. A lot of stuff the compiler emits these days also causes many of the types to not be correctly removed even though they aren't used.

So there are many C++ compiler things that seem to be keeping things alive in the DWARF output by dsymutil these days that used to not be there before. When ODR works correctly, the first instance of a type gets emitted from the first object file file that has it, and then all other .o files that have this same type remove it and have all CU references within changed to a DW_FORM_ref_addr inter CU reference. We should make sure that this isn't happening to many clang types and causing us to not unique types.

When the type uniquing works correctly it really reduces the output DWARF size. Back at Apple the WebCore.framework DWARF debug info went from 1.8GB down to 450MB.

As an example of C++ compiler type bloat, you can see this if you do "#include <cstdio>". If you compile this with a recent clang:

#include <stdio.h>

int main() {
    printf("hello\n");
    return 0;
}

The DWARF looks very small:

0x0000000b: DW_TAG_compile_unit
              DW_AT_producer	("Apple clang version 12.0.0 (clang-1200.0.32.29)")
              DW_AT_language	(DW_LANG_C_plus_plus_11)
              DW_AT_name	("main.cpp")
              DW_AT_LLVM_sysroot	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk")
              DW_AT_APPLE_sdk	("MacOSX.sdk")
              DW_AT_stmt_list	(0x00000000)
              DW_AT_comp_dir	("/Users/gclayton/Documents/src/args")
              DW_AT_low_pc	(0x0000000100003f60)
              DW_AT_high_pc	(0x0000000100003f8a)

0x00000032:   DW_TAG_subprogram
                DW_AT_low_pc	(0x0000000100003f60)
                DW_AT_high_pc	(0x0000000100003f8a)
                DW_AT_frame_base	(DW_OP_reg6 RBP)
                DW_AT_name	("main")
                DW_AT_decl_file	("/Users/gclayton/Documents/src/args/main.cpp")
                DW_AT_decl_line	(4)
                DW_AT_type	(0x000000000000004b "int")
                DW_AT_external	(true)

0x0000004b:   DW_TAG_base_type
                DW_AT_name	("int")
                DW_AT_encoding	(DW_ATE_signed)
                DW_AT_byte_size	(0x04)

0x00000052:   NULL

But if you include the "cstdio" file instead:

#include <cstdio>

int main() {
    printf("hello\n");
    return 0;
}

This ends up having tons of extra type stuff that really bloats the DWARF. The C++ header ends up with full definitions for every function in stdio.h and data types (like "FILE"!) for functions that are not used in the .o file. In this case the C++ code add DW_TAG_imported_declaration to all these types and that ends up keeping all of these extra types alive since something refers to it and ends up with way toe much stuff in the final dSYM. We can see the DW_TAG_imported_declaration below:

0x00000032:   DW_TAG_namespace
                DW_AT_name	("std")

0x00000037:     DW_TAG_namespace
                  DW_AT_name	("__1")
                  DW_AT_export_symbols	(true)

0x0000003c:       DW_TAG_imported_declaration
                    DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstdio")
                    DW_AT_decl_line	(109)
                    DW_AT_import	(0x0000000000000195)
`

And later we see:

0x00000193:       NULL

0x00000194:     NULL

0x00000195:   DW_TAG_typedef
                DW_AT_type	(0x00000000000001a0 "__sFILE")
                DW_AT_name	("FILE")
                DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h")
                DW_AT_decl_line	(157)

0x000001a0:   DW_TAG_structure_type
                DW_AT_calling_convention	(DW_CC_pass_by_value)
                DW_AT_name	("__sFILE")
                DW_AT_byte_size	(0x98)
                DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h")
                DW_AT_decl_line	(126)

0x000001a9:     DW_TAG_member
                  DW_AT_name	("_p")
                  DW_AT_type	(0x000000000000029a "unsigned char*")
                  DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h")
                  DW_AT_decl_line	(127)
                  DW_AT_data_member_location	(0x00)
....

According to the problem of imported declarations - this patch does the same as the current dsymutil.
The solution for that problem would be nice to have. Though this is out of the scope of this patch.

c) .debug_frame. Already generated CIE records are not reused between object files

Can't we unique these when the .o file gets linked on the main thread?

We can, but we need to re-parse generated debug_frame then.
Probably, If it would not take a lot of time, we might to try.

e) .debug_names is not supported by this patch.

Any reason why this can't be done? I believe that dsymutil just generates the .debug_names (or Apple accelerator tables) after you have written all of the DWARF out by traversing it no? If this doesn't happen, it can be added with the --update option to dsymutil, but again, this takes time.

There is no fundamental reason. I think it might be supported.
I just need more time to check that code and understand how to implement it properly.
I would update this patch with implementation later.

[avl]

@JDevlieghere @aprantl @echristo @dblaikie @clayborg

What do you think of this patch? Is it OK in general and Is it worth to divide it into smaller patches and integrate?

[dblaikie]

A non-deterministic behavior is probably not acceptable - it'd certainly be no good at Google if we ever wanted to use this for DWARF linking. Not sure how the Apple folks feel about it for classic dsymutil.

I believe lld's multithreading still produces deterministic output, for instance.

[avl]

yep. this implementation is non-deterministic.
I tried to preserve the current ODR algorithm(which relies on concrete type sequences).

I thought about following improvement(though not for this patch),
which should be deterministic even in multi-thread mode:

Implement artificial type table with type merging:

Source DWARF:

0x0000000b: DW_TAG_compile_unit 
              DW_AT_name = "module1.cpp"
                            
0x0000002e: DW_TAG_structure_type 
              DW_AT_name = "S"

              NULL

0x0000008c:   DW_TAG_subprogram 

0x000000aa:     DW_TAG_variable  
                  DW_AT_name = "s"
                  DW_AT_type (0x0000002e "S")

            NULL

0x0000012c: DW_TAG_compile_unit 
              DW_AT_name "module2.cpp"

0x0000014f:   DW_TAG_structure_type 
                DW_AT_name = "S"

0x00000157:     DW_TAG_structure_type
                  DW_AT_name = "Nested"

0x00000160: DW_TAG_subprogram 

0x00000168:   DW_TAG_variable  
                   DW_AT_name = "n"
                   DW_AT_type (0x00000157 "Nested")

            NULL

New DWARF:

0x0000000b: DW_TAG_compile_unit
              DW_AT_name = "__type_table"

0x0000002e:   DW_TAG_structure_type 
                DW_AT_name = "S"

0x00000036:     DW_TAG_structure_type
                DW_AT_name = "Nested"

              NULL

0x0000012c: DW_TAG_compile_unit
              DW_AT_name = "module1.cpp"

0x0000014c:   DW_TAG_subprogram 

0x00000150:     DW_TAG_variable  
                  DW_AT_name = "s"
                  DW_AT_type (0x0000002e "S")

            NULL

0x0000018c: DW_TAG_compile_unit
              DW_AT_name "module2.cpp"

0x000001bc:   DW_TAG_subprogram 

0x000001c0:     DW_TAG_variable  
                  DW_AT_name = "n"
                  DW_AT_type (0x00000036 "Nested")

            NULL

In this example, "New DWARF is the result of ODR unique algorithm differrent from the current one.
The current ODR algorithm is - remember the first type occurrence and set all following references(to the same type) into that first met type.
The new ODR algorithm might be: move all types into separate artificial compile unit containing type table.
In this case type table might be always created in the same form despite the type processing order.

As additional bonus type merging might save noticeable amount of space.

[clayborg]

In D96035#2579263, @avl wrote:

@JDevlieghere @aprantl @echristo @dblaikie @clayborg

What do you think of this patch? Is it OK in general and Is it worth to divide it into smaller patches and integrate?

The original dsymutil (pre llvm version) that I wrote was deterministic, and so is the current one. That being said, I am not too worried about this because the quality of the resulting debug info is the main goal here.

My main concerns are:

• The final .debug_str should only have one copy of a string. It should be possible to create individual string tables and merge them in the last loop you have that fixes up offsets right?
• I am unclear on how ODR uniquing works here without cross talking between threads and slowing everything down. Since they currently create new DW_FORM_ref_addr that reference other already streamed out CUs, how does that work in this patch?
• I am unclear on what state the inter CU references are in in this patch when they are in the same .o file from LTO. Do they work? Are you loading all CUs into memory? I would be ok with loading all CUs from the same file at one time.

I do your idea of creating a new compile unit for types in order to make ODR better as I have thought of this. A few extra ideas on this:

• it might be best to just create type units on the fly instead of making fake compile units?
• if we don't want to create/use type units then:
  • create a new DW_TAG_compile_unit whose source path is the DW_AT_decl_file for the type in question
  • Bonus points if we can add all types form the same DW_AT_decl_file to the same DW_TAG_compile_unit
  • Some type class declarations that use compiler default created constructors, copy constructor, assignment operators might omit these member functions in various different compile units if they were or were not used by that compile unit. If we can augment the uniqued type definition to include these, that would be a huge win. Right now when ODR finds type "class A" with no compiler created constructor, it will re-emit a full type definition because DWARF needs to be able to reference it _if_ a previous copy did not include the needed method

[JDevlieghere]

In D96035#2579270, @dblaikie wrote:

A non-deterministic behavior is probably not acceptable - it'd certainly be no good at Google if we ever wanted to use this for DWARF linking. Not sure how the Apple folks feel about it for classic dsymutil.

Yes, this would be a non-starter.

[avl]

The final .debug_str should only have one copy of a string. It should be possible to create individual string tables and merge them in the last loop you have that fixes up offsets right?

Right. The final .debug_str has only one copy of a string.
The implementation is a bit different though:

While cloning compile unit, DWARFLinker remembers the pointer to the string and reference to the string attribute:

// create delayed task for setting proper offset value for the string
// attribute.
CU.noteOffsetData(It, *String);

After cloning is complete, DWARFLinker puts remembered strings into the common StringPool and attribute is updated:

void DWARFLinker::assignDelayedOffsets(CompileUnit &CU) {
  std::unique_lock<std::mutex> Guard(StringPoolMutex);

  for (const CompileUnit::OffsetsData &Data : CU.getStringsOffsetData())
    setStringsOffset(Data.AttrToPatch, Data.String);
}

This way common resource TheOffsetsStringPool is locked only once per compile unit.
After all compile units are processed the .debug_str is generated from the TheOffsetsStringPool
which has only one copy of each string.

I am unclear on how ODR uniquing works here without cross talking between threads and slowing everything down. Since they currently create new DW_FORM_ref_addr that reference other already streamed out CUs, how does that work in this patch?

ODR uniquing cross talks between threads. It is implemented this way:

During analyzing declaration context stage DWARFLinker remembers properties of each declaration context for compile unit:

// In multy-threaded case it is in-effective to lock&use
// common resource ODRContexts.getChildDeclContext() for
// every type declaration context. Instead, we remember
// DeclarationContextKey and create DeclarationContext and
// update Info.Ctxt field later. That way we lock
// ODRContexts.getChildDeclContext() only once for each compile
// unit.

CU.noteContextData(Current.ContextKey, &Info);

Later, after all types are analyzed, corresponding type declaration contexts are created in the common
ODRContexts pool.

void DWARFLinker::assignDelayedDeclContexts(CompileUnit &CU) {
  std::unique_lock<std::mutex> Guard(ContextsMutex);

  for (const CompileUnit::DeclContextKeyData &Data : CU.getDeclContextData())
    setDeclContext(Data.Key, Data.Info);
}

At that stage, Declaration contexts(which are already is in ODRContexts pool) might be reused.

I am unclear on what state the inter CU references are in in this patch when they are in the same .o file from LTO. Do they work? Are you loading all CUs into memory? I would be ok with loading all CUs from the same file at one time.

inter CU references are supported. This patch tries to load/unload not dependent compile units and load at one time only inter-connected compile units.
Let`s we have following dependencies:

CU1 references CU2, CU2 references CU4, CU4 references CU3, CU5(does not reference other CUs)

1. first pass. CU1 loaded. It is seen that it needs CU2, so left CU1 in memory. CU2 loaded. It is seen that it needs CU4, so left CU2 in memory. CU3 loaded. processed. unloaded. CU4 loaded. It is seen that it needs CU3, so left CU4 in memory. CU5 loaded. processed. unloaded.
2. second pass. CU1 processed. CU2 processed. CU3 loaded. processed. CU4 processed.

The reason why I try to avoid loading all CUs from the same file at one time, this utility - https://reviews.llvm.org/D86539.
Implementation which loads all CUs from the same file at one time would require a lot of memory if it optimize single binary(not a set of object files).

it might be best to just create type units on the fly instead of making fake compile units?

Type units might be created, but they require additional space. We might lose some size improvement.

create a new DW_TAG_compile_unit whose source path is the DW_AT_decl_file for the type in question

I thought about single artificial compile unit for all types, but yes we might implement it this way.

Some type class declarations that use compiler default created constructors, copy constructor, assignment operators might omit these member functions in various different compile units if they were or were not used by that compile unit. If we can augment the uniqued type definition to include these, that would be a huge win.

yes. That is the idea for type merging. it might save a lot of space.

[avl]

In D96035#2582232, @JDevlieghere wrote:

In D96035#2579270, @dblaikie wrote:

A non-deterministic behavior is probably not acceptable - it'd certainly be no good at Google if we ever wanted to use this for DWARF linking. Not sure how the Apple folks feel about it for classic dsymutil.

Yes, this would be a non-starter.

even if it IS deterministic in --num-threads=1, and is Not deterministic in multi-thread case?

Might changing the ODR uniquing algorithm be a solution then? (That is what I described in some previous message - do not reuse existing type declarations, but generate an artificial compile unit containing the types)? Such artificial compile unit containing the types might be generated in the deterministic way.

[echristo]

In D96035#2582780, @avl wrote:

In D96035#2582232, @JDevlieghere wrote:

In D96035#2579270, @dblaikie wrote:

A non-deterministic behavior is probably not acceptable - it'd certainly be no good at Google if we ever wanted to use this for DWARF linking. Not sure how the Apple folks feel about it for classic dsymutil.

Yes, this would be a non-starter.

even if it IS deterministic in --num-threads=1, and is Not deterministic in multi-thread case?

Yes :)

[avl]

In D96035#2582811, @echristo wrote:

In D96035#2582780, @avl wrote:

In D96035#2582232, @JDevlieghere wrote:

In D96035#2579270, @dblaikie wrote:

A non-deterministic behavior is probably not acceptable - it'd certainly be no good at Google if we ever wanted to use this for DWARF linking. Not sure how the Apple folks feel about it for classic dsymutil.

Yes, this would be a non-starter.

even if it IS deterministic in --num-threads=1, and is Not deterministic in multi-thread case?

Yes :)

I see. I tried to preserve existing idea of ODR uniquing in this patch. To make it in multi-thread way and to have a deterministic behavior - Could we consider to change current ODR uniquing algorithm?

[dblaikie]

In D96035#2582780, @avl wrote:

In D96035#2582232, @JDevlieghere wrote:

In D96035#2579270, @dblaikie wrote:

A non-deterministic behavior is probably not acceptable - it'd certainly be no good at Google if we ever wanted to use this for DWARF linking. Not sure how the Apple folks feel about it for classic dsymutil.

Yes, this would be a non-starter.

even if it IS deterministic in --num-threads=1, and is Not deterministic in multi-thread case?

Might changing the ODR uniquing algorithm be a solution then? (That is what I described in some previous message - do not reuse existing type declarations, but generate an artificial compile unit containing the types)? Such artificial compile unit containing the types might be generated in the deterministic way.

I'm not sure why that would necessarily be better/faster - it'd still require two passes, right? One to collect the types, then another to emit the unit with the types and the units referencing that?

If it requires two passes, what about one pass that decides which of the type definitions to keep in the unit that defined them, and which to remove/make a reference to the kept ones? That could then potentially produce the same kind of (& possibly exactly the same) output as is the case today, rather than introducing a new CU?

[avl]

I'm not sure why that would necessarily be better/faster - it'd still require two passes, right? One to collect the types, then another to emit the unit with the types and the units referencing that?

If it requires two passes, what about one pass that decides which of the type definitions to keep in the unit that defined them, and which to remove/make a reference to the kept ones? That could then potentially produce the same kind of (& possibly exactly the same) output as is the case today, rather than introducing a new CU?

I think I did not understand the idea. Current processing looks similar to the above description.
For the single compile unit, we do the declaration context analysis step that decides which of the type definitions to keep in the unit that defined them, and which to remove/make a reference to the kept ones. Later we emit the body of the compile unit based on the results of the declaration context analysis step.

When we decide which of the type definitions to keep in the unit we use ODR uniquing algorithm which sets the first met type definition as canonical and uses it later for type references in other compile units.

But we do not have a fixed order of compile units, they are processed in parallel. If both CU1 and CU2 have the same type definition, depending on the real order, canonical type definition might be set for CU1 or CU2. How could we avoid that non-determinism using additional pass?

Speaking of the solution with artificial CU it has several advantages before the current ODR algorithm:

1. Since DWARFLinker generates artificial CU then it might generate it in a deterministic way. Despite the real order of processed types, we might always generate them in the same form(sorted by name or another).

2. Current ODR algorithm makes CU processing be depending on common resource - ODR Declaration context. That slowdowns parallel execution(threads wait for each other). If we would generate separate artificial CU for types then processing of current CU might be done without waiting for ODR Declaration context. it might increase effective CPU utilization ratio.

3. Since we generate types - it is possible to implement type merging, which might save a significant amount of space.

[dblaikie]

In D96035#2583116, @avl wrote:

I'm not sure why that would necessarily be better/faster - it'd still require two passes, right? One to collect the types, then another to emit the unit with the types and the units referencing that?

If it requires two passes, what about one pass that decides which of the type definitions to keep in the unit that defined them, and which to remove/make a reference to the kept ones? That could then potentially produce the same kind of (& possibly exactly the same) output as is the case today, rather than introducing a new CU?

I think I did not understand the idea. Current processing looks similar to the above description.
For the single compile unit, we do the declaration context analysis step that decides which of the type definitions to keep in the unit that defined them, and which to remove/make a reference to the kept ones. Later we emit the body of the compile unit based on the results of the declaration context analysis step.

When we decide which of the type definitions to keep in the unit we use ODR uniquing algorithm which sets the first met type definition as canonical and uses it later for type references in other compile units.

But we do not have a fixed order of compile units, they are processed in parallel. If both CU1 and CU2 have the same type definition, depending on the real order, canonical type definition might be set for CU1 or CU2. How could we avoid that non-determinism using additional pass?

Comparing your proposal - avoiding nondeterminism by sorting the types by name in the new type-holding CU, we could do something similar, but instead sorting the CUs a type appears in to pick which of those existing locations to be the canonical home. (or, indeed, could sort by the original linear visitation order)

eg: multithreaded walk over each CU and find each type (this applies to both your proposal and what I'm suggesting here, I think) - then, rather than sorting by name and putting them in a new CU, sort by the "index" of the CU the type appeared in (where the index is the order the CUs would've been visited in in the current linear algorithm) then place/preserve the canonical type in the CU with the lowest index where the type appears?

Then the second pass goes and emits the CUs, consulting this list of type homes to determine whether this type should be emitted in the CU, or reference a copy emitted elsewhere.

I think there might be merit to the approach you're suggesting too - but I'm less sure it's necessary to significantly alter the output scheme to achieve the benefits of parallelism.

(as for type merging - that might also be possible with the scheme I'm proposing - if we're rewriting DIEs anyway, seems plausible we could add new child DIEs to canonical type type DIEs)

[clayborg]

One idea is for each type we want to unique, each CU as it is being parsed on its own thread creates a type map that maps "type compile unit path" (which is DW_AT_decl_file of that type) to a list of uniqued types with decl context info. This would allow all types with the same DW_AT_decl_file to be stored in the same type compile unit. As we emit the debug info for a normal compile unit, any type references are changed to be DW_AT_ref_addr references with values that will need to be fixed up. After all normal CUs have been emitted, combine all of the CU specific type maps together, and now emit the type compile units (carefully to include all needed children in each type to make the most complete version of the type possible and emit them) in a sorted order to allow determinism. Then we fix up all DW_AT_ref_addr references in the normal CUs to point to the one type definition from the type compile units.

Sorry if this is restating what has already been said/implied!

[avl]

Comparing your proposal - avoiding nondeterminism by sorting the types by name in the new type-holding CU, we could do something similar, but instead sorting the CUs a type appears in to pick which of those existing locations to be the canonical home. (or, indeed, could sort by the original linear visitation order)

eg: multithreaded walk over each CU and find each type (this applies to both your proposal and what I'm suggesting here, I think) - then, rather than sorting by name and putting them in a new CU, sort by the "index" of the CU the type appeared in (where the index is the order the CUs would've been visited in in the current linear algorithm) then place/preserve the canonical type in the CU with the lowest index where the type appears?

Then the second pass goes and emits the CUs, consulting this list of type homes to determine whether this type should be emitted in the CU, or reference a copy emitted elsewhere.

My understanding is that this way assumes all CUs should be loaded into the memory and it does extra pass. i.e.

1. the first pass enumerates in a multithreaded manner all object files, all compile units, and creates an indexed map(the list of type homes). (In the result all CUs from all object files are loaded into the memory at the same time. The indexed map is also in the memory).

2. the second pass enumerates in a multithreaded manner all object files, all compile units, and emits bodies(consulting the list of type homes to determine whether this type should be emitted in the CU, or reference a copy emitted elsewhere).

3. Patch sizes/offsets/references after individual CU bodies are glued into the resulting file.

The scheme implemented in this patch and which might be done with additional compile unit keeping types - visit CU only once and then it might be unloaded from the memory:

1. the first pass enumerates in a multithreaded manner all object files, all compile units. Each CU is loaded, analyzed for types(there would be created a list of attribute referencing types), emitted. types moved into the artificial CU. unloaded.

2. Emit artificial CU(After the first pass is finished all CU are unloaded from the memory, except the artificial one).

3. Patch sizes/offsets/references after individual CU bodies are glued into the resulting file. (At this stage type's references from already emitted CU bodies would be patched to proper offsets inside artificial CU).

this scheme does not need to have two passes and it does not need to load all CUs into the memory at the same time.

(as for type merging - that might also be possible with the scheme I'm proposing - if we're rewriting DIEs anyway, seems plausible we could add new child DIEs to canonical type type DIEs)

That would be one more reason to keep CUs in memory. When we rewrite a canonical DIE in some CU we would need to gather all children from all other CUs.

The artificial CU(from the scheme with separate CU keeping types) would have merged type so it does not require to keep source CU.

[avl]

One idea is for each type we want to unique, each CU as it is being parsed on its own thread creates a type map that maps "type compile unit path" (which is DW_AT_decl_file of that type) to a list of uniqued types with decl context info. This would allow all types with the same DW_AT_decl_file to be stored in the same type compile unit. As we emit the debug info for a normal compile unit, any type references are changed to be DW_AT_ref_addr references with values that will need to be fixed up. After all normal CUs have been emitted, combine all of the CU specific type maps together, and now emit the type compile units (carefully to include all needed children in each type to make the most complete version of the type possible and emit them) in a sorted order to allow determinism. Then we fix up all DW_AT_ref_addr references in the normal CUs to point to the one type definition from the type compile units.

Sorry if this is restating what has already been said/implied!

yep. That is pretty match the same scenario which I talked about.

The difference is that you propose to create type units(DW_TAG_type_unit) for the types while my suggestion is to use artificial compile unit(that would save space needed by type units headers).

This would allow all types with the same DW_AT_decl_file to be stored in the same type compile unit.

Do you mean DW_TAG_type_unit or DW_TAG_compile_unit here?

It looks like we could not put several types into the same type unit:

"A type unit entry owns debugging information entries that
represent the definition of a single type, plus additional debugging information
entries that may be necessary to include as part of the definition of the type."

Why it would be good to group types on DW_AT_decl_file basis?

[friss]

In D96035#2584686, @avl wrote:

The scheme implemented in this patch and which might be done with additional compile unit keeping types - visit CU only once and then it might be unloaded from the memory:

1. the first pass enumerates in a multithreaded manner all object files, all compile units. Each CU is loaded, analyzed for types(there would be created a list of attribute referencing types), emitted. types moved into the artificial CU. unloaded.

How would you guarantee that the artificial CU is always emitted in the same order? It seems like you need to stash the types somewhere, then create a deterministic ordering and then emit it, but I don't remember whether that's doable with LLVM's DIEs (It's been a while since I touched this code).
Conceptually, I think the idea of moving types to a "central" location is valid, it seems semantically equivalent to the kind of debug info we would get with -gmodules. Another kind of approach would be to gather enough information to create a dependency graph of CUs and then orchestrate parallel processing according to this. Even if this would limit the parallelism at the beginning of a link, I think it's very hard to intuitively figure out whether it would perform worse than the approach above.

[avl]

How would you guarantee that the artificial CU is always emitted in the same order? It seems like you need to stash the types somewhere, then create a deterministic ordering and then emit it, but I don't remember whether that's doable with LLVM's DIEs (It's been a while since I touched this code).

We might sort types on name basis inside that artificial CU. In this case it would always be emitted in the same order.

[friss]

In D96035#2585324, @avl wrote:

How would you guarantee that the artificial CU is always emitted in the same order? It seems like you need to stash the types somewhere, then create a deterministic ordering and then emit it, but I don't remember whether that's doable with LLVM's DIEs (It's been a while since I touched this code).

We might sort types on name basis inside that artificial CU. In this case it would always be emitted in the same order.

I don't think finding how to sort them is the complicated part. In my mind the issue is keeping them around until you can sort them as I'm not sure DIEs can exist not attached to a CU. Also, it's not just the one type, but the transitive closure of all its dependencies that you need to consider which makes things (much) more complicated. As I said before, I haven't worked on this in a long time, so feel free to shoot my concerns down!

[clayborg]

In D96035#2585694, @friss wrote:

In D96035#2585324, @avl wrote:

How would you guarantee that the artificial CU is always emitted in the same order? It seems like you need to stash the types somewhere, then create a deterministic ordering and then emit it, but I don't remember whether that's doable with LLVM's DIEs (It's been a while since I touched this code).

We might sort types on name basis inside that artificial CU. In this case it would always be emitted in the same order.

I don't think finding how to sort them is the complicated part. In my mind the issue is keeping them around until you can sort them as I'm not sure DIEs can exist not attached to a CU. Also, it's not just the one type, but the transitive closure of all its dependencies that you need to consider which makes things (much) more complicated. As I said before, I haven't worked on this in a long time, so feel free to shoot my concerns down!

I agree with Fred in that it might be too expensive to keep all of the CUs around so that we can later copy type data out of them at the end into uniqued compile units.

For testing DWARF, a while back I created a DWARF generator set of classes:

llvm/unittests/DebugInfo/DWARF/DwarfGenerator.h
llvm/unittests/DebugInfo/DWARF/DwarfGenerator.cpp

This allows the creation of stand alone DWARF using a new set of DWARF classes:

llvm::dwarfgen::DIE
llvm::dwarfgen::CompileUnit
llvm::dwarfgen::Generator

It might allow us to copy the types into a llvm::dwarfgen::Generator instance and might easy the ability to add and remove DIEs to parent DIEs, like adding new methods to classes and slowly adding new types a compile unit DIE. Check the test files in llvm/unittests/DebugInfo/DWARF that include the DwarfGenerator.h file to see examples of tests that create DWARF using this generator. The nice thing is all references to other DIEs are actual pointers to llvm::dwarfgen::DIE objects, so you can insert and remove DIEs as needed to anything within a llvm::dwarfgen::CompileUnit, even across llvm::dwarfgen::CompileUnit objects if needed, and when you generate the DWARF, it all gets finalized at that point. So maybe we can use the DWARF generator for all types. When each compile unit is done being parsed on its thread, it can enter a type copying phase where it makes sure all of the uniqued types it needs are copied into a single smart class that uses the DWARF generator. Then we can unload the compile units from the .o files right away when we are done.

[dblaikie]

In D96035#2584686, @avl wrote:

Comparing your proposal - avoiding nondeterminism by sorting the types by name in the new type-holding CU, we could do something similar, but instead sorting the CUs a type appears in to pick which of those existing locations to be the canonical home. (or, indeed, could sort by the original linear visitation order)
eg: multithreaded walk over each CU and find each type (this applies to both your proposal and what I'm suggesting here, I think) - then, rather than sorting by name and putting them in a new CU, sort by the "index" of the CU the type appeared in (where the index is the order the CUs would've been visited in in the current linear algorithm) then place/preserve the canonical type in the CU with the lowest index where the type appears?
Then the second pass goes and emits the CUs, consulting this list of type homes to determine whether this type should be emitted in the CU, or reference a copy emitted elsewhere.

My understanding is that this way assumes all CUs should be loaded into the memory and it does extra pass. i.e.

Sounds like your proposal would require that too - or require reloading the CUs twice? I think either strategy (keeping them loaded, or reloading them) could be used for either of our proposed directions (creating a separate unit, or using the existing units)?

1. the first pass enumerates in a multithreaded manner all object files, all compile units, and creates an indexed map(the list of type homes). (In the result all CUs from all object files are loaded into the memory at the same time. The indexed map is also in the memory).

2. the second pass enumerates in a multithreaded manner all object files, all compile units, and emits bodies(consulting the list of type homes to determine whether this type should be emitted in the CU, or reference a copy emitted elsewhere).

3. Patch sizes/offsets/references after individual CU bodies are glued into the resulting file.

The scheme implemented in this patch and which might be done with additional compile unit keeping types - visit CU only once and then it might be unloaded from the memory:

1. the first pass enumerates in a multithreaded manner all object files, all compile units. Each CU is loaded, analyzed for types(there would be created a list of attribute referencing types), emitted. types moved into the artificial CU. unloaded.

"types moved into the artificial CU" means what exactly? Copying them type DIE tree into some intermediate, in-memory representation?

Ah, yes, though keeping them in memory may be expensive - it might be cheaper to rewalk the units/know the offsets to the DIEs and reload them in some way.

2. Emit artificial CU(After the first pass is finished all CU are unloaded from the memory, except the artificial one).

3. Patch sizes/offsets/references after individual CU bodies are glued into the resulting file. (At this stage type's references from already emitted CU bodies would be patched to proper offsets inside artificial CU).

this scheme does not need to have two passes and it does not need to load all CUs into the memory at the same time.

(as for type merging - that might also be possible with the scheme I'm proposing - if we're rewriting DIEs anyway, seems plausible we could add new child DIEs to canonical type type DIEs)

That would be one more reason to keep CUs in memory. When we rewrite a canonical DIE in some CU we would need to gather all children from all other CUs.

The artificial CU(from the scheme with separate CU keeping types) would have merged type so it does not require to keep source CU.

Yes, if we're merging types it becomes somewhat more complicated - again, could do a two pass (without necessarily keeping all CUs in memory) - go quickly skim the CUs, for each type, record the type name and member list - then merge those lists, keeping the type record for the lowest indexed CU, and a merged member list that mentions which offset/CU each member comes from. This is essentially the same data you'd have to keep in memory (either fully in memory, or some kind of skeleton data that refers to the CUs/input file itself, so it can be reparsed as needed) for the "emit a separate unit full of types at the end" approach.

And in any case, you couldn't emit the units in parallel in the first pass, because you wouldn't know which offsets to write them to, right? (because the size of units will be changing during this process)

So I'm not sure where the parallelism comes into your scheme (& even in my scheme it'd be a bit non-trivial - I guess you'd have to record all the DIE references in each unit that might become cross-unit references (so you know you'll have to change their encoded size) and all the types (so you know whether that'll get bigger (if members are merged in) or smaller (if the type is removed in favor of referencing a type in another unit) - not sure there's a huge difference in performance/complexity between the two, perhaps.

[avl]

I agree with Fred in that it might be too expensive to keep all of the CUs around so that we can later copy type data out of them at the end into uniqued compile units.

It might allow us to copy the types into a llvm::dwarfgen::Generator instance and might easy the ability to add and remove DIEs to parent DIEs, like adding new methods to classes and slowly adding new types a compile unit DIE.

When each compile unit is done being parsed on its thread, it can enter a type copying phase where it makes sure all of the uniqued types it needs are copied into a single smart class that uses the DWARF generator. Then we can unload the compile units from the .o files right away when we are done.

Agreed. That is exactly solution which I am talking about. I did not examined whether llvm::dwarfgen::Generator is suitable for that yet but I would be happy to use it if it is.

[avl]

Sounds like your proposal would require that too - or require reloading the CUs twice? I think either strategy (keeping them loaded, or reloading them) could be used for either of our proposed directions (creating a separate unit, or using the existing units)?

No. My solution does not require to keep all CUs in memory or to reload them twice(for the purpose of ODR types deduplication).

It loads CU, analyzes types, creates list of type references, removes types. Emit DWARF for that CU. unload CU.

It does not need to load that CU again or keep input DWARF of that CU in the memory.
It only needs to fix-up remembered type references in generated DWARF after all CUs are processed and artificial CU is built.
It keeps all types in memory but that requires less space than all CUs.

"types moved into the artificial CU" means what exactly? Copying them type DIE tree into some intermediate, in-memory representation?

yes.

Ah, yes, though keeping them in memory may be expensive - it might be cheaper to rewalk the units/know the offsets to the DIEs and reload them in some way.

I think it would not be cheaper to load CU from the disk again. But we can do experiment and select the more effective solution. i.e. solution with artificial CU allows us to have a choice.

If we would implement solution which load all CUs in memory or reload them from disk - then we would not have a choice.

Yes, if we're merging types it becomes somewhat more complicated - again, could do a two pass (without necessarily keeping all CUs in memory) - go quickly skim the CUs, for each type, record the type name and member list - then merge those lists, keeping the type record for the lowest indexed CU, and a merged member list that mentions which offset/CU each member comes from. This is essentially the same data you'd have to keep in memory (either fully in memory, or some kind of skeleton data that refers to the CUs/input file itself, so it can be reparsed as needed) for the "emit a separate unit full of types at the end" approach.
And in any case, you couldn't emit the units in parallel in the first pass, because you wouldn't know which offsets to write them to, right? (because the size of units will be changing during this process)

Not exactly right. This prototype uses exactly the scheme with one pass(which loads/parse/handle/unload CU only once). It processes units in parallel, generates resulting DWARF and the list of references which should be patched. After all CUs are processed and final sizes are known. It has additional pass which writes correct offsets. That additional pass is _much_ cheaper than load/parse/handle/unload CUs again.

So I'm not sure where the parallelism comes into your scheme (& even in my scheme it'd be a bit non-trivial - I guess you'd have to record all the DIE references in each unit that might become cross-unit references (so you know you'll have to change their encoded size) and all the types (so you know whether that'll get bigger (if members are merged in) or smaller (if the type is removed in favor of referencing a type in another unit) - not sure there's a huge difference in performance/complexity between the two, perhaps.

right. That is the scheme(remember cross-unit references and patch them later) is used by this patch(except that it does not do types merging and does not have separate CU keeping types). This scheme allows to process CU separately, in parallel. It gives us more than 2x performance improvement. It also might be improved more(theoretically, If threads inter-dependency would be decreased).

[dblaikie]

In D96035#2587154, @avl wrote:

Sounds like your proposal would require that too - or require reloading the CUs twice? I think either strategy (keeping them loaded, or reloading them) could be used for either of our proposed directions (creating a separate unit, or using the existing units)?

No. My solution does not require to keep all CUs in memory or to reload them twice(for the purpose of ODR types deduplication).

It loads CU, analyzes types, creates list of type references, removes types. Emit DWARF for that CU. unload CU.

It does not need to load that CU again or keep input DWARF of that CU in the memory.
It only needs to fix-up remembered type references in generated DWARF after all CUs are processed and artificial CU is built.
It keeps all types in memory but that requires less space than all CUs.

Yeah, having to keep the types in memory is the bit I'm getting at - but yes, it's not all CUs. If they're being unloaded/reloaded though, it might be less clear whether it's so costly to preserve the existing output behavior. Though I don't personally have a problem with creating a synthetic/arbitrary CU to put all the types in either. But other folks (Apple folks with a vested interest in the dsymutil behavior) might disagree.

"types moved into the artificial CU" means what exactly? Copying them type DIE tree into some intermediate, in-memory representation?

yes.

Ah, yes, though keeping them in memory may be expensive - it might be cheaper to rewalk the units/know the offsets to the DIEs and reload them in some way.

I think it would not be cheaper to load CU from the disk again. But we can do experiment and select the more effective solution. i.e. solution with artificial CU allows us to have a choice.

If we would implement solution which load all CUs in memory or reload them from disk - then we would not have a choice.

Yes, if we're merging types it becomes somewhat more complicated - again, could do a two pass (without necessarily keeping all CUs in memory) - go quickly skim the CUs, for each type, record the type name and member list - then merge those lists, keeping the type record for the lowest indexed CU, and a merged member list that mentions which offset/CU each member comes from. This is essentially the same data you'd have to keep in memory (either fully in memory, or some kind of skeleton data that refers to the CUs/input file itself, so it can be reparsed as needed) for the "emit a separate unit full of types at the end" approach.
And in any case, you couldn't emit the units in parallel in the first pass, because you wouldn't know which offsets to write them to, right? (because the size of units will be changing during this process)

Not exactly right. This prototype uses exactly the scheme with one pass(which loads/parse/handle/unload CU only once). It processes units in parallel, generates resulting DWARF

By "generating resulting DWARF" I guess that has to be buffered in memory? (how could two CUs be written to the final output file at the same time? You wouldn't know where to write to because you wouldn't know how big the previous unit would be if you haven't finished processing it)

[avl]

Yeah, having to keep the types in memory is the bit I'm getting at - but yes, it's not all CUs. If they're being unloaded/reloaded though, it might be less clear whether it's so costly to preserve the existing output behavior. Though I don't personally have a problem with creating a synthetic/arbitrary CU to put all the types in either. But other folks (Apple folks with a vested interest in the dsymutil behavior) might disagree.

I see, thanks. I tend to implement such a behavior (creating a synthetic/arbitrary CU to put all the types in either) since I believe that would be the most effective way.
(the practice might show if it is). It would be good to know whether that solution is acceptable/unacceptable before starting implementation. @JDevlieghere @aprantl @friss Would you find "implementation moving types into separate synthetic/arbitrary CU" useful and worth integrating into upstream dsymutil?(if it would show good performance/size results being implemented)

By "generating resulting DWARF" I guess that has to be buffered in memory? (how could two CUs be written to the final output file at the same time? You wouldn't know where to write to because you wouldn't know how big the previous unit would be if you haven't finished processing it)

Right. The DWARF for each CU is buffered in memory. And after all CUs are processed - They are glued together into the final full resulting DWARF file. That is exactly how this implementation works.

[dblaikie]

By "generating resulting DWARF" I guess that has to be buffered in memory? (how could two CUs be written to the final output file at the same time? You wouldn't know where to write to because you wouldn't know how big the previous unit would be if you haven't finished processing it)

Right. The DWARF for each CU is buffered in memory. And after all CUs are processed - They are glued together into the final full resulting DWARF file. That is exactly how this implementation works.

That seems quite expensive :s

"how this implementation works" - the patch in this review? The existing checked-in implementation? (apologies as I'm not especially familiar with the implementation of dsymutil as it is today)

[avl]

That seems quite expensive :s

yep, it requires additional space. Though it allows to have benefits:

• it allows separate parallel processing(this patch has more than 2x performance improvement).
• plain DWARF requires less space than parsed/generated(DWARFDie/DIE).

"how this implementation works" - the patch in this review? The existing checked-in implementation? (apologies as I'm not especially familiar with the implementation of dsymutil as it is today)

yes, the patch in this review.

[avl]

Changes from the previous version of the patch:

1. It produces predictable results when working in multi-thread mode.
2. It does types merging. Types merging allows reducing the size of .debug_info table by 40% less in size.

[avl]

@JDevlieghere @aprantl @dblaikie @echristo @clayborg @friss

Would you mind taking a look at this patch, please? Changes from the first version of the patch:

1. It produces predictable results when working in multi-thread mode.
2. It does types merging. Types merging allows reducing the size of .debug_info table by 40% less in size.

[avl]

Example of dwarf output with type merging(created from odr-uniquing.cpp dsymutil lit test) : https://reviews.llvm.org/P8270

[avl]

@dblaikie I copied your message from the another review.

(I'm not especially invested in DWARFLinker, since it's mostly for MachO dsymutil linking (though generalizing it to work for dwp could be interesting, might be an alternative solution (rather than going to more like gold-dwp) to addressing some overhead/scalability issues in llvm-dwp))

The type merging approach sounds OK to me, broadly speaking.
The overhead at low parallelism seems unfortunate - can the algorithm be made more amenable to single threaded performance as well? (is there some design tradeoff that could happen at low >parallelism that'd keep mostly the same approach? Or if we want low-parallelism performance are we going to end up maintaining the two different approaches entirely?)
What's the explanation for the difference/improvement in total output size with type uniquing enabled? I guess existing dsymutil is only structural, not odr-based uniquing or something?

[avl]

(I'm not especially invested in DWARFLinker, since it's mostly for MachO dsymutil linking (though generalizing it to work for dwp could be interesting, might be an alternative solution (rather than going to more like gold-dwp) to addressing some overhead/scalability issues in llvm-dwp))

The type merging approach sounds OK to me, broadly speaking.
The overhead at low parallelism seems unfortunate - can the algorithm be made more amenable to single threaded performance as well?

yes it can. I think that performance might be improved for 1 thread case at least to 1.5X of upstream dsymutil. May be better.

(is there some design tradeoff that could happen at low >parallelism that'd keep mostly the same approach? Or if we want low-parallelism performance are we going to end up maintaining the two different approaches entirely?)

Types merging plus deterministic output require doing more work than current dsymutil solution. That is the reason why 1-2 threads version works slower. But there is a set of things which might improve performance.

If deterministic output is the requirement then we would not be able to support two different approaches since they would be binary incompatible.

What's the explanation for the difference/improvement in total output size with type uniquing enabled? I guess existing dsymutil is only structural, not odr-based uniquing or something?

Upstream dsymutil solution could not deduplicate partially defined types. Some type class declarations might omit member functions in various different compile units. Upstream dsymutil does not change them and leave inplace. Types merging solution is able to remove all such copies and to have only one class declaration containing all member functions. That allows to save space.

[dblaikie]

In D96035#2932140, @avl wrote:

(I'm not especially invested in DWARFLinker, since it's mostly for MachO dsymutil linking (though generalizing it to work for dwp could be interesting, might be an alternative solution (rather than going to more like gold-dwp) to addressing some overhead/scalability issues in llvm-dwp))

The type merging approach sounds OK to me, broadly speaking.
The overhead at low parallelism seems unfortunate - can the algorithm be made more amenable to single threaded performance as well?

yes it can. I think that performance might be improved for 1 thread case at least to 1.5X of upstream dsymutil. May be better.

(is there some design tradeoff that could happen at low >parallelism that'd keep mostly the same approach? Or if we want low-parallelism performance are we going to end up maintaining the two different approaches entirely?)

Types merging plus deterministic output require doing more work than current dsymutil solution. That is the reason why 1-2 threads version works slower. But there is a set of things which might improve performance.

Fair enough - maybe there's room for the overhead to be low enough that it's worth the extra benefits of smaller output, such that it's not worth keeping the old implementation (ie: its time/space tradeoff isn't so interesting to be worth keeping). That'll be up to the Apple folks, pretty much.

If deterministic output is the requirement then we would not be able to support two different approaches since they would be binary incompatible.

So long as the different behaviors aren't chosen organically (based on number of cores available, etc), but instead by a flag - that would be OK. The goal is to be deterministic based on inputs/command line - but different command lines can produce different behaviors.

What's the explanation for the difference/improvement in total output size with type uniquing enabled? I guess existing dsymutil is only structural, not odr-based uniquing or something?

Upstream dsymutil solution could not deduplicate partially defined types. Some type class declarations might omit member functions in various different compile units. Upstream dsymutil does not change them and leave inplace. Types merging solution is able to remove all such copies and to have only one class declaration containing all member functions. That allows to save space.

Yeah, figured something like that - thanks for explaining!

[avl]

@JDevlieghere @aprantl @clayborg @friss @echristo

Folks, What is your opinion on this patch? Would it be useful to integrate?

The patch implements two features:

1. Types deduplication by merging. It merges all partial type definitions/declarations into a single description to avoid types duplications. This approach allows the generation .debug_info table of 50% less in size.

2. Parallel execution. It allows speed-up execution if the hardware has available threads.

Current size/performance results are(compared with llvm upstream dsymutil):

• .debug_info table of 40% less in size.
• single-threaded mode works 1.7x slower.
• multy-thread mode works up to 2x faster.

There is a number of things which might improve the result:

1. Deduplicate types defined in DW_TAG_subprograms with DW_AT_specification attribute.

2. (probably)Deduplicate abstract inline instances.

3. Use another memory pool for dies. Instead of BumpPtrAllocator use a pre-allocated pool. It would reduce data fragmentation.

4. Current version of the patch uses two different memory pools(because of interfaces dependencies) with duplicated data: MTSafeStringTable and NonRelocatableStringpool. Using only one kind of pool would help to reduce memory usage and to improve performance numbers.

5. Improve Dies navigation - https://reviews.llvm.org/D102634#inline-971772.

6. Generate index tables in parallel.

7. Avoid gluing generated tables into a single file. Use tables, generated for compilation units, as the output. It would allow avoiding data copying. Consumers might write compilation units tables directly so we do not need to create an intermediate output file.

So, I think we might have(after above is done) a single-threaded performance to be 1.3-1.5 slower than current dsymutil, multi-threaded performance up to 3x faster.

[dblaikie]

In D96035#2985299, @avl wrote:

@JDevlieghere @aprantl @clayborg @friss @echristo

Yeah, especially @aprantl & @JDevlieghere

Folks, What is your opinion on this patch? Would it be useful to integrate?

The patch implements two features:

1. Types deduplication by merging. It merges all partial type definitions/declarations into a single description to avoid types duplications. This approach allows the generation .debug_info table of 50% less in size.

2. Parallel execution. It allows speed-up execution if the hardware has available threads.

Current size/performance results are(compared with llvm upstream dsymutil):

• .debug_info table of 40% less in size.

Out of curiosity - what's the total size of the clang dsym file with/without this patch/feature?

• single-threaded mode works 1.7x slower.
• multy-thread mode works up to 2x faster.

With how many threads? (& what sort of setup are the speed measurements done with - writing to (& maybe reading from) a fast SSD or temporary/RAM filesystem?)

There is a number of things which might improve the result:

1. Deduplicate types defined in DW_TAG_subprograms with DW_AT_specification attribute.

2. (probably)Deduplicate abstract inline instances.

3. Use another memory pool for dies. Instead of BumpPtrAllocator use a pre-allocated pool. It would reduce data fragmentation.

4. Current version of the patch uses two different memory pools(because of interfaces dependencies) with duplicated data: MTSafeStringTable and NonRelocatableStringpool. Using only one kind of pool would help to reduce memory usage and to improve performance numbers.

5. Improve Dies navigation - https://reviews.llvm.org/D102634#inline-971772.

6. Generate index tables in parallel.

7. Avoid gluing generated tables into a single file. Use tables, generated for compilation units, as the output. It would allow avoiding data copying. Consumers might write compilation units tables directly so we do not need to create an intermediate output file.

So, I think we might have(after above is done) a single-threaded performance to be 1.3-1.5 slower than current dsymutil, multi-threaded performance up to 3x faster.

It may be necessary to lower the overhead further, though I'm not sure.

Alternatively: perhaps the new architecture could present a different tradeoff: what's the performance of parallelism (on a fast storage) without any deduplication? Can it run faster but produce a larger output (or is the output such a bottleneck that that isn't the case)?

[avl]

Current size/performance results are(compared with llvm upstream dsymutil):

    .debug_info table of 40% less in size.

Out of curiosity - what's the total size of the clang dsym file with/without this patch/feature?

dsymutil --use-dlnext clang
ls -lh clang.dSYM/Contents/Resources/DWARF/clang
758M

dsymutil clang
ls -lh clang.dSYM/Contents/Resources/DWARF/clang
955M

i.e. total size of the clang dsym file become 20% smaller with --use-dlnext option.

single-threaded mode works 1.7x slower.
multy-thread mode works up to 2x faster.

With how many threads? (& what sort of setup are the speed measurements done with - writing to (& maybe reading from) a fast SSD or temporary/RAM filesystem?)

Measurements were done on Darwin 24-core 64G system using regular disk(not SSD/ not temporary/RAM filesystem). ~2x improvement is for 16 cores.

|----------------------------------------------------------------------
|       |           dsymutil           |     dsymutil --use-dlnext    |
|-------|------------------------------|------------------------------|
|       |exec time|  memory  | DWARF(*)|exec time|  memory  |  DWARF  |
|       |   sec   |    GB    |    MB   |   sec   |    GB    |   MB    |
|-------|------------------------------|------------------------------|
|threads|         |          |         |         |          |         |
|-------|------------------------------|------------------------------|
|   1   |   155   |   15.8   |   465   |   269   |   16.1   |   273   |
|-------|------------------------------|------------------------------|
|   2   |    99   |   17.5   |   465   |   154   |   16.1   |   273   |
|-------|------------------------------|------------------------------|
|   4   |    99   |   17.5   |   465   |    96   |   16.5   |   273   |
|-------|------------------------------|------------------------------|
|   8   |    99   |   17.5   |   465   |    65   |   16.5   |   273   |
|-------|------------------------------|------------------------------|
|  16   |    99   |   17.5   |   465   |    52   |   16.5   |   273   |
|---------------------------------------------------------------------|

Alternatively: perhaps the new architecture could present a different tradeoff: what's the performance of parallelism (on a fast storage) without any deduplication? Can it run faster but produce a >larger output (or is the output such a bottleneck that that isn't the case)?

For the case when type deduplication is not done the performance numbers(for regular storage) look better(if compared with no ODR case, the performance improvement seen starting from 2 cores):

|----------------------------------------------------------------------
|       |           dsymutil           |     dsymutil --use-dlnext    |
|-------|------------------------------|------------------------------|
|       |exec time|  memory  |  DWARF  |exec time|  memory  |  DWARF  |
|       |   sec   |    GB    |    MB   |   sec   |    GB    |   MB    |
|-------|------------------------------|------------------------------|
|threads|         |          |         |         |          |         |
|-------|------------------------------|------------------------------|
|   1   |   224   |   15.9   |   1400  |   250   |   19.5   |   1400  |
|-------|------------------------------|------------------------------|
|   2   |   214   |   17.7   |   1400  |   144   |   19.5   |   1400  |
|-------|------------------------------|------------------------------|
|   4   |   214   |   17.7   |   1400  |    90   |   19.5   |   1400  |
|-------|------------------------------|------------------------------|
|   8   |   214   |   17.7   |   1400  |    62   |    20    |   1400  |
|-------|------------------------------|------------------------------|
|  16   |   214   |   17.7   |   1400  |    51   |    20    |   1400  |
|---------------------------------------------------------------------|

output is not a bottleneck here. The execution time directly depends on the amount of source DWARF which should be analyzed/cloned(The upstream dsymutil in ODR deduplication mode skips analyzing/clonning for some dies thus it works faster. When it does not skip types, like in no ODR case, it has similar single thread performance).

Anyway, there is additional set of things which might improve performance/memory requirements/output size for all modes(single-thread/multi-thread/ODR/no-ODR).

[clayborg]

In D96035#2985299, @avl wrote:

@JDevlieghere @aprantl @clayborg @friss @echristo

Folks, What is your opinion on this patch? Would it be useful to integrate?

It is very compelling! I do think it would be useful to integrate if there are perf and size wins as you are finding.

The patch implements two features:

1. Types deduplication by merging. It merges all partial type definitions/declarations into a single description to avoid types duplications. This approach allows the generation .debug_info table of 50% less in size.

Very nice! I will apply this patch and do some testing. This can help eliminate issues we have in LLDB when you have many copies of a type, and any compiler generated methods (constructors, copy constructors, destructors, etc) are only defined in some of the copies of the type. Were you able to find all such methods and merge them in? I am guessing you didn't end up using the DWARFGenerator for this?

2. Parallel execution. It allows speed-up execution if the hardware has available threads.

We have this in normal dsymutil right now right? You have just made it faster? Or made it work with the new approach you have in this patch?

Current size/performance results are(compared with llvm upstream dsymutil):

• .debug_info table of 40% less in size.

That is great! Anything we can do to reduce the dups in DWARF even more than what we already had in dsymutil is great. I know there were a bunch of things that could cause types to be emitted multiple times in the current dsymutil that were not allowing us to reduce the DWARF size as much as we would have liked to.

• single-threaded mode works 1.7x slower.
• multy-thread mode works up to 2x faster.

Nice.

There is a number of things which might improve the result:

1. Deduplicate types defined in DW_TAG_subprograms with DW_AT_specification attribute.

2. (probably)Deduplicate abstract inline instances.

3. Use another memory pool for dies. Instead of BumpPtrAllocator use a pre-allocated pool. It would reduce data fragmentation.

4. Current version of the patch uses two different memory pools(because of interfaces dependencies) with duplicated data: MTSafeStringTable and NonRelocatableStringpool. Using only one kind of pool would help to reduce memory usage and to improve performance numbers.

5. Improve Dies navigation - https://reviews.llvm.org/D102634#inline-971772.

6. Generate index tables in parallel.

7. Avoid gluing generated tables into a single file. Use tables, generated for compilation units, as the output. It would allow avoiding data copying. Consumers might write compilation units tables directly so we do not need to create an intermediate output file.

So, I think we might have(after above is done) a single-threaded performance to be 1.3-1.5 slower than current dsymutil, multi-threaded performance up to 3x faster.

You say 3x here and 2x above. If it is faster, then all is good. By default dsymutil runs with multiple threads, so the single threaded performance doesn't matter to me.

[avl]

The patch implements two features:

    Types deduplication by merging. It merges all partial type definitions/declarations into a single description to avoid types duplications. This approach allows the generation .debug_info table of 50% less in size.

Very nice! I will apply this patch and do some testing. This can help eliminate issues we have in LLDB when you have many copies of a type, and any compiler generated methods (constructors, copy >constructors, destructors, etc) are only defined in some of the copies of the type. Were you able to find all such methods and merge them in?

yep, all such methods should be merged into the single type description(as soon as method name was properly identified - it either should have explicit name(DW_AT_name, DW_AT_linkage_name), either it should have implicit unique identifiers(DW_AT_decl_file, DW_AT_decl_line)).

I am guessing you didn't end up using the DWARFGenerator for this?

I did not use DWARFGenerator because it heavily depends on AsmPrinter, MC/* classes. It might be reused though, If "DIE tree building" part would be separated from "writing output" part.
Implementation of "DIE tree building" part in this patch uses the same idea as DWARFGenerator.

Another thing is that DWARFGenerator is not ready to work in multi-thread mode(i.e. to build dies tree in parallel).

DWARFGenerator might be refactored so that above problems would be resolved. In that case this patch would be able to use DWARFGenerator.

Parallel execution. It allows speed-up execution if the hardware has available threads.

We have this in normal dsymutil right now right? You have just made it faster? Or made it work with the new approach you have in this patch?

This patch uses the new approach. Current upstream dsymutil is able to utilize only two threads. It processes compile units sequentially, but is able to analyze compile unit and clone previous compile unit in parallel.

This patch is able to utilize all available threads. It is able to process each compile unit separately and in parallel.
Thus, it might perform better when available threads exist because more threads are used.

You say 3x here and 2x above. If it is faster, then all is good. By default dsymutil runs with multiple threads, so the single threaded performance doesn't matter to me.

2x - is the performance result for this patch for 16 threads.
3x - is my estimation for this patch plus set of improvements(mentioned above) for 16 threads.

[avl]

yep, all such methods should be merged into the single type description(as soon as method name was properly identified - it either should have explicit name(DW_AT_name, DW_AT_linkage_name), >either it should have implicit unique identifiers(DW_AT_decl_file, DW_AT_decl_line)).

There exist restriction currently - This patch does not de-duplicate types from DW_TAG_subprograms with DW_AT_specification attribute. It might be done and will improve size/performance numbers.

[JDevlieghere]

In D96035#2931137, @avl wrote:

@JDevlieghere @aprantl @dblaikie @echristo @clayborg @friss

Would you mind taking a look at this patch, please? Changes from the first version of the patch:

Disclaimer: I haven't had a chance to look at the code itself yet, but I have a few high level questions.

1. It produces predictable results when working in multi-thread mode.

This means that the generated DWARF is identical regardless of the number of threads, right?

2. It does types merging. Types merging allows reducing the size of .debug_info table by 40% less in size.

Very exciting!

My main concern at this point is how you are planning to qualify this change. I see you copied the existing tests (which I assume was necessary because you had to modify them to match the newly generated output) which is a great start. The LLDB test suite runs most tests with a dSYM and another good test would be to run it with the --use-dlnext flag. However, both of these are still pretty artificial test cases. Anecdotally, we often find bugs in large projects that don't show up in smaller ones, and semantically comparing the DWARF by hand is basically impossible. When we re-implemented dsymutil on top of llvm, we compared the hash of the generated DWARF to make sure it was identical (bugs in the original implementation included). I'm curious what your thoughts are on this.

[avl]

It produces predictable results when working in multi-thread mode.

This means that the generated DWARF is identical regardless of the number of threads, right?

Right. Generated DWARF is identical regardless of the number of threads.

My main concern at this point is how you are planning to qualify this change. I see you copied the existing tests (which I assume was necessary because you had to modify them to match the newly generated output) which is a great start. The LLDB test suite runs most tests with a dSYM and another good test would be to run it with the --use-dlnext flag. However, both of these are still pretty artificial test cases. Anecdotally, we often find bugs in large projects that don't show up in smaller ones, and semantically comparing the DWARF by hand is basically impossible. When we re-implemented dsymutil on top of llvm, we compared the hash of the generated DWARF to make sure it was identical (bugs in the original implementation included). I'm curious what your thoughts are on this.

yeah, approach of comparing output with original implementation is impossible in this case since output has been changed radically.

I have a thought of implementing a set of tests using debugger scripts. Something similar to this - https://reviews.llvm.org/D74169#change-ZwZOgdhWYOcz . So that such tests cover all/most aspects of DWARF specification. These tests would not depend on the concrete dsymutil binary format but will show whether debug information is correctly seen by debug info consumer(lldb).

Speaking of bugs found in large projects that don't show up in smaller ones: I think is the only way to create wider testing suite. i.e. if current test suite misses some cases then we need to add them(either upfront, either after the fact). Another possibility is to make llvm-dwarfdump --verify to be able to catch more bugs.

The LLDB test suite runs most tests with a dSYM and another good test would be to run it with the --use-dlnext flag.

Did not do this yet, but will check.

[avl]

rebased.

[avl]

rebased. fixed a couple of minor issues.

[avl]

1. added deduplication of types defined inside subprograms.
2. implemented several performance optimizations.
3. supported "-Xclang -gsimple-template-names=simple".
4. The debug types dump was implemented for llvm-dwarfdump.

[avl]

@JDevlieghere @aprantl @dblaikie @clayborg @friss @echristo

Please consider this new version of the patch. It has several improvements and I think it is in a ready state now.

Changes from the previous version:

1. It deduplicates types defined inside subprograms. This change allows reducing the size of the resulting dwarf even more: .debug_info table is 67% less in size now.

2. Several performance optimizations were implemented. It is 2.1x faster than upstream dsymutil in multi-thread ODR mode(16 threads). It is 1.4x slower in single-thread ODR mode. For non-ODR mode, it is 4.5x faster in multi-thread mode(16 threads) and has the same performance in single-thread mode. See comparison tables 1 and 2.

3. It is compatible with "-Xclang -gsimple-template-names=simple". The previous version recognizes types by their names(utilizing the fact that names contain template parameters). This version takes into account DW_TAG_template_type_parameter and DW_TAG_template_value_parameter. It allows taking advantage created by "-Xclang -gsimple-template-names=simple": smaller size of debug_str section and performance improvement(due to smaller strings). Table 3 compares output results for debug info built with "-Xclang -debug-forward-template-params", processed by current upstream dsymutil and debug info built with "-Xclang -debug-forward-template-params -Xclang -gsimple-template-names=simple", processed by new version from this patch. The overall binary is 37% less in size.

4. Because of point 3, It is now a requirement to build debug info with "-Xclang -debug-forward-template-params". Without specifying "-Xclang -debug-forward-template-params" the output would be correct but it would be non-deterministic.

5. The debug types dump was implemented for llvm-dwarfdump(--odr-compat-dump), which allows comparing results generated by the current upstream dsymutil and the new version from this patch. Dumps, done for debug-info created by dsymutil with/without "--use-dlnext" options, might be compared and should match.

Testing:

1. It passed llvm/test/tools/dsymutil lit tests.
2. It passed check-llvm, check-clang, check-lldb.
3. It passed llvm-dwarfdump --verify for all llvm&clang binaries.
4. It passed llvm-dwarfdump --odr-compat-dump for all llvm&clang binaries:

dsymutil clang 
llvm-dwarfdump --odr-compat-dump clang | md5
27e2af4662bf44adeccf158072e6ebd7
dsymutil --use-dlnext clang
llvm-dwarfdump --odr-compat-dump clang | md5
27e2af4662bf44adeccf158072e6ebd7

5. It produces deterministic output:

dsymutil --use-dlnext --num-threads 1 bugpoint
md5 bugpoint.dSYM/Contents/Resources/DWARF/bugpoint
a1b113c6998a6ca23084e7a9bc929184
dsymutil --use-dlnext --num-threads 2 bugpoint
md5 bugpoint.dSYM/Contents/Resources/DWARF/bugpoint
a1b113c6998a6ca23084e7a9bc929184
dsymutil --use-dlnext --num-threads 3 bugpoint
md5 bugpoint.dSYM/Contents/Resources/DWARF/bugpoint
a1b113c6998a6ca23084e7a9bc929184
dsymutil --use-dlnext --num-threads 7 bugpoint
md5 bugpoint.dSYM/Contents/Resources/DWARF/bugpoint
a1b113c6998a6ca23084e7a9bc929184

There still exists room for improvements. The single-thread/multi-thread performance might be improved, run-time memory requirements might be decreased. I think it would be better to do these additional improvements after main part is integrated.

Performance results for this patch for the clang binary(Darwin 24-core 64G):

Table 1. clang binary, ODR deduplication is ON:

|----------------------------------------------------------------------
|       |           dsymutil           |     dsymutil --use-dlnext    |
|-------|------------------------------|------------------------------|
|       |exec time|  memory  | DWARF(*)|exec time|  memory  |  DWARF  |
|       |   sec   |    GB    |    MB   |   sec   |    GB    |   MB    |
|-------|------------------------------|------------------------------|
|threads|         |          |         |         |          |         |
|-------|------------------------------|------------------------------|
|   1   |   159   |   16.5   |   485   |   220   |   12.6   |   157   |
|-------|------------------------------|------------------------------|
|   2   |    99   |   17.8   |   485   |   129   |   12.6   |   157   |
|-------|------------------------------|------------------------------|
|   4   |    99   |   17.8   |   485   |    83   |   12.6   |   157   |
|-------|------------------------------|------------------------------|
|   8   |    99   |   17.8   |   485   |    57   |   12.9   |   157   |
|-------|------------------------------|------------------------------|
|  16   |    99   |   17.8   |   485   |    47   |   13.1   |   157   |
|---------------------------------------------------------------------|

Table 2. clang binary, ODR deduplication is OFF:

|----------------------------------------------------------------------
|       |           dsymutil           |     dsymutil --use-dlnext    |
|-------|------------------------------|------------------------------|
|       |exec time|  memory  |  DWARF  |exec time|  memory  |  DWARF  |
|       |   sec   |    GB    |    MB   |   sec   |    GB    |   MB    |
|-------|------------------------------|------------------------------|
|threads|         |          |         |         |          |         |
|-------|------------------------------|------------------------------|
|   1   |   224   |   16.2   |   1450  |   224   |   15.0   |   1460  |
|-------|------------------------------|------------------------------|
|   2   |   218   |    18    |   1450  |   131   |   15.7   |   1460  |
|-------|------------------------------|------------------------------|
|   4   |   218   |    18    |   1450  |    83   |   15.9   |   1460  |
|-------|------------------------------|------------------------------|
|   8   |   218   |    18    |   1450  |    57   |   16.3   |   1460  |
|-------|------------------------------|------------------------------|
|  16   |   218   |    18    |   1450  |    47   |   16.6   |   1460  |
|---------------------------------------------------------------------|

Table 3. clang binary, ODR deduplication is ON
(dsymutil+"-Xclang -gsimple-template-names=simple"
ws dsymutil+"-Xclang -debug-forward-template-params -Xclang -gsimple-template-names=simple"):

|----------------------------------------------------------------------
|       |           dsymutil           |     dsymutil --use-dlnext    |
|-------|------------------------------|------------------------------|
|       |exec time|  memory  |DWARF(**)|exec time|  memory  |  DWARF  |
|       |   sec   |    GB    |    MB   |   sec   |    GB    |   MB    |
|-------|------------------------------|------------------------------|
|threads|         |          |         |         |          |         |
|-------|------------------------------|------------------------------|
|   1   |   159   |   16.5   |   485   |   216   |   12.0   |   157   |
|       |         |          |   258   |         |          |   210   |
|-------|------------------------------|------------------------------|
|   2   |    99   |   17.8   |   485   |   126   |   12.1   |   157   |
|       |         |          |   258   |         |          |   210   |
|-------|------------------------------|------------------------------|
|   4   |    99   |   17.8   |   485   |    80   |   12.1   |   157   |
|       |         |          |   258   |         |          |   210   |
|-------|------------------------------|------------------------------|
|   8   |    99   |   17.8   |   485   |    55   |   12.3   |   157   |
|       |         |          |   258   |         |          |   210   |
|-------|------------------------------|------------------------------|
|  16   |    99   |   17.8   |   485   |    44   |   12.4   |   157   |
|       |         |          |   258   |         |          |   210   |
|---------------------------------------------------------------------|

(**) DWARF is the size of .debug_info(first) and debug_str(second) section.

[avl]

rebased.

[clayborg]

I am taking at look at the current state of the patch and results this week. I am very interested in this. If anyone has any ideas on validation, please share.

[clayborg]

So I am currently just looking at the content of the DWARF that is produced when using "--use-dlnext". Right now the main issue is it seems that types declared in an anonymous namespace in implementation files are all being moved to the "__type_table" die:

0x0000000b: DW_TAG_compile_unit
              DW_AT_producer	("DWARFLinker lib")
              DW_AT_language	(DW_LANG_C_plus_plus_14)
              DW_AT_name	("__type_table")
              DW_AT_stmt_list	(0x00000000)
              DW_AT_comp_dir	("")

One quick example is LVBase from APValue.cpp line 165:

namespace {
  struct LVBase {
    APValue::LValueBase Base;
    CharUnits Offset;
    unsigned PathLength;
    bool IsNullPtr : 1;
    bool IsOnePastTheEnd : 1;
  };
}

Which produces a type in the "__type_table" compile unit:

0x000000a6:   DW_TAG_namespace
                DW_AT_decl_file	("/Users/gclayton/Documents/src/lldb/clean/Debug/lib/libclangAST.a(APValue.cpp.o)")

0x000000a9:     DW_TAG_structure_type
                  DW_AT_calling_convention	(DW_CC_pass_by_value)
                  DW_AT_name	("LVBase")
                  DW_AT_byte_size	(0x20)
                  DW_AT_decl_file	("/Users/gclayton/Documents/src/lldb/clean/llvm-project/clang/lib/AST/APValue.cpp")
                  DW_AT_decl_line	(166)

0x000000b3:       DW_TAG_subprogram
                    DW_AT_name	("LVBase")
                    DW_AT_declaration	(true)
                    DW_AT_artificial	(true)
                    DW_AT_external	(true)

I would think we should leave these types in the original compile unit as that will be the only compile unit that will use these types. Right now you are moving them to the "__type_table" compile unit so they can be uniqued, but they don't need to be uniqued. This is pretty C++ specific though.

The namespace also has the object file path as the decl file. If we are going to move the type, wouldn't it be better to use the compile unit's path like:

0x000000a6:   DW_TAG_namespace
                DW_AT_decl_file	("/Users/gclayton/Documents/src/lldb/clean/llvm-project/clang/lib/AST/APValue.cpp")

[clayborg]

So an easy detection method to not move types that are defined in an implementation file would be if a type is DW_AT_decl_file matches the DW_TAG_compile_unit source file, then don't move it. If the type's DW_AT_decl_file doesn't match, that put that type into the "__type_table" DW_TAG_compile_unit.

Another way to know if a type should be moved to the "__type_table" would be to detect if the type is defined multiple times, and if it isn't don't move it. If it is, then move it. Not sure how easy that would be to detect with your current codebase though.

[clayborg]

Looking at a few normal CUs after this I see that there are a lot of imported declarations we can probably get rid of if no one references them:

0x04517e06:   DW_TAG_namespace
                DW_AT_name	("std")

0x04517e0b:     DW_TAG_namespace
                  DW_AT_name	("__1")
                  DW_AT_export_symbols	(true)

0x04517e10:       DW_TAG_namespace
                    DW_AT_name	("chrono")

0x04517e15:         DW_TAG_imported_module
                      DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/chrono")
                      DW_AT_decl_line	(2923)
                      DW_AT_import	(0x0000000001b05f36)

0x04517e1d:         NULL

0x04517e1e:       DW_TAG_imported_declaration
                    DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/cstddef")
                    DW_AT_decl_line	(49)
                    DW_AT_import	(0x0000000003fe0299)

0x04517e25:       DW_TAG_imported_declaration
                    DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/cstddef")
                    DW_AT_decl_line	(50)
                    DW_AT_import	(0x0000000003fe0375)

0x04517e2c:       DW_TAG_imported_declaration
                    DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/cstddef")
                    DW_AT_decl_line	(53)
                    DW_AT_import	(0x0000000003fe019b)

0x04517e33:       DW_TAG_imported_declaration
                    DW_AT_decl_file	("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/cstdlib")
                    DW_AT_decl_line	(99)
                    DW_AT_import	(0x0000000003fe0375)

... <many many more>

These imported declarations amounted to 3% of overall CU size for a single compile unit that I tested. I know that the stock dsymutil whitelists all of these which causes these and all the types the reference to be kept, so if we can get rid of any DW_TAG_imported_declaration DIEs that are not referenced, we might also be able to get rid of the type that they reference as well as all types from a C++ header file that have corresponding C header files like "cstdlib" -> "stdlib.h", they will have this issue and waste space.

[clayborg]

One issue I do see with this approach as well is the first "type_table" is about 69MB for a dSYM file for LLDB.framework in LLDB. Any types that are accessed from debug info in LLDB will end up causing this entire DW_TAG_compile_unit to be parsed which can be expensive as LLDB parses DWARF partially and only parses what it needs to when loading DWARF. Is there any way to split up the "type_table" into multiple DW_TAG_compile_unit entries whose DW_AT_name matches the DW_AT_decl_file of the root type that is being uniqued? That would allow smaller compile units to be parsed and keep memory footprint down in debuggers.

[dblaikie]

These imported declarations amounted to 3% of overall CU size for a single compile unit that I tested. I know that the stock dsymutil whitelists all of these which causes these and all the types the reference to be kept, so if we can get rid of any DW_TAG_imported_declaration DIEs that are not referenced, we might also be able to get rid of the type that they reference as well as all types from a C++ header file that have corresponding C header files like "cstdlib" -> "stdlib.h", they will have this issue and waste space.

This is a more fundamental issue with Clang's debug info (I say this as the person who implemented the support for imported declarations in Clang) - Clang doesn't track the use of a using decl (& for a using directive, that brings in a whole namespace, there's no DWARF that would reference that anyway (eg: a variable is of some type - so you have to refer to the type DIE, there's nowhere to mention/reference an imported namespace from that association)) so Clang's debug info describes the type referenced via a using decl in the source without using the using decl in the DWARF:

$ cat decl.cpp
namespace ns {
struct t1 { };
}
using ns::t1;
t1 v1;
$ clang++-tot decl.cpp -g -c && llvm-dwarfdump-tot decl.o | grep "DW_AT_type\|DW_TAG"
0x0000000b: DW_TAG_compile_unit
0x0000001e:   DW_TAG_variable
                DW_AT_type      (0x00000038 "ns::t1")
0x00000033:   DW_TAG_namespace
0x00000038:     DW_TAG_structure_type
0x00000042:   DW_TAG_imported_declaration

So imported modules are never referenced from other DWARF - if you want any of them to persist you have to treat them all as roots, you can't GC them.
And for imported declarations, at least those generated from Clang you have to do the same until that bug is fixed in Clang to actually reference the using declarations.

In any case I assume this issue is mostly orthogonal to this proposal (ie: I expect that's an existing issue in dsymutil, not a new regression introduced by this patch), yeah?

In D96035#3221098, @clayborg wrote:

One issue I do see with this approach as well is the first "type_table" is about 69MB for a dSYM file for LLDB.framework in LLDB. Any types that are accessed from debug info in LLDB will end up causing this entire DW_TAG_compile_unit to be parsed which can be expensive as LLDB parses DWARF partially and only parses what it needs to when loading DWARF. Is there any way to split up the "type_table" into multiple DW_TAG_compile_unit entries whose DW_AT_name matches the DW_AT_decl_file of the root type that is being uniqued? That would allow smaller compile units to be parsed and keep memory footprint down in debuggers.

Splitting them up seems OK to me. Though there's some tradeoff - encodings can be more dense if clusters of types are in the same unit (since they can use shorter unit-local references, rather than full sec_offset references) and th extra unit headers. Probably groups of types in units would be good - a basic heuristic would be X types per unit, but a more advanced one would depend on the size of the types (& even more advanced would group related types together to benefit from more local references - but I doubt that's worth it).

[avl]

In D96035#3221040, @clayborg wrote:

So I am currently just looking at the content of the DWARF that is produced when using "--use-dlnext". Right now the main issue is it seems that types declared in an anonymous namespace in implementation files are all being moved to the "__type_table" die:

0x0000000b: DW_TAG_compile_unit
              DW_AT_producer	("DWARFLinker lib")
              DW_AT_language	(DW_LANG_C_plus_plus_14)
              DW_AT_name	("__type_table")
              DW_AT_stmt_list	(0x00000000)
              DW_AT_comp_dir	("")

One quick example is LVBase from APValue.cpp line 165:

namespace {
  struct LVBase {
    APValue::LValueBase Base;
    CharUnits Offset;
    unsigned PathLength;
    bool IsNullPtr : 1;
    bool IsOnePastTheEnd : 1;
  };
}

Which produces a type in the "__type_table" compile unit:

0x000000a6:   DW_TAG_namespace
                DW_AT_decl_file	("/Users/gclayton/Documents/src/lldb/clean/Debug/lib/libclangAST.a(APValue.cpp.o)")

0x000000a9:     DW_TAG_structure_type
                  DW_AT_calling_convention	(DW_CC_pass_by_value)
                  DW_AT_name	("LVBase")
                  DW_AT_byte_size	(0x20)
                  DW_AT_decl_file	("/Users/gclayton/Documents/src/lldb/clean/llvm-project/clang/lib/AST/APValue.cpp")
                  DW_AT_decl_line	(166)

0x000000b3:       DW_TAG_subprogram
                    DW_AT_name	("LVBase")
                    DW_AT_declaration	(true)
                    DW_AT_artificial	(true)
                    DW_AT_external	(true)

I would think we should leave these types in the original compile unit as that will be the only compile unit that will use these types. Right now you are moving them to the "__type_table" compile unit so they can be uniqued, but they don't need to be uniqued. This is pretty C++ specific though.

The reason why the content of anonymous namespace(as well as types defined inside subprograms) is moved into the type table - is handling parameterized types.
If we have a type defined inside an anonymous namespace and this type is not moved into the type table and this type is used as a parameter of another class then we could not move that another class into the type table also. That another class might be used in other places and have other dependencies. Finally, the type from the anonymous namespace might prevent moving a lot of classes into the type table(current design decision is that type table does not have references into other compile units). Please check the following example:

---

cat main.cpp

char foo1 (void);
long foo2 (void);

int main ( void ) {
  return foo1() + foo2();
}

---

cat a.h

struct A {
  template <class T>
  T foo () {
    return T();
  }
};

---

cat foo1.cpp

#include "a.h"

char foo1 (void) {
  return A().foo<char>(); 
}

---

cat foo2.cpp

#include "a.h"

namespace {

struct B {
  long m1;
};

B internal (void) {
  return A().foo<B>();
}

}

long foo2 (void ) {
  return internal().m1;
}

In dwarf we have a following die :

DW_TAG_structure_type
  DW_AT_calling_convention        (DW_CC_pass_by_value)
  DW_AT_name      ("A")
  DW_AT_byte_size (0x01)
  DW_AT_decl_file ("/home/avl/test_templates1/./a.h")
  DW_AT_decl_line (1)

  DW_TAG_subprogram
    DW_AT_linkage_name    ("_ZN1A3fooIN12_GLOBAL__N_11BEEET_v")
    DW_AT_name    ("foo<(anonymous namespace)::B>")
    DW_AT_decl_file       ("/home/avl/test_templates1/./a.h")
    DW_AT_decl_line       (3)
    DW_AT_type    (0x0000016d "(anonymous namespace)::B")
    DW_AT_declaration     (true)

    DW_TAG_template_type_parameter
      DW_AT_type  (0x0000016d "(anonymous namespace)::B")
      DW_AT_name  ("T")

    DW_TAG_formal_parameter
      DW_AT_type  (0x000001b3 "A *")
      DW_AT_artificial    (true)

    NULL
  NULL

The method "foo<(anonymous namespace)::B>" prevents moving "A"(from foo2.cpp) into the type table.
The "A" structure might contain a lot of data and many other types might depend on "A".
Finally a big piece of DWARF(from NON-anonymous namespaces) would be left in place
and is not de duplicated.

To resolve this problem this patch moves all types into the type table. So that "anonymous"
types would not prevent other types from de duplication.

There probably might be implemented smarter solution, which will be able to "split" type "A" so that all not depending part of it would be moved into the type table but
part related to the types defined in anonymous namespace would left in place. That is more complex solution and is not implemented yet. This patch implements types merging. Types splitting might be done if current solution is not good enough.

The namespace also has the object file path as the decl file. If we are going to move the type, wouldn't it be better to use the compile unit's path like:

0x000000a6:   DW_TAG_namespace
                DW_AT_decl_file	("/Users/gclayton/Documents/src/lldb/clean/llvm-project/clang/lib/AST/APValue.cpp")

yes, it supposed to be compile unit path. will check it.

[jhenderson]

llvm/tools/llvm-dwarfdump/llvm-dwarfdump.cpp

252 ↗

(On Diff #396957)

[dblaikie]

The reason why the content of anonymous namespace(as well as types defined inside subprograms) is moved into the type table - is handling parameterized types.
If we have a type defined inside an anonymous namespace and this type is not moved into the type table and this type is used as a parameter of another class then we could not move that another class into the type table also. That another class might be used in other places and have other dependencies. Finally, the type from the anonymous namespace might prevent moving a lot of classes into the type table(current design decision is that type table does not have references into other compile units). Please check the following example:

You can use cross-unit references (DW_FORM_sec_offset) to have one type/DIE in one unit refer to a type/DIE in another unit. So they could still be moved to/handled in a common way, but split up into multiple units to avoid pathological cases in DWARF consumers that might tend to load whole units at a time. Would that be feasible? (not sure if such consumers also then have common pathologies where they load all units that a unit references via DW_FORM_sec_offset... if they do, then the solution becomes more difficult and would involve either putting every type in a separate unit or finding groups of related types to put together, rather than arbitrary grouping - @clayborg do you know if lldb has that sort of behavior? (where it'd try to load all referenced units from a unit it's loading))

[avl]

In D96035#3223263, @dblaikie wrote:

You can use cross-unit references (DW_FORM_sec_offset) to have one type/DIE in one unit refer to a type/DIE in another unit. So they could still be moved to/handled in a common way, but split up into multiple units to avoid pathological cases in DWARF consumers that might tend to load whole units at a time. Would that be feasible? (not sure if such consumers also then have common pathologies where they load all units that a unit references via DW_FORM_sec_offset... if they do, then the solution becomes more difficult and would involve either putting every type in a separate unit or finding groups of related types to put together, rather than arbitrary grouping - @clayborg do you know if lldb has that sort of behavior? (where it'd try to load all referenced units from a unit it's loading))

If I understood correctly - you are talking here more about splitting the current global type table between several artificial type tables.
Which is an additional problem, but not what I was talking about. I tried to describe situation when a class ("A") contains information for global
namespace and for anonymous namespace:

DW_TAG_structure_type                                               | global&anonymous
  DW_AT_calling_convention        (DW_CC_pass_by_value)             | global&anonymous
  DW_AT_name      ("A")                                             | global&anonymous
  DW_AT_byte_size (0x01)                                            | global&anonymous
  DW_AT_decl_file ("/home/avl/test_templates1/./a.h")               | global&anonymous
  DW_AT_decl_line (1)                                               | global&anonymous

  DW_TAG_subprogram                                                 | anonymous
    DW_AT_linkage_name    ("_ZN1A3fooIN12_GLOBAL__N_11BEEET_v")     | anonymous
    DW_AT_name    ("foo<(anonymous namespace)::B>")                 | anonymous
    DW_AT_decl_file       ("/home/avl/test_templates1/./a.h")       | anonymous
    DW_AT_decl_line       (3)                                       | anonymous
    DW_AT_type    (0x0000016d "(anonymous namespace)::B")           | anonymous
    DW_AT_declaration     (true)                                    | anonymous

    DW_TAG_template_type_parameter                                  | anonymous
      DW_AT_type  (0x0000016d "(anonymous namespace)::B")           | anonymous
      DW_AT_name  ("T")                                             | anonymous

    DW_TAG_formal_parameter                                         | anonymous
      DW_AT_type  (0x000001b3 "A *")                                | anonymous
      DW_AT_artificial    (true)                                    | anonymous

    NULL                                                            | anonymous
    
  DW_TAG_subprogram                                                 | global
    DW_AT_linkage_name    ("another member of A")                   | global
    DW_AT_name    ("another member of A")                           | global
    
  NULL                                                              | global&anonymous

Thus it should be split into two parts. One should go to the global type table:

DW_TAG_structure_type                                               
  DW_AT_calling_convention        (DW_CC_pass_by_value)             
  DW_AT_name      ("A")                                             
  DW_AT_byte_size (0x01)                                           
  DW_AT_decl_file ("/home/avl/test_templates1/./a.h")              
  DW_AT_decl_line (1)                                              

  DW_TAG_subprogram                                                
    DW_AT_linkage_name    ("another member of A")                  
    DW_AT_name    ("another member of A")                          
    
  NULL

Another part should be left into the owning compilation unit.

DW_TAG_structure_type                                             
DW_AT_calling_convention        (DW_CC_pass_by_value)             
DW_AT_name      ("A")                                             
DW_AT_byte_size (0x01)                                            
DW_AT_decl_file ("/home/avl/test_templates1/./a.h")               
DW_AT_decl_line (1)                                               

DW_TAG_subprogram                                                 
  DW_AT_linkage_name    ("_ZN1A3fooIN12_GLOBAL__N_11BEEET_v")     
  DW_AT_name    ("foo<(anonymous namespace)::B>")                 
  DW_AT_decl_file       ("/home/avl/test_templates1/./a.h")       
  DW_AT_decl_line       (3)                                       
  DW_AT_type    (0x0000016d "(anonymous namespace)::B")           
  DW_AT_declaration     (true)                                    

  DW_TAG_template_type_parameter                                  
    DW_AT_type  (0x0000016d "(anonymous namespace)::B")           
    DW_AT_name  ("T")                                             

  DW_TAG_formal_parameter                                         
    DW_AT_type  (0x000001b3 "A *")                                
    DW_AT_artificial    (true)                                    

  NULL                                                            
  
NULL

That kind of splitting is not implemented by current patch.
Also that solution would require more space since it duplicates structure description.

Or are you suggesting another solution for that case?

[avl]

In D96035#3221098, @clayborg wrote:

One issue I do see with this approach as well is the first "type_table" is about 69MB for a dSYM file for LLDB.framework in LLDB. Any types that are accessed from debug info in LLDB will end up causing this entire DW_TAG_compile_unit to be parsed which can be expensive as LLDB parses DWARF partially and only parses what it needs to when loading DWARF. Is there any way to split up the "type_table" into multiple DW_TAG_compile_unit entries whose DW_AT_name matches the DW_AT_decl_file of the root type that is being uniqued? That would allow smaller compile units to be parsed and keep memory footprint down in debuggers.

I think that it could be done in general. Though I am not sure about the concrete scheme. Following are reasons why I implemented it as a single type unit:

1. Fragmentation. clang binary has ~1600 compilation units. If every unit would have a separate type unit, then unit header, unit die, namespace dies, base types, line table headers would be duplicated. Types distributed between units should also be duplicated(if they have a template parameter from another unit, similar to the above example for an anonymous type).

2. Cross CU references. The current type table does not have outside references. That allows processing the table by one pass. If we allow outside references then we need to accumulate such inter-connected units in memory first and then process them. I do not know how that is currently implemented in lldb - Probably there is some efficient scheme to handle inter-connected units.

Probably, we might create some scheme with several type units. F.e. similar to what David described: "a basic heuristic would be X types per unit, but a more advanced one would depend on the size of the types (& even more advanced would group related types together to benefit from more local references - but I doubt that's worth it)."

[dblaikie]

In D96035#3223521, @avl wrote:

In D96035#3223263, @dblaikie wrote:

You can use cross-unit references (DW_FORM_sec_offset) to have one type/DIE in one unit refer to a type/DIE in another unit. So they could still be moved to/handled in a common way, but split up into multiple units to avoid pathological cases in DWARF consumers that might tend to load whole units at a time. Would that be feasible? (not sure if such consumers also then have common pathologies where they load all units that a unit references via DW_FORM_sec_offset... if they do, then the solution becomes more difficult and would involve either putting every type in a separate unit or finding groups of related types to put together, rather than arbitrary grouping - @clayborg do you know if lldb has that sort of behavior? (where it'd try to load all referenced units from a unit it's loading))

If I understood correctly - you are talking here more about splitting the current global type table between several artificial type tables.
Which is an additional problem, but not what I was talking about. I tried to describe situation when a class ("A") contains information for global
namespace and for anonymous namespace:

DW_TAG_structure_type                                               | global&anonymous
  DW_AT_calling_convention        (DW_CC_pass_by_value)             | global&anonymous
  DW_AT_name      ("A")                                             | global&anonymous
  DW_AT_byte_size (0x01)                                            | global&anonymous
  DW_AT_decl_file ("/home/avl/test_templates1/./a.h")               | global&anonymous
  DW_AT_decl_line (1)                                               | global&anonymous

  DW_TAG_subprogram                                                 | anonymous
    DW_AT_linkage_name    ("_ZN1A3fooIN12_GLOBAL__N_11BEEET_v")     | anonymous
    DW_AT_name    ("foo<(anonymous namespace)::B>")                 | anonymous
    DW_AT_decl_file       ("/home/avl/test_templates1/./a.h")       | anonymous
    DW_AT_decl_line       (3)                                       | anonymous
    DW_AT_type    (0x0000016d "(anonymous namespace)::B")           | anonymous
    DW_AT_declaration     (true)                                    | anonymous

    DW_TAG_template_type_parameter                                  | anonymous
      DW_AT_type  (0x0000016d "(anonymous namespace)::B")           | anonymous
      DW_AT_name  ("T")                                             | anonymous

    DW_TAG_formal_parameter                                         | anonymous
      DW_AT_type  (0x000001b3 "A *")                                | anonymous
      DW_AT_artificial    (true)                                    | anonymous

    NULL                                                            | anonymous
    
  DW_TAG_subprogram                                                 | global
    DW_AT_linkage_name    ("another member of A")                   | global
    DW_AT_name    ("another member of A")                           | global
    
  NULL                                                              | global&anonymous

Thus it should be split into two parts. One should go to the global type table:

DW_TAG_structure_type                                               
  DW_AT_calling_convention        (DW_CC_pass_by_value)             
  DW_AT_name      ("A")                                             
  DW_AT_byte_size (0x01)                                           
  DW_AT_decl_file ("/home/avl/test_templates1/./a.h")              
  DW_AT_decl_line (1)                                              

  DW_TAG_subprogram                                                
    DW_AT_linkage_name    ("another member of A")                  
    DW_AT_name    ("another member of A")                          
    
  NULL

Another part should be left into the owning compilation unit.

DW_TAG_structure_type                                             
DW_AT_calling_convention        (DW_CC_pass_by_value)             
DW_AT_name      ("A")                                             
DW_AT_byte_size (0x01)                                            
DW_AT_decl_file ("/home/avl/test_templates1/./a.h")               
DW_AT_decl_line (1)                                               

DW_TAG_subprogram                                                 
  DW_AT_linkage_name    ("_ZN1A3fooIN12_GLOBAL__N_11BEEET_v")     
  DW_AT_name    ("foo<(anonymous namespace)::B>")                 
  DW_AT_decl_file       ("/home/avl/test_templates1/./a.h")       
  DW_AT_decl_line       (3)                                       
  DW_AT_type    (0x0000016d "(anonymous namespace)::B")           
  DW_AT_declaration     (true)                                    

  DW_TAG_template_type_parameter                                  
    DW_AT_type  (0x0000016d "(anonymous namespace)::B")           
    DW_AT_name  ("T")                                             

  DW_TAG_formal_parameter                                         
    DW_AT_type  (0x000001b3 "A *")                                
    DW_AT_artificial    (true)                                    

  NULL                                                            
  
NULL

That kind of splitting is not implemented by current patch.
Also that solution would require more space since it duplicates structure description.

Or are you suggesting another solution for that case?

Oh, sorry, I wasn't discussing that case/didn't understand that's what was being discussed. I think for that case it's fine to move the translation-unit-local/private linkage/anonymous-ish part of the type into the global type table along with the rest of the type.

I think @clayborg's point was mostly for a purely translation-unit-local entity it could stay in the translation unit that defined it - reducing the need for cross-CU referencing to refer to the type and for work needed to move the type around, etc.

In D96035#3223734, @avl wrote:

In D96035#3221098, @clayborg wrote:

One issue I do see with this approach as well is the first "type_table" is about 69MB for a dSYM file for LLDB.framework in LLDB. Any types that are accessed from debug info in LLDB will end up causing this entire DW_TAG_compile_unit to be parsed which can be expensive as LLDB parses DWARF partially and only parses what it needs to when loading DWARF. Is there any way to split up the "type_table" into multiple DW_TAG_compile_unit entries whose DW_AT_name matches the DW_AT_decl_file of the root type that is being uniqued? That would allow smaller compile units to be parsed and keep memory footprint down in debuggers.

I think that it could be done in general. Though I am not sure about the concrete scheme. Following are reasons why I implemented it as a single type unit:

1. Fragmentation. clang binary has ~1600 compilation units. If every unit would have a separate type unit, then unit header, unit die, namespace dies, base types, line table headers would be duplicated. Types distributed between units should also be duplicated(if they have a template parameter from another unit, similar to the above example for an anonymous type).

Yeah, I think one type per unit would be more expensive than would be desirable - like type units which have a lot of overhead for this sort of reason.

I don't understand the second part "Types distributed between units should also be duplicated" - types can refer to types in other units (using DW_FORM_sec_offset the same as the way that types will be referenced from the non-type contexts where the type needs to be referred to (eg: a compilation unit with a subprogram that needs to refer to the return type in the global type table/type unit/whatever)), I wouldn't advocate for any duplication.

2. Cross CU references. The current type table does not have outside references. That allows processing the table by one pass. If we allow outside references then we need to accumulate such inter-connected units in memory first and then process them. I do not know how that is currently implemented in lldb - Probably there is some efficient scheme to handle inter-connected units.

If the types can all be generated in one pass today - they could be generated in one pass over multiple units as well. I don't follow why that'd require multiple passes. (laying out one unit with multiple types requires laying out all the DIEs to know the values of unit-relative offsets - putting those DIEs in several units doesn't make that situation worse in terms of the amount of data that needs to be kept around - it'd mean laying out multiple units to determine where the offset values are across those units)

Probably, we might create some scheme with several type units. F.e. similar to what David described: "a basic heuristic would be X types per unit, but a more advanced one would depend on the size of the types (& even more advanced would group related types together to benefit from more local references - but I doubt that's worth it)."

Yep - X types would be a good first approximation. Could probably do the "X bytes or DIEs per grouping" might not be too expensive either.

[avl]

If the types can all be generated in one pass today - they could be generated in one pass over multiple units as well. I don't follow why that'd require multiple passes. (laying out one unit with multiple types requires laying out all the DIEs to know the values of unit-relative offsets - putting those DIEs in several units doesn't make that situation worse in terms of the amount of data that needs to be kept around - it'd mean laying out multiple units to determine where the offset values are across those units)

Oh sorry, I wrote not clearly. My statement is not about generation. This is about reading generated dwarf. If the resulting DWARF would be closely coupled by inter-CU references then it would be hard to read&process it. It would also be hard to implement parallel separate compile units processing(because units depend on each other). To effectively process DWARF, there would be necessary to load nearly all compile&type units into the memory. An extreme scenario of such usage is applying DWARFLinker to the already linked DWARF.

With current implementation only the "type_table" unit needs to be loaded forever, other units might be loaded/unloaded independently(because "type table" cannot reference other compile&type units).
If we would generate DWARF which has type units that reference other type units and usual compile units - then, in the worst case, we would need to load nearly all of them to be able to process(and since dependency is not known upfront we need a pass to discover dependency and then a pass which loads units and processes).

Probably, lldb does not have processing which requires loading all dependent units at once. And thus, it is not so important for lldb.

But from the point of view general DWARF reader it is better to have less coupled DWARF. That is why the current design&implementation of the patch assumes that the "type table" does not reference other units.

At first glance, we might divide types into not depending groups. So that it still be possible to not have references between type units
and be able to load smaller groups at once.

[avl]

I don't understand the second part "Types distributed between units should also be duplicated" - types can refer to types in other units (using DW_FORM_sec_offset the same as the way that types will be referenced from the non-type contexts where the type needs to be referred to (eg: a compilation unit with a subprogram that needs to refer to the return type in the global type table/type unit/whatever)), I wouldn't advocate for any duplication.

The case with parametrized function member does not have a good solution if separated between compilation units.
It either requires to duplicate part of the description(that is a solution which I was talking about trying to avoid inter-CU references),
either introduces inter-CU reference(that is solution which you are talking about(DW_FORM_sec_offset/DW_FORM_ref_addr)).
I think it would be good if we will use a solution that does not require duplicating data AND does not
require using inter-CU reference(other that non-typeCU->typeCU):

cat main.cpp

char foo1 (void);
long foo2 (void);

int main ( void ) {
  return foo1() + foo2();
}

cat a.h

struct A {
  template <class T>
  T foo () {
    return T();
  }

  int createi () {
    return 0;
   }

  float createf ( ) {
    return 0.0;
  }
};

cat foo1.cpp

#include "a.h"

char foo1 (void) {
  return A().foo<char>(); 
}

cat foo2.cpp

#include "a.h"

namespace name1 {

struct B {
  long m1;
};

B internal (void) {
  return A().foo<B>();
}

}

long foo2 (void ) {
  return name1::internal().m1;
}

int foo3 () {

 A var;

 return var.createi() + var.createf();
}

If struct "A" and struct B are placed into the different compilation units, then we have two alternatives on how to represent struct "A".
First alternative assumes duplication of struct "A" definition while second alternative assumes inter-CU reference:

1. The first compilation unit("foo1.cpp") does not have a definition for "foo<name1::B>". The definition for "foo<name1::B>" is located into the second compilation unit("foo2.cpp"). In this case we have a duplicated definition of struct "A".

DW_TAG_compile_unit
  DW_AT_name "foo1.cpp"
  
  DW_TAG_structure_type   <<<<<
    DW_AT_name "A"        <<<<<

    DW_TAG_subprogram
      DW_AT_name "foo<char>"

    DW_TAG_subprogram
      DW_AT_name "createi"

    DW_TAG_subprogram
      DW_AT_name "createf"

  NULL
  
  
DW_TAG_compile_unit
  DW_AT_name "foo2.cpp"
  
  DW_TAG_structure_type    <<<<<<
    DW_AT_name "A"         <<<<<<
    
    DW_TAG_subprogram
      DW_AT_name "foo<name1::B>"

        DW_TAG_template_type_parameter
          DW_AT_type "name1::B"

  DW_TAG_namespace
    DW_AT_name "name1"

  DW_TAG_structure_type
    DW_AT_name "B"

  NULL

2. All definitions are inside the first unit("foo1.cpp"), but we have a cross CU reference from the first unit to the second.

DW_TAG_compile_unit
  DW_AT_name "foo1.cpp"
  
  DW_TAG_structure_type
    DW_AT_name "A"

    DW_TAG_subprogram
      DW_AT_name "foo<char>"

    DW_TAG_subprogram
      DW_AT_name "createi"

    DW_TAG_subprogram
      DW_AT_name "createf"

    DW_TAG_subprogram
      DW_AT_name "foo<name1::B>"

        DW_TAG_template_type_parameter
          DW_AT_type "name1::B"   <<<<<<

  NULL
  
  
DW_TAG_compile_unit
  DW_AT_name "foo2.cpp"
  
  DW_TAG_namespace
    DW_AT_name "name1"

  DW_TAG_structure_type   <<<<<<
    DW_AT_name "B"

  NULL

The solution from this patch solves both problems by putting all data in a single compilation unit.
Thus we do not need to duplicate data and create only non-typeCU->typeCU inter-CU references.
If we want to split "type table" into a set of smaller units to minimize peak memory usage then we probably could
group types so that all dependent types would be located in a single unit.
That way we probably could avoid duplication and inter-CU references.
Avoiding inter-CU references might help to create more effective implementation of an DWARF reader.

[avl]

@dblaikie: I think @clayborg's point was mostly for a purely translation-unit-local entity it could stay in the translation unit that defined it - reducing the need for cross-CU referencing to refer to the type and for work needed to move the type around, etc.

right, the purely translation-unit-local entity might be left into the original compile unit(not moved into the type table).
The current implementation(for simplicity) could not do this. This might be done as an improvement.

@clayborg: These imported declarations amounted to 3% of overall CU size for a single compile unit that I tested. I know that the stock dsymutil whitelists all of these which causes these and all the types the reference to be kept, so if we can get rid of any DW_TAG_imported_declaration DIEs that are not referenced, we might also be able to get rid of the type that they reference as well as all types from a C++ header file that have corresponding C header files like "cstdlib" -> "stdlib.h", they will have this issue and waste space.

@dblaikie: So imported modules are never referenced from other DWARF - if you want any of them to persist you have to treat them all as roots, you can't GC them.
@dblaikie: And for imported declarations, at least those generated from Clang you have to do the same until that bug is fixed in Clang to actually reference the using declarations.

@dblaikie: In any case I assume this issue is mostly orthogonal to this proposal (ie: I expect that's an existing issue in dsymutil, not a new regression introduced by this patch), yeah?

This patch should leave imported declaration inplace. i.e. they are placed similarly to how current dsymutil does this.

[dblaikie]

In D96035#3226125, @avl wrote:

I don't understand the second part "Types distributed between units should also be duplicated" - types can refer to types in other units (using DW_FORM_sec_offset the same as the way that types will be referenced from the non-type contexts where the type needs to be referred to (eg: a compilation unit with a subprogram that needs to refer to the return type in the global type table/type unit/whatever)), I wouldn't advocate for any duplication.

The case with parametrized function member does not have a good solution if separated between compilation units.
It either requires to duplicate part of the description(that is a solution which I was talking about trying to avoid inter-CU references),
either introduces inter-CU reference(that is solution which you are talking about(DW_FORM_sec_offset/DW_FORM_ref_addr)).
I think it would be good if we will use a solution that does not require duplicating data AND does not
require using inter-CU reference(other that non-typeCU->typeCU):

cat main.cpp

char foo1 (void);
long foo2 (void);

int main ( void ) {
  return foo1() + foo2();
}

cat a.h

struct A {
  template <class T>
  T foo () {
    return T();
  }

  int createi () {
    return 0;
   }

  float createf ( ) {
    return 0.0;
  }
};

cat foo1.cpp

#include "a.h"

char foo1 (void) {
  return A().foo<char>(); 
}

cat foo2.cpp

#include "a.h"

namespace name1 {

struct B {
  long m1;
};

B internal (void) {
  return A().foo<B>();
}

}

long foo2 (void ) {
  return name1::internal().m1;
}

int foo3 () {

 A var;

 return var.createi() + var.createf();
}

If struct "A" and struct B are placed into the different compilation units, then we have two alternatives on how to represent struct "A".
First alternative assumes duplication of struct "A" definition while second alternative assumes inter-CU reference:

1. The first compilation unit("foo1.cpp") does not have a definition for "foo<name1::B>". The definition for "foo<name1::B>" is located into the second compilation unit("foo2.cpp"). In this case we have a duplicated definition of struct "A".

DW_TAG_compile_unit
  DW_AT_name "foo1.cpp"
  
  DW_TAG_structure_type   <<<<<
    DW_AT_name "A"        <<<<<

    DW_TAG_subprogram
      DW_AT_name "foo<char>"

    DW_TAG_subprogram
      DW_AT_name "createi"

    DW_TAG_subprogram
      DW_AT_name "createf"

  NULL
  
  
DW_TAG_compile_unit
  DW_AT_name "foo2.cpp"
  
  DW_TAG_structure_type    <<<<<<
    DW_AT_name "A"         <<<<<<
    
    DW_TAG_subprogram
      DW_AT_name "foo<name1::B>"

        DW_TAG_template_type_parameter
          DW_AT_type "name1::B"

  DW_TAG_namespace
    DW_AT_name "name1"

  DW_TAG_structure_type
    DW_AT_name "B"

  NULL

2. All definitions are inside the first unit("foo1.cpp"), but we have a cross CU reference from the first unit to the second.

DW_TAG_compile_unit
  DW_AT_name "foo1.cpp"
  
  DW_TAG_structure_type
    DW_AT_name "A"

    DW_TAG_subprogram
      DW_AT_name "foo<char>"

    DW_TAG_subprogram
      DW_AT_name "createi"

    DW_TAG_subprogram
      DW_AT_name "createf"

    DW_TAG_subprogram
      DW_AT_name "foo<name1::B>"

        DW_TAG_template_type_parameter
          DW_AT_type "name1::B"   <<<<<<

  NULL
  
  
DW_TAG_compile_unit
  DW_AT_name "foo2.cpp"
  
  DW_TAG_namespace
    DW_AT_name "name1"

  DW_TAG_structure_type   <<<<<<
    DW_AT_name "B"

  NULL

The solution from this patch solves both problems by putting all data in a single compilation unit.
Thus we do not need to duplicate data and create only non-typeCU->typeCU inter-CU references.
If we want to split "type table" into a set of smaller units to minimize peak memory usage then we probably could
group types so that all dependent types would be located in a single unit.
That way we probably could avoid duplication and inter-CU references.
Avoiding inter-CU references might help to create more effective implementation of an DWARF reader.

This doesn't address the case where struct A is a local type (in an anonymous namespace). Moving it out of the translation/compile unit it's defined it will break the representation (because the name is only valid within that compile unit - so you might end up pulling two types with the same name into the type table unit, making it hard for a DWARF consumer to do correct name lookup/know which version of the type the user is talking about in a given context).

What we could do is a third option - not duplicating the whole definition of A, but only part of the definition.

Take a look at the DWARF Clang produces for something like this, for instance:

struct t1 {
  virtual void f1(); // home this type into another translation unit
  template<typename T>
  void f1() { }
};
namespace {
struct t2 { };
}
int main() {
  t1().f1<t2>();
}

The DWARF for this will include a declaration of t1 (so, some duplication - but not all the members, etc) with a declaration of the f1<t2> member in that t1 declaration (& then an out of line definition that refers to that declaration) (in this case we could potentially also include a DW_AT_specification in the declaration that points to the definition in the type table, if that helps consumers significantly - so they don't have to do name lookup to figure out that they're the same thing).

Speaking of all that - how are member function definitions working in this current proposal? Do they use sec_offsets to refer to the declaration in the type table, or do they already do something like ^ with a type declaration in the same unit to house the member function declaration?

[avl]

! In D96035#3232592, @dblaikie wrote:

This doesn't address the case where struct A is a local type (in an anonymous namespace). Moving it out of the translation/compile unit it's defined it will break the representation (because the name is only valid within that compile unit - so you might end up pulling two types with the same name into the type table unit, making it hard for a DWARF consumer to do correct name lookup/know which version of the type the user is talking about in a given context).

This patch put such types into the type table separately. It creates two different namespaces with different DW_AT_decl_file attribute:

DW_TAG_compile_unit
  DW_AT_name "__type_table"
  
  DW_TAG_namespace
    DW_AT_decl_file "compile_unit1.cpp"
    
    DW_TAG_class_type
      DW_AT_name "B"
    NULL
  NULL

  DW_TAG_namespace
    DW_AT_decl_file "compile_unit2.cpp"
    
    DW_TAG_class_type
      DW_AT_name "B"
    NULL
  NULL

NULL

What we could do is a third option - not duplicating the whole definition of A, but only part of the definition.

Take a look at the DWARF Clang produces for something like this, for instance:

struct t1 {
  virtual void f1(); // home this type into another translation unit
  template<typename T>
  void f1() { }
};
namespace {
struct t2 { };
}
int main() {
  t1().f1<t2>();
}

The DWARF for this will include a declaration of t1 (so, some duplication - but not all the members, etc) with a declaration of the f1<t2> member in that t1 declaration (& then an out of line definition that refers to that declaration) (in this case we could potentially also include a DW_AT_specification in the declaration that points to the definition in the type table, if that helps consumers significantly - so they don't have to do name lookup to figure out that they're the same thing).

yes. I assumed the similar approach here - https://reviews.llvm.org/D96035#3222090 (part about types splitting).

If type splitting were supported then this approach allows leaving anonymous types inside the compilation unit at the cost of little duplication and without unnecessary inter-CU references. It also allows decreasing the overall size of "type table". The current patch does not support type splitting. I suggest implementing it as an additional improvement.

Types splitting is necessary because originally generated "declaration of t1 (so, some duplication - but not all the members, etc) with a declaration of the f1<t2> member in that t1 declaration" might contain other members. So we need to move these other members into the type table and leave "declaration of the f1<t2> member" in place. Otherwise, (because other members will stay not deduplicated) the size of resulting DWARF would be bigger.

Speaking of all that - how are member function definitions working in this current proposal? Do they use sec_offsets to refer to the declaration in the type table, or do they already do something like ^ with a type declaration in the same unit to house the member function declaration?

All member function definitions are merged into the type which is in type table. Later they are referenced(using DW_FORM_ref_addr) from compile units:

DW_TAG_compile_unit
  DW_AT_name "__type_table"

  DW_TAG_structure_type
    DW_AT_name "t1"
      
    DW_TAG_subprogram
      DW_AT_name "t1"
    NULL

    DW_TAG_subprogram
      DW_AT_name "f1"
      
      DW_TAG_template_type_parameter
        DW_AT_type "__type_table::main.cpp::t2"  // I used names instead offsets, in fact this is DW_FORM_ref_addr referencing into "__type_table"
    NULL
  NULL
  
  DW_TAG_namespace
    DW_AT_decl_file "main.cpp"
    
    DW_TAG_struct_type
      DW_AT_name "t2"
    NULL
  NULL

NULL

DW_TAG_compile_unit
  DW_AT_name "main.cpp"
  
  DW_TAG_subprogram
    DW_AT_name "main"
  NULL

  DW_TAG_subprogram
    DW_AT_name "t1"
    DW_AT_specification "__type_table::t1::t1" // I used names instead offsets, in fact this is DW_FORM_ref_addr referencing into "__type_table"
  NULL

  DW_TAG_subprogram
    DW_AT_name "f1"
    DW_AT_specification "__type_table::t1::f1" // I used names instead offsets, in fact this is DW_FORM_ref_addr referencing into "__type_table"
  NULL

NULL

[dblaikie]

In D96035#3234040, @avl wrote:

! In D96035#3232592, @dblaikie wrote:

This doesn't address the case where struct A is a local type (in an anonymous namespace). Moving it out of the translation/compile unit it's defined it will break the representation (because the name is only valid within that compile unit - so you might end up pulling two types with the same name into the type table unit, making it hard for a DWARF consumer to do correct name lookup/know which version of the type the user is talking about in a given context).

This patch put such types into the type table separately. It creates two different namespaces with different DW_AT_decl_file attribute:

DW_TAG_compile_unit
  DW_AT_name "__type_table"
  
  DW_TAG_namespace
    DW_AT_decl_file "compile_unit1.cpp"
    
    DW_TAG_class_type
      DW_AT_name "B"
    NULL
  NULL

  DW_TAG_namespace
    DW_AT_decl_file "compile_unit2.cpp"
    
    DW_TAG_class_type
      DW_AT_name "B"
    NULL
  NULL

NULL

Yeah, I don't think that's enough to address the scoping issue - which file they're written in doesn't define their scope of name lookup, unfortunately. (eg: you could #include a file into another file - the file the code is written in doesn't tell you whether another entity defined in a different file can see that name (because the files could be included into the same translation unit)) - local types, I think, have to stay in the compile unit that defined them - I don't think there's any other solution to the language scoping.

What we could do is a third option - not duplicating the whole definition of A, but only part of the definition.

Take a look at the DWARF Clang produces for something like this, for instance:

struct t1 {
  virtual void f1(); // home this type into another translation unit
  template<typename T>
  void f1() { }
};
namespace {
struct t2 { };
}
int main() {
  t1().f1<t2>();
}

The DWARF for this will include a declaration of t1 (so, some duplication - but not all the members, etc) with a declaration of the f1<t2> member in that t1 declaration (& then an out of line definition that refers to that declaration) (in this case we could potentially also include a DW_AT_specification in the declaration that points to the definition in the type table, if that helps consumers significantly - so they don't have to do name lookup to figure out that they're the same thing).

yes. I assumed the similar approach here - https://reviews.llvm.org/D96035#3222090 (part about types splitting).

If type splitting were supported then this approach allows leaving anonymous types inside the compilation unit at the cost of little duplication and without unnecessary inter-CU references. It also allows decreasing the overall size of "type table". The current patch does not support type splitting. I suggest implementing it as an additional improvement.

Types splitting is necessary because originally generated "declaration of t1 (so, some duplication - but not all the members, etc) with a declaration of the f1<t2> member in that t1 declaration" might contain other members. So we need to move these other members into the type table and leave "declaration of the f1<t2> member" in place. Otherwise, (because other members will stay not deduplicated) the size of resulting DWARF would be bigger.

Yep, that sounds right to me. To the best of my knowledge, the sort of things that may need to get "left behind" in the compilation unit/not moved to the type table, would be member templates like the one we've been discussing, and nested types (eg: you could have a pimpl idiom - struct t1 { struct t2; }; ... namespace { struct t3 { }; } struct t1::t2 { t3 m; }; so the whole definition of t2 would have to stay in the compilation unit with t3, I think - because it needs to reference t3. The type table could include a declaration of t2, since t1 might include pointers to t2 or other references that only require a declaration)

The nice thing about this particular way of splitting - is that there's loads of existing precedent for it. This sort of DWARF (declarations with members) is generated by clang all the time for homed debug info. (though, that said, lldb isn't /super/ fond of this sort of debug info - so it wouldn't surprise me if it'd need some fixes to handle this sort of thing in a dsym, but we might be lucky there & it already works because at least all the types are still defined within the single dsym)

Speaking of all that - how are member function definitions working in this current proposal? Do they use sec_offsets to refer to the declaration in the type table, or do they already do something like ^ with a type declaration in the same unit to house the member function declaration?

All member function definitions are merged into the type which is in type table. Later they are referenced(using DW_FORM_ref_addr) from compile units:

DW_TAG_compile_unit
  DW_AT_name "__type_table"

  DW_TAG_structure_type
    DW_AT_name "t1"
      
    DW_TAG_subprogram
      DW_AT_name "t1"
    NULL

    DW_TAG_subprogram
      DW_AT_name "f1"
      
      DW_TAG_template_type_parameter
        DW_AT_type "__type_table::main.cpp::t2"  // I used names instead offsets, in fact this is DW_FORM_ref_addr referencing into "__type_table"
    NULL
  NULL
  
  DW_TAG_namespace
    DW_AT_decl_file "main.cpp"
    
    DW_TAG_struct_type
      DW_AT_name "t2"
    NULL
  NULL

NULL

DW_TAG_compile_unit
  DW_AT_name "main.cpp"
  
  DW_TAG_subprogram
    DW_AT_name "main"
  NULL

  DW_TAG_subprogram
    DW_AT_name "t1"
    DW_AT_specification "__type_table::t1::t1" // I used names instead offsets, in fact this is DW_FORM_ref_addr referencing into "__type_table"
  NULL

  DW_TAG_subprogram
    DW_AT_name "f1"
    DW_AT_specification "__type_table::t1::f1" // I used names instead offsets, in fact this is DW_FORM_ref_addr referencing into "__type_table"
  NULL

NULL

Ah, OK - thanks for the details!

[dblaikie]

(I should say, whether or not it's necessary/prerequisite to handle the anonymous types more accurately, through type splitting - I don't have an opinion on - dsymutil isn't something I use/have any investment in - I'm just discussing the DWARF representational tools and techniques that might be used to address some of these limitations. Folks who own/maintain/ship dsymutil are the ones to make any final choices about what is/isn't acceptable here)

[avl]

rebased. removed usages of NonRelocatableStringpool.

[avl]

rebased.
refactored.
implemented performance improvements(it is 1.3x slower in single thread mode, 2.4x faster im multi-thread mode).
implemented types splitting.

[dblaikie]

In D96035#3284247, @avl wrote:

rebased.
refactored.
implemented performance improvements(it is 1.3x slower in single thread mode, 2.4x faster im multi-thread mode).

1.3x slower means what took 1second before now takes 2.3 seconds?

implemented types splitting.

This means that anonymous/non-linkage types are correctly kept in their original CUs? Does it mean other things too/instead? (splitting up types into chunks to avoid exceptionally large units containing all types?)

[MaskRay]

In D96035#3284832, @dblaikie wrote:

In D96035#3284247, @avl wrote:

rebased.
refactored.
implemented performance improvements(it is 1.3x slower in single thread mode, 2.4x faster im multi-thread mode).

1.3x slower means what took 1second before now takes 2.3 seconds?

(https://randomascii.wordpress.com/2018/02/04/what-we-talk-about-when-we-talk-about-performance/ The ambiguity made me start to use "1.3x as fast" )

implemented types splitting.

This means that anonymous/non-linkage types are correctly kept in their original CUs? Does it mean other things too/instead? (splitting up types into chunks to avoid exceptionally large units containing all types?)

[avl]

In D96035#3284832, @dblaikie wrote:

In D96035#3284247, @avl wrote:

rebased.
refactored.
implemented performance improvements(it is 1.3x slower in single thread mode, 2.4x faster im multi-thread mode).

1.3x slower means what took 1second before now takes 2.3 seconds?

No, sorry, I meant what took 1second before now takes 1*1.3 = 1.3sec.

implemented types splitting.

This means that anonymous/non-linkage types are correctly kept in their original CUs?

yes. anonymous/non-linkage types are correctly kept in their original CUs. These are tests for that functionality:

odr-nested-types1.test
odr-types-in-subprogram1.test

Does it mean other things too/instead? (splitting up types into chunks to avoid exceptionally large units containing all types?)

splitting up types into chunks is not implemented yet. I think it would take some time. I would prefer to implement it as an separate improvement for this patch.

[clayborg]

In D96035#3284921, @avl wrote:

In D96035#3284832, @dblaikie wrote:

In D96035#3284247, @avl wrote:

rebased.
refactored.
implemented performance improvements(it is 1.3x slower in single thread mode, 2.4x faster im multi-thread mode).

1.3x slower means what took 1second before now takes 2.3 seconds?

No, sorry, I meant what took 1second before now takes 1*1.3 = 1.3sec.

implemented types splitting.

This means that anonymous/non-linkage types are correctly kept in their original CUs?

yes. anonymous/non-linkage types are correctly kept in their original CUs. These are tests for that functionality:

odr-nested-types1.test
odr-types-in-subprogram1.test

Does it mean other things too/instead? (splitting up types into chunks to avoid exceptionally large units containing all types?)

splitting up types into chunks is not implemented yet. I think it would take some time. I would prefer to implement it as an separate improvement for this patch.

I do believe that splitting types up into a compile unit that matches the DW_AT_decl_file would make this patch really hard to resist as it then makes the DWARF the best it can be. The nice thing is that if this is done it makes it very easy to tell where a type should be defined. So if the type's DW_AT_decl_file matches the current CU or if this is an anonymous namespace, then the type stays where it is. If it doesn't match, then it gets moved to a new compile unit. I don't know exactly how complex this would be, but it seems like it shouldn't be too hard. The huge type unit has the ability to greatly impact debugger performance as the code stands now because as soon as the debugger needs any type, it will have to parse all of the DIEs in the type compile unit. LLDB parses DWARF lazily and only pulls in what it needs, but with these binaries we would need to parse some 60MB of type DIEs as soon as anyone needs a type.

[dblaikie]

In D96035#3285620, @clayborg wrote:

In D96035#3284921, @avl wrote:

In D96035#3284832, @dblaikie wrote:

In D96035#3284247, @avl wrote:

rebased.
refactored.
implemented performance improvements(it is 1.3x slower in single thread mode, 2.4x faster im multi-thread mode).

1.3x slower means what took 1second before now takes 2.3 seconds?

No, sorry, I meant what took 1second before now takes 1*1.3 = 1.3sec.

implemented types splitting.

This means that anonymous/non-linkage types are correctly kept in their original CUs?

yes. anonymous/non-linkage types are correctly kept in their original CUs. These are tests for that functionality:

odr-nested-types1.test
odr-types-in-subprogram1.test

Does it mean other things too/instead? (splitting up types into chunks to avoid exceptionally large units containing all types?)

splitting up types into chunks is not implemented yet. I think it would take some time. I would prefer to implement it as an separate improvement for this patch.

I do believe that splitting types up into a compile unit that matches the DW_AT_decl_file would make this patch really hard to resist as it then makes the DWARF the best it can be.

I think the major reason it'd be resisted would be the cost - increasing dsymutil time significantly would be a hard sell. I think it's pretty understandable that the sort of parallel processing @avl's experimenting with here wouldn't be able to do that efficiently.

Arbitrarily grouping types into smallish buckets to reduce the overhead for DWARF consumers seems more feasible, though, I think.

[avl]

In D96035#3285620, @clayborg wrote:

I do believe that splitting types up into a compile unit that matches the DW_AT_decl_file would make this patch really hard to resist as it then makes the DWARF the best it can be. The nice thing is that if this is done it makes it very easy to tell where a type should be defined. So if the type's DW_AT_decl_file matches the current CU or if this is an anonymous namespace, then the type stays where it is. If it doesn't match, then it gets moved to a new compile unit. I don't know exactly how complex this would be, but it seems like it shouldn't be too hard. The huge type unit has the ability to greatly impact debugger performance as the code stands now because as soon as the debugger needs any type, it will have to parse all of the DIEs in the type compile unit. LLDB parses DWARF lazily and only pulls in what it needs, but with these binaries we would need to parse some 60MB of type DIEs as soon as anyone needs a type.

There are some disadvantages with creating additional compilation units for each source compile unit:

1. Fragmentation and duplication. It would be necessary to duplicate: unit header, unit die, namespace dies, base types, line table headers, line table files, abbreviation table. clang has approx 1600 compilation units. So we need to duplicate all the above information for each of them. At the end of all, we might lose some DWARF size achievements.

2. Clously coupled references. If all types would be placed in separate compilation units matched with the original unit of declaration then types would reference each other. As the result, It would be hard to process such units in a parallel manner(independently). This limits the acceleration that can be achieved by parallelization. This patch tries to avoid cross-CU references. Only one type is allowed: non-type-CU -> type-CU.

What about the following solution: Current type table unit(let`s say 60MB) would be divided into several buckets(let`s say 16) of independent types. Each bucket is placed in separate artificial compilation unit. So that there would not be references between units, there would not be a lot of duplicated information. The size of each separate type unit would be around 4MB(it would help to lldb to not parse much). Can this be a good solution? It looks like it allows to keep benefits(small final size of overall DWARF file, simple references, small size of each compile unit). It also would probably help to speed up multi-thread execution of DWARFLinker(if all type units would be generated in parallel) but I am afraid it would slow down single-thread execution.

[dblaikie]

In D96035#3287393, @avl wrote:

In D96035#3285620, @clayborg wrote:

I do believe that splitting types up into a compile unit that matches the DW_AT_decl_file would make this patch really hard to resist as it then makes the DWARF the best it can be. The nice thing is that if this is done it makes it very easy to tell where a type should be defined. So if the type's DW_AT_decl_file matches the current CU or if this is an anonymous namespace, then the type stays where it is. If it doesn't match, then it gets moved to a new compile unit. I don't know exactly how complex this would be, but it seems like it shouldn't be too hard. The huge type unit has the ability to greatly impact debugger performance as the code stands now because as soon as the debugger needs any type, it will have to parse all of the DIEs in the type compile unit. LLDB parses DWARF lazily and only pulls in what it needs, but with these binaries we would need to parse some 60MB of type DIEs as soon as anyone needs a type.

There are some disadvantages with creating additional compilation units for each source compile unit:

1. Fragmentation and duplication. It would be necessary to duplicate: unit header, unit die, namespace dies, base types, line table headers, line table files, abbreviation table. clang has approx 1600 compilation units. So we need to duplicate all the above information for each of them. At the end of all, we might lose some DWARF size achievements.

Oh, yeah, too many units sounds unfortunate for sure. Currently we'd have one CU per .cpp file, and with "putting types in a synthetic unit that matches their decl_file" we'd have an additional one CU per .h file - which, admittedly, is probably a bit more than doubling the number of units. (template instantiations, for instance, might all be grouped together in the same unit, since they're defined in the same header)

I don't know that this makes the DWARF 'the best it can be' - it's not clear to me what expressive power is provided by the types being in units that match source file names (indeed the information is provided in the DWARF either way - a consumer can treat types defined in a certain file in a certain way even if they're all in one big CU).

2. Clously coupled references. If all types would be placed in separate compilation units matched with the original unit of declaration then types would reference each other. As the result, It would be hard to process such units in a parallel manner(independently). This limits the acceleration that can be achieved by parallelization. This patch tries to avoid cross-CU references. Only one type is allowed: non-type-CU -> type-CU.

Yeah.

What about the following solution: Current type table unit(let`s say 60MB) would be divided into several buckets(let`s say 16) of independent types. Each bucket is placed in separate artificial compilation unit. So that there would not be references between units, there would not be a lot of duplicated information. The size of each separate type unit would be around 4MB(it would help to lldb to not parse much). Can this be a good solution? It looks like it allows to keep benefits(small final size of overall DWARF file, simple references, small size of each compile unit). It also would probably help to speed up multi-thread execution of DWARFLinker(if all type units would be generated in parallel) but I am afraid it would slow down single-thread execution.

Finding independent buckets of types sounds difficult/algorithmically complicated. But maybe that's feasible? I'm not sure. I was thinking more "emit all the types the same way you do currently, except into multiple unit "Chunks"" - ie: the code already handles type-to-type references within the single type-CU, so I don't understand (maybe I'm missing something) why it would be difficult to treat that "type-CU" as actually being "multiple type CUs" with arbitrary cross-referencing within that collection of type CUs. Then the chunks/buckets are chosen arbitrarily - admittedly that means longer encoding (sec_offset references are longer than the unit-relative references) than if you can group the types together into isolated/only-self-referencing groups - so maybe the extra space savings is worth the work to create those isolated groups? Naively, I would not have expected it to be worth that much.

[avl]

Finding independent buckets of types sounds difficult/algorithmically complicated. But maybe that's feasible? I'm not sure.

I do not have a ready implementation. So I do not know whether implementation would be acceptable. And even whether it is possible to divide whole type table onto 16 independent parts. That needs some investigation.

I was thinking more "emit all the types the same way you do currently, except into multiple unit "Chunks"" - ie: the code already handles type-to-type references within the single type-CU, so I don't understand (maybe I'm missing something) why it would be difficult to treat that "type-CU" as actually being "multiple type CUs" with arbitrary cross-referencing within that collection of type CUs. Then the chunks/buckets are chosen arbitrarily - admittedly that means longer encoding (sec_offset references are longer than the unit-relative references) than if you can group the types together into isolated/only-self-referencing groups - so maybe the extra space savings is worth the work to create those isolated groups? Naively, I would not have expected it to be worth that much.

Current implementation deals with one big type unit. This implementation loads all types in memory. The similar implementation might be done for "multiple type CUs". But if we had several independent type CU we might be able to not load all type units CU into the memory but to process them separately and throw out of memory. This would allow to reduce memory requirements.

So yes we might change current implementation to generate several type CUs referencing each other. It would have the same properties as current implementation with the exception that lldb would be able to load it by smaller chunks. If we would be able to divide whole type unit onto several independent parts then we might use more efficient implementation(which uses less memory f.e.)

[avl]

I`ve got following message in the mailbox from @clayborg . I believe it was sent here but it was not appeared because of some error. So I duplicate it here.

On Feb 1, 2022, at 9:26 AM, David Blaikie via Phabricator <[e-mail]> wrote:

dblaikie added a comment.

In D96035#3287393 https://reviews.llvm.org/D96035#3287393, @avl wrote:

In D96035#3285620 https://reviews.llvm.org/D96035#3285620, @clayborg wrote:

I do believe that splitting types up into a compile unit that matches the DW_AT_decl_file would make this patch really hard to resist as it then makes the DWARF the best it can be. The nice thing is that if this is done it makes it very easy to tell where a type should be defined. So if the type's DW_AT_decl_file matches the current CU or if this is an anonymous namespace, then the type stays where it is. If it doesn't match, then it gets moved to a new compile unit. I don't know exactly how complex this would be, but it seems like it shouldn't be too hard. The huge type unit has the ability to greatly impact debugger performance as the code stands now because as soon as the debugger needs any type, it will have to parse all of the DIEs in the type compile unit. LLDB parses DWARF lazily and only pulls in what it needs, but with these binaries we would need to parse some 60MB of type DIEs as soon as anyone needs a type.

There are some disadvantages with creating additional compilation units for each source compile unit:

1. Fragmentation and duplication. It would be necessary to duplicate: unit header, unit die, namespace dies, base types, line table headers, line table files, abbreviation table. clang has approx 1600 compilation units. So we need to duplicate all the above information for each of them. At the end of all, we might lose some DWARF size achievements.

base types wouldn't need to be duplicated right? We can put any types with no decl file into the current type unit you created and have everyone use absolute links to them? I was not a fan of the abbreviation tables being split up, I believe the llvm dsymutil has done this for a while now, as it leads to duplication as you already mention. I would prefer on big .debug_abbrev section if possible so all compile units can re-use, but I understand that this would slow things down.

Oh, yeah, too many units sounds unfortunate for sure. Currently we'd have one CU per .cpp file, and with "putting types in a synthetic unit that matches their decl_file" we'd have an additional one CU per .h file - which, admittedly, is probably a bit more than doubling the number of units. (template instantiations, for instance, might all be grouped together in the same unit, since they're defined in the same header)

So 11 bytes for a compile unit plus a few for the main DIE doesn't sound too bad to me when everything would be perfectly organized in the DWARF. Since there would all types would be have matching DW_AT_decl_file attributes, the line table should be very small, with hopefully just a few files. I would love to see a file done like this just so we can compare the sizes of an actual file to see if we are using up that much more space.

I don't know that this makes the DWARF 'the best it can be' - it's not clear to me what expressive power is provided by the types being in units that match source file names (indeed the information is provided in the DWARF either way - a consumer can treat types defined in a certain file in a certain way even if they're all in one big CU).

DWARF has always been a "shove a bunch of stuff into one big blob per compile unit" with no regard to how it will be used in the linking phase since DWARF was created. Linkers are expected to concatenate and relocate and call it a day. This leaves a ton of debug info in the linked executable that isn't useful, like functions that should have been dead stripped, but weren't because of the limitations of the DWARF format and what can be done quickly by linkers that don't understand the format. It is not easy for do anything with DWARF and it isn't a great format for just accessing a thing or two here and there since it is a serialized stream of conciousness of what is in compile unit with no real organization other than "I just put it all in one bucket".

I love the idea of having the DWARF being really well organized from a consumer stand point. Not many producers are thinking about how this information is consumed, just how quick and fast we can make it or link it. There is definitely value to doing things quickly, don't get me wrong, so for the compile/edit/debug cycle we need to keep things fast. But for when you want to create a build and archive it, I would rather spend a few more cycles getting the DWARF in the best shape it can be in so that it can be consumed efficiently by symbolication servers that want to use it, or debuggers to debug a released build than saving a few seconds or bytes in the file.

2. Clously coupled references. If all types would be placed in separate compilation units matched with the original unit of declaration then types would reference each other. As the result, It would be hard to process such units in a parallel manner(independently). This limits the acceleration that can be achieved by parallelization. This patch tries to avoid cross-CU references. Only one type is allowed: non-type-CU -> type-CU.

Currently you have one compile unit that all threads must access and register their types with right? I don't see how splitting this up would cause more delays than all threads vying for access to the one and only type unit? Seems like it would actually reduce lock contention, but I haven't fully looked at your current solution, so forgive my saying so if this is way off.

Why are we trying to avoid cross CU references with DW_FORM_ref_addr? This is placing the emphasis on linking speed once again over how this information will be accessed by consumers. And one might say that bad DWARF organization can cause more delays when these DWARF files are used for debugging or symolication by having to parse all type DIEs for all types just because it was faster to create the DWARF file in the first place.

Yeah.

What about the following solution: Current type table unit(let`s say 60MB) would be divided into several buckets(let`s say 16) of independent types. Each bucket is placed in separate artificial compilation unit. So that there would not be references between units, there would not be a lot of duplicated information. The size of each separate type unit would be around 4MB(it would help to lldb to not parse much). Can this be a good solution? It looks like it allows to keep benefits(small final size of overall DWARF file, simple references, small size of each compile unit). It also would probably help to speed up multi-thread execution of DWARFLinker(if all type units would be generated in parallel) but I am afraid it would slow down single-thread execution.

Does anyone cares about single thread execution time? I personally don't.

Finding independent buckets of types sounds difficult/algorithmically complicated. But maybe that's feasible? I'm not sure. I was thinking more "emit all the types the same way you do currently, except into multiple unit "Chunks"" - ie: the code already handles type-to-type references within the single type-CU, so I don't understand (maybe I'm missing something) why it would be difficult to treat that "type-CU" as actually being "multiple type CUs" with arbitrary cross-referencing within that collection of type CUs. Then the chunks/buckets are chosen arbitrarily - admittedly that means longer encoding (sec_offset references are longer than the unit-relative references) than if you can group the types together into isolated/only-self-referencing groups - so maybe the extra space savings is worth the work to create those isolated groups? Naively, I would not have expected it to be worth that much.

DW_FORM_ref_addr is 32 bits for DWARF32, and not many use DWARF64 as it has other size increases, so using these to do cross CU references is just as cheap as a CU relative reference. And I haven't seen any compilers take advantage of using DW_FORM_ref1 or DW_FORM_ref2, they just use DW_FORM_ref4 by default which is the same size as DW_FORM_ref_addr.

I would love to see the "each type gets put into their own compile unit based on the DW_AT_decl_file of each type", just to see what this does to the size of the DWARF compared to the currently solution. If the size increase is too much, I would drop my suggestion. Is trying this out hard to do and not worrying at all about performance?

One other solution would be to group types into common subdirectories based off of the DW_AT_decl_file. The idea being and all STL types could be in one common compile unit for anything in with a Dw_AT_decl_file that starts with a common directory prefix? This would allow "vector", "list" and "map" types for the STL to all be in a a specific type compile unit? So the idea would be to strip the file base name from the DW_AT_decl_file and group all types from all header files in that directory into the same type unit?

[avl]

base types wouldn't need to be duplicated right? We can put any types with no decl file into the current type unit you created and have everyone use absolute links to them?

I am currently implementing an optimization when basic types are always put in the start of current CU and then it is possible to use DW_FORM_ref1 reference to access them. If that optimization become useful then it would be necessary to duplicate them. But without such optimization, there is no need to duplicate basic types.

I was not a fan of the abbreviation tables being split up, I believe the llvm dsymutil has done this for a while now, as it leads to duplication as you already mention. I would prefer on big .debug_abbrev section if possible so all compile units can re-use, but I understand that this would slow things down.

The problem with abbreviation table is not that it affects performance.
I think it is possible to implement global lock-free fast abbreviation table.
The problem is deterministic output. If several threads would write into the
single global abbreviation table then order of elements inside abbreviation table
would be non-deterministic. Thus, I do not see currently how parallel processing
of compilation units might be done using global abbreviation table.

Without requirement of deterministic output - global abbreviation table might be effectively implemented.

So 11 bytes for a compile unit plus a few for the main DIE doesn't sound too bad to me when everything would be perfectly organized in the DWARF. Since there would all types would be have matching DW_AT_decl_file attributes, the line table should be very small, with hopefully just a few files. I would love to see a file done like this just so we can compare the sizes of an actual file to see if we are using up that much more space.

I would love to see the "each type gets put into their own compile unit based on the DW_AT_decl_file of each type", just to see what this does to the size of the DWARF compared to the currently solution. If the size increase is too much, I would drop my suggestion. Is trying this out hard to do and not worrying at all about performance?

also several namespace dies, line table header, abbreviation table, duplicated info inside line table - all of them together might take a significant amount. I would try to model that case and see the real numbers.

Currently you have one compile unit that all threads must access and register their types with right? I don't see how splitting this up would cause more delays than all threads vying for access to the one and only type unit? Seems like it would actually reduce lock contention, but I haven't fully looked at your current solution, so forgive my saying so if this is way off.

It`s not about delays. I am trying to make compile units independent.

Why are we trying to avoid cross CU references with DW_FORM_ref_addr? This is placing the emphasis on linking speed once again over how this information will be accessed by consumers. And one might say that bad DWARF organization can cause more delays when these DWARF files are used for debugging or symolication by having to parse all type DIEs for all types just because it was faster to create the DWARF file in the first place.

Generally speaking, cross CU references makes CU depending on each other. That complicates parallel processing. If CU not depend on others then it could be processed and thrown out. Otherwise, If it depends on other CU`s we need to load all dependent CU`s into the memory and process them together.

If we can separate monolithic type CU into several independent units it would help to "debugging or symbolication" to not parse all type DIEs and it would help other tools(like dsymutil) to process incoming DWARF fast with minimum used RAM.

Does anyone cares about single thread execution time? I personally don't.

If we do not then things are easier :-)

One other solution would be to group types into common subdirectories based off of the DW_AT_decl_file. The idea being and all STL types could be in one common compile unit for anything in with a Dw_AT_decl_file that starts with a common directory prefix? This would allow "vector", "list" and "map" types for the STL to all be in a a specific type compile unit? So the idea would be to strip the file base name from the DW_AT_decl_file and group all types from all header files in that directory into the same type unit?

My current attempt is to try to avoid cross-CU referencies. Let me do some investigation of possible strategies.
Probably, that suggestion(to group types on common directory prefix) would be close to what I am trying to achieve.
Which size of type unit is acceptable? If all STL types require f.e. 4MB - is it OK?

[dblaikie]

Does anyone cares about single thread execution time? I personally don't.

If we do not then things are easier :-)

I think @JDevlieghere and @aprantl probably care about it just from a regression perspective - but I could be wrong. (I thought they'd expressed some concerns about that in this/other reviews previously).

Apple folks ^ have you got a sense of if/how much of a single threaded regression you'd want to accept in this work, for some multithreaded gains?

[avl]

@clayborg I`ve done some research and have a couple of questions. Would you mind looking at them, please?

1. First thing is that I tried to model the case with separate compilation units for each declaration file and it seems that the overall impact is about 0,78% of debug_info+debug_abbrev size of the clang binary. Size of debug_info+debug_abbrev size is 158.7MB. Additional size, required for multiple compilation units, is 1.23MB(2051*(100+500)). 2051 is the number of files used in decl_file attributes, 100 bytes is the size of the compilation unit header, compilation unit die, size of namespace dies, line table header, 500 is the size of separate abbreviation table.

2. Second thing is that I divided all types into independent buckets. The current type table for the clang binary takes approx 40MB. The maximal size of a bucket containing dependent types is 1.6Mb. So, if the original type table would be divided into multiple compilation units based on types dependency then each type unit might be 1,6Mb or less.
   
   Dividing types into independent buckets has several advantages: since they are not dependent on each other it is possible to handle them in a parallel/independent way. Another thing is that it potentially minimizes the number of loads which should be done. It loads all dependent types at once. In case types are not divided into independent buckets we would need to load all dependencies by several loads(i.e. whenever cross-CU reference is encountered).

Assuming we decided to split the current global type table on the "decl_file" basis as you suggested, What do you think of the following questions:

1. How the following situation should be handled:

Source DWARF:

      DW_TAG_compile_unit
        DW_AT_name "cu1"
     
0x100:  DW_TAG_namespace   <<<<<<<<<<<<<<<<<<<
          DW_AT_name "namespace1"
       
          DW_TAG_structure
            DW_AT_name "S1"
            DW_AT_decl_file "file1"
          NULL
         
          DW_TAG_structure
            DW_AT_name "S2"
            DW_AT_decl_file "file2"
          NULL
        NULL
     
        DW_TAG_import 0x100   <<<<<<<<<<<<<<<<<<<
      NULL

Result DWARF:

      DW_TAG_compile_unit
        DW_AT_name "type_table_file1"

0x200:  DW_TAG_namespace  <<<<<<<<<<<<<<<<<<<
          DW_AT_name "namespace1"
          
          DW_TAG_structure
            DW_AT_name "S1"
            DW_AT_decl_file "file1"
          NULL
          
        NULL

      DW_TAG_compile_unit
        DW_AT_name "type_table_file2"

0x300:  DW_TAG_namespace   <<<<<<<<<<<<<<<<<<<
          DW_AT_name "namespace1"

          DW_TAG_structure
            DW_AT_name "S2"
            DW_AT_decl_file "file2"
          NULL
          
        NULL

     DW_TAG_compile_unit
        DW_AT_name "cu1"

        DW_TAG_import 0x200 or 0x300 ?    <<<<<<<<<<<<<<<<<<<

Which offset corresponding to the "namespace1" should be used? Any of them?

2. Would it be OK to split DW_TAG_module ?

Source DWARF:

DW_TAG_compile_unit
  DW_AT_name "cu1"
  
  DW_TAG_module             <<<<<<<<<<<<<<<<<<< 
    DW_AT_name "module1"    <<<<<<<<<<<<<<<<<<<
    
    DW_TAG_structure
      DW_AT_name "S1"
      DW_AT_decl_file "file1"
    NULL
   
    DW_TAG_structure
      DW_AT_name "S2"
      DW_AT_decl_file "file2"
    NULL

  NULL
  
NULL

Result DWARF:

DW_TAG_compile_unit
  DW_AT_name "type_table_file1"

  DW_TAG_module            <<<<<<<<<<<<<<<<<<<
    DW_AT_name "module1"   <<<<<<<<<<<<<<<<<<<
    
    DW_TAG_structure
      DW_AT_name "S1"
      DW_AT_decl_file "file1"
    NULL
    
  NULL
  
NULL
   
DW_TAG_compile_unit
  DW_AT_name "type_table_file2"

  DW_TAG_module            <<<<<<<<<<<<<<<<<<< 
    DW_AT_name "module1"   <<<<<<<<<<<<<<<<<<<
    
    DW_TAG_structure
      DW_AT_name "S2"
      DW_AT_decl_file "file2"
    NULL
    
  NULL
  
NULL

Is it OK, that DW_TAG_module would be split?

3. Only root types should be moved into compile unit for corresponding "decl_file", right?

DW_TAG_compile_unit
  DW_AT_name "cu1"
  
  DW_TAG_class_type
    DW_AT_name "class1" 
    DW_AT_decl_file "file1"
    
    DW_TAG_subroutine
      DW_AT_name "method1"
      DW_AT_decl_file "file1"    <<<<<<<<<<<<<<<<<<<<<
    NULL
    
    DW_TAG_subroutine
      DW_AT_name "method2"
      DW_AT_decl_file "file2"     <<<<<<<<<<<<<<<<<<<<<
    NULL          
  NULL
NULL

i.e. "method1" and "method2" both should be placed into the compile unit for "file1", right?

4. what do you think: Would it be good to split current monolithic type table not on "decl_file" basis but on "buckets of dependend types" basis?

[clayborg]

In D96035#3314256, @avl wrote:

@clayborg I`ve done some research and have a couple of questions. Would you mind looking at them, please?

1. First thing is that I tried to model the case with separate compilation units for each declaration file and it seems that the overall impact is about 0,78% of debug_info+debug_abbrev size of the clang binary. Size of debug_info+debug_abbrev size is 158.7MB. Additional size, required for multiple compilation units, is 1.23MB(2051*(100+500)). 2051 is the number of files used in decl_file attributes, 100 bytes is the size of the compilation unit header, compilation unit die, size of namespace dies, line table header, 500 is the size of separate abbreviation table.

Thanks for trying this!

Under 1% (0.78%) , I would be ok with this overhead for really clean DWARF. To clarify: was this a all types for a specific DW_AT_decl_file in it own compile unit? Or was this for all files in the same directory?

2. Second thing is that I divided all types into independent buckets. The current type table for the clang binary takes approx 40MB. The maximal size of a bucket containing dependent types is 1.6Mb. So, if the original type table would be divided into multiple compilation units based on types dependency then each type unit might be 1,6Mb or less.
   
   Dividing types into independent buckets has several advantages: since they are not dependent on each other it is possible to handle them in a parallel/independent way. Another thing is that it potentially minimizes the number of loads which should be done. It loads all dependent types at once. In case types are not divided into independent buckets we would need to load all dependencies by several loads(i.e. whenever cross-CU reference is encountered).

Assuming we decided to split the current global type table on the "decl_file" basis as you suggested, What do you think of the following questions:

1. How the following situation should be handled:

Source DWARF:

      DW_TAG_compile_unit
        DW_AT_name "cu1"
     
0x100:  DW_TAG_namespace   <<<<<<<<<<<<<<<<<<<
          DW_AT_name "namespace1"
       
          DW_TAG_structure
            DW_AT_name "S1"
            DW_AT_decl_file "file1"
          NULL
         
          DW_TAG_structure
            DW_AT_name "S2"
            DW_AT_decl_file "file2"
          NULL
        NULL
     
        DW_TAG_import 0x100   <<<<<<<<<<<<<<<<<<<
      NULL

Result DWARF:

      DW_TAG_compile_unit
        DW_AT_name "type_table_file1"

0x200:  DW_TAG_namespace  <<<<<<<<<<<<<<<<<<<
          DW_AT_name "namespace1"
          
          DW_TAG_structure
            DW_AT_name "S1"
            DW_AT_decl_file "file1"
          NULL
          
        NULL

      DW_TAG_compile_unit
        DW_AT_name "type_table_file2"

0x300:  DW_TAG_namespace   <<<<<<<<<<<<<<<<<<<
          DW_AT_name "namespace1"

          DW_TAG_structure
            DW_AT_name "S2"
            DW_AT_decl_file "file2"
          NULL
          
        NULL

     DW_TAG_compile_unit
        DW_AT_name "cu1"

        DW_TAG_import 0x200 or 0x300 ?    <<<<<<<<<<<<<<<<<<<

We might have to include them both right? Do we actually see DWARF like this? But my initial guess is we would need them both if clients needed access to "S1" or "S2". Or if we can track what that DWARF accesses within the DW_TAG_import, maybe only the ones that are needed.

Which offset corresponding to the "namespace1" should be used? Any of them?

2. Would it be OK to split DW_TAG_module ?


Source DWARF:

DW_TAG_compile_unit
  DW_AT_name "cu1"
  
  DW_TAG_module             <<<<<<<<<<<<<<<<<<< 
    DW_AT_name "module1"    <<<<<<<<<<<<<<<<<<<
    
    DW_TAG_structure
      DW_AT_name "S1"
      DW_AT_decl_file "file1"
    NULL
   
    DW_TAG_structure
      DW_AT_name "S2"
      DW_AT_decl_file "file2"
    NULL

  NULL
  
NULL

Result DWARF:

DW_TAG_compile_unit
  DW_AT_name "type_table_file1"

  DW_TAG_module            <<<<<<<<<<<<<<<<<<<
    DW_AT_name "module1"   <<<<<<<<<<<<<<<<<<<
    
    DW_TAG_structure
      DW_AT_name "S1"
      DW_AT_decl_file "file1"
    NULL
    
  NULL
  
NULL
   
DW_TAG_compile_unit
  DW_AT_name "type_table_file2"

  DW_TAG_module            <<<<<<<<<<<<<<<<<<< 
    DW_AT_name "module1"   <<<<<<<<<<<<<<<<<<<
    
    DW_TAG_structure
      DW_AT_name "S2"
      DW_AT_decl_file "file2"
    NULL
    
  NULL
  
NULL

Is it OK, that DW_TAG_module would be split?

I believe it would be ok to split the module. I don't believe anyone refers to the module directly do they? Only things within and as long as we fix up those direct references then that should be ok. I have to admit I am not familiar with the DW_TAG_module as we don't use it much in LLDB AFAIK.

3. Only root types should be moved into compile unit for corresponding "decl_file", right?

DW_TAG_compile_unit
  DW_AT_name "cu1"
  
  DW_TAG_class_type
    DW_AT_name "class1" 
    DW_AT_decl_file "file1"
    
    DW_TAG_subroutine
      DW_AT_name "method1"
      DW_AT_decl_file "file1"    <<<<<<<<<<<<<<<<<<<<<
    NULL
    
    DW_TAG_subroutine
      DW_AT_name "method2"
      DW_AT_decl_file "file2"     <<<<<<<<<<<<<<<<<<<<<
    NULL          
  NULL
NULL

i.e. "method1" and "method2" both should be placed into the compile unit for "file1", right?

Yes, The entire contents of a class should go into the DW_AT_decl_file for the class itself. DW_TAG_subprogram entries that belong to the class should live with the class.

4. what do you think: Would it be good to split current monolithic type table not on "decl_file" basis but on "buckets of dependend types" basis?

If every type can do into its own compile unit for only an extra 0.78% cost, I find this totally acceptable and I would vote for this!

I am very excited about this feature and I will push for adoption of this within Facebook ASAP if this is viable, faster and also much better organized! The one thing to think about is seeing if we can make this work with "--update" option to dsymutil (which reparses the DWARF and re-emits it and the accelerator tables. This --update option could then be used to post produce debug innfo before it gets permanently archived to disk, so the smaller and better organized the better! So if we can get this working we will have a huge set of code to run this through and test it with!

[avl]

Thanks for trying this!

Under 1% (0.78%) , I would be ok with this overhead for really clean DWARF. To clarify: was this a all types for a specific DW_AT_decl_file in it own compile unit? Or was this for all files in the same directory?

That was all types for a specific DW_AT_decl_file in it own compile unit. I did not take into account duplication of file names in the line table, but I think overall size would still be within 1%(if there would not be other duplicated things).

Things might be much worse if we would need to duplicate this kind of dies for each compilation unit :

DW_TAG_const_type 
                DW_AT_type (0x000000000000004a "llvm::Twine")

DW_TAG_pointer_type
                DW_AT_type (0x000000000003c122 "const char")

We might have to include them both right? Do we actually see DWARF like this? But my initial guess is we would need them both if clients needed access to "S1" or "S2". Or if we can track what that DWARF accesses within the DW_TAG_import, maybe only the ones that are needed.

If we are going to include them both - then(in the real examples) it means inserting a dozens of DW_TAG_imported_module dies(for each used type).

We see DWARF like this very often:

using namespace llvm;

SmallSet<int, 3> Set;
SmallVector<const int *, 8> Ptrs;

Originally it looks like this:

      DW_TAG_compile_unit
        DW_AT_name "driver_cc1.cpp"
        
0x100:  DW_TAG_namespace   <<<<<<<<<<<<<<<<<<<
          DW_AT_name "llvm"
       
          DW_TAG_class_type
            DW_AT_name "SmallSet<int, 3>"
            DW_AT_decl_file "SmallSet.h"
          NULL
         
          DW_TAG_class_type
            DW_AT_name "SmallVector<const int *, 8>"
            DW_AT_decl_file "SmallVector.h"
          NULL
        NULL
     
        DW_TAG_imported_module
          DW_AT_import 0x100   <<<<<<<<<<<<<<<<<<<
      NULL

with new scheme it would be:

      DW_TAG_compile_unit
        DW_AT_name "type_table_SmallSet"

0x200:  DW_TAG_namespace  <<<<<<<<<<<<<<<<<<<
          DW_AT_name "llvm"
          
          DW_TAG_structure
            DW_AT_name "SmallSet<int, 3>"
            DW_AT_decl_file "SmallSet.h"
          NULL
          
        NULL

      DW_TAG_compile_unit
        DW_AT_name "type_table_SmallVector"

0x300:  DW_TAG_namespace   <<<<<<<<<<<<<<<<<<<
          DW_AT_name "llvm"

          DW_TAG_structure
            DW_AT_name "SmallVector<const int *, 8>"
            DW_AT_decl_file "SmallVector.h"
          NULL
          
        NULL

     DW_TAG_compile_unit
        DW_AT_name "driver_cc1.cpp"

        DW_TAG_imported_module   <<<<<<<<<<<<<<<<<<<
          DW_AT_import 0x200           <<<<<<<<<<<<<<<<<<<
        DW_TAG_imported_module    <<<<<<<<<<<<<<<<<<<
          DW_AT_import 0x300            <<<<<<<<<<<<<<<<<<<

It looks really overweight. These DW_TAG_imported_module dies will noticeable increase added size.

I am very excited about this feature and I will push for adoption of this within Facebook ASAP if this is viable, faster and also much better organized!

Thanks!

Thing, which I did not understand yet - why "splitting on decl_file basis" is so good? I understand the problem with big "type_table": we do not want to parse a lot of DWARF if lazily use only part of it. But, if less fragmented solution exists, would it be good to use it instead of "splitting on decl_file basis"?
Or "splitting on decl_file basis" is still better for some reason?

The one thing to think about is seeing if we can make this work with "--update" option to dsymutil (which reparses the DWARF and re-emits it and the accelerator tables. This --update option could then be used to post produce debug innfo before it gets permanently archived to disk, so the smaller and better organized the better! So if we can get this working we will have a huge set of code to run this through and test it with!

yes, I think we could make it possible that "--update" (or another replacement option) would reorganize DWARF. Though it would be good if we would do at as an additional change after this patch.

[clayborg]

Does the patch in this diff already contain the changes where each type in a separate compile unit that matches the DW_AT_decl_file? I would love to start running some tests, we have some dSYM files that are too large and are exceeding the 4GB boundary that I would like to test this patch on.

[avl]

In D96035#3370504, @clayborg wrote:

Does the patch in this diff already contain the changes where each type in a separate compile unit that matches the DW_AT_decl_file? I would love to start running some tests, we have some dSYM files that are too large and are exceeding the 4GB boundary that I would like to test this patch on.

No, it does not yet. Implementing this needs quite serious refactoring and would require some time(a month or two).

Another thing is that I am not sure that is the right way to go. This change seems would introduce significant fragmentation. My initial estimation(1%) was based on the assumption that a small number of DIEs would be duplicated. The more I think about it the more sources of duplication become clear. f.e. it would be necessary to insert a lot of DW_TAG_imported_module DIEs(see https://reviews.llvm.org/D96035#3314256 for the example ). Also, the implementation should be significantly complicated to add this kind of fragmentation. As a result size of the final DWARF would be increased, performance results for the patch would be worse(especially for single-thread mode, which seems to be still important).

I do not understand currently why "splitting on decl_file basis" is so good? (So that it worth degradation of performance and size results).

As an alternative, I am thinking of splitting type_table on a namespace basis. It would avoid the necessity of inserting multiple DW_TAG_imported_module DIEs. In clang, it would result in three compile units 17MB, 16MB, 7MB instead of one of 40Mb. What do you think of that idea - splitting the current type table into smaller units on a namespace basis?

[avl]

rebased. supported .debug_names.

[JDevlieghere]

In D96035#3291217, @dblaikie wrote:

Does anyone cares about single thread execution time? I personally don't.

If we do not then things are easier :-)

I think @JDevlieghere and @aprantl probably care about it just from a regression perspective - but I could be wrong. (I thought they'd expressed some concerns about that in this/other reviews previously).

Apple folks ^ have you got a sense of if/how much of a single threaded regression you'd want to accept in this work, for some multithreaded gains?

If it's purely time, I don't care too much about single threaded performance. When the lookForDiesToKeep algorithm was recursive we would occasionally run out of stack space because std::threads (underlying LLVM's thread pool) have a ridiculously small stack and no way to increase it. So we had this workaround in place for a while that if you specified -j1 it would not avoid the thread pool altogether. But since I converted that algorithm to a work list/state machine that's not really a concern anymore. The thing I did express concerns about what an earlier revision of this change that caused non-determinism in multithreaded mode which is not really negotiable but IIUC that's no longer the case with the current patch.

[JDevlieghere]

I'm (still) very excited about this. I'm really happy that this lives behind a flag which addresses most of my original concerns. Making this the default in dsymutil would require a bunch of qualification but this approach allows me to do that gradually. One thing I suggested earlier I think was to run the lldb test suite with the new flag and see if that breaks anything. Did you ever get around to that? That should give us an initial level of confidence. Jim expressed some concerns about trying to squeeze everything in the same compilation unit. Greg, do you share that concern? Do we expect this will negatively impact lldb's performance? It would probably be worthwhile to do a few performance experiments here as well to make sure this doesn't regress debugging performance.

I'm also very excited that Sony is considering upstreaming llvm-dva (based on Paul's comment in D124082). In the past we've often stuck to binary compatibility because there's no great way to qualify big changes to the algorithm. But as long as all of this lives behind a flag that's not a dealbreaker.

If we're happy with the approach I think we should land this an iterate on smaller changes that will be much easier to review. One reason I've been so quiet in this review is that it takes a long time to page in everything.

[avl]

I'm (still) very excited about this. I'm really happy that this lives behind a flag which addresses most of my original concerns. Making this the default in dsymutil would require a bunch of qualification but this approach allows me to do that gradually. One thing I suggested earlier I think was to run the lldb test suite with the new flag and see if that breaks anything. Did you ever get around to that?

yes, I did. This is the set of tests passed by this patch:

It passed llvm/test/tools/dsymutil lit tests.
It passed check-llvm, check-clang, check-lldb.
It passed llvm-dwarfdump --verify for all llvm&clang binaries.
It passed llvm-dwarfdump --types-compat-dump for all llvm&clang binaries.

Jim expressed some concerns about trying to squeeze everything in the same compilation unit. Greg, do you share that concern? Do we expect this will negatively impact lldb's performance? It would probably be worthwhile to do a few performance experiments here as well to make sure this doesn't regress debugging performance.

I will check the performance results and post them here. Though I think the performance degradation would not be significant.

Anyway, I think it might be a good idea to split a single type unit into several units. The question is - by which criteria? @clayborg suggested dividing it by declaration file basis. So that a separate unit would be created for every declaration file. I think that solution creates significant fragmentation. That fragmentation would lead to increased resulting DWARF size and complications of implementation.

I would prefer to integrate the current version of the patch and add functionality for separate type units later. Doing things in that way has the advantage of iterative development. If the patch would be integrated earlier then it would be tested/probed by a bigger amount of people. Also, it might be faster improved by several people. Another thing is that we will have more time to create a good concept for separated type units.

If we're happy with the approach I think we should land this an iterate on smaller changes that will be much easier to review. One reason I've been so quiet in this review is that it takes a long time to page in everything.

If we`re happy with the current approach I am ready to start the creation of separated patches.
I suggest integrating this version of the patch and do not wait for "several type units" implementation.
"several type units" implementation might be integrated as an addition to this patch.

[avl]

It would probably be worthwhile to do a few performance experiments here as well to make sure this doesn't regress debugging performance.

please, consider run-time/memory results for lldb used with clang built with various dsymutil options:

                        |    time, sec  |  Memory, Gbytes | .debug_info, bytes |
------------------------|---------------|-----------------|--------------------|
                        |               |                 |                    |
clang-original-noodr    |     8.58      |     1.82        |    1461723785      |
                        |               |                 |                    |
------------------------|---------------|--------------------------------------|
                        |               |                 |                    |
clang-original-odr      |     8.41      |     1.76        |     484472706      |
                        |               |                 |                    |
------------------------|---------------|--------------------------------------|
                        |               |                 |                    |
clang-dlnext-odr        |     8.75      |     1.98        |     155118011      |   <<<< single type unit size is 40495379 bytes.
                        |               |                 |                    |
------------------------|---------------|--------------------------------------|

tested lldb version:

lldb --version
lldb version 15.0.0git (https://github.com/llvm/llvm-project.git revision d9cea8d3a8fff86672174780312674871729578c)
  clang revision d9cea8d3a8fff86672174780312674871729578c
  llvm revision d9cea8d3a8fff86672174780312674871729578c

lldb run command:

/usr/bin/time -l lldb clang --source lldb-commands

set of commands passed to lldb through --source option:

b main
r
n
n
n
fr v
exit

If above set of commands is not very relevant for testing lldb performance then I will retest with another set.

[clayborg]

My main concern with creating one large unit for all types is the performance of the debugger when consuming this, but we don't need to fix this in the first iteration and can work on improving this later after we prove it is a problem. As long as the DWARF is in good shape and we are properly uniquing types in a much better way than before I am good.

Can we name the option something other than "--use-dlnext"? This option name won't mean much to users. Not sure what a good option name would be though but something that doesn't talk about internal classes that will be used would be better IMHO.

[JDevlieghere]

In D96035#3504468, @clayborg wrote:

Can we name the option something other than "--use-dlnext"? This option name won't mean much to users. Not sure what a good option name would be though but something that doesn't talk about internal classes that will be used would be better IMHO.

Agreed. How about a flag to specify the dwarf linker: --dwarf-linker={dsymutil,dwarfutil}. The respective tools would have the respective linkers as their default and we could add more options later (e.g. dwarfutil with separete type units).

With the flag renames this LGTM.

[avl]

In D96035#3505700, @JDevlieghere wrote:

In D96035#3504468, @clayborg wrote:

Can we name the option something other than "--use-dlnext"? This option name won't mean much to users. Not sure what a good option name would be though but something that doesn't talk about internal classes that will be used would be better IMHO.

Agreed. How about a flag to specify the dwarf linker: --dwarf-linker={dsymutil,dwarfutil}. The respective tools would have the respective linkers as their default and we could add more options later (e.g. dwarfutil with separete type units).

With the flag renames this LGTM.

probably, we would like to use both dwarflinkers with dsymutil and dwarfutil. Thus, we do not want to name tools in the option. How about these variants:

--dwarf-linker={classic,mt}  (mt is multy-thread)

--dwarf-linker={classic,with-joined-types}

--dwarf-linker={classic,experimental}

?

Also, to check my understanding, We are not going to integrate this patch as a single huge change. Instead, I will separate it to the series of reviews and we are going to integrated them one by one, correct?

[clayborg]

probably, we would like to use both dwarflinkers with dsymutil and dwarfutil. Thus, we do not want to name tools in the option. How about these variants:

--dwarf-linker={classic,mt}  (mt is multy-thread)

--dwarf-linker={classic,with-joined-types}

--dwarf-linker={classic,experimental}

How about:

--linker=apple
--linker=llvm

The "apple" would be the classic Apple linker with all of its compatability
The new "llvm" would be the new linker with the better type uniquing + other features that will get added. We would need these described in the option help text.

Any multi-threaded options should be separate options if they are not already covered by existing options.

Do we have the ability to change how types are emitted? Like, can we use the "classic" linker with new type uniquing options ("with-joined-types" or "experimental")? I am trying to figure out the three options you specified above

[avl]

How about:

--linker=apple
--linker=llvm

The "apple" would be the classic Apple linker with all of its compatability
The new "llvm" would be the new linker with the better type uniquing + other features that will get added. We would need these described in the option help text.

These are fine to me.

Any multi-threaded options should be separate options if they are not already covered by existing options.

Do we have the ability to change how types are emitted? Like, can we use the "classic" linker with new type uniquing options ("with-joined-types" or "experimental")? I am trying to figure out the three options you specified above

No, we do not have the ability to change how types are emitted. "classic"/"apple" linker generates it one way. "experimental"/"llvm" generates it in other way.

These variants: "mt", "with-joined-types", "experimental" are different suggestions for the name of new linker. There should be used only one of them.

"mt" - supposed to note that new linker utilizes multi-thread execution heavier.
"with-joined-types" - supposed to note that new linker uses other approach to type deduplication.
"experimental" - supposed to note that new linker is an experiment yet.

[dreampiggy]

Hi.

I try to pick the code for testing the performance. However, I found the new dwarflinker-next, will crash at some specify code path

See crash stacktrace here:

I try to debug and found the reason:

during analysis stage, NamesGenerator.assignName(CU, DieIdx) can not assign a valid name for some unhandled DIEEntry tag, like

+ DW_TAG_thrown_type

This will cause the final CU.TypeInfo.TypeName be null, and crash at the TypePool.insert during ODR stage

Any solution for this ? Should we just mark it as bad type with the raw hex with the default case (current not)? Or add some more handle cases ?

[avl]

In D96035#3681927, @dreampiggy wrote:

Hi.

I try to pick the code for testing the performance. However, I found the new dwarflinker-next, will crash at some specify code path

Hi @dreampiggy,

The patch is a bit outdated. The upstream code has been changed since I built it last time. I will rebase it.

See crash stacktrace here:

I try to debug and found the reason:

during analysis stage, NamesGenerator.assignName(CU, DieIdx) can not assign a valid name for some unhandled DIEEntry tag, like

+ DW_TAG_thrown_type

This will cause the final CU.TypeInfo.TypeName be null, and crash at the TypePool.insert during ODR stage

Any solution for this ? Should we just mark it as bad type with the raw hex with the default case (current not)? Or add some more handle cases ?

We need to handle more cases. DW_TAG_thrown_type is currently not handled by new DWARFLinker(as well as by old DWARFLinker). It should be handled at least inside SyntheticTypeNameBuilder::analyzeDieForTypeName().
I will rebase and add handling of DW_TAG_thrown_type in a week.
Thanks for the analysis.

[dreampiggy]

DW_TAG_thrown_type is currently not handled by new DWARFLinker(as well as by old DWARFLinker)

Yes. But seems the old DWARFLinker DIECloner does not hit any assert and crash. Seems the new "artificial compile unit" based on types name will fail on this case ?

[avl]

In D96035#3684041, @dreampiggy wrote:

DW_TAG_thrown_type is currently not handled by new DWARFLinker(as well as by old DWARFLinker)

Yes. But seems the old DWARFLinker DIECloner does not hit any assert and crash. Seems the new "artificial compile unit" based on types name will fail on this case ?

yes, it will fail. Thus, it is necessary to add(other than adding specific handling for DW_TAG_thrown_type) default handler for unknown DW_TAG_* die. So that the patch do not crash in such a case. Will do this with nearest rebasing.

[clayborg]

In D96035#3685472, @avl wrote:

In D96035#3684041, @dreampiggy wrote:

DW_TAG_thrown_type is currently not handled by new DWARFLinker(as well as by old DWARFLinker)

Yes. But seems the old DWARFLinker DIECloner does not hit any assert and crash. Seems the new "artificial compile unit" based on types name will fail on this case ?

yes, it will fail. Thus, it is necessary to add(other than adding specific handling for DW_TAG_thrown_type) default handler for unknown DW_TAG_* die. So that the patch do not crash in such a case. Will do this with nearest rebasing.

It should be as easy as classifying DW_TAG_thrown_type as a type die right?

[dblaikie]

In D96035#3685472, @avl wrote:

In D96035#3684041, @dreampiggy wrote:

DW_TAG_thrown_type is currently not handled by new DWARFLinker(as well as by old DWARFLinker)

Yes. But seems the old DWARFLinker DIECloner does not hit any assert and crash. Seems the new "artificial compile unit" based on types name will fail on this case ?

yes, it will fail. Thus, it is necessary to add(other than adding specific handling for DW_TAG_thrown_type) default handler for unknown DW_TAG_* die. So that the patch do not crash in such a case. Will do this with nearest rebasing.

So it sounds like there's a list of attributes that may be type attributes - is it possible to detect type references by the type of the referenced DIE instead of the attribute used to reference them? (or both?) or would DW_AT_sibling break that by causing a sibling attribute to be treated differently just because the sibling was a type?

[avl]

In D96035#3685745, @dblaikie wrote:

In D96035#3685472, @avl wrote:

In D96035#3684041, @dreampiggy wrote:

DW_TAG_thrown_type is currently not handled by new DWARFLinker(as well as by old DWARFLinker)

Yes. But seems the old DWARFLinker DIECloner does not hit any assert and crash. Seems the new "artificial compile unit" based on types name will fail on this case ?

yes, it will fail. Thus, it is necessary to add(other than adding specific handling for DW_TAG_thrown_type) default handler for unknown DW_TAG_* die. So that the patch do not crash in such a case. Will do this with nearest rebasing.

So it sounds like there's a list of attributes that may be type attributes - is it possible to detect type references by the type of the referenced DIE instead of the attribute used to reference them? (or both?) or would DW_AT_sibling break that by causing a sibling attribute to be treated differently just because the sibling was a type?

This exact case is not about how to detect whether an attribute references type DIE or not.
(Though attributes referencing type die currently implemented as the predefined list: DW_AT_type, DW_AT_containing_type, DW_AT_specification, DW_AT_abstract_origin, DW_AT_import.)
This case is about how to decide which DIE referencing type DIE should be deduplicated.

f.e. The patch deduplicates this kind of DIE:

DW_TAG_pointer_type 
   DW_AT_type -> ref to type DIE.

To be able to deduplicate above DW_TAG_pointer_type, the patch creates a type descriptor
for the DIE DW_TAG_pointer_type. Any reference to the above pointer will be resolved to the
reference to this type descriptor.

There are some DIEs referencing type DIEs that should not be deduplicated.
f.e. DW_TAG_call_site or DW_TAG_call_site_parameter. Since they are used in
subroutines that are not moved into the "artificial type unit".

Currently there is a list of such DIEs inside SyntheticTypeNameBuilder::analyzeDieForTypeName().
DIEs handled in analyzeDieForTypeName() will be assigned a name and maybe deduplicated later.
DW_TAG_thrown_type was missed there. Probably, instead of just a list of DIEs that might be deduplicated,
it is possible to create a more general rule. f.e. if some DIE is a child of another DIE that is already known
to be deduplicated then such DIE should be considered as a deduplication candidate also. This rule should
cover the DW_TAG_thrown_type case.

Speaking of determining attributes referencing type DIEs: solution detecting type references by the type of the referenced DIE might be too expensive from the performance point of view. Resolving DIE references is quite expensive operation.

[avl]

In D96035#3685513, @clayborg wrote:

In D96035#3685472, @avl wrote:

In D96035#3684041, @dreampiggy wrote:

DW_TAG_thrown_type is currently not handled by new DWARFLinker(as well as by old DWARFLinker)

Yes. But seems the old DWARFLinker DIECloner does not hit any assert and crash. Seems the new "artificial compile unit" based on types name will fail on this case ?

yes, it will fail. Thus, it is necessary to add(other than adding specific handling for DW_TAG_thrown_type) default handler for unknown DW_TAG_* die. So that the patch do not crash in such a case. Will do this with nearest rebasing.

It should be as easy as classifying DW_TAG_thrown_type as a type die right?

Yes, we can classify DW_TAG_thrown_type as a deduplication candidate and this concrete case should be cured.

Additionally we can extend the rule for deduplication candidates in such way: All children of the known deduplication candidate should also be considered as a deduplication candidates - that allows to not create a fixed list.

[dblaikie]

In D96035#3686890, @avl wrote:

In D96035#3685745, @dblaikie wrote:

In D96035#3685472, @avl wrote:

In D96035#3684041, @dreampiggy wrote:

DW_TAG_thrown_type is currently not handled by new DWARFLinker(as well as by old DWARFLinker)

Yes. But seems the old DWARFLinker DIECloner does not hit any assert and crash. Seems the new "artificial compile unit" based on types name will fail on this case ?

yes, it will fail. Thus, it is necessary to add(other than adding specific handling for DW_TAG_thrown_type) default handler for unknown DW_TAG_* die. So that the patch do not crash in such a case. Will do this with nearest rebasing.

So it sounds like there's a list of attributes that may be type attributes - is it possible to detect type references by the type of the referenced DIE instead of the attribute used to reference them? (or both?) or would DW_AT_sibling break that by causing a sibling attribute to be treated differently just because the sibling was a type?

This exact case is not about how to detect whether an attribute references type DIE or not.
(Though attributes referencing type die currently implemented as the predefined list: DW_AT_type, DW_AT_containing_type, DW_AT_specification, DW_AT_abstract_origin, DW_AT_import.)
This case is about how to decide which DIE referencing type DIE should be deduplicated.

f.e. The patch deduplicates this kind of DIE:

DW_TAG_pointer_type 
   DW_AT_type -> ref to type DIE.

To be able to deduplicate above DW_TAG_pointer_type, the patch creates a type descriptor
for the DIE DW_TAG_pointer_type. Any reference to the above pointer will be resolved to the
reference to this type descriptor.

There are some DIEs referencing type DIEs that should not be deduplicated.
f.e. DW_TAG_call_site or DW_TAG_call_site_parameter. Since they are used in
subroutines that are not moved into the "artificial type unit".

Currently there is a list of such DIEs inside SyntheticTypeNameBuilder::analyzeDieForTypeName().
DIEs handled in analyzeDieForTypeName() will be assigned a name and maybe deduplicated later.
DW_TAG_thrown_type was missed there. Probably, instead of just a list of DIEs that might be deduplicated,
it is possible to create a more general rule. f.e. if some DIE is a child of another DIE that is already known
to be deduplicated then such DIE should be considered as a deduplication candidate also. This rule should
cover the DW_TAG_thrown_type case.

Speaking of determining attributes referencing type DIEs: solution detecting type references by the type of the referenced DIE might be too expensive from the performance point of view. Resolving DIE references is quite expensive operation.

Ah, OK - I'm mostly following. Thanks for helping me understand.

[avl]

rebased, changed use-dlnext into --linker=[Apple,LLVM], added handling DW_TAG_thrown_type, added handling default case.

llvm/include/llvm/CodeGen/NonRelocatableStringpool.h

public:
   StringRef internString(StringRef S);
 
   uint64_t getSize() { return CurrentEndOffset; }
 
   /// Return the list of strings to be emitted. This does not contain the
   /// strings which were added via internString only.
   std::vector<DwarfStringPoolEntryRef> getEntriesForEmission() const;
 
+  /// Clear contents.
+  void clear() {
+    Strings.clear();
+    CurrentEndOffset = 0;
+    NumEntries = 0;
+    Translator = nullptr;
+  }
+
 private:
   MapTy Strings;
   uint64_t CurrentEndOffset = 0;
   unsigned NumEntries = 0;
   DwarfStringPoolEntryRef EmptyString;
   std::function<StringRef(StringRef Input)> Translator;
 };
 

llvm/include/llvm/DWARFLinker/DWARFLinker.h

virtual bool applyValidRelocs(MutableArrayRef<char> Data, uint64_t BaseOffset,
                                 bool IsLittleEndian) = 0;
 
   /// Relocate the given address offset if a valid relocation exists.
   virtual llvm::Expected<uint64_t> relocateIndexedAddr(uint64_t StartOffset,
                                                        uint64_t EndOffset) = 0;
 
   /// Returns all valid functions address ranges(i.e., those ranges
   /// which points to sections with code).
-  virtual RangesTy &getValidAddressRanges() = 0;
+  virtual void getValidAddressRanges(RangesTy *&Ranges) = 0;
 
   /// Erases all data.
   virtual void clear() = 0;
 };
 
 /// DwarfEmitter presents interface to generate all debug info tables.
 class DwarfEmitter {
 public:

 };
 
 using UnitListTy = std::vector<std::unique_ptr<CompileUnit>>;
 
 /// this class represents DWARF information for source file
 /// and it`s address map.
 class DWARFFile {
 public:
-  DWARFFile(StringRef Name, DWARFContext *Dwarf, AddressesMap *Addresses,
-            const std::vector<std::string> &Warnings)
-      : FileName(Name), Dwarf(Dwarf), Addresses(Addresses), Warnings(Warnings) {
-  }
+  DWARFFile(StringRef Name, sys::TimePoint<std::chrono::seconds> Timestamp,
+            std::unique_ptr<DWARFContext> Dwarf,
+            std::unique_ptr<AddressesMap> Addresses,
+            const std::vector<std::string> &Warnings,
+            std::function<void(StringRef FileName,
+                               sys::TimePoint<std::chrono::seconds> Timestamp)>)
+      : FileName(Name), Timestamp(Timestamp), Dwarf(std::move(Dwarf)),
+        Addresses(std::move(Addresses)), Warnings(Warnings) {}
 
   /// object file name.
   StringRef FileName;
+  /// object file timestamp.
+  sys::TimePoint<std::chrono::seconds> Timestamp;
   /// source DWARF information.
-  DWARFContext *Dwarf = nullptr;
+  std::unique_ptr<DWARFContext> Dwarf;
   /// helpful address information(list of valid address ranges, relocations).
-  AddressesMap *Addresses = nullptr;
+  std::unique_ptr<AddressesMap> Addresses;
   /// warnings for object file.
   const std::vector<std::string> &Warnings;
 };
 
 typedef std::function<void(const Twine &Warning, StringRef Context,
                            const DWARFDie *DIE)>
     messageHandler;
 typedef std::function<ErrorOr<DWARFFile &>(StringRef ContainerName,

llvm/include/llvm/DWARFLinkerNext/DWARFLinker.h

 //===- DWARFLinker.h --------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#ifndef LLVM_DWARFLINKER_DWARFLINKER_H
-#define LLVM_DWARFLINKER_DWARFLINKER_H
+#ifndef LLVM_DWARFLINKERNEXT_DWARFLINKER_H
+#define LLVM_DWARFLINKERNEXT_DWARFLINKER_H
 
 #include "llvm/CodeGen/AccelTable.h"
 #include "llvm/CodeGen/NonRelocatableStringpool.h"
-#include "llvm/DWARFLinker/DWARFLinkerDeclContext.h"
+#include "llvm/DWARFLinkerNext/DWARFLinkerDeclContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFCompileUnit.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/MC/MCDwarf.h"
+#include <atomic>
 #include <map>
+#include <mutex>
 
 namespace llvm {
 
-enum class DwarfLinkerClient { Dsymutil, LLD, General };
+namespace dwarflinker {
 
-/// The kind of accelerator tables we should emit.
-enum class AccelTableKind {
-  Apple,   ///< .apple_names, .apple_namespaces, .apple_types, .apple_objc.
-  Dwarf,   ///< DWARF v5 .debug_names.
-  Default, ///< Dwarf for DWARF5 or later, Apple otherwise.
-};
+enum class DwarfLinkerClient { Dsymutil, LLD, General };
 
 /// Partial address range. Besides an offset, only the
 /// HighPC is stored. The structure is stored in a map where the LowPC is the
 /// key.
 struct ObjFileAddressRange {
   /// Function HighPC.
   uint64_t HighPC;
   /// Offset to apply to the linked address.
   /// should be 0 for not-linked object file.
   int64_t Offset;
 
   ObjFileAddressRange(uint64_t EndPC, int64_t Offset)
       : HighPC(EndPC), Offset(Offset) {}
 
   ObjFileAddressRange() : HighPC(0), Offset(0) {}
 };
 
 /// Map LowPC to ObjFileAddressRange.
 using RangesTy = std::map<uint64_t, ObjFileAddressRange>;
 
 /// AddressesMap represents information about valid addresses used
 /// by debug information. Valid addresses are those which points to
 /// live code sections. i.e. relocations for these addresses point
 /// into sections which would be/are placed into resulting binary.
+///
+/// May be used asynchroniously for reading.
 class AddressesMap {
 public:
   virtual ~AddressesMap();
 
   /// Returns true if represented addresses are from linked file.
   /// Returns false if represented addresses are from not-linked
   /// object file.
   virtual bool areRelocationsResolved() const = 0;
public:
   ///
   /// \returns true and sets Info.InDebugMap if it is the case.
   virtual bool hasLiveAddressRange(const DWARFDie &DIE,
                                    CompileUnit::DIEInfo &Info) = 0;
 
   /// Apply the valid relocations to the buffer \p Data, taking into
   /// account that Data is at \p BaseOffset in the debug_info section.
   ///
+  /// This function must be called with monotonic \p BaseOffset values.
+  ///
   /// \returns true whether any reloc has been applied.
   virtual bool applyValidRelocs(MutableArrayRef<char> Data, uint64_t BaseOffset,
                                 bool IsLittleEndian) = 0;
 
   /// Relocate the given address offset if a valid relocation exists.
   virtual llvm::Expected<uint64_t> relocateIndexedAddr(uint64_t StartOffset,
                                                        uint64_t EndOffset) = 0;
 
   /// Returns all valid functions address ranges(i.e., those ranges
   /// which points to sections with code).
-  virtual RangesTy &getValidAddressRanges() = 0;
+  virtual void getValidAddressRanges(RangesTy *&Ranges) = 0;
 
   /// Erases all data.
   virtual void clear() = 0;
 };
 
 /// DwarfEmitter presents interface to generate all debug info tables.
+///
+/// Can not be used asynchroniously.
 class DwarfEmitter {
 public:
   virtual ~DwarfEmitter();
 
+  virtual const Triple &getTargetTriple() const = 0;
+
   /// Emit DIE containing warnings.
   virtual void emitPaperTrailWarningsDie(DIE &Die) = 0;
 
   /// Emit section named SecName with data SecData.
   virtual void emitSectionContents(StringRef SecData, StringRef SecName) = 0;
 
   /// Emit the abbreviation table \p Abbrevs to the debug_abbrev section.
   virtual void
   emitAbbrevs(const std::vector<std::unique_ptr<DIEAbbrev>> &Abbrevs,
               unsigned DwarfVersion) = 0;
 
   /// Emit the string table described by \p Pool.
   virtual void emitStrings(const NonRelocatableStringpool &Pool) = 0;
 
   /// Emit DWARF debug names.
-  virtual void
-  emitDebugNames(AccelTable<DWARF5AccelTableStaticData> &Table) = 0;
+  virtual void emitDebugNames(AccelTable<DWARF5AccelTableStaticData> &) {}
 
   /// Emit Apple namespaces accelerator table.
   virtual void
-  emitAppleNamespaces(AccelTable<AppleAccelTableStaticOffsetData> &Table) = 0;
+  emitAppleNamespaces(AccelTable<AppleAccelTableStaticOffsetData> &) {}
 
   /// Emit Apple names accelerator table.
-  virtual void
-  emitAppleNames(AccelTable<AppleAccelTableStaticOffsetData> &Table) = 0;
+  virtual void emitAppleNames(AccelTable<AppleAccelTableStaticOffsetData> &) {}
 
   /// Emit Apple Objective-C accelerator table.
-  virtual void
-  emitAppleObjc(AccelTable<AppleAccelTableStaticOffsetData> &Table) = 0;
+  virtual void emitAppleObjc(AccelTable<AppleAccelTableStaticOffsetData> &) {}
 
   /// Emit Apple type accelerator table.
-  virtual void
-  emitAppleTypes(AccelTable<AppleAccelTableStaticTypeData> &Table) = 0;
+  virtual void emitAppleTypes(AccelTable<AppleAccelTableStaticTypeData> &) {}
 
   /// Emit debug_ranges for \p FuncRange by translating the
   /// original \p Entries.
   virtual void emitRangesEntries(
       int64_t UnitPcOffset, uint64_t OrigLowPc,
       const FunctionIntervals::const_iterator &FuncRange,
       const std::vector<DWARFDebugRangeList::RangeListEntry> &Entries,
       unsigned AddressSize) = 0;
public:
   /// Emit an FDE with data \p Bytes.
   virtual void emitFDE(uint32_t CIEOffset, uint32_t AddreSize, uint32_t Address,
                        StringRef Bytes) = 0;
 
   /// Emit the debug_loc contribution for \p Unit by copying the entries from
   /// \p Dwarf and offsetting them. Update the location attributes to point to
   /// the new entries.
   virtual void emitLocationsForUnit(
-      const CompileUnit &Unit, DWARFContext &Dwarf,
+      CompileUnit &Unit, DWARFContext &Dwarf,
       std::function<void(StringRef, SmallVectorImpl<uint8_t> &)>
           ProcessExpr) = 0;
 
   /// Emit the compilation unit header for \p Unit in the
   /// debug_info section.
   ///
   /// As a side effect, this also switches the current Dwarf version
   /// of the MC layer to the one of U.getOrigUnit().
-  virtual void emitCompileUnitHeader(CompileUnit &Unit,
-                                     unsigned DwarfVersion) = 0;
+  virtual void emitCompileUnitHeader(CompileUnit &Unit) = 0;
 
   /// Recursively emit the DIE tree rooted at \p Die.
   virtual void emitDIE(DIE &Die) = 0;
 
   /// Returns size of generated .debug_line section.
   virtual uint64_t getLineSectionSize() const = 0;
 
   /// Returns size of generated .debug_frame section.
   virtual uint64_t getFrameSectionSize() const = 0;
 
   /// Returns size of generated .debug_ranges section.
   virtual uint64_t getRangesSectionSize() const = 0;
 
   /// Returns size of generated .debug_info section.
   virtual uint64_t getDebugInfoSectionSize() const = 0;
 };
 
 using UnitListTy = std::vector<std::unique_ptr<CompileUnit>>;
+using UnloadCallback = std::function<void(
+    StringRef FileName, sys::TimePoint<std::chrono::seconds> Timestamp)>;
 
 /// this class represents DWARF information for source file
 /// and it`s address map.
+///
+/// May be used asynchroniously for reading.
 class DWARFFile {
 public:
-  DWARFFile(StringRef Name, DWARFContext *Dwarf, AddressesMap *Addresses,
-            const std::vector<std::string> &Warnings)
-      : FileName(Name), Dwarf(Dwarf), Addresses(Addresses), Warnings(Warnings) {
+  DWARFFile(StringRef Name, sys::TimePoint<std::chrono::seconds> Timestamp,
+            std::unique_ptr<DWARFContext> Dwarf,
+            std::unique_ptr<AddressesMap> Addresses,
+            const std::vector<std::string> &Warnings, UnloadCallback unloadFunc)
+      : FileName(Name), Timestamp(Timestamp), Dwarf(std::move(Dwarf)),
+        Addresses(std::move(Addresses)), Warnings(Warnings),
+        unloadFunc(unloadFunc) {
+    if (Dwarf)
+      Endianess = Dwarf->isLittleEndian() ? support::endianness::little
+                                          : support::endianness::big;
   }
 
   /// object file name.
   StringRef FileName;
+  /// object file timestamp.
+  sys::TimePoint<std::chrono::seconds> Timestamp;
   /// source DWARF information.
-  DWARFContext *Dwarf = nullptr;
+  std::unique_ptr<DWARFContext> Dwarf;
   /// helpful address information(list of valid address ranges, relocations).
-  AddressesMap *Addresses = nullptr;
+  std::unique_ptr<AddressesMap> Addresses;
   /// warnings for object file.
   const std::vector<std::string> &Warnings;
+  /// Endiannes of source DWARF information.
+  support::endianness Endianess = support::endianness::little;
+
+  /// Callback to the module keeping object file to unload.
+  UnloadCallback unloadFunc;
+
+  /// Unloads object file and corresponding AddressesMap and Dwarf Context.
+  void unload() {
+    Addresses.reset();
+    Dwarf.reset();
+
+    if (unloadFunc)
+      unloadFunc(FileName, Timestamp);
+  }
 };
 
 typedef std::function<void(const Twine &Warning, StringRef Context,
                            const DWARFDie *DIE)>
     messageHandler;
 typedef std::function<ErrorOr<DWARFFile &>(StringRef ContainerName,
                                            StringRef Path)>
     objFileLoader;
 typedef std::map<std::string, std::string> swiftInterfacesMap;
 typedef std::map<std::string, std::string> objectPrefixMap;
 
-/// The core of the Dwarf linking logic.
-///
-/// The generation of the dwarf information from the object files will be
-/// driven by the selection of 'root DIEs', which are DIEs that
-/// describe variables or functions that resolves to the corresponding
-/// code section(and thus have entries in the Addresses map). All the debug
-/// information that will be generated(the DIEs, but also the line
-/// tables, ranges, ...) is derived from that set of root DIEs.
-///
-/// The root DIEs are identified because they contain relocations that
-/// points to code section(the low_pc for a function, the location for
-/// a variable). These relocations are called ValidRelocs in the
-/// AddressesInfo and are gathered as a very first step when we start
-/// processing a object file.
-class DWARFLinker {
-public:
-  DWARFLinker(DwarfEmitter *Emitter,
-              DwarfLinkerClient ClientID = DwarfLinkerClient::General)
-      : TheDwarfEmitter(Emitter), DwarfLinkerClientID(ClientID) {}
+class DwarfStreamer;
 
-  /// Add object file to be linked.
-  void addObjectFile(DWARFFile &File);
+/// Hold the input and output of the debug info size in bytes.
+struct DebugInfoSize {
+  uint64_t Input;
+  uint64_t Output;
+};
 
-  /// Link debug info for added objFiles. Object
-  /// files are linked all together.
-  bool link();
+/// Keeps track of data associated with one object during linking.
+struct LinkContext {
+  struct FDERecordPatch {
+    FDERecordPatch(uint64_t OffsetToFDERecord)
+        : OffsetToFDERecord(OffsetToFDERecord) {}
+    /// Offset to the FDE record which need to update reference to CIE record.
+    uint64_t OffsetToFDERecord;
+  };
+  /// Object file descriptor.
+  DWARFFile &File;
 
-  /// A number of methods setting various linking options:
+  /// Set of Compilation Units(may be accessed asynchroniously for reading).
+  UnitListTy CompileUnits;
 
-  /// Allows to generate log of linking process to the standard output.
-  void setVerbosity(bool Verbose) { Options.Verbose = Verbose; }
+  /// Set of Compile Units for modules. May be updated asynchroniously.
+  UnitListTy ModulesCompileUnits;
+  std::mutex ModulesCompileUnitsMutex;
 
-  /// Print statistics to standard output.
-  void setStatistics(bool Statistics) { Options.Statistics = Statistics; }
+  /// Semaphore for Compile Units line tables.
+  std::mutex LineTablesMutex;
 
-  /// Do not emit linked dwarf info.
-  void setNoOutput(bool NoOut) { Options.NoOutput = NoOut; }
+  /// Size of Debug info before optimizing.
+  uint64_t OriginalDebugInfoSize = 0;
 
-  /// Do not unique types according to ODR.
-  void setNoODR(bool NoODR) { Options.NoODR = NoODR; }
+  /// Set of patches for FDE records(containing references to the CIE
+  /// records, which should be updated). After we glued resulting file
+  /// from the pieces corresponding to the source object files, we need
+  /// to update references to the CIE records with proper offsets.
+  std::vector<FDERecordPatch> FDERecords;
 
-  /// update existing DWARF info(for the linked binary).
-  void setUpdate(bool Update) { Options.Update = Update; }
+  /// Data, common for all compilation units.
+  SmallString<4096> OutDebugInfoBytes;
 
-  /// Use specified number of threads for parallel files linking.
-  void setNumThreads(unsigned NumThreads) { Options.Threads = NumThreads; }
+  support::endianness Endianess = support::endianness::little;
 
-  /// Set kind of accelerator tables to be generated.
-  void setAccelTableKind(AccelTableKind Kind) {
-    Options.TheAccelTableKind = Kind;
-  }
+  std::array<uint64_t, CompileUnit::TableKindsNum> TableOffsets = {0};
 
-  /// Set prepend path for clang modules.
-  void setPrependPath(const std::string &Ppath) { Options.PrependPath = Ppath; }
+  LinkContext(DWARFFile &File) : File(File) {
+    if (File.Dwarf) {
+      if (!File.Dwarf->compile_units().empty())
+        CompileUnits.reserve(File.Dwarf->getNumCompileUnits());
 
-  /// Set translator which would be used for strings.
-  void
-  setStringsTranslator(std::function<StringRef(StringRef)> StringsTranslator) {
-    this->StringsTranslator = StringsTranslator;
+      Endianess = File.Dwarf->isLittleEndian() ? support::endianness::little
+                                               : support::endianness::big;
     }
-
-  /// Set estimated objects files amount, for preliminary data allocation.
-  void setEstimatedObjfilesAmount(unsigned ObjFilesNum) {
-    ObjectContexts.reserve(ObjFilesNum);
   }
 
-  /// Set warning handler which would be used to report warnings.
-  void setWarningHandler(messageHandler Handler) {
-    Options.WarningHandler = Handler;
+  /// Add Compile Unit corresponding to the module.
+  void addModulesCompileUnit(std::unique_ptr<CompileUnit> Unit) {
+    std::unique_lock<std::mutex> Guard(ModulesCompileUnitsMutex);
+
+    ModulesCompileUnits.insert(ModulesCompileUnits.begin(), std::move(Unit));
   }
 
-  /// Set error handler which would be used to report errors.
-  void setErrorHandler(messageHandler Handler) {
-    Options.ErrorHandler = Handler;
+  /// Returns set of offsets to FDE records(referencing CIE records)
+  /// which should be updated after final offsets are become known.
+  std::vector<FDERecordPatch> getFDERecordsPatches() {
+    return std::move(FDERecords);
   }
 
-  /// Set object files loader which would be used to load
-  /// additional objects for splitted dwarf.
-  void setObjFileLoader(objFileLoader Loader) {
-    Options.ObjFileLoader = Loader;
+  /// Remember FDE record(we need to update CIE record offset in it).
+  /// It should be updated after final offsets are become known.
+  void noteFDERecordToPatch(FDERecordPatch Patch) {
+    FDERecords.emplace_back(Patch);
   }
 
-  /// Set map for Swift interfaces.
-  void setSwiftInterfacesMap(swiftInterfacesMap *Map) {
-    Options.ParseableSwiftInterfaces = Map;
+  /// Returns Endiannes of the source DWARF information.
+  support::endianness getEndianness() { return Endianess; }
+
+  /// Link context might contain debug tables which are common for the whole
+  /// object files(invariant tables, debug_frame). This method returns start
+  /// offset inside final DWARF file for such tables.
+  uint64_t getStartOffset(CompileUnit::DebugTableKind TableKind) const {
+    return TableOffsets[static_cast<typename std::underlying_type<
+        CompileUnit::DebugTableKind>::type>(TableKind)];
   }
 
-  /// Set prefix map for objects.
-  void setObjectPrefixMap(objectPrefixMap *Map) {
-    Options.ObjectPrefixMap = Map;
+  /// Set start offset for debug tables common for the whole object file.
+  void setStartOffset(CompileUnit::DebugTableKind TableKind, uint64_t Offset) {
+    TableOffsets[static_cast<typename std::underlying_type<
+        CompileUnit::DebugTableKind>::type>(TableKind)] = Offset;
   }
 
+  CompileUnit *getUnitForOffset(CompileUnit &CU, uint64_t Offset) const;
+};
+
+/// The core of the Dwarf linking logic.
+///
+/// The generation of the dwarf information from the object files will be
+/// driven by the selection of 'root DIEs', which are DIEs that
+/// describe variables or functions that resolves to the corresponding
+/// code section(and thus have entries in the Addresses map). All the debug
+/// information that will be generated(the DIEs, but also the line
+/// tables, ranges, ...) is derived from that set of root DIEs.
+///
+/// The root DIEs are identified because they contain relocations that
+/// points to code section(the low_pc for a function, the location for
+/// a variable). These relocations are gathered as a very first step
+/// when we start processing a object file by AddressesMap.
+///
+/// Can not be used asynchroniously.
+class DWARFLinker {
 private:
   /// Flags passed to DwarfLinker::lookForDIEsToKeep
   enum TraversalFlags {
     TF_Keep = 1 << 0,            ///< Mark the traversed DIEs as kept.
     TF_InFunctionScope = 1 << 1, ///< Current scope is a function scope.
     TF_DependencyWalk = 1 << 2,  ///< Walking the dependencies of a kept DIE.
     TF_ParentWalk = 1 << 3,      ///< Walking up the parents of a kept DIE.
     TF_ODR = 1 << 4,             ///< Use the ODR while keeping dependents.
     TF_SkipPC = 1 << 5,          ///< Skip all location attributes.
   };
 
-  /// The  distinct types of work performed by the work loop.
+  /// The distinct types of work performed by the work loop.
   enum class WorklistItemType {
     /// Given a DIE, look for DIEs to be kept.
     LookForDIEsToKeep,
     /// Given a DIE, look for children of this DIE to be kept.
     LookForChildDIEsToKeep,
     /// Given a DIE, look for DIEs referencing this DIE to be kept.
     LookForRefDIEsToKeep,
     /// Given a DIE, look for parent DIEs to be kept.
WorklistItem(DWARFDie Die, CompileUnit &CU, WorklistItemType T,
                  CompileUnit::DIEInfo *OtherInfo = nullptr)
         : Die(Die), Type(T), CU(CU), Flags(0), OtherInfo(OtherInfo) {}
 
     WorklistItem(unsigned AncestorIdx, CompileUnit &CU, unsigned Flags)
         : Die(), Type(WorklistItemType::LookForParentDIEsToKeep), CU(CU),
           Flags(Flags), AncestorIdx(AncestorIdx) {}
   };
 
+  using AttributeSpec = DWARFAbbreviationDeclaration::AttributeSpec;
+
+  /// Information gathered and exchanged between the various
+  /// clone*Attributes helpers about the attributes of a particular DIE.
+  ///
+  /// Can not be used asynchroniously.
+  struct AttributesInfo {
+    /// Names.
+    StringRef Name;
+    StringRef MangledName;
+    StringRef NameWithoutTemplate;
+
+    /// Offsets in the string pool.
+    uint32_t NameOffset = 0;
+    uint32_t MangledNameOffset = 0;
+
+    /// Value of AT_low_pc in the input DIE
+    uint64_t OrigLowPc = std::numeric_limits<uint64_t>::max();
+
+    /// Value of AT_high_pc in the input DIE
+    uint64_t OrigHighPc = 0;
+
+    /// Value of DW_AT_call_return_pc in the input DIE
+    uint64_t OrigCallReturnPc = 0;
+
+    /// Value of DW_AT_call_pc in the input DIE
+    uint64_t OrigCallPc = 0;
+
+    /// Offset to apply to PC addresses inside a function.
+    int64_t PCOffset = 0;
+
+    /// Does the DIE have a low_pc attribute?
+    bool HasLowPc = false;
+
+    /// Is this DIE only a declaration?
+    bool IsDeclaration = false;
+
+    /// Attributes which must be updated with proper offsets.
+    /// When we glue final file from the pieces corresponding
+    /// to the compile units, we know final offsets. Some attributes
+    /// should be updated with these offsets. Following members
+    /// keeps information about such attributes.
+    std::vector<CompileUnit::RangeAttributePatchInfo> RangeAttrReferences;
+    std::vector<CompileUnit::LocationAttributePatchInfo> LocationAttrReferences;
+    std::vector<CompileUnit::DieRefAttrPatchInfo> AttrReferences;
+
+    AttributesInfo() = default;
+  };
+
+  /// linking options
+  struct DWARFLinkerOptions {
+    /// Generate processing log to the standard output.
+    bool Verbose = false;
+
+    /// Print statistics.
+    bool Statistics = false;
+
+    /// Skip emitting output
+    bool NoOutput = false;
+
+    /// Do not unique types according to ODR
+    bool NoODR = false;
+
+    /// Update
+    bool RemoveObsoleteDebugInfo = true;
+
+    /// Number of threads.
+    unsigned Threads = 1;
+
+    /// Generate index tables.
+    bool GenerateAppleNamespaces = false;
+    bool GenerateAppleNames = false;
+    bool GenerateAppleTypes = false;
+    bool GenerateAppleObjC = false;
+    bool GeneratePubNames = false;
+    bool GeneratePubTypes = false;
+    bool GenerateDebugNames = false;
+
+    /// Prepend path for the clang modules.
+    std::string PrependPath;
+
+    /// warning handler(it might be called asynchronously).
+    messageHandler WarningHandler = nullptr;
+
+    /// error handler(it might be called asynchronously).
+    messageHandler ErrorHandler = nullptr;
+
+    /// Callback for load object file(it might be called asynchronously).
+    objFileLoader ObjFileLoader = nullptr;
+
+    /// A list of all .swiftinterface files referenced by the debug
+    /// info, mapping Module name to path on disk. The entries need to
+    /// be uniqued and sorted and there are only few entries expected
+    /// per compile unit, which is why this is a std::map.
+    /// this is dsymutil specific fag.
+    ///
+    /// (it might be called asynchronously).
+    swiftInterfacesMap *ParseableSwiftInterfaces = nullptr;
+
+    /// A list of remappings to apply to file paths.
+    ///
+    /// (it might be called asynchronously).
+    objectPrefixMap *ObjectPrefixMap = nullptr;
+  } Options;
+
+public:
+  DWARFLinker(DwarfEmitter *Emitter,
+              DwarfLinkerClient ClientID = DwarfLinkerClient::General,
+              std::function<StringRef(StringRef)> StringsTranslator = nullptr);
+
+  /// Add object file \p File to be linked.
+  void addObjectFile(DWARFFile &File);
+
+  /// Link debug info for added objFiles. Object
+  /// files are linked all together.
+  bool link();
+
+  /// \defgroup Methods setting various linking options:
+  ///
+  /// @{
+  ///
+
+  /// Allows to generate log of linking process to the standard output.
+  void setVerbosity(bool Verbose) { Options.Verbose = Verbose; }
+
+  /// Print statistics to standard output.
+  void setStatistics(bool Statistics) { Options.Statistics = Statistics; }
+
+  /// Do not emit linked dwarf info.
+  void setNoOutput(bool NoOut) { Options.NoOutput = NoOut; }
+
+  /// Do not unique types according to ODR.
+  void setNoODR(bool NoODR) { Options.NoODR = NoODR; }
+
+  /// update existing DWARF info(for the linked binary).
+  void setRemoveObsoleteDebugInfo(bool RemoveObsoleteDebugInfo) {
+    Options.RemoveObsoleteDebugInfo = RemoveObsoleteDebugInfo;
+  }
+
+  /// Use specified number of threads for parallel files linking.
+  void setNumThreads(unsigned NumThreads) { Options.Threads = NumThreads; }
+
+  /// Set generation Apple namespaces On/Off.
+  void setGenerateAppleNamespaces(bool generate) {
+    Options.GenerateAppleNamespaces = generate;
+  }
+
+  /// Set generation Apple names On/Off.
+  void setGenerateAppleNames(bool generate) {
+    Options.GenerateAppleNames = generate;
+  }
+
+  /// Set generation Apple types On/Off.
+  void setGenerateAppleTypes(bool generate) {
+    Options.GenerateAppleTypes = generate;
+  }
+
+  /// Set generation Apple ObjC On/Off.
+  void setGenerateAppleObjC(bool generate) {
+    Options.GenerateAppleObjC = generate;
+  }
+
+  /// Set generation Pubnames On/Off.
+  void setGeneratePubNames(bool generate) {
+    Options.GeneratePubNames = generate;
+  }
+
+  /// Set generation Pubtypes On/Off.
+  void setGeneratePubTypes(bool generate) {
+    Options.GeneratePubTypes = generate;
+  }
+
+  /// Set generation Debug names On/Off.
+  void setGenerateDebugNames(bool generate) {
+    Options.GenerateDebugNames = generate;
+  }
+
+  /// Set prepend path for clang modules.
+  void setPrependPath(const std::string &Ppath) { Options.PrependPath = Ppath; }
+
+  /// Set warning handler which would be used to report warnings.
+  void setWarningHandler(messageHandler Handler) {
+    Options.WarningHandler = Handler;
+  }
+
+  /// Set error handler which would be used to report errors.
+  void setErrorHandler(messageHandler Handler) {
+    Options.ErrorHandler = Handler;
+  }
+
+  /// Set object files loader which would be used to load
+  /// additional objects for splitted dwarf.
+  void setObjFileLoader(objFileLoader Loader) {
+    Options.ObjFileLoader = Loader;
+  }
+
+  /// Set map for Swift interfaces.
+  void setSwiftInterfacesMap(swiftInterfacesMap *Map) {
+    Options.ParseableSwiftInterfaces = Map;
+  }
+
+  /// Set prefix map for objects.
+  void setObjectPrefixMap(objectPrefixMap *Map) {
+    Options.ObjectPrefixMap = Map;
+  }
+
+  /// @}
+
+protected:
+  /// \defgroup Methods used to link the debug information
+  ///
+  /// @{
+
+  /// Returns next available unique Compile Unit ID.
+  unsigned getNextUniqueUnitID() { return UniqueUnitID.fetch_add(1); }
+
+  /// Returns true if DWARFLinker is in multy-thread mode.
+  bool IsMT() { return Options.Threads != 1; }
+
+  /// Returns true if all compile units for specified \p Context were cloned.
+  /// If there are unprocessed inter-CU references after all compile units
+  /// are done then we need to do additional pass to handle these inter-CU
+  /// references.
+  bool IsClonningComplete(LinkContext &Context) const;
+
+  /// Link single object file from specified \p Context.
+  void linkObjectFile(LinkContext &Context);
+
+  /// Generate linked output for single Compile Unit \p CU from the specified \p
+  /// Context.
+  void processSingleCompileUnit(LinkContext &Context, CompileUnit &CU);
+
+  /// Copy invariant debug sections from the specified \p Dwarf context into
+  /// the output \p Emitter.
+  void copyInvariantDebugSection(DWARFContext &Dwarf, DwarfEmitter &Emitter);
+
+  /// Emit warnings as Dwarf compile units to leave a trail after linking.
+  void emitPaperTrailWarnings(CompileUnit &CU, DwarfEmitter &Emitter);
+
+  /// Resolve the DIE attribute reference that has been extracted in \p
+  /// RefValue. The resulting DIE might be in another CompileUnit which is
+  /// stored into \p RefCU.
+  /// \returns null if resolving fails for any reason.
+  DWARFDie resolveDIEReference(const LinkContext &Context, CompileUnit &U,
+                               const DWARFFormValue &RefValue,
+                               const DWARFDie &DIE, CompileUnit *&RefCU);
+
   /// returns true if we need to translate strings.
-  bool needToTranslateStrings() { return StringsTranslator != nullptr; }
+  bool needToTranslateStrings() { return NeedToTranslateStrings; }
 
+  /// Reports Warning.
   void reportWarning(const Twine &Warning, const DWARFFile &File,
                      const DWARFDie *DIE = nullptr) const {
     if (Options.WarningHandler != nullptr)
       Options.WarningHandler(Warning, File.FileName, DIE);
   }
 
+  /// Reports Warning.
+  void reportWarning(const Twine &Warning, StringRef FileName,
+                     const DWARFDie *DIE = nullptr) const {
+    if (Options.WarningHandler != nullptr)
+      Options.WarningHandler(Warning, FileName, DIE);
+  }
+
+  /// Reports Error.
   void reportError(const Twine &Warning, const DWARFFile &File,
                    const DWARFDie *DIE = nullptr) const {
     if (Options.ErrorHandler != nullptr)
       Options.ErrorHandler(Warning, File.FileName, DIE);
   }
 
-  /// Remembers the oldest and newest DWARF version we've seen in a unit.
-  void updateDwarfVersion(unsigned Version) {
-    MaxDwarfVersion = std::max(MaxDwarfVersion, Version);
-    MinDwarfVersion = std::min(MinDwarfVersion, Version);
+  /// Reports Error.
+  void reportError(const Twine &Warning, StringRef FileName,
+                   const DWARFDie *DIE = nullptr) const {
+    if (Options.ErrorHandler != nullptr)
+      Options.ErrorHandler(Warning, FileName, DIE);
   }
 
-  /// Remembers the kinds of accelerator tables we've seen in a unit.
-  void updateAccelKind(DWARFContext &Dwarf);
+  /// @}
 
-  /// Emit warnings as Dwarf compile units to leave a trail after linking.
-  bool emitPaperTrailWarnings(const DWARFFile &File,
-                              OffsetsStringPool &StringPool);
+  /// \defgroup Methods used to search for Clang modules.
+  ///
+  /// @{
+  ///
+  /// If this compile unit is really a skeleton CU that points to a
+  /// clang module, register it in ClangModules and return true.
+  ///
+  /// A skeleton CU is a CU without children, a DW_AT_gnu_dwo_name
+  /// pointing to the module, and a DW_AT_gnu_dwo_id with the module
+  /// hash.
+  bool registerModuleReference(LinkContext &Context, DWARFUnit &CU,
+                               const DWARFFile &File, unsigned Indent = 0,
+                               bool Quiet = false);
 
-  void copyInvariantDebugSection(DWARFContext &Dwarf);
+  /// Recursively add the debug info in this clang module .pcm
+  /// file (and all the modules imported by it in a bottom-up fashion)
+  /// to link context's modules compile units list.
+  Error loadClangModule(LinkContext &Context, DWARFUnit &CU, StringRef FilePath,
+                        StringRef ModuleName, uint64_t DwoId,
+                        const DWARFFile &File, unsigned Indent = 0,
+                        bool Quiet = false);
 
-  /// Keeps track of data associated with one object during linking.
-  struct LinkContext {
-    DWARFFile &File;
-    UnitListTy CompileUnits;
-    bool Skip = false;
+  /// @}
 
-    LinkContext(DWARFFile &File) : File(File) {}
+  /// \defgroup Methods used to analyzing of type declaration contexts.
+  ///
+  /// @{
 
-    /// Clear part of the context that's no longer needed when we're done with
-    /// the debug object.
-    void clear() {
-      CompileUnits.clear();
-      File.Addresses->clear();
-    }
-  };
+  /// Non-recursive helper to build the global DeclContext information and
+  /// gather the child->parent relationships in the original compile unit.
+  ///
+  /// This function uses the same work list approach as lookForDIEsToKeep.
+  ///
+  /// \return true when this DIE and all of its children are only
+  /// forward declarations to types defined in external clang modules
+  /// (i.e., forward declarations that are children of a DW_TAG_module).
+  bool analyzeContextInfo(const LinkContext &Context, const DWARFDie &DIE,
+                          unsigned ParentIdx, CompileUnit &CU,
+                          swiftInterfacesMap *ParseableSwiftInterfaces,
+                          bool &HasUnresolvedRefs);
+  void analyzeImportedModule(const DWARFDie &DIE, CompileUnit &CU,
+                             swiftInterfacesMap *ParseableSwiftInterfaces);
+
+  /// Create DeclarationContext specified by \p Key ans set it into \p Info.
+  void setDeclContext(const DeclContextKey *Key, CompileUnit::DIEInfo *Info);
+
+  /// Set Declaration Contexts(accummulated through analyzing stage) from the \p
+  /// CU to the corresponding DIEInfo structures.
+  void assignDelayedDeclContexts(CompileUnit &CU);
 
-  /// Called before emitting object data
-  void cleanupAuxiliarryData(LinkContext &Context);
+  /// @}
+
+  /// \defgroup Methods used for analyzing liveness information.
+  ///
+  /// Recursively walk the \p DIE tree and look for DIEs to
+  /// keep. Store that information in \p CU's DIEInfo.
+  ///
+  /// @{
 
   /// Look at the parent of the given DIE and decide whether they should be
   /// kept.
   void lookForParentDIEsToKeep(unsigned AncestorIdx, CompileUnit &CU,
                                unsigned Flags,
                                SmallVectorImpl<WorklistItem> &Worklist);
 
   /// Look at the children of the given DIE and decide whether they should be
   /// kept.
   void lookForChildDIEsToKeep(const DWARFDie &Die, CompileUnit &CU,
                               unsigned Flags,
                               SmallVectorImpl<WorklistItem> &Worklist);
 
   /// Look at DIEs referenced by the given DIE and decide whether they should be
   /// kept. All DIEs referenced though attributes should be kept.
-  void lookForRefDIEsToKeep(const DWARFDie &Die, CompileUnit &CU,
-                            unsigned Flags, const UnitListTy &Units,
-                            const DWARFFile &File,
+  bool lookForRefDIEsToKeep(const DWARFDie &Die, CompileUnit &CU,
+                            unsigned Flags, const LinkContext &Context,
                             SmallVectorImpl<WorklistItem> &Worklist);
 
-  /// \defgroup FindRootDIEs Find DIEs corresponding to Address map entries.
-  ///
-  /// @{
-  /// Recursively walk the \p DIE tree and look for DIEs to
-  /// keep. Store that information in \p CU's DIEInfo.
-  ///
-  /// The return value indicates whether the DIE is incomplete.
-  void lookForDIEsToKeep(AddressesMap &RelocMgr, RangesTy &Ranges,
-                         const UnitListTy &Units, const DWARFDie &DIE,
-                         const DWARFFile &File, CompileUnit &CU,
-                         unsigned Flags);
+  /// Returns false if there were encountered unresolved Die references.
+  bool lookForDIEsToKeep(const LinkContext &Context, const DWARFDie &DIE,
+                         CompileUnit &CU, unsigned Flags);
 
-  /// If this compile unit is really a skeleton CU that points to a
-  /// clang module, register it in ClangModules and return true.
-  ///
-  /// A skeleton CU is a CU without children, a DW_AT_gnu_dwo_name
-  /// pointing to the module, and a DW_AT_gnu_dwo_id with the module
-  /// hash.
-  bool registerModuleReference(DWARFDie CUDie, const DWARFUnit &Unit,
-                               const DWARFFile &File,
-                               OffsetsStringPool &OffsetsStringPool,
-                               DeclContextTree &ODRContexts,
-                               uint64_t ModulesEndOffset, unsigned &UnitID,
-                               bool IsLittleEndian, unsigned Indent = 0,
-                               bool Quiet = false);
-
-  /// Recursively add the debug info in this clang module .pcm
-  /// file (and all the modules imported by it in a bottom-up fashion)
-  /// to Units.
-  Error loadClangModule(DWARFDie CUDie, StringRef FilePath,
-                        StringRef ModuleName, uint64_t DwoId,
-                        const DWARFFile &File,
-                        OffsetsStringPool &OffsetsStringPool,
-                        DeclContextTree &ODRContexts, uint64_t ModulesEndOffset,
-                        unsigned &UnitID, bool IsLittleEndian,
-                        unsigned Indent = 0, bool Quiet = false);
-
-  /// Mark the passed DIE as well as all the ones it depends on as kept.
-  void keepDIEAndDependencies(AddressesMap &RelocMgr, RangesTy &Ranges,
-                              const UnitListTy &Units, const DWARFDie &DIE,
-                              CompileUnit::DIEInfo &MyInfo,
-                              const DWARFFile &File, CompileUnit &CU,
-                              bool UseODR);
-
-  unsigned shouldKeepDIE(AddressesMap &RelocMgr, RangesTy &Ranges,
-                         const DWARFDie &DIE, const DWARFFile &File,
-                         CompileUnit &Unit, CompileUnit::DIEInfo &MyInfo,
-                         unsigned Flags);
+  unsigned shouldKeepDIE(const DWARFDie &DIE, CompileUnit &Unit,
+                         CompileUnit::DIEInfo &MyInfo, unsigned Flags);
 
   /// Check if a variable describing DIE should be kept.
   /// \returns updated TraversalFlags.
-  unsigned shouldKeepVariableDIE(AddressesMap &RelocMgr, const DWARFDie &DIE,
+  unsigned shouldKeepVariableDIE(const DWARFDie &DIE, CompileUnit &CU,
                                  CompileUnit::DIEInfo &MyInfo, unsigned Flags);
 
-  unsigned shouldKeepSubprogramDIE(AddressesMap &RelocMgr, RangesTy &Ranges,
-                                   const DWARFDie &DIE, const DWARFFile &File,
-                                   CompileUnit &Unit,
+  unsigned shouldKeepSubprogramDIE(const DWARFDie &DIE, CompileUnit &CU,
                                    CompileUnit::DIEInfo &MyInfo,
                                    unsigned Flags);
 
-  /// Resolve the DIE attribute reference that has been extracted in \p
-  /// RefValue. The resulting DIE might be in another CompileUnit which is
-  /// stored into \p ReferencedCU. \returns null if resolving fails for any
-  /// reason.
-  DWARFDie resolveDIEReference(const DWARFFile &File, const UnitListTy &Units,
-                               const DWARFFormValue &RefValue,
-                               const DWARFDie &DIE, CompileUnit *&RefCU);
-
   /// @}
 
-  /// \defgroup Methods used to link the debug information
+  /// \defgroup Methods used to clone debug information.
+  ///
+  /// DIECloner is a helper to clone single CU.
   ///
   /// @{
-
-  struct DWARFLinkerOptions;
-
+  ///
   class DIECloner {
+    const LinkContext &Context;
     DWARFLinker &Linker;
-    DwarfEmitter *Emitter;
-    DWARFFile &ObjFile;
-
-    /// Allocator used for all the DIEValue objects.
-    BumpPtrAllocator &DIEAlloc;
-
-    std::vector<std::unique_ptr<CompileUnit>> &CompileUnits;
+    CompileUnit &CU;
 
-    bool Update;
+    const DWARFLinkerOptions &Options;
 
   public:
-    DIECloner(DWARFLinker &Linker, DwarfEmitter *Emitter, DWARFFile &ObjFile,
-              BumpPtrAllocator &DIEAlloc,
-              std::vector<std::unique_ptr<CompileUnit>> &CompileUnits,
-              bool Update)
-        : Linker(Linker), Emitter(Emitter), ObjFile(ObjFile),
-          DIEAlloc(DIEAlloc), CompileUnits(CompileUnits), Update(Update) {}
+    DIECloner(CompileUnit &CU, const LinkContext &Context, DWARFLinker &Linker,
+              raw_pwrite_stream &Out, const DWARFLinkerOptions &Options);
+    ~DIECloner();
 
+    void cloneCU();
+
+  private:
     /// Recursively clone \p InputDIE into an tree of DIE objects
     /// where useless (as decided by lookForDIEsToKeep()) bits have been
     /// stripped out and addresses have been rewritten according to the
     /// address map.
     ///
     /// \param OutOffset is the offset the cloned DIE in the output
     /// compile unit.
     /// \param PCOffset (while cloning a function scope) is the offset
     /// applied to the entry point of the function to get the linked address.
     /// \param Die the output DIE to use, pass NULL to create one.
     /// \returns the root of the cloned tree or null if nothing was selected.
-    DIE *cloneDIE(const DWARFDie &InputDIE, const DWARFFile &File,
-                  CompileUnit &U, OffsetsStringPool &StringPool,
-                  int64_t PCOffset, uint32_t OutOffset, unsigned Flags,
-                  bool IsLittleEndian, DIE *Die = nullptr);
-
-    /// Construct the output DIE tree by cloning the DIEs we
-    /// chose to keep above. If there are no valid relocs, then there's
-    /// nothing to clone/emit.
-    uint64_t cloneAllCompileUnits(DWARFContext &DwarfContext,
-                                  const DWARFFile &File,
-                                  OffsetsStringPool &StringPool,
-                                  bool IsLittleEndian);
-
-  private:
-    using AttributeSpec = DWARFAbbreviationDeclaration::AttributeSpec;
-
-    /// Information gathered and exchanged between the various
-    /// clone*Attributes helpers about the attributes of a particular DIE.
-    struct AttributesInfo {
-      /// Names.
-      DwarfStringPoolEntryRef Name, MangledName, NameWithoutTemplate;
-
-      /// Offsets in the string pool.
-      uint32_t NameOffset = 0;
-      uint32_t MangledNameOffset = 0;
-
-      /// Value of AT_low_pc in the input DIE
-      uint64_t OrigLowPc = std::numeric_limits<uint64_t>::max();
-
-      /// Value of AT_high_pc in the input DIE
-      uint64_t OrigHighPc = 0;
-
-      /// Value of DW_AT_call_return_pc in the input DIE
-      uint64_t OrigCallReturnPc = 0;
-
-      /// Value of DW_AT_call_pc in the input DIE
-      uint64_t OrigCallPc = 0;
-
-      /// Offset to apply to PC addresses inside a function.
-      int64_t PCOffset = 0;
-
-      /// Does the DIE have a low_pc attribute?
-      bool HasLowPc = false;
-
-      /// Does the DIE have a ranges attribute?
-      bool HasRanges = false;
-
-      /// Is this DIE only a declaration?
-      bool IsDeclaration = false;
-
-      AttributesInfo() = default;
-    };
+    DIE *cloneDIE(const DWARFDie &InputDIE, int64_t PCOffset,
+                  uint32_t OutOffset, unsigned Flags, DIE *Die = nullptr);
 
     /// Helper for cloneDIE.
     unsigned cloneAttribute(DIE &Die, const DWARFDie &InputDIE,
-                            const DWARFFile &File, CompileUnit &U,
-                            OffsetsStringPool &StringPool,
                             const DWARFFormValue &Val,
                             const AttributeSpec AttrSpec, unsigned AttrSize,
-                            AttributesInfo &AttrInfo, bool IsLittleEndian);
+                            AttributesInfo &AttrInfo, uint64_t AttrOutOffset);
 
-    /// Clone a string attribute described by \p AttrSpec and add
-    /// it to \p Die.
+    /// Clone a string attribute described by \p AttrSpec and add it to \p Die.
     /// \returns the size of the new attribute.
-    unsigned cloneStringAttribute(DIE &Die, AttributeSpec AttrSpec,
-                                  const DWARFFormValue &Val, const DWARFUnit &U,
-                                  OffsetsStringPool &StringPool,
+    unsigned cloneStringAttribute(DIE &Die, const DWARFDie &InputDIE,
+                                  AttributeSpec AttrSpec,
+                                  const DWARFFormValue &Val,
                                   AttributesInfo &Info);
 
-    /// Clone an attribute referencing another DIE and add
-    /// it to \p Die.
+    /// Clone an attribute referencing another DIE and add it to \p Die.
     /// \returns the size of the new attribute.
-    unsigned cloneDieReferenceAttribute(DIE &Die, const DWARFDie &InputDIE,
-                                        AttributeSpec AttrSpec,
+    unsigned cloneDieReferenceAttribute(DIE &Die, AttributeSpec AttrSpec,
                                         unsigned AttrSize,
                                         const DWARFFormValue &Val,
-                                        const DWARFFile &File,
-                                        CompileUnit &Unit);
+                                        AttributesInfo &Info,
+                                        uint64_t AttrOutOffset);
 
     /// Clone a DWARF expression that may be referencing another DIE.
     void cloneExpression(DataExtractor &Data, DWARFExpression Expression,
-                         const DWARFFile &File, CompileUnit &Unit,
                          SmallVectorImpl<uint8_t> &OutputBuffer);
 
-    /// Clone an attribute referencing another DIE and add
-    /// it to \p Die.
+    /// Clone an attribute referencing another DIE and add it to \p Die.
     /// \returns the size of the new attribute.
-    unsigned cloneBlockAttribute(DIE &Die, const DWARFFile &File,
-                                 CompileUnit &Unit, AttributeSpec AttrSpec,
-                                 const DWARFFormValue &Val, unsigned AttrSize,
-                                 bool IsLittleEndian);
+    unsigned cloneBlockAttribute(DIE &Die, AttributeSpec AttrSpec,
+                                 const DWARFFormValue &Val, unsigned AttrSize);
 
-    /// Clone an attribute referencing another DIE and add
-    /// it to \p Die.
+    /// Clone an attribute referencing another DIE and add it to \p Die.
     /// \returns the size of the new attribute.
     unsigned cloneAddressAttribute(DIE &Die, AttributeSpec AttrSpec,
                                    const DWARFFormValue &Val,
-                                   const CompileUnit &Unit,
                                    AttributesInfo &Info);
 
     /// Clone a scalar attribute  and add it to \p Die.
     /// \returns the size of the new attribute.
     unsigned cloneScalarAttribute(DIE &Die, const DWARFDie &InputDIE,
-                                  const DWARFFile &File, CompileUnit &U,
                                   AttributeSpec AttrSpec,
                                   const DWARFFormValue &Val, unsigned AttrSize,
-                                  AttributesInfo &Info);
+                                  AttributesInfo &Info, uint64_t AttrOutOffset);
 
     /// Get the potential name and mangled name for the entity
-    /// described by \p Die and store them in \Info if they are not
+    /// described by \p Die and store them in \p Info if they are not
     /// already there.
     /// \returns is a name was found.
     bool getDIENames(const DWARFDie &Die, AttributesInfo &Info,
-                     OffsetsStringPool &StringPool, bool StripTemplate = false);
-
-    /// Create a copy of abbreviation Abbrev.
-    void copyAbbrev(const DWARFAbbreviationDeclaration &Abbrev, bool hasODR);
+                     bool StripTemplate = false);
 
     uint32_t hashFullyQualifiedName(DWARFDie DIE, CompileUnit &U,
                                     const DWARFFile &File,
                                     int RecurseDepth = 0);
 
     /// Helper for cloneDIE.
-    void addObjCAccelerator(CompileUnit &Unit, const DIE *Die,
-                            DwarfStringPoolEntryRef Name,
-                            OffsetsStringPool &StringPool, bool SkipPubSection);
-  };
-
-  /// Assign an abbreviation number to \p Abbrev
-  void assignAbbrev(DIEAbbrev &Abbrev);
+    void addObjCAccelerator(CompileUnit &Unit, const DIE *Die, StringRef Name,
+                            bool SkipPubSection);
 
-  /// Compute and emit debug_ranges section for \p Unit, and
-  /// patch the attributes referencing it.
-  void patchRangesForUnit(const CompileUnit &Unit, DWARFContext &Dwarf,
-                          const DWARFFile &File) const;
-
-  /// Generate and emit the DW_AT_ranges attribute for a compile_unit if it had
-  /// one.
-  void generateUnitRanges(CompileUnit &Unit) const;
-
-  /// Extract the line tables from the original dwarf, extract the relevant
-  /// parts according to the linked function ranges and emit the result in the
-  /// debug_line section.
-  void patchLineTableForUnit(CompileUnit &Unit, DWARFContext &OrigDwarf,
-                             const DWARFFile &File);
-
-  /// Emit the accelerator entries for \p Unit.
-  void emitAcceleratorEntriesForUnit(CompileUnit &Unit);
-  void emitDwarfAcceleratorEntriesForUnit(CompileUnit &Unit);
-  void emitAppleAcceleratorEntriesForUnit(CompileUnit &Unit);
-
-  /// Patch the frame info for an object file and emit it.
-  void patchFrameInfoForObject(const DWARFFile &, RangesTy &Ranges,
-                               DWARFContext &, unsigned AddressSize);
-
-  /// FoldingSet that uniques the abbreviations.
-  FoldingSet<DIEAbbrev> AbbreviationsSet;
-
-  /// Storage for the unique Abbreviations.
-  /// This is passed to AsmPrinter::emitDwarfAbbrevs(), thus it cannot be
-  /// changed to a vector of unique_ptrs.
-  std::vector<std::unique_ptr<DIEAbbrev>> Abbreviations;
+    /// Emits all debug info corresponding to the cloned compile unit.
+    std::unique_ptr<DwarfStreamer> Emitter;
 
     /// DIELoc objects that need to be destructed (but not freed!).
     std::vector<DIELoc *> DIELocs;
 
     /// DIEBlock objects that need to be destructed (but not freed!).
     std::vector<DIEBlock *> DIEBlocks;
 
     /// Allocator used for all the DIEValue objects.
     BumpPtrAllocator DIEAlloc;
-  /// @}
+  };
 
-  DwarfEmitter *TheDwarfEmitter;
-  std::vector<LinkContext> ObjectContexts;
+  /// Compute and emit debug_ranges section for \p Unit, and
+  /// patch the attributes referencing it.
+  void patchRangesForUnit(CompileUnit &Unit, DwarfEmitter &Emitter) const;
 
-  unsigned MaxDwarfVersion = 0;
-  unsigned MinDwarfVersion = std::numeric_limits<unsigned>::max();
+  /// Extract the line tables from the original dwarf, extract the relevant
+  /// parts according to the linked function ranges and emit the result in the
+  /// debug_line section.
+  void patchLineTableForUnit(CompileUnit &CU, DwarfEmitter &Emitter);
 
-  bool AtLeastOneAppleAccelTable = false;
-  bool AtLeastOneDwarfAccelTable = false;
+  /// Patch the frame info for an object file and emit it.
+  void patchFrameInfoForObject(LinkContext &Context, RangesTy &Ranges,
+                               unsigned AddressSize, DwarfEmitter &Emitter);
 
-  /// The CIEs that have been emitted in the output section. The actual CIE
-  /// data serves a the key to this StringMap, this takes care of comparing the
-  /// semantics of CIEs defined in different object files.
-  StringMap<uint32_t> EmittedCIEs;
-
-  /// Offset of the last CIE that has been emitted in the output
-  /// debug_frame section.
-  uint32_t LastCIEOffset = 0;
+  /// Store specified string \p String into the global string
+  /// pool(TheOffsetsStringPool). Assign the string offset to the
+  /// specified \p It Die Value.
+  void setStringsOffset(DIE::value_iterator It, const char *String);
+
+  /// Store strings accumulated for \p CU at clonning stage into the global
+  /// string pool (TheOffsetsStringPool). Assign offsets to the
+  /// corresponding Die Values.
+  void assignDelayedOffsets(CompileUnit &CU);
 
-  /// Apple accelerator tables.
-  AccelTable<DWARF5AccelTableStaticData> DebugNames;
-  AccelTable<AppleAccelTableStaticOffsetData> AppleNames;
-  AccelTable<AppleAccelTableStaticOffsetData> AppleNamespaces;
-  AccelTable<AppleAccelTableStaticOffsetData> AppleObjc;
-  AccelTable<AppleAccelTableStaticTypeData> AppleTypes;
+  /// @}
 
-  /// Mapping the PCM filename to the DwoId.
-  StringMap<uint64_t> ClangModules;
+  /// \defgroup Methods used to assemble final file from cloned
+  /// compile units.
+  ///
+  /// @{
 
-  DwarfLinkerClient DwarfLinkerClientID;
+  /// Assemble all CompileUnits into the single output DWARF File.
+  void glueCompileUnitsAndWriteToTheOutput();
 
-  std::function<StringRef(StringRef)> StringsTranslator = nullptr;
+  /// Assign offsets for Compile Units sections.
+  void assignOffsets(StringMap<DebugInfoSize> &);
+  void
+  assignOffsets(LinkContext &Context,
+                std::array<uint64_t, CompileUnit::TableKindsNum> &TableOffsets);
+  void
+  assignOffsets(CompileUnit &CU,
+                std::array<uint64_t, CompileUnit::TableKindsNum> &TableOffsets);
 
-  /// linking options
-  struct DWARFLinkerOptions {
-    /// Generate processing log to the standard output.
-    bool Verbose = false;
+  /// Patch Offsets and Size fields.
+  void patchOffsetsAndSizes();
+  void patchOffsetsAndSizes(LinkContext &Context);
+  void patchOffsetsAndSizes(CompileUnit &CU);
+  void patchCompileUnit(CompileUnit &CU, StringRef Data);
+  void patchAranges(CompileUnit &CU, StringRef Data);
+  void patchFrames(LinkContext &Context, StringRef Data);
+
+  /// Write data to the resulting output.
+  void writeDWARFToTheOutput();
+  void writeDWARFToTheOutput(CompileUnit &CU, bool &HasAbbreviations);
+  void writeDWARFToTheOutput(LinkContext &CU);
+
+  /// Print statistic for processed Debug Info.
+  void printStatistic(const StringMap<DebugInfoSize> &SizeByObject);
+
+  /// Build Accelerator tables.
+  void buildAcceleratorTables();
+  void buildAcceleratorTables(CompileUnit &CU);
+
+  /// Increase Object File reference counter. Archive files might contain
+  /// several object files. Thus, we count number of object files inside
+  /// an archive, so that we could safely unload archive if all it`s object
+  /// files are already processed.
+  void addBinHolderNameRef(DWARFFile &File);
+
+  /// Decrease Object file reference counter and unload Object File
+  /// if there are no more referencies.
+  void unloadObjectFile(DWARFFile &File);
+  /// @}
 
-    /// Print statistics.
-    bool Statistics = false;
+  /// \defgroup Members accessed asinchroniously.
+  ///
+  /// @{
 
-    /// Skip emitting output
-    bool NoOutput = false;
+  /// Unique ID for compile unit.
+  std::atomic<unsigned> UniqueUnitID;
 
-    /// Do not unique types according to ODR
-    bool NoODR = false;
+  /// Mapping the PCM filename to the DwoId.
+  StringMap<uint64_t> ClangModules;
+  std::mutex ClangModulesMutex;
 
-    /// Update
-    bool Update = false;
+  /// BinaryHolderRefCounter counts whether references to the binary
+  /// files exist.
+  DenseMap<StringRef, uint32_t> BinaryHolderRefCounter;
+  std::mutex BinaryHolderRefMutex;
 
-    /// Number of threads.
-    unsigned Threads = 1;
+  /// The Dwarf string pool is used for emission.
+  OffsetsStringPool TheOffsetsStringPool;
+  std::mutex StringPoolMutex;
 
-    /// The accelerator table kind
-    AccelTableKind TheAccelTableKind = AccelTableKind::Default;
+  /// ODR Contexts for the data deduplication.
+  DeclContextTree ODRContexts;
+  std::mutex ContextsMutex;
 
-    /// Prepend path for the clang modules.
-    std::string PrependPath;
+  // no need semaphore, since it is immutable:
+  std::vector<std::unique_ptr<LinkContext>> ObjectContexts;
 
-    // warning handler
-    messageHandler WarningHandler = nullptr;
+  bool NeedToTranslateStrings = false;
 
-    // error handler
-    messageHandler ErrorHandler = nullptr;
+  DwarfLinkerClient DwarfLinkerClientID;
 
-    objFileLoader ObjFileLoader = nullptr;
+  /// @}
 
-    /// A list of all .swiftinterface files referenced by the debug
-    /// info, mapping Module name to path on disk. The entries need to
-    /// be uniqued and sorted and there are only few entries expected
-    /// per compile unit, which is why this is a std::map.
-    /// this is dsymutil specific fag.
-    swiftInterfacesMap *ParseableSwiftInterfaces = nullptr;
+  /// \defgroup Members accessed sequentially.
+  ///
+  /// @{
 
-    /// A list of remappings to apply to file paths.
-    objectPrefixMap *ObjectPrefixMap = nullptr;
-  } Options;
+  /// Apple accelerator tables.
+  AccelTable<DWARF5AccelTableStaticData> DebugNames;
+  AccelTable<AppleAccelTableStaticOffsetData> AppleNames;
+  AccelTable<AppleAccelTableStaticOffsetData> AppleNamespaces;
+  AccelTable<AppleAccelTableStaticOffsetData> AppleObjc;
+  AccelTable<AppleAccelTableStaticTypeData> AppleTypes;
+
+  DwarfEmitter *TheDwarfEmitter;
+  /// @}
 };
 
+} // end namespace dwarflinker
+
 } // end namespace llvm
 
-#endif // LLVM_DWARFLINKER_DWARFLINKER_H
+#endif // LLVM_DWARFLINKERNEXT_DWARFLINKER_H

llvm/include/llvm/DWARFLinkerNext/DWARFLinkerCompileUnit.h

+//===- DWARFLinkerCompileUnit.h ---------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DWARFLINKERNEXT_DWARFLINKERCOMPILEUNIT_H
+#define LLVM_DWARFLINKERNEXT_DWARFLINKERCOMPILEUNIT_H
+
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/IntervalMap.h"
+#include "llvm/CodeGen/DIE.h"
+#include "llvm/CodeGen/NonRelocatableStringpool.h"
+#include "llvm/DebugInfo/DWARF/DWARFUnit.h"
+#include "llvm/Support/DataExtractor.h"
+#include <atomic>
+#include <mutex>
+
+namespace llvm {
+
+namespace dwarflinker {
+
+class DeclContext;
+class DWARFFile;
+class DeclContextTree;
+
+template <typename KeyT, typename ValT>
+using HalfOpenIntervalMap =
+    IntervalMap<KeyT, ValT, IntervalMapImpl::NodeSizer<KeyT, ValT>::LeafSize,
+                IntervalMapHalfOpenInfo<KeyT>>;
+
+using FunctionIntervals = HalfOpenIntervalMap<uint64_t, int64_t>;
+
+/// DeclContextKey keeps all meaningful fields for DeclContext.
+/// It is used to memorize context fields for types used in current compile
+/// unit.
+class DeclContextKey {
+public:
+  DeclContextKey() {}
+  DeclContextKey(unsigned Hash, uint32_t Line, uint32_t ByteSize, uint16_t Tag,
+                 StringRef Name, StringRef File,
+                 unsigned ParentQualifiedNameHash, bool IsInvalid,
+                 bool IsDefinedInClangModule)
+      : Hash(Hash), Line(Line), ByteSize(ByteSize), Tag(Tag), Name(Name),
+        File(File), ParentQualifiedNameHash(ParentQualifiedNameHash),
+        IsInvalid(IsInvalid), IsDefinedInClangModule(IsDefinedInClangModule) {}
+
+  unsigned Hash = 0;
+  uint32_t Line = 0;
+  uint32_t ByteSize = 0;
+  uint16_t Tag = 0;
+  StringRef Name;
+  StringRef File;
+  unsigned ParentQualifiedNameHash = 0;
+  unsigned IsInvalid : 1;
+  unsigned IsDefinedInClangModule : 1;
+};
+
+/// Info type for the DenseMap storing the DeclContextKey pointers.
+struct DeclKeyMapInfo : private DenseMapInfo<DeclContextKey *> {
+  using DenseMapInfo<DeclContextKey *>::getEmptyKey;
+  using DenseMapInfo<DeclContextKey *>::getTombstoneKey;
+
+  static unsigned getHashValue(const DeclContextKey *Ctxt) {
+    return Ctxt->Hash;
+  }
+
+  static bool isEqual(const DeclContextKey *LHS, const DeclContextKey *RHS) {
+    if (RHS == getEmptyKey() || RHS == getTombstoneKey())
+      return RHS == LHS;
+    return LHS->Hash == RHS->Hash &&
+           LHS->ParentQualifiedNameHash == RHS->ParentQualifiedNameHash &&
+           LHS->Line == RHS->Line && LHS->ByteSize == RHS->ByteSize &&
+           LHS->Name == RHS->Name && LHS->File == RHS->File;
+  }
+};
+
+/// Stores all information relating to a compile unit, be it in its original
+/// instance in the object file to its brand new cloned and generated DIE tree.
+class CompileUnit {
+public:
+  /// The stages of new compile unit processing.
+  enum class Stages {
+    Created,
+    SrcDWARFLoaded,
+    ModulesLoaded,
+    TypesDeclarationsFound,
+    LivenessAnalysisDone,
+    Cloned,
+  };
+
+  /// The debug tables containing pieces or related
+  /// information for the current compilation unit.
+  enum class DebugTableKind : uint8_t {
+    DebugInfo = 0,
+    DebugLine = 1,
+    DebugFrame = 2,
+    DebugRange = 3,
+    DebugLoc = 4,
+    DebugARanges = 5,
+    DebugAbbrev = 6,
+  };
+  const static size_t TableKindsNum = 7;
+
+  /// Information gathered about a DIE in the object file.
+  struct DIEInfo {
+    DIEInfo() {
+      Keep = false;
+      InDebugMap = false;
+      Prune = false;
+      IncompleteDeclaration = false;
+    }
+    DIEInfo(const DIEInfo &Info) {
+      AddrAdjust = Info.AddrAdjust;
+      Ctxt.store(Info.Ctxt);
+      ClonedDieOffset = Info.ClonedDieOffset;
+      ParentIdx = Info.ParentIdx;
+      Keep = Info.Keep;
+      InDebugMap = Info.InDebugMap;
+      Prune = Info.Prune;
+      IncompleteDeclaration = Info.IncompleteDeclaration;
+    }
+    /// Address offset to apply to the described entity.
+    int64_t AddrAdjust = 0;
+
+    /// ODR Declaration context.
+    std::atomic<const DeclContext *> Ctxt = {nullptr};
+
+    /// Offset to cloned version of that DIE.
+    uint64_t ClonedDieOffset = 0;
+
+    /// The index of this DIE's parent.
+    uint32_t ParentIdx = 0;
+
+    /// Is the DIE part of the linked output?
+    bool Keep : 1;
+
+    /// Was this DIE's entity found in the map?
+    bool InDebugMap : 1;
+
+    /// Is this a pure forward declaration we can strip?
+    bool Prune : 1;
+
+    /// Does DIE transitively refer an incomplete decl?
+    bool IncompleteDeclaration : 1;
+  };
+
+  /// This structure keeps offset to DW_AT_range attribute and
+  /// value which should be updated when final offsets would be known.
+  struct RangeAttributePatchInfo {
+    RangeAttributePatchInfo(uint64_t CULocalOffsetValue, uint64_t OffsetToAttr)
+        : CULocalOffsetValue(CULocalOffsetValue), OffsetToAttr(OffsetToAttr) {}
+    uint64_t CULocalOffsetValue;
+    uint64_t OffsetToAttr;
+  };
+
+  /// This structure keeps offset to DW_AT_location attribute and
+  /// value which should be updated when final offsets would be known.
+  struct LocationAttributePatchInfo {
+    LocationAttributePatchInfo(uint64_t OffsetToAttr,
+                               uint64_t CULocalOffsetValue, int64_t PCValue)
+        : OffsetToAttr(OffsetToAttr), CULocalOffsetValue(CULocalOffsetValue),
+          PCValue(PCValue) {}
+    uint64_t OffsetToAttr;
+    uint64_t CULocalOffsetValue;
+    int64_t PCValue;
+  };
+
+  /// This structure keeps offset to Die reference attribute and
+  /// its target(CU/Die index) which should be updated when final offsets would
+  /// be known.
+  struct DieRefAttrPatchInfo {
+    DieRefAttrPatchInfo(uint64_t RefDieIdx, const CompileUnit *RefCU,
+                        uint64_t OffsetToAttr, bool IsLocal)
+        : RefDieIdx(RefDieIdx), RefCU(RefCU), OffsetToAttr(OffsetToAttr),
+          IsCULocal(IsLocal) {}
+    uint64_t RefDieIdx = 0;
+    const CompileUnit *RefCU = nullptr;
+    uint64_t OffsetToAttr = 0;
+    bool IsCULocal = false;
+  };
+
+  /// This structure keeps pointer to string and pointer to die string
+  /// attribute. That string should be added to the global string pool later and
+  /// it`s offset should be assigned to the die string attribute.
+  struct OffsetsData {
+    OffsetsData(DIE::value_iterator AttrToPatch, const char *String)
+        : AttrToPatch(AttrToPatch), String(String) {}
+    DIE::value_iterator AttrToPatch;
+    const char *String;
+  };
+
+  /// This structure keeps pointer to type declaration context key
+  /// and pointer to DIE info. The type declaration context key would be used
+  /// later for adding type declaration into global DeclContextTree and created
+  /// context declaration would be assigned to the DIE info.
+  struct DeclContextKeyData {
+    DeclContextKeyData(const DeclContextKey *Key, CompileUnit::DIEInfo *Info)
+        : Key(Key), Info(Info) {}
+    const DeclContextKey *Key = nullptr;
+    CompileUnit::DIEInfo *Info = nullptr;
+  };
+
+  /// This structure keeps fields which would be used for creating accelerator
+  /// table.
+  struct AccelInfo {
+    /// Name of the entry.
+    DwarfStringPoolEntryRef Name;
+
+    /// Tag of the DIE this entry describes.
+    dwarf::Tag Tag;
+
+    /// Output offset of the DIE this entry describes.
+    uint64_t OutOffset;
+
+    /// Hash of the fully qualified name.
+    uint32_t QualifiedNameHash;
+
+    /// Emit this entry only in the apple_* sections.
+    bool SkipPubSection;
+
+    /// Is this an ObjC class implementation?
+    bool ObjcClassImplementation;
+
+    AccelInfo(DwarfStringPoolEntryRef Name, dwarf::Tag Tag, uint64_t OutOffset,
+              bool SkipPubSection = false)
+        : Name(Name), Tag(Tag), OutOffset(OutOffset),
+          SkipPubSection(SkipPubSection) {}
+
+    AccelInfo(DwarfStringPoolEntryRef Name, dwarf::Tag Tag, uint64_t OutOffset,
+              uint32_t QualifiedNameHash, bool ObjCClassIsImplementation)
+        : Name(Name), Tag(Tag), OutOffset(OutOffset),
+          QualifiedNameHash(QualifiedNameHash), SkipPubSection(false),
+          ObjcClassImplementation(ObjCClassIsImplementation) {}
+  };
+
+  CompileUnit(unsigned ID, bool, StringRef ClangModuleName, DWARFFile &File);
+
+  CompileUnit(DWARFUnit &OrigUnit, unsigned ID, bool CanUseODR,
+              StringRef ClangModuleName, DWARFFile &File);
+
+  /// Marks compile units which does not have a pair in the source debug info.
+  bool isArtificialCU() const { return OrigUnit == None; }
+
+  /// Endiannes for the compile unit.
+  support::endianness getEndianness() const { return Endianess; }
+
+  /// Returns DWARF version.
+  uint16_t getVersion() const { return Version; }
+
+  /// Returns size of compile unit info.
+  uint16_t getHeaderSize() const { return Version >= 5 ? 12 : 11; }
+
+  /// Returns stage of overall processing.
+  Stages getStage() const { return Stage; }
+
+  /// Set stage of overall processing.
+  void setStage(Stages Stage) { this->Stage = Stage; }
+
+  /// Returns true if there are references to this compile unit
+  /// from other comile units.
+  bool hasExternalReferences() const {
+    std::unique_lock<std::mutex> Guard(ExternalReferencesMutex);
+
+    return !ExternalReferences.empty();
+  }
+
+  /// Recognises the table name and match it with the DebugTableKind.
+  static Optional<DebugTableKind> parseDebugTableName(StringRef Name);
+
+  /// Returns paired compile unit from input DWARF.
+  DWARFUnit &getOrigUnit() const { return **OrigUnit; }
+
+  /// Unique id.
+  unsigned getUniqueID() const { return ID; }
+
+  /// Create a Die for this compile unit.
+  void createOutputDIE() {
+    NewUnit.emplace(getOrigUnit().getUnitDIE().getTag());
+  }
+
+  /// Returns Die for this compile unit.
+  DIE *getOutputUnitDIE() const {
+    if (NewUnit)
+      return &const_cast<BasicDIEUnit &>(*NewUnit).getUnitDie();
+    return nullptr;
+  }
+
+  /// Returns true id ODR deduplication should be done for this compile unit.
+  bool hasODR() const { return HasODR; }
+
+  /// Returns true if this compile unit is from Clang module.
+  bool isClangModule() const { return !ClangModuleName.empty(); }
+
+  uint16_t getLanguage();
+  /// Return the DW_AT_LLVM_sysroot of the compile unit or an empty StringRef.
+  StringRef getSysRoot();
+
+  /// Returns Clang module name;
+  const std::string &getClangModuleName() const { return ClangModuleName; }
+
+  /// Group of functions returning DIE info.
+  ///
+  /// \p Idx index of the DIE
+  /// \returns DIEInfo descriptor.
+  DIEInfo &getInfo(unsigned Idx) { return Info[Idx]; }
+
+  /// \p Idx index of the DIE
+  /// \returns DIEInfo descriptor.
+  const DIEInfo &getInfo(unsigned Idx) const { return Info[Idx]; }
+
+  /// \p Die
+  /// \returns DIEInfo descriptor.
+  DIEInfo &getInfo(const DWARFDie &Die) {
+    unsigned Idx = getOrigUnit().getDIEIndex(Die);
+    return Info[Idx];
+  }
+
+  /// \p Die
+  /// \returns DIEInfo descriptor.
+  const DIEInfo &getInfo(const DWARFDie &Die) const {
+    unsigned Idx = getOrigUnit().getDIEIndex(Die);
+    return Info[Idx];
+  }
+
+  /// Put reference from other CU into the list of external references.
+  void addLiveReference(uint64_t OrigOffset) {
+    std::unique_lock<std::mutex> Guard(ExternalReferencesMutex);
+
+    ExternalReferences.insert(OrigOffset);
+  }
+
+  /// Returns set of references to dies of this compile unit
+  /// from other compile units.
+  DenseSet<uint64_t> getLiveReferences() {
+    std::unique_lock<std::mutex> Guard(ExternalReferencesMutex);
+
+    return std::move(ExternalReferences);
+  }
+
+  /// Returns offset to the start of specified \p TableKind debug table,
+  /// corresponding to this compile unit.
+  uint64_t getStartOffset(DebugTableKind TableKind) const {
+    return TableOffsets[static_cast<
+        typename std::underlying_type<DebugTableKind>::type>(TableKind)];
+  }
+
+  /// Set offset to the start of specified \p TableKind debug table,
+  /// corresponding to this compile unit.
+  void setStartOffset(DebugTableKind TableKind, uint64_t Offset) {
+    TableOffsets[static_cast<
+        typename std::underlying_type<DebugTableKind>::type>(TableKind)] =
+        Offset;
+  }
+
+  /// Set size of this(newly generated) compile unit.
+  void setUnitSize(uint64_t UnitSize) { this->UnitSize = UnitSize; }
+
+  /// Returns size of this(newly generated) compile unit.
+  uint64_t getUnitSize() const { return UnitSize; }
+
+  /// Returns offset to the DW_AT_stmt_list attribute.
+  Optional<uint64_t> getOffsetToStmtList() const { return OffsetToStmtList; }
+
+  /// Set offset to the DW_AT_stmt_list attribute.
+  void setOffsetToStmtList(uint64_t Offset) { OffsetToStmtList = Offset; }
+
+  /// Returns value of DW_AT_low_pc attribute.
+  uint64_t getLowPc() const { return LowPc; }
+
+  /// Returns value of DW_AT_high_pc attribute.
+  uint64_t getHighPc() const { return HighPc; }
+
+  /// Returns true if there is a label corresponding to the specified \p Addr.
+  bool hasLabelAt(uint64_t Addr) const { return Labels.count(Addr); }
+
+  /// Returns patch information for unit`s DW_AT_range attribute.
+  Optional<RangeAttributePatchInfo> getUnitRangesAttribute() const {
+    return UnitRangeAttribute;
+  }
+
+  /// Returns function ranges of this compile unit.
+  const FunctionIntervals &getFunctionRanges() const { return Ranges; }
+
+  /// Returns list of patches for DW_AT_range attributes of this
+  /// compile units dies.
+  std::vector<RangeAttributePatchInfo> &getRangesAttributes() {
+    return RangeAttributes;
+  }
+
+  /// Returns list of patches for DW_AT_location attributes of this
+  /// compile units dies.
+  std::vector<LocationAttributePatchInfo> &getLocationAttributes() {
+    return LocationAttributes;
+  }
+
+  /// Mark every DIE in this unit as kept. This function also
+  /// marks variables as InDebugMap so that they appear in the
+  /// reconstructed accelerator tables.
+  void markEverythingAsKept();
+
+  /// Keep track of a reference to DIE info \p RefDieIdx in \p RefUnit by \p
+  /// Attr. The attribute should be fixed up later to point to the absolute
+  /// offset of \p Die in the debug_info section or to the canonical offset of
+  /// \p Ctxt if it is non-null.
+  void noteDieReference(CompileUnit::DieRefAttrPatchInfo Data);
+
+  /// Returns list of patches for reference attributes of dies of this compile
+  /// unit.
+  std::vector<CompileUnit::DieRefAttrPatchInfo> getDiesReferences();
+
+  /// Do not clear Input dies after the compile unit is cloned.
+  void setKeepInputDies() { KeepInputDies = true; }
+
+  /// Returns true if Input dies should not be cleared after the compile unit is
+  /// cloned.
+  bool needToKeepInputDies() { return KeepInputDies; }
+
+  /// Add the low_pc of a label that is relocated by applying
+  /// offset \p PCOffset.
+  void addLabelLowPc(uint64_t LabelLowPc, int64_t PcOffset);
+
+  /// Add a function range [\p LowPC, \p HighPC) that is relocated by applying
+  /// offset \p PCOffset.
+  void addFunctionRange(uint64_t LowPC, uint64_t HighPC, int64_t PCOffset);
+
+  /// Keep track of a DW_AT_range attribute that we will need to patch up later.
+  void noteRangeAttribute(const DIE &Die, RangeAttributePatchInfo AttrOffsets);
+
+  /// Keep track of a location attribute pointing to a location list in the
+  /// debug_loc section.
+  void noteLocationAttribute(uint64_t AttrOffset, uint64_t Offset,
+                             int64_t PcOffset);
+
+  /// Add a name accelerator entry for \a Die with \a Name.
+  void addNamespaceAccelerator(const DIE *Die, StringRef Name);
+
+  /// Add a name accelerator entry for \a Die with \a Name.
+  void addNameAccelerator(const DIE *Die, StringRef Name,
+                          bool SkipPubnamesSection = false);
+
+  /// Add various accelerator entries for \p Die with \p Name which is stored
+  /// in the string table at \p Offset. \p Name must be an Objective-C
+  /// selector.
+  void addObjCAccelerator(const DIE *Die, StringRef Name,
+                          bool SkipPubnamesSection = false);
+
+  /// Add a type accelerator entry for \p Die with \p Name which is stored in
+  /// the string table at \p Offset.
+  void addTypeAccelerator(const DIE *Die, StringRef Name,
+                          bool ObjcClassImplementation,
+                          uint32_t QualifiedNameHash);
+
+  /// Returns stored info for creating apple_names/debug_pubnames accelerator
+  /// table.
+  const std::vector<AccelInfo> &getPubnames() const { return Pubnames; }
+
+  /// Returns stored info for creating apple_types/debug_pubtypes accelerator
+  /// table.
+  const std::vector<AccelInfo> &getPubtypes() const { return Pubtypes; }
+
+  /// Returns stored info for creating apple_namespaces accelerator table.
+  const std::vector<AccelInfo> &getNamespaces() const { return Namespaces; }
+
+  /// Returns stored info for creating apple_objc accelerator table.
+  const std::vector<AccelInfo> &getObjC() const { return ObjC; }
+
+  /// Initialize state of compile unit in the start of overall unit processing.
+  void initIteration();
+
+  /// Cleanup unneeded resources after compile unit is cloned.
+  void cleanupDataAfterClonning();
+
+  /// Cleanup unneeded resources after accelerator tables are built.
+  void cleanupAcceleratorTables();
+
+  MCSymbol *getLabelBegin() { return LabelBegin; }
+  void setLabelBegin(MCSymbol *S) { LabelBegin = S; }
+
+  /// Remember type declaration fields, to create declaration context later.
+  void noteContextData(const DeclContextKey *Key, CompileUnit::DIEInfo *Info) {
+    DeclContextData.emplace_back(Key, Info);
+  }
+
+  /// Get stored declaration contexts fields.
+  std::vector<DeclContextKeyData> getDeclContextData() {
+    return std::move(DeclContextData);
+  }
+
+  /// Remember pointer to string and die string attribute. Specified
+  /// \p String would added to the global string pool later and
+  /// \p AttrToPatch would be updated accordingly.
+  void noteOffsetData(DIE::value_iterator AttrToPatch, const char *String) {
+    StringOffsetsData.emplace_back(AttrToPatch, String);
+  }
+
+  /// Get stored strings and attributes to update.
+  std::vector<OffsetsData> getStringsOffsetData() {
+    return std::move(StringOffsetsData);
+  }
+
+  /// Adds \p Abbrev into unit`s abbreviation table.
+  void assignAbbrev(DIEAbbrev &Abbrev);
+
+  /// Returns name of the file for the \p FileIdx
+  /// from the unit`s line table.
+  Optional<StringRef> getFileName(unsigned FileIdx);
+
+  /// Store DeclContextKey body into the Keys container.
+  DeclContextKey *insertKey(DeclContextKey &Key) {
+    DenseSet<DeclContextKey *, DeclKeyMapInfo>::iterator It = Keys.find(&Key);
+    if (It == Keys.end()) {
+      DeclContextKey *NewKey = new (Allocator)
+          DeclContextKey(Key.Hash, Key.Line, Key.ByteSize, Key.Tag, Key.Name,
+                         Key.File, Key.ParentQualifiedNameHash, Key.IsInvalid,
+                         Key.IsDefinedInClangModule);
+
+      It = Keys.insert(NewKey).first;
+    }
+
+    return *It;
+  };
+
+  /// Returns DWARFFile containing this compile unit.
+  const DWARFFile &getContaingFile() const { return ContaingFile; }
+
+  StringRef internString(StringRef S) { return Strings.internString(S); }
+
+  /// Retuirns abbreviations for this compile unit.
+  const std::vector<std::unique_ptr<DIEAbbrev>> &getAbbreviations() {
+    return Abbreviations;
+  }
+
+  /// Returns generated elf file keeping debug tables for this compile unit.
+  SmallString<0> &getDwarfBits() { return OutDebugInfoBytes; }
+
+  /// Erases generated elf file keeping debug tables for this compile unit.
+  void eraseDwarfBits() { OutDebugInfoBytes = SmallString<0>(); }
+
+private:
+  /// \defgroup Members accessed sequentially.
+  ///
+  /// @{
+
+  /// Pointer to the paired compile unit from the input DWARF.
+  Optional<DWARFUnit *> OrigUnit;
+
+  /// Unique ID for the compile unit.
+  unsigned ID = 0;
+
+  /// DWARF version.
+  uint16_t Version = 0;
+
+  /// DIE info indexed by DIE index.
+  std::vector<DIEInfo> Info;
+
+  /// DIE for newly generated compile unit.
+  Optional<BasicDIEUnit> NewUnit;
+  MCSymbol *LabelBegin = nullptr;
+
+  uint64_t LowPc = std::numeric_limits<uint64_t>::max();
+  uint64_t HighPc = 0;
+
+  FunctionIntervals::Allocator RangeAlloc;
+
+  /// The ranges in that interval map are the PC ranges for
+  /// functions in this unit, associated with the PC offset to apply
+  /// to the addresses to get the linked address.
+  FunctionIntervals Ranges;
+
+  /// The DW_AT_low_pc of each DW_TAG_label.
+  SmallDenseMap<uint64_t, uint64_t, 1> Labels;
+
+  /// DW_AT_ranges attributes to patch after we have gathered
+  /// all the unit's function addresses.
+  /// @{
+  std::vector<RangeAttributePatchInfo> RangeAttributes;
+  Optional<RangeAttributePatchInfo> UnitRangeAttribute;
+  /// @}
+
+  /// Location attributes that need to be transferred from the
+  /// original debug_loc section to the linked one. They are stored
+  /// along with the PC offset that is to be applied to their
+  /// function's address.
+  std::vector<LocationAttributePatchInfo> LocationAttributes;
+
+  /// Accelerator entries for the unit, both for the pub*
+  /// sections and the apple* ones.
+  /// @{
+  std::vector<AccelInfo> Pubnames;
+  std::vector<AccelInfo> Pubtypes;
+  std::vector<AccelInfo> Namespaces;
+  std::vector<AccelInfo> ObjC;
+  /// @}
+
+  /// Is this unit subject to the ODR rule?
+  bool HasODR = false;
+
+  /// The DW_AT_language of this unit.
+  uint16_t Language = 0;
+
+  /// The DW_AT_LLVM_sysroot of this unit.
+  std::string SysRoot;
+
+  /// If this is a Clang module, this holds the module's name.
+  std::string ClangModuleName;
+
+  /// FoldingSet that uniques the abbreviations.
+  FoldingSet<DIEAbbrev> AbbreviationsSet;
+
+  /// Storage for the unique Abbreviations.
+  /// This is passed to AsmPrinter::emitDwarfAbbrevs(), thus it cannot be
+  /// changed to a vector of unique_ptrs.
+  std::vector<std::unique_ptr<DIEAbbrev>> Abbreviations;
+
+  /// String pool.
+  NonRelocatableStringpool Strings;
+
+  /// Cached resolved paths from the line table.
+  /// The key is <UniqueUnitID, FileIdx>.
+  using ResolvedPathsMap = DenseMap<unsigned, StringRef>;
+  ResolvedPathsMap ResolvedFullPaths;
+  StringMap<std::string> ResolvedParentPaths;
+
+  /// Data for delayed creation of Declaration contexts.
+  std::vector<DeclContextKeyData> DeclContextData;
+
+  /// Data for delayed creation of String pool entries.
+  std::vector<OffsetsData> StringOffsetsData;
+
+  /// Bodies for DeclContextKeys.
+  DenseSet<DeclContextKey *, DeclKeyMapInfo> Keys;
+  BumpPtrAllocator Allocator;
+
+  /// Offset to DW_AT_stmt_list.
+  Optional<uint64_t> OffsetToStmtList;
+
+  /// DWARFFile containing this compile unit.
+  DWARFFile &ContaingFile;
+
+  /// Offsets to the debug tables corresponding to this compile unit.
+  std::array<uint64_t, CompileUnit::TableKindsNum> TableOffsets = {0};
+  uint64_t UnitSize = 0;
+
+  /// Elf file containg generated debug tables for this compile unit.
+  SmallString<0> OutDebugInfoBytes;
+
+  /// Endiannes for this compile unit.
+  support::endianness Endianess = support::endianness::little;
+
+  /// A list of attributes to fixup with the absolute offset of
+  /// a DIE in the debug_info section.
+  ///
+  /// The offsets for the attributes in this array couldn't be set while
+  /// cloning because for references the target DIE's offset
+  /// isn't known when you emit the reference attribute.
+  std::vector<DieRefAttrPatchInfo> DIEReferences;
+  /// @}
+
+  /// \defgroup Members accessed asinchroniously.
+  ///
+  /// @{
+  /// This flag tells whether input dies could be cleared after
+  /// compile unit is fully processed.
+  std::atomic<bool> KeepInputDies = {false};
+
+  /// Offsets to the Dies referenced from the other compile units.
+  DenseSet<uint64_t> ExternalReferences;
+  mutable std::mutex ExternalReferencesMutex;
+
+  /// This field keeps current stage of overall compile unit processing.
+  std::atomic<Stages> Stage;
+  /// @}
+};
+
+} // end of namespace dwarflinker
+
+} // end namespace llvm
+
+#endif // LLVM_DWARFLINKERNEXT_DWARFLINKERCOMPILEUNIT_H

llvm/include/llvm/DWARFLinkerNext/DWARFLinkerDeclContext.h

+//===- DWARFLinkerDeclContext.h ---------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DWARFLINKERNEXT_DWARFLINKERDECLCONTEXT_H
+#define LLVM_DWARFLINKERNEXT_DWARFLINKERDECLCONTEXT_H
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/DenseMapInfo.h"
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/CodeGen/NonRelocatableStringpool.h"
+#include "llvm/DWARFLinkerNext/DWARFLinkerCompileUnit.h"
+#include "llvm/DebugInfo/DWARF/DWARFDebugLine.h"
+#include "llvm/DebugInfo/DWARF/DWARFDie.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+
+namespace llvm {
+
+namespace dwarflinker {
+
+struct DeclMapInfo;
+struct LinkContext;
+
+/// A DeclContext is a named program scope that is used for ODR uniquing of
+/// types.
+///
+/// The set of DeclContext for the ODR-subject parts of a Dwarf link is
+/// expanded (and uniqued) with each new object file processed. We need to
+/// determine the context of each DIE in an linked object file to see if the
+/// corresponding type has already been emitted.
+///
+/// The contexts are conceptually organized as a tree (eg. a function scope is
+/// contained in a namespace scope that contains other scopes), but
+/// storing/accessing them in an actual tree is too inefficient: we need to be
+/// able to very quickly query a context for a given child context by name.
+/// Storing a StringMap in each DeclContext would be too space inefficient.
+///
+/// The solution here is to give each DeclContext a link to its parent (this
+/// allows to walk up the tree), but to query the existence of a specific
+/// DeclContext using a separate DenseMap keyed on the hash of the fully
+/// qualified name of the context.
+class DeclContext {
+public:
+  using Map = DenseSet<DeclContext *, DeclMapInfo>;
+
+  DeclContext() : DefinedInClangModule(0) {}
+
+  DeclContext(DeclContextKey Key)
+      : QualifiedNameHash(Key.Hash),
+        ParentQualifiedNameHash(Key.ParentQualifiedNameHash), Line(Key.Line),
+        ByteSize(Key.ByteSize), Tag(Key.Tag), Name(Key.Name), File(Key.File),
+        DefinedInClangModule(0) {}
+
+  DeclContext(unsigned Hash, uint32_t Line, uint32_t ByteSize, uint16_t Tag,
+              StringRef Name, StringRef File, unsigned ParentQualifiedNameHash)
+      : QualifiedNameHash(Hash),
+        ParentQualifiedNameHash(ParentQualifiedNameHash), Line(Line),
+        ByteSize(ByteSize), Tag(Tag), Name(Name), File(File),
+        DefinedInClangModule(0) {}
+
+  DeclContextKey getKey() {
+    return DeclContextKey(QualifiedNameHash, Line, ByteSize, Tag, Name, File,
+                          ParentQualifiedNameHash, true, DefinedInClangModule);
+  }
+
+  uint32_t getQualifiedNameHash() const { return QualifiedNameHash; }
+
+  bool setCanonicalInfo(uint64_t DieIndex, CompileUnit *CU) {
+    if (!HasCanonicalDie) {
+      std::unique_lock<std::mutex> Guard(CanonicalInfoMutex);
+
+      if (!HasCanonicalDie) {
+        assert(CU);
+        CanonicalDIEIndex = DieIndex;
+        CanonicalCU = CU;
+        HasCanonicalDie = true;
+        return true;
+      }
+    }
+
+    return false;
+  }
+
+  bool hasCanonicalDie() const { return HasCanonicalDie; }
+  uint32_t getCanonicalDIEIdx() const { return CanonicalDIEIndex; }
+  CompileUnit *getCanonicalCU() const { return CanonicalCU; }
+
+  bool isDefinedInClangModule() const { return DefinedInClangModule; }
+  void setDefinedInClangModule(bool Val) { DefinedInClangModule = Val; }
+
+  uint16_t getTag() const { return Tag; }
+
+private:
+  friend DeclMapInfo;
+
+  /// \defgroup Members accessed sequentially.
+  ///
+  /// @{
+  unsigned QualifiedNameHash = 0;
+  unsigned ParentQualifiedNameHash = 0;
+  uint32_t Line = 0;
+  uint32_t ByteSize = 0;
+  uint16_t Tag = dwarf::DW_TAG_compile_unit;
+  StringRef Name;
+  StringRef File;
+  /// @}
+
+  /// \defgroup Members accessed asinchroniously.
+  ///
+  /// @{
+  std::atomic<bool> DefinedInClangModule = {false};
+  std::atomic<bool> HasCanonicalDie = {false};
+  uint32_t CanonicalDIEIndex = 0;
+  CompileUnit *CanonicalCU = nullptr;
+  std::mutex CanonicalInfoMutex;
+  /// @}
+};
+
+/// This class gives a tree-like API to the DenseMap that stores the
+/// DeclContext objects. It holds the BumpPtrAllocator where these objects will
+/// be allocated.
+class DeclContextTree {
+public:
+  DeclContextKey *getChildDeclContextKey(const LinkContext &Context,
+                                         DeclContextKey *ContextKey,
+                                         const DWARFDie &DIE, CompileUnit &Unit,
+                                         bool InClangModule,
+                                         bool &IsUnresolved);
+
+  /// Get the child of \a Context described by \a DIE in \a Unit. The
+  /// required strings will be interned in \a StringPool.
+  /// \returns The child DeclContext along with one bit that is set if
+  /// this context is invalid.
+  ///
+  /// An invalid context means it shouldn't be considered for uniquing, but its
+  /// not returning null, because some children of that context might be
+  /// uniquing candidates.
+  DeclContext *getChildDeclContext(const DeclContextKey &ContextKey);
+
+  DeclContextKey &getRootKey() { return RootKey; }
+
+  void clear() {
+    Contexts.clear();
+    Allocator.Reset();
+    StringPool.clear();
+  }
+
+private:
+  BumpPtrAllocator Allocator;
+  DeclContextKey RootKey;
+  DeclContext::Map Contexts;
+
+  /// String pool keeping type name and file name bodies.
+  NonRelocatableStringpool StringPool;
+};
+
+/// Info type for the DenseMap storing the DeclContext pointers.
+struct DeclMapInfo : private DenseMapInfo<DeclContext *> {
+  using DenseMapInfo<DeclContext *>::getEmptyKey;
+  using DenseMapInfo<DeclContext *>::getTombstoneKey;
+
+  static unsigned getHashValue(const DeclContext *Ctxt) {
+    return Ctxt->QualifiedNameHash;
+  }
+
+  static bool isEqual(const DeclContext *LHS, const DeclContext *RHS) {
+    if (RHS == getEmptyKey() || RHS == getTombstoneKey())
+      return RHS == LHS;
+    return LHS->QualifiedNameHash == RHS->QualifiedNameHash &&
+           LHS->ParentQualifiedNameHash == RHS->ParentQualifiedNameHash &&
+           LHS->Line == RHS->Line && LHS->ByteSize == RHS->ByteSize &&
+           LHS->Name == RHS->Name && LHS->File == RHS->File;
+  }
+};
+
+} // end of namespace dwarflinker
+
+} // end namespace llvm
+
+#endif // LLVM_DWARFLINKERNEXT_DWARFLINKERDECLCONTEXT_H

llvm/include/llvm/DWARFLinkerNext/DWARFStreamer.h

+//===- DwarfStreamer.h ------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DWARFLINKERNEXT_DWARFSTREAMER_H
+#define LLVM_DWARFLINKERNEXT_DWARFSTREAMER_H
+
+#include "llvm/CodeGen/AccelTable.h"
+#include "llvm/CodeGen/AsmPrinter.h"
+#include "llvm/DWARFLinkerNext/DWARFLinker.h"
+#include "llvm/MC/MCAsmInfo.h"
+#include "llvm/MC/MCContext.h"
+#include "llvm/MC/MCInstrInfo.h"
+#include "llvm/MC/MCObjectFileInfo.h"
+#include "llvm/MC/MCRegisterInfo.h"
+#include "llvm/Target/TargetMachine.h"
+
+namespace llvm {
+
+///   User of DwarfStreamer should call initialization code
+///   for AsmPrinter:
+///
+///   InitializeAllTargetInfos();
+///   InitializeAllTargetMCs();
+///   InitializeAllTargets();
+///   InitializeAllAsmPrinters();
+
+class MCCodeEmitter;
+
+namespace dwarflinker {
+
+enum class OutputFileType {
+  Object,
+  Assembly,
+};
+
+/// The Dwarf streaming logic.
+///
+/// All interactions with the MC layer that is used to build the debug
+/// information binary representation are handled in this class.
+class DwarfStreamer : public DwarfEmitter {
+public:
+  DwarfStreamer(OutputFileType OutFileType, raw_pwrite_stream &OutFile,
+                std::function<StringRef(StringRef Input)> Translator,
+                messageHandler Error, messageHandler Warning)
+      : OutFile(OutFile), OutFileType(OutFileType), Translator(Translator),
+        ErrorHandler(Error), WarningHandler(Warning) {}
+
+  bool init(Triple TheTriple);
+  const Triple &getTargetTriple() const override {
+    return MOFI->getTargetTriple();
+  }
+
+  /// Dump the file to the disk.
+  void finish();
+
+  AsmPrinter &getAsmPrinter() const { return *Asm; }
+
+  /// Set the current output section to debug_info and change
+  /// the MC Dwarf version to \p DwarfVersion.
+  void switchToDebugInfoSection(unsigned DwarfVersion);
+
+  /// Emit the compilation unit header for \p Unit in the
+  /// debug_info section.
+  ///
+  /// As a side effect, this also switches the current Dwarf version
+  /// of the MC layer to the one of U.getOrigUnit().
+  void emitCompileUnitHeader(CompileUnit &Unit) override;
+
+  /// Recursively emit the DIE tree rooted at \p Die.
+  void emitDIE(DIE &Die) override;
+
+  /// Emit the abbreviation table \p Abbrevs to the debug_abbrev section.
+  void emitAbbrevs(const std::vector<std::unique_ptr<DIEAbbrev>> &Abbrevs,
+                   unsigned DwarfVersion) override;
+
+  /// Emit DIE containing warnings.
+  void emitPaperTrailWarningsDie(DIE &Die) override;
+
+  /// Emit contents of section SecName From Obj.
+  void emitSectionContents(StringRef SecData, StringRef SecName) override;
+
+  /// Emit the string table described by \p Pool.
+  void emitStrings(const NonRelocatableStringpool &Pool) override;
+
+  /// Emit the swift_ast section stored in \p Buffer.
+  void emitSwiftAST(StringRef Buffer);
+
+  /// Emit debug_ranges for \p FuncRange by translating the
+  /// original \p Entries.
+  void emitRangesEntries(
+      int64_t UnitPcOffset, uint64_t OrigLowPc,
+      const FunctionIntervals::const_iterator &FuncRange,
+      const std::vector<DWARFDebugRangeList::RangeListEntry> &Entries,
+      unsigned AddressSize) override;
+
+  /// Emit debug_aranges entries for \p Unit and if \p DoRangesSection is true,
+  /// also emit the debug_ranges entries for the DW_TAG_compile_unit's
+  /// DW_AT_ranges attribute.
+  void emitUnitRangesEntries(CompileUnit &Unit, bool DoRangesSection) override;
+
+  uint64_t getRangesSectionSize() const override { return RangesSectionSize; }
+
+  /// Emit the debug_loc contribution for \p Unit by copying the entries from
+  /// \p Dwarf and offsetting them. Update the location attributes to point to
+  /// the new entries.
+  void emitLocationsForUnit(
+      CompileUnit &Unit, DWARFContext &Dwarf,
+      std::function<void(StringRef, SmallVectorImpl<uint8_t> &)> ProcessExpr)
+      override;
+
+  /// Emit the line table described in \p Rows into the debug_line section.
+  void emitLineTableForUnit(MCDwarfLineTableParams Params,
+                            StringRef PrologueBytes, unsigned MinInstLength,
+                            std::vector<DWARFDebugLine::Row> &Rows,
+                            unsigned AdddressSize) override;
+
+  /// Copy the debug_line over to the updated binary while unobfuscating the
+  /// file names and directories.
+  void translateLineTable(DataExtractor LineData, uint64_t Offset) override;
+
+  uint64_t getLineSectionSize() const override { return LineSectionSize; }
+
+  /// Emit the .debug_pubnames contribution for \p Unit.
+  void emitPubNamesForUnit(const CompileUnit &Unit) override;
+
+  /// Emit the .debug_pubtypes contribution for \p Unit.
+  void emitPubTypesForUnit(const CompileUnit &Unit) override;
+
+  /// Emit a CIE.
+  void emitCIE(StringRef CIEBytes) override;
+
+  /// Emit an FDE with data \p Bytes.
+  void emitFDE(uint32_t CIEOffset, uint32_t AddreSize, uint32_t Address,
+               StringRef Bytes) override;
+
+  /// Emit DWARF debug names.
+  void emitDebugNames(AccelTable<DWARF5AccelTableStaticData> &Table) override;
+
+  /// Emit Apple namespaces accelerator table.
+  void emitAppleNamespaces(
+      AccelTable<AppleAccelTableStaticOffsetData> &Table) override;
+
+  /// Emit Apple names accelerator table.
+  void
+  emitAppleNames(AccelTable<AppleAccelTableStaticOffsetData> &Table) override;
+
+  /// Emit Apple Objective-C accelerator table.
+  void
+  emitAppleObjc(AccelTable<AppleAccelTableStaticOffsetData> &Table) override;
+
+  /// Emit Apple type accelerator table.
+  void
+  emitAppleTypes(AccelTable<AppleAccelTableStaticTypeData> &Table) override;
+
+  uint64_t getFrameSectionSize() const override { return FrameSectionSize; }
+
+  uint64_t getDebugInfoSectionSize() const override {
+    return DebugInfoSectionSize;
+  }
+
+private:
+  inline void error(const Twine &Error, StringRef Context = "") {
+    if (ErrorHandler)
+      ErrorHandler(Error, Context, nullptr);
+  }
+
+  inline void warn(const Twine &Warning, StringRef Context = "") {
+    if (WarningHandler)
+      WarningHandler(Warning, Context, nullptr);
+  }
+
+  /// \defgroup MCObjects MC layer objects constructed by the streamer
+  /// @{
+  std::unique_ptr<MCRegisterInfo> MRI;
+  std::unique_ptr<MCAsmInfo> MAI;
+  std::unique_ptr<MCObjectFileInfo> MOFI;
+  std::unique_ptr<MCContext> MC;
+  MCAsmBackend *MAB; // Owned by MCStreamer
+  std::unique_ptr<MCInstrInfo> MII;
+  std::unique_ptr<MCSubtargetInfo> MSTI;
+  MCInstPrinter *MIP; // Owned by AsmPrinter
+  MCCodeEmitter *MCE; // Owned by MCStreamer
+  MCStreamer *MS;     // Owned by AsmPrinter
+  std::unique_ptr<TargetMachine> TM;
+  std::unique_ptr<AsmPrinter> Asm;
+  /// @}
+
+  /// The output file we stream the linked Dwarf to.
+  raw_pwrite_stream &OutFile;
+  OutputFileType OutFileType = OutputFileType::Object;
+  std::function<StringRef(StringRef Input)> Translator;
+
+  uint64_t RangesSectionSize = 0;
+  uint64_t LocSectionSize = 0;
+  uint64_t LineSectionSize = 0;
+  uint64_t FrameSectionSize = 0;
+  uint64_t DebugInfoSectionSize = 0;
+
+  /// Keep track of emitted CUs and their Unique ID.
+  struct EmittedUnit {
+    unsigned ID;
+    MCSymbol *LabelBegin;
+  };
+  std::vector<EmittedUnit> EmittedUnits;
+
+  /// Emit the pubnames or pubtypes section contribution for \p
+  /// Unit into \p Sec. The data is provided in \p Names.
+  void emitPubSectionForUnit(MCSection *Sec, StringRef Name,
+                             const CompileUnit &Unit,
+                             const std::vector<CompileUnit::AccelInfo> &Names);
+
+  messageHandler ErrorHandler = nullptr;
+  messageHandler WarningHandler = nullptr;
+};
+
+} // end of namespace dwarflinker
+
+} // end namespace llvm
+
+#endif // LLVM_DWARFLINKERNEXT_DWARFSTREAMER_H

llvm/lib/CMakeLists.txt

 include(LLVM-Build)
 
 # `Demangle', `Support' and `TableGen' libraries are added on the top-level
 # CMakeLists.txt
 
 add_subdirectory(IR)
 add_subdirectory(FuzzMutate)
 add_subdirectory(FileCheck)
 add_subdirectory(InterfaceStub)
 add_subdirectory(IRReader)
 add_subdirectory(CodeGen)
 add_subdirectory(BinaryFormat)
 add_subdirectory(Bitcode)
 add_subdirectory(Bitstream)
 add_subdirectory(DWARFLinker)
+add_subdirectory(DWARFLinkerNext)
 add_subdirectory(Extensions)
 add_subdirectory(Frontend)
 add_subdirectory(Transforms)
 add_subdirectory(Linker)
 add_subdirectory(Analysis)
 add_subdirectory(LTO)
 add_subdirectory(MC)
 add_subdirectory(MCA)

llvm/lib/DWARFLinker/DWARFLinker.cpp

 /// ReferencedCU. \returns null if resolving fails for any reason.
 DWARFDie DWARFLinker::resolveDIEReference(const DWARFFile &File,
                                           const UnitListTy &Units,
                                           const DWARFFormValue &RefValue,
                                           const DWARFDie &DIE,
                                           CompileUnit *&RefCU) {
   assert(RefValue.isFormClass(DWARFFormValue::FC_Reference));
   uint64_t RefOffset = *RefValue.getAsReference();
-  if ((RefCU = getUnitForOffset(Units, RefOffset)))
+  if ((RefCU = getUnitForOffset(Units, RefOffset))) {
     if (const auto RefDie = RefCU->getOrigUnit().getDIEForOffset(RefOffset)) {
       // In a file with broken references, an attribute might point to a NULL
       // DIE.
       if (!RefDie.isNULL())
         return RefDie;
     }
+  }
 
   reportWarning("could not find referenced DIE", File, &DIE);
   return DWARFDie();
 }
 
 /// \returns whether the passed \a Attr type might contain a DIE reference
 /// suitable for ODR uniquing.
 static bool isODRAttribute(uint16_t Attr) {
void DWARFLinker::patchLineTableForUnit(CompileUnit &Unit,
   if (!StmtList)
     return;
 
   // Update the cloned DW_AT_stmt_list with the correct debug_line offset.
   if (auto *OutputDIE = Unit.getOutputUnitDIE())
     patchStmtList(*OutputDIE,
                   DIEInteger(TheDwarfEmitter->getLineSectionSize()));
 
-  RangesTy &Ranges = File.Addresses->getValidAddressRanges();
+  RangesTy *Ranges;
+  File.Addresses->getValidAddressRanges(Ranges);
 
   // Parse the original line info for the unit.
   DWARFDebugLine::LineTable LineTable;
   uint64_t StmtOffset = *StmtList;
   DWARFDataExtractor LineExtractor(
       OrigDwarf.getDWARFObj(), OrigDwarf.getDWARFObj().getLineSection(),
       OrigDwarf.isLittleEndian(), Unit.getOrigUnit().getAddressByteSize());
   if (needToTranslateStrings())
if (CurrRange == InvalidRange || Row.Address.Address < CurrRange.start() ||
         CurrRange = InvalidRange;
         if (StopAddress != -1ULL) {
           // Try harder by looking in the Address ranges map.
           // There are corner cases where this finds a
           // valid entry. It's unclear if this is right or wrong, but
           // for now do as dsymutil.
           // FIXME: Understand exactly what cases this addresses and
           // potentially remove it along with the Ranges map.
-          auto Range = Ranges.lower_bound(Row.Address.Address);
-          if (Range != Ranges.begin() && Range != Ranges.end())
+          auto Range = Ranges->lower_bound(Row.Address.Address);
+          if (Range != Ranges->begin() && Range != Ranges->end())
             --Range;
 
-          if (Range != Ranges.end() && Range->first <= Row.Address.Address &&
+          if (Range != Ranges->end() && Range->first <= Row.Address.Address &&
               Range->second.HighPC >= Row.Address.Address) {
             StopAddress = Row.Address.Address + Range->second.Offset;
           }
         }
       }
       if (StopAddress != -1ULL && !Seq.empty()) {
         // Insert end sequence row with the computed end address, but
         // the same line as the previous one.
auto CloneLambda = [&](size_t I) {
     // Note that this loop can not be merged with the previous one because
     // cross-cu references require the ParentIdx to be setup for every CU in
     // the object file before calling this.
     if (LLVM_UNLIKELY(Options.Update)) {
       for (auto &CurrentUnit : OptContext.CompileUnits)
         CurrentUnit->markEverythingAsKept();
       copyInvariantDebugSection(*OptContext.File.Dwarf);
     } else {
+      RangesTy *Ranges;
+      OptContext.File.Addresses->getValidAddressRanges(Ranges);
       for (auto &CurrentUnit : OptContext.CompileUnits)
-        lookForDIEsToKeep(*OptContext.File.Addresses,
-                          OptContext.File.Addresses->getValidAddressRanges(),
+        lookForDIEsToKeep(*OptContext.File.Addresses, *Ranges,
                           OptContext.CompileUnits,
                           CurrentUnit->getOrigUnit().getUnitDIE(),
                           OptContext.File, *CurrentUnit, 0);
     }
 
     // The calls to applyValidRelocs inside cloneDIE will walk the reloc
     // array again (in the same way findValidRelocsInDebugInfo() did). We
     // need to reset the NextValidReloc index to the beginning.
     if (OptContext.File.Addresses->hasValidRelocs() ||
         LLVM_UNLIKELY(Options.Update)) {
       SizeByObject[OptContext.File.FileName].Input =
           getDebugInfoSize(*OptContext.File.Dwarf);
       SizeByObject[OptContext.File.FileName].Output =
           DIECloner(*this, TheDwarfEmitter, OptContext.File, DIEAlloc,
                     OptContext.CompileUnits, Options.Update)
               .cloneAllCompileUnits(*OptContext.File.Dwarf, OptContext.File,
                                     OffsetsStringPool,
                                     OptContext.File.Dwarf->isLittleEndian());
     }
     if (!Options.NoOutput && !OptContext.CompileUnits.empty() &&
-        LLVM_LIKELY(!Options.Update))
+        LLVM_LIKELY(!Options.Update)) {
+      RangesTy *Ranges;
+      OptContext.File.Addresses->getValidAddressRanges(Ranges);
+
       patchFrameInfoForObject(
-          OptContext.File, OptContext.File.Addresses->getValidAddressRanges(),
-          *OptContext.File.Dwarf,
+          OptContext.File, *Ranges, *OptContext.File.Dwarf,
           OptContext.CompileUnits[0]->getOrigUnit().getAddressByteSize());
+    }
 
     // Clean-up before starting working on the next object.
     cleanupAuxiliarryData(OptContext);
   };
 
   auto EmitLambda = [&]() {
     // Emit everything that's global.
     if (!Options.NoOutput) {

llvm/lib/DWARFLinkerNext/CMakeLists.txt

+add_llvm_component_library(LLVMDWARFLinkerNext
+  DWARFLinkerCompileUnit.cpp
+  DWARFLinkerDeclContext.cpp
+  DWARFLinker.cpp
+  DWARFStreamer.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${LLVM_MAIN_INCLUDE_DIR}/llvm/DWARFLinkerNext
+
+  DEPENDS
+  intrinsics_gen
+
+  LINK_COMPONENTS
+  BinaryFormat
+  DebugInfoDWARF
+  AsmPrinter
+  CodeGen
+  MC
+  Object
+  Support
+  )

llvm/lib/DWARFLinkerNext/DWARFLinker.cpp

+//=== DWARFLinker.cpp -----------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DWARFLinkerNext/DWARFLinker.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/BitVector.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/Triple.h"
+#include "llvm/CodeGen/NonRelocatableStringpool.h"
+#include "llvm/DWARFLinkerNext/DWARFLinkerDeclContext.h"
+#include "llvm/DWARFLinkerNext/DWARFStreamer.h"
+#include "llvm/DebugInfo/DWARF/DWARFAbbreviationDeclaration.h"
+#include "llvm/DebugInfo/DWARF/DWARFContext.h"
+#include "llvm/DebugInfo/DWARF/DWARFDataExtractor.h"
+#include "llvm/DebugInfo/DWARF/DWARFDebugLine.h"
+#include "llvm/DebugInfo/DWARF/DWARFDebugRangeList.h"
+#include "llvm/DebugInfo/DWARF/DWARFDie.h"
+#include "llvm/DebugInfo/DWARF/DWARFFormValue.h"
+#include "llvm/DebugInfo/DWARF/DWARFSection.h"
+#include "llvm/DebugInfo/DWARF/DWARFUnit.h"
+#include "llvm/Support/DataExtractor.h"
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/ErrorOr.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/LEB128.h"
+#include "llvm/Support/Parallel.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/ThreadPool.h"
+#include <vector>
+
+namespace llvm {
+
+namespace dwarflinker {
+
+/// Compute the total size of the debug info.
+static uint64_t getDebugInfoSize(DWARFContext &Dwarf) {
+  uint64_t Size = 0;
+  for (auto &Unit : Dwarf.compile_units()) {
+    Size += Unit->getLength();
+  }
+  return Size;
+}
+
+/// Similar to DWARFUnitSection::getUnitForOffset(), but returning our
+/// CompileUnit object instead.
+CompileUnit *LinkContext::getUnitForOffset(CompileUnit &CurrentCU,
+                                           uint64_t Offset) const {
+  if (CurrentCU.isClangModule())
+    return &CurrentCU;
+
+  auto CU = llvm::upper_bound(
+      CompileUnits, Offset,
+      [](uint64_t LHS, const std::unique_ptr<CompileUnit> &RHS) {
+        return LHS < RHS->getOrigUnit().getNextUnitOffset();
+      });
+
+  return CU != CompileUnits.end() ? CU->get() : nullptr;
+}
+
+DWARFLinker::DWARFLinker(DwarfEmitter *Emitter, DwarfLinkerClient ClientID,
+                         std::function<StringRef(StringRef)> StringsTranslator)
+    : UniqueUnitID(0), TheOffsetsStringPool(StringsTranslator, true),
+      NeedToTranslateStrings(StringsTranslator != nullptr),
+      DwarfLinkerClientID(ClientID), TheDwarfEmitter(Emitter) {}
+
+DWARFDie DWARFLinker::resolveDIEReference(const LinkContext &Context,
+                                          CompileUnit &CU,
+                                          const DWARFFormValue &RefValue,
+                                          const DWARFDie &DIE,
+                                          CompileUnit *&RefCU) {
+  assert(RefValue.isFormClass(DWARFFormValue::FC_Reference));
+  uint64_t RefOffset = *RefValue.getAsReference();
+
+  if ((RefCU = Context.getUnitForOffset(CU, RefOffset)))
+    if (const auto RefDie = RefCU->getOrigUnit().getDIEForOffset(RefOffset)) {
+      // In a file with broken references, an attribute might point to a NULL
+      // DIE.
+      if (!RefDie.isNULL())
+        return RefDie;
+    }
+
+  reportWarning("could not find referenced DIE", CU.getContaingFile(), &DIE);
+  return DWARFDie();
+}
+
+/// \returns whether the passed \a Attr type might contain a DIE reference
+/// suitable for ODR uniquing.
+static bool isODRAttribute(uint16_t Attr) {
+  switch (Attr) {
+  default:
+    return false;
+  case dwarf::DW_AT_type:
+  case dwarf::DW_AT_containing_type:
+  case dwarf::DW_AT_specification:
+  case dwarf::DW_AT_abstract_origin:
+  case dwarf::DW_AT_import:
+    return true;
+  }
+  llvm_unreachable("Improper attribute.");
+}
+
+static bool isTypeTag(uint16_t Tag) {
+  switch (Tag) {
+  case dwarf::DW_TAG_array_type:
+  case dwarf::DW_TAG_class_type:
+  case dwarf::DW_TAG_enumeration_type:
+  case dwarf::DW_TAG_pointer_type:
+  case dwarf::DW_TAG_reference_type:
+  case dwarf::DW_TAG_string_type:
+  case dwarf::DW_TAG_structure_type:
+  case dwarf::DW_TAG_subroutine_type:
+  case dwarf::DW_TAG_typedef:
+  case dwarf::DW_TAG_union_type:
+  case dwarf::DW_TAG_ptr_to_member_type:
+  case dwarf::DW_TAG_set_type:
+  case dwarf::DW_TAG_subrange_type:
+  case dwarf::DW_TAG_base_type:
+  case dwarf::DW_TAG_const_type:
+  case dwarf::DW_TAG_constant:
+  case dwarf::DW_TAG_file_type:
+  case dwarf::DW_TAG_namelist:
+  case dwarf::DW_TAG_packed_type:
+  case dwarf::DW_TAG_volatile_type:
+  case dwarf::DW_TAG_restrict_type:
+  case dwarf::DW_TAG_atomic_type:
+  case dwarf::DW_TAG_interface_type:
+  case dwarf::DW_TAG_unspecified_type:
+  case dwarf::DW_TAG_shared_type:
+    return true;
+  default:
+    break;
+  }
+  return false;
+}
+
+AddressesMap::~AddressesMap() {}
+
+DwarfEmitter::~DwarfEmitter() {}
+
+static Optional<StringRef> StripTemplateParameters(StringRef Name) {
+  // We are looking for template parameters to strip from Name. e.g.
+  //
+  //  operator<<B>
+  //
+  // We look for > at the end but if it does not contain any < then we
+  // have something like operator>>. We check for the operator<=> case.
+  if (!Name.endswith(">") || Name.count("<") == 0 || Name.endswith("<=>"))
+    return {};
+
+  // How many < until we have the start of the template parameters.
+  size_t NumLeftAnglesToSkip = 1;
+
+  // If we have operator<=> then we need to skip its < as well.
+  NumLeftAnglesToSkip += Name.count("<=>");
+
+  size_t RightAngleCount = Name.count('>');
+  size_t LeftAngleCount = Name.count('<');
+
+  // If we have more < than > we have operator< or operator<<
+  // we to account for their < as well.
+  if (LeftAngleCount > RightAngleCount)
+    NumLeftAnglesToSkip += LeftAngleCount - RightAngleCount;
+
+  size_t StartOfTemplate = 0;
+  while (NumLeftAnglesToSkip--)
+    StartOfTemplate = Name.find('<', StartOfTemplate) + 1;
+
+  return Name.substr(0, StartOfTemplate - 1);
+}
+
+bool DWARFLinker::DIECloner::getDIENames(const DWARFDie &Die,
+                                         AttributesInfo &Info,
+                                         bool StripTemplate) {
+  // This function will be called on DIEs having low_pcs and
+  // ranges. As getting the name might be more expansive, filter out
+  // blocks directly.
+  if (Die.getTag() == dwarf::DW_TAG_lexical_block)
+    return false;
+
+  if (Info.MangledName.empty())
+    Info.MangledName = Die.getLinkageName();
+
+  if (Info.Name.empty())
+    Info.Name = Die.getShortName();
+
+  if (Info.MangledName.empty())
+    Info.MangledName = Info.Name;
+
+  if (StripTemplate && !Info.Name.empty() && Info.MangledName != Info.Name) {
+    // StringRef Name = Info.Name.getString();
+    if (Optional<StringRef> StrippedName = StripTemplateParameters(Info.Name))
+      Info.NameWithoutTemplate = *StrippedName;
+  }
+
+  return !Info.Name.empty() || !Info.MangledName.empty();
+}
+
+/// Resolve the relative path to a build artifact referenced by DWARF by
+/// applying DW_AT_comp_dir.
+static void resolveRelativeObjectPath(SmallVectorImpl<char> &Buf, DWARFDie CU) {
+  sys::path::append(Buf, dwarf::toString(CU.find(dwarf::DW_AT_comp_dir), ""));
+}
+
+/// Collect references to parseable Swift interfaces in imported
+/// DW_TAG_module blocks.
+void DWARFLinker::analyzeImportedModule(
+    const DWARFDie &DIE, CompileUnit &CU,
+    swiftInterfacesMap *ParseableSwiftInterfaces) {
+  if (CU.getLanguage() != dwarf::DW_LANG_Swift)
+    return;
+
+  if (!ParseableSwiftInterfaces)
+    return;
+
+  StringRef Path = dwarf::toStringRef(DIE.find(dwarf::DW_AT_LLVM_include_path));
+  if (!Path.endswith(".swiftinterface"))
+    return;
+  // Don't track interfaces that are part of the SDK.
+  StringRef SysRoot = dwarf::toStringRef(DIE.find(dwarf::DW_AT_LLVM_sysroot));
+  if (SysRoot.empty())
+    SysRoot = CU.getSysRoot();
+  if (!SysRoot.empty() && Path.startswith(SysRoot))
+    return;
+  if (Optional<DWARFFormValue> Val = DIE.find(dwarf::DW_AT_name))
+    if (Optional<const char *> Name = Val->getAsCString()) {
+      auto &Entry = (*ParseableSwiftInterfaces)[*Name];
+      // The prepend path is applied later when copying.
+      DWARFDie CUDie = CU.getOrigUnit().getUnitDIE();
+      SmallString<128> ResolvedPath;
+      if (sys::path::is_relative(Path))
+        resolveRelativeObjectPath(ResolvedPath, CUDie);
+      sys::path::append(ResolvedPath, Path);
+      if (!Entry.empty() && Entry != ResolvedPath && Options.WarningHandler)
+        Options.WarningHandler(
+            Twine("Conflicting parseable interfaces for Swift Module ") +
+                *Name + ": " + Entry + " and " + Path,
+            CU.getContaingFile().FileName, &DIE);
+      Entry = std::string(ResolvedPath.str());
+    }
+}
+
+/// The distinct types of work performed by the work loop in
+/// analyzeContextInfo.
+enum class ContextWorklistItemType : uint8_t {
+  AnalyzeContextInfo,
+  UpdateChildPruning,
+  UpdatePruning,
+};
+
+/// This class represents an item in the work list. The type defines what kind
+/// of work needs to be performed when processing the current item. Everything
+/// but the Type and Die fields are optional based on the type.
+struct ContextWorklistItem {
+  DWARFDie Die;
+  unsigned ParentIdx;
+  union {
+    CompileUnit::DIEInfo *OtherInfo;
+    DeclContextKey *ContextKey;
+  };
+  ContextWorklistItemType Type;
+  bool InImportedModule;
+
+  ContextWorklistItem(DWARFDie Die, ContextWorklistItemType T,
+                      CompileUnit::DIEInfo *OtherInfo = nullptr)
+      : Die(Die), ParentIdx(0), OtherInfo(OtherInfo), Type(T),
+        InImportedModule(false) {}
+
+  ContextWorklistItem(DWARFDie Die, DeclContextKey *ContextKeyPtr,
+                      unsigned ParentIdx, bool InImportedModule)
+      : Die(Die), ParentIdx(ParentIdx), ContextKey(ContextKeyPtr),
+        Type(ContextWorklistItemType::AnalyzeContextInfo),
+        InImportedModule(InImportedModule) {}
+};
+
+static bool updatePruning(const DWARFDie &Die, CompileUnit &CU) {
+  CompileUnit::DIEInfo &Info = CU.getInfo(Die);
+
+  // Prune this DIE if it is either a forward declaration inside a
+  // DW_TAG_module or a DW_TAG_module that contains nothing but
+  // forward declarations.
+  Info.Prune &= (Die.getTag() == dwarf::DW_TAG_module) ||
+                (isTypeTag(Die.getTag()) &&
+                 dwarf::toUnsigned(Die.find(dwarf::DW_AT_declaration), 0));
+
+  // Only prune forward declarations inside a DW_TAG_module for which a
+  // definition exists elsewhere.
+  Info.Prune &=
+      Info.Ctxt && Info.Ctxt.load()->hasCanonicalDie() &&
+      Info.Ctxt.load()->getCanonicalCU()->getUniqueID() != CU.getUniqueID();
+
+  return Info.Prune;
+}
+
+static void updateChildPruning(const DWARFDie &Die, CompileUnit &CU,
+                               CompileUnit::DIEInfo &ChildInfo) {
+  CompileUnit::DIEInfo &Info = CU.getInfo(Die);
+  Info.Prune &= ChildInfo.Prune;
+}
+
+/// Recursive helper to build the global DeclContext information and
+/// gather the child->parent relationships in the original compile unit.
+///
+/// This function uses the same work list approach as lookForDIEsToKeep.
+///
+/// \return true when this DIE and all of its children are only
+/// forward declarations to types defined in external clang modules
+/// (i.e., forward declarations that are children of a DW_TAG_module).
+bool DWARFLinker::analyzeContextInfo(
+    const LinkContext &Context, const DWARFDie &DIE, unsigned ParentIdx,
+    CompileUnit &CU, swiftInterfacesMap *ParseableSwiftInterfaces,
+    bool &HasUnresolvedRefs) {
+  // LIFO work list.
+  std::vector<ContextWorklistItem> Worklist;
+  DeclContextKey *RootKey = CU.insertKey(ODRContexts.getRootKey());
+  Worklist.emplace_back(DIE, RootKey, ParentIdx, CU.isClangModule());
+
+  while (!Worklist.empty()) {
+    ContextWorklistItem Current = Worklist.back();
+    Worklist.pop_back();
+
+    switch (Current.Type) {
+    case ContextWorklistItemType::UpdatePruning:
+      updatePruning(Current.Die, CU);
+      continue;
+    case ContextWorklistItemType::UpdateChildPruning:
+      updateChildPruning(Current.Die, CU, *Current.OtherInfo);
+      continue;
+    case ContextWorklistItemType::AnalyzeContextInfo:
+      break;
+    }
+
+    unsigned Idx = CU.getOrigUnit().getDIEIndex(Current.Die);
+    CompileUnit::DIEInfo &Info = CU.getInfo(Idx);
+
+    // Clang imposes an ODR on modules(!) regardless of the language:
+    //  "The module-id should consist of only a single identifier,
+    //   which provides the name of the module being defined. Each
+    //   module shall have a single definition."
+    //
+    // This does not extend to the types inside the modules:
+    //  "[I]n C, this implies that if two structs are defined in
+    //   different submodules with the same name, those two types are
+    //   distinct types (but may be compatible types if their
+    //   definitions match)."
+    //
+    // We treat non-C++ modules like namespaces for this reason.
+    if (Current.Die.getTag() == dwarf::DW_TAG_module &&
+        Current.ParentIdx == 0 &&
+        dwarf::toString(Current.Die.find(dwarf::DW_AT_name), "") !=
+            CU.getClangModuleName()) {
+      Current.InImportedModule = true;
+      analyzeImportedModule(Current.Die, CU, ParseableSwiftInterfaces);
+    }
+
+    Info.ParentIdx = Current.ParentIdx;
+    bool InClangModule = CU.isClangModule() || Current.InImportedModule;
+    Info.Ctxt = nullptr;
+    if (CU.hasODR() || InClangModule) {
+      if (Current.ContextKey) {
+
+        bool IsUnresolved = false;
+        DeclContextKey *NewContextKey = ODRContexts.getChildDeclContextKey(
+            Context, Current.ContextKey, Current.Die, CU, InClangModule,
+            IsUnresolved);
+
+        if (IsUnresolved) {
+          // Current.Die references other compile unit, which is not loaded yet.
+          // Stop type declaration analyzing for now.
+          HasUnresolvedRefs = true;
+          return false;
+        }
+
+        if (NewContextKey) {
+          Current.ContextKey = NewContextKey;
+
+          if (IsMT()) {
+            // In multy-threaded case it is in-effective to lock&use
+            // common resource ODRContexts.getChildDeclContext() for
+            // every type declaration context. Instead, we remember
+            // DeclarationContextKey and create DeclarationContext and
+            // update Info.Ctxt field later. That way we lock
+            // ODRContexts.getChildDeclContext() only once for each compile
+            // unit.
+
+            CU.noteContextData(Current.ContextKey, &Info);
+          } else {
+            setDeclContext(Current.ContextKey, &Info);
+          }
+        }
+      }
+    }
+    Info.Prune = Current.InImportedModule;
+    // Add children in reverse order to the worklist to effectively process
+    // them in order.
+    Worklist.emplace_back(Current.Die, ContextWorklistItemType::UpdatePruning);
+    for (DWARFDie Child = Current.Die.getLastChild(); Child.isValid();
+         Child = Child.getPreviousSibling()) {
+      CompileUnit::DIEInfo &ChildInfo = CU.getInfo(Child);
+      Worklist.emplace_back(
+          Current.Die, ContextWorklistItemType::UpdateChildPruning, &ChildInfo);
+      Worklist.emplace_back(Child, Current.ContextKey, Idx,
+                            Current.InImportedModule);
+    }
+  }
+
+  return CU.getInfo(DIE).Prune;
+}
+
+static bool dieNeedsChildrenToBeMeaningful(uint32_t Tag) {
+  switch (Tag) {
+  default:
+    return false;
+  case dwarf::DW_TAG_class_type:
+  case dwarf::DW_TAG_common_block:
+  case dwarf::DW_TAG_lexical_block:
+  case dwarf::DW_TAG_structure_type:
+  case dwarf::DW_TAG_subprogram:
+  case dwarf::DW_TAG_subroutine_type:
+  case dwarf::DW_TAG_union_type:
+    return true;
+  }
+  llvm_unreachable("Invalid Tag");
+}
+
+/// Check if a variable describing DIE should be kept.
+/// \returns updated TraversalFlags.
+unsigned DWARFLinker::shouldKeepVariableDIE(const DWARFDie &DIE,
+                                            CompileUnit &CU,
+                                            CompileUnit::DIEInfo &MyInfo,
+                                            unsigned Flags) {
+  const auto *Abbrev = DIE.getAbbreviationDeclarationPtr();
+
+  // Global variables with constant value can always be kept.
+  if (!(Flags & TF_InFunctionScope) &&
+      Abbrev->findAttributeIndex(dwarf::DW_AT_const_value)) {
+    MyInfo.InDebugMap = true;
+    return Flags | TF_Keep;
+  }
+
+  // See if there is a relocation to a valid debug map entry inside
+  // this variable's location. The order is important here. We want to
+  // always check if the variable has a valid relocation, so that the
+  // DIEInfo is filled. However, we don't want a static variable in a
+  // function to force us to keep the enclosing function.
+  if (!CU.getContaingFile().Addresses->hasLiveMemoryLocation(DIE, MyInfo) ||
+      (Flags & TF_InFunctionScope))
+    return Flags;
+
+  if (Options.Verbose) {
+    outs() << "Keeping variable DIE:";
+    DIDumpOptions DumpOpts;
+    DumpOpts.ChildRecurseDepth = 0;
+    DumpOpts.Verbose = Options.Verbose;
+    DIE.dump(outs(), 8 /* Indent */, DumpOpts);
+  }
+
+  return Flags | TF_Keep;
+}
+
+/// Check if a function describing DIE should be kept.
+/// \returns updated TraversalFlags.
+unsigned DWARFLinker::shouldKeepSubprogramDIE(const DWARFDie &DIE,
+                                              CompileUnit &CU,
+                                              CompileUnit::DIEInfo &MyInfo,
+                                              unsigned Flags) {
+  Flags |= TF_InFunctionScope;
+
+  auto LowPc = dwarf::toAddress(DIE.find(dwarf::DW_AT_low_pc));
+  if (!LowPc)
+    return Flags;
+
+  assert(LowPc.hasValue() && "low_pc attribute is not an address.");
+  if (!CU.getContaingFile().Addresses->hasLiveAddressRange(DIE, MyInfo))
+    return Flags;
+
+  if (Options.Verbose) {
+    outs() << "Keeping subprogram DIE:";
+    DIDumpOptions DumpOpts;
+    DumpOpts.ChildRecurseDepth = 0;
+    DumpOpts.Verbose = Options.Verbose;
+    DIE.dump(outs(), 8 /* Indent */, DumpOpts);
+  }
+
+  if (DIE.getTag() == dwarf::DW_TAG_label) {
+    if (CU.hasLabelAt(*LowPc))
+      return Flags;
+
+    DWARFUnit &OrigUnit = CU.getOrigUnit();
+    // FIXME: dsymutil-classic compat. dsymutil-classic doesn't consider labels
+    // that don't fall into the CU's aranges. This is wrong IMO. Debug info
+    // generation bugs aside, this is really wrong in the case of labels, where
+    // a label marking the end of a function will have a PC == CU's high_pc.
+    if (dwarf::toAddress(OrigUnit.getUnitDIE().find(dwarf::DW_AT_high_pc))
+            .getValueOr(UINT64_MAX) <= LowPc)
+      return Flags;
+    CU.addLabelLowPc(*LowPc, MyInfo.AddrAdjust);
+    return Flags | TF_Keep;
+  }
+
+  Flags |= TF_Keep;
+
+  Optional<uint64_t> HighPc = DIE.getHighPC(*LowPc);
+  if (!HighPc) {
+    reportWarning("Function without high_pc. Range will be discarded.\n",
+                  CU.getContaingFile(), &DIE);
+    return Flags;
+  }
+
+  // Replace the debug map range with a more accurate one.
+  dwarflinker::RangesTy *Ranges;
+  CU.getContaingFile().Addresses->getValidAddressRanges(Ranges);
+  (*Ranges)[*LowPc] = ObjFileAddressRange(*HighPc, MyInfo.AddrAdjust);
+  CU.addFunctionRange(*LowPc, *HighPc, MyInfo.AddrAdjust);
+  return Flags;
+}
+
+/// Check if a DIE should be kept.
+/// \returns updated TraversalFlags.
+unsigned DWARFLinker::shouldKeepDIE(const DWARFDie &DIE, CompileUnit &CU,
+                                    CompileUnit::DIEInfo &MyInfo,
+                                    unsigned Flags) {
+  switch (DIE.getTag()) {
+  case dwarf::DW_TAG_constant:
+  case dwarf::DW_TAG_variable:
+    return shouldKeepVariableDIE(DIE, CU, MyInfo, Flags);
+  case dwarf::DW_TAG_subprogram:
+  case dwarf::DW_TAG_label:
+    return shouldKeepSubprogramDIE(DIE, CU, MyInfo, Flags);
+  case dwarf::DW_TAG_base_type:
+    // DWARF Expressions may reference basic types, but scanning them
+    // is expensive. Basic types are tiny, so just keep all of them.
+  case dwarf::DW_TAG_imported_module:
+  case dwarf::DW_TAG_imported_declaration:
+  case dwarf::DW_TAG_imported_unit:
+    // We always want to keep these.
+    return Flags | TF_Keep;
+  default:
+    break;
+  }
+
+  return Flags;
+}
+
+/// Helper that updates the completeness of the current DIE based on the
+/// completeness of one of its children. It depends on the incompleteness of
+/// the children already being computed.
+static void updateChildIncompleteness(const DWARFDie &Die, CompileUnit &CU,
+                                      CompileUnit::DIEInfo &ChildInfo) {
+  switch (Die.getTag()) {
+  case dwarf::DW_TAG_structure_type:
+  case dwarf::DW_TAG_class_type:
+    break;
+  default:
+    return;
+  }
+
+  CompileUnit::DIEInfo &MyInfo = CU.getInfo(Die);
+
+  if (ChildInfo.IncompleteDeclaration || ChildInfo.Prune)
+    MyInfo.IncompleteDeclaration = true;
+}
+
+/// Helper that updates the completeness of the current DIE based on the
+/// completeness of the DIEs it references. It depends on the incompleteness of
+/// the referenced DIE already being computed.
+static void updateRefIncompleteness(const DWARFDie &Die, CompileUnit &CU,
+                                    const CompileUnit::DIEInfo &RefInfo) {
+  switch (Die.getTag()) {
+  case dwarf::DW_TAG_typedef:
+  case dwarf::DW_TAG_member:
+  case dwarf::DW_TAG_reference_type:
+  case dwarf::DW_TAG_ptr_to_member_type:
+  case dwarf::DW_TAG_pointer_type:
+    break;
+  default:
+    return;
+  }
+
+  CompileUnit::DIEInfo &MyInfo = CU.getInfo(Die);
+
+  if (MyInfo.IncompleteDeclaration)
+    return;
+
+  if (RefInfo.IncompleteDeclaration)
+    MyInfo.IncompleteDeclaration = true;
+}
+
+/// Look at the children of the given DIE and decide whether they should be
+/// kept.
+void DWARFLinker::lookForChildDIEsToKeep(
+    const DWARFDie &Die, CompileUnit &CU, unsigned Flags,
+    SmallVectorImpl<WorklistItem> &Worklist) {
+  // The TF_ParentWalk flag tells us that we are currently walking up the
+  // parent chain of a required DIE, and we don't want to mark all the children
+  // of the parents as kept (consider for example a DW_TAG_namespace node in
+  // the parent chain). There are however a set of DIE types for which we want
+  // to ignore that directive and still walk their children.
+  if (dieNeedsChildrenToBeMeaningful(Die.getTag()))
+    Flags &= ~DWARFLinker::TF_ParentWalk;
+
+  // We're finished if this DIE has no children or we're walking the parent
+  // chain.
+  if (!Die.hasChildren() || (Flags & DWARFLinker::TF_ParentWalk))
+    return;
+
+  // Add children in reverse order to the worklist to effectively process them
+  // in order.
+  for (auto Child : reverse(Die.children())) {
+    // Add a worklist item before every child to calculate incompleteness right
+    // after the current child is processed.
+    CompileUnit::DIEInfo &ChildInfo = CU.getInfo(Child);
+    Worklist.emplace_back(Die, CU, WorklistItemType::UpdateChildIncompleteness,
+                          &ChildInfo);
+    Worklist.emplace_back(Child, CU, Flags);
+  }
+}
+
+/// Look at DIEs referenced by the given DIE and decide whether they should be
+/// kept. All DIEs referenced though attributes should be kept.
+bool DWARFLinker::lookForRefDIEsToKeep(
+    const DWARFDie &Die, CompileUnit &CU, unsigned Flags,
+    const LinkContext &Context, SmallVectorImpl<WorklistItem> &Worklist) {
+  bool Res = true;
+  bool UseOdr = (Flags & DWARFLinker::TF_DependencyWalk)
+                    ? (Flags & DWARFLinker::TF_ODR)
+                    : CU.hasODR();
+  DWARFUnit &Unit = CU.getOrigUnit();
+  DWARFDataExtractor Data = Unit.getDebugInfoExtractor();
+  const auto *Abbrev = Die.getAbbreviationDeclarationPtr();
+  uint64_t Offset = Die.getOffset() + getULEB128Size(Abbrev->getCode());
+
+  SmallVector<std::pair<DWARFDie, CompileUnit &>, 4> ReferencedDIEs;
+  for (const auto &AttrSpec : Abbrev->attributes()) {
+    DWARFFormValue Val(AttrSpec.Form);
+    if (!Val.isFormClass(DWARFFormValue::FC_Reference) ||
+        AttrSpec.Attr == dwarf::DW_AT_sibling) {
+      DWARFFormValue::skipValue(AttrSpec.Form, Data, &Offset,
+                                Unit.getFormParams());
+      continue;
+    }
+
+    Val.extractValue(Data, &Offset, Unit.getFormParams(), &Unit);
+    assert(Val.isFormClass(DWARFFormValue::FC_Reference));
+    uint64_t RefOffset = *Val.getAsReference();
+
+    if (CompileUnit *ReferencedCU = Context.getUnitForOffset(CU, RefOffset)) {
+      // Check whether referenced CU is equal to the current CU, or whether
+      // liveness analysis is done for referenced CU(i.e. we have correct
+      // IncompleteDeclaration field value).
+      if (ReferencedCU->getUniqueID() == CU.getUniqueID() ||
+          ReferencedCU->getStage() >=
+              CompileUnit::Stages::LivenessAnalysisDone) {
+        if (const DWARFDie RefDie =
+                ReferencedCU->getOrigUnit().getDIEForOffset(RefOffset)) {
+          // In a file with broken references, an attribute might point to a
+          // NULL DIE.
+          if (RefDie.isNULL())
+            reportWarning("could not find referenced DIE", CU.getContaingFile(),
+                          &Die);
+          else {
+            CompileUnit::DIEInfo &RefInfo = ReferencedCU->getInfo(RefDie);
+            if (AttrSpec.Form != dwarf::DW_FORM_ref_addr && RefInfo.Ctxt &&
+                RefInfo.Ctxt.load()->hasCanonicalDie())
+              continue;
+
+            // Keep a module forward declaration if there is no definition.
+            if (!(isODRAttribute(AttrSpec.Attr) && RefInfo.Ctxt &&
+                  RefInfo.Ctxt.load()->hasCanonicalDie()))
+              RefInfo.Prune = false;
+
+            if (ReferencedCU->getUniqueID() == CU.getUniqueID())
+              ReferencedDIEs.emplace_back(RefDie, *ReferencedCU);
+            else {
+              // update IncompleteDeclaration field based on
+              // information from the RefInfo.
+              updateRefIncompleteness(Die, CU, RefInfo);
+            }
+          }
+        } else {
+          reportWarning("could not find referenced DIE", CU.getContaingFile(),
+                        &Die);
+        }
+      } else {
+        // if referenced CU is not analyzed yet, set flag
+        // keeping input Dies(so that we do not reload them during next pass)
+        // and add referenced offset as live external reference to ReferencedCU.
+        Res = false;
+        ReferencedCU->addLiveReference(RefOffset);
+        ReferencedCU->setKeepInputDies();
+        CU.setKeepInputDies();
+      }
+    } else {
+      reportWarning("could not find referenced Compile Unit",
+                    CU.getContaingFile(), &Die);
+    }
+  }
+
+  unsigned ODRFlag = UseOdr ? DWARFLinker::TF_ODR : 0;
+
+  // Add referenced DIEs in reverse order to the worklist to effectively
+  // process them in order.
+  for (auto &P : reverse(ReferencedDIEs)) {
+    // Add a worklist item before every child to calculate incompleteness right
+    // after the current child is processed.
+    CompileUnit::DIEInfo &Info = P.second.getInfo(P.first);
+    Worklist.emplace_back(Die, CU, WorklistItemType::UpdateRefIncompleteness,
+                          &Info);
+    Worklist.emplace_back(P.first, P.second,
+                          DWARFLinker::TF_Keep |
+                              DWARFLinker::TF_DependencyWalk | ODRFlag);
+  }
+
+  return Res;
+}
+
+/// Look at the parent of the given DIE and decide whether they should be kept.
+void DWARFLinker::lookForParentDIEsToKeep(
+    unsigned AncestorIdx, CompileUnit &CU, unsigned Flags,
+    SmallVectorImpl<WorklistItem> &Worklist) {
+  // Stop if we encounter an ancestor that's already marked as kept.
+  if (CU.getInfo(AncestorIdx).Keep)
+    return;
+
+  DWARFUnit &Unit = CU.getOrigUnit();
+  DWARFDie ParentDIE = Unit.getDIEAtIndex(AncestorIdx);
+  Worklist.emplace_back(CU.getInfo(AncestorIdx).ParentIdx, CU, Flags);
+  Worklist.emplace_back(ParentDIE, CU, Flags);
+}
+
+/// Recursively walk the \p DIE tree and look for DIEs to keep. Store that
+/// information in \p CU's DIEInfo.
+///
+/// This function is the entry point of the DIE selection algorithm. It is
+/// expected to walk the DIE tree in file order and (though the mediation of
+/// its helper) call hasValidRelocation() on each DIE that might be a 'root
+/// DIE' (See DwarfLinker class comment).
+///
+/// While walking the dependencies of root DIEs, this function is also called,
+/// but during these dependency walks the file order is not respected. The
+/// TF_DependencyWalk flag tells us which kind of traversal we are currently
+/// doing.
+///
+/// The recursive algorithm is implemented iteratively as a work list because
+/// very deep recursion could exhaust the stack for large projects. The work
+/// list acts as a scheduler for different types of work that need to be
+/// performed.
+///
+/// The recursive nature of the algorithm is simulated by running the "main"
+/// algorithm (LookForDIEsToKeep) followed by either looking at more DIEs
+/// (LookForChildDIEsToKeep, LookForRefDIEsToKeep, LookForParentDIEsToKeep) or
+/// fixing up a computed property (UpdateChildIncompleteness,
+/// UpdateRefIncompleteness).
+///
+/// Returns false if DIE has inter-CU references and referenced CU is not
+/// analyzed yet.
+bool DWARFLinker::lookForDIEsToKeep(const LinkContext &Context,
+                                    const DWARFDie &Die, CompileUnit &CU,
+                                    unsigned Flags) {
+  // LIFO work list.
+  SmallVector<WorklistItem, 4> Worklist;
+  Worklist.emplace_back(Die, CU, Flags);
+
+  bool Res = true;
+
+  for (uint64_t DieOffset : CU.getLiveReferences()) {
+    if (const auto RefDie = CU.getOrigUnit().getDIEForOffset(DieOffset)) {
+      // In a file with broken references, an attribute might point to a NULL
+      // DIE.
+      if (!RefDie.isNULL()) {
+        Worklist.emplace_back(RefDie, CU,
+                              TF_Keep | TF_DependencyWalk |
+                                  (CU.hasODR() ? TF_ODR : 0));
+      } else {
+        reportWarning("DIE contains broken reference attribute",
+                      CU.getContaingFile(), &Die);
+      }
+    } else {
+      reportWarning("DIE contains broken reference attribute",
+                    CU.getContaingFile(), &Die);
+    }
+  }
+
+  while (!Worklist.empty()) {
+    WorklistItem Current = Worklist.pop_back_val();
+
+    // Look at the worklist type to decide what kind of work to perform.
+    switch (Current.Type) {
+    case WorklistItemType::UpdateChildIncompleteness:
+      updateChildIncompleteness(Current.Die, Current.CU, *Current.OtherInfo);
+      continue;
+    case WorklistItemType::UpdateRefIncompleteness:
+      updateRefIncompleteness(Current.Die, Current.CU, *Current.OtherInfo);
+      continue;
+    case WorklistItemType::LookForChildDIEsToKeep:
+      lookForChildDIEsToKeep(Current.Die, Current.CU, Current.Flags, Worklist);
+      continue;
+    case WorklistItemType::LookForRefDIEsToKeep:
+      if (!lookForRefDIEsToKeep(Current.Die, Current.CU, Current.Flags, Context,
+                                Worklist))
+        Res = false;
+      continue;
+    case WorklistItemType::LookForParentDIEsToKeep:
+      lookForParentDIEsToKeep(Current.AncestorIdx, Current.CU, Current.Flags,
+                              Worklist);
+      continue;
+    case WorklistItemType::LookForDIEsToKeep:
+      break;
+    }
+
+    unsigned Idx = Current.CU.getOrigUnit().getDIEIndex(Current.Die);
+    CompileUnit::DIEInfo &MyInfo = Current.CU.getInfo(Idx);
+
+    if (MyInfo.Prune)
+      continue;
+
+    // If the Keep flag is set, we are marking a required DIE's dependencies.
+    // If our target is already marked as kept, we're all set.
+    bool AlreadyKept = MyInfo.Keep;
+    if ((Current.Flags & TF_DependencyWalk) && AlreadyKept)
+      continue;
+
+    // We must not call shouldKeepDIE while called from keepDIEAndDependencies,
+    // because it would screw up the relocation finding logic.
+    if (!(Current.Flags & TF_DependencyWalk))
+      Current.Flags =
+          shouldKeepDIE(Current.Die, Current.CU, MyInfo, Current.Flags);
+
+    // Finish by looking for child DIEs. Because of the LIFO worklist we need
+    // to schedule that work before any subsequent items are added to the
+    // worklist.
+    Worklist.emplace_back(Current.Die, Current.CU, Current.Flags,
+                          WorklistItemType::LookForChildDIEsToKeep);
+
+    if (AlreadyKept || !(Current.Flags & TF_Keep))
+      continue;
+
+    // If it is a newly kept DIE mark it as well as all its dependencies as
+    // kept.
+    MyInfo.Keep = true;
+
+    // We're looking for incomplete types.
+    MyInfo.IncompleteDeclaration =
+        Current.Die.getTag() != dwarf::DW_TAG_subprogram &&
+        Current.Die.getTag() != dwarf::DW_TAG_member &&
+        dwarf::toUnsigned(Current.Die.find(dwarf::DW_AT_declaration), 0);
+
+    // After looking at the parent chain, look for referenced DIEs. Because of
+    // the LIFO worklist we need to schedule that work before any subsequent
+    // items are added to the worklist.
+    Worklist.emplace_back(Current.Die, Current.CU, Current.Flags,
+                          WorklistItemType::LookForRefDIEsToKeep);
+
+    bool UseOdr = (Current.Flags & TF_DependencyWalk) ? (Current.Flags & TF_ODR)
+                                                      : Current.CU.hasODR();
+    unsigned ODRFlag = UseOdr ? TF_ODR : 0;
+    unsigned ParFlags = TF_ParentWalk | TF_Keep | TF_DependencyWalk | ODRFlag;
+
+    // Now schedule the parent walk.
+    Worklist.emplace_back(MyInfo.ParentIdx, Current.CU, ParFlags);
+  }
+
+  return Res;
+}
+
+unsigned DWARFLinker::DIECloner::cloneStringAttribute(DIE &Die,
+                                                      const DWARFDie &InputDIE,
+                                                      AttributeSpec AttrSpec,
+                                                      const DWARFFormValue &Val,
+                                                      AttributesInfo &Info) {
+  Optional<const char *> String;
+  if (AttrSpec.Form == dwarf::DW_FORM_string) {
+    // In this case Val points to inplace copy of the string(inside DIECopy)
+    // done by cloneDIE. We are going to store pointer to the string,
+    // so take a string value from the InputDie.
+    Optional<DWARFFormValue> StrVal = InputDIE.find(AttrSpec.Attr);
+    if (StrVal)
+      String = StrVal->getAsCString();
+  } else
+    String = Val.getAsCString();
+
+  if (!String)
+    return 0;
+
+  // Update attributes info.
+  if (AttrSpec.Attr == dwarf::DW_AT_name)
+    Info.Name = *String;
+  else if (AttrSpec.Attr == dwarf::DW_AT_MIPS_linkage_name ||
+           AttrSpec.Attr == dwarf::DW_AT_linkage_name)
+    Info.MangledName = *String;
+
+  if (Linker.IsMT()) {
+    // In multy-threaded case it is in-effective to lock&use
+    // common resource TheOffsetsStringPool for every string
+    // attribute. Instead, we remember string attribute and
+    // put it into the string pool and update later. That way
+    // we lock TheOffsetsStringPool only once for each compile
+    // unit.
+
+    // create string attribute with dummy string offset
+    DIE::value_iterator It =
+        Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+                     dwarf::DW_FORM_strp, DIEInteger(0xbaddef));
+
+    // create delayed task for setting proper offset value for the string
+    // attribute.
+    CU.noteOffsetData(It, *String);
+  } else {
+    // add string to the pool and set proper offset.
+    uint64_t Offset = Linker.TheOffsetsStringPool.getEntry(*String).getOffset();
+    Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr), dwarf::DW_FORM_strp,
+                 DIEInteger(Offset));
+  }
+
+  return 4;
+}
+
+unsigned DWARFLinker::DIECloner::cloneDieReferenceAttribute(
+    DIE &Die, AttributeSpec AttrSpec, unsigned AttrSize,
+    const DWARFFormValue &Val, AttributesInfo &Info, uint64_t AttrOutOffset) {
+  const DWARFUnit &U = CU.getOrigUnit();
+  uint64_t RefOffset = *Val.getAsReference();
+
+  // If the referenced DIE is not found,  drop the attribute.
+  if (AttrSpec.Attr == dwarf::DW_AT_sibling)
+    return 0;
+
+  const CompileUnit *RefCU = Context.getUnitForOffset(CU, RefOffset);
+  if (!RefCU)
+    return 0;
+
+  DWARFDie RefDie = RefCU->getOrigUnit().getDIEForOffset(RefOffset);
+  if (!RefDie)
+    return 0;
+
+  const CompileUnit::DIEInfo &RefInfo = RefCU->getInfo(RefDie);
+
+  bool IsModuleRef = RefInfo.Ctxt && RefInfo.Ctxt.load()->hasCanonicalDie() &&
+                     RefInfo.Ctxt.load()->isDefinedInClangModule();
+
+  // Check for canonical Die for ODR attribute.
+  if (isODRAttribute(AttrSpec.Attr) && (CU.hasODR() || IsModuleRef)) {
+    // We cannot currently rely on a DIEEntry to emit ref_addr
+    // references, because the implementation calls back to DwarfDebug
+    // to find the unit offset. (We don't have a DwarfDebug)
+    // FIXME: we should be able to design DIEEntry reliance on
+    // DwarfDebug away.
+    uint64_t Attr;
+    Attr = 0xBADDEF;
+
+    Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+                 dwarf::DW_FORM_ref_addr, DIEInteger(Attr));
+
+    // We remember reference attribute to update it later when
+    // final offsets are known.
+    if (RefInfo.Ctxt && RefInfo.Ctxt.load()->hasCanonicalDie()) {
+      // Remember offset to the canonical Die.
+      Info.AttrReferences.emplace_back(
+          RefInfo.Ctxt.load()->getCanonicalDIEIdx(),
+          RefInfo.Ctxt.load()->getCanonicalCU(), AttrOutOffset, false);
+    } else {
+      // Remember offset to the referenced Die.
+      Info.AttrReferences.emplace_back(RefCU->getOrigUnit().getDIEIndex(RefDie),
+                                       RefCU, AttrOutOffset, false);
+    }
+
+    return U.getRefAddrByteSize();
+  }
+
+  // If referenced Die is already cloned and this is local reference then
+  // we might use already known ClonedDieOffset.
+  if (AttrSpec.Form != dwarf::DW_FORM_ref_addr && RefInfo.ClonedDieOffset) {
+    Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+                 dwarf::Form(AttrSpec.Form),
+                 DIEInteger(RefInfo.ClonedDieOffset));
+    return AttrSize;
+  }
+
+  // For other cases we need to remember reference attribute to update it later
+  // when final offsets are known.
+  Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+               dwarf::Form(AttrSpec.Form), DIEInteger(RefInfo.ClonedDieOffset));
+
+  Info.AttrReferences.emplace_back(RefCU->getOrigUnit().getDIEIndex(RefDie),
+                                   RefCU, AttrOutOffset,
+                                   AttrSpec.Form != dwarf::DW_FORM_ref_addr);
+  return AttrSize;
+}
+
+void DWARFLinker::DIECloner::cloneExpression(
+    DataExtractor &Data, DWARFExpression Expression,
+    SmallVectorImpl<uint8_t> &OutputBuffer) {
+  using Encoding = DWARFExpression::Operation::Encoding;
+
+  uint64_t OpOffset = 0;
+  for (auto &Op : Expression) {
+    auto Description = Op.getDescription();
+    // DW_OP_const_type is variable-length and has 3
+    // operands. DWARFExpression thus far only supports 2.
+    auto Op0 = Description.Op[0];
+    auto Op1 = Description.Op[1];
+    if ((Op0 == Encoding::BaseTypeRef && Op1 != Encoding::SizeNA) ||
+        (Op1 == Encoding::BaseTypeRef && Op0 != Encoding::Size1))
+      Linker.reportWarning("Unsupported DW_OP encoding.", CU.getContaingFile());
+
+    if ((Op0 == Encoding::BaseTypeRef && Op1 == Encoding::SizeNA) ||
+        (Op1 == Encoding::BaseTypeRef && Op0 == Encoding::Size1)) {
+      // This code assumes that the other non-typeref operand fits into 1 byte.
+      assert(OpOffset < Op.getEndOffset());
+      uint32_t ULEBsize = Op.getEndOffset() - OpOffset - 1;
+      assert(ULEBsize <= 16);
+
+      // Copy over the operation.
+      OutputBuffer.push_back(Op.getCode());
+      uint64_t RefOffset;
+      if (Op1 == Encoding::SizeNA) {
+        RefOffset = Op.getRawOperand(0);
+      } else {
+        OutputBuffer.push_back(Op.getRawOperand(0));
+        RefOffset = Op.getRawOperand(1);
+      }
+      uint32_t Offset = 0;
+      // Look up the base type. For DW_OP_convert, the operand may be 0 to
+      // instead indicate the generic type. The same holds for
+      // DW_OP_reinterpret, which is currently not supported.
+      if (RefOffset > 0 || Op.getCode() != dwarf::DW_OP_convert) {
+        auto RefDie = CU.getOrigUnit().getDIEForOffset(RefOffset);
+        CompileUnit::DIEInfo &Info = CU.getInfo(RefDie);
+        uint64_t ClonedDieOffset = Info.ClonedDieOffset;
+        if (ClonedDieOffset != 0)
+          Offset = ClonedDieOffset;
+        else
+          Linker.reportWarning(
+              "base type ref doesn't point to DW_TAG_base_type.",
+              CU.getContaingFile());
+      }
+      uint8_t ULEB[16];
+      unsigned RealSize = encodeULEB128(Offset, ULEB, ULEBsize);
+      if (RealSize > ULEBsize) {
+        // Emit the generic type as a fallback.
+        RealSize = encodeULEB128(0, ULEB, ULEBsize);
+        Linker.reportWarning("base type ref doesn't fit.",
+                             CU.getContaingFile());
+      }
+      assert(RealSize == ULEBsize && "padding failed");
+      ArrayRef<uint8_t> ULEBbytes(ULEB, ULEBsize);
+      OutputBuffer.append(ULEBbytes.begin(), ULEBbytes.end());
+    } else {
+      // Copy over everything else unmodified.
+      StringRef Bytes = Data.getData().slice(OpOffset, Op.getEndOffset());
+      OutputBuffer.append(Bytes.begin(), Bytes.end());
+    }
+    OpOffset = Op.getEndOffset();
+  }
+}
+
+unsigned DWARFLinker::DIECloner::cloneBlockAttribute(DIE &Die,
+                                                     AttributeSpec AttrSpec,
+                                                     const DWARFFormValue &Val,
+                                                     unsigned AttrSize) {
+  DIEValueList *Attr;
+  DIEValue Value;
+  DIELoc *Loc = nullptr;
+  DIEBlock *Block = nullptr;
+  if (AttrSpec.Form == dwarf::DW_FORM_exprloc) {
+    Loc = new (DIEAlloc) DIELoc;
+    DIELocs.push_back(Loc);
+  } else {
+    Block = new (DIEAlloc) DIEBlock;
+    DIEBlocks.push_back(Block);
+  }
+  Attr = Loc ? static_cast<DIEValueList *>(Loc)
+             : static_cast<DIEValueList *>(Block);
+
+  if (Loc)
+    Value = DIEValue(dwarf::Attribute(AttrSpec.Attr),
+                     dwarf::Form(AttrSpec.Form), Loc);
+  else
+    Value = DIEValue(dwarf::Attribute(AttrSpec.Attr),
+                     dwarf::Form(AttrSpec.Form), Block);
+
+  // If the block is a DWARF Expression, clone it into the temporary
+  // buffer using cloneExpression(), otherwise copy the data directly.
+  SmallVector<uint8_t, 32> Buffer;
+  ArrayRef<uint8_t> Bytes = *Val.getAsBlock();
+  if (DWARFAttribute::mayHaveLocationDescription(AttrSpec.Attr) &&
+      (Val.isFormClass(DWARFFormValue::FC_Block) ||
+       Val.isFormClass(DWARFFormValue::FC_Exprloc))) {
+    DWARFUnit &OrigUnit = CU.getOrigUnit();
+    DataExtractor Data(StringRef((const char *)Bytes.data(), Bytes.size()),
+                       CU.getEndianness() == support::endianness::little,
+                       OrigUnit.getAddressByteSize());
+    DWARFExpression Expr(Data, OrigUnit.getAddressByteSize(),
+                         OrigUnit.getFormParams().Format);
+    cloneExpression(Data, Expr, Buffer);
+    Bytes = Buffer;
+  }
+  for (auto Byte : Bytes)
+    Attr->addValue(DIEAlloc, static_cast<dwarf::Attribute>(0),
+                   dwarf::DW_FORM_data1, DIEInteger(Byte));
+
+  // FIXME: If DIEBlock and DIELoc just reuses the Size field of
+  // the DIE class, this "if" could be replaced by
+  // Attr->setSize(Bytes.size()).
+  if (Loc)
+    Loc->setSize(Bytes.size());
+  else
+    Block->setSize(Bytes.size());
+
+  Die.addValue(DIEAlloc, Value);
+  return AttrSize;
+}
+
+unsigned
+DWARFLinker::DIECloner::cloneAddressAttribute(DIE &Die, AttributeSpec AttrSpec,
+                                              const DWARFFormValue &Val,
+                                              AttributesInfo &Info) {
+  if (!Options.RemoveObsoleteDebugInfo) {
+    if (AttrSpec.Attr == dwarf::DW_AT_low_pc)
+      Info.HasLowPc = true;
+    Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+                 dwarf::Form(AttrSpec.Form), DIEInteger(Val.getRawUValue()));
+    return CU.getOrigUnit().getAddressByteSize();
+  }
+
+  dwarf::Form Form = AttrSpec.Form;
+  uint64_t Addr = 0;
+  if (Form == dwarf::DW_FORM_addrx) {
+    if (Optional<uint64_t> AddrOffsetSectionBase =
+            CU.getOrigUnit().getAddrOffsetSectionBase()) {
+      uint64_t StartOffset = *AddrOffsetSectionBase + Val.getRawUValue();
+      uint64_t EndOffset = StartOffset + CU.getOrigUnit().getAddressByteSize();
+      if (llvm::Expected<uint64_t> RelocAddr =
+              CU.getContaingFile().Addresses->relocateIndexedAddr(StartOffset,
+                                                                  EndOffset))
+        Addr = *RelocAddr;
+      else
+        Linker.reportWarning(toString(RelocAddr.takeError()),
+                             CU.getContaingFile());
+    } else
+      Linker.reportWarning("no base offset for address table",
+                           CU.getContaingFile());
+
+    // If this is an indexed address emit the debug_info address.
+    Form = dwarf::DW_FORM_addr;
+  } else
+    Addr = *Val.getAsAddress();
+
+  if (AttrSpec.Attr == dwarf::DW_AT_low_pc) {
+    if (Die.getTag() == dwarf::DW_TAG_inlined_subroutine ||
+        Die.getTag() == dwarf::DW_TAG_lexical_block)
+      // The low_pc of a block or inline subroutine might get
+      // relocated because it happens to match the low_pc of the
+      // enclosing subprogram. To prevent issues with that, always use
+      // the low_pc from the input DIE if relocations have been applied.
+      Addr = (Info.OrigLowPc != std::numeric_limits<uint64_t>::max()
+                  ? Info.OrigLowPc
+                  : Addr) +
+             Info.PCOffset;
+    else if (Die.getTag() == dwarf::DW_TAG_compile_unit) {
+      Addr = CU.getLowPc();
+      if (Addr == std::numeric_limits<uint64_t>::max())
+        return 0;
+    }
+    Info.HasLowPc = true;
+  } else if (AttrSpec.Attr == dwarf::DW_AT_high_pc) {
+    if (Die.getTag() == dwarf::DW_TAG_compile_unit) {
+      if (uint64_t HighPc = CU.getHighPc())
+        Addr = HighPc;
+      else
+        return 0;
+    } else
+      // If we have a high_pc recorded for the input DIE, use
+      // it. Otherwise (when no relocations where applied) just use the
+      // one we just decoded.
+      Addr = (Info.OrigHighPc ? Info.OrigHighPc : Addr) + Info.PCOffset;
+  } else if (AttrSpec.Attr == dwarf::DW_AT_call_return_pc) {
+    // Relocate a return PC address within a call site entry.
+    if (Die.getTag() == dwarf::DW_TAG_call_site)
+      Addr = (Info.OrigCallReturnPc ? Info.OrigCallReturnPc : Addr) +
+             Info.PCOffset;
+  } else if (AttrSpec.Attr == dwarf::DW_AT_call_pc) {
+    // Relocate the address of a branch instruction within a call site entry.
+    if (Die.getTag() == dwarf::DW_TAG_call_site)
+      Addr = (Info.OrigCallPc ? Info.OrigCallPc : Addr) + Info.PCOffset;
+  }
+
+  Die.addValue(DIEAlloc, static_cast<dwarf::Attribute>(AttrSpec.Attr),
+               static_cast<dwarf::Form>(Form), DIEInteger(Addr));
+  return CU.getOrigUnit().getAddressByteSize();
+}
+
+unsigned DWARFLinker::DIECloner::cloneScalarAttribute(
+    DIE &Die, const DWARFDie &InputDIE, AttributeSpec AttrSpec,
+    const DWARFFormValue &Val, unsigned AttrSize, AttributesInfo &Info,
+    uint64_t AttrOutOffset) {
+  uint64_t Value;
+
+  if (!Options.RemoveObsoleteDebugInfo) {
+    if (auto OptionalValue = Val.getAsUnsignedConstant())
+      Value = *OptionalValue;
+    else if (auto OptionalValue = Val.getAsSignedConstant())
+      Value = *OptionalValue;
+    else if (auto OptionalValue = Val.getAsSectionOffset())
+      Value = *OptionalValue;
+    else {
+      Linker.reportWarning(
+          "Unsupported scalar attribute form. Dropping attribute.",
+          CU.getContaingFile(), &InputDIE);
+      return 0;
+    }
+    if (AttrSpec.Attr == dwarf::DW_AT_declaration && Value)
+      Info.IsDeclaration = true;
+    Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+                 dwarf::Form(AttrSpec.Form), DIEInteger(Value));
+    return AttrSize;
+  }
+
+  if (AttrSpec.Attr == dwarf::DW_AT_high_pc &&
+      Die.getTag() == dwarf::DW_TAG_compile_unit) {
+    if (CU.getLowPc() == -1ULL)
+      return 0;
+    // Dwarf >= 4 high_pc is an size, not an address.
+    Value = CU.getHighPc() - CU.getLowPc();
+  } else if (AttrSpec.Form == dwarf::DW_FORM_sec_offset)
+    Value = *Val.getAsSectionOffset();
+  else if (AttrSpec.Form == dwarf::DW_FORM_sdata)
+    Value = *Val.getAsSignedConstant();
+  else if (auto OptionalValue = Val.getAsUnsignedConstant())
+    Value = *OptionalValue;
+  else {
+    Linker.reportWarning(
+        "Unsupported scalar attribute form. Dropping attribute.",
+        CU.getContaingFile(), &InputDIE);
+    return 0;
+  }
+
+  Die.addValue(DIEAlloc, dwarf::Attribute(AttrSpec.Attr),
+               dwarf::Form(AttrSpec.Form), DIEInteger(Value));
+
+  if (AttrSpec.Attr == dwarf::DW_AT_ranges)
+    Info.RangeAttrReferences.emplace_back(Value, AttrOutOffset);
+
+  // A more generic way to check for location attributes would be
+  // nice, but it's very unlikely that any other attribute needs a
+  // location list.
+  // FIXME: use DWARFAttribute::mayHaveLocationDescription().
+  else if (AttrSpec.Attr == dwarf::DW_AT_location ||
+           AttrSpec.Attr == dwarf::DW_AT_frame_base) {
+    Info.LocationAttrReferences.emplace_back(AttrOutOffset, Value,
+                                             Info.PCOffset);
+  } else if (AttrSpec.Attr == dwarf::DW_AT_declaration && Value)
+    Info.IsDeclaration = true;
+
+  return AttrSize;
+}
+
+/// Clone \p InputDIE's attribute described by \p AttrSpec with
+/// value \p Val, and add it to \p Die.
+/// \returns the size of the cloned attribute.
+unsigned DWARFLinker::DIECloner::cloneAttribute(
+    DIE &Die, const DWARFDie &InputDIE, const DWARFFormValue &Val,
+    const AttributeSpec AttrSpec, unsigned AttrSize, AttributesInfo &Info,
+    uint64_t AttrOutOffset) {
+  switch (AttrSpec.Form) {
+  case dwarf::DW_FORM_strp:
+  case dwarf::DW_FORM_string:
+  case dwarf::DW_FORM_strx:
+  case dwarf::DW_FORM_strx1:
+  case dwarf::DW_FORM_strx2:
+  case dwarf::DW_FORM_strx3:
+  case dwarf::DW_FORM_strx4:
+    return cloneStringAttribute(Die, InputDIE, AttrSpec, Val, Info);
+  case dwarf::DW_FORM_ref_addr:
+  case dwarf::DW_FORM_ref1:
+  case dwarf::DW_FORM_ref2:
+  case dwarf::DW_FORM_ref4:
+  case dwarf::DW_FORM_ref8:
+    return cloneDieReferenceAttribute(Die, AttrSpec, AttrSize, Val, Info,
+                                      AttrOutOffset);
+  case dwarf::DW_FORM_block:
+  case dwarf::DW_FORM_block1:
+  case dwarf::DW_FORM_block2:
+  case dwarf::DW_FORM_block4:
+  case dwarf::DW_FORM_exprloc:
+    return cloneBlockAttribute(Die, AttrSpec, Val, AttrSize);
+  case dwarf::DW_FORM_addr:
+  case dwarf::DW_FORM_addrx:
+    return cloneAddressAttribute(Die, AttrSpec, Val, Info);
+  case dwarf::DW_FORM_data1:
+  case dwarf::DW_FORM_data2:
+  case dwarf::DW_FORM_data4:
+  case dwarf::DW_FORM_data8:
+  case dwarf::DW_FORM_udata:
+  case dwarf::DW_FORM_sdata:
+  case dwarf::DW_FORM_sec_offset:
+  case dwarf::DW_FORM_flag:
+  case dwarf::DW_FORM_flag_present:
+    return cloneScalarAttribute(Die, InputDIE, AttrSpec, Val, AttrSize, Info,
+                                AttrOutOffset);
+  default:
+    Linker.reportWarning("Unsupported attribute form " +
+                             dwarf::FormEncodingString(AttrSpec.Form) +
+                             " in cloneAttribute. Dropping.",
+                         CU.getContaingFile(), &InputDIE);
+  }
+
+  return 0;
+}
+
+static bool isObjCSelector(StringRef Name) {
+  return Name.size() > 2 && (Name[0] == '-' || Name[0] == '+') &&
+         (Name[1] == '[');
+}
+
+void DWARFLinker::DIECloner::addObjCAccelerator(CompileUnit &CU, const DIE *Die,
+                                                StringRef Name,
+                                                bool SkipPubSection) {
+  assert(isObjCSelector(Name) && "not an objc selector");
+  // Objective C method or class function.
+  // "- [Class(Category) selector :withArg ...]"
+  StringRef ClassNameStart(StringRef(Name).drop_front(2));
+  size_t FirstSpace = ClassNameStart.find(' ');
+  if (FirstSpace == StringRef::npos)
+    return;
+
+  StringRef SelectorStart(ClassNameStart.data() + FirstSpace + 1);
+  if (!SelectorStart.size())
+    return;
+
+  StringRef Selector(SelectorStart.data(), SelectorStart.size() - 1);
+  CU.addNameAccelerator(Die, Selector, SkipPubSection);
+
+  // Add an entry for the class name that points to this
+  // method/class function.
+  StringRef ClassName(ClassNameStart.data(), FirstSpace);
+  CU.addObjCAccelerator(Die, ClassName, SkipPubSection);
+
+  if (ClassName[ClassName.size() - 1] == ')') {
+    size_t OpenParens = ClassName.find('(');
+    if (OpenParens != StringRef::npos) {
+      StringRef ClassNameNoCategory(ClassName.data(), OpenParens);
+      CU.addObjCAccelerator(Die, ClassNameNoCategory, SkipPubSection);
+
+      std::string MethodNameNoCategory(Name.data(), OpenParens + 2);
+      // FIXME: The missing space here may be a bug, but
+      //        dsymutil-classic also does it this way.
+      MethodNameNoCategory.append(std::string(SelectorStart));
+
+      CU.addNameAccelerator(Die, CU.internString(MethodNameNoCategory),
+                            SkipPubSection);
+    }
+  }
+}
+
+static bool
+shouldSkipAttribute(DWARFAbbreviationDeclaration::AttributeSpec AttrSpec,
+                    uint16_t Tag, bool InDebugMap, bool SkipPC,
+                    bool InFunctionScope) {
+  switch (AttrSpec.Attr) {
+  default:
+    return false;
+  case dwarf::DW_AT_low_pc:
+  case dwarf::DW_AT_high_pc:
+  case dwarf::DW_AT_ranges:
+    return SkipPC;
+  case dwarf::DW_AT_str_offsets_base:
+    // FIXME: Use the string offset table with Dwarf 5.
+    return true;
+  case dwarf::DW_AT_location:
+  case dwarf::DW_AT_frame_base:
+    // FIXME: for some reason dsymutil-classic keeps the location attributes
+    // when they are of block type (i.e. not location lists). This is totally
+    // wrong for globals where we will keep a wrong address. It is mostly
+    // harmless for locals, but there is no point in keeping these anyway when
+    // the function wasn't linked.
+    return (SkipPC || (!InFunctionScope && Tag == dwarf::DW_TAG_variable &&
+                       !InDebugMap)) &&
+           !DWARFFormValue(AttrSpec.Form).isFormClass(DWARFFormValue::FC_Block);
+  }
+}
+
+DIE *DWARFLinker::DIECloner::cloneDIE(const DWARFDie &InputDIE,
+                                      int64_t PCOffset, uint32_t OutOffset,
+                                      unsigned Flags, DIE *Die) {
+  DWARFUnit &U = CU.getOrigUnit();
+  unsigned Idx = U.getDIEIndex(InputDIE);
+  CompileUnit::DIEInfo &Info = CU.getInfo(Idx);
+
+  // Should the DIE appear in the output?
+  if (!Info.Keep)
+    return nullptr;
+
+  uint64_t Offset = InputDIE.getOffset();
+  assert((!Die || Idx == 0) && "DIE is already cloned.");
+
+  // The DIE might have been already created by a forward reference
+  // (see cloneDieReferenceAttribute()).
+  if (!Die)
+    Die = DIE::get(DIEAlloc, dwarf::Tag(InputDIE.getTag()));
+
+  assert(Die->getTag() == InputDIE.getTag());
+  Die->setOffset(OutOffset);
+  CU.getInfo(Idx).ClonedDieOffset = OutOffset;
+  if ((CU.hasODR() || CU.isClangModule()) && !Info.IncompleteDeclaration &&
+      Die->getTag() != dwarf::DW_TAG_namespace && Info.Ctxt &&
+      Info.Ctxt != CU.getInfo(Info.ParentIdx).Ctxt &&
+      !Info.Ctxt.load()->hasCanonicalDie()) {
+    // We are about to emit a DIE that is the root of its own valid
+    // DeclContext tree. Make the current offset the canonical offset
+    // for this context.
+    const_cast<DeclContext *>(Info.Ctxt.load())->setCanonicalInfo(Idx, &CU);
+  }
+
+  // Extract and clone every attribute.
+  DWARFDataExtractor Data = U.getDebugInfoExtractor();
+  // Point to the next DIE (generally there is always at least a NULL
+  // entry after the current one). If this is a lone
+  // DW_TAG_compile_unit without any children, point to the next unit.
+  uint64_t NextOffset = (Idx + 1 < U.getNumDIEs())
+                            ? U.getDIEAtIndex(Idx + 1).getOffset()
+                            : U.getNextUnitOffset();
+  AttributesInfo AttrInfo;
+
+  // We could copy the data only if we need to apply a relocation to it. After
+  // testing, it seems there is no performance downside to doing the copy
+  // unconditionally, and it makes the code simpler.
+  SmallString<40> DIECopy(Data.getData().substr(Offset, NextOffset - Offset));
+  Data =
+      DWARFDataExtractor(DIECopy, Data.isLittleEndian(), Data.getAddressSize());
+
+  // Modify the copy with relocated addresses.
+  if (CU.getContaingFile().Addresses->areRelocationsResolved() &&
+      CU.getContaingFile().Addresses->applyValidRelocs(DIECopy, Offset,
+                                                       Data.isLittleEndian())) {
+    // If we applied relocations, we store the value of high_pc that was
+    // potentially stored in the input DIE. If high_pc is an address
+    // (Dwarf version == 2), then it might have been relocated to a
+    // totally unrelated value (because the end address in the object
+    // file might be start address of another function which got moved
+    // independently by the linker). The computation of the actual
+    // high_pc value is done in cloneAddressAttribute().
+    AttrInfo.OrigHighPc =
+        dwarf::toAddress(InputDIE.find(dwarf::DW_AT_high_pc), 0);
+    // Also store the low_pc. It might get relocated in an
+    // inline_subprogram that happens at the beginning of its
+    // inlining function.
+    AttrInfo.OrigLowPc = dwarf::toAddress(InputDIE.find(dwarf::DW_AT_low_pc),
+                                          std::numeric_limits<uint64_t>::max());
+    AttrInfo.OrigCallReturnPc =
+        dwarf::toAddress(InputDIE.find(dwarf::DW_AT_call_return_pc), 0);
+    AttrInfo.OrigCallPc =
+        dwarf::toAddress(InputDIE.find(dwarf::DW_AT_call_pc), 0);
+  }
+
+  // Reset the Offset to 0 as we will be working on the local copy of
+  // the data.
+  Offset = 0;
+
+  const auto *Abbrev = InputDIE.getAbbreviationDeclarationPtr();
+  Offset += getULEB128Size(Abbrev->getCode());
+
+  // We are entering a subprogram. Get and propagate the PCOffset.
+  if (Die->getTag() == dwarf::DW_TAG_subprogram)
+    PCOffset = Info.AddrAdjust;
+  AttrInfo.PCOffset = PCOffset;
+
+  if (Abbrev->getTag() == dwarf::DW_TAG_subprogram) {
+    Flags |= TF_InFunctionScope;
+    if (!Info.InDebugMap && Options.RemoveObsoleteDebugInfo)
+      Flags |= TF_SkipPC;
+  }
+
+  for (const auto &AttrSpec : Abbrev->attributes()) {
+    if (Options.RemoveObsoleteDebugInfo &&
+        shouldSkipAttribute(AttrSpec, Die->getTag(), Info.InDebugMap,
+                            Flags & TF_SkipPC, Flags & TF_InFunctionScope)) {
+      DWARFFormValue::skipValue(AttrSpec.Form, Data, &Offset,
+                                U.getFormParams());
+      continue;
+    }
+
+    if (Die->getTag() == dwarf::DW_TAG_compile_unit &&
+        AttrSpec.Attr == dwarf::DW_AT_stmt_list) {
+      CU.setOffsetToStmtList(OutOffset);
+    }
+
+    DWARFFormValue Val(AttrSpec.Form);
+    uint64_t AttrSize = Offset;
+    Val.extractValue(Data, &Offset, U.getFormParams(), &U);
+    AttrSize = Offset - AttrSize;
+
+    OutOffset += cloneAttribute(*Die, InputDIE, Val, AttrSpec, AttrSize,
+                                AttrInfo, OutOffset);
+  }
+
+  // Look for accelerator entries.
+  uint16_t Tag = InputDIE.getTag();
+  // FIXME: This is slightly wrong. An inline_subroutine without a
+  // low_pc, but with AT_ranges might be interesting to get into the
+  // accelerator tables too. For now stick with dsymutil's behavior.
+  if ((Info.InDebugMap || AttrInfo.HasLowPc ||
+       AttrInfo.RangeAttrReferences.size() > 0) &&
+      Tag != dwarf::DW_TAG_compile_unit &&
+      getDIENames(InputDIE, AttrInfo,
+                  Tag != dwarf::DW_TAG_inlined_subroutine)) {
+    if (Options.GeneratePubNames || Options.GenerateAppleNames ||
+        Options.GenerateDebugNames) {
+      if (!AttrInfo.MangledName.empty() &&
+          AttrInfo.MangledName != AttrInfo.Name)
+        CU.addNameAccelerator(Die, AttrInfo.MangledName,
+                              Tag == dwarf::DW_TAG_inlined_subroutine);
+      if (!AttrInfo.Name.empty()) {
+        if (!AttrInfo.NameWithoutTemplate.empty())
+          CU.addNameAccelerator(Die, AttrInfo.NameWithoutTemplate,
+                                /* SkipPubSection */ true);
+        CU.addNameAccelerator(Die, AttrInfo.Name,
+                              Tag == dwarf::DW_TAG_inlined_subroutine);
+      }
+    }
+    if (Options.GenerateAppleObjC)
+      if (!AttrInfo.Name.empty() && isObjCSelector(AttrInfo.Name))
+        addObjCAccelerator(CU, Die, AttrInfo.Name, /* SkipPubSection =*/true);
+  } else if (Tag == dwarf::DW_TAG_namespace &&
+             (Options.GenerateAppleNamespaces || Options.GenerateDebugNames)) {
+    if (AttrInfo.Name.empty())
+      AttrInfo.Name = "(anonymous namespace)";
+    CU.addNamespaceAccelerator(Die, AttrInfo.Name);
+  } else if (Options.GeneratePubTypes || Options.GenerateAppleTypes) {
+    if (isTypeTag(Tag) && !AttrInfo.IsDeclaration &&
+        getDIENames(InputDIE, AttrInfo) && !AttrInfo.Name.empty() &&
+        AttrInfo.Name[0]) {
+      uint32_t Hash =
+          hashFullyQualifiedName(InputDIE, CU, CU.getContaingFile());
+      uint64_t RuntimeLang =
+          dwarf::toUnsigned(InputDIE.find(dwarf::DW_AT_APPLE_runtime_class))
+              .getValueOr(0);
+      bool ObjCClassIsImplementation =
+          (RuntimeLang == dwarf::DW_LANG_ObjC ||
+           RuntimeLang == dwarf::DW_LANG_ObjC_plus_plus) &&
+          dwarf::toUnsigned(
+              InputDIE.find(dwarf::DW_AT_APPLE_objc_complete_type))
+              .getValueOr(0);
+      CU.addTypeAccelerator(Die, AttrInfo.Name, ObjCClassIsImplementation,
+                            Hash);
+    }
+  }
+
+  // Determine whether there are any children that we want to keep.
+  bool HasChildren = false;
+  for (auto Child : InputDIE.children()) {
+    unsigned Idx = U.getDIEIndex(Child);
+    if (CU.getInfo(Idx).Keep) {
+      HasChildren = true;
+      break;
+    }
+  }
+
+  DIEAbbrev NewAbbrev = Die->generateAbbrev();
+  if (HasChildren)
+    NewAbbrev.setChildrenFlag(dwarf::DW_CHILDREN_yes);
+  // Assign a permanent abbrev number
+  CU.assignAbbrev(NewAbbrev);
+  Die->setAbbrevNumber(NewAbbrev.getNumber());
+
+  uint64_t SizeOfAbbreviationNumer = getULEB128Size(Die->getAbbrevNumber());
+
+  // We need to update offsets to remembered attributes after we know
+  // the size of the abbreviation number.
+  for (CompileUnit::DieRefAttrPatchInfo AttrRef : AttrInfo.AttrReferences) {
+    AttrRef.OffsetToAttr += SizeOfAbbreviationNumer;
+
+    CU.noteDieReference(AttrRef);
+  }
+
+  for (CompileUnit::RangeAttributePatchInfo AttrOffsets :
+       AttrInfo.RangeAttrReferences) {
+    AttrOffsets.OffsetToAttr += SizeOfAbbreviationNumer;
+    CU.noteRangeAttribute(*Die, AttrOffsets);
+  }
+
+  for (CompileUnit::LocationAttributePatchInfo LocAttr :
+       AttrInfo.LocationAttrReferences) {
+    LocAttr.OffsetToAttr += SizeOfAbbreviationNumer;
+    CU.noteLocationAttribute(LocAttr.OffsetToAttr, LocAttr.CULocalOffsetValue,
+                             LocAttr.PCValue);
+  }
+
+  if (Die->getTag() == dwarf::DW_TAG_compile_unit) {
+    if (Optional<uint64_t> Offset = CU.getOffsetToStmtList())
+      CU.setOffsetToStmtList(*Offset + SizeOfAbbreviationNumer);
+  }
+
+  // Add the size of the abbreviation number to the output offset.
+  OutOffset += SizeOfAbbreviationNumer;
+
+  if (!HasChildren) {
+    // Update our size.
+    Die->setSize(OutOffset - Die->getOffset());
+    return Die;
+  }
+
+  // Recursively clone children.
+  for (auto Child : InputDIE.children()) {
+    if (DIE *Clone = cloneDIE(Child, PCOffset, OutOffset, Flags)) {
+      Die->addChild(Clone);
+      OutOffset = Clone->getOffset() + Clone->getSize();
+    }
+  }
+
+  // Account for the end of children marker.
+  OutOffset += sizeof(int8_t);
+  // Update our size.
+  Die->setSize(OutOffset - Die->getOffset());
+  return Die;
+}
+
+/// Patch the input object file relevant debug_ranges entries
+/// and emit them in the output file. Update the relevant attributes
+/// to point at the new entries.
+void DWARFLinker::patchRangesForUnit(CompileUnit &CU,
+                                     DwarfEmitter &Emitter) const {
+  DWARFDebugRangeList RangeList;
+  const auto &FunctionRanges = CU.getFunctionRanges();
+  unsigned AddressSize = CU.getOrigUnit().getAddressByteSize();
+  DWARFDataExtractor RangeExtractor(
+      CU.getContaingFile().Dwarf->getDWARFObj(),
+      CU.getContaingFile().Dwarf->getDWARFObj().getRangesSection(),
+      CU.getContaingFile().Dwarf->isLittleEndian(), AddressSize);
+  auto InvalidRange = FunctionRanges.end(), CurrRange = InvalidRange;
+  DWARFUnit &OrigUnit = CU.getOrigUnit();
+  auto OrigUnitDie = OrigUnit.getUnitDIE(false);
+  uint64_t OrigLowPc =
+      dwarf::toAddress(OrigUnitDie.find(dwarf::DW_AT_low_pc), -1ULL);
+  // Ranges addresses are based on the unit's low_pc. Compute the
+  // offset we need to apply to adapt to the new unit's low_pc.
+  int64_t UnitPcOffset = 0;
+  if (OrigLowPc != -1ULL)
+    UnitPcOffset = int64_t(OrigLowPc) - CU.getLowPc();
+
+  for (auto &RangeAttributeOffsets : CU.getRangesAttributes()) {
+    uint64_t Offset = RangeAttributeOffsets.CULocalOffsetValue;
+    RangeAttributeOffsets.CULocalOffsetValue = Emitter.getRangesSectionSize();
+    if (Error E = RangeList.extract(RangeExtractor, &Offset)) {
+      llvm::consumeError(std::move(E));
+      reportWarning("invalid range list ignored.", CU.getContaingFile());
+      RangeList.clear();
+    }
+    const auto &Entries = RangeList.getEntries();
+    if (!Entries.empty()) {
+      const DWARFDebugRangeList::RangeListEntry &First = Entries.front();
+
+      if (CurrRange == InvalidRange ||
+          First.StartAddress + OrigLowPc < CurrRange.start() ||
+          First.StartAddress + OrigLowPc >= CurrRange.stop()) {
+        CurrRange = FunctionRanges.find(First.StartAddress + OrigLowPc);
+        if (CurrRange == InvalidRange ||
+            CurrRange.start() > First.StartAddress + OrigLowPc) {
+          reportWarning("no mapping for range.", CU.getContaingFile());
+          continue;
+        }
+      }
+    }
+
+    Emitter.emitRangesEntries(UnitPcOffset, OrigLowPc, CurrRange, Entries,
+                              AddressSize);
+  }
+}
+
+/// Insert the new line info sequence \p Seq into the current
+/// set of already linked line info \p Rows.
+static void insertLineSequence(std::vector<DWARFDebugLine::Row> &Seq,
+                               std::vector<DWARFDebugLine::Row> &Rows) {
+  if (Seq.empty())
+    return;
+
+  if (!Rows.empty() && Rows.back().Address < Seq.front().Address) {
+    llvm::append_range(Rows, Seq);
+    Seq.clear();
+    return;
+  }
+
+  object::SectionedAddress Front = Seq.front().Address;
+  auto InsertPoint = partition_point(
+      Rows, [=](const DWARFDebugLine::Row &O) { return O.Address < Front; });
+
+  // FIXME: this only removes the unneeded end_sequence if the
+  // sequences have been inserted in order. Using a global sort like
+  // described in patchLineTableForUnit() and delaying the end_sequene
+  // elimination to emitLineTableForUnit() we can get rid of all of them.
+  if (InsertPoint != Rows.end() && InsertPoint->Address == Front &&
+      InsertPoint->EndSequence) {
+    *InsertPoint = Seq.front();
+    Rows.insert(InsertPoint + 1, Seq.begin() + 1, Seq.end());
+  } else {
+    Rows.insert(InsertPoint, Seq.begin(), Seq.end());
+  }
+
+  Seq.clear();
+}
+
+/// Extract the line table for \p CU, and
+/// recreate a relocated version of these for the address ranges that
+/// are present in the binary.
+void DWARFLinker::patchLineTableForUnit(CompileUnit &CU,
+                                        DwarfEmitter &Emitter) {
+  DWARFDie CUDie = CU.getOrigUnit().getUnitDIE();
+  auto StmtList = dwarf::toSectionOffset(CUDie.find(dwarf::DW_AT_stmt_list));
+  if (!StmtList)
+    return;
+
+  RangesTy *Ranges;
+  CU.getContaingFile().Addresses->getValidAddressRanges(Ranges);
+
+  // Parse the original line info for the unit.
+  DWARFDebugLine::LineTable LineTable;
+  uint64_t StmtOffset = *StmtList;
+  DWARFDataExtractor LineExtractor(
+      CU.getContaingFile().Dwarf->getDWARFObj(),
+      CU.getContaingFile().Dwarf->getDWARFObj().getLineSection(),
+      CU.getContaingFile().Dwarf->isLittleEndian(),
+      CU.getOrigUnit().getAddressByteSize());
+  if (needToTranslateStrings())
+    return Emitter.translateLineTable(LineExtractor, StmtOffset);
+
+  if (Error Err = LineTable.parse(
+          LineExtractor, &StmtOffset, *CU.getContaingFile().Dwarf,
+          &CU.getOrigUnit(), CU.getContaingFile().Dwarf->getWarningHandler()))
+    CU.getContaingFile().Dwarf->getWarningHandler()(std::move(Err));
+
+  // This vector is the output line table.
+  std::vector<DWARFDebugLine::Row> NewRows;
+  NewRows.reserve(LineTable.Rows.size());
+
+  // Current sequence of rows being extracted, before being inserted
+  // in NewRows.
+  std::vector<DWARFDebugLine::Row> Seq;
+  const auto &FunctionRanges = CU.getFunctionRanges();
+  auto InvalidRange = FunctionRanges.end(), CurrRange = InvalidRange;
+
+  // FIXME: This logic is meant to generate exactly the same output as
+  // Darwin's classic dsymutil. There is a nicer way to implement this
+  // by simply putting all the relocated line info in NewRows and simply
+  // sorting NewRows before passing it to emitLineTableForUnit. This
+  // should be correct as sequences for a function should stay
+  // together in the sorted output. There are a few corner cases that
+  // look suspicious though, and that required to implement the logic
+  // this way. Revisit that once initial validation is finished.
+
+  // Iterate over the object file line info and extract the sequences
+  // that correspond to linked functions.
+  for (auto &Row : LineTable.Rows) {
+    // Check whether we stepped out of the range. The range is
+    // half-open, but consider accept the end address of the range if
+    // it is marked as end_sequence in the input (because in that
+    // case, the relocation offset is accurate and that entry won't
+    // serve as the start of another function).
+    if (CurrRange == InvalidRange || Row.Address.Address < CurrRange.start() ||
+        Row.Address.Address > CurrRange.stop() ||
+        (Row.Address.Address == CurrRange.stop() && !Row.EndSequence)) {
+      // We just stepped out of a known range. Insert a end_sequence
+      // corresponding to the end of the range.
+      uint64_t StopAddress = CurrRange != InvalidRange
+                                 ? CurrRange.stop() + CurrRange.value()
+                                 : -1ULL;
+      CurrRange = FunctionRanges.find(Row.Address.Address);
+      bool CurrRangeValid =
+          CurrRange != InvalidRange && CurrRange.start() <= Row.Address.Address;
+      if (!CurrRangeValid) {
+        CurrRange = InvalidRange;
+        if (StopAddress != -1ULL) {
+          // Try harder by looking in the Address ranges map.
+          // There are corner cases where this finds a
+          // valid entry. It's unclear if this is right or wrong, but
+          // for now do as dsymutil.
+          // FIXME: Understand exactly what cases this addresses and
+          // potentially remove it along with the Ranges map.
+          auto Range = Ranges->lower_bound(Row.Address.Address);
+          if (Range != Ranges->begin() && Range != Ranges->end())
+            --Range;
+
+          if (Range != Ranges->end() && Range->first <= Row.Address.Address &&
+              Range->second.HighPC >= Row.Address.Address) {
+            StopAddress = Row.Address.Address + Range->second.Offset;
+          }
+        }
+      }
+      if (StopAddress != -1ULL && !Seq.empty()) {
+        // Insert end sequence row with the computed end address, but
+        // the same line as the previous one.
+        auto NextLine = Seq.back();
+        NextLine.Address.Address = StopAddress;
+        NextLine.EndSequence = 1;
+        NextLine.PrologueEnd = 0;
+        NextLine.BasicBlock = 0;
+        NextLine.EpilogueBegin = 0;
+        Seq.push_back(NextLine);
+        insertLineSequence(Seq, NewRows);
+      }
+
+      if (!CurrRangeValid)
+        continue;
+    }
+
+    // Ignore empty sequences.
+    if (Row.EndSequence && Seq.empty())
+      continue;
+
+    // Relocate row address and add it to the current sequence.
+    Row.Address.Address += CurrRange.value();
+    Seq.emplace_back(Row);
+
+    if (Row.EndSequence)
+      insertLineSequence(Seq, NewRows);
+  }
+
+  // Finished extracting, now emit the line tables.
+  // FIXME: LLVM hard-codes its prologue values. We just copy the
+  // prologue over and that works because we act as both producer and
+  // consumer. It would be nicer to have a real configurable line
+  // table emitter.
+  if (LineTable.Prologue.getVersion() < 2 ||
+      LineTable.Prologue.getVersion() > 5 ||
+      LineTable.Prologue.DefaultIsStmt != DWARF2_LINE_DEFAULT_IS_STMT ||
+      LineTable.Prologue.OpcodeBase > 13)
+    reportWarning("line table parameters mismatch. Cannot emit.",
+                  CU.getContaingFile());
+  else {
+    uint32_t PrologueEnd = *StmtList + 10 + LineTable.Prologue.PrologueLength;
+    // DWARF v5 has an extra 2 bytes of information before the header_length
+    // field.
+    if (LineTable.Prologue.getVersion() == 5)
+      PrologueEnd += 2;
+    StringRef LineData =
+        CU.getContaingFile().Dwarf->getDWARFObj().getLineSection().Data;
+    MCDwarfLineTableParams Params;
+    Params.DWARF2LineOpcodeBase = LineTable.Prologue.OpcodeBase;
+    Params.DWARF2LineBase = LineTable.Prologue.LineBase;
+    Params.DWARF2LineRange = LineTable.Prologue.LineRange;
+    Emitter.emitLineTableForUnit(Params,
+                                 LineData.slice(*StmtList + 4, PrologueEnd),
+                                 LineTable.Prologue.MinInstLength, NewRows,
+                                 CU.getOrigUnit().getAddressByteSize());
+  }
+}
+
+/// Read the frame info stored in the object, and emit the
+/// patched frame descriptions for the resulting file.
+///
+/// This is actually pretty easy as the data of the CIEs and FDEs can
+/// be considered as black boxes and moved as is. The only thing to do
+/// is to patch the addresses in the headers.
+void DWARFLinker::patchFrameInfoForObject(LinkContext &Context,
+                                          RangesTy &Ranges, unsigned AddrSize,
+                                          DwarfEmitter &Emitter) {
+  StringRef FrameData =
+      Context.File.Dwarf->getDWARFObj().getFrameSection().Data;
+  if (FrameData.empty())
+    return;
+
+  DataExtractor Data(FrameData, Context.File.Dwarf->isLittleEndian(), 0);
+  uint64_t InputOffset = 0;
+
+  // Store the data of the CIEs defined in this object, keyed by their
+  // offsets.
+  DenseMap<uint64_t, StringRef> LocalCIES;
+
+  /// The CIEs that have been emitted in the output section. The actual CIE
+  /// data serves a the key to this StringMap.
+  StringMap<uint32_t> EmittedCIEs;
+
+  while (Data.isValidOffset(InputOffset)) {
+    uint64_t EntryOffset = InputOffset;
+    uint32_t InitialLength = Data.getU32(&InputOffset);
+    if (InitialLength == 0xFFFFFFFF)
+      return reportWarning("Dwarf64 bits no supported", Context.File);
+
+    uint32_t CIEId = Data.getU32(&InputOffset);
+    if (CIEId == 0xFFFFFFFF) {
+      // This is a CIE, store it.
+      StringRef CIEData = FrameData.substr(EntryOffset, InitialLength + 4);
+      LocalCIES[EntryOffset] = CIEData;
+      // The -4 is to account for the CIEId we just read.
+      InputOffset += InitialLength - 4;
+      continue;
+    }
+
+    uint32_t Loc = Data.getUnsigned(&InputOffset, AddrSize);
+
+    // Some compilers seem to emit frame info that doesn't start at
+    // the function entry point, thus we can't just lookup the address
+    // in the debug map. Use the AddressInfo's range map to see if the FDE
+    // describes something that we can relocate.
+    auto Range = Ranges.upper_bound(Loc);
+    if (Range != Ranges.begin())
+      --Range;
+    if (Range == Ranges.end() || Range->first > Loc ||
+        Range->second.HighPC <= Loc) {
+      // The +4 is to account for the size of the InitialLength field itself.
+      InputOffset = EntryOffset + InitialLength + 4;
+      continue;
+    }
+
+    // This is an FDE, and we have a mapping.
+    // Have we already emitted a corresponding CIE?
+    StringRef CIEData = LocalCIES[CIEId];
+    if (CIEData.empty())
+      return reportWarning("Inconsistent debug_frame content. Dropping.",
+                           Context.File);
+
+    uint64_t OffsetToCIERecord = Emitter.getFrameSectionSize();
+
+    // Look if we already emitted a CIE that corresponds to the
+    // referenced one (the CIE data is the key of that lookup).
+    auto IteratorInserted =
+        EmittedCIEs.insert(std::make_pair(CIEData, OffsetToCIERecord));
+    OffsetToCIERecord = IteratorInserted.first->getValue();
+
+    // If there is no CIE yet for this ID, emit it.
+    if (IteratorInserted.second)
+      Emitter.emitCIE(CIEData);
+
+    uint64_t OffsetToFDERecord = Emitter.getFrameSectionSize();
+    // Remember offset to the FDE record, so that we might update
+    // field referencing CIE record, when final offsets are known.
+    Context.noteFDERecordToPatch(
+        LinkContext::FDERecordPatch(OffsetToFDERecord));
+
+    // Emit the FDE with updated address and CIE pointer.
+    // (4 + AddrSize) is the size of the CIEId + initial_location
+    // fields that will get reconstructed by emitFDE().
+    unsigned FDERemainingBytes = InitialLength - (4 + AddrSize);
+    Emitter.emitFDE(OffsetToCIERecord, AddrSize, Loc + Range->second.Offset,
+                    FrameData.substr(InputOffset, FDERemainingBytes));
+    InputOffset += FDERemainingBytes;
+  }
+}
+
+uint32_t DWARFLinker::DIECloner::hashFullyQualifiedName(DWARFDie DIE,
+                                                        CompileUnit &U,
+                                                        const DWARFFile &File,
+                                                        int ChildRecurseDepth) {
+  const char *Name = nullptr;
+  DWARFUnit *OrigUnit = &U.getOrigUnit();
+  CompileUnit *CU = &U;
+  Optional<DWARFFormValue> Ref;
+
+  while (1) {
+    if (const char *CurrentName = DIE.getName(DINameKind::ShortName))
+      Name = CurrentName;
+
+    if (!(Ref = DIE.find(dwarf::DW_AT_specification)) &&
+        !(Ref = DIE.find(dwarf::DW_AT_abstract_origin)))
+      break;
+
+    if (!Ref->isFormClass(DWARFFormValue::FC_Reference))
+      break;
+
+    CompileUnit *RefCU;
+    if (auto RefDIE =
+            Linker.resolveDIEReference(Context, U, *Ref, DIE, RefCU)) {
+      CU = RefCU;
+      OrigUnit = &RefCU->getOrigUnit();
+      DIE = RefDIE;
+    }
+  }
+
+  unsigned Idx = OrigUnit->getDIEIndex(DIE);
+  if (!Name && DIE.getTag() == dwarf::DW_TAG_namespace)
+    Name = "(anonymous namespace)";
+
+  if (CU->getInfo(Idx).ParentIdx == 0 ||
+      // FIXME: dsymutil-classic compatibility. Ignore modules.
+      CU->getOrigUnit().getDIEAtIndex(CU->getInfo(Idx).ParentIdx).getTag() ==
+          dwarf::DW_TAG_module)
+    return djbHash(Name ? Name : "", djbHash(ChildRecurseDepth ? "" : "::"));
+
+  DWARFDie Die = OrigUnit->getDIEAtIndex(CU->getInfo(Idx).ParentIdx);
+  return djbHash(
+      (Name ? Name : ""),
+      djbHash((Name ? "::" : ""),
+              hashFullyQualifiedName(Die, *CU, File, ++ChildRecurseDepth)));
+}
+
+static uint64_t getDwoId(const DWARFDie &CUDie) {
+  if (Optional<uint64_t> DwoId = dwarf::toUnsigned(
+          CUDie.find({dwarf::DW_AT_dwo_id, dwarf::DW_AT_GNU_dwo_id})))
+    return *DwoId;
+  return 0;
+}
+
+static std::string remapPath(StringRef Path,
+                             const objectPrefixMap &ObjectPrefixMap) {
+  if (ObjectPrefixMap.empty())
+    return Path.str();
+
+  SmallString<256> p = Path;
+  for (const auto &Entry : ObjectPrefixMap)
+    if (llvm::sys::path::replace_path_prefix(p, Entry.first, Entry.second))
+      break;
+  return p.str().str();
+}
+
+bool DWARFLinker::registerModuleReference(LinkContext &Context, DWARFUnit &CU,
+                                          const DWARFFile &File,
+                                          unsigned Indent, bool Quiet) {
+  DWARFDie CUDie = CU.getUnitDIE();
+
+  std::string PCMfile = dwarf::toString(
+      CUDie.find({dwarf::DW_AT_dwo_name, dwarf::DW_AT_GNU_dwo_name}), "");
+  if (PCMfile.empty())
+    return false;
+  if (Options.ObjectPrefixMap)
+    PCMfile = remapPath(PCMfile, *Options.ObjectPrefixMap);
+
+  // Clang module DWARF skeleton CUs abuse this for the path to the module.
+  uint64_t DwoId = getDwoId(CUDie);
+
+  std::string Name = dwarf::toString(CUDie.find(dwarf::DW_AT_name), "");
+  if (Name.empty()) {
+    if (!Quiet)
+      reportWarning("Anonymous module skeleton CU for " + PCMfile, File);
+    return true;
+  }
+
+  if (!Quiet && Options.Verbose) {
+    outs().indent(Indent);
+    outs() << "Found clang module reference " << PCMfile;
+  }
+
+  {
+    std::unique_lock<std::mutex> Guard(ClangModulesMutex);
+
+    auto Cached = ClangModules.find(PCMfile);
+    if (Cached != ClangModules.end()) {
+      // FIXME: Until PR27449 (https://llvm.org/bugs/show_bug.cgi?id=27449) is
+      // fixed in clang, only warn about DWO_id mismatches in verbose mode.
+      // ASTFileSignatures will change randomly when a module is rebuilt.
+      if (!Quiet && Options.Verbose && (Cached->second != DwoId))
+        reportWarning(
+            Twine("hash mismatch: this object file was built against a "
+                  "different version of the module ") +
+                PCMfile,
+            File);
+      if (!Quiet && Options.Verbose)
+        outs() << " [cached].\n";
+      return true;
+    }
+    if (!Quiet && Options.Verbose)
+      outs() << " ...\n";
+
+    // Cyclic dependencies are disallowed by Clang, but we still
+    // shouldn't run into an infinite loop, so mark it as processed now.
+    ClangModules.insert({PCMfile, DwoId});
+  }
+
+  if (Error E = loadClangModule(Context, CU, PCMfile, Name, DwoId, File,
+                                Indent + 2, Quiet)) {
+    consumeError(std::move(E));
+    return false;
+  }
+  return true;
+}
+
+Error DWARFLinker::loadClangModule(LinkContext &Context, DWARFUnit &CU,
+                                   StringRef Filename, StringRef ModuleName,
+                                   uint64_t DwoId, const DWARFFile &File,
+                                   unsigned Indent, bool Quiet) {
+  /// Using a SmallString<0> because loadClangModule() is recursive.
+  SmallString<0> Path(Options.PrependPath);
+  if (sys::path::is_relative(Filename))
+    resolveRelativeObjectPath(Path, CU.getUnitDIE());
+  sys::path::append(Path, Filename);
+  // Don't use the cached binary holder because we have no thread-safety
+  // guarantee and the lifetime is limited.
+
+  if (Options.ObjFileLoader == nullptr)
+    return Error::success();
+
+  auto ErrOrObj = Options.ObjFileLoader(File.FileName, Path);
+  if (!ErrOrObj)
+    return Error::success();
+
+  std::unique_ptr<CompileUnit> Unit;
+  for (const auto &ChildCU : ErrOrObj->Dwarf->compile_units()) {
+    // Recursively get all modules imported by this one.
+    auto CUDie = ChildCU->getUnitDIE(false);
+    if (!CUDie)
+      continue;
+    if (!registerModuleReference(Context, *ChildCU, File, Indent, Quiet)) {
+      if (Unit) {
+        std::string Err =
+            (Filename +
+             ": Clang modules are expected to have exactly 1 compile unit.\n")
+                .str();
+        reportError(Err, File);
+        return make_error<StringError>(Err, inconvertibleErrorCode());
+      }
+      // FIXME: Until PR27449 (https://llvm.org/bugs/show_bug.cgi?id=27449) is
+      // fixed in clang, only warn about DWO_id mismatches in verbose mode.
+      // ASTFileSignatures will change randomly when a module is rebuilt.
+      uint64_t PCMDwoId = getDwoId(CUDie);
+      if (PCMDwoId != DwoId) {
+        if (!Quiet && Options.Verbose)
+          reportWarning(
+              Twine("hash mismatch: this object file was built against a "
+                    "different version of the module ") +
+                  Filename,
+              File);
+        std::unique_lock<std::mutex> Guard(ClangModulesMutex);
+
+        // Update the cache entry with the DwoId of the module loaded from disk.
+        ClangModules[Filename] = PCMDwoId;
+      }
+
+      if (!ChildCU->getUnitDIE().hasChildren())
+        return Error::success();
+
+      // Add compile unit corresponding to this module.
+      Unit =
+          std::make_unique<CompileUnit>(*ChildCU, getNextUniqueUnitID(),
+                                        !Options.NoODR, ModuleName, *ErrOrObj);
+      Unit->setStage(CompileUnit::Stages::ModulesLoaded);
+
+      Context.addModulesCompileUnit(std::move(Unit));
+    }
+  }
+
+  return Error::success();
+}
+
+void DWARFLinker::emitPaperTrailWarnings(CompileUnit &CU,
+                                         DwarfEmitter &Emitter) {
+  BumpPtrAllocator DIEAlloc;
+  std::vector<DIEBlock *> DIEBlocks;
+
+  CU.getInfo(0).Keep = true;
+  DIE *CUDie = DIE::get(DIEAlloc, dwarf::DW_TAG_compile_unit);
+  CUDie->setOffset(CU.getHeaderSize());
+  StringRef Producer;
+  StringRef WarningHeader;
+
+  std::unique_lock<std::mutex> Guard(StringPoolMutex);
+
+  switch (DwarfLinkerClientID) {
+  case DwarfLinkerClient::Dsymutil:
+    Producer = TheOffsetsStringPool.internString("dsymutil");
+    WarningHeader = "dsymutil_warning";
+    break;
+
+  default:
+    Producer = TheOffsetsStringPool.internString("dwarfopt");
+    WarningHeader = "dwarfopt_warning";
+    break;
+  }
+
+  StringRef FileName =
+      TheOffsetsStringPool.internString(CU.getContaingFile().FileName);
+  CUDie->addValue(
+      DIEAlloc, dwarf::DW_AT_producer, dwarf::DW_FORM_strp,
+      DIEInteger(TheOffsetsStringPool.getEntry(Producer).getOffset()));
+  DIEBlock *String = new (DIEAlloc) DIEBlock();
+  DIEBlocks.push_back(String);
+  for (auto &C : FileName)
+    String->addValue(DIEAlloc, dwarf::Attribute(0), dwarf::DW_FORM_data1,
+                     DIEInteger(C));
+  String->addValue(DIEAlloc, dwarf::Attribute(0), dwarf::DW_FORM_data1,
+                   DIEInteger(0));
+
+  CUDie->addValue(DIEAlloc, dwarf::DW_AT_name, dwarf::DW_FORM_string, String);
+  for (const auto &Warning : CU.getContaingFile().Warnings) {
+    DIE &ConstDie = CUDie->addChild(DIE::get(DIEAlloc, dwarf::DW_TAG_constant));
+    ConstDie.addValue(
+        DIEAlloc, dwarf::DW_AT_name, dwarf::DW_FORM_strp,
+        DIEInteger(TheOffsetsStringPool.getEntry(WarningHeader).getOffset()));
+    ConstDie.addValue(DIEAlloc, dwarf::DW_AT_artificial, dwarf::DW_FORM_flag,
+                      DIEInteger(1));
+    ConstDie.addValue(
+        DIEAlloc, dwarf::DW_AT_const_value, dwarf::DW_FORM_strp,
+        DIEInteger(TheOffsetsStringPool.getEntry(Warning).getOffset()));
+  }
+  unsigned Size = 4 /* FORM_strp */ + FileName.size() + 1 +
+                  CU.getContaingFile().Warnings.size() * (4 + 1 + 4) +
+                  1 /* End of children */;
+  DIEAbbrev Abbrev = CUDie->generateAbbrev();
+  CU.assignAbbrev(Abbrev);
+  CUDie->setAbbrevNumber(Abbrev.getNumber());
+  Size += getULEB128Size(Abbrev.getNumber());
+  // Abbreviation ordering needed for classic compatibility.
+  for (auto &Child : CUDie->children()) {
+    Abbrev = Child.generateAbbrev();
+    CU.assignAbbrev(Abbrev);
+    Child.setAbbrevNumber(Abbrev.getNumber());
+    Size += getULEB128Size(Abbrev.getNumber());
+  }
+  CUDie->setSize(Size);
+  Emitter.emitPaperTrailWarningsDie(*CUDie);
+
+  uint64_t UnitSize = 0;
+  UnitSize += CU.getHeaderSize();
+  UnitSize += CUDie->getSize();
+  CU.setUnitSize(UnitSize);
+
+  Emitter.emitAbbrevs(CU.getAbbreviations(), CU.getVersion());
+
+  for (auto I = DIEBlocks.begin(), E = DIEBlocks.end(); I != E; ++I)
+    (*I)->~DIEBlock();
+}
+
+DWARFLinker::DIECloner::DIECloner(CompileUnit &CU, const LinkContext &Context,
+                                  DWARFLinker &Linker, raw_pwrite_stream &Out,
+                                  const DWARFLinkerOptions &Options)
+    : Context(Context), Linker(Linker), CU(CU), Options(Options) {
+  auto ReportWarn = [&](const Twine &message, StringRef, const DWARFDie *) {
+    Linker.reportWarning(message, CU.getContaingFile());
+  };
+
+  if (!Linker.TheDwarfEmitter)
+    return;
+
+  Emitter = std::make_unique<DwarfStreamer>(
+      OutputFileType::Object, Out, [](StringRef input) { return input; },
+      ReportWarn, ReportWarn);
+
+  if (!Emitter->init(Linker.TheDwarfEmitter->getTargetTriple()))
+    return;
+}
+
+DWARFLinker::DIECloner::~DIECloner() {
+  // release all memory used for cloning compile unit.
+  for (auto I = DIEBlocks.begin(), E = DIEBlocks.end(); I != E; ++I)
+    (*I)->~DIEBlock();
+  for (auto I = DIELocs.begin(), E = DIELocs.end(); I != E; ++I)
+    (*I)->~DIELoc();
+
+  DIEBlocks.clear();
+  DIELocs.clear();
+  DIEAlloc.Reset();
+
+  if (Emitter)
+    Emitter->finish();
+}
+
+void DWARFLinker::DIECloner::cloneCU() {
+  auto InputDIE = CU.getOrigUnit().getUnitDIE();
+  if (!InputDIE)
+    return;
+
+  if (CU.getInfo(0).Keep) {
+    // Clone the InputDIE into your Unit DIE in our compile unit since it
+    // already has a DIE inside of it.
+    CU.createOutputDIE();
+    cloneDIE(InputDIE, 0 /* PC offset */,
+             CU.getHeaderSize() /* Unit Header size */, 0,
+             CU.getOutputUnitDIE());
+
+    Linker.assignDelayedOffsets(CU);
+  }
+
+  if (Options.NoOutput || !CU.getInfo(0).Keep)
+    return;
+
+  assert(Emitter);
+
+  if (Options.RemoveObsoleteDebugInfo || Linker.needToTranslateStrings())
+    Linker.patchLineTableForUnit(CU, *Emitter);
+
+  if (Options.RemoveObsoleteDebugInfo) {
+    // Emit all the compile unit's debug information.
+    Emitter->emitUnitRangesEntries(
+        CU, static_cast<bool>(CU.getUnitRangesAttribute()));
+
+    Linker.patchRangesForUnit(CU, *Emitter);
+    auto ProcessExpr = [&](StringRef Bytes, SmallVectorImpl<uint8_t> &Buffer) {
+      DWARFUnit &OrigUnit = CU.getOrigUnit();
+
+      DataExtractor Data(Bytes,
+                         CU.getEndianness() == support::endianness::little,
+                         OrigUnit.getAddressByteSize());
+      cloneExpression(Data,
+                      DWARFExpression(Data, OrigUnit.getAddressByteSize(),
+                                      OrigUnit.getFormParams().Format),
+                      Buffer);
+    };
+    Emitter->emitLocationsForUnit(CU, *CU.getContaingFile().Dwarf, ProcessExpr);
+  }
+
+  Emitter->emitAbbrevs(CU.getAbbreviations(), CU.getVersion());
+
+  assert(Emitter->getDebugInfoSectionSize() ==
+         CU.getStartOffset(CompileUnit::DebugTableKind::DebugInfo));
+
+  Emitter->emitCompileUnitHeader(CU);
+  Emitter->emitDIE(*CU.getOutputUnitDIE());
+
+#ifndef NDEBUG
+  uint64_t UnitSize = 0;
+  if (DIE *OutUnitDie = CU.getOutputUnitDIE()) {
+    UnitSize += CU.getHeaderSize() /* Header size */;
+    UnitSize += OutUnitDie->getSize();
+  }
+  assert(Emitter->getDebugInfoSectionSize() == UnitSize);
+#endif
+
+  CU.setUnitSize(Emitter->getDebugInfoSectionSize());
+}
+
+void DWARFLinker::copyInvariantDebugSection(DWARFContext &Dwarf,
+                                            DwarfEmitter &Emitter) {
+  if (!needToTranslateStrings())
+    Emitter.emitSectionContents(Dwarf.getDWARFObj().getLineSection().Data,
+                                "debug_line");
+  Emitter.emitSectionContents(Dwarf.getDWARFObj().getLocSection().Data,
+                              "debug_loc");
+  Emitter.emitSectionContents(Dwarf.getDWARFObj().getRangesSection().Data,
+                              "debug_ranges");
+  Emitter.emitSectionContents(Dwarf.getDWARFObj().getFrameSection().Data,
+                              "debug_frame");
+  Emitter.emitSectionContents(Dwarf.getDWARFObj().getArangesSection(),
+                              "debug_aranges");
+}
+
+void DWARFLinker::addObjectFile(DWARFFile &File) {
+  ObjectContexts.emplace_back(std::make_unique<LinkContext>(File));
+}
+
+// ------------------------------------------------------------------
+//
+// The overall linking process looks like this:
+//
+// parrallel_for_each(ObjectFile) {
+//   for_each (Compile Unit) {
+//     1. load Clang modules.
+//   }
+//
+//   parrallel_for_each(Compile Unit) {
+//     1. load input DWARF for Compile Unit.
+//     2. report warnings for Clang modules.
+//     3. search for type declarations.
+//     4. analyze live DIEs.
+//     5. clone DIEs(Generate output DIEs and resulting DWARF tables).
+//        The result is in OutDebugInfoBytes, which is ELF file containg
+//        DWARF tables corresponding for current compile unit.
+//     6. cleanup Input and Output DIEs.
+//   }
+//
+//   deallocate loaded Object file.
+// }
+//
+// for_each (ObjectFile) {
+//   for_each (Compile Unit) {
+//     1. set offsets to Compile Units DWARF tables.
+//     2. patch size/offsets fields.
+//     3. generate index tables.
+//     4. move DWARF tables of compile units into the resulting file.
+//    }
+// }
+//
+// i.e. every compile unit is processed separately, visited only once
+// (except case inter-CU references exist), and used data are freed
+// after the compile unit is processed. The resulting file is glued
+// from generated debug tables, corresponding to separate compile units.
+//
+// Handling inter-CU references: inter-CU references are hard to process
+// using only one pass. f.e. if CU1 references CU100 and CU100 references
+// CU1, we could not finish handling of CU1 until we finished CU100.
+// Thus we either need to load all CUs into the memory, either load CUs
+// several times. So we discover dependencies during the first pass(so that
+// depending compile units stay loaded into the memory). And then, during the
+// second pass it handles inter-connected compile units.
+
+bool DWARFLinker::link() {
+  assert(Options.NoOutput || TheDwarfEmitter);
+
+  // reset compile unit unique ID counter.
+  UniqueUnitID = 0;
+
+  if (Options.Verbose && Options.Threads != 1) {
+    Options.Threads = 1;
+    reportWarning(
+        "set number of threads to 1 to make --verbose to work properly", "");
+  }
+
+  for (std::unique_ptr<LinkContext> &Context : ObjectContexts) {
+    if (Options.Verbose) {
+      if (DwarfLinkerClientID == DwarfLinkerClient::Dsymutil)
+        outs() << "DEBUG MAP OBJECT: " << Context->File.FileName << "\n";
+      else
+        outs() << "OBJECT FILE: " << Context->File.FileName << "\n";
+    }
+
+    if (Context->File.Dwarf) {
+      for (const std::unique_ptr<DWARFUnit> &OrigCU :
+           Context->File.Dwarf->compile_units()) {
+        if (Options.Verbose) {
+          outs() << "Input compilation unit:";
+          DIDumpOptions DumpOpts;
+          DumpOpts.ChildRecurseDepth = 0;
+          DumpOpts.Verbose = Options.Verbose;
+          OrigCU->getUnitDIE().dump(outs(), 0, DumpOpts);
+        }
+      }
+    }
+
+    // count references to source files.
+    addBinHolderNameRef(Context->File);
+  }
+
+  parallel::strategy = hardware_concurrency(Options.Threads);
+
+  if (Options.Threads == 1) {
+    for (std::unique_ptr<LinkContext> &Context : ObjectContexts)
+      linkObjectFile(*Context);
+  } else {
+    ThreadPool Pool(parallel::strategy);
+    for (std::unique_ptr<LinkContext> &Context : ObjectContexts) {
+      Pool.async([&]() { linkObjectFile(*Context); });
+    }
+    Pool.wait();
+  }
+
+  ODRContexts.clear();
+
+  // Assemble final file after debug info for each compile unit is generated.
+  glueCompileUnitsAndWriteToTheOutput();
+
+  return true;
+}
+
+bool DWARFLinker::IsClonningComplete(LinkContext &Context) const {
+  for (const auto &CU : Context.CompileUnits) {
+    if (CU->getStage() == CompileUnit::Stages::Cloned) {
+      if (CU->hasExternalReferences())
+        return false;
+    } else
+      return false;
+  }
+
+  return true;
+}
+
+void DWARFLinker::setDeclContext(const DeclContextKey *Key,
+                                 CompileUnit::DIEInfo *Info) {
+  if (!Key->IsInvalid) {
+    // Create type declaration context corresponding to the Key
+    // and set it to Info->Ctxt.
+    Info->Ctxt = ODRContexts.getChildDeclContext(*Key);
+    const_cast<DeclContext *>(Info->Ctxt.load())
+        ->setDefinedInClangModule(Key->IsDefinedInClangModule);
+    assert(Info->Ctxt);
+  }
+}
+
+void DWARFLinker::assignDelayedDeclContexts(CompileUnit &CU) {
+  std::unique_lock<std::mutex> Guard(ContextsMutex);
+
+  for (const CompileUnit::DeclContextKeyData &Data : CU.getDeclContextData())
+    setDeclContext(Data.Key, Data.Info);
+}
+
+void DWARFLinker::setStringsOffset(DIE::value_iterator AttrToPatch,
+                                   const char *String) {
+  const auto &Old = *AttrToPatch;
+  assert(Old.getType() == DIEValue::isInteger);
+
+  // Put specified String into the global string pool and set it`s
+  // offset for the specified Attribute.
+  uint64_t Offset = TheOffsetsStringPool.getEntry(String).getOffset();
+  *AttrToPatch =
+      DIEValue(Old.getAttribute(), Old.getForm(), DIEInteger(Offset));
+}
+
+void DWARFLinker::assignDelayedOffsets(CompileUnit &CU) {
+  std::unique_lock<std::mutex> Guard(StringPoolMutex);
+
+  for (const CompileUnit::OffsetsData &Data : CU.getStringsOffsetData())
+    setStringsOffset(Data.AttrToPatch, Data.String);
+}
+
+void DWARFLinker::linkObjectFile(LinkContext &Context) {
+  if (Context.File.Warnings.empty()) {
+    if (!Context.File.Dwarf)
+      return;
+
+    // Look for modules and load them.
+    for (const std::unique_ptr<DWARFUnit> &OrigCU :
+         Context.File.Dwarf->compile_units()) {
+      auto CUDie = OrigCU->getUnitDIE();
+
+      if (CUDie && Options.RemoveObsoleteDebugInfo)
+        registerModuleReference(Context, *OrigCU, Context.File);
+    }
+
+    // Process modules compile units in parallel.
+    parallelForEach(Context.ModulesCompileUnits,
+                    [&](std::unique_ptr<CompileUnit> &CU) {
+                      processSingleCompileUnit(Context, *CU);
+                    });
+
+    // Check for live relocations. If there is no any live relocation then we
+    // can skip entire object file.
+    if (Options.RemoveObsoleteDebugInfo &&
+        !Context.File.Addresses->hasValidRelocs()) {
+      if (Options.Verbose)
+        outs() << "No valid relocations found. Skipping.\n";
+      return;
+    }
+
+    Context.OriginalDebugInfoSize = getDebugInfoSize(*Context.File.Dwarf);
+
+    // Create CompileUnit structures to keep information about source
+    // DWARFUnit`s
+    for (const auto &OrigCU : Context.File.Dwarf->compile_units()) {
+      Context.CompileUnits.emplace_back(std::make_unique<CompileUnit>(
+          *OrigCU, getNextUniqueUnitID(),
+          !Options.NoODR && Options.RemoveObsoleteDebugInfo, "", Context.File));
+      Context.CompileUnits.back()->getOrigUnit().getUnitDIE();
+    };
+
+    // Process compile units in parallel. We might do additional passes,
+    // if there are unprocessed inter-CU references.
+    while (!IsClonningComplete(Context)) {
+      parallelForEach(Context.CompileUnits,
+                      [&](std::unique_ptr<CompileUnit> &CU) {
+                        processSingleCompileUnit(Context, *CU);
+                      });
+    }
+  }
+
+  if (!Options.NoOutput) {
+    assert(TheDwarfEmitter);
+    auto ReportWarn = [&](const Twine &message, StringRef, const DWARFDie *) {
+      reportWarning(message, Context.File);
+    };
+
+    if (!Context.File.Warnings.empty()) {
+      // Write generated debug info(for warnings) into the DWARF file of
+      // corresponding artificial compile unit.
+      Context.CompileUnits.push_back(std::make_unique<CompileUnit>(
+          getNextUniqueUnitID(),
+          !Options.NoODR && Options.RemoveObsoleteDebugInfo, "", Context.File));
+
+      raw_svector_ostream stream(Context.CompileUnits.back()->getDwarfBits());
+
+      DwarfStreamer Emitter(
+          OutputFileType::Object, stream, [](StringRef input) { return input; },
+          ReportWarn, ReportWarn);
+
+      if (!Emitter.init(TheDwarfEmitter->getTargetTriple()))
+        return;
+
+      emitPaperTrailWarnings(*Context.CompileUnits.back(), Emitter);
+
+      Emitter.finish();
+    } else if (!Options.RemoveObsoleteDebugInfo) {
+      // Copy invariant debug tables into the DWARF file of
+      // link context.
+      raw_svector_ostream stream(Context.OutDebugInfoBytes);
+
+      DwarfStreamer Emitter(
+          OutputFileType::Object, stream, [](StringRef input) { return input; },
+          ReportWarn, ReportWarn);
+
+      if (!Emitter.init(TheDwarfEmitter->getTargetTriple()))
+        return;
+
+      copyInvariantDebugSection(*Context.File.Dwarf, Emitter);
+
+      Emitter.finish();
+    } else if (!Context.CompileUnits.empty()) {
+      // Generate debug info for .debug_frame and write it into the DWARF
+      // file of link context.
+      raw_svector_ostream stream(Context.OutDebugInfoBytes);
+
+      DwarfStreamer Emitter(
+          OutputFileType::Object, stream, [](StringRef input) { return input; },
+          ReportWarn, ReportWarn);
+
+      if (!Emitter.init(TheDwarfEmitter->getTargetTriple()))
+        return;
+      RangesTy *Ranges;
+
+      Context.File.Addresses->getValidAddressRanges(Ranges);
+
+      patchFrameInfoForObject(
+          Context, *Ranges,
+          Context.CompileUnits[0]->getOrigUnit().getAddressByteSize(), Emitter);
+
+      Emitter.finish();
+    }
+  }
+
+  // we could not simply unload object file here.
+  // Object file might be a member of an archive.
+  // Unloading object file in such case would also
+  // unload containing archive. Thus we count references
+  // to the archive and unload it when there are no
+  // more unprocessed archive members.
+  unloadObjectFile(Context.File);
+}
+
+static bool isArchive(StringRef Name) { return Name.endswith(")"); }
+
+static StringRef getBinaryHolderName(StringRef Name) {
+  if (isArchive(Name))
+    return Name.substr(0, Name.rfind('('));
+  else
+    return Name;
+}
+
+void DWARFLinker::addBinHolderNameRef(DWARFFile &File) {
+  BinaryHolderRefCounter[getBinaryHolderName(File.FileName)]++;
+}
+
+void DWARFLinker::unloadObjectFile(DWARFFile &File) {
+  std::unique_lock<std::mutex> Guard(BinaryHolderRefMutex);
+
+  assert(BinaryHolderRefCounter[getBinaryHolderName(File.FileName)] != 0);
+
+  BinaryHolderRefCounter[getBinaryHolderName(File.FileName)]--;
+
+  if (BinaryHolderRefCounter[getBinaryHolderName(File.FileName)] == 0)
+    File.unload();
+}
+
+void DWARFLinker::processSingleCompileUnit(LinkContext &Context,
+                                           CompileUnit &CU) {
+  CU.initIteration();
+
+  // load input Dies for CU.
+  if (CU.getStage() != CompileUnit::Stages::Cloned) {
+    CU.getOrigUnit().getUnitDIE(false);
+
+    // pre-load compile units line table.
+    if (CU.getContaingFile().Dwarf) {
+      std::unique_lock<std::mutex> Guard(Context.LineTablesMutex);
+      CU.getContaingFile().Dwarf->getLineTableForUnit(&CU.getOrigUnit());
+    }
+  }
+
+  while (CU.getStage() != CompileUnit::Stages::Cloned) {
+    switch (CU.getStage()) {
+    case CompileUnit::Stages::Created: {
+      DWARFDie CUDie = CU.getOrigUnit().getUnitDIE();
+      if (!CUDie)
+        CU.setStage(CompileUnit::Stages::LivenessAnalysisDone);
+      else
+        CU.setStage(CompileUnit::Stages::SrcDWARFLoaded);
+    } break;
+
+    case CompileUnit::Stages::SrcDWARFLoaded: {
+      if (Options.RemoveObsoleteDebugInfo) {
+        // The registerModuleReference() condition effectively skips
+        // over fully resolved skeleton units. This second pass of
+        // registerModuleReferences doesn't do any new work, but it
+        // will collect top-level errors, which are suppressed. Module
+        // warnings were already displayed in the first iteration.
+        if (registerModuleReference(Context, CU.getOrigUnit(),
+                                    CU.getContaingFile(), 0, true)) {
+          CU.setStage(CompileUnit::Stages::Cloned);
+          return;
+        }
+        CU.setStage(CompileUnit::Stages::ModulesLoaded);
+      } else
+        CU.setStage(CompileUnit::Stages::TypesDeclarationsFound);
+    } break;
+
+    case CompileUnit::Stages::ModulesLoaded: {
+      bool HasUnresolvedRefs = false;
+      analyzeContextInfo(Context, CU.getOrigUnit().getUnitDIE(), 0, CU,
+                         Options.ParseableSwiftInterfaces, HasUnresolvedRefs);
+      if (HasUnresolvedRefs) {
+        // if analyzeContextInfo set true for HasUnresolvedRefs then this
+        // compile unit has inter-CU reference to the compile unit which is not
+        // analyzed yet. Stop processing of current unit until referenced
+        // unit would be analyzed.
+
+        return;
+      }
+      assignDelayedDeclContexts(CU);
+      CU.setStage(CompileUnit::Stages::TypesDeclarationsFound);
+    } break;
+
+    case CompileUnit::Stages::TypesDeclarationsFound: {
+      // Then mark all the DIEs that need to be present in the generated output
+      // and collect some information about them.
+      // Note that this loop can not be merged with the previous one because
+      // cross-cu references require the ParentIdx to be setup for every CU in
+      // the object file before calling this.
+      if (!Options.RemoveObsoleteDebugInfo || CU.isClangModule()) {
+        CU.markEverythingAsKept();
+      } else {
+        RangesTy *Ranges;
+        CU.getContaingFile().Addresses->getValidAddressRanges(Ranges);
+        if (!lookForDIEsToKeep(Context, CU.getOrigUnit().getUnitDIE(false), CU,
+                               0)) {
+          // if lookForDIEsToKeep returned false then this compile unit
+          // has inter-CU reference to the compile unit which is not
+          // analyzed yet. Stop processing of current unit until referenced
+          // unit would be analyzed.
+
+          return;
+        }
+      }
+
+      CU.setStage(CompileUnit::Stages::LivenessAnalysisDone);
+    } break;
+
+    case CompileUnit::Stages::LivenessAnalysisDone: {
+      if (CU.isClangModule() || !Options.RemoveObsoleteDebugInfo ||
+          CU.getContaingFile().Addresses->hasValidRelocs()) {
+        raw_svector_ostream stream(CU.getDwarfBits());
+
+        DIECloner(CU, Context, *this, stream, Options).cloneCU();
+      }
+
+      CU.setStage(CompileUnit::Stages::Cloned);
+    } break;
+
+    case CompileUnit::Stages::Cloned:
+      assert(false);
+    }
+  }
+
+  CU.cleanupDataAfterClonning();
+}
+
+void DWARFLinker::patchCompileUnit(CompileUnit &CU, StringRef Data) {
+  assert(Data.size() >= sizeof(int32_t));
+
+  // Update the compile unit size.
+  {
+    char *PatchLocation = const_cast<char *>(Data.data());
+    uint64_t PatchValue = CU.getUnitSize() - 4;
+    support::endian::write32(PatchLocation, PatchValue, CU.getEndianness());
+  }
+
+  // Update offset into the abbreviation table.
+  {
+    char *PatchLocation =
+        const_cast<char *>(Data.data() + (CU.getVersion() >= 5 ? 8 : 6));
+    uint64_t PatchValue =
+        CU.getStartOffset(CompileUnit::DebugTableKind::DebugAbbrev);
+    support::endian::write32(PatchLocation, PatchValue, CU.getEndianness());
+  }
+
+  // Update Dies references.
+  for (const CompileUnit::DieRefAttrPatchInfo &Ref : CU.getDiesReferences()) {
+    if (!Ref.RefCU)
+      continue;
+
+    const CompileUnit::DIEInfo &RefDieInfo = Ref.RefCU->getInfo(Ref.RefDieIdx);
+
+    uint64_t Offset = RefDieInfo.ClonedDieOffset;
+    if (!Ref.IsCULocal)
+      Offset +=
+          Ref.RefCU->getStartOffset(CompileUnit::DebugTableKind::DebugInfo);
+
+    char *PatchLocation = const_cast<char *>(Data.data() + Ref.OffsetToAttr);
+    uint64_t PatchValue = Offset;
+    support::endian::write32(PatchLocation, PatchValue, CU.getEndianness());
+  };
+
+  if (CU.isArtificialCU() || !Options.RemoveObsoleteDebugInfo)
+    return;
+
+  // Update the cloned DW_AT_stmt_list with the correct debug_line offset.
+  if (Optional<uint64_t> Offset = CU.getOffsetToStmtList()) {
+    assert(Data.size() >= (*Offset + sizeof(int32_t)));
+    char *PatchLocation = const_cast<char *>(Data.data() + *Offset);
+    uint64_t PatchValue =
+        CU.getStartOffset(CompileUnit::DebugTableKind::DebugLine);
+    support::endian::write32(PatchLocation, PatchValue, CU.getEndianness());
+  }
+
+  // Update compile unit DW_AT_range attribute.
+  Optional<CompileUnit::RangeAttributePatchInfo> Offsets =
+      CU.getUnitRangesAttribute();
+  if (Offsets) {
+    assert(Data.size() >= (Offsets->OffsetToAttr + sizeof(int32_t)));
+    char *PatchLocation =
+        const_cast<char *>(Data.data() + Offsets->OffsetToAttr);
+    uint64_t PatchValue =
+        CU.getStartOffset(CompileUnit::DebugTableKind::DebugRange);
+    support::endian::write32(PatchLocation, PatchValue, CU.getEndianness());
+  }
+
+  // Update Dies DW_AT_range attributes.
+  for (const auto &RangeOffset : CU.getRangesAttributes()) {
+    char *PatchLocation =
+        const_cast<char *>(Data.data() + RangeOffset.OffsetToAttr);
+    uint64_t PatchValue =
+        RangeOffset.CULocalOffsetValue +
+        CU.getStartOffset(CompileUnit::DebugTableKind::DebugRange);
+    support::endian::write32(PatchLocation, PatchValue, CU.getEndianness());
+  }
+
+  // Update Dies DW_AT_location attributes.
+  for (const CompileUnit::LocationAttributePatchInfo &LocPatch :
+       CU.getLocationAttributes()) {
+    char *PatchLocation =
+        const_cast<char *>(Data.data() + LocPatch.OffsetToAttr);
+    uint64_t PatchValue =
+        LocPatch.CULocalOffsetValue +
+        CU.getStartOffset(CompileUnit::DebugTableKind::DebugLoc);
+    support::endian::write32(PatchLocation, PatchValue, CU.getEndianness());
+  }
+}
+
+void DWARFLinker::patchAranges(CompileUnit &CU, StringRef Data) {
+  if (CU.isArtificialCU())
+    return;
+
+  const int OffsetToCompileUnitField = sizeof(uint32_t) + sizeof(uint16_t);
+  assert(Data.size() >= (OffsetToCompileUnitField + sizeof(int32_t)));
+
+  char *PatchLocation =
+      const_cast<char *>(Data.data() + OffsetToCompileUnitField);
+  uint64_t PatchValue =
+      CU.getStartOffset(CompileUnit::DebugTableKind::DebugInfo);
+  support::endian::write32(PatchLocation, PatchValue, CU.getEndianness());
+}
+
+void DWARFLinker::patchFrames(LinkContext &Context, StringRef Data) {
+  for (LinkContext::FDERecordPatch Patch : Context.getFDERecordsPatches()) {
+    // 4 is the offset from the start of FDE record to the field referencing CIE
+    // record.
+    uint64_t OffsetToTheFieldReferencingCIE = Patch.OffsetToFDERecord + 4;
+    char *PatchLocation =
+        const_cast<char *>(Data.data() + OffsetToTheFieldReferencingCIE);
+
+    uint64_t CieOffsetValue =
+        support::endian::read32(PatchLocation, Context.getEndianness());
+
+    uint64_t PatchValue =
+        CieOffsetValue +
+        Context.getStartOffset(CompileUnit::DebugTableKind::DebugFrame);
+    support::endian::write32(PatchLocation, PatchValue,
+                             Context.getEndianness());
+  }
+}
+
+/// Enumerates all debug_info sections from \p Obj and calls
+/// \p Handler for each of them.
+void enumerateDebugSections(
+    object::ObjectFile &Obj,
+    std::function<void(StringRef Name, StringRef Data)> Handler) {
+  for (const object::SectionRef &Sect : Obj.sections()) {
+    Expected<StringRef> SectNameOrErr = Sect.getName();
+    if (!SectNameOrErr)
+      consumeError(SectNameOrErr.takeError());
+
+    if (!Sect.isDebugSection(*SectNameOrErr))
+      continue;
+
+    Expected<StringRef> Data = Sect.getContents();
+    if (!Data)
+      consumeError(SectNameOrErr.takeError());
+    StringRef Name =
+        SectNameOrErr->substr(SectNameOrErr->find_first_not_of("._z"));
+    Handler(Name, *Data);
+  }
+}
+
+void DWARFLinker::glueCompileUnitsAndWriteToTheOutput() {
+  if (Options.NoOutput)
+    return;
+
+  // For each object file map how many bytes were emitted.
+  StringMap<DebugInfoSize> SizeByObject;
+
+  // Go through all object files, all compile units and assign
+  // offsets to them.
+  assignOffsets(SizeByObject);
+
+  // Patch size/offsets fields according to the assigned offsets.
+  patchOffsetsAndSizes();
+
+  // Build accelerator tables.
+  buildAcceleratorTables();
+
+  // Write debug tables from all object files/compile units into the
+  // resulting file.
+  writeDWARFToTheOutput();
+
+  if (Options.Statistics)
+    printStatistic(SizeByObject);
+}
+
+void DWARFLinker::printStatistic(const StringMap<DebugInfoSize> &SizeByObject) {
+  // Create a vector sorted in descending order by output size.
+  std::vector<std::pair<StringRef, DebugInfoSize>> Sorted;
+  for (auto &E : SizeByObject)
+    Sorted.emplace_back(E.first(), E.second);
+  llvm::sort(Sorted, [](auto &LHS, auto &RHS) {
+    return LHS.second.Output > RHS.second.Output;
+  });
+
+  auto ComputePercentange = [](int64_t Input, int64_t Output) -> float {
+    const float Difference = Output - Input;
+    const float Sum = Input + Output;
+    if (Sum == 0)
+      return 0;
+    return (Difference / (Sum / 2));
+  };
+
+  int64_t InputTotal = 0;
+  int64_t OutputTotal = 0;
+  const char *FormatStr = "{0,-45} {1,10}b  {2,10}b {3,8:P}\n";
+
+  // Print header.
+  outs() << ".debug_info section size (in bytes)\n";
+  outs() << "----------------------------------------------------------------"
+            "---------------\n";
+  outs() << "Filename                                           Object       "
+            "  dSYM   Change\n";
+  outs() << "----------------------------------------------------------------"
+            "---------------\n";
+
+  // Print body.
+  for (auto &E : Sorted) {
+    InputTotal += E.second.Input;
+    OutputTotal += E.second.Output;
+    llvm::outs() << formatv(
+        FormatStr, sys::path::filename(E.first).take_back(45), E.second.Input,
+        E.second.Output, ComputePercentange(E.second.Input, E.second.Output));
+  }
+  // Print total and footer.
+  outs() << "----------------------------------------------------------------"
+            "---------------\n";
+  llvm::outs() << formatv(FormatStr, "Total", InputTotal, OutputTotal,
+                          ComputePercentange(InputTotal, OutputTotal));
+  outs() << "----------------------------------------------------------------"
+            "---------------\n\n";
+}
+
+void DWARFLinker::assignOffsets(StringMap<DebugInfoSize> &SizeByObject) {
+  std::array<uint64_t, CompileUnit::TableKindsNum> TableOffsets = {0};
+  uint64_t DebugInfoSizeInTheStartOfContext = 0;
+
+  for (const std::unique_ptr<LinkContext> &Context : ObjectContexts) {
+    DebugInfoSizeInTheStartOfContext = TableOffsets[static_cast<
+        typename std::underlying_type<CompileUnit::DebugTableKind>::type>(
+        CompileUnit::DebugTableKind::DebugInfo)];
+
+    assignOffsets(*Context, TableOffsets);
+
+    for (std::unique_ptr<CompileUnit> &CU : Context->ModulesCompileUnits)
+      assignOffsets(*CU, TableOffsets);
+
+    for (std::unique_ptr<CompileUnit> &CU : Context->CompileUnits)
+      assignOffsets(*CU, TableOffsets);
+
+    uint64_t DebugInfoSizeInTheEndOfContext = TableOffsets[static_cast<
+        typename std::underlying_type<CompileUnit::DebugTableKind>::type>(
+        CompileUnit::DebugTableKind::DebugInfo)];
+
+    SizeByObject[Context->File.FileName].Input = Context->OriginalDebugInfoSize;
+    SizeByObject[Context->File.FileName].Output =
+        (DebugInfoSizeInTheEndOfContext - DebugInfoSizeInTheStartOfContext);
+  }
+}
+
+void DWARFLinker::assignOffsets(
+    LinkContext &Context,
+    std::array<uint64_t, CompileUnit::TableKindsNum> &TableOffsets) {
+  for (size_t i = 0; i < TableOffsets.size(); i++)
+    Context.setStartOffset(static_cast<CompileUnit::DebugTableKind>(i),
+                           TableOffsets[i]);
+
+  if (Context.OutDebugInfoBytes.empty())
+    return;
+
+  MemoryBufferRef Mem(Context.OutDebugInfoBytes, "obj");
+  Expected<std::unique_ptr<object::ObjectFile>> Obj =
+      object::ObjectFile::createObjectFile(Mem);
+  if (!Obj) {
+    consumeError(Obj.takeError());
+    return;
+  }
+
+  // track sizes of the tables of current unit.
+  enumerateDebugSections(**Obj, [&](StringRef Name, StringRef Data) {
+    if (Optional<CompileUnit::DebugTableKind> TableKind =
+            CompileUnit::parseDebugTableName(Name)) {
+      TableOffsets[static_cast<typename std::underlying_type<
+          CompileUnit::DebugTableKind>::type>(*TableKind)] += Data.size();
+    }
+  });
+}
+
+void DWARFLinker::assignOffsets(
+    CompileUnit &CU,
+    std::array<uint64_t, CompileUnit::TableKindsNum> &TableOffsets) {
+  // set offsets to the current unit.
+  for (size_t i = 0; i < TableOffsets.size(); i++)
+    CU.setStartOffset(static_cast<CompileUnit::DebugTableKind>(i),
+                      TableOffsets[i]);
+
+  if (CU.getDwarfBits().empty())
+    return;
+
+  MemoryBufferRef Mem(CU.getDwarfBits(), "obj");
+  Expected<std::unique_ptr<object::ObjectFile>> Obj =
+      object::ObjectFile::createObjectFile(Mem);
+  if (!Obj) {
+    consumeError(Obj.takeError());
+    return;
+  }
+
+  // track sizes of the tables of current unit.
+  enumerateDebugSections(**Obj, [&](StringRef Name, StringRef Data) {
+    if (Optional<CompileUnit::DebugTableKind> TableKind =
+            CompileUnit::parseDebugTableName(Name)) {
+      TableOffsets[static_cast<typename std::underlying_type<
+          CompileUnit::DebugTableKind>::type>(*TableKind)] += Data.size();
+    }
+  });
+}
+
+void DWARFLinker::patchOffsetsAndSizes() {
+  if (Options.NoOutput)
+    return;
+
+  // TODO: offsets might be patched in parrallel
+  for (const std::unique_ptr<LinkContext> &Context : ObjectContexts) {
+    patchOffsetsAndSizes(*Context);
+
+    for (std::unique_ptr<CompileUnit> &CU : Context->ModulesCompileUnits)
+      patchOffsetsAndSizes(*CU);
+
+    for (std::unique_ptr<CompileUnit> &CU : Context->CompileUnits)
+      patchOffsetsAndSizes(*CU);
+  }
+}
+
+void DWARFLinker::patchOffsetsAndSizes(LinkContext &Context) {
+  MemoryBufferRef Mem(Context.OutDebugInfoBytes, "obj");
+  Expected<std::unique_ptr<object::ObjectFile>> Obj =
+      object::ObjectFile::createObjectFile(Mem);
+  if (!Obj) {
+    consumeError(Obj.takeError());
+    return;
+  }
+
+  enumerateDebugSections(**Obj, [&](StringRef Name, StringRef Data) {
+    if (Name == "debug_frame")
+      patchFrames(Context, Data);
+  });
+}
+
+void DWARFLinker::patchOffsetsAndSizes(CompileUnit &CU) {
+  if (CU.getDwarfBits().empty())
+    return;
+
+  MemoryBufferRef Mem(CU.getDwarfBits(), "obj");
+  Expected<std::unique_ptr<object::ObjectFile>> Obj =
+      object::ObjectFile::createObjectFile(Mem);
+  if (!Obj) {
+    consumeError(Obj.takeError());
+    return;
+  }
+
+  enumerateDebugSections(**Obj, [&](StringRef Name, StringRef Data) {
+    if (Name == "debug_info")
+      patchCompileUnit(CU, Data);
+    else if (Name == "debug_aranges")
+      patchAranges(CU, Data);
+  });
+}
+
+void DWARFLinker::buildAcceleratorTables() {
+  for (const std::unique_ptr<LinkContext> &Context : ObjectContexts) {
+    for (std::unique_ptr<CompileUnit> &CU : Context->ModulesCompileUnits)
+      buildAcceleratorTables(*CU);
+
+    for (std::unique_ptr<CompileUnit> &CU : Context->CompileUnits)
+      buildAcceleratorTables(*CU);
+  }
+}
+
+void DWARFLinker::buildAcceleratorTables(CompileUnit &CU) {
+  // TODO: index tables might be generated in parallel.
+  if (Options.GenerateAppleNamespaces)
+    for (const auto &Namespace : CU.getNamespaces())
+      AppleNamespaces.addName(
+          TheOffsetsStringPool.getEntry(Namespace.Name.getString()),
+          Namespace.OutOffset +
+              CU.getStartOffset(CompileUnit::DebugTableKind::DebugInfo));
+
+  if (Options.GenerateAppleNames)
+    for (const auto &Pubname : CU.getPubnames())
+      AppleNames.addName(
+          TheOffsetsStringPool.getEntry(Pubname.Name.getString()),
+          Pubname.OutOffset +
+              CU.getStartOffset(CompileUnit::DebugTableKind::DebugInfo));
+
+  if (Options.GenerateAppleTypes)
+    for (const auto &Pubtype : CU.getPubtypes())
+      AppleTypes.addName(
+          TheOffsetsStringPool.getEntry(Pubtype.Name.getString()),
+          Pubtype.OutOffset +
+              CU.getStartOffset(CompileUnit::DebugTableKind::DebugInfo),
+          Pubtype.Tag,
+          Pubtype.ObjcClassImplementation ? dwarf::DW_FLAG_type_implementation
+                                          : 0,
+          Pubtype.QualifiedNameHash);
+
+  if (Options.GenerateAppleObjC)
+    for (const auto &ObjC : CU.getObjC())
+      AppleObjc.addName(
+          TheOffsetsStringPool.getEntry(ObjC.Name.getString()),
+          ObjC.OutOffset +
+              CU.getStartOffset(CompileUnit::DebugTableKind::DebugInfo));
+
+  if (Options.GeneratePubNames && TheDwarfEmitter)
+    TheDwarfEmitter->emitPubNamesForUnit(CU);
+
+  if (Options.GeneratePubTypes && TheDwarfEmitter)
+    TheDwarfEmitter->emitPubTypesForUnit(CU);
+
+  if (Options.GenerateDebugNames) {
+    for (const auto &Namespace : CU.getNamespaces())
+      DebugNames.addName(
+          TheOffsetsStringPool.getEntry(Namespace.Name.getString()),
+          Namespace.OutOffset, Namespace.Tag, CU.getUniqueID());
+    for (const auto &Pubname : CU.getPubnames())
+      DebugNames.addName(
+          TheOffsetsStringPool.getEntry(Pubname.Name.getString()),
+          Pubname.OutOffset, Pubname.Tag, CU.getUniqueID());
+    for (const auto &Pubtype : CU.getPubtypes())
+      DebugNames.addName(
+          TheOffsetsStringPool.getEntry(Pubtype.Name.getString()),
+          Pubtype.OutOffset, Pubtype.Tag, CU.getUniqueID());
+  }
+
+  CU.cleanupAcceleratorTables();
+}
+
+void DWARFLinker::writeDWARFToTheOutput() {
+  bool HasAbbreviations = false;
+
+  for (const std::unique_ptr<LinkContext> &Context : ObjectContexts) {
+    writeDWARFToTheOutput(*Context);
+
+    for (std::unique_ptr<CompileUnit> &CU : Context->ModulesCompileUnits)
+      writeDWARFToTheOutput(*CU, HasAbbreviations);
+
+    for (std::unique_ptr<CompileUnit> &CU : Context->CompileUnits)
+      writeDWARFToTheOutput(*CU, HasAbbreviations);
+  }
+
+  // generate empty abbrev
+  if (!HasAbbreviations) {
+    std::vector<std::unique_ptr<DIEAbbrev>> Abbreviations;
+    TheDwarfEmitter->emitAbbrevs(Abbreviations, 3);
+  }
+
+  // put sections common to all compilation units.
+  TheDwarfEmitter->emitStrings(TheOffsetsStringPool);
+
+  if (Options.GenerateAppleNamespaces)
+    TheDwarfEmitter->emitAppleNamespaces(AppleNamespaces);
+
+  if (Options.GenerateAppleNames)
+    TheDwarfEmitter->emitAppleNames(AppleNames);
+
+  if (Options.GenerateAppleTypes)
+    TheDwarfEmitter->emitAppleTypes(AppleTypes);
+
+  if (Options.GenerateAppleObjC)
+    TheDwarfEmitter->emitAppleObjc(AppleObjc);
+
+  if (Options.GenerateDebugNames)
+    TheDwarfEmitter->emitDebugNames(DebugNames);
+}
+
+void DWARFLinker::writeDWARFToTheOutput(CompileUnit &CU,
+                                        bool &HasAbbreviations) {
+  if (CU.getDwarfBits().empty())
+    return;
+
+  if (!HasAbbreviations && CU.getAbbreviations().size() > 0)
+    HasAbbreviations = true;
+
+  MemoryBufferRef Mem(CU.getDwarfBits(), "obj");
+  Expected<std::unique_ptr<object::ObjectFile>> Obj =
+      object::ObjectFile::createObjectFile(Mem);
+  if (!Obj) {
+    consumeError(Obj.takeError());
+    return;
+  }
+
+  // now glue sections and put them into output DwarfEmitter.
+  enumerateDebugSections(**Obj, [&](StringRef Name, StringRef Data) {
+    TheDwarfEmitter->emitSectionContents(Data, Name);
+  });
+
+  CU.eraseDwarfBits();
+}
+
+void DWARFLinker::writeDWARFToTheOutput(LinkContext &Context) {
+  if (Context.OutDebugInfoBytes.empty())
+    return;
+
+  MemoryBufferRef Mem(Context.OutDebugInfoBytes, "obj");
+  Expected<std::unique_ptr<object::ObjectFile>> Obj =
+      object::ObjectFile::createObjectFile(Mem);
+  if (!Obj) {
+    consumeError(Obj.takeError());
+    return;
+  }
+
+  // now glue sections and put them into output DwarfEmitter.
+  enumerateDebugSections(**Obj, [&](StringRef Name, StringRef Data) {
+    TheDwarfEmitter->emitSectionContents(Data, Name);
+  });
+
+  Context.OutDebugInfoBytes = SmallString<0>();
+}
+
+} // end of namespace dwarflinker
+
+} // namespace llvm

llvm/lib/DWARFLinkerNext/DWARFLinkerCompileUnit.cpp

+//===- DWARFLinkerCompileUnit.cpp -----------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DWARFLinkerNext/DWARFLinkerCompileUnit.h"
+#include "llvm/DWARFLinkerNext/DWARFLinker.h"
+#include "llvm/DWARFLinkerNext/DWARFLinkerDeclContext.h"
+
+namespace llvm {
+
+namespace dwarflinker {
+
+/// Check if the DIE at \p Idx is in the scope of a function.
+static bool inFunctionScope(CompileUnit &U, unsigned Idx) {
+  while (Idx) {
+    if (U.getOrigUnit().getDIEAtIndex(Idx).getTag() == dwarf::DW_TAG_subprogram)
+      return true;
+    Idx = U.getInfo(Idx).ParentIdx;
+  }
+  return false;
+}
+
+CompileUnit::CompileUnit(unsigned ID, bool, StringRef ClangModuleName,
+                         DWARFFile &File)
+    : ID(ID), Ranges(RangeAlloc), ClangModuleName(ClangModuleName),
+      ContaingFile(File), Stage(Stages::Created) {
+  Info.resize(1);
+  Version = 3;
+}
+
+CompileUnit::CompileUnit(DWARFUnit &OrigUnit, unsigned ID, bool CanUseODR,
+                         StringRef ClangModuleName, DWARFFile &File)
+    : OrigUnit(&OrigUnit), ID(ID), Ranges(RangeAlloc),
+      ClangModuleName(ClangModuleName), ContaingFile(File),
+      Stage(Stages::Created) {
+  auto CUDie = OrigUnit.getUnitDIE();
+  if (!CUDie)
+    return;
+
+  if (File.Dwarf)
+    Endianess = File.Dwarf->isLittleEndian() ? support::endianness::little
+                                             : support::endianness::big;
+
+  Version = OrigUnit.getVersion();
+
+  if (auto Lang = dwarf::toUnsigned(CUDie.find(dwarf::DW_AT_language)))
+    HasODR = CanUseODR && (*Lang == dwarf::DW_LANG_C_plus_plus ||
+                           *Lang == dwarf::DW_LANG_C_plus_plus_03 ||
+                           *Lang == dwarf::DW_LANG_C_plus_plus_11 ||
+                           *Lang == dwarf::DW_LANG_C_plus_plus_14 ||
+                           *Lang == dwarf::DW_LANG_ObjC_plus_plus);
+
+  DeclContextData.reserve(1000);
+  StringOffsetsData.reserve(1000);
+}
+
+uint16_t CompileUnit::getLanguage() {
+  if (!Language) {
+    DWARFDie CU = getOrigUnit().getUnitDIE();
+    Language = dwarf::toUnsigned(CU.find(dwarf::DW_AT_language), 0);
+  }
+  return Language;
+}
+
+StringRef CompileUnit::getSysRoot() {
+  if (SysRoot.empty()) {
+    DWARFDie CU = getOrigUnit().getUnitDIE();
+    SysRoot = dwarf::toStringRef(CU.find(dwarf::DW_AT_LLVM_sysroot)).str();
+  }
+  return SysRoot;
+}
+
+void CompileUnit::noteDieReference(CompileUnit::DieRefAttrPatchInfo Data) {
+  DIEReferences.emplace_back(Data);
+}
+
+std::vector<CompileUnit::DieRefAttrPatchInfo> CompileUnit::getDiesReferences() {
+  return std::move(DIEReferences);
+}
+
+void CompileUnit::initIteration() {
+  // if exist external references to this compile unit, then we need to
+  // begin processing for this unit again, starting from TypesDeclarationsFound
+  // stage.
+  if (getStage() == CompileUnit::Stages::Cloned) {
+    if (hasExternalReferences())
+      setStage(CompileUnit::Stages::TypesDeclarationsFound);
+    else
+      return;
+  }
+  Info.resize(getOrigUnit().getNumDIEs());
+
+  eraseDwarfBits();
+  OutDebugInfoBytes.reserve(4000);
+
+  if (getStage() < Stages::LivenessAnalysisDone) {
+    Ranges.clear();
+    Labels.clear();
+  }
+
+  if (getStage() < Stages::Cloned) {
+    Pubnames.clear();
+    Pubtypes.clear();
+    Namespaces.clear();
+    ObjC.clear();
+    RangeAttributes.clear();
+    UnitRangeAttribute.reset();
+    LocationAttributes.clear();
+  }
+}
+
+void CompileUnit::cleanupDataAfterClonning() {
+  assert(DeclContextData.size() == 0);
+  assert(StringOffsetsData.size() == 0);
+
+  NewUnit.reset();
+  ResolvedFullPaths.shrink_and_clear();
+  ResolvedParentPaths.clear();
+  Keys.clear();
+  Allocator.Reset();
+
+  if (!needToKeepInputDies() && OrigUnit)
+    (*OrigUnit)->clear();
+
+  AbbreviationsSet.clear();
+}
+
+void CompileUnit::cleanupAcceleratorTables() {
+  Pubnames.clear();
+  Pubnames.shrink_to_fit();
+
+  Pubtypes.clear();
+  Pubtypes.shrink_to_fit();
+
+  Namespaces.clear();
+  Namespaces.shrink_to_fit();
+
+  ObjC.clear();
+  ObjC.shrink_to_fit();
+
+  Strings.clear();
+}
+
+void CompileUnit::addLabelLowPc(uint64_t LabelLowPc, int64_t PcOffset) {
+  Labels.insert({LabelLowPc, PcOffset});
+}
+
+void CompileUnit::addFunctionRange(uint64_t FuncLowPc, uint64_t FuncHighPc,
+                                   int64_t PcOffset) {
+  //  Don't add empty ranges to the interval map.  They are a problem because
+  //  the interval map expects half open intervals. This is safe because they
+  //  are empty anyway.
+  if (FuncHighPc != FuncLowPc)
+    Ranges.insert(FuncLowPc, FuncHighPc, PcOffset);
+  this->LowPc = std::min(LowPc, FuncLowPc + PcOffset);
+  this->HighPc = std::max(HighPc, FuncHighPc + PcOffset);
+}
+
+void CompileUnit::noteRangeAttribute(const DIE &Die,
+                                     RangeAttributePatchInfo AttrOutOffsets) {
+  if (Die.getTag() != dwarf::DW_TAG_compile_unit)
+    RangeAttributes.push_back(AttrOutOffsets);
+  else
+    UnitRangeAttribute = AttrOutOffsets;
+}
+
+void CompileUnit::noteLocationAttribute(uint64_t AttrOutOffset, uint64_t Offset,
+                                        int64_t PcOffset) {
+  LocationAttributes.emplace_back(
+      LocationAttributePatchInfo{AttrOutOffset, Offset, PcOffset});
+}
+
+void CompileUnit::markEverythingAsKept() {
+  unsigned Idx = 0;
+
+  for (auto &I : Info) {
+    // Mark everything that wasn't explicit marked for pruning.
+    I.Keep = !I.Prune;
+    auto DIE = getOrigUnit().getDIEAtIndex(Idx++);
+
+    // Try to guess which DIEs must go to the accelerator tables. We do that
+    // just for variables, because functions will be handled depending on
+    // whether they carry a DW_AT_low_pc attribute or not.
+    if (DIE.getTag() != dwarf::DW_TAG_variable &&
+        DIE.getTag() != dwarf::DW_TAG_constant)
+      continue;
+
+    Optional<DWARFFormValue> Value;
+    if (!(Value = DIE.find(dwarf::DW_AT_location))) {
+      if ((Value = DIE.find(dwarf::DW_AT_const_value)) &&
+          !inFunctionScope(*this, I.ParentIdx))
+        I.InDebugMap = true;
+      continue;
+    }
+    if (auto Block = Value->getAsBlock()) {
+      if (Block->size() > getOrigUnit().getAddressByteSize() &&
+          (*Block)[0] == dwarf::DW_OP_addr)
+        I.InDebugMap = true;
+    }
+  }
+}
+
+void CompileUnit::addNamespaceAccelerator(const DIE *Die, StringRef Name) {
+  Namespaces.emplace_back(Strings.getEntry(Name), Die->getTag(),
+                          Die->getOffset());
+}
+
+void CompileUnit::addObjCAccelerator(const DIE *Die, StringRef Name,
+                                     bool SkipPubSection) {
+  ObjC.emplace_back(Strings.getEntry(Name), Die->getTag(), Die->getOffset(),
+                    SkipPubSection);
+}
+
+void CompileUnit::addNameAccelerator(const DIE *Die, StringRef Name,
+                                     bool SkipPubSection) {
+  Pubnames.emplace_back(Strings.getEntry(Name), Die->getTag(), Die->getOffset(),
+                        SkipPubSection);
+}
+
+void CompileUnit::addTypeAccelerator(const DIE *Die, StringRef Name,
+                                     bool ObjcClassImplementation,
+                                     uint32_t QualifiedNameHash) {
+  Pubtypes.emplace_back(Strings.getEntry(Name), Die->getTag(), Die->getOffset(),
+                        QualifiedNameHash, ObjcClassImplementation);
+}
+
+Optional<CompileUnit::DebugTableKind>
+CompileUnit::parseDebugTableName(StringRef Name) {
+  if (Name == "debug_info")
+    return CompileUnit::DebugTableKind::DebugInfo;
+  else if (Name == "debug_line")
+    return CompileUnit::DebugTableKind::DebugLine;
+  else if (Name == "debug_frame")
+    return CompileUnit::DebugTableKind::DebugFrame;
+  else if (Name == "debug_ranges")
+    return CompileUnit::DebugTableKind::DebugRange;
+  else if (Name == "debug_loc")
+    return CompileUnit::DebugTableKind::DebugLoc;
+  else if (Name == "debug_aranges")
+    return CompileUnit::DebugTableKind::DebugARanges;
+  else if (Name == "debug_abbrev")
+    return CompileUnit::DebugTableKind::DebugAbbrev;
+
+  return None;
+}
+
+void CompileUnit::assignAbbrev(DIEAbbrev &Abbrev) {
+  // Check the set for priors.
+  FoldingSetNodeID ID;
+  Abbrev.Profile(ID);
+  void *InsertToken;
+
+  DIEAbbrev *InSet = AbbreviationsSet.FindNodeOrInsertPos(ID, InsertToken);
+  // If it's newly added.
+  if (InSet) {
+    // Assign existing abbreviation number.
+    Abbrev.setNumber(InSet->getNumber());
+  } else {
+    // Add to abbreviation list.
+    Abbreviations.push_back(
+        std::make_unique<DIEAbbrev>(Abbrev.getTag(), Abbrev.hasChildren()));
+    for (const auto &Attr : Abbrev.getData())
+      Abbreviations.back()->AddAttribute(Attr.getAttribute(), Attr.getForm());
+    AbbreviationsSet.InsertNode(Abbreviations.back().get(), InsertToken);
+    // Assign the unique abbreviation number.
+    Abbrev.setNumber(Abbreviations.size());
+    Abbreviations.back()->setNumber(Abbreviations.size());
+  }
+}
+
+Optional<StringRef> CompileUnit::getFileName(unsigned FileIdx) {
+  if (const auto *LT =
+          getOrigUnit().getContext().getLineTableForUnit(&getOrigUnit())) {
+    if (LT->hasFileAtIndex(FileIdx)) {
+
+      // Cache the resolved paths based on the index in the line table,
+      // because calling realpath is expensive.
+      ResolvedPathsMap::const_iterator It = ResolvedFullPaths.find(FileIdx);
+      if (It == ResolvedFullPaths.end()) {
+        std::string OrigFileName;
+        bool FoundFileName = LT->getFileNameByIndex(
+            FileIdx, getOrigUnit().getCompilationDir(),
+            DILineInfoSpecifier::FileLineInfoKind::AbsoluteFilePath,
+            OrigFileName);
+        (void)FoundFileName;
+        assert(FoundFileName && "Must get file name from line table");
+
+        // Second level of caching, this time based on the file's parent
+        // path.
+        StringRef FileName = sys::path::filename(OrigFileName);
+        StringRef ParentPath = sys::path::parent_path(OrigFileName);
+
+        // If the ParentPath has not yet been resolved, resolve and cache it for
+        // future look-ups.
+        StringMap<std::string>::iterator ParentIt =
+            ResolvedParentPaths.find(ParentPath);
+        if (ParentIt == ResolvedParentPaths.end()) {
+          SmallString<256> RealPath;
+          sys::fs::real_path(ParentPath, RealPath);
+          ParentIt = ResolvedParentPaths
+                         .insert({ParentPath, std::string(RealPath.c_str(),
+                                                          RealPath.size())})
+                         .first;
+        }
+
+        // Join the file name again with the resolved path.
+        SmallString<256> ResolvedPath(ParentIt->second);
+        sys::path::append(ResolvedPath, FileName);
+
+        It = ResolvedFullPaths
+                 .insert(std::make_pair(FileIdx,
+                                        Strings.internString(ResolvedPath)))
+                 .first;
+      }
+
+      return It->second;
+    }
+  }
+
+  return None;
+}
+
+} // end of namespace dwarflinker
+
+} // namespace llvm

llvm/lib/DWARFLinkerNext/DWARFLinkerDeclContext.cpp

+//===- DWARFLinkerDeclContext.cpp -----------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DWARFLinkerNext/DWARFLinkerDeclContext.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/DWARFLinkerNext/DWARFLinker.h"
+#include "llvm/DebugInfo/DWARF/DWARFContext.h"
+#include "llvm/DebugInfo/DWARF/DWARFDie.h"
+#include "llvm/DebugInfo/DWARF/DWARFUnit.h"
+
+namespace llvm {
+
+namespace dwarflinker {
+
+// DieNameResolver is a helper class which searches for type name
+// and checks for unresolved inter-CU references.
+class DieNameResolver {
+  const LinkContext &Context;
+  CompileUnit &CU;
+  const DWARFDie ¨
+  bool IsUnresolved = false;
+
+public:
+  DieNameResolver(const LinkContext &Context, CompileUnit &CU,
+                  const DWARFDie &DIE)
+      : Context(Context), CU(CU), DIE(DIE) {}
+
+  StringRef resolve() {
+    // Check for linkage name.
+    if (Optional<DWARFFormValue> Val = findRecursively(
+            {dwarf::DW_AT_MIPS_linkage_name, dwarf::DW_AT_linkage_name})) {
+      if (IsUnresolved)
+        return StringRef();
+
+      if (Optional<const char *> String = dwarf::toString(*Val))
+        return StringRef(*String);
+    }
+
+    // Check for short name.
+    if (Optional<DWARFFormValue> Val = findRecursively(dwarf::DW_AT_name)) {
+      if (IsUnresolved)
+        return StringRef();
+
+      if (Optional<const char *> String = dwarf::toString(*Val))
+        return StringRef(*String);
+    }
+
+    return StringRef();
+  }
+
+  bool isUnresolved() const { return IsUnresolved; }
+
+protected:
+  DWARFDie getAttributeValueAsReferencedDie(dwarf::Attribute Attr) {
+    if (Optional<DWARFFormValue> F = DIE.find(Attr))
+      return getAttributeValueAsReferencedDie(*F);
+    return DWARFDie();
+  }
+
+  DWARFDie getAttributeValueAsReferencedDie(const DWARFFormValue &V) {
+    if (auto SpecRef = V.getAsRelativeReference()) {
+      if (SpecRef->Unit)
+        return SpecRef->Unit->getDIEForOffset(SpecRef->Unit->getOffset() +
+                                              SpecRef->Offset);
+
+      if (CompileUnit *RefCU = Context.getUnitForOffset(CU, SpecRef->Offset)) {
+        if (RefCU->getStage() < CompileUnit::Stages::SrcDWARFLoaded) {
+          RefCU->setKeepInputDies();
+          IsUnresolved = true;
+          return DWARFDie();
+        }
+
+        return RefCU->getOrigUnit().getDIEForOffset(SpecRef->Offset);
+      }
+    }
+    return DWARFDie();
+  }
+
+  Optional<DWARFFormValue> findRecursively(ArrayRef<dwarf::Attribute> Attrs) {
+    SmallVector<DWARFDie, 3> Worklist;
+    Worklist.push_back(DIE);
+
+    // Keep track if DIEs already seen to prevent infinite recursion.
+    // Empirically we rarely see a depth of more than 3 when dealing with valid
+    // DWARF. This corresponds to following the DW_AT_abstract_origin and
+    // DW_AT_specification just once.
+    SmallSet<DWARFDie, 3> Seen;
+    Seen.insert(DIE);
+
+    while (!Worklist.empty()) {
+      DWARFDie Die = Worklist.pop_back_val();
+
+      if (!Die.isValid())
+        continue;
+
+      if (auto Value = Die.find(Attrs))
+        return Value;
+
+      if (auto D = getAttributeValueAsReferencedDie(
+              llvm::dwarf::DW_AT_abstract_origin)) {
+        if (IsUnresolved)
+          return None;
+
+        if (Seen.insert(D).second)
+          Worklist.push_back(D);
+      }
+
+      if (auto D = getAttributeValueAsReferencedDie(
+              llvm::dwarf::DW_AT_specification)) {
+        if (IsUnresolved)
+          return None;
+
+        if (Seen.insert(D).second)
+          Worklist.push_back(D);
+      }
+    }
+
+    return None;
+  }
+};
+
+DeclContextKey *DeclContextTree::getChildDeclContextKey(
+    const LinkContext &Context, DeclContextKey *ContextKey, const DWARFDie &DIE,
+    CompileUnit &CU, bool DefinedInClangModule, bool &IsUnresolved) {
+  assert(ContextKey);
+
+  IsUnresolved = false;
+  unsigned Tag = DIE.getTag();
+
+  // FIXME: dsymutil-classic compat: We should bail out here if we
+  // have a specification or an abstract_origin. We will get the
+  // parent context wrong here.
+
+  switch (Tag) {
+  default:
+    // By default stop gathering child contexts.
+    return nullptr;
+  case dwarf::DW_TAG_module:
+    break;
+  case dwarf::DW_TAG_compile_unit: {
+    ContextKey->IsInvalid = true;
+    return ContextKey;
+  }
+  case dwarf::DW_TAG_subprogram:
+    // Do not unique anything inside CU local functions.
+    if ((ContextKey->Tag == dwarf::DW_TAG_namespace ||
+         ContextKey->Tag == dwarf::DW_TAG_compile_unit) &&
+        !dwarf::toUnsigned(DIE.find(dwarf::DW_AT_external), 0))
+      return nullptr;
+    LLVM_FALLTHROUGH;
+  case dwarf::DW_TAG_member:
+  case dwarf::DW_TAG_namespace:
+  case dwarf::DW_TAG_structure_type:
+  case dwarf::DW_TAG_class_type:
+  case dwarf::DW_TAG_union_type:
+  case dwarf::DW_TAG_enumeration_type:
+  case dwarf::DW_TAG_typedef:
+    // Artificial things might be ambiguous, because they might be created on
+    // demand. For example implicitly defined constructors are ambiguous
+    // because of the way we identify contexts, and they won't be generated
+    // every time everywhere.
+    if (dwarf::toUnsigned(DIE.find(dwarf::DW_AT_artificial), 0))
+      return nullptr;
+    break;
+  }
+
+  StringRef NameRef;
+  StringRef FileRef;
+
+  DieNameResolver NameResolver(Context, CU, DIE);
+  NameRef = NameResolver.resolve();
+
+  if (NameResolver.isUnresolved()) {
+    // DIE references other compile unit, which is not loaded yet.
+    // Stop type declaration analyzing for now.
+    CU.setKeepInputDies();
+    IsUnresolved = true;
+    return nullptr;
+  }
+
+  bool IsAnonymousNamespace = NameRef.empty() && Tag == dwarf::DW_TAG_namespace;
+  if (IsAnonymousNamespace) {
+    // FIXME: For dsymutil-classic compatibility. I think uniquing within
+    // anonymous namespaces is wrong. There is no ODR guarantee there.
+    NameRef = "(anonymous namespace)";
+  }
+
+  if (Tag != dwarf::DW_TAG_class_type && Tag != dwarf::DW_TAG_structure_type &&
+      Tag != dwarf::DW_TAG_union_type &&
+      Tag != dwarf::DW_TAG_enumeration_type && NameRef.empty())
+    return nullptr;
+
+  unsigned Line = 0;
+  unsigned ByteSize = std::numeric_limits<uint32_t>::max();
+
+  if (!DefinedInClangModule) {
+    // Gather some discriminating data about the DeclContext we will be
+    // creating: File, line number and byte size. This shouldn't be necessary,
+    // because the ODR is just about names, but given that we do some
+    // approximations with overloaded functions and anonymous namespaces, use
+    // these additional data points to make the process safer.
+    //
+    // This is disabled for clang modules, because forward declarations of
+    // module-defined types do not have a file and line.
+    ByteSize = dwarf::toUnsigned(DIE.find(dwarf::DW_AT_byte_size),
+                                 std::numeric_limits<uint64_t>::max());
+    if (Tag != dwarf::DW_TAG_namespace || IsAnonymousNamespace) {
+      if (unsigned FileNum =
+              dwarf::toUnsigned(DIE.find(dwarf::DW_AT_decl_file), 0)) {
+        // FIXME: dsymutil-classic compatibility. I'd rather not
+        // unique anything in anonymous namespaces, but if we do, then
+        // verify that the file and line correspond.
+        if (IsAnonymousNamespace)
+          FileNum = 1;
+
+        if (Optional<StringRef> FileName = CU.getFileName(FileNum)) {
+          Line = dwarf::toUnsigned(DIE.find(dwarf::DW_AT_decl_line), 0);
+          FileRef = *FileName;
+        }
+      }
+    }
+  }
+
+  if (!Line && NameRef.empty())
+    return nullptr;
+
+  // We hash NameRef, which is the mangled name, in order to get most
+  // overloaded functions resolve correctly.
+  //
+  // Strictly speaking, hashing the Tag is only necessary for a
+  // DW_TAG_module, to prevent uniquing of a module and a namespace
+  // with the same name.
+  //
+  // FIXME: dsymutil-classic won't unique the same type presented
+  // once as a struct and once as a class. Using the Tag in the fully
+  // qualified name hash to get the same effect.
+  unsigned Hash = hash_combine(ContextKey->Hash, Tag, NameRef);
+
+  // FIXME: dsymutil-classic compatibility: when we don't have a name,
+  // use the filename.
+  if (IsAnonymousNamespace)
+    Hash = hash_combine(Hash, FileRef);
+
+  bool IsInvalid = false;
+  // FIXME: dsymutil-classic compatibility. Union types aren't
+  // uniques, but their children might be.
+  if ((Tag == dwarf::DW_TAG_subprogram &&
+       ContextKey->Tag != dwarf::DW_TAG_structure_type &&
+       ContextKey->Tag != dwarf::DW_TAG_class_type) ||
+      (Tag == dwarf::DW_TAG_union_type)) {
+    IsInvalid = true;
+  }
+
+  DeclContextKey NewKey(Hash, Line, ByteSize, Tag, NameRef, FileRef,
+                        ContextKey->Hash, IsInvalid, DefinedInClangModule);
+  return CU.insertKey(NewKey);
+}
+
+DeclContext *
+DeclContextTree::getChildDeclContext(const DeclContextKey &ContextKey) {
+  DeclContext Key(ContextKey);
+  auto ContextIter = Contexts.find(&Key);
+
+  if (ContextIter == Contexts.end()) {
+    // The context wasn't found.
+    // We need to store ContextKey.Name and ContextKey.File in the StringPool
+    // since lifetime of DeclContextTree is longer than lifetime
+    // of ContextKey which points to CU local data.
+    bool Inserted;
+    DeclContext *NewContext = new (Allocator)
+        DeclContext(ContextKey.Hash, ContextKey.Line, ContextKey.ByteSize,
+                    ContextKey.Tag, StringPool.internString(ContextKey.Name),
+                    StringPool.internString(ContextKey.File),
+                    ContextKey.ParentQualifiedNameHash);
+    std::tie(ContextIter, Inserted) = Contexts.insert(NewContext);
+    assert(Inserted && "Failed to insert DeclContext");
+    (void)Inserted;
+  } else if (ContextKey.Tag != dwarf::DW_TAG_namespace) {
+    /// In the current implementation, we don't handle overloaded functions
+    /// well, because the argument types are not taken into account when
+    /// computing the DeclContext tree.
+    ///
+    /// Some of this is mitigated byt using mangled names that do contain the
+    /// arguments types, but sometimes (e.g. with function templates) we don't
+    /// have that. In that case, just do not unique anything that refers to the
+    /// contexts we are not able to distinguish.
+    ///
+    /// If a context that is not a namespace appears twice in the same CU, we
+    /// know it is ambiguous.
+    ///
+    /// TODO: implement building unambiguous type declarations.
+  }
+  assert(ContextIter != Contexts.end());
+
+  return *ContextIter;
+}
+
+} // end of namespace dwarflinker
+
+} // namespace llvm

llvm/lib/DWARFLinkerNext/DWARFStreamer.cpp

+//===- DwarfStreamer.cpp --------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DWARFLinkerNext/DWARFStreamer.h"
+#include "llvm/ADT/Triple.h"
+#include "llvm/CodeGen/NonRelocatableStringpool.h"
+#include "llvm/DWARFLinkerNext/DWARFLinkerCompileUnit.h"
+#include "llvm/DebugInfo/DWARF/DWARFContext.h"
+#include "llvm/MC/MCAsmBackend.h"
+#include "llvm/MC/MCCodeEmitter.h"
+#include "llvm/MC/MCDwarf.h"
+#include "llvm/MC/MCObjectWriter.h"
+#include "llvm/MC/MCSection.h"
+#include "llvm/MC/MCStreamer.h"
+#include "llvm/MC/MCSubtargetInfo.h"
+#include "llvm/MC/MCSymbol.h"
+#include "llvm/MC/MCTargetOptions.h"
+#include "llvm/MC/MCTargetOptionsCommandFlags.h"
+#include "llvm/Support/LEB128.h"
+#include "llvm/Support/TargetRegistry.h"
+#include "llvm/Target/TargetOptions.h"
+
+namespace llvm {
+
+namespace dwarflinker {
+
+bool DwarfStreamer::init(Triple TheTriple) {
+  std::string ErrorStr;
+  std::string TripleName;
+  StringRef Context = "dwarf streamer init";
+
+  // Get the target.
+  const Target *TheTarget =
+      TargetRegistry::lookupTarget(TripleName, TheTriple, ErrorStr);
+  if (!TheTarget)
+    return error(ErrorStr, Context), false;
+  TripleName = TheTriple.getTriple();
+
+  // Create all the MC Objects.
+  MRI.reset(TheTarget->createMCRegInfo(TripleName));
+  if (!MRI)
+    return error(Twine("no register info for target ") + TripleName, Context),
+           false;
+
+  MCTargetOptions MCOptions = mc::InitMCTargetOptionsFromFlags();
+  MAI.reset(TheTarget->createMCAsmInfo(*MRI, TripleName, MCOptions));
+  if (!MAI)
+    return error("no asm info for target " + TripleName, Context), false;
+
+  MOFI.reset(new MCObjectFileInfo);
+  MC.reset(new MCContext(MAI.get(), MRI.get(), MOFI.get()));
+  MOFI->InitMCObjectFileInfo(TheTriple, /*PIC*/ false, *MC);
+
+  MSTI.reset(TheTarget->createMCSubtargetInfo(TripleName, "", ""));
+  if (!MSTI)
+    return error("no subtarget info for target " + TripleName, Context), false;
+
+  MAB = TheTarget->createMCAsmBackend(*MSTI, *MRI, MCOptions);
+  if (!MAB)
+    return error("no asm backend for target " + TripleName, Context), false;
+
+  MII.reset(TheTarget->createMCInstrInfo());
+  if (!MII)
+    return error("no instr info info for target " + TripleName, Context), false;
+
+  MCE = TheTarget->createMCCodeEmitter(*MII, *MRI, *MC);
+  if (!MCE)
+    return error("no code emitter for target " + TripleName, Context), false;
+
+  switch (OutFileType) {
+  case OutputFileType::Assembly: {
+    MIP = TheTarget->createMCInstPrinter(TheTriple, MAI->getAssemblerDialect(),
+                                         *MAI, *MII, *MRI);
+    MS = TheTarget->createAsmStreamer(
+        *MC, std::make_unique<formatted_raw_ostream>(OutFile), true, true, MIP,
+        std::unique_ptr<MCCodeEmitter>(MCE), std::unique_ptr<MCAsmBackend>(MAB),
+        true);
+    break;
+  }
+  case OutputFileType::Object: {
+    MS = TheTarget->createMCObjectStreamer(
+        TheTriple, *MC, std::unique_ptr<MCAsmBackend>(MAB),
+        MAB->createObjectWriter(OutFile), std::unique_ptr<MCCodeEmitter>(MCE),
+        *MSTI, MCOptions.MCRelaxAll, MCOptions.MCIncrementalLinkerCompatible,
+        /*DWARFMustBeAtTheEnd*/ false);
+    break;
+  }
+  }
+
+  if (!MS)
+    return error("no object streamer for target " + TripleName, Context), false;
+
+  // Finally create the AsmPrinter we'll use to emit the DIEs.
+  TM.reset(TheTarget->createTargetMachine(TripleName, "", "", TargetOptions(),
+                                          None));
+  if (!TM)
+    return error("no target machine for target " + TripleName, Context), false;
+
+  Asm.reset(TheTarget->createAsmPrinter(*TM, std::unique_ptr<MCStreamer>(MS)));
+  if (!Asm)
+    return error("no asm printer for target " + TripleName, Context), false;
+
+  RangesSectionSize = 0;
+  LocSectionSize = 0;
+  LineSectionSize = 0;
+  FrameSectionSize = 0;
+  DebugInfoSectionSize = 0;
+
+  return true;
+}
+
+void DwarfStreamer::finish() { MS->Finish(); }
+
+void DwarfStreamer::switchToDebugInfoSection(unsigned DwarfVersion) {
+  MS->SwitchSection(MOFI->getDwarfInfoSection());
+  MC->setDwarfVersion(DwarfVersion);
+}
+
+/// Emit the compilation unit header for \p Unit in the debug_info section.
+///
+/// A Dwarf 4 section header is encoded as:
+///  uint32_t   Unit length (omitting this field)
+///  uint16_t   Version
+///  uint32_t   Abbreviation table offset
+///  uint8_t    Address size
+///
+/// Leading to a total of 11 bytes.
+///
+/// A Dwarf 5 section header is encoded as:
+///  uint32_t   Unit length (omitting this field)
+///  uint16_t   Version
+///  uint8_t    Unit type
+///  uint8_t    Address size
+///  uint32_t   Abbreviation table offset
+/// Leading to a total of 12 bytes.
+void DwarfStreamer::emitCompileUnitHeader(CompileUnit &Unit) {
+  unsigned Version = Unit.getVersion();
+  switchToDebugInfoSection(Version);
+
+  /// The start of the unit within its section.
+  Unit.setLabelBegin(Asm->createTempSymbol("cu_begin"));
+  Asm->OutStreamer->emitLabel(Unit.getLabelBegin());
+
+  // Emit size of content not including length itself. The size has already
+  // been computed in CompileUnit::computeOffsets(). Subtract 4 to that size to
+  // account for the length field.
+  Asm->emitInt32(-4);
+  Asm->emitInt16(Version);
+
+  if (Version >= 5) {
+    Asm->emitInt8(dwarf::DW_UT_compile);
+    Asm->emitInt8(Unit.getOrigUnit().getAddressByteSize());
+    // We share one abbreviations table across all units so it's always at the
+    // start of the section.
+    Asm->emitInt32(0);
+  } else {
+    // We share one abbreviations table across all units so it's always at the
+    // start of the section.
+    Asm->emitInt32(0);
+    Asm->emitInt8(Unit.getOrigUnit().getAddressByteSize());
+  }
+  DebugInfoSectionSize += Unit.getHeaderSize();
+
+  // Remember this CU.
+  EmittedUnits.push_back({Unit.getUniqueID(), Unit.getLabelBegin()});
+}
+
+/// Emit the \p Abbrevs array as the shared abbreviation table
+/// for the linked Dwarf file.
+void DwarfStreamer::emitAbbrevs(
+    const std::vector<std::unique_ptr<DIEAbbrev>> &Abbrevs,
+    unsigned DwarfVersion) {
+  MS->SwitchSection(MOFI->getDwarfAbbrevSection());
+  MC->setDwarfVersion(DwarfVersion);
+  Asm->emitDwarfAbbrevs(Abbrevs);
+}
+
+/// Recursively emit the DIE tree rooted at \p Die.
+void DwarfStreamer::emitDIE(DIE &Die) {
+  MS->SwitchSection(MOFI->getDwarfInfoSection());
+  Asm->emitDwarfDIE(Die);
+  DebugInfoSectionSize += Die.getSize();
+}
+
+/// Emit contents of section SecName From Obj.
+void DwarfStreamer::emitSectionContents(StringRef SecData, StringRef SecName) {
+  MCSection *Section =
+      StringSwitch<MCSection *>(SecName)
+          .Case("debug_info", MC->getObjectFileInfo()->getDwarfInfoSection())
+          .Case("debug_line", MC->getObjectFileInfo()->getDwarfLineSection())
+          .Case("debug_loc", MC->getObjectFileInfo()->getDwarfLocSection())
+          .Case("debug_ranges",
+                MC->getObjectFileInfo()->getDwarfRangesSection())
+          .Case("debug_frame", MC->getObjectFileInfo()->getDwarfFrameSection())
+          .Case("debug_aranges",
+                MC->getObjectFileInfo()->getDwarfARangesSection())
+          .Case("debug_abbrev",
+                MC->getObjectFileInfo()->getDwarfAbbrevSection())
+          .Default(nullptr);
+
+  if (Section) {
+    MS->SwitchSection(Section);
+
+    MS->emitBytes(SecData);
+  }
+}
+
+/// Emit DIE containing warnings.
+void DwarfStreamer::emitPaperTrailWarningsDie(DIE &Die) {
+  switchToDebugInfoSection(/* Version */ 2);
+
+  Asm->emitInt32(11 + Die.getSize() - 4);
+  Asm->emitInt16(2);
+  Asm->emitInt32(0);
+  Asm->emitInt8(MOFI->getTargetTriple().isArch64Bit() ? 8 : 4);
+
+  DebugInfoSectionSize += 11;
+  emitDIE(Die);
+}
+
+/// Emit the debug_str section stored in \p Pool.
+void DwarfStreamer::emitStrings(const NonRelocatableStringpool &Pool) {
+  Asm->OutStreamer->SwitchSection(MOFI->getDwarfStrSection());
+  std::vector<DwarfStringPoolEntryRef> Entries = Pool.getEntriesForEmission();
+  for (auto Entry : Entries) {
+    // Emit the string itself.
+    Asm->OutStreamer->emitBytes(Entry.getString());
+    // Emit a null terminator.
+    Asm->emitInt8(0);
+  }
+}
+
+void DwarfStreamer::emitDebugNames(
+    AccelTable<DWARF5AccelTableStaticData> &Table) {
+  if (EmittedUnits.empty())
+    return;
+
+  // Build up data structures needed to emit this section.
+  std::vector<MCSymbol *> CompUnits;
+  DenseMap<unsigned, size_t> UniqueIdToCuMap;
+  unsigned Id = 0;
+  for (auto &CU : EmittedUnits) {
+    CompUnits.push_back(CU.LabelBegin);
+    // We might be omitting CUs, so we need to remap them.
+    UniqueIdToCuMap[CU.ID] = Id++;
+  }
+
+  Asm->OutStreamer->SwitchSection(MOFI->getDwarfDebugNamesSection());
+  emitDWARF5AccelTable(
+      Asm.get(), Table, CompUnits,
+      [&UniqueIdToCuMap](const DWARF5AccelTableStaticData &Entry) {
+        return UniqueIdToCuMap[Entry.getCUIndex()];
+      });
+}
+
+void DwarfStreamer::emitAppleNamespaces(
+    AccelTable<AppleAccelTableStaticOffsetData> &Table) {
+  Asm->OutStreamer->SwitchSection(MOFI->getDwarfAccelNamespaceSection());
+  auto *SectionBegin = Asm->createTempSymbol("namespac_begin");
+  Asm->OutStreamer->emitLabel(SectionBegin);
+  emitAppleAccelTable(Asm.get(), Table, "namespac", SectionBegin);
+}
+
+void DwarfStreamer::emitAppleNames(
+    AccelTable<AppleAccelTableStaticOffsetData> &Table) {
+  Asm->OutStreamer->SwitchSection(MOFI->getDwarfAccelNamesSection());
+  auto *SectionBegin = Asm->createTempSymbol("names_begin");
+  Asm->OutStreamer->emitLabel(SectionBegin);
+  emitAppleAccelTable(Asm.get(), Table, "names", SectionBegin);
+}
+
+void DwarfStreamer::emitAppleObjc(
+    AccelTable<AppleAccelTableStaticOffsetData> &Table) {
+  Asm->OutStreamer->SwitchSection(MOFI->getDwarfAccelObjCSection());
+  auto *SectionBegin = Asm->createTempSymbol("objc_begin");
+  Asm->OutStreamer->emitLabel(SectionBegin);
+  emitAppleAccelTable(Asm.get(), Table, "objc", SectionBegin);
+}
+
+void DwarfStreamer::emitAppleTypes(
+    AccelTable<AppleAccelTableStaticTypeData> &Table) {
+  Asm->OutStreamer->SwitchSection(MOFI->getDwarfAccelTypesSection());
+  auto *SectionBegin = Asm->createTempSymbol("types_begin");
+  Asm->OutStreamer->emitLabel(SectionBegin);
+  emitAppleAccelTable(Asm.get(), Table, "types", SectionBegin);
+}
+
+/// Emit the swift_ast section stored in \p Buffers.
+void DwarfStreamer::emitSwiftAST(StringRef Buffer) {
+  MCSection *SwiftASTSection = MOFI->getDwarfSwiftASTSection();
+  SwiftASTSection->setAlignment(Align(32));
+  MS->SwitchSection(SwiftASTSection);
+  MS->emitBytes(Buffer);
+}
+
+/// Emit the debug_range section contents for \p FuncRange by
+/// translating the original \p Entries. The debug_range section
+/// format is totally trivial, consisting just of pairs of address
+/// sized addresses describing the ranges.
+void DwarfStreamer::emitRangesEntries(
+    int64_t UnitPcOffset, uint64_t OrigLowPc,
+    const FunctionIntervals::const_iterator &FuncRange,
+    const std::vector<DWARFDebugRangeList::RangeListEntry> &Entries,
+    unsigned AddressSize) {
+  MS->SwitchSection(MC->getObjectFileInfo()->getDwarfRangesSection());
+
+  // Offset each range by the right amount.
+  int64_t PcOffset = Entries.empty() ? 0 : FuncRange.value() + UnitPcOffset;
+  for (const auto &Range : Entries) {
+    if (Range.isBaseAddressSelectionEntry(AddressSize)) {
+      warn("unsupported base address selection operation",
+           "emitting debug_ranges");
+      break;
+    }
+    // Do not emit empty ranges.
+    if (Range.StartAddress == Range.EndAddress)
+      continue;
+
+    // All range entries should lie in the function range.
+    if (!(Range.StartAddress + OrigLowPc >= FuncRange.start() &&
+          Range.EndAddress + OrigLowPc <= FuncRange.stop()))
+      warn("inconsistent range data.", "emitting debug_ranges");
+    MS->emitIntValue(Range.StartAddress + PcOffset, AddressSize);
+    MS->emitIntValue(Range.EndAddress + PcOffset, AddressSize);
+    RangesSectionSize += 2 * AddressSize;
+  }
+
+  // Add the terminator entry.
+  MS->emitIntValue(0, AddressSize);
+  MS->emitIntValue(0, AddressSize);
+  RangesSectionSize += 2 * AddressSize;
+}
+
+/// Emit the debug_aranges contribution of a unit and
+/// if \p DoDebugRanges is true the debug_range contents for a
+/// compile_unit level DW_AT_ranges attribute (Which are basically the
+/// same thing with a different base address).
+/// Just aggregate all the ranges gathered inside that unit.
+void DwarfStreamer::emitUnitRangesEntries(CompileUnit &Unit,
+                                          bool DoDebugRanges) {
+  unsigned AddressSize = Unit.getOrigUnit().getAddressByteSize();
+  // Gather the ranges in a vector, so that we can simplify them. The
+  // IntervalMap will have coalesced the non-linked ranges, but here
+  // we want to coalesce the linked addresses.
+  std::vector<std::pair<uint64_t, uint64_t>> Ranges;
+  const auto &FunctionRanges = Unit.getFunctionRanges();
+
+  for (auto Range = FunctionRanges.begin(), End = FunctionRanges.end();
+       Range != End; ++Range)
+    Ranges.push_back(std::make_pair(Range.start() + Range.value(),
+                                    Range.stop() + Range.value()));
+
+  // The object addresses where sorted, but again, the linked
+  // addresses might end up in a different order.
+  llvm::sort(Ranges);
+
+  if (!Ranges.empty()) {
+    MS->SwitchSection(MC->getObjectFileInfo()->getDwarfARangesSection());
+
+    MCSymbol *BeginLabel = Asm->createTempSymbol("Barange");
+    MCSymbol *EndLabel = Asm->createTempSymbol("Earange");
+
+    unsigned HeaderSize =
+        sizeof(int32_t) + // Size of contents (w/o this field
+        sizeof(int16_t) + // DWARF ARange version number
+        sizeof(int32_t) + // Offset of CU in the .debug_info section
+        sizeof(int8_t) +  // Pointer Size (in bytes)
+        sizeof(int8_t);   // Segment Size (in bytes)
+
+    unsigned TupleSize = AddressSize * 2;
+    unsigned Padding = offsetToAlignment(HeaderSize, Align(TupleSize));
+
+    Asm->emitLabelDifference(EndLabel, BeginLabel, 4); // Arange length
+    Asm->OutStreamer->emitLabel(BeginLabel);
+    Asm->emitInt16(dwarf::DW_ARANGES_VERSION); // Version number
+    Asm->emitInt32(Unit.getStartOffset(
+        CompileUnit::DebugTableKind::DebugInfo)); // Corresponding unit's offset
+    Asm->emitInt8(AddressSize);                   // Address size
+    Asm->emitInt8(0);                             // Segment size
+
+    Asm->OutStreamer->emitFill(Padding, 0x0);
+
+    for (auto Range = Ranges.begin(), End = Ranges.end(); Range != End;
+         ++Range) {
+      uint64_t RangeStart = Range->first;
+      MS->emitIntValue(RangeStart, AddressSize);
+      while ((Range + 1) != End && Range->second == (Range + 1)->first)
+        ++Range;
+      MS->emitIntValue(Range->second - RangeStart, AddressSize);
+    }
+
+    // Emit terminator
+    Asm->OutStreamer->emitIntValue(0, AddressSize);
+    Asm->OutStreamer->emitIntValue(0, AddressSize);
+    Asm->OutStreamer->emitLabel(EndLabel);
+  }
+
+  if (!DoDebugRanges)
+    return;
+
+  MS->SwitchSection(MC->getObjectFileInfo()->getDwarfRangesSection());
+  // Offset each range by the right amount.
+  int64_t PcOffset = -Unit.getLowPc();
+  // Emit coalesced ranges.
+  for (auto Range = Ranges.begin(), End = Ranges.end(); Range != End; ++Range) {
+    MS->emitIntValue(Range->first + PcOffset, AddressSize);
+    while (Range + 1 != End && Range->second == (Range + 1)->first)
+      ++Range;
+    MS->emitIntValue(Range->second + PcOffset, AddressSize);
+    RangesSectionSize += 2 * AddressSize;
+  }
+
+  // Add the terminator entry.
+  MS->emitIntValue(0, AddressSize);
+  MS->emitIntValue(0, AddressSize);
+  RangesSectionSize += 2 * AddressSize;
+}
+
+/// Emit location lists for \p Unit and update attributes to point to the new
+/// entries.
+void DwarfStreamer::emitLocationsForUnit(
+    CompileUnit &Unit, DWARFContext &Dwarf,
+    std::function<void(StringRef, SmallVectorImpl<uint8_t> &)> ProcessExpr) {
+  auto &Attributes = Unit.getLocationAttributes();
+
+  if (Attributes.empty())
+    return;
+
+  MS->SwitchSection(MC->getObjectFileInfo()->getDwarfLocSection());
+
+  unsigned AddressSize = Unit.getOrigUnit().getAddressByteSize();
+  uint64_t BaseAddressMarker = (AddressSize == 8)
+                                   ? std::numeric_limits<uint64_t>::max()
+                                   : std::numeric_limits<uint32_t>::max();
+  const DWARFSection &InputSec = Dwarf.getDWARFObj().getLocSection();
+  DataExtractor Data(InputSec.Data, Dwarf.isLittleEndian(), AddressSize);
+  DWARFUnit &OrigUnit = Unit.getOrigUnit();
+  auto OrigUnitDie = OrigUnit.getUnitDIE(false);
+  int64_t UnitPcOffset = 0;
+  if (auto OrigLowPc = dwarf::toAddress(OrigUnitDie.find(dwarf::DW_AT_low_pc)))
+    UnitPcOffset = int64_t(*OrigLowPc) - Unit.getLowPc();
+
+  SmallVector<uint8_t, 32> Buffer;
+  for (auto &Attr : Attributes) {
+    uint64_t Offset = Attr.CULocalOffsetValue;
+    Attr.CULocalOffsetValue = LocSectionSize;
+    // This is the quantity to add to the old location address to get
+    // the correct address for the new one.
+    int64_t LocPcOffset = Attr.PCValue + UnitPcOffset;
+    while (Data.isValidOffset(Offset)) {
+      uint64_t Low = Data.getUnsigned(&Offset, AddressSize);
+      uint64_t High = Data.getUnsigned(&Offset, AddressSize);
+      LocSectionSize += 2 * AddressSize;
+      // End of list entry.
+      if (Low == 0 && High == 0) {
+        Asm->OutStreamer->emitIntValue(0, AddressSize);
+        Asm->OutStreamer->emitIntValue(0, AddressSize);
+        break;
+      }
+      // Base address selection entry.
+      if (Low == BaseAddressMarker) {
+        Asm->OutStreamer->emitIntValue(BaseAddressMarker, AddressSize);
+        Asm->OutStreamer->emitIntValue(High + Attr.PCValue, AddressSize);
+        LocPcOffset = 0;
+        continue;
+      }
+      // Location list entry.
+      Asm->OutStreamer->emitIntValue(Low + LocPcOffset, AddressSize);
+      Asm->OutStreamer->emitIntValue(High + LocPcOffset, AddressSize);
+      uint64_t Length = Data.getU16(&Offset);
+      Asm->OutStreamer->emitIntValue(Length, 2);
+      // Copy the bytes into to the buffer, process them, emit them.
+      Buffer.reserve(Length);
+      Buffer.resize(0);
+      StringRef Input = InputSec.Data.substr(Offset, Length);
+      ProcessExpr(Input, Buffer);
+      Asm->OutStreamer->emitBytes(
+          StringRef((const char *)Buffer.data(), Length));
+      Offset += Length;
+      LocSectionSize += Length + 2;
+    }
+  }
+}
+
+void DwarfStreamer::emitLineTableForUnit(MCDwarfLineTableParams Params,
+                                         StringRef PrologueBytes,
+                                         unsigned MinInstLength,
+                                         std::vector<DWARFDebugLine::Row> &Rows,
+                                         unsigned PointerSize) {
+  // Switch to the section where the table will be emitted into.
+  MS->SwitchSection(MC->getObjectFileInfo()->getDwarfLineSection());
+  MCSymbol *LineStartSym = MC->createTempSymbol();
+  MCSymbol *LineEndSym = MC->createTempSymbol();
+
+  // The first 4 bytes is the total length of the information for this
+  // compilation unit (not including these 4 bytes for the length).
+  Asm->emitLabelDifference(LineEndSym, LineStartSym, 4);
+  Asm->OutStreamer->emitLabel(LineStartSym);
+  // Copy Prologue.
+  MS->emitBytes(PrologueBytes);
+  LineSectionSize += PrologueBytes.size() + 4;
+
+  SmallString<128> EncodingBuffer;
+  raw_svector_ostream EncodingOS(EncodingBuffer);
+
+  if (Rows.empty()) {
+    // We only have the dummy entry, dsymutil emits an entry with a 0
+    // address in that case.
+    MCDwarfLineAddr::Encode(*MC, Params, std::numeric_limits<int64_t>::max(), 0,
+                            EncodingOS);
+    MS->emitBytes(EncodingOS.str());
+    LineSectionSize += EncodingBuffer.size();
+    MS->emitLabel(LineEndSym);
+    return;
+  }
+
+  // Line table state machine fields
+  unsigned FileNum = 1;
+  unsigned LastLine = 1;
+  unsigned Column = 0;
+  unsigned IsStatement = 1;
+  unsigned Isa = 0;
+  uint64_t Address = -1ULL;
+
+  unsigned RowsSinceLastSequence = 0;
+
+  for (unsigned Idx = 0; Idx < Rows.size(); ++Idx) {
+    auto &Row = Rows[Idx];
+
+    int64_t AddressDelta;
+    if (Address == -1ULL) {
+      MS->emitIntValue(dwarf::DW_LNS_extended_op, 1);
+      MS->emitULEB128IntValue(PointerSize + 1);
+      MS->emitIntValue(dwarf::DW_LNE_set_address, 1);
+      MS->emitIntValue(Row.Address.Address, PointerSize);
+      LineSectionSize += 2 + PointerSize + getULEB128Size(PointerSize + 1);
+      AddressDelta = 0;
+    } else {
+      AddressDelta = (Row.Address.Address - Address) / MinInstLength;
+    }
+
+    // FIXME: code copied and transformed from MCDwarf.cpp::EmitDwarfLineTable.
+    // We should find a way to share this code, but the current compatibility
+    // requirement with classic dsymutil makes it hard. Revisit that once this
+    // requirement is dropped.
+
+    if (FileNum != Row.File) {
+      FileNum = Row.File;
+      MS->emitIntValue(dwarf::DW_LNS_set_file, 1);
+      MS->emitULEB128IntValue(FileNum);
+      LineSectionSize += 1 + getULEB128Size(FileNum);
+    }
+    if (Column != Row.Column) {
+      Column = Row.Column;
+      MS->emitIntValue(dwarf::DW_LNS_set_column, 1);
+      MS->emitULEB128IntValue(Column);
+      LineSectionSize += 1 + getULEB128Size(Column);
+    }
+
+    // FIXME: We should handle the discriminator here, but dsymutil doesn't
+    // consider it, thus ignore it for now.
+
+    if (Isa != Row.Isa) {
+      Isa = Row.Isa;
+      MS->emitIntValue(dwarf::DW_LNS_set_isa, 1);
+      MS->emitULEB128IntValue(Isa);
+      LineSectionSize += 1 + getULEB128Size(Isa);
+    }
+    if (IsStatement != Row.IsStmt) {
+      IsStatement = Row.IsStmt;
+      MS->emitIntValue(dwarf::DW_LNS_negate_stmt, 1);
+      LineSectionSize += 1;
+    }
+    if (Row.BasicBlock) {
+      MS->emitIntValue(dwarf::DW_LNS_set_basic_block, 1);
+      LineSectionSize += 1;
+    }
+
+    if (Row.PrologueEnd) {
+      MS->emitIntValue(dwarf::DW_LNS_set_prologue_end, 1);
+      LineSectionSize += 1;
+    }
+
+    if (Row.EpilogueBegin) {
+      MS->emitIntValue(dwarf::DW_LNS_set_epilogue_begin, 1);
+      LineSectionSize += 1;
+    }
+
+    int64_t LineDelta = int64_t(Row.Line) - LastLine;
+    if (!Row.EndSequence) {
+      MCDwarfLineAddr::Encode(*MC, Params, LineDelta, AddressDelta, EncodingOS);
+      MS->emitBytes(EncodingOS.str());
+      LineSectionSize += EncodingBuffer.size();
+      EncodingBuffer.resize(0);
+      Address = Row.Address.Address;
+      LastLine = Row.Line;
+      RowsSinceLastSequence++;
+    } else {
+      if (LineDelta) {
+        MS->emitIntValue(dwarf::DW_LNS_advance_line, 1);
+        MS->emitSLEB128IntValue(LineDelta);
+        LineSectionSize += 1 + getSLEB128Size(LineDelta);
+      }
+      if (AddressDelta) {
+        MS->emitIntValue(dwarf::DW_LNS_advance_pc, 1);
+        MS->emitULEB128IntValue(AddressDelta);
+        LineSectionSize += 1 + getULEB128Size(AddressDelta);
+      }
+      MCDwarfLineAddr::Encode(*MC, Params, std::numeric_limits<int64_t>::max(),
+                              0, EncodingOS);
+      MS->emitBytes(EncodingOS.str());
+      LineSectionSize += EncodingBuffer.size();
+      EncodingBuffer.resize(0);
+      Address = -1ULL;
+      LastLine = FileNum = IsStatement = 1;
+      RowsSinceLastSequence = Column = Isa = 0;
+    }
+  }
+
+  if (RowsSinceLastSequence) {
+    MCDwarfLineAddr::Encode(*MC, Params, std::numeric_limits<int64_t>::max(), 0,
+                            EncodingOS);
+    MS->emitBytes(EncodingOS.str());
+    LineSectionSize += EncodingBuffer.size();
+    EncodingBuffer.resize(0);
+  }
+
+  MS->emitLabel(LineEndSym);
+}
+
+/// Copy the debug_line over to the updated binary while unobfuscating the file
+/// names and directories.
+void DwarfStreamer::translateLineTable(DataExtractor Data, uint64_t Offset) {
+  MS->SwitchSection(MC->getObjectFileInfo()->getDwarfLineSection());
+  StringRef Contents = Data.getData();
+
+  // We have to deconstruct the line table header, because it contains to
+  // length fields that will need to be updated when we change the length of
+  // the files and directories in there.
+  unsigned UnitLength = Data.getU32(&Offset);
+  uint64_t UnitEnd = Offset + UnitLength;
+  MCSymbol *BeginLabel = MC->createTempSymbol();
+  MCSymbol *EndLabel = MC->createTempSymbol();
+  unsigned Version = Data.getU16(&Offset);
+
+  if (Version > 5) {
+    warn("Unsupported line table version: dropping contents and not "
+         "unobfsucating line table.");
+    return;
+  }
+
+  Asm->emitLabelDifference(EndLabel, BeginLabel, 4);
+  Asm->OutStreamer->emitLabel(BeginLabel);
+  Asm->emitInt16(Version);
+  LineSectionSize += 6;
+
+  MCSymbol *HeaderBeginLabel = MC->createTempSymbol();
+  MCSymbol *HeaderEndLabel = MC->createTempSymbol();
+  Asm->emitLabelDifference(HeaderEndLabel, HeaderBeginLabel, 4);
+  Asm->OutStreamer->emitLabel(HeaderBeginLabel);
+  Offset += 4;
+  LineSectionSize += 4;
+
+  uint64_t AfterHeaderLengthOffset = Offset;
+  // Skip to the directories.
+  Offset += (Version >= 4) ? 5 : 4;
+  unsigned OpcodeBase = Data.getU8(&Offset);
+  Offset += OpcodeBase - 1;
+  Asm->OutStreamer->emitBytes(Contents.slice(AfterHeaderLengthOffset, Offset));
+  LineSectionSize += Offset - AfterHeaderLengthOffset;
+
+  // Offset points to the first directory.
+  while (const char *Dir = Data.getCStr(&Offset)) {
+    if (Dir[0] == 0)
+      break;
+
+    StringRef Translated = Translator(Dir);
+    Asm->OutStreamer->emitBytes(Translated);
+    Asm->emitInt8(0);
+    LineSectionSize += Translated.size() + 1;
+  }
+  Asm->emitInt8(0);
+  LineSectionSize += 1;
+
+  while (const char *File = Data.getCStr(&Offset)) {
+    if (File[0] == 0)
+      break;
+
+    StringRef Translated = Translator(File);
+    Asm->OutStreamer->emitBytes(Translated);
+    Asm->emitInt8(0);
+    LineSectionSize += Translated.size() + 1;
+
+    uint64_t OffsetBeforeLEBs = Offset;
+    Asm->emitULEB128(Data.getULEB128(&Offset));
+    Asm->emitULEB128(Data.getULEB128(&Offset));
+    Asm->emitULEB128(Data.getULEB128(&Offset));
+    LineSectionSize += Offset - OffsetBeforeLEBs;
+  }
+  Asm->emitInt8(0);
+  LineSectionSize += 1;
+
+  Asm->OutStreamer->emitLabel(HeaderEndLabel);
+
+  // Copy the actual line table program over.
+  Asm->OutStreamer->emitBytes(Contents.slice(Offset, UnitEnd));
+  LineSectionSize += UnitEnd - Offset;
+
+  Asm->OutStreamer->emitLabel(EndLabel);
+  Offset = UnitEnd;
+}
+
+/// Emit the pubnames or pubtypes section contribution for \p
+/// Unit into \p Sec. The data is provided in \p Names.
+void DwarfStreamer::emitPubSectionForUnit(
+    MCSection *Sec, StringRef SecName, const CompileUnit &Unit,
+    const std::vector<CompileUnit::AccelInfo> &Names) {
+  if (Names.empty())
+    return;
+
+  // Start the dwarf pubnames section.
+  Asm->OutStreamer->SwitchSection(Sec);
+  MCSymbol *BeginLabel = Asm->createTempSymbol("pub" + SecName + "_begin");
+  MCSymbol *EndLabel = Asm->createTempSymbol("pub" + SecName + "_end");
+
+  bool HeaderEmitted = false;
+  // Emit the pubnames for this compilation unit.
+  for (const auto &Name : Names) {
+    if (Name.SkipPubSection)
+      continue;
+
+    if (!HeaderEmitted) {
+      // Emit the header.
+      Asm->emitLabelDifference(EndLabel, BeginLabel, 4); // Length
+      Asm->OutStreamer->emitLabel(BeginLabel);
+      Asm->emitInt16(dwarf::DW_PUBNAMES_VERSION); // Version
+      Asm->emitInt32(Unit.getStartOffset(
+          CompileUnit::DebugTableKind::DebugInfo)); // Unit offset
+      Asm->emitInt32(Unit.getUnitSize());           // Size
+      HeaderEmitted = true;
+    }
+    Asm->emitInt32(Name.OutOffset);
+
+    // Emit the string itself.
+    Asm->OutStreamer->emitBytes(Name.Name.getString());
+    // Emit a null terminator.
+    Asm->emitInt8(0);
+  }
+
+  if (!HeaderEmitted)
+    return;
+  Asm->emitInt32(0); // End marker.
+  Asm->OutStreamer->emitLabel(EndLabel);
+}
+
+/// Emit .debug_pubnames for \p Unit.
+void DwarfStreamer::emitPubNamesForUnit(const CompileUnit &Unit) {
+  emitPubSectionForUnit(MC->getObjectFileInfo()->getDwarfPubNamesSection(),
+                        "names", Unit, Unit.getPubnames());
+}
+
+/// Emit .debug_pubtypes for \p Unit.
+void DwarfStreamer::emitPubTypesForUnit(const CompileUnit &Unit) {
+  emitPubSectionForUnit(MC->getObjectFileInfo()->getDwarfPubTypesSection(),
+                        "types", Unit, Unit.getPubtypes());
+}
+
+/// Emit a CIE into the debug_frame section.
+void DwarfStreamer::emitCIE(StringRef CIEBytes) {
+  MS->SwitchSection(MC->getObjectFileInfo()->getDwarfFrameSection());
+
+  MS->emitBytes(CIEBytes);
+  FrameSectionSize += CIEBytes.size();
+}
+
+/// Emit a FDE into the debug_frame section. \p FDEBytes
+/// contains the FDE data without the length, CIE offset and address
+/// which will be replaced with the parameter values.
+void DwarfStreamer::emitFDE(uint32_t CIEOffset, uint32_t AddrSize,
+                            uint32_t Address, StringRef FDEBytes) {
+  MS->SwitchSection(MC->getObjectFileInfo()->getDwarfFrameSection());
+
+  MS->emitIntValue(FDEBytes.size() + 4 + AddrSize, 4);
+  MS->emitIntValue(CIEOffset, 4);
+  MS->emitIntValue(Address, AddrSize);
+  MS->emitBytes(FDEBytes);
+  FrameSectionSize += FDEBytes.size() + 8 + AddrSize;
+}
+
+} // end of namespace dwarflinker
+
+} // namespace llvm

llvm/test/tools/dsymutil/X86/basic-linking-x86.test

 RUN: cat %p/../Inputs/basic.macho.x86_64 > %t1
 RUN: dsymutil -f -oso-prepend-path=%p/.. %t1
 RUN: llvm-dwarfdump -a %t1.dwarf | FileCheck %s
 RUN: dsymutil -f -o %t2 -oso-prepend-path=%p/.. %p/../Inputs/basic.macho.x86_64
 RUN: llvm-dwarfdump -a %t2 | FileCheck %s
 RUN: dsymutil -f -o - -oso-prepend-path=%p/.. %p/../Inputs/basic.macho.x86_64 | llvm-dwarfdump -a - | FileCheck %s --check-prefixes=CHECK,BASIC
 RUN: dsymutil -f -o - -oso-prepend-path=%p/.. %p/../Inputs/basic-archive.macho.x86_64 | llvm-dwarfdump -a - | FileCheck %s --check-prefixes=CHECK,ARCHIVE
 RUN: dsymutil -dump-debug-map -oso-prepend-path=%p/.. %p/../Inputs/basic.macho.x86_64 | dsymutil -f -y -o - - | llvm-dwarfdump -a - | FileCheck %s --check-prefixes=CHECK,BASIC
 RUN: dsymutil -dump-debug-map -oso-prepend-path=%p/.. %p/../Inputs/basic-archive.macho.x86_64 | dsymutil -f -o - -y - | llvm-dwarfdump -a - | FileCheck %s --check-prefixes=CHECK,ARCHIVE
 
 CHECK: file format Mach-O 64-bit x86-64
 
 CHECK: debug_info contents
 
-CHECK:  Compile Unit:
+CHECK:  0x[[CU1_OFFSET:[0-9a-f]*]]: Compile Unit:
 
 CHECK:  DW_TAG_compile_unit
 CHECK:    DW_AT_producer ("Apple LLVM version 6.0 (clang-600.0.39) (based on LLVM 3.5svn)")
 CHECK:    DW_AT_language (DW_LANG_C99)
 CHECK:    DW_AT_name ("basic1.c")
 CHECK:    DW_AT_stmt_list (0x00000000)
 CHECK:    DW_AT_comp_dir ("/Inputs")
 CHECK:    DW_AT_low_pc (0x0000000100000ea0)
 CHECK:    DW_TAG_subprogram
 CHECK:    DW_AT_name ("main")
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
 CHECK:      DW_AT_decl_line (23)
 CHECK:      DW_AT_prototyped (0x01)
-CHECK:      DW_AT_type (0x00000063
+CHECK:      DW_AT_type (0x[[INT_OFFSET:[0-9a-f]*]]
 CHECK:      DW_AT_external (0x01)
 CHECK:      DW_AT_accessibility (DW_ACCESS_public)
 CHECK:      DW_AT_low_pc (0x0000000100000ea0)
 CHECK:      DW_AT_high_pc (0x0000000100000ec4)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_name ("argc")
 CHECK:        DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
 CHECK:        DW_AT_decl_line (23)
-CHECK:        DW_AT_type (0x00000063
+CHECK:        DW_AT_type (0x[[INT_OFFSET]]
 CHECK:        DW_AT_location (DW_OP_fbreg -8)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_name ("argv")
 CHECK:        DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
 CHECK:        DW_AT_decl_line (23)
-CHECK:        DW_AT_type (0x0000006a
+CHECK:        DW_AT_type (0x[[CONSTCHARPTR_PTR_OFFSET:[0-9a-f]*]]
 CHECK:        DW_AT_location (DW_OP_fbreg -16)
 CHECK:      NULL
 CHECK:    DW_TAG_base_type
 CHECK:      DW_AT_name ("int")
 CHECK:      DW_AT_encoding (DW_ATE_signed)
 CHECK:      DW_AT_byte_size (0x04)
 CHECK:    DW_TAG_pointer_type
-CHECK:      DW_AT_type (0x0000006f
+CHECK:      DW_AT_type (0x[[CONSTCHARPTR_OFFSET:[0-9a-f]*]]
 CHECK:    DW_TAG_pointer_type
-CHECK:      DW_AT_type (0x00000074
+CHECK:      DW_AT_type (0x[[CONSTCHAR_OFFSET:[0-9a-f]*]]
 CHECK:    DW_TAG_const_type
-CHECK:      DW_AT_type (0x00000079
+CHECK:      DW_AT_type (0x[[CHAR_OFFSET:[0-9a-f]*]]
 CHECK:    DW_TAG_base_type
 CHECK:      DW_AT_name ("char")
 CHECK:      DW_AT_encoding (DW_ATE_signed_char)
 CHECK:      DW_AT_byte_size (0x01)
 CHECK:    NULL
 
-CHECK:  Compile Unit:
+CHECK:  0x[[CU2_OFFSET:[0-9a-f]*]]: Compile Unit:
 
 CHECK:  DW_TAG_compile_unit
 CHECK:   DW_AT_producer ("Apple LLVM version 6.0 (clang-600.0.39) (based on LLVM 3.5svn)")
 CHECK:   DW_AT_name ("basic2.c")
 CHECK:   DW_AT_stmt_list (0x0000003f)
 CHECK:   DW_AT_comp_dir ("/Inputs")
 CHECK:   DW_AT_low_pc (0x0000000100000ed0)
 CHECK:    DW_TAG_base_type
 CHECK:      DW_AT_name ("int")
 CHECK:    DW_TAG_variable
 CHECK:      DW_AT_name ("private_int")
-CHECK:      DW_AT_type (0x000000a7
+CHECK:      DW_AT_type (0x[[PRIVATE_INT_OFFSET:[0-9a-f]*]]
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic2.c")
 BASIC:      DW_AT_location (DW_OP_addr 0x100001008)
 ARCHIVE:    DW_AT_location (DW_OP_addr 0x100001004)
 CHECK:    DW_TAG_variable
 CHECK:      DW_AT_name ("baz")
-CHECK:      DW_AT_type (0x000000a7
+CHECK:      DW_AT_type (0x[[PRIVATE_INT_OFFSET]]
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic2.c")
 CHECK:      DW_AT_location (DW_OP_addr 0x100001000)
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("foo")
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic2.c")
-CHECK:      DW_AT_type (0x000000a7
+CHECK:      DW_AT_type (0x[[PRIVATE_INT_OFFSET]]
 CHECK:      DW_AT_low_pc (0x0000000100000ed0)
 CHECK:      DW_AT_high_pc (0x0000000100000f19)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_name ("arg")
-CHECK:        DW_AT_type (0x000000a7
+CHECK:        DW_AT_type (0x[[PRIVATE_INT_OFFSET]]
 CHECK:        DW_AT_location (DW_OP_fbreg -4)
 CHECK:      NULL
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("inc")
-CHECK:      DW_AT_type (0x000000a7
+CHECK:      DW_AT_type (0x[[PRIVATE_INT_OFFSET]]
 CHECK:      DW_AT_low_pc (0x0000000100000f20)
 CHECK:      DW_AT_high_pc (0x0000000100000f37)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:    NULL
 
-CHECK:  Compile Unit:
+CHECK:  0x[[CU3_OFFSET:[0-9a-f]*]]: Compile Unit:
 
 CHECK:  DW_TAG_compile_unit
 CHECK:    DW_AT_producer ("Apple LLVM version 6.0 (clang-600.0.39) (based on LLVM 3.5svn)")
 CHECK:    DW_AT_name ("basic3.c")
 CHECK:    DW_AT_stmt_list (0x00000093)
 CHECK:    DW_AT_comp_dir ("/Inputs")
 CHECK:    DW_AT_low_pc (0x0000000100000f40)
 CHECK:    DW_TAG_variable
 CHECK:      DW_AT_name ("val")
-CHECK:      DW_AT_type (0x00000162
+CHECK:      DW_AT_type (0x[[VAL_OFFSET:[0-9a-f]*]]
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic3.c")
 BASIC:      DW_AT_location (DW_OP_addr 0x100001004)
 ARCHIVE:    DW_AT_location (DW_OP_addr 0x100001008)
 CHECK:    DW_TAG_volatile_type
-CHECK:      DW_AT_type (0x00000167
+CHECK:      DW_AT_type (0x[[VOLATILE_INT_OFFSET:[0-9a-f]*]]
 CHECK:    DW_TAG_base_type
 CHECK:      DW_AT_name ("int")
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("bar")
-CHECK:      DW_AT_type (0x00000167
+CHECK:      DW_AT_type (0x[[VOLATILE_INT_OFFSET]]
 CHECK:      DW_AT_low_pc (0x0000000100000f40)
 CHECK:      DW_AT_high_pc (0x0000000100000f84)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_name ("arg")
-CHECK:        DW_AT_type (0x00000167
+CHECK:        DW_AT_type (0x[[VOLATILE_INT_OFFSET]]
 CHECK:        DW_AT_location (DW_OP_fbreg -8)
 CHECK:      NULL
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("inc")
-CHECK:      DW_AT_type (0x00000167
+CHECK:      DW_AT_type (0x[[VOLATILE_INT_OFFSET]]
 CHECK:      DW_AT_low_pc (0x0000000100000f90)
 CHECK:      DW_AT_high_pc (0x0000000100000fa9)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 
 CHECK:    NULL
 
 CHECK-NOT: .debug_loc contents
 
 CHECK:.debug_aranges contents:
-CHECK-NEXT:Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x00000000, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT:Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU1_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT:[0x0000000100000ea0, 0x0000000100000ec4)
-CHECK-NEXT:Address Range Header: length = 0x0000003c, format = DWARF32, version = 0x0002, cu_offset = 0x00000081, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT:Address Range Header: length = 0x0000003c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU2_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT:[0x0000000100000ed0, 0x0000000100000f19)
 CHECK-NEXT:[0x0000000100000f20, 0x0000000100000f37)
-CHECK-NEXT:Address Range Header: length = 0x0000003c, format = DWARF32, version = 0x0002, cu_offset = 0x00000126, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT:Address Range Header: length = 0x0000003c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU3_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT:[0x0000000100000f40, 0x0000000100000f84)
 CHECK-NEXT:[0x0000000100000f90, 0x0000000100000fa9)
 
 CHECK: .debug_line contents:
 CHECK: file_names[  1]:
 CHECK-NEXT: name: "basic1.c"
 CHECK-NEXT: dir_index: 0
 CHECK: Address            Line   Column File   ISA Discriminator Flags

llvm/test/tools/dsymutil/X86/basic-lto-dw4-linking-x86.test

 RUN: dsymutil -f -o - -oso-prepend-path=%p/.. %p/../Inputs/basic-lto-dw4.macho.x86_64 | llvm-dwarfdump -a - | FileCheck %s
 
 CHECK: file format Mach-O 64-bit x86-64
 
 CHECK: debug_info contents
 
-CHECK:  Compile Unit: {{.*}} version = 0x0004
+CHECK:  0x[[CU1_OFFSET:[0-9a-f]*]]: Compile Unit: {{.*}} version = 0x0004
 CHECK:  DW_TAG_compile_unit
 CHECK:    DW_AT_producer ("clang version 3.7.0 ")
 CHECK:    DW_AT_language (DW_LANG_C99)
 CHECK:    DW_AT_name ("basic1.c")
 CHECK:    DW_AT_stmt_list (0x00000000)
 CHECK:    DW_AT_comp_dir ("/Inputs")
 CHECK:    DW_AT_low_pc (0x0000000100000f40)
 CHECK:    DW_AT_high_pc (0x0000000100000f4b)
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_low_pc (0x0000000100000f40)
 CHECK:      DW_AT_high_pc (0x0000000100000f4b)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_AT_name ("main")
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
 CHECK:      DW_AT_prototyped (true)
-CHECK:      DW_AT_type (0x00000000000000a1
+CHECK:      DW_AT_type (0x[[SUBPROGRAM_OFFSET:[0-9a-f]*]]
 CHECK:      DW_AT_external (true)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_location (DW_OP_reg5 RDI, DW_OP_piece 0x4)
 CHECK:        DW_AT_name ("argc")
 CHECK:        DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
-CHECK:        DW_AT_type (0x00000000000000a1
+CHECK:        DW_AT_type (0x[[SUBPROGRAM_OFFSET]]
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_location (DW_OP_reg4 RSI)
 CHECK:        DW_AT_name ("argv")
-CHECK:        DW_AT_type (0x00000060
+CHECK:        DW_AT_type (0x[[CONSTCHARPTR_PTR_OFFSET:[0-9a-f]*]]
 CHECK:      NULL
 CHECK:    DW_TAG_pointer_type
-CHECK:       DW_AT_type (0x00000065
+CHECK:       DW_AT_type (0x[[PTR_TYPE:[0-9a-f]*]]
 CHECK:    DW_TAG_pointer_type
 CHECK:    DW_TAG_const_type
 CHECK:    DW_TAG_base_type
 CHECK:      DW_AT_name ("char")
 CHECK:      DW_AT_encoding (DW_ATE_signed_char)
 CHECK:      DW_AT_byte_size (0x01)
 CHECK:    NULL
 
-CHECK:  Compile Unit:{{.*}} version = 0x0004
+CHECK:  0x[[CU2_OFFSET:[0-9a-f]*]]: Compile Unit:{{.*}} version = 0x0004
 
 CHECK:  DW_TAG_compile_unit
 CHECK:    DW_AT_producer ("clang version 3.7.0 ")
 CHECK:    DW_AT_language (DW_LANG_C99)
 CHECK:    DW_AT_name ("basic2.c")
 CHECK:    DW_AT_stmt_list (0x00000044)
 CHECK:    DW_AT_low_pc (0x0000000100000f50)
 CHECK:    DW_AT_high_pc (0x0000000100000f87)
 CHECK:    DW_TAG_base_type
 CHECK:      DW_AT_name ("int")
 CHECK:    DW_TAG_variable
 CHECK:      DW_AT_name ("baz")
 CHECK:      DW_AT_location (DW_OP_addr 0x100001000)
 CHECK:    DW_TAG_variable
 CHECK:      DW_AT_name ("private_int")
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic2.c")
 CHECK:      DW_AT_location (DW_OP_addr 0x100001008)
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("inc")
-CHECK:      DW_AT_type (0x000000a1
+CHECK:      DW_AT_type (0x[[INC_OFFSET:[0-9a-f]*]]
 CHECK:      DW_AT_inline (DW_INL_inlined)
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_low_pc (0x0000000100000f50)
 CHECK:      DW_AT_high_pc (0x0000000100000f87)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_AT_name ("foo")
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic2.c")
 CHECK:      DW_AT_prototyped (true)
-CHECK:      DW_AT_type (0x000000a1
+CHECK:      DW_AT_type (0x[[INC_OFFSET]]
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_location (0x00000000
 CHECK:          [0x0000000100000f50, 0x0000000100000f5c): DW_OP_reg5 RDI, DW_OP_piece 0x4)
 CHECK:        DW_AT_name ("arg")
-CHECK:        DW_AT_type (0x000000a1
+CHECK:        DW_AT_type (0x[[INC_OFFSET]]
 CHECK:      DW_TAG_inlined_subroutine
-CHECK:        DW_AT_abstract_origin (0x000000d2 "inc")
+CHECK:        DW_AT_abstract_origin (0x[[ABSTRACT_OFFSET:[0-9a-f]*]] "inc")
 CHECK:        DW_AT_low_pc (0x0000000100000f61)
 CHECK:        DW_AT_high_pc (0x0000000100000f70)
 CHECK:      NULL
 CHECK:    NULL
 
-CHECK:  Compile Unit: {{.*}} version = 0x0004
+CHECK:  0x[[CU3_OFFSET:[0-9a-f]*]]: Compile Unit: {{.*}} version = 0x0004
 
 CHECK:  DW_TAG_compile_unit
 CHECK:    DW_AT_producer ("clang version 3.7.0 ")
 CHECK:    DW_AT_name ("basic3.c")
 CHECK:    DW_AT_stmt_list (0x0000009a)
 CHECK:    DW_AT_low_pc (0x0000000100000f90)
 CHECK:    DW_AT_high_pc (0x0000000100000fb4)
 CHECK:    DW_TAG_variable

 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_AT_name ("bar")
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_location (0x00000025
 CHECK:          [0x0000000100000f90, 0x0000000100000f9f): DW_OP_reg5 RDI, DW_OP_piece 0x4
 CHECK:          [0x0000000100000fa9, 0x0000000100000fad): DW_OP_reg5 RDI, DW_OP_piece 0x4)
 CHECK:        DW_AT_name ("arg")
 CHECK:      DW_TAG_inlined_subroutine
-CHECK:        DW_AT_abstract_origin (0x0000015f "inc")
+CHECK:        DW_AT_abstract_origin (0x[[ABSTRACT2_OFFSET:[0-9a-f]*]] "inc")
 CHECK:        DW_AT_ranges (0x00000000
 CHECK:           [0x0000000100000f94, 0x0000000100000f9a)
 CHECK:           [0x0000000100000f9f, 0x0000000100000fa7))
 
 CHECK:      NULL
 CHECK:    NULL
 
 
 CHECK: .debug_loc contents:
 CHECK-NEXT: 0x00000000:
 CHECK-NEXT:              (0x0000000000000000, 0x000000000000000c): DW_OP_reg5 RDI, DW_OP_piece 0x4
 CHECK-NOT: :
 CHECK:      0x00000025:
 CHECK-NEXT:              (0x0000000000000000, 0x000000000000000f): DW_OP_reg5 RDI, DW_OP_piece 0x4
 CHECK-NEXT:              (0x0000000000000019, 0x000000000000001d): DW_OP_reg5 RDI, DW_OP_piece 0x4
 
 
 CHECK: .debug_aranges contents:
-CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x00000000, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU1_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT: [0x0000000100000f40,  0x0000000100000f4b)
-CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x00000077, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU2_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT: [0x0000000100000f50, 0x0000000100000f87)
-CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x0000011b, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU3_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT: [0x0000000100000f90, 0x0000000100000fb4)
 
 CHECK: .debug_line contents:
 CHECK: file_names[  1]:
 CHECK-NEXT: name: "basic1.c"
 CHECK-NEXT: dir_index: 0
 CHECK: Address            Line   Column File   ISA Discriminator Flags
 CHECK-NEXT: ------------------ ------ ------ ------ --- ------------- -------------

llvm/test/tools/dsymutil/X86/basic-lto-linking-x86.test

 RUN: dsymutil -f -o - -oso-prepend-path=%p/.. %p/../Inputs/basic-lto.macho.x86_64 | llvm-dwarfdump -a - | FileCheck %s
 RUN: dsymutil -oso-prepend-path=%p/.. -dump-debug-map %p/../Inputs/basic-lto.macho.x86_64 | dsymutil -f -o - -y - | llvm-dwarfdump -a - | FileCheck %s
 
 CHECK: file format Mach-O 64-bit x86-64
 
 CHECK: debug_info contents
 
-CHECK:  Compile Unit:
+CHECK:  0x[[CU1_OFFSET:[0-9a-f]*]]: Compile Unit:
 
 CHECK:  DW_TAG_compile_unit
 CHECK:    DW_AT_producer ("Apple LLVM version 6.0 (clang-600.0.39) (based on LLVM 3.5svn)")
 CHECK:    DW_AT_language (DW_LANG_C99)
 CHECK:    DW_AT_name ("basic1.c")
 CHECK:    DW_AT_stmt_list (0x00000000)
 CHECK:    DW_AT_comp_dir ("/Inputs")
 CHECK:    DW_AT_low_pc (0x0000000100000f40)
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("main")
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic1.c")
 CHECK:      DW_AT_decl_line (23)
 CHECK:      DW_AT_prototyped (0x01)
-CHECK:      DW_AT_type (0x00000063
+CHECK:      DW_AT_type (0x[[INT_OFFSET:[0-9a-f]*]]
 CHECK:      DW_AT_external (0x01)
 CHECK:      DW_AT_accessibility (DW_ACCESS_public)
 CHECK:      DW_AT_low_pc (0x0000000100000f40)
 CHECK:      DW_AT_high_pc (0x0000000100000f4b)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_name ("argc")
-CHECK:        DW_AT_type (0x00000063
+CHECK:        DW_AT_type (0x[[INT_OFFSET]]
 CHECK:        DW_AT_location (DW_OP_reg5 RDI, DW_OP_piece 0x4)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_name ("argv")
-CHECK:        DW_AT_type (0x0000006a
+CHECK:        DW_AT_type (0x[[CONSTCHARPTR_PTR_OFFSET:[0-9a-f]*]]
 CHECK:        DW_AT_location (DW_OP_reg4 RSI)
 CHECK:      NULL
 CHECK:    DW_TAG_base_type
 CHECK:      DW_AT_name ("int")
 CHECK:      DW_AT_encoding (DW_ATE_signed)
 CHECK:      DW_AT_byte_size (0x04)
 CHECK:    DW_TAG_pointer_type
-CHECK:      DW_AT_type (0x0000006f
+CHECK:      DW_AT_type (0x[[CONSTCHARPTR_OFFSET:[0-9a-f]*]]
 CHECK:    DW_TAG_pointer_type
-CHECK:      DW_AT_type (0x00000074
+CHECK:      DW_AT_type (0x[[CONSTCHAR_OFFSET:[0-9a-f]*]]
 CHECK:    DW_TAG_const_type
-CHECK:      DW_AT_type (0x00000079
+CHECK:      DW_AT_type (0x[[CHAR_OFFSET:[0-9a-f]*]]
 CHECK:    DW_TAG_base_type
 CHECK:      DW_AT_name ("char")
 CHECK:      DW_AT_encoding (DW_ATE_signed_char)
 CHECK:      DW_AT_byte_size (0x01)
 CHECK:    NULL
 
-CHECK:  Compile Unit:
+CHECK:  0x[[CU2_OFFSET:[0-9a-f]*]]: Compile Unit:
 
 CHECK:  DW_TAG_compile_unit
 CHECK:    DW_AT_producer ("Apple LLVM version 6.0 (clang-600.0.39) (based on LLVM 3.5svn)")
 CHECK:    DW_AT_name ("basic2.c")
 CHECK:    DW_AT_stmt_list (0x0000003e)
 CHECK:    DW_AT_comp_dir ("/Inputs")
 CHECK:   DW_AT_low_pc (0x0000000100000f50)
 CHECK:    DW_TAG_variable
 CHECK:      DW_AT_name ("private_int")
-CHECK:      DW_AT_type (0x0000000000000063
+CHECK:      DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic2.c")
 CHECK:      DW_AT_location (DW_OP_addr 0x100001008)
 CHECK:    DW_TAG_variable
 CHECK:      DW_AT_name ("baz")
-CHECK:      DW_AT_type (0x0000000000000063
+CHECK:      DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:      DW_AT_location (DW_OP_addr 0x100001000)
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("foo")
-CHECK:      DW_AT_type (0x0000000000000063
+CHECK:      DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:      DW_AT_low_pc (0x0000000100000f50)
 CHECK:      DW_AT_high_pc (0x0000000100000f89)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_name ("arg")
-CHECK:        DW_AT_type (0x0000000000000063
+CHECK:        DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:        DW_AT_location (0x00000000
 CHECK:          [0x0000000100000f50, 0x0000000100000f5e): DW_OP_reg5 RDI, DW_OP_piece 0x4)
 CHECK:[[INC1:0x[0-9a-f]*]]{{.*}}DW_TAG_inlined_subroutine
-CHECK:        DW_AT_abstract_origin (0x00000128 "inc")
+CHECK:        DW_AT_abstract_origin (0x[[INC_OFFSET:[0-9a-f]*]] "inc")
 CHECK:        DW_AT_low_pc (0x0000000100000f63)
 CHECK:        DW_AT_high_pc (0x0000000100000f72)
 CHECK:        DW_AT_call_line (20)
 CHECK:      NULL
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("inc")
-CHECK:      DW_AT_type (0x0000000000000063
+CHECK:      DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:      DW_AT_inline (DW_INL_inlined)
 CHECK:    NULL
 
-CHECK:  Compile Unit:
+CHECK:  0x[[CU3_OFFSET:[0-9a-f]*]]: Compile Unit:
 
 CHECK:  DW_TAG_compile_unit
 CHECK:    DW_AT_producer ("Apple LLVM version 6.0 (clang-600.0.39) (based on LLVM 3.5svn)")
 CHECK:    DW_AT_name ("basic3.c")
 CHECK:    DW_AT_stmt_list (0x0000007e)
 CHECK:    DW_AT_comp_dir ("/Inputs")
 CHECK:    DW_AT_low_pc (0x0000000100000f90)
 CHECK:    DW_TAG_variable
 CHECK:      DW_AT_name ("val")
-CHECK:      DW_AT_type (0x00000176
+CHECK:      DW_AT_type (0x[[VOLATILE_INT_OFFSET:[0-9a-f]*]]
 CHECK:      DW_AT_decl_file ("/Inputs{{[/\\]}}basic3.c")
 CHECK:      DW_AT_location (DW_OP_addr 0x100001004)
 CHECK:    DW_TAG_volatile_type
-CHECK:      DW_AT_type (0x0000000000000063
+CHECK:      DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("bar")
-CHECK:      DW_AT_type (0x0000000000000063
+CHECK:      DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:      DW_AT_low_pc (0x0000000100000f90)
 CHECK:      DW_AT_high_pc (0x0000000100000fb4)
 CHECK:      DW_AT_frame_base (DW_OP_reg6 RBP)
 CHECK:      DW_TAG_formal_parameter
 CHECK:        DW_AT_name ("arg")
-CHECK:        DW_AT_type (0x0000000000000063
+CHECK:        DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:        DW_AT_location (0x00000025
 CHECK:          [0x0000000100000f90, 0x0000000100000f9f): DW_OP_reg5 RDI, DW_OP_piece 0x4
 CHECK:          [0x0000000100000fa9, 0x0000000100000fad): DW_OP_reg5 RDI, DW_OP_piece 0x4)
 CHECK:      DW_TAG_lexical_block
 CHECK:        DW_AT_low_pc (0x0000000100000f94)
 CHECK:        DW_AT_high_pc (0x0000000100000fa7)
 CHECK:[[INC2:0x[0-9a-f]*]]{{.*}}DW_TAG_inlined_subroutine
-CHECK:          DW_AT_abstract_origin (0x000001d4 "inc")
+CHECK:          DW_AT_abstract_origin (0x[[INC_OFFSET:[0-9a-f]*]] "inc")
 CHECK:          DW_AT_ranges (0x00000000
 CHECK:             [0x0000000100000f94, 0x0000000100000f9a)
 CHECK:             [0x0000000100000f9f, 0x0000000100000fa7))
 CHECK:        NULL
 CHECK:      NULL
 CHECK:    DW_TAG_subprogram
 CHECK:      DW_AT_name ("inc")
-CHECK:      DW_AT_type (0x0000000000000063
+CHECK:      DW_AT_type (0x{{0*}}[[INT_OFFSET]]
 CHECK:    NULL
 
 CHECK: .debug_loc contents:
 CHECK-NEXT: 0x00000000:
 CHECK-NEXT:              (0x0000000000000000, 0x000000000000000e): DW_OP_reg5 RDI, DW_OP_piece 0x4
 CHECK-NOT: :
 CHECK:      0x00000025:
 CHECK-NEXT:              (0x0000000000000000, 0x000000000000000f): DW_OP_reg5 RDI, DW_OP_piece 0x4
 CHECK-NEXT:              (0x0000000000000019, 0x000000000000001d): DW_OP_reg5 RDI, DW_OP_piece 0x4
 
 CHECK: .debug_aranges contents:
-CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x00000000, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU1_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT: [0x0000000100000f40, 0x0000000100000f4b)
-CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x00000081, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU2_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT: [0x0000000100000f50, 0x0000000100000f89)
-CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x0000013a, addr_size = 0x08, seg_size = 0x00
+CHECK-NEXT: Address Range Header: length = 0x0000002c, format = DWARF32, version = 0x0002, cu_offset = 0x[[CU3_OFFSET]], addr_size = 0x08, seg_size = 0x00
 CHECK-NEXT: [0x0000000100000f90, 0x0000000100000fb4)
 
 
 CHECK: .debug_line contents
 CHECK: file_names[  1]:
 CHECK-NEXT: name: "basic1.c"
 CHECK-NEXT: dir_index: 0
 CHECK: 0x0000000100000f40     23      0      1   0             0  is_stmt

 CHECK-NEXT: length = 0x0000001e, format = DWARF32, version = 0x0002, unit_offset = 0x0000013a, unit_size = 0x000000ac
 CHECK-NEXT: Offset     Name
 CHECK-NEXT: 0x00000026 "val"
 CHECK-NEXT: 0x00000045 "bar"
 
 CHECK: .debug_pubtypes contents:
 CHECK-NEXT: length = 0x0000001f, format = DWARF32, version = 0x0002, unit_offset = 0x00000000, unit_size = 0x00000081
 CHECK-NEXT: Offset     Name
-CHECK-NEXT: 0x00000063 "int"
+CHECK-NEXT: 0x[[INT_OFFSET]] "int"
 CHECK-NEXT: 0x00000079 "char"
 
 CHECK: .apple_names contents:
 CHECK-NEXT: Header {
 CHECK-NEXT:   Magic: 0x48415348
 CHECK-NEXT:   Version: 0x1
 CHECK-NEXT:   Hash function: 0x0
 CHECK-NEXT:   Bucket count: 7
 CHECK-NEXT:   Hashes count: 7
 CHECK-NEXT:   HeaderData length: 12
 CHECK-NEXT: }
 CHECK-NEXT: DIE offset base: 0
 CHECK-NEXT: Number of atoms: 1
 CHECK-NEXT: Atoms [
 CHECK-NEXT:   Atom 0 {
 CHECK-NEXT:     Type: DW_ATOM_die_offset
 CHECK-NEXT:     Form: DW_FORM_data4
 CHECK-NEXT:   }
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 0 [
 CHECK-NEXT:   Hash 0xb8860c2 [
 CHECK-NEXT:     Name@0x74 {
-CHECK-NEXT:       String: 0x0000007e "baz"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "baz"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x000000c0
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT:   Hash 0xb88801f [
 CHECK-NEXT:     Name@0x84 {
-CHECK-NEXT:       String: 0x0000008a "inc"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "inc"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x00000110
 CHECK-NEXT:       ]
 CHECK-NEXT:       Data 1 [
 CHECK-NEXT:         Atom[0]: 0x000001c7
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 1 [
 CHECK-NEXT:   EMPTY
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 2 [
 CHECK-NEXT:   Hash 0xfed12c6a [
 CHECK-NEXT:     Name@0x98 {
-CHECK-NEXT:       String: 0x00000072 "private_int"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "private_int"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x000000a7
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 3 [
 CHECK-NEXT:   Hash 0xb88b5c8 [
 CHECK-NEXT:     Name@0xa8 {
-CHECK-NEXT:       String: 0x00000097 "val"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "val"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x00000160
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT:   Hash 0x7c9a7f6a [
 CHECK-NEXT:     Name@0xb8 {
-CHECK-NEXT:       String: 0x00000051 "main"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "main"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x00000026
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 4 [
 CHECK-NEXT:   EMPTY
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 5 [
 CHECK-NEXT:   Hash 0xb887389 [
 CHECK-NEXT:     Name@0xc8 {
-CHECK-NEXT:       String: 0x00000082 "foo"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "foo"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x000000d9
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 6 [
 CHECK-NEXT:   Hash 0xb8860ba [
 CHECK-NEXT:     Name@0xd8 {
-CHECK-NEXT:       String: 0x0000009b "bar"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "bar"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x0000017f
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT: ]
 
 CHECK: apple_types contents:

 CHECK-NEXT:   Atom 3 {
 CHECK-NEXT:     Type: DW_ATOM_qual_name_hash
 CHECK-NEXT:     Form: DW_FORM_data4
 CHECK-NEXT:   }
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 0 [
 CHECK-NEXT:   Hash 0xb888030 [
 CHECK-NEXT:     Name@0x44 {
-CHECK-NEXT:       String: 0x00000060 "int"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "int"
 CHECK-NEXT:       Data 0 [
-CHECK-NEXT:         Atom[0]: 0x00000063
+CHECK-NEXT:         Atom[0]: 0x[[INT_OFFSET]]
 CHECK-NEXT:         Atom[1]: 0x0024
 CHECK-NEXT:         Atom[2]: 0x00
 CHECK-NEXT:         Atom[3]: 0x0c3a28a4
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 1 [
 CHECK-NEXT:   Hash 0x7c952063 [
 CHECK-NEXT:     Name@0x5b {
-CHECK-NEXT:       String: 0x00000064 "char"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "char"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x00000079
 CHECK-NEXT:         Atom[1]: 0x0024
 CHECK-NEXT:         Atom[2]: 0x00
 CHECK-NEXT:         Atom[3]: 0x937bd757
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]

llvm/test/tools/dsymutil/X86/objc.test

 RUN: dsymutil -f -oso-prepend-path=%p/.. %p/../Inputs/objc.macho.x86_64 -o - \
 RUN:   | llvm-dwarfdump -apple-types -apple-objc - | FileCheck %s
 
 CHECK: .apple_types contents:
-CHECK: String: 0x00000066 "A"
+CHECK: String: 0x{{[0-9a-f]*}} "A"
 CHECK-NEXT: Data 0 [
 CHECK-NEXT:   Atom[0]: 0x0000012d
 CHECK-NEXT:   Atom[1]: 0x0013 (DW_TAG_structure_type)
 CHECK-NEXT:   Atom[2]: 0x02
 CHECK-NEXT:   Atom[3]: 0x0b87b15a
 CHECK-NEXT: ]
 
 CHECK: .apple_objc contents:

 CHECK-NEXT:   Atom 0 {
 CHECK-NEXT:     Type: DW_ATOM_die_offset
 CHECK-NEXT:     Form: DW_FORM_data4
 CHECK-NEXT:   }
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 0 [
 CHECK-NEXT:   Hash 0x2b5e6 [
 CHECK-NEXT:     Name@0x38 {
-CHECK-NEXT:       String: 0x00000066 "A"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "A"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x00000027
 CHECK-NEXT:       ]
 CHECK-NEXT:       Data 1 [
 CHECK-NEXT:         Atom[0]: 0x0000007a
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT: ]
 CHECK-NEXT: Bucket 1 [
 CHECK-NEXT:   Hash 0x3fa0f4b5 [
 CHECK-NEXT:     Name@0x4c {
-CHECK-NEXT:       String: 0x0000009d "A(Category)"
+CHECK-NEXT:       String: 0x{{[0-9a-f]*}} "A(Category)"
 CHECK-NEXT:       Data 0 [
 CHECK-NEXT:         Atom[0]: 0x0000007a
 CHECK-NEXT:       ]
 CHECK-NEXT:     }
 CHECK-NEXT:   ]
 CHECK-NEXT: ]

llvm/test/tools/dsymutil/X86/update-one-CU.test

 RUN: dsymutil -oso-prepend-path=%p/.. %p/../Inputs/objc.macho.x86_64 -o %t.dSYM
 RUN: dsymutil -update %t.dSYM
 RUN: llvm-dwarfdump -apple-types -apple-objc %t.dSYM | FileCheck %s
 
 CHECK: .apple_types contents:
 CHECK: Hash 0x2b5e6 [
 CHECK-NEXT:    Name@0x145 {
-CHECK-NEXT:      String: 0x00000066 "A"
+CHECK-NEXT:      String: 0x{{[0-9a-f]*}} "A"
 CHECK-NEXT:      Data 0 [
 CHECK-NEXT:        Atom[0]: 0x0000012d
 CHECK-NEXT:        Atom[1]: 0x0013
 CHECK-NEXT:        Atom[2]: 0x02
 CHECK-NEXT:        Atom[3]: 0x0b87b15a
 CHECK-NEXT:      ]
 CHECK-NEXT:    }
 CHECK-NEXT:  ]
 
 CHECK: .apple_objc contents:
 CHECK: Hash 0x2b5e6
 CHECK-NEXT: Name@0x38 {
-CHECK-NEXT:   String: 0x00000066 "A"
+CHECK-NEXT:   String: 0x{{[0-9a-f]*}} "A"
 CHECK-NEXT:   Data 0 [
 CHECK-NEXT:     Atom[0]: 0x00000027
 CHECK-NEXT:   ]
 CHECK-NEXT:   Data 1 [
 CHECK-NEXT:     Atom[0]: 0x0000007a
 CHECK-NEXT:   ]
 CHECK-NEXT: }
 CHECK: Hash 0x3fa0f4b5
 CHECK-NEXT: Name@0x4c {
-CHECK-NEXT:   String: 0x0000009d "A(Category)"
+CHECK-NEXT:   String: 0x{{[0-9a-f]*}} "A(Category)"
 CHECK-NEXT:   Data 0 [
 CHECK-NEXT:     Atom[0]: 0x0000007a
 CHECK-NEXT:   ]
 CHECK-NEXT: }

llvm/tools/dsymutil/BinaryHolder.h

public:
   private:
     std::vector<std::unique_ptr<object::Archive>> Archives;
     DenseMap<KeyTy, ObjectEntry> MemberCache;
     std::mutex MemberCacheMutex;
   };
 
   Expected<const ObjectEntry &>
   getObjectEntry(StringRef Filename, TimestampTy Timestamp = TimestampTy());
+  void eraseObjectEntry(StringRef Filename,
+                        TimestampTy Timestamp = TimestampTy());
 
   void clear();
 
 private:
   /// Cache of static archives. Objects that are part of a static archive are
   /// stored under this object, rather than in the map below.
   StringMap<ArchiveEntry> ArchiveCache;
   std::mutex ArchiveCacheMutex;

llvm/tools/dsymutil/BinaryHolder.cpp

if (Err) {
       ObjectCache.erase(Filename);
       return std::move(Err);
     }
   }
 
   return ObjectCache[Filename];
 }
 
+void BinaryHolder::eraseObjectEntry(StringRef Filename, TimestampTy Timestamp) {
+  if (Verbose)
+    WithColor::note() << "erasing '" << Filename << "' from cache\n";
+
+  if (isArchive(Filename)) {
+    StringRef ArchiveFilename = getArchiveAndObjectName(Filename).first;
+    std::lock_guard<std::mutex> Lock(ArchiveCacheMutex);
+    ArchiveCache.erase(ArchiveFilename);
+    return;
+  }
+
+  std::lock_guard<std::mutex> Lock(ObjectCacheMutex);
+  ObjectCache.erase(Filename);
+}
+
 void BinaryHolder::clear() {
   std::lock_guard<std::mutex> ArchiveLock(ArchiveCacheMutex);
   std::lock_guard<std::mutex> ObjectLock(ObjectCacheMutex);
   ArchiveCache.clear();
   ObjectCache.clear();
 }
 
 } // namespace dsymutil
 } // namespace llvm

llvm/tools/dsymutil/CMakeLists.txt

 set(LLVM_TARGET_DEFINITIONS Options.td)
 tablegen(LLVM Options.inc -gen-opt-parser-defs)
 add_public_tablegen_target(DsymutilTableGen)
 
 set(LLVM_LINK_COMPONENTS
   AllTargetsCodeGens
   AllTargetsDescs
   AllTargetsInfos
   AsmPrinter
   DebugInfoDWARF
   DWARFLinker
+  DWARFLinkerNext
   MC
   Object
   CodeGen
   Option
   Remarks
   Support
   Target
   )

llvm/tools/dsymutil/DwarfLinkerForBinary.h

 
 #include "BinaryHolder.h"
 #include "DebugMap.h"
 #include "LinkUtils.h"
 #include "llvm/DWARFLinker/DWARFLinker.h"
 #include "llvm/DWARFLinker/DWARFLinkerCompileUnit.h"
 #include "llvm/DWARFLinker/DWARFLinkerDeclContext.h"
 #include "llvm/DWARFLinker/DWARFStreamer.h"
+#include "llvm/DWARFLinkerNext/DWARFLinker.h"
+#include "llvm/DWARFLinkerNext/DWARFLinkerCompileUnit.h"
+#include "llvm/DWARFLinkerNext/DWARFLinkerDeclContext.h"
+#include "llvm/DWARFLinkerNext/DWARFStreamer.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/Remarks/RemarkFormat.h"
 #include "llvm/Remarks/RemarkLinker.h"
 
 namespace llvm {
 namespace dsymutil {
 
 /// The core of the Dsymutil Dwarf linking logic.
enum TraversalFlags {
     TF_InFunctionScope = 1 << 1, ///< Current scope is a function scope.
     TF_DependencyWalk = 1 << 2,  ///< Walking the dependencies of a kept DIE.
     TF_ParentWalk = 1 << 3,      ///< Walking up the parents of a kept DIE.
     TF_ODR = 1 << 4,             ///< Use the ODR while keeping dependents.
     TF_SkipPC = 1 << 5,          ///< Skip all location attributes.
   };
 
 private:
+  bool linkImpl(const DebugMap &);
+  bool newLinkImpl(const DebugMap &);
 
   /// Keeps track of relocations.
-  class AddressManager : public AddressesMap {
+  class AddressManager : public virtual AddressesMap,
+                         public virtual dwarflinker::AddressesMap {
     struct ValidReloc {
       uint64_t Offset;
       uint32_t Size;
       uint64_t Addend;
       const DebugMapObject::DebugMapEntry *Mapping;
 
       ValidReloc(uint64_t Offset, uint32_t Size, uint64_t Addend,
                  const DebugMapObject::DebugMapEntry *Mapping)
class AddressManager : public virtual AddressesMap,
     /// The valid relocations for the current DebugMapObject.
     /// This vector is sorted by relocation offset.
     /// {
     std::vector<ValidReloc> ValidDebugInfoRelocs;
     std::vector<ValidReloc> ValidDebugAddrRelocs;
     /// }
 
     RangesTy AddressRanges;
+    dwarflinker::RangesTy AddressRangesForNewLibrary;
 
     StringRef SrcFileName;
 
     /// Returns list of valid relocations from \p Relocs,
     /// between \p StartOffset and \p NextOffset.
     ///
     /// \returns true if any relocation is found.
     std::vector<ValidReloc>
     getRelocations(const std::vector<ValidReloc> &Relocs, uint64_t StartPos,
                    uint64_t EndPos);
 
     /// Resolve specified relocation \p Reloc.
     ///
     /// \returns resolved value.
     uint64_t relocate(const ValidReloc &Reloc) const;
 
     /// Fill \p Info with address information for the specified \p Reloc.
     void fillDieInfo(const ValidReloc &Reloc, CompileUnit::DIEInfo &Info);
+    void fillDieInfo(const ValidReloc &Reloc,
+                     dwarflinker::CompileUnit::DIEInfo &Info);
 
     /// Print contents of debug map entry for the specified \p Reloc.
     void printReloc(const ValidReloc &Reloc);
 
   public:
-    AddressManager(DwarfLinkerForBinary &Linker, const object::ObjectFile &Obj,
-                   const DebugMapObject &DMO)
+    AddressManager(bool isNewDwarfLinkerLibrary, DwarfLinkerForBinary &Linker,
+                   const object::ObjectFile &Obj, const DebugMapObject &DMO)
         : Linker(Linker), SrcFileName(DMO.getObjectFilename()) {
       findValidRelocsInDebugSections(Obj, DMO);
 
       // Iterate over the debug map entries and put all the ones that are
       // functions (because they have a size) into the Ranges map. This map is
       // very similar to the FunctionRanges that are stored in each unit, with 2
       // notable differences:
       //
       //  1. Obviously this one is global, while the other ones are per-unit.
       //
       //  2. This one contains not only the functions described in the DIE
       //     tree, but also the ones that are only in the debug map.
       //
       // The latter information is required to reproduce dsymutil's logic while
       // linking line tables. The cases where this information matters look like
       // bugs that need to be investigated, but for now we need to reproduce
       // dsymutil's behavior.
       // FIXME: Once we understood exactly if that information is needed,
       // maybe totally remove this (or try to use it to do a real
       // -gline-tables-only on Darwin.
+      if (isNewDwarfLinkerLibrary) {
         for (const auto &Entry : DMO.symbols()) {
           const auto &Mapping = Entry.getValue();
           if (Mapping.Size && Mapping.ObjectAddress)
             AddressRanges[*Mapping.ObjectAddress] = ObjFileAddressRange(
                 *Mapping.ObjectAddress + Mapping.Size,
                 int64_t(Mapping.BinaryAddress) - *Mapping.ObjectAddress);
         }
+      } else {
+        for (const auto &Entry : DMO.symbols()) {
+          const auto &Mapping = Entry.getValue();
+          if (Mapping.Size && Mapping.ObjectAddress)
+            AddressRangesForNewLibrary[*Mapping.ObjectAddress] =
+                dwarflinker::ObjFileAddressRange(
+                    *Mapping.ObjectAddress + Mapping.Size,
+                    int64_t(Mapping.BinaryAddress) - *Mapping.ObjectAddress);
+        }
+      }
     }
     virtual ~AddressManager() override { clear(); }
 
     virtual bool areRelocationsResolved() const override { return true; }
 
     bool hasValidRelocs() override {
       return !ValidDebugInfoRelocs.empty() || !ValidDebugAddrRelocs.empty();
     }
public:
 
     /// Checks that there is a relocation in the \p Relocs array against a
     /// debug map entry between \p StartOffset and \p NextOffset.
     ///
     /// \returns true and sets Info.InDebugMap if it is the case.
     bool hasValidRelocationAt(const std::vector<ValidReloc> &Relocs,
                               uint64_t StartOffset, uint64_t EndOffset,
                               CompileUnit::DIEInfo &Info);
+    bool hasValidRelocationAt(const std::vector<ValidReloc> &Relocs,
+                              uint64_t StartOffset, uint64_t EndOffset,
+                              dwarflinker::CompileUnit::DIEInfo &Info);
 
     bool hasLiveMemoryLocation(const DWARFDie &DIE,
                                CompileUnit::DIEInfo &Info) override;
     bool hasLiveAddressRange(const DWARFDie &DIE,
                              CompileUnit::DIEInfo &Info) override;
 
+    bool
+    hasLiveMemoryLocation(const DWARFDie &DIE,
+                          dwarflinker::CompileUnit::DIEInfo &Info) override;
+    bool hasLiveAddressRange(const DWARFDie &DIE,
+                             dwarflinker::CompileUnit::DIEInfo &Info) override;
+
     bool applyValidRelocs(MutableArrayRef<char> Data, uint64_t BaseOffset,
                           bool IsLittleEndian) override;
 
     llvm::Expected<uint64_t> relocateIndexedAddr(uint64_t StartOffset,
                                                  uint64_t EndOffset) override;
 
-    RangesTy &getValidAddressRanges() override { return AddressRanges; }
+    void getValidAddressRanges(RangesTy *&Ranges) override {
+      Ranges = &AddressRanges;
+    }
+    void getValidAddressRanges(dwarflinker::RangesTy *&Ranges) override {
+      Ranges = &AddressRangesForNewLibrary;
+    }
 
     void clear() override {
       AddressRanges.clear();
+      AddressRangesForNewLibrary.clear();
       ValidDebugInfoRelocs.clear();
       ValidDebugAddrRelocs.clear();
     }
   };
 
 private:
   /// \defgroup Helpers Various helper methods.
   ///
   /// @{
-  bool createStreamer(const Triple &TheTriple, raw_fd_ostream &OutFile);
+  template <class DS, class OFT>
+  bool createStreamer(const Triple &TheTriple, raw_fd_ostream &OutFile,
+                      std::unique_ptr<DS> &Streamer, OFT OutFileType);
 
   /// Attempt to load a debug object from disk.
   ErrorOr<const object::ObjectFile &> loadObject(const DebugMapObject &Obj,
                                                  const Triple &triple);
-  ErrorOr<DWARFFile &> loadObject(const DebugMapObject &Obj,
-                                  const DebugMap &DebugMap,
-                                  remarks::RemarkLinker &RL);
+
+  template <class DF>
+  ErrorOr<DF &> loadObject(const DebugMapObject &Obj, const DebugMap &DebugMap,
+                           remarks::RemarkLinker &RL,
+                           std::vector<std::unique_ptr<DF>> &ObjectsForLinking,
+                           bool isNewDwarfLinkerLibrary);
 
   raw_fd_ostream &OutFile;
   BinaryHolder &BinHolder;
   LinkOptions Options;
-  std::unique_ptr<DwarfStreamer> Streamer;
-  std::vector<std::unique_ptr<DWARFFile>> ObjectsForLinking;
-  std::vector<std::unique_ptr<DWARFContext>> ContextForLinking;
-  std::vector<std::unique_ptr<AddressManager>> AddressMapForLinking;
   std::vector<std::string> EmptyWarnings;
 
   /// A list of all .swiftinterface files referenced by the debug
   /// info, mapping Module name to path on disk. The entries need to
   /// be uniqued and sorted and there are only few entries expected
   /// per compile unit, which is why this is a std::map.
   std::map<std::string, std::string> ParseableSwiftInterfaces;
 
   bool ModuleCacheHintDisplayed = false;
   bool ArchiveHintDisplayed = false;
+
+  std::mutex ObjLoaderMutex;
 };
 
 } // end namespace dsymutil
 } // end namespace llvm
 
 #endif // LLVM_TOOLS_DSYMUTIL_DWARFLINKER_H

llvm/tools/dsymutil/DwarfLinkerForBinary.cpp

 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/BinaryFormat/MachO.h"
 #include "llvm/CodeGen/AccelTable.h"
 #include "llvm/CodeGen/AsmPrinter.h"
 #include "llvm/CodeGen/DIE.h"
 #include "llvm/CodeGen/NonRelocatableStringpool.h"
 #include "llvm/Config/config.h"
 #include "llvm/DWARFLinker/DWARFLinkerDeclContext.h"
+#include "llvm/DWARFLinkerNext/DWARFLinkerDeclContext.h"
 #include "llvm/DebugInfo/DIContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFAbbreviationDeclaration.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/DebugInfo/DWARF/DWARFDataExtractor.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugLine.h"
 #include "llvm/DebugInfo/DWARF/DWARFDebugRangeList.h"
 #include "llvm/DebugInfo/DWARF/DWARFDie.h"
 #include "llvm/DebugInfo/DWARF/DWARFFormValue.h"
 #include "llvm/DebugInfo/DWARF/DWARFSection.h"
 #include "llvm/DebugInfo/DWARF/DWARFUnit.h"
 #include "llvm/MC/MCAsmBackend.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCCodeEmitter.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCDwarf.h"
 #include "llvm/MC/MCInstrInfo.h"
 #include "llvm/MC/MCObjectFileInfo.h"
 #include "llvm/MC/MCObjectWriter.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/MC/MCSection.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSubtargetInfo.h"
 #include "llvm/MC/MCTargetOptions.h"
+#include "llvm/MC/MCTargetOptionsCommandFlags.h"
 #include "llvm/Object/MachO.h"
 #include "llvm/Object/ObjectFile.h"
 #include "llvm/Object/SymbolicFile.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/DJB.h"
 #include "llvm/Support/DataExtractor.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/LEB128.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include "llvm/Support/WithColor.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
-#include "llvm/MC/MCTargetOptionsCommandFlags.h"
 #include <algorithm>
 #include <cassert>
 #include <cinttypes>
 #include <climits>
 #include <cstdint>
 #include <cstdlib>
 #include <cstring>
 #include <limits>
void DwarfLinkerForBinary::reportWarning(const Twine &Warning,
   DIDumpOptions DumpOpts;
   DumpOpts.ChildRecurseDepth = 0;
   DumpOpts.Verbose = Options.Verbose;
 
   WithColor::note() << "    in DIE:\n";
   DIE->dump(errs(), 6 /* Indent */, DumpOpts);
 }
 
+template <class DS, class OFT>
 bool DwarfLinkerForBinary::createStreamer(const Triple &TheTriple,
-                                          raw_fd_ostream &OutFile) {
+                                          raw_fd_ostream &OutFile,
+                                          std::unique_ptr<DS> &Streamer,
+                                          OFT OutFileType) {
   if (Options.NoOutput)
     return true;
 
-  Streamer = std::make_unique<DwarfStreamer>(
-      Options.FileType, OutFile, Options.Translator, Options.Minimize,
+  Streamer = std::make_unique<DS>(
+      OutFileType, OutFile, Options.Translator, Options.Minimize,
       [&](const Twine &Error, StringRef Context, const DWARFDie *) {
         error(Error, Context);
       },
       [&](const Twine &Warning, StringRef Context, const DWARFDie *) {
         warn(Warning, Context);
       });
   return Streamer->init(TheTriple);
 }
if (EC)
     return errorCodeToError(EC);
 
   if (Error E = RL.serialize(OS, Options.RemarksFormat))
     return E;
 
   return Error::success();
 }
 
-ErrorOr<DWARFFile &>
-DwarfLinkerForBinary::loadObject(const DebugMapObject &Obj,
-                                 const DebugMap &DebugMap,
-                                 remarks::RemarkLinker &RL) {
+template <class DF>
+ErrorOr<DF &> DwarfLinkerForBinary::loadObject(
+    const DebugMapObject &Obj, const DebugMap &DebugMap,
+    remarks::RemarkLinker &RL,
+    std::vector<std::unique_ptr<DF>> &ObjectsForLinking,
+    bool isNewDwarfLinkerLibrary) {
   auto ErrorOrObj = loadObject(Obj, DebugMap.getTriple());
 
   if (ErrorOrObj) {
-    ContextForLinking.push_back(
-        std::unique_ptr<DWARFContext>(DWARFContext::create(*ErrorOrObj)));
-    AddressMapForLinking.push_back(
-        std::make_unique<AddressManager>(*this, *ErrorOrObj, Obj));
-
-    ObjectsForLinking.push_back(std::make_unique<DWARFFile>(
-        Obj.getObjectFilename(), ContextForLinking.back().get(),
-        AddressMapForLinking.back().get(),
-        Obj.empty() ? Obj.getWarnings() : EmptyWarnings));
+    ObjectsForLinking.push_back(std::make_unique<DF>(
+        Obj.getObjectFilename(), Obj.getTimestamp(),
+        DWARFContext::create(*ErrorOrObj),
+        std::make_unique<AddressManager>(isNewDwarfLinkerLibrary, *this,
+                                         *ErrorOrObj, Obj),
+        Obj.empty() ? Obj.getWarnings() : EmptyWarnings,
+        [&](StringRef FileName,
+            sys::TimePoint<std::chrono::seconds> Timestamp) {
+          BinHolder.eraseObjectEntry(FileName, Timestamp);
+        }));
 
     Error E = RL.link(*ErrorOrObj);
     if (Error NewE = handleErrors(
             std::move(E), [&](std::unique_ptr<FileError> EC) -> Error {
               return remarksErrorHandler(Obj, *this, std::move(EC));
             }))
       return errorToErrorCode(std::move(NewE));
 
     return *ObjectsForLinking.back();
   }
 
   return ErrorOrObj.getError();
 }
 
 bool DwarfLinkerForBinary::link(const DebugMap &Map) {
-  if (!createStreamer(Map.getTriple(), OutFile))
-    return false;
+  if (Options.UseDwarfLinkerNext)
+    return newLinkImpl(Map);
 
-  ObjectsForLinking.clear();
-  ContextForLinking.clear();
-  AddressMapForLinking.clear();
+  return linkImpl(Map);
+}
+
+bool DwarfLinkerForBinary::linkImpl(const DebugMap &Map) {
+  std::unique_ptr<DwarfStreamer> Streamer;
+  if (!Options.NoOutput) {
+    Streamer = std::make_unique<DwarfStreamer>(
+        Options.FileType, OutFile, Options.Translator, Options.Minimize,
+        [&](const Twine &Error, StringRef Context, const DWARFDie *) {
+          error(Error, Context);
+        },
+        [&](const Twine &Warning, StringRef Context, const DWARFDie *) {
+          warn(Warning, Context);
+        });
+    if (!Streamer->init(Map.getTriple()))
+      return false;
+  }
 
   DebugMap DebugMap(Map.getTriple(), Map.getBinaryPath());
 
+  std::vector<std::unique_ptr<DWARFFile>> ObjectsForLinking;
+
   DWARFLinker GeneralLinker(Streamer.get(), DwarfLinkerClient::Dsymutil);
 
   remarks::RemarkLinker RL;
   if (!Options.RemarksPrependPath.empty())
     RL.setExternalFilePrependPath(Options.RemarksPrependPath);
   GeneralLinker.setObjectPrefixMap(&Options.ObjectPrefixMap);
 
   std::function<StringRef(StringRef)> TranslationLambda = [&](StringRef Input) {
GeneralLinker.setWarningHandler(
       [&](const Twine &Warning, StringRef Context, const DWARFDie *DIE) {
         reportWarning(Warning, Context, DIE);
       });
   GeneralLinker.setErrorHandler(
       [&](const Twine &Error, StringRef Context, const DWARFDie *) {
         error(Error, Context);
       });
   GeneralLinker.setObjFileLoader(
-      [&DebugMap, &RL, this](StringRef ContainerName,
-                             StringRef Path) -> ErrorOr<DWARFFile &> {
+      [&](StringRef ContainerName, StringRef Path) -> ErrorOr<DWARFFile &> {
+        std::unique_lock<std::mutex> Guard(ObjLoaderMutex);
+
         auto &Obj = DebugMap.addDebugMapObject(
             Path, sys::TimePoint<std::chrono::seconds>(), MachO::N_OSO);
 
-        if (auto ErrorOrObj = loadObject(Obj, DebugMap, RL)) {
+        if (ErrorOr<DWARFFile &> ErrorOrObj = loadObject<DWARFFile>(
+                Obj, DebugMap, RL, ObjectsForLinking, false)) {
           return *ErrorOrObj;
         } else {
           // Try and emit more helpful warnings by applying some heuristics.
           StringRef ObjFile = ContainerName;
           bool IsClangModule = sys::path::extension(Path).equals(".pcm");
           bool IsArchive = ObjFile.endswith(")");
 
           if (IsClangModule) {
if (Obj->getType() == MachO::N_AST) {
 
       // Copy the module into the .swift_ast section.
       if (!Options.NoOutput)
         Streamer->emitSwiftAST((*ErrorOrMem)->getBuffer());
 
       continue;
     }
 
-    if (auto ErrorOrObj = loadObject(*Obj, Map, RL))
+    if (auto ErrorOrObj = loadObject(*Obj, Map, RL, ObjectsForLinking, false))
       GeneralLinker.addObjectFile(*ErrorOrObj);
     else {
       ObjectsForLinking.push_back(std::make_unique<DWARFFile>(
-          Obj->getObjectFilename(), nullptr, nullptr,
-          Obj->empty() ? Obj->getWarnings() : EmptyWarnings));
+          Obj->getObjectFilename(), Obj->getTimestamp(), nullptr, nullptr,
+          Obj->empty() ? Obj->getWarnings() : EmptyWarnings,
+          [&](StringRef, sys::TimePoint<std::chrono::seconds>) {}));
       GeneralLinker.addObjectFile(*ObjectsForLinking.back());
     }
   }
 
   // link debug info for loaded object files.
   GeneralLinker.link();
 
   StringRef ArchName = Map.getTriple().getArchName();
if (Map.getTriple().isOSDarwin() && !Map.getBinaryPath().empty() &&
     return MachOUtils::generateDsymCompanion(
         Options.VFS, Map, Options.Translator,
         *Streamer->getAsmPrinter().OutStreamer, OutFile);
 
   Streamer->finish();
   return true;
 }
 
+bool DwarfLinkerForBinary::newLinkImpl(const DebugMap &Map) {
+  dwarflinker::OutputFileType OutFileType;
+  if (Options.FileType == OutputFileType::Object)
+    OutFileType = dwarflinker::OutputFileType::Object;
+  else
+    OutFileType = dwarflinker::OutputFileType::Assembly;
+
+  std::unique_ptr<dwarflinker::DwarfStreamer> Streamer;
+
+  if (!Options.NoOutput) {
+    Streamer = std::make_unique<dwarflinker::DwarfStreamer>(
+        OutFileType, OutFile, Options.Translator,
+        [&](const Twine &Error, StringRef Context, const DWARFDie *) {
+          error(Error, Context);
+        },
+        [&](const Twine &Warning, StringRef Context, const DWARFDie *) {
+          warn(Warning, Context);
+        });
+    if (!Streamer->init(Map.getTriple()))
+      return false;
+  }
+
+  DebugMap DebugMap(Map.getTriple(), Map.getBinaryPath());
+
+  std::vector<std::unique_ptr<dwarflinker::DWARFFile>> ObjectsForLinking;
+
+  bool AtLeastOneDwarfAccelTable = false;
+  bool AtLeastOneAppleAccelTable = false;
+
+  std::function<StringRef(StringRef)> TranslationLambda = [&](StringRef Input) {
+    assert(Options.Translator);
+    return Options.Translator(Input);
+  };
+
+  dwarflinker::DWARFLinker GeneralLinker(
+      Streamer.get(), dwarflinker::DwarfLinkerClient::Dsymutil,
+      Options.Translator ? TranslationLambda : nullptr);
+
+  remarks::RemarkLinker RL;
+  if (!Options.RemarksPrependPath.empty())
+    RL.setExternalFilePrependPath(Options.RemarksPrependPath);
+  GeneralLinker.setObjectPrefixMap(&Options.ObjectPrefixMap);
+
+  GeneralLinker.setVerbosity(Options.Verbose);
+  GeneralLinker.setStatistics(Options.Statistics);
+  GeneralLinker.setNoOutput(Options.NoOutput);
+  GeneralLinker.setNoODR(Options.NoODR);
+  GeneralLinker.setRemoveObsoleteDebugInfo(!Options.Update);
+  GeneralLinker.setNumThreads(Options.Threads);
+  GeneralLinker.setPrependPath(Options.PrependPath);
+  GeneralLinker.setWarningHandler(
+      [&](const Twine &Warning, StringRef Context, const DWARFDie *DIE) {
+        reportWarning(Warning, Context, DIE);
+      });
+  GeneralLinker.setErrorHandler(
+      [&](const Twine &Error, StringRef Context, const DWARFDie *) {
+        error(Error, Context);
+      });
+  GeneralLinker.setObjFileLoader(
+      [&](StringRef ContainerName,
+          StringRef Path) -> ErrorOr<dwarflinker::DWARFFile &> {
+        std::unique_lock<std::mutex> Guard(ObjLoaderMutex);
+
+        auto &Obj = DebugMap.addDebugMapObject(
+            Path, sys::TimePoint<std::chrono::seconds>(), MachO::N_OSO);
+
+        if (ErrorOr<dwarflinker::DWARFFile &> ErrorOrObj =
+                loadObject<dwarflinker::DWARFFile>(Obj, DebugMap, RL,
+                                                   ObjectsForLinking, true)) {
+          return *ErrorOrObj;
+        } else {
+          // Try and emit more helpful warnings by applying some heuristics.
+          StringRef ObjFile = ContainerName;
+          bool IsClangModule = sys::path::extension(Path).equals(".pcm");
+          bool IsArchive = ObjFile.endswith(")");
+
+          if (IsClangModule) {
+            StringRef ModuleCacheDir = sys::path::parent_path(Path);
+            if (sys::fs::exists(ModuleCacheDir)) {
+              // If the module's parent directory exists, we assume that the
+              // module cache has expired and was pruned by clang.  A more
+              // adventurous dsymutil would invoke clang to rebuild the module
+              // now.
+              if (!ModuleCacheHintDisplayed) {
+                WithColor::note()
+                    << "The clang module cache may have expired since "
+                       "this object file was built. Rebuilding the "
+                       "object file will rebuild the module cache.\n";
+                ModuleCacheHintDisplayed = true;
+              }
+            } else if (IsArchive) {
+              // If the module cache directory doesn't exist at all and the
+              // object file is inside a static library, we assume that the
+              // static library was built on a different machine. We don't want
+              // to discourage module debugging for convenience libraries within
+              // a project though.
+              if (!ArchiveHintDisplayed) {
+                WithColor::note()
+                    << "Linking a static library that was built with "
+                       "-gmodules, but the module cache was not found.  "
+                       "Redistributable static libraries should never be "
+                       "built with module debugging enabled.  The debug "
+                       "experience will be degraded due to incomplete "
+                       "debug information.\n";
+                ArchiveHintDisplayed = true;
+              }
+            }
+          }
+
+          return ErrorOrObj.getError();
+        }
+
+        llvm_unreachable("Unhandled DebugMap object");
+      });
+  GeneralLinker.setSwiftInterfacesMap(&ParseableSwiftInterfaces);
+
+  for (const auto &Obj : Map.objects()) {
+    // N_AST objects (swiftmodule files) should get dumped directly into the
+    // appropriate DWARF section.
+    if (Obj->getType() == MachO::N_AST) {
+      if (Options.Verbose)
+        outs() << "DEBUG MAP OBJECT: " << Obj->getObjectFilename() << "\n";
+
+      StringRef File = Obj->getObjectFilename();
+      auto ErrorOrMem = MemoryBuffer::getFile(File);
+      if (!ErrorOrMem) {
+        warn("Could not open '" + File + "'\n");
+        continue;
+      }
+      sys::fs::file_status Stat;
+      if (auto Err = sys::fs::status(File, Stat)) {
+        warn(Err.message());
+        continue;
+      }
+      if (!Options.NoTimestamp) {
+        // The modification can have sub-second precision so we need to cast
+        // away the extra precision that's not present in the debug map.
+        auto ModificationTime =
+            std::chrono::time_point_cast<std::chrono::seconds>(
+                Stat.getLastModificationTime());
+        if (ModificationTime != Obj->getTimestamp()) {
+          // Not using the helper here as we can easily stream TimePoint<>.
+          WithColor::warning()
+              << File << ": timestamp mismatch between swift interface file ("
+              << sys::TimePoint<>(Obj->getTimestamp()) << ") and debug map ("
+              << sys::TimePoint<>(Obj->getTimestamp()) << ")\n";
+          continue;
+        }
+      }
+
+      // Copy the module into the .swift_ast section.
+      if (!Options.NoOutput)
+        Streamer->emitSwiftAST((*ErrorOrMem)->getBuffer());
+
+      continue;
+    }
+
+    if (auto ErrorOrObj = loadObject(*Obj, Map, RL, ObjectsForLinking, true)) {
+      GeneralLinker.addObjectFile(*ErrorOrObj);
+
+      if (Options.TheAccelTableKind == AccelTableKind::Default &&
+          ErrorOrObj->Dwarf) {
+        auto &DwarfObj = ErrorOrObj->Dwarf->getDWARFObj();
+
+        if (!AtLeastOneDwarfAccelTable &&
+            (!DwarfObj.getAppleNamesSection().Data.empty() ||
+             !DwarfObj.getAppleTypesSection().Data.empty() ||
+             !DwarfObj.getAppleNamespacesSection().Data.empty() ||
+             !DwarfObj.getAppleObjCSection().Data.empty()))
+          AtLeastOneAppleAccelTable = true;
+
+        if (!AtLeastOneDwarfAccelTable &&
+            !DwarfObj.getNamesSection().Data.empty())
+          AtLeastOneDwarfAccelTable = true;
+      }
+    } else {
+      ObjectsForLinking.push_back(std::make_unique<dwarflinker::DWARFFile>(
+          Obj->getObjectFilename(), Obj->getTimestamp(), nullptr, nullptr,
+          Obj->empty() ? Obj->getWarnings() : EmptyWarnings,
+          [&](StringRef, sys::TimePoint<std::chrono::seconds>) {}));
+      GeneralLinker.addObjectFile(*ObjectsForLinking.back());
+    }
+  }
+
+  // If we haven't decided on an accelerator table kind yet, we base ourselves
+  // on the DWARF we have seen so far. At this point we haven't pulled in debug
+  // information from modules yet, so it is technically possible that they
+  // would affect the decision. However, as they're built with the same
+  // compiler and flags, it is safe to assume that they will follow the
+  // decision made here.
+  if (Options.TheAccelTableKind == llvm::AccelTableKind::Apple) {
+    GeneralLinker.setGenerateAppleNamespaces(true);
+    GeneralLinker.setGenerateAppleNames(true);
+    GeneralLinker.setGenerateAppleTypes(true);
+    GeneralLinker.setGenerateAppleObjC(true);
+    GeneralLinker.setGeneratePubNames(!Options.Minimize);
+    GeneralLinker.setGeneratePubTypes(!Options.Minimize);
+  } else if (Options.TheAccelTableKind == llvm::AccelTableKind::Dwarf) {
+    GeneralLinker.setGenerateDebugNames(true);
+  } else {
+    if (AtLeastOneDwarfAccelTable && !AtLeastOneAppleAccelTable) {
+      GeneralLinker.setGenerateDebugNames(true);
+    } else {
+      GeneralLinker.setGenerateAppleNamespaces(true);
+      GeneralLinker.setGenerateAppleNames(true);
+      GeneralLinker.setGenerateAppleTypes(true);
+      GeneralLinker.setGenerateAppleObjC(true);
+      GeneralLinker.setGeneratePubNames(!Options.Minimize);
+      GeneralLinker.setGeneratePubTypes(!Options.Minimize);
+    }
+  }
+
+  // link debug info for loaded object files.
+  GeneralLinker.link();
+
+  StringRef ArchName = Map.getTriple().getArchName();
+  if (Error E = emitRemarks(Options, Map.getBinaryPath(), ArchName, RL))
+    return error(toString(std::move(E)));
+
+  if (Options.NoOutput)
+    return true;
+
+  if (Options.ResourceDir && !ParseableSwiftInterfaces.empty()) {
+    StringRef ArchName = Triple::getArchTypeName(Map.getTriple().getArch());
+    if (auto E =
+            copySwiftInterfaces(ParseableSwiftInterfaces, ArchName, Options))
+      return error(toString(std::move(E)));
+  }
+
+  if (Map.getTriple().isOSDarwin() && !Map.getBinaryPath().empty() &&
+      Options.FileType == OutputFileType::Object)
+    return MachOUtils::generateDsymCompanion(
+        Options.VFS, Map, Options.Translator,
+        *Streamer->getAsmPrinter().OutStreamer, OutFile);
+
+  Streamer->finish();
+  return true;
+}
+
 static bool isMachOPairedReloc(uint64_t RelocType, uint64_t Arch) {
   switch (Arch) {
   case Triple::x86:
     return RelocType == MachO::GENERIC_RELOC_SECTDIFF ||
            RelocType == MachO::GENERIC_RELOC_LOCAL_SECTDIFF;
   case Triple::x86_64:
     return RelocType == MachO::X86_64_RELOC_SUBTRACTOR;
   case Triple::arm:

 void DwarfLinkerForBinary::AddressManager::fillDieInfo(
     const ValidReloc &Reloc, CompileUnit::DIEInfo &Info) {
   Info.AddrAdjust = relocate(Reloc);
   if (Reloc.Mapping->getValue().ObjectAddress)
     Info.AddrAdjust -= uint64_t(*Reloc.Mapping->getValue().ObjectAddress);
   Info.InDebugMap = true;
 }
 
+void DwarfLinkerForBinary::AddressManager::fillDieInfo(
+    const ValidReloc &Reloc, dwarflinker::CompileUnit::DIEInfo &Info) {
+  Info.AddrAdjust = relocate(Reloc);
+  if (Reloc.Mapping->getValue().ObjectAddress)
+    Info.AddrAdjust -= uint64_t(*Reloc.Mapping->getValue().ObjectAddress);
+  Info.InDebugMap = true;
+}
+
 bool DwarfLinkerForBinary::AddressManager::hasValidRelocationAt(
     const std::vector<ValidReloc> &AllRelocs, uint64_t StartOffset,
     uint64_t EndOffset, CompileUnit::DIEInfo &Info) {
   std::vector<ValidReloc> Relocs =
       getRelocations(AllRelocs, StartOffset, EndOffset);
 
   if (Relocs.size() == 0)
     return false;
 
   if (Linker.Options.Verbose)
     printReloc(Relocs[0]);
   fillDieInfo(Relocs[0], Info);
 
   return true;
 }
 
+bool DwarfLinkerForBinary::AddressManager::hasValidRelocationAt(
+    const std::vector<ValidReloc> &AllRelocs, uint64_t StartOffset,
+    uint64_t EndOffset, dwarflinker::CompileUnit::DIEInfo &Info) {
+  std::vector<ValidReloc> Relocs =
+      getRelocations(AllRelocs, StartOffset, EndOffset);
+
+  if (Relocs.size() == 0)
+    return false;
+
+  if (Linker.Options.Verbose)
+    printReloc(Relocs[0]);
+  fillDieInfo(Relocs[0], Info);
+
+  return true;
+}
+
 /// Get the starting and ending (exclusive) offset for the
 /// attribute with index \p Idx descibed by \p Abbrev. \p Offset is
 /// supposed to point to the position of the first attribute described
 /// by \p Abbrev.
 /// \return [StartOffset, EndOffset) as a pair.
 static std::pair<uint64_t, uint64_t>
 getAttributeOffsets(const DWARFAbbreviationDeclaration *Abbrev, unsigned Idx,
                     uint64_t Offset, const DWARFUnit &Unit) {
if (Optional<uint64_t> AddrOffsetSectionBase =
                                   MyInfo);
     } else
       Linker.reportWarning("no base offset for address table", SrcFileName);
   }
 
   return false;
 }
 
+bool DwarfLinkerForBinary::AddressManager::hasLiveMemoryLocation(
+    const DWARFDie &DIE, dwarflinker::CompileUnit::DIEInfo &MyInfo) {
+
+  const auto *Abbrev = DIE.getAbbreviationDeclarationPtr();
+
+  Optional<uint32_t> LocationIdx =
+      Abbrev->findAttributeIndex(dwarf::DW_AT_location);
+  if (!LocationIdx)
+    return false;
+
+  uint64_t Offset = DIE.getOffset() + getULEB128Size(Abbrev->getCode());
+  uint64_t LocationOffset, LocationEndOffset;
+  std::tie(LocationOffset, LocationEndOffset) =
+      getAttributeOffsets(Abbrev, *LocationIdx, Offset, *DIE.getDwarfUnit());
+
+  // FIXME: Support relocations debug_addr.
+  return hasValidRelocationAt(ValidDebugInfoRelocs, LocationOffset,
+                              LocationEndOffset, MyInfo);
+}
+
+bool DwarfLinkerForBinary::AddressManager::hasLiveAddressRange(
+    const DWARFDie &DIE, dwarflinker::CompileUnit::DIEInfo &MyInfo) {
+  const auto *Abbrev = DIE.getAbbreviationDeclarationPtr();
+
+  Optional<uint32_t> LowPcIdx = Abbrev->findAttributeIndex(dwarf::DW_AT_low_pc);
+  if (!LowPcIdx)
+    return false;
+
+  dwarf::Form Form = Abbrev->getFormByIndex(*LowPcIdx);
+
+  if (Form == dwarf::DW_FORM_addr) {
+    uint64_t Offset = DIE.getOffset() + getULEB128Size(Abbrev->getCode());
+    uint64_t LowPcOffset, LowPcEndOffset;
+    std::tie(LowPcOffset, LowPcEndOffset) =
+        getAttributeOffsets(Abbrev, *LowPcIdx, Offset, *DIE.getDwarfUnit());
+    return hasValidRelocationAt(ValidDebugInfoRelocs, LowPcOffset,
+                                LowPcEndOffset, MyInfo);
+  }
+
+  if (Form == dwarf::DW_FORM_addrx) {
+    Optional<DWARFFormValue> AddrValue = DIE.find(dwarf::DW_AT_low_pc);
+    if (Optional<uint64_t> AddrOffsetSectionBase =
+            DIE.getDwarfUnit()->getAddrOffsetSectionBase()) {
+      uint64_t StartOffset = *AddrOffsetSectionBase + AddrValue->getRawUValue();
+      uint64_t EndOffset =
+          StartOffset + DIE.getDwarfUnit()->getAddressByteSize();
+      return hasValidRelocationAt(ValidDebugAddrRelocs, StartOffset, EndOffset,
+                                  MyInfo);
+    } else
+      Linker.reportWarning("no base offset for address table", SrcFileName);
+  }
+
+  return false;
+}
+
 uint64_t
 DwarfLinkerForBinary::AddressManager::relocate(const ValidReloc &Reloc) const {
   return Reloc.Mapping->getValue().BinaryAddress + Reloc.Addend;
 }
 
 /// Apply the valid relocations found by findValidRelocs() to
 /// the buffer \p Data, taking into account that Data is at \p BaseOffset
 /// in the debug_info section.

llvm/tools/dsymutil/LinkUtils.h

struct LinkOptions {
   bool Update = false;
 
   /// Minimize
   bool Minimize = false;
 
   /// Do not check swiftmodule timestamp
   bool NoTimestamp = false;
 
+  /// Use DWARFLinkerNext
+  bool UseDwarfLinkerNext = false;
+
   /// Number of threads.
   unsigned Threads = 1;
 
   // Output file type.
   OutputFileType FileType = OutputFileType::Object;
 
   /// The accelerator table kind
   AccelTableKind TheAccelTableKind;

llvm/tools/dsymutil/Options.td

def remarks_prepend_path: Separate<["--", "-"], "remarks-prepend-path">,
   Group<grp_general>;
 def: Joined<["--", "-"], "remarks-prepend-path=">, Alias<remarks_prepend_path>;
 
 def remarks_output_format: Separate<["--", "-"], "remarks-output-format">,
   MetaVarName<"<format>">,
   HelpText<"Specify the format to be used when serializing the linked remarks.">,
   Group<grp_general>;
 def: Joined<["--", "-"], "remarks-output-format=">, Alias<remarks_output_format>;
+
+def use_dlnext: F<"use-dlnext">,
+  HelpText<"Use DWARFLinkerNext.">,
+  Group<grp_general>;

llvm/tools/dsymutil/dsymutil.cpp

static Expected<DsymutilOptions> getOptions(opt::InputArgList &Args) {
 
   Options.LinkOpts.Minimize = Args.hasArg(OPT_minimize);
   Options.LinkOpts.NoODR = Args.hasArg(OPT_no_odr);
   Options.LinkOpts.NoOutput = Args.hasArg(OPT_no_output);
   Options.LinkOpts.NoTimestamp = Args.hasArg(OPT_no_swiftmodule_timestamp);
   Options.LinkOpts.Update = Args.hasArg(OPT_update);
   Options.LinkOpts.Verbose = Args.hasArg(OPT_verbose);
   Options.LinkOpts.Statistics = Args.hasArg(OPT_statistics);
+  Options.LinkOpts.UseDwarfLinkerNext = Args.hasArg(OPT_use_dlnext);
 
   if (opt::Arg *ReproducerPath = Args.getLastArg(OPT_use_reproducer)) {
     Options.ReproMode = ReproducerMode::Use;
     Options.ReproducerPath = ReproducerPath->getValue();
   }
 
   if (Args.hasArg(OPT_gen_reproducer))
     Options.ReproMode = ReproducerMode::Generate;