This is an archive of the discontinued LLVM Phabricator instance.

[Debuginfo][llvm-dwarfutil] llvm-dwarfutil dsymutil-like tool for ELF.
ClosedPublic

Authored by avl on Aug 25 2020, 7:04 AM.

Details

Summary

This patch implements proposal https://lists.llvm.org/pipermail/llvm-dev/2020-August/144579.html
llvm-dwarfutil - is a tool that is used for processing debug info(DWARF) located in built binary files to improve debug info quality, reduce debug info size. The patch currently implements smaller set of command-line options(comparing to the proposal):

./llvm-dwarfutil [options] <input file> <output file>

  --garbage-collection    Do garbage collection for debug info(default)
  -j <value>              Alias for --num-threads
  --no-garbage-collection Don`t do garbage collection for debug info
  --no-odr-deduplication  Don`t do ODR deduplication for debug types
  --no-odr                Alias for --no-odr-deduplication
  --no-separate-debug-file
                          Create single output file, containing debug tables(default)
  --num-threads <threads> Number of available threads for multi-threaded execution. Defaults to the number of cores on the current machine
  --odr-deduplication     Do ODR deduplication for debug types(default)
  --odr                   Alias for --odr-deduplication
  --separate-debug-file   Create two output files: file w/o debug tables and file with debug tables
  --tombstone [bfd,maxpc,exec,universal]
                          Tombstone value used as a marker of invalid address(default: universal)
    =bfd - Zero for all addresses and [1,1] for DWARF v4 (or less) address ranges and exec
    =maxpc - Minus 1 for all addresses and minus 2 for DWARF v4 (or less) address ranges
    =exec - Match with address ranges of executable sections
    =universal - Both: bfd and maxpc

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jhenderson added inline comments.Apr 6 2022, 12:43 AM
llvm/test/tools/llvm-dwarfutil/ELF/verify.test
12–14 ↗(On Diff #420641)

Check the error messages.

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
117–122

Test case?

119

Don't think you need upper-case here.

142

SectionRef is designed to be trivially copyable and doesn't really need the const & bit.

161–164

Not tested.

175

Test case?

183

Test case?

201–208

It's more common to have the public interface first in the class, I find.

221

Do you need the trailing return type?

246

Ditto.

272

Is SmallString really the right answer here for such a large base size?

294

Delete this blank line.

303

Ditto.

333

Is the trailing return type needed?

361

Ditto.

397

Ditto.

407

Test case?

417–418

Test case?

487

Test case?

513

Test case?

avl added inline comments.Apr 6 2022, 2:09 AM
llvm/lib/DWARFLinker/DWARFLinker.cpp
438–439

right. I used the same style like in the rest of that file. It does not look good to have a mix of styles. I think it should be separate commit for DWARFLinker library changing style of error messages.

llvm/test/tools/llvm-dwarfutil/ELF/copy-itself.test
7 ↗(On Diff #420641)

Ok.

8–10 ↗(On Diff #420641)

i.e. I need to move this check into the "standard" copy test file. OK.

12–27 ↗(On Diff #420641)

It actually does not duplicate coverage. Using combination of these options(--garbage-collection, --no-garbage-collection, --separate-debug-file) leads to crossing different code paths:

llvm-dwwarfutil.cpp:219
llvm-dwwarfutil.cpp:244
llvm-dwwarfutil.cpp:331
llvm-dwwarfutil.cpp:359
llvm-dwwarfutil.cpp:395

Thus it looks good if all of them would be covered.

llvm/test/tools/llvm-dwarfutil/ELF/copy.test
1–2 ↗(On Diff #420641)

It seems like gc-default.test covers much of this test's behaviour? Do we need both?

Probably, part for garbage-collection case is doubled and then might be removed. But part for no-garbage-collection looks correct and is not covered by gc-default.test.

17–18 ↗(On Diff #420641)

It looks like it does not pass:

diff --git a/llvm/test/tools/llvm-dwarfutil/ELF/copy.test b/llvm/test/tools/llvm-dwarfutil/ELF/copy.test
index 9ca78308f5da..9e9eb8b84750 100644
--- a/llvm/test/tools/llvm-dwarfutil/ELF/copy.test
+++ b/llvm/test/tools/llvm-dwarfutil/ELF/copy.test
@@ -14,7 +14,7 @@
 # RUN: llvm-objdump --headers %t1 | FileCheck -check-prefix=CHECK-NON-DEBUG %s
 # RUN: llvm-dwarfdump -a %t1.debug | FileCheck -check-prefix=CHECK-DEBUG %s
 
-# RUN: llvm-dwarfutil %t.o --separate-debug-file --no-separate-debug-file %t1
+# RUN: llvm-dwarfutil %t.o --separate-debug-file %t1
 # RUN: llvm-dwarfdump -a %t1 | FileCheck -check-prefix=CHECK-DEBUG %s
 
 # RUN: llvm-dwarfutil --no-garbage-collection %t.o %t1
llvm-lit  copy.test 
-- Testing: 1 tests, 1 workers --
FAIL: LLVM :: tools/llvm-dwarfutil/ELF/copy.test (1 of 1)
llvm/test/tools/llvm-dwarfutil/ELF/odr-class.test
11 ↗(On Diff #420641)

OK

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
333

I think it is always useful to type it explicitly. It clearly shows return type. Otherwise you need to compile the lambda by itself. If it is against of guideline - then would remove it.

avl updated this revision to Diff 421013.Apr 6 2022, 2:27 PM

addressed comments.

Right now if any DWARF section isn't supported, or if a DW_FORM is not supported, this patch will still emit a file and return a non-zero exit status.

So if this patch isn't going to support any of the DWARF sections that are required, it must exit with a valid error message and return a non zero exit status. The things that come to mind are:

  • anything with an older .debug_types section should return an error stating that debug types support is not supported
  • any newer DWARF with type units in the .debug_info section should return an error stating that debug types support is not supported
  • If fission is not supported, we need to return an error if we find any fission related sections (".debug_addr", etc)
  • if any DWARF section is required for debugging (like ".debug_loclists" or ".debug_rnglists") and not getting re-linked by this tool and if those sections become invalid after the DWARF is changed, then we need to return an error
  • if the line tables are too new and the DWARF linker can't update them return an error

All of this stems from Apple not having to support anything but what the Apple compilers emit. Many compilers that emit ELF files and linkers that link ELF have many other options that can be enabled for DWARF. If we aren't going to support them, we can't just emit some warnings and hope the user knows better, we need to return an error and set the exit status to a non zero value. We will need tests to cover these errors as well.

llvm/lib/DWARFLinker/DWARFLinker.cpp
437

Can we improve this check to see if the range already exists and not warn if we try to add the same range that is already in there or if the range is a subrange of an existing range?

avl added inline comments.Apr 6 2022, 2:37 PM
llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
175

It is quite tricky to create a test case here. Changing this to asserts does not look right. I think it is safe to leave it as is.

272

raw_svector_ostream uses SmallString. If that constant looks too large let`s consider to make it smaller?

avl added a comment.EditedApr 6 2022, 2:44 PM

Right now if any DWARF section isn't supported, or if a DW_FORM is not supported, this patch will still emit a file and return a non-zero exit status.

So if this patch isn't going to support any of the DWARF sections that are required, it must exit with a valid error message and return a non zero exit status. The things that come to mind are:

  • anything with an older .debug_types section should return an error stating that debug types support is not supported
  • any newer DWARF with type units in the .debug_info section should return an error stating that debug types support is not supported
  • If fission is not supported, we need to return an error if we find any fission related sections (".debug_addr", etc)
  • if any DWARF section is required for debugging (like ".debug_loclists" or ".debug_rnglists") and not getting re-linked by this tool and if those sections become invalid after the DWARF is changed, then we need to return an error
  • if the line tables are too new and the DWARF linker can't update them return an error

All of this stems from Apple not having to support anything but what the Apple compilers emit. Many compilers that emit ELF files and linkers that link ELF have many other options that can be enabled for DWARF. If we aren't going to support them, we can't just emit some warnings and hope the user knows better, we need to return an error and set the exit status to a non zero value. We will need tests to cover these errors as well.

My intention was to limit first version to functionality presented in current dsymutil/DWARFLinker. And to add improvements incrementally. But If we want to have that extended check in the first version then I will add it.

My own preference is to integrate that first version of the tool and add improvements incrementally. f.e. adding such extended check would require changing interface of DWARFLinker. Use "Error DWARFLinker::link()" instead of "bool DWARFLinker::link()". It would be better to make such change separately.

In D86539#3434454, @avl wrote:

Right now if any DWARF section isn't supported, or if a DW_FORM is not supported, this patch will still emit a file and return a non-zero exit status.

So if this patch isn't going to support any of the DWARF sections that are required, it must exit with a valid error message and return a non zero exit status. The things that come to mind are:

  • anything with an older .debug_types section should return an error stating that debug types support is not supported
  • any newer DWARF with type units in the .debug_info section should return an error stating that debug types support is not supported
  • If fission is not supported, we need to return an error if we find any fission related sections (".debug_addr", etc)
  • if any DWARF section is required for debugging (like ".debug_loclists" or ".debug_rnglists") and not getting re-linked by this tool and if those sections become invalid after the DWARF is changed, then we need to return an error
  • if the line tables are too new and the DWARF linker can't update them return an error

All of this stems from Apple not having to support anything but what the Apple compilers emit. Many compilers that emit ELF files and linkers that link ELF have many other options that can be enabled for DWARF. If we aren't going to support them, we can't just emit some warnings and hope the user knows better, we need to return an error and set the exit status to a non zero value. We will need tests to cover these errors as well.

My intention was to limit first version to functionality presented in current dsymutil/DWARFLinker. And to add improvements incrementally. But If we want to have that extended check in the first version then I will add it.

My own preference is to integrate that first version of the tool and add improvements incrementally. f.e. adding such extended check would require changing interface of DWARFLinker. Use "Error DWARFLinker::link()" instead of "bool DWARFLinker::link()". It would be better to make such change separately.

A little background: I am very interested in this patch and I have been trying to get statistics with the resulting binary from this patch so I can evaluate if we want to use it in production on linked binaries produced by our build system. I'd like to see how long it takes to run and also how much smaller debug info becomes. If I run this currently I have no way to know if we are producing anything valid and wether or not I can log the results since we always produce something and have no exit status. I also can't trust the resulting debug info sizes at all since we might be removing all sorts of extra DWARF sections that the existing DWARF linker doesn't handle, or worse yet we just don't clone an attribute and we are missing inline debug info. So it is hard to evaluate this right now as it is. I do think we should try to return errors for files we can't handle though as it would be great if people trying this out could evaluate it and know if it is working for them or if there are things that aren't handled.

In D86539#3434454, @avl wrote:

Right now if any DWARF section isn't supported, or if a DW_FORM is not supported, this patch will still emit a file and return a non-zero exit status.

So if this patch isn't going to support any of the DWARF sections that are required, it must exit with a valid error message and return a non zero exit status. The things that come to mind are:

  • anything with an older .debug_types section should return an error stating that debug types support is not supported
  • any newer DWARF with type units in the .debug_info section should return an error stating that debug types support is not supported
  • If fission is not supported, we need to return an error if we find any fission related sections (".debug_addr", etc)
  • if any DWARF section is required for debugging (like ".debug_loclists" or ".debug_rnglists") and not getting re-linked by this tool and if those sections become invalid after the DWARF is changed, then we need to return an error
  • if the line tables are too new and the DWARF linker can't update them return an error

All of this stems from Apple not having to support anything but what the Apple compilers emit. Many compilers that emit ELF files and linkers that link ELF have many other options that can be enabled for DWARF. If we aren't going to support them, we can't just emit some warnings and hope the user knows better, we need to return an error and set the exit status to a non zero value. We will need tests to cover these errors as well.

My intention was to limit first version to functionality presented in current dsymutil/DWARFLinker. And to add improvements incrementally. But If we want to have that extended check in the first version then I will add it.

My own preference is to integrate that first version of the tool and add improvements incrementally. f.e. adding such extended check would require changing interface of DWARFLinker. Use "Error DWARFLinker::link()" instead of "bool DWARFLinker::link()". It would be better to make such change separately.

A little background: I am very interested in this patch and I have been trying to get statistics with the resulting binary from this patch so I can evaluate if we want to use it in production on linked binaries produced by our build system. I'd like to see how long it takes to run and also how much smaller debug info becomes. If I run this currently I have no way to know if we are producing anything valid and wether or not I can log the results since we always produce something and have no exit status. I also can't trust the resulting debug info sizes at all since we might be removing all sorts of extra DWARF sections that the existing DWARF linker doesn't handle, or worse yet we just don't clone an attribute and we are missing inline debug info. So it is hard to evaluate this right now as it is. I do think we should try to return errors for files we can't handle though as it would be great if people trying this out could evaluate it and know if it is working for them or if there are things that aren't handled.

Sure enough, though that doesn't necessarily have to be in the first version that's committed. It's work in progress - if this gets committed sooner then you can pitch in and help add some of that functionality, which might be more productive. Error handling/ensuring invalid/unsupported inputs don't result in "successful" garbage output is certainly important/something to keep in mind.

jhenderson added inline comments.Apr 7 2022, 2:37 AM
llvm/lib/DWARFLinker/DWARFLinker.cpp
438–439

Agreed

llvm/test/tools/llvm-dwarfutil/ELF/Inputs/archive.a
1 ↗(On Diff #421013)

Why can't this file be generated on the fly? It's also a bit misleading calling it archive.a when it's a thin archive.

llvm/test/tools/llvm-dwarfutil/ELF/copy-itself.test
12–27 ↗(On Diff #420641)

I'm not sure what you mean by "crossing different code paths" here exactly, but writeToOutput's behaviour is already tested, and a single test case is sufficient to show that writing to the input file via this method is sufficient, assuming that code calling that function doesn't do anything odd like open the input file again etc. Given that it doesn't currently, a test isn't needed until someone adds that functionality. Your non self-copy tests are sufficient to show that output is written correctly when those options are used.

llvm/test/tools/llvm-dwarfutil/ELF/copy.test
11 ↗(On Diff #421013)

Rewording as suggested changes the emphasis slightly to be more natural, namely that the second copy is the thing that is uncertain, and the first is the "correct" version.

16 ↗(On Diff #421013)

Shouldn't these cases be in a dedicated test file for --separate-debug-file?

23 ↗(On Diff #421013)

We already have a test case for the explicit option. The interesting bit here is the last-one-wins behaviour, so just focus on that in this comment.

28–29 ↗(On Diff #421013)
31 ↗(On Diff #421013)

Use cmp rather than diff for binary files. Applies throughout.

17–18 ↗(On Diff #420641)

I somehow completely misread the test (mistaking --no-garbage-collection for --no-separate-debug-file).

llvm/test/tools/llvm-dwarfutil/ELF/error-corrupted-input-file.test
6 ↗(On Diff #421013)

The format would usually be error: 'archive.a': corrupted input file. See createFileError (which you should be using).

That being said, how is this input file "corrupt"? The file itself is a perfectly valid thin archive file. Shouldn't this just be something like "unsupported file format" or similar?

llvm/test/tools/llvm-dwarfutil/ELF/error-separate-file-stdout.test
7 ↗(On Diff #421013)

<stdin> is typically used for error messages where the file is read from stdin (please add code and a test case that does this). It makes sense to use the same style for stdout (i.e. <stdout>) where that's relevant, but actually here, I think it would be clearer to say "unable to write to stdout when --separate-debug-file specified" or something equivalent.

llvm/test/tools/llvm-dwarfutil/ELF/gc-class.test
9 ↗(On Diff #421013)

I don't think you need to reference "--garbage-collection" in all of these comments in this file, since it's not all that relevant to the test case.

13 ↗(On Diff #421013)

Could simplify to "Check --odr alias." No need to say "explicit" here since the use of an alias implies it's explicit anyway.

17–19 ↗(On Diff #421013)

I don't think you need to check the "last one wins" behaviour of aliases. It should be sufficient to check that for the primary versions of the options (and I'd do that before the aliases) and then show that aliases produce identical output to their main version.

21 ↗(On Diff #421013)
29 ↗(On Diff #421013)
llvm/test/tools/llvm-dwarfutil/ELF/verbose.test
10 ↗(On Diff #421013)

I'm not convinced you should have a warning here, because otherwise the --verbose option will always produce a warning unless an explicit --num-threads option is specified. I'm not sure though. Perhaps documenting it in the CommandGuide and help text would be sufficient?

llvm/test/tools/llvm-dwarfutil/error-invalid-format.test
8 ↗(On Diff #420641)

Not addressed?

llvm/test/tools/llvm-dwarfutil/error-no-input-file.test
6 ↗(On Diff #420641)

Not addressed.

Also, this needs to use a lit substitution for the message format. See the %errc* substitutions.

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
175

Looks like you made a test case after all :-)

272

Yeah, I think smaller may be better. As things stand, this will create a ~100kb string on the stack, which is rather a lot of stack used in one go, but someone with better undersetanding of the code may have a more informed opinion.

333

Otherwise you need to compile the lambda by itself

I don't understand this comment.

If it is against of guideline - then would remove it.

It's because it's superfluous noise - I'm not sure there's a specific coding standard on it though. Since this lambda is being passed directly into a function, you don't need to know what its return type is due to how you use it later. Within the function, it's clear that the return type is Error, so it doesn't really help with readability of the function either, in my opinion. Consequently, the extra code means it's harder for me to read everything.

417–418

Not addressed?

avl marked an inline comment as done.Apr 7 2022, 8:02 AM

A little background: I am very interested in this patch and I have been trying to get statistics with the resulting binary from this patch so I can evaluate if we want to use it in production on linked binaries produced by our build system. I'd like to see how long it takes to run and also how much smaller debug info becomes. If I run this currently I have no way to know if we are producing anything valid and wether or not I can log the results since we always produce something and have no exit status. I also can't trust the resulting debug info sizes at all since we might be removing all sorts of extra DWARF sections that the existing DWARF linker doesn't handle, or worse yet we just don't clone an attribute and we are missing inline debug info. So it is hard to evaluate this right now as it is. I do think we should try to return errors for files we can't handle though as it would be great if people trying this out could evaluate it and know if it is working for them or if there are things that aren't handled.

Ok, I will prepare followup patch implementing such a behavior.

avl added inline comments.Apr 7 2022, 8:43 AM
llvm/test/tools/llvm-dwarfutil/ELF/copy-itself.test
12–27 ↗(On Diff #420641)

If the implementation would be changed later - and instead of writeToOutput there would be used another code in some places - then we will not have a check that proves everything is working as expected(if we will remove these tests).

llvm/test/tools/llvm-dwarfutil/ELF/copy.test
16 ↗(On Diff #421013)

From my point of view there is no difference whether we have dedicated test file for --separate-debug-file or it is tested inside other test cases. If you think we need one - I will do it.

llvm/test/tools/llvm-dwarfutil/ELF/error-corrupted-input-file.test
6 ↗(On Diff #421013)

The same error might occur if the source file is corrupted. Possible good message would be :

error: 'archive.a': unsupported or corrupted input file

?

llvm/test/tools/llvm-dwarfutil/ELF/verbose.test
10 ↗(On Diff #421013)

If that warning looks annoying - yes we can avoid it and document the behavior. To check my understanding - the warning should not be shown in all cases: in default "llvm-dwarfutil %t.o %t1 --verbose" and in the explicit : "llvm-dwarfutil %t.o %t1 --verbose --num-threads 10", right ?

llvm/test/tools/llvm-dwarfutil/error-invalid-format.test
8 ↗(On Diff #420641)

it looks addressed. error-invalid-format.test - is the file name. It probably should be in quotas(using createFileError) like you said in other comment.

llvm/test/tools/llvm-dwarfutil/error-no-input-file.test
6 ↗(On Diff #420641)

not-existed - is the file name.

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
175

no. there is no test case for it still.

333

Otherwise you need to compile the lambda by itself

I don't understand this comment.

I mean that the type of returning value is not clear while someone inspecting the code. Someone need to check types of expressions in the "return" clauses, type of argument where lambda is used, etc...

If it is against of guideline - then would remove it.

It's because it's superfluous noise - I'm not sure there's a specific coding standard on it though. Since this lambda is being passed directly into a function, you don't need to know what its return type is due to how you use it later. Within the function, it's clear that the return type is Error, so it doesn't really help with readability of the function either, in my opinion. Consequently, the extra code means it's harder for me to read everything.

My understanding is opposite. It is not really clear whether return type is Error, or instead it is Expected<something> or something else.. If returning type is hidden it requires less letters but more actions to understand.

so I think the benefit of using explicit return value is more than created noise.

417–418

yep, it is not addressed yet. Returning type of linkDebugInfo is kind of artificial at the moment. I will change it in followup review. Currently it is hard to create a testcase when it returns false.

avl added inline comments.Apr 7 2022, 11:45 AM
llvm/lib/DWARFLinker/DWARFLinker.cpp
437

@clayborg

Can we improve this check to see if the range already exists and not warn if we try to add the same range that is already in there or if the range is a subrange of an existing range?

I did not get the idea. Every overlapping range is such that "it is already in there or if the range is a subrange of an existing range". Probably you mean: not to report the same range twice ?

clayborg added inline comments.Apr 7 2022, 1:43 PM
llvm/lib/DWARFLinker/DWARFLinker.cpp
437

The idea here is that the "Ranges" ivar in the CompileUnit class I am assuming is used to create the DW_AT_ranges for a compile unit. If you have two functions that LTO combined into a single address range, you currently will get an "Overlapping address range [0x1000-0x2000), [0x1000, 0x2000). Range will be discarded." warning, which is not true at all in this case and is a bad warning. What is actually happening is that the range is already in the "Ranges" and it won't be discarded and everything is fine.

We also get errors in DWARF a lot of the time due to no smart DWARF linkers out there, and you might have a range that is already inside the "Ranges" like if you have a DW_TAG_subprogram with a range of [0x1000-0x2000) and another with [0x1100-0x1200). In this case "Ranges" will already contain the second range and nothing needs to be added to "Ranges". This is bad DWARF of course, but it happens quite a bit as I have taken part in the "llvm-dwarfdump --verify" patches and we would get a error for overlapping ranges here.

So there are 3 cases I can think of:

  • The same range tries to get added to "Ranges" and that is ok since it is already there
  • A range that tries to get added is already fully contained within "Ranges", which should also be ok since "Ranges" already contains that range, though it is an indication of bad DWARF.
  • You have ranges that intersect with "Ranges" but are not fully contained. like if "Ranges" contained "[0x1000-0x02000) and you tried to add [0xf00-0x1100] or [0x1100-0x2100)

If our goal is to do the best job possible with "CompileUnit::Ranges" so that we can produce a valid DW_AT_ranges attribute, then we should try and do our best to keep it up to date. Right now with the current code, we will happily link the DWARF for the 3rd case above, but we will create in incomplete DW_AT_ranges by ejecting either of the overlapping ranges like [0xf00-0x1100] or [0x1100-0x2100).

clayborg added inline comments.Apr 7 2022, 1:48 PM
llvm/lib/DWARFLinker/DWARFLinker.cpp
437

Also, I believe "Ranges" will combine sequential ranges from reading the header file, so if "Ranges" already fully contains the range, we do nothing and don't warn, and if a range isn't fully contained but intersects, we should extend the range it overlaps with, and if it doesn't overlap of course we just add it.

jhenderson added inline comments.Apr 8 2022, 12:37 AM
llvm/test/tools/llvm-dwarfutil/ELF/copy-itself.test
12–27 ↗(On Diff #420641)

And at that point, it would be up to the developer to consider the new behaviour and how it needs testing though. Regardless, that code will still be tested by basic testing for each of the different options - the combination of them doesn't change the fact that it gets called ultimately. Combining the options doesn't change the code path of the area under test to something that can't be hit by it. If we followed your logic, we'd write tests for every possible combination of tool options for every tool, which is clearly impractical.

llvm/test/tools/llvm-dwarfutil/ELF/error-corrupted-input-file.test
6 ↗(On Diff #421013)

This error can only be hit if the object is a file type supported by Binary that is not an object type. Unless the input file is somehow corrupted to make type identification pass but for the wrong kind of file, then the code wouldn't get to this error - it would fail during Binary construction, which is an earlier error. If it does get corrupted in such a way as to look like another object file kind, I don't think it's unreasonable to assume it is that object file type and state simply "unsupported" (otherwise we have to state "corrupt or unsupported" for every instance where we currently just say "unsupported"). Is there another case you had in mind for "corrupt"?

llvm/test/tools/llvm-dwarfutil/ELF/verbose.test
10 ↗(On Diff #421013)

I think default behaviour (no --num-threads specified) should not produce a warning. If --num-threads is specified explicitly, I think a warning makes sense.

llvm/test/tools/llvm-dwarfutil/error-invalid-format.test
8 ↗(On Diff #420641)

Sorry, missed that somehow, but see my comment elsewhere about filenames appearing after "error" (use createFileError).

llvm/test/tools/llvm-dwarfutil/error-no-input-file.test
6 ↗(On Diff #420641)

Same comment as above - use createFileError (as well as the %errc* substitution).

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
175

verbose.test line 24 checks "Verifying DWARF..." though?

333

I've created a Discourse thread (https://discourse.llvm.org/t/coding-standard-for-lambda-trailing-return-types/61569) to discuss the general principle, because I don't agree with you, and I think it's clear we need a general standard on this topic.

I mean that the type of returning value is not clear while someone inspecting the code. Someone need to check types of expressions in the "return" clauses, type of argument where lambda is used, etc...

In this particular context, when does somebody need to know the return type? How do return Err; and return Error::success(); not express the return type clearly?

My understanding is opposite. It is not really clear whether return type is Error, or instead it is Expected<something> or something else.

The use of Error::success() rules out Expected<T> as a return type. Instead it would return a concrete value. Again though, my question is, why do you need to know the return type in this context?

I argue the following for this specific case:

  1. The lambda is being passed directly into a function so we don't have to worry about capturing it in a variable and ensuring the variable's type matches that of the lambda return value.
  2. The function is small. If someone needs to modify it, they'll need to understand what the function does anyway, and it's obvious from what is being returned: the use of Err as a variable name, immediately after assigning it to an Error variable, combined with Error::success().
  3. The compiler will tell you if you get something wrong when modifying the code (either the function signature changing or somebody modifying the lambda). If developing in an IDE, this error may even appear as you are typing, without the need to explicitly write the compiler.
417–418

Okay, I take it this isn't the code path that is hit if debug data in the input is malformed?

avl added inline comments.Apr 8 2022, 3:22 AM
llvm/test/tools/llvm-dwarfutil/ELF/copy-itself.test
12–27 ↗(On Diff #420641)

not every possible but every relevant. We have several paths which could lead to writing to the file - it would be good if all of these paths would be covered by test.

But if we want to reduce that level of testing - I will left only one case.

llvm/test/tools/llvm-dwarfutil/ELF/error-corrupted-input-file.test
6 ↗(On Diff #421013)

Ok, would use "unsupported file format".

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
175

my comment was about 181 line :

return createError("unsupported file: '" + FileName + "'");

this line is not covered by test and it is tricky to do it.

333

I've created a Discourse thread (https://discourse.llvm.org/t/coding-standard-for-lambda-trailing-return-types/61569) to discuss the general principle, because I don't agree with you, and I think it's clear we need a general standard on this topic.

Thanks. I will join to this thread.

I mean that the type of returning value is not clear while someone inspecting the code. Someone need to check types of expressions in the "return" clauses, type of argument where lambda is used, etc...

In this particular context, when does somebody need to know the return type? How do return Err; and return Error::success(); not express the return type clearly?

In this particular context it is more or less obvious. But in general it is very subjective - whether it is obvious or not. C++ is very complex language. If you see return true - it does not mean return value is bool. if you see return foo() - it is not seen which type it is. I think it is easier to just have a return type than think whether it is obvious or not.

My understanding is opposite. It is not really clear whether return type is Error, or instead it is Expected<something> or something else.

The use of Error::success() rules out Expected<T> as a return type. Instead it would return a concrete value. Again though, my question is, why do you need to know the return type in this context?

In this particular context it is more or less obvious(Though it still requires to check several places(return Err, Error Err, return Error::success()) instead of one(-> Error)). The thing which does not worth a time, from my opinion -> thinking whether it is enough obvious or not.

I argue the following for this specific case:

  1. The lambda is being passed directly into a function so we don't have to worry about capturing it in a variable and ensuring the variable's type matches that of the lambda return value.
  2. The function is small. If someone needs to modify it, they'll need to understand what the function does anyway, and it's obvious from what is being returned: the use of Err as a variable name, immediately after assigning it to an Error variable, combined with Error::success().
  3. The compiler will tell you if you get something wrong when modifying the code (either the function signature changing or somebody modifying the lambda). If developing in an IDE, this error may even appear as you are typing, without the need to explicitly write the compiler.

My main point is that it would be easier to not waste a time on thinking whether each concrete case is obvious enough or not.

417–418

right. malformed input would not lead to return false here.(Though it looks logical if it is)

jhenderson added inline comments.Apr 8 2022, 3:40 AM
llvm/test/tools/llvm-dwarfutil/ELF/copy-itself.test
12–27 ↗(On Diff #420641)

To be clear, I agree that all call sites for writeToOutput need testing (i.e. the ones you've highlighted above). But I don't think they need testing for the "copy self" case, because that's not significantly different to the "copy to other file" case, as long as there is a test that shows "copy self" in general works.

llvm/test/tools/llvm-dwarfutil/ELF/copy.test
16 ↗(On Diff #421013)

Normally, I'd have a dedicated test file per option, with possible additional ones for testing option interactions where needed. --[no-]separate-debug-file is a distinct option, hence I think it should be tested in a separate file, rather than mixing multiple unrelated test axes in one file.

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
175

Digging into the code some more, the "unsupported file" case can only be hit if the created Binary instance is not an object, but is a recognised Binary type. For this to happen, the output file would have to have been written in an invalid state, no? That seems like an internal error within the program, so possibly should be an llvm_unreachable or similar?

333

My point is less if it's obvious and more whether it matters. In my experience, in 99% of cases, it doesn't with regards to lambdas because of how they are used. Anyway, let's see how that discussion in Discourse pans out. I'm not opposed to trailing return types in some cases where it really isn't obvious and it does matter, but I don't believe that to be the case here. It sounds to me like you're advocating for "always have return types".

avl updated this revision to Diff 421615.Apr 8 2022, 12:20 PM

addressed comments.

not yet addressed comments: overlapped address ranges, returning type of lambda,
check for unsupported debug tables.

avl updated this revision to Diff 421775.Apr 10 2022, 1:47 AM

added comment for overlapping address ranges workaround.

avl edited the summary of this revision. (Show Details)Apr 10 2022, 12:14 PM
jhenderson added inline comments.Apr 10 2022, 11:17 PM
llvm/lib/DWARFLinker/DWARFLinker.cpp
440
llvm/test/tools/llvm-dwarfutil/ELF/error-unsupported-input-file.test
5–7 ↗(On Diff #421775)

Optional to ensure you are checking the file name and not accidentally something else.

llvm/test/tools/llvm-dwarfutil/ELF/gc-class.test
25 ↗(On Diff #421775)
llvm/test/tools/llvm-dwarfutil/ELF/reading-from-stdinput.test
4 ↗(On Diff #421775)

Up to you on this one, but I'd have this in the copy.test file, and then you can do llvm-dwarfutil --no-garbage-collection - %t1 < [input-file] where [input-file] is the same object used in the existing test cases in that file.

24 ↗(On Diff #421775)

Something to consider if you prefer to keep separate, or indeed more generally: move the YAML into a separate file under an Inputs subdirectory so that it can be referenced by multiple test cases. That saves duplicating identical YAML across different test files.

llvm/test/tools/llvm-dwarfutil/ELF/separate-debug-file.test
1 ↗(On Diff #421775)
22–23 ↗(On Diff #421775)

Did you mean cp here and not cmp? If so, I'd move it to where you need it rather than tagging it on the end of this test block, as it makes it look like it is important for the test case it is attached to.

39 ↗(On Diff #421775)

Nit: too many blank lines (1 is enough).

llvm/tools/llvm-dwarfutil/Error.h
26

If you swap the arguments and make Prefix default to empty, you can avoid the strange empty string at several call sites.

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
191

I think it might be more useful in the future to explain with a comment why we can't get here, in case something does manage to slip through and a developer needs to investigate. The error message could probably also do with explaining that it's an internal issue, e.g. "tool unexpectedly did not emit a supported object file".

avl updated this revision to Diff 422029.Apr 11 2022, 1:39 PM

addressed comments.

avl added inline comments.Apr 11 2022, 1:43 PM
llvm/lib/DWARFLinker/DWARFLinker.cpp
437

@clayborg The patch which implements that behavior is D123469.

Just to be clear: I don't have the time, or really the knowledge, to review the bulk of the core behaviour of this change. Someone else will need to review that.

llvm/lib/DWARFLinker/DWARFLinker.cpp
440

Apologies, missed one bit in my previous suggestion.

llvm/test/tools/llvm-dwarfutil/ELF/Inputs/common.yaml
8–13 ↗(On Diff #422029)

Assuming you need a .text section, do you actually need it to have anything other than something like the below?

Name:    .text
Type:    SHT_PROGBITS
Address: 0x1000
Size:    0x10
llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
192–195

Slightly more natural sounding phrasing suggested. Please reflow the comment if needed.

453–454

Could we refactor it out first please? Alternatively, does the behaviour need to be in the initial landing of this patch, or can it be added later?

529

Many tools have the error() function call std::exit(1) to end the program without needing an explicit return EXIT_FAILURE. I'm okay either way with it, but wanted to raise common practice with you here.

avl added inline comments.Apr 12 2022, 1:47 AM
llvm/test/tools/llvm-dwarfutil/ELF/Inputs/common.yaml
8–13 ↗(On Diff #422029)

we need Flags and probably AddressAlign:, but Content might be replaced with Size.

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
453–454

I think it is better to land it in this form. It is useful functionality and it is better to have it from start. As to the refactoring restoreStatOnFile - I am going to create a patch for soon.

529

The reason to not calling std::exit(1) is that error() function is used as a error reporting function for DWARFLinker. DWARFLinker in it`s current state does not stop if any error encountered. As an alternative I can create separate error reporting function for DWARFLinker.

avl updated this revision to Diff 422157.Apr 12 2022, 3:02 AM

addressed comments.

avl added a comment.Apr 12 2022, 11:22 AM
In D86539#3434454, @avl wrote:

Right now if any DWARF section isn't supported, or if a DW_FORM is not supported, this patch will still emit a file and return a non-zero exit status.

So if this patch isn't going to support any of the DWARF sections that are required, it must exit with a valid error message and return a non zero exit status. The things that come to mind are:

  • anything with an older .debug_types section should return an error stating that debug types support is not supported
  • any newer DWARF with type units in the .debug_info section should return an error stating that debug types support is not supported
  • If fission is not supported, we need to return an error if we find any fission related sections (".debug_addr", etc)
  • if any DWARF section is required for debugging (like ".debug_loclists" or ".debug_rnglists") and not getting re-linked by this tool and if those sections become invalid after the DWARF is changed, then we need to return an error
  • if the line tables are too new and the DWARF linker can't update them return an error

All of this stems from Apple not having to support anything but what the Apple compilers emit. Many compilers that emit ELF files and linkers that link ELF have many other options that can be enabled for DWARF. If we aren't going to support them, we can't just emit some warnings and hope the user knows better, we need to return an error and set the exit status to a non zero value. We will need tests to cover these errors as well.

My intention was to limit first version to functionality presented in current dsymutil/DWARFLinker. And to add improvements incrementally. But If we want to have that extended check in the first version then I will add it.

My own preference is to integrate that first version of the tool and add improvements incrementally. f.e. adding such extended check would require changing interface of DWARFLinker. Use "Error DWARFLinker::link()" instead of "bool DWARFLinker::link()". It would be better to make such change separately.

A little background: I am very interested in this patch and I have been trying to get statistics with the resulting binary from this patch so I can evaluate if we want to use it in production on linked binaries produced by our build system. I'd like to see how long it takes to run and also how much smaller debug info becomes. If I run this currently I have no way to know if we are producing anything valid and wether or not I can log the results since we always produce something and have no exit status. I also can't trust the resulting debug info sizes at all since we might be removing all sorts of extra DWARF sections that the existing DWARF linker doesn't handle, or worse yet we just don't clone an attribute and we are missing inline debug info. So it is hard to evaluate this right now as it is. I do think we should try to return errors for files we can't handle though as it would be great if people trying this out could evaluate it and know if it is working for them or if there are things that aren't handled.

@clayborg The D123623 implements displaying error messages for unsupported tables. Would you mind to check it, please?

jhenderson added inline comments.Apr 13 2022, 12:27 AM
llvm/test/tools/llvm-dwarfutil/ELF/Inputs/common.yaml
8–13 ↗(On Diff #422029)

Is 1b significant as a size? Could it be empty? 1? 0x10?

llvm/tools/llvm-dwarfutil/DebugInfoLinker.cpp
233

Canonical style is as suggested.

llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
193
453–454

As to the refactoring restoreStatOnFile - I am going to create a patch for soon.

Not sure if we're talking past each other hum. I'm simply requesting the restoreStatOnFile patch be created and then this patch rebased on top of it, rather than duplicating code that will be resolved by the other patch.

529

I've got no preference, so do whatever you prefer. The one thing I would say: if you've printed error: xxx to stderr, people will expect the the tool to return with a non-zero exit code.

avl added inline comments.Apr 13 2022, 1:20 AM
llvm/test/tools/llvm-dwarfutil/ELF/Inputs/common.yaml
8–13 ↗(On Diff #422029)

It could not be empty. It also should be not less then referenced:

Values:
  - CStr: foo1
  - Value:  0x1000
  - Value:  0x10  <<<<<<<<<
llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
453–454

I understand. This way current patch will depend on the patch for restoreStatOnFile. It looks non practical for me because it would delay review. This restoreStatOnFile is not an important interface thing which will make a lot of problems later. That is why I think it is better to not create patch dependencies and that it might be OK to do both things independently.

But if you prefer to refactor restoreStatOnFile first - will rebase this patch on requested refactoring.

529

that is already done:

inline void error(Error Err, StringRef Prefix = "") {
  handleAllErrors(std::move(Err), [&](ErrorInfoBase &Info) {
    WithColor::error(errs(), Prefix) << Info.message() << '\n';
  });
  std::exit(EXIT_FAILURE);
}
jhenderson added inline comments.Apr 13 2022, 1:33 AM
llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
453–454

I'd always prefer avoiding incurring technical debt like this when the way forward is not particularly complex. Besides, it's not really delaying this patch, as nobody has given the thumbs up to the core behaviour anyway (and it's not something that I'm prepared to do due to my lack of familiarity and time).

529

Yes, I saw, after I made this comment, but left the comment there as it's still relevant (just information now rather than something you need to change).

avl updated this revision to Diff 424594.Apr 22 2022, 1:38 PM

rebased on top of D123821.

jhenderson added inline comments.Apr 24 2022, 11:23 PM
llvm/tools/llvm-dwarfutil/llvm-dwarfutil.cpp
499

Similar to how I suggested the class be renamed to FilePermissionsApplier from FilePermissionsCarrier, I think you should use the word "Applier" in the variable name instead of "Carrier". I also think you should add the "OrErr" suffix, like in BinOrErr, so the new name would be PermsApplierOrErr.

avl updated this revision to Diff 424866.Apr 25 2022, 4:15 AM

renamed variable.

avl added a comment.Apr 26 2022, 3:29 AM

@aprantl @JDevlieghere Could you take a look at this patch, since it affects dsymutil, please?

avl updated this revision to Diff 441782.Jul 1 2022, 1:39 PM

rebased.

avl updated this revision to Diff 441965.Jul 3 2022, 10:20 AM

rebased.

avl added a comment.Jul 4 2022, 11:20 AM

@clayborg Greg, since you`ve already tested this patch, Would you mind setting final approval for this, please?

The dsymutil-specific part is already integrated by separate patches: D125492, D125474, D125469.

The suggestions which you`ve done previously - already have corresponding patches: D123469, D123623.

Also, this patch is reviewed by @jhenderson from the point of view of llvm tools requirements.

clayborg accepted this revision.Jul 7 2022, 2:12 PM

I look forward to using and improving this tool!

This revision is now accepted and ready to land.Jul 7 2022, 2:12 PM

What will this tool do if the "--odr-deduplication" and "--garbage-collection" are not specified?

One other quick suggestion for the "bfd" value for the "--tombstone" option, should we state that the value is zero? Or maybe even change "bfd" to "zero"?

avl added a comment.Jul 8 2022, 11:46 AM

@clayborg @jhenderson Thank you and all for the review!

What will this tool do if the "--odr-deduplication" and "--garbage-collection" are not specified?

The default state for --garbage-collection is enabled.
The default state for --odr-deduplication is enabled.

Thus if above options are not specified the tool would enable them.

One other quick suggestion for the "bfd" value for the "--tombstone" option, should we state that the value is zero? Or maybe even change "bfd" to "zero"?

Following is the description for bfd value from documentation section:

- `bfd`: zero for all addresses and [1,1] for DWARF v4 (or less) address ranges.

Do you think this description should be changed?

In D86539#3639533, @avl wrote:

@clayborg @jhenderson Thank you and all for the review!

What will this tool do if the "--odr-deduplication" and "--garbage-collection" are not specified?

The default state for --garbage-collection is enabled.
The default state for --odr-deduplication is enabled.

Ah, these take a value like "--garbage-collection=0" or "--garbage-collection=1"? Then this is ok.

Thus if above options are not specified the tool would enable them.

One other quick suggestion for the "bfd" value for the "--tombstone" option, should we state that the value is zero? Or maybe even change "bfd" to "zero"?

Following is the description for bfd value from documentation section:

- `bfd`: zero for all addresses and [1,1] for DWARF v4 (or less) address ranges.

Do you think this description should be changed?

We currently have:

=bfd                  -   BFD default value.
=maxpc             -   Max PC values(-1/-2).

I was suggesting just adding something like:

=bfd                  -   BFD default value (0).
=maxpc             -   Max PC values (-1/-2).
In D86539#3639533, @avl wrote:

@clayborg @jhenderson Thank you and all for the review!

What will this tool do if the "--odr-deduplication" and "--garbage-collection" are not specified?

The default state for --garbage-collection is enabled.
The default state for --odr-deduplication is enabled.

Ah, these take a value like "--garbage-collection=0" or "--garbage-collection=1"? Then this is ok.

For user-facing utilities, the convention is --(no-)?foobar. It seems that --(no-)?garbage-collection is implemented in this patch, which is nice.

avl added a comment.Jul 11 2022, 5:26 AM
In D86539#3639533, @avl wrote:

@clayborg @jhenderson Thank you and all for the review!

What will this tool do if the "--odr-deduplication" and "--garbage-collection" are not specified?

The default state for --garbage-collection is enabled.
The default state for --odr-deduplication is enabled.

Ah, these take a value like "--garbage-collection=0" or "--garbage-collection=1"? Then this is ok.

not exactly. These are in form --[no-]garbage-collection.

Thus if above options are not specified the tool would enable them.

One other quick suggestion for the "bfd" value for the "--tombstone" option, should we state that the value is zero? Or maybe even change "bfd" to "zero"?

Following is the description for bfd value from documentation section:

- `bfd`: zero for all addresses and [1,1] for DWARF v4 (or less) address ranges.

Do you think this description should be changed?

We currently have:

=bfd                  -   BFD default value.
=maxpc             -   Max PC values(-1/-2).

I was suggesting just adding something like:

=bfd                  -   BFD default value (0).
=maxpc             -   Max PC values (-1/-2).

Oh, this is about review summary, which is a bit outdated. Previously, It matched with text shown by --help option. But current llvm-dwarfutil shows shorter variant:

--tombstone [bfd,maxpc,exec,universal]
                        Tombstone value used as a marker of invalid address(default: universal)

I will add descriptions for --tombstone values for 'help' output back and make them match with command line guide(
llvm-dwarfutil.rst) and update review summary:

--tombstone [bfd,maxpc,exec,universal]
                        Tombstone value used as a marker of invalid address(default: universal)

  =bfd            -  zero for all addresses and [1,1] for DWARF v4 (or less) address ranges, and exec.
  =maxpc       -  -1 for all addresses and -2 for DWARF v4 (or less) address ranges.
  =exec          - match with address ranges of executable sections.
  =universal   -   Both: bfd and maxpc.
avl updated this revision to Diff 444276.Jul 13 2022, 8:32 AM

added descriptions for --tombstone option values into help output.

avl edited the summary of this revision. (Show Details)Jul 13 2022, 8:33 AM
clayborg accepted this revision.Jul 13 2022, 11:37 AM
This revision was landed with ongoing or failed builds.Jul 19 2022, 1:24 AM
This revision was automatically updated to reflect the committed changes.
thakis added a subscriber: thakis.Jul 19 2022, 8:51 AM
thakis added inline comments.
llvm/tools/llvm-dwarfutil/CMakeLists.txt
7

Since you link to this, do you still need the AllTargets* below? Isn't LLVM_TARGETS_TO_BUILD a superset?

avl added inline comments.Jul 20 2022, 11:56 AM
llvm/tools/llvm-dwarfutil/CMakeLists.txt
7

Indeed. Thank you for the catch! Will remove AllTargets* below.

thakis added inline comments.Aug 3 2022, 6:18 AM
llvm/tools/llvm-dwarfutil/CMakeLists.txt
7

Did you have a chance to try this? Did it not work?

avl added inline comments.Aug 3 2022, 12:53 PM
llvm/tools/llvm-dwarfutil/CMakeLists.txt
7

I was busy with other tasks, sorry. Will do this tomorrow.