It's already possible to run the DWARF verifier on the linked dsymutil output. This patch extends this functionality and makes it possible to run the DWARF verifier on the input as well. A new option --verify-dwarf allows you to specify input, output, all and none. The existing --verify flag remains unchanged and acts and alias for --verify-dwarf=output. Input verification issues do not result in a non-zero exit code because dsymutil is capable of taking invalid DWARF as input and producing valid DWARF as output.
Details
Diff Detail
Event Timeline
llvm/tools/dsymutil/Options.td | ||
---|---|---|
37 | there is an extra dash here: ---verify-dwarf |
llvm/tools/dsymutil/BinaryHolder.cpp | ||
---|---|---|
274 | I have two suggestions/questions: 1. It would probably be better to do input verification inside DWARFLinker library(DWARFLinker::link()). Then all users of the library will be able to use this functionality. 2. Would it be useful to have failure in verification to be fatal error ? | |
llvm/tools/dsymutil/dsymutil.cpp | ||
645 | It makes sense to display message to the user when Options.Verify is ignored because of Options.LinkOpts.NoOutput. Probably verifyOptions() is a good place for it. |
llvm/tools/dsymutil/BinaryHolder.cpp | ||
---|---|---|
274 | I think moving it into the DWARFLinker is a good idea, we already have a DWARFContext there. I don't think we should make input verification failure a fatal error. Depending on the issue dsymutil is capable of generating a valid dSYM from invalid DWARF. |
llvm/tools/dsymutil/BinaryHolder.cpp | ||
---|---|---|
274 |
There is a final set of errors which dsymutil could cure and generate a valid DWARF. The example of such errors is "Overlapping address range". Before dsymutil, there are a lot of address ranges starting from zero(these are address ranges related to deleted code). So input verification would report error "Overlapping address range". dsymutil removes DWARF which uses these address ranges and final DWARF become valid. Other kinds of errors could not be cured by dsymutil. And then it would be useful to fail if input contains such errors. Probably, we could add a parameter to the DWARFContext.verify() - SkipDWARLinkerKnownErrors. So that if verifier encounter only known errors it would report success, otherwise it would report failure and dsymutil would fail during input validation ? It looks useful if dsymutil would fail on input verification if input DWARF contains errors which could not be fixed/correctly processed by dsymutil. |
I really like this patch as it will make detecting LTO issues easier as there are often issues in the incoming .o files.
One thing to think about is we want to only verify unique object files one time. In older LTO based code I remember seeing that same "lto.o" file in the debug map multiple time with many different sources file (N_SO) entries with the same path to the same .o file (N_OSO). If this would happen, would this patch end up running the verifier on the same file over and over?
llvm/tools/dsymutil/BinaryHolder.cpp | ||
---|---|---|
274 | I believe that detection of overlapping address ranges is disabled or .o files since it is known to have unrelocated addresses. | |
llvm/tools/dsymutil/Options.td | ||
45 | If we aren't going to have these errors be fatal, can we possibly add an option that _can_ make verification errors cause it to be? --verify-errors-fatal |
llvm/tools/dsymutil/BinaryHolder.cpp | ||
---|---|---|
274 | probably, yes. My point is that we can ignore errors, which could be cured by dsymutil, and we could stop verifying if the error, which could not be fixed by dsymutil, was met. |
llvm/include/llvm/DWARFLinker/DWARFLinker.h | ||
---|---|---|
275–276 ↗ | (On Diff #407767) | It looks useful to explicitly show that library verifies input DWARF. |
787–788 ↗ | (On Diff #407767) | It looks useful to explicitly show that library verifies input DWARF. |
llvm/lib/DWARFLinker/DWARFLinker.cpp | ||
2643 ↗ | (On Diff #407767) | It is probably better to use DWARFLinker::reportWarning method instead of direct call to WithColor::warning(). |
llvm/test/tools/dsymutil/X86/verify.test | ||
18 | it would be good to test -verify-dwarf=none and -verify-dwarf=bad_value. | |
llvm/tools/dsymutil/LinkUtils.h | ||
33–34 ↗ | (On Diff #407767) | |
llvm/tools/dsymutil/dsymutil.cpp | ||
88 | it is not expected to have a lot of values here. Probably, uint8_t instead of unsigned would be better. | |
91 | All = Input | Output, may be useful later in getVerifyKind(). | |
271 | static_cast<bool>(Options.Verify & DWARFVerify::Input); looks overweight. Probably implement a helper ?: inline bool FlagIsSet (DWARFVerify Flags, DWARFVerify SingleFlag) { return static_cast<uint8_t>(Flags) & static_cast<uint8_t>(SingleFlag); } Options.LinkOpts.Verify = FlagIsSet(Options.Verify, DWARFVerify::Input); |
Thank you Jonas! There are still several inline nits...
llvm/include/llvm/DWARFLinker/DWARFLinker.h | ||
---|---|---|
787 ↗ | (On Diff #407972) | |
llvm/tools/dsymutil/LinkUtils.h | ||
33 ↗ | (On Diff #407972) | |
llvm/tools/dsymutil/dsymutil.cpp | ||
88 | ||
95 | now it is not used and probably might be deleted. | |
100 | now it is not used and probably might be deleted. | |
105 | should start from the small letter. | |
240 |
Most of the patch looks fine, just wanted to make sure we aren't changing it so that "--verify" can't be specified without a now required argument as it would break people's scripts for anyone that is using this flag. We are using on all iOS DWARF output right now at Facebook/Meta. I would rather us add a new "--verify-input" that can be specified, and add leave "--verify" to mean verify the output, or make a "--verify-output" as well and alias "--verify" to that flag.
llvm/tools/dsymutil/dsymutil.cpp | ||
---|---|---|
262 | Is this now changing it so "--verify" no longer works and now requires an additional argument? It would hate to break anyone's scripts that might be currently using the "--verify" option by changing its semantics. |
This will teach me to read all of the posts and the original description before commenting... If everything is as it says in the description this looks good to me.
Yup.
@clayborg Do you still want a flag to make verification issues fatal? I'm happy to put that in a follow up patch.
Actually no, so much DWARF can have issues that might not fully pass verification that I am not sure it would be usable, so lets hold off for now.
it would be good to test -verify-dwarf=none and -verify-dwarf=bad_value.