This is an archive of the discontinued LLVM Phabricator instance.

Macro Debug Info support in Clang
ClosedPublic

Authored by aaboud on Jan 13 2016, 12:34 AM.

Diff Detail

Repository
rL LLVM

Event Timeline

aaboud updated this revision to Diff 44715.Jan 13 2016, 12:34 AM
aaboud retitled this revision from to Macro Debug Info support in Clang.
aaboud updated this object.
aaboud added reviewers: dblaikie, rsmith, aprantl, probinson.
aaboud added a subscriber: cfe-commits.
aaboud added inline comments.Jan 13 2016, 12:43 AM
include/clang/AST/ASTConsumer.h
163 ↗(On Diff #44715)

Richard,
I know that you suggested not to introduce the Preprocessor in anyway to the ASTConsumer.
However, as I could not understand the other way to support macros in Clang without applying a huge redesign, I decided to upload the code I suggested in the proposal.
http://lists.llvm.org/pipermail/llvm-dev/2015-November/092449.html

If you still think that this approach is not the right one, I would appreciate it if you can explain again how to implement the macro support differently.

aprantl edited edge metadata.Jan 13 2016, 9:14 AM

Added a few stylistic comments and request for more documentation.

lib/CodeGen/MacroPPCallbacks.cpp
1 ↗(On Diff #44715)

Please run your patch through clang-format.

62 ↗(On Diff #44715)

Comment why SkipFiles is initialized to 2?

74 ↗(On Diff #44715)

Comments?

91 ↗(On Diff #44715)

Please add a comment explaining this condition.

lib/CodeGen/MacroPPCallbacks.h
34 ↗(On Diff #44715)

Please comment the purpose of these fields.

45 ↗(On Diff #44715)

Since r237417 LLVM enabled autobrief, so it's no longer recommended to use \brief for one-liners.

aaboud updated this revision to Diff 46034.Jan 26 2016, 12:41 PM
aaboud edited edge metadata.
aaboud marked 6 inline comments as done.

Added comments explaining the implementation.

aprantl added inline comments.Feb 1 2016, 9:43 AM
lib/CodeGen/CGDebugInfo.cpp
2099 ↗(On Diff #46034)

By using a bool for IsUndef, we'll end up with somewhat ugly call sites like

DI->CreateMacro(Parent, /*IsUndef*/ true, ...);

so we might as well just pass in the dwarf constant

DI->CreateMacro(Parent, llvm::dwarf::DW_MACINFO_undef, ...);

directly.

aaboud updated this revision to Diff 47997.Feb 15 2016, 10:11 AM
aaboud marked an inline comment as done.

Applied Adrian comment.

rs added a subscriber: rs.Jul 22 2016, 10:16 AM

Hi Amjad,

are you still planning on getting this patch and https://reviews.llvm.org/D16077 committed ? It looks like these two patches are final pieces in the puzzle to get macro information in the DWARF debug output.

I've downloaded the diffs and applied them myself on my local checkout and they seem to work fine. If you would like me to upload the rebased patches onto phabricator to save you the trouble of having to the fix conflicts downstream then let me know.

Thanks,
Ranjeet

aaboud updated this revision to Diff 65551.Jul 26 2016, 10:28 AM
aaboud added reviewers: erichkeane, bwyma.

Updated patch to top of trunk (r276746) - Thanks to Ranjeet Singh.

Please, provide your feedback.

rsmith added inline comments.Nov 3 2016, 5:23 PM
include/clang/AST/ASTConsumer.h
163 ↗(On Diff #44715)

Don't add this to ASTConsumer; an AST consumer and a PP consumer are two largely separate ideas. Instead, you could create and register a PPCallbacks object directly from CodeGenAction::CreateASTConsumer. In fact, if you look there, a PPCallbacks subclass is already registered (for code coverage); you can do something similar to register your MacroPPCallbacks object.

lib/CodeGen/CMakeLists.txt
76 ↗(On Diff #65551)

This file is missing from the diff.

lib/CodeGen/CodeGenModule.cpp
28 ↗(On Diff #65551)

This file is missing from the diff.

aaboud added a comment.Nov 3 2016, 5:47 PM

Thanks Richard for reviewing the patch and for the comments.
Please, see answers below.

Regards,
Amjad

include/clang/AST/ASTConsumer.h
163 ↗(On Diff #44715)

Thanks, I will try this approach, it is indeed cleaner.

lib/CodeGen/CMakeLists.txt
76 ↗(On Diff #65551)

Sorry about that, it seems when I updated the code these files were removed from the patch.
Also, I need to improve the implementation in these two files.

By the way, you can find an old version of them in the previous diff.

I will update them and upload them to the review together with the above suggestion of yours.

aaboud updated this revision to Diff 82696.Dec 29 2016, 2:13 PM

Improved code based on Richard's comments.
Removed the change to ASTConsumer, now it does not need to know anything about PP consumer.
However, I still needed to change the CodeGenerator ASTConsumer to return the CGDebugInfo class.

Richard, please review the new patch.
This time I made sure to add the "MacroPPCallbacks.*" new files.

Richard, can you please review the new patch?

Kindly reminder, I am still waiting for a review on this new patch.

Thanks

mkuper resigned from this revision.Jan 30 2017, 4:23 PM

I admit it would be nice if someone reviewed this, but I'm really the wrong person for this code area. :-)

aprantl added inline comments.Jan 31 2017, 9:54 AM
lib/CodeGen/MacroPPCallbacks.cpp
31 ↗(On Diff #82696)

std::next() ?

44 ↗(On Diff #82696)

LLVM style prefers comments to be full sentences and above the code.

83 ↗(On Diff #82696)

I think there is an LLVM_FALLTHROUGH macro (or something to that end)

154 ↗(On Diff #82696)

#include as opposed to #import? If so, why?

62 ↗(On Diff #44715)

Thanks. I think this comment should now go into the header and be promoted to the Docygen comment for the CommandIncludeFiles. Also consider doing CommandIncludeFiles = 0 in the class definition.

lib/CodeGen/ModuleBuilder.cpp
68 ↗(On Diff #82696)

What does Ref stand for?

rsmith added inline comments.Jan 31 2017, 1:37 PM
include/clang/CodeGen/ModuleBuilder.h
71 ↗(On Diff #82696)

Typo 'calss'

72 ↗(On Diff #82696)

What is the returned value in that case? A reference to a null pointer?

Generally, I'm not particularly happy with this approach of exposing a reference to an internal pointer as the way of propagating the CGDebugInfo to the macro generator as needed. Instead, how about giving the MacroPPCallbacks object a pointer to the CodeGenerator itself, and having it ask the CodeGenModule for the debug info object when it needs it?

73 ↗(On Diff #82696)

caller -> the caller's

74 ↗(On Diff #82696)

start using -> starting to use

lib/CodeGen/MacroPPCallbacks.cpp
1 ↗(On Diff #82696)

Wrong file header (should be .cpp, and you don't need the "-*- C++ -*-" in a .cpp file.

aaboud marked 7 inline comments as done.Feb 1 2017, 10:25 AM

Thanks Adrian and Richard for the comments, I appreciate that.
I am uploading a new patch that address most of the comments.

Adrian, I answered some of your comments below.

Thanks,
Amjad

lib/CodeGen/MacroPPCallbacks.cpp
31 ↗(On Diff #82696)

I am not that satisfied with this function, and I hope we can do it in a better way.
This function is based on a code I took from this a static function in file "lib\Frontend\PrintPreprocessedOutput.cpp":

/// PrintMacroDefinition - Print a macro definition in a form that will be
/// properly accepted back as a definition.
static void PrintMacroDefinition(const IdentifierInfo &II, const MacroInfo &MI,
                                 Preprocessor &PP, raw_ostream &OS) {
44 ↗(On Diff #82696)

As I said above, this code is taken from "lib\Frontend\PrintPreprocessedOutput.cpp"
Also, I ran clang-format on it, and it did not suggest to change this line.

But, I do not mind fixing this, if it is the preferred comment style.

154 ↗(On Diff #82696)

I am not familiar with "#import" keyword.

  1. Is it handled differently by clang that #include?
  2. Do we need to generate macro debug info for it?

The reason I have this switch below, is that I am not sure if we call this "InclusionDirective" callback for identifiers that are not "#include".

However, I see no harm of updating the LastHashLoc every time we enter this function. The value will not be used unless it is set just before calling "FileChanged"callback.

lib/CodeGen/ModuleBuilder.cpp
68 ↗(On Diff #82696)

Ref - stands for reference, I think I do not need the "Ref" prefix for this member, only for the get function.
But, I will follow Richards suggestion that will eliminate the need for this member.

aaboud updated this revision to Diff 86669.Feb 1 2017, 10:26 AM
aaboud marked an inline comment as done.

Address Adrian and Richard comments.

Please, review the new patch and let me know if I need to change anything else.

Thanks,
Amjad

Thanks! Couple more inline comments.

lib/CodeGen/CGDebugInfo.h
415 ↗(On Diff #86669)

It would be good to be a bit more descriptive here. If the comment just repeats the name of the function it is not worth keeping,
/// Create debug info for a macro (definition/use/?).

420 ↗(On Diff #86669)

E.g.,
/// Create debug info for a file referenced by an #include directive.

lib/CodeGen/MacroPPCallbacks.cpp
88 ↗(On Diff #86669)

I think the entire function can be replaced with

if ((Status == MainFileScope) ||
    (Status == CommandLineScopeIncludes && CommandIncludeFiles))
  return Loc;
return SourceLocation();
100 ↗(On Diff #86669)

Could you try splitting this function up into two functions?

MacroPPCallbacks::FileEntered()
MacroPPCallbacks::FileExited()

and the common code factored out into a helper? The control flow here seems unnecessarily complex with all the flags.

177 ↗(On Diff #86669)

calculat*e*MacroDefinition
(is there a better word than calculate?)

44 ↗(On Diff #82696)

I think it would be a good idea to clean this up. Feel free to either do this now or in a separate commit.

154 ↗(On Diff #82696)

#import comes from Objective-C. It is equivalent to #include + #pragma once.
I'm curious whether this comment means that it isn't handled?

aaboud added a comment.Feb 1 2017, 2:17 PM

Thanks Adrian for the fast response.
See some answers below.

I will update the code and upload a new patch soon.

lib/CodeGen/MacroPPCallbacks.cpp
88 ↗(On Diff #86669)

Sure, will do.
The reason I have a switch, because I had more cases during the implementation that I get rid of.

100 ↗(On Diff #86669)

Will try restructure.
The way I implemented this is by having stages (parsing levels).
The control flow is to identify the current stage and decide how to continue from there.

I will see what I can do to make the code much clear.

177 ↗(On Diff #86669)

How about:
GenerateMacroDefinition
PrintMacroDefinition
ParseMacroDefinition

Do you think any of the above would be better choice?

154 ↗(On Diff #82696)

I will remove the switch and update the LastHashLoc unconditionally.
I did not check Objective-C debug info, but I assume it will work fine once I change these lines.

aaboud updated this revision to Diff 87020.Feb 3 2017, 2:03 PM
aaboud marked 4 inline comments as done.

Addressed Adrian's comments.
Please, let me know if I missed any.

Also, I improved the code and added explanation about the file inclusion structure that is being handled in the macro callbacks.

Few more inline comments.
I think the code itself is looking good now. Why doesn't this patch include any test cases? I would have expected some that exercise the command line include handling, etc., ...

lib/CodeGen/MacroPPCallbacks.h
35 ↗(On Diff #87020)

s/was/were/

37 ↗(On Diff #87020)

What about EnteredCommandLineIncludes or EnteredCommandLineIncludeFiles?

aaboud updated this revision to Diff 87408.Feb 7 2017, 5:59 AM
aaboud marked 2 inline comments as done.

Addressed Adrian last comments.
Added a LIT tests that covers all the macro kinds:

  1. built-in (define)
  2. command-line (define, include, undef)
  3. main source (define, include, undef)

Checked the above with and without PCH include.

Notice, that current implementation does not support debug info for macro definition and inclusion generated during the PCH file creation.
To support that, we need to extend the PCH format to preserve this information.
If you believe this feature is important, I will open a bugzilla ticket and we can handle it separately.

Addressed Adrian last comments.
Added a LIT tests that covers all the macro kinds:

  1. built-in (define)
  2. command-line (define, include, undef)
  3. main source (define, include, undef)

Checked the above with and without PCH include.

Notice, that current implementation does not support debug info for macro definition and inclusion generated during the PCH file creation.
To support that, we need to extend the PCH format to preserve this information.
If you believe this feature is important, I will open a bugzilla ticket and we can handle it separately.

That would be a good idea and important to fix. I don't think we want the the generated code (or the debug info for that matter) to change just because the user decides to switch to PCH. Ideally turning on PCH should be transparent.

I have one final question: What's the impact of this change on, e.g., building clang?

  • How much does the build directory grow?
  • Is there any noticeable compile time regression?
test/CodeGen/debug-info-macro.c
14 ↗(On Diff #87408)

Alternatively, you may be able to use #line to force an easily recognizable line number (unless that messes with the macro debug info, or you can use FileCheck's [[@LINE-offset]] feature.

Addressed Adrian last comments.
Added a LIT tests that covers all the macro kinds:

  1. built-in (define)
  2. command-line (define, include, undef)
  3. main source (define, include, undef)

Checked the above with and without PCH include.

Notice, that current implementation does not support debug info for macro definition and inclusion generated during the PCH file creation.
To support that, we need to extend the PCH format to preserve this information.
If you believe this feature is important, I will open a bugzilla ticket and we can handle it separately.

That would be a good idea and important to fix. I don't think we want the the generated code (or the debug info for that matter) to change just because the user decides to switch to PCH. Ideally turning on PCH should be transparent.

I have one final question: What's the impact of this change on, e.g., building clang?

  • How much does the build directory grow?
  • Is there any noticeable compile time regression?

I will run self-build and come back with answers for these questions.
I am just wondering, if I build in debug mode, will it use the flag "-fstandalone-debug"?
This flag is set by default only for DarwinClang and FreeBSD, is there a way to assure it will be used with self-build?
Notice that without this flag, there will be no debug info macro generated.

aaboud added a comment.Feb 8 2017, 2:25 PM

How much does the build directory grow?
Is there any noticeable compile time regression?

I build clang in release mode using GCC, then used that build to build clang in debug mode with "-fstandalone-debug" flag, one time with my changes and another time without my changes.

Without my changes

Clang executable Size: 1,565MB
Build time: 17m57.504s

Withmy changes

Clang executable Size: 2,043MB
Build time: 17m58.008s

Do, build time is the same, but binary size was increased about 30%.
However, remember that we are emitting macro debug info in dwarf4 format, once we support emitting it in dwarf5 format the size should be reduced significantly.

I'm not sure it makes sense to motivate this feature with "-fstandalone-debug" flag.

Is "-fmacro-debug" flag sounds good?
Should we extend DebugInfoKind enumeration to support the debug info macro? Or we should add a new option to CodeGenOptions?

aaboud updated this revision to Diff 87796.Feb 9 2017, 5:52 AM

Added flag to control macro debug info generation:

  1. -fdebug-macro (-fno-debug-macro) for Clang driver
  2. -debug-info-macro for Clang compiler (-cc1)

Also, made sure FE will discard this flag when debug info is not required, i.e. with -g0.
Should we discard the macro debug info when "-gline-tables-only" is used? or it is up to the user not to use "-fdebug-macro" in this case?

I also extended the test to check all combinations of using -debug-info-macro flag with -debug-info-kind flag.

aaboud added a comment.Feb 9 2017, 6:03 AM

How are you measuring the build time? Total time for, say "ninja clang" with full parallelism? That'd be hard to measure the actual impact (since it could be the link time or other things are dominating, etc). If you have a reliable way to time (I'm assuming Intel has lots of tools for measuring compiler performance) the compilation, get a sense of the variance, etc (& link time, to get a sense of how much the larger inputs affect linker performance) would be useful.

I did a manual measurement, but I believe it is representative.
I used "time" tool to measure the whole time of the build (compile + link).

  • time make -j8

Also, notice that I limited the targets to X86 only, so it was not a full build of LLVM.

  • -DLLVM_OPTIMIZED_TABLEGEN=ON -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_CXX_FLAGS="-fstandalone-debug" -DCMAKE_C_FLAGS="-fstandalone-debug"

However, as I already said, it should answer the time change and size question fairly enough.

Does GCC have a command line option for this that we could mirror?

GCC emits macro debug info when when build with -g3.

Please also add driver testcase (e.g., to test/Driver/debug-options.c).
Technically clang does accept -g3 as an option, so we could wire it up for compatibility. I would still keep the -fmacro-debug driver option though.

docs/UsersManual.rst
1692 ↗(On Diff #87796)

Debug info for C preprocessor macros ...

1697 ↗(On Diff #87796)

Generate debug info for preprocessor macros. ...

aaboud updated this revision to Diff 87819.Feb 9 2017, 9:05 AM
aaboud marked 2 inline comments as done.

Added test cases for driver debug info macro flags in test/Driver/debug-options.c
Applied some changes to the documentation suggested by Ardian.

aprantl accepted this revision.Feb 9 2017, 9:14 AM

Could you also add support for -g3? It looks like we are supporting -g0 and -g1, so we should probably also mirror -g3.

Thanks a lot for all your work!

This revision is now accepted and ready to land.Feb 9 2017, 9:14 AM
This revision was automatically updated to reflect the committed changes.
dlj added a subscriber: dlj.Feb 9 2017, 4:05 PM
dlj added inline comments.
cfe/trunk/lib/CodeGen/MacroPPCallbacks.cpp
125

As a heads up... this fails under -Werror:

llvm/tools/clang/lib/CodeGen/MacroPPCallbacks.cpp:125:3: error: default label in switch which covers all enumeration values [-Werror,-Wcovered-switch-default]

default:
^

1 error generated.