Submitted.

And thanks a lot for cleaning up, really appreciate it!

I apologize for not tagging appropriately - is that process documented somewhere?

Thanks, yes, I did not intend to delete the else. This only triggers with fuzzing with rather involved inputs, thus I wasn't able to create a nice enough unit test.

I am so sorry, thanks for sending out the patch already and fixing the
layout!

Thank you!! Sorry for forgetting that I needed to do this, <shamecube>

Answering the fundamental design question first.

Remove superfluous include.

Address review comments.

Address review comments.

Address review comments.

Add comment.

Address reviewer comments.

Address review comments.

Undo changes to ownership of initial set of FormatTokens.

Address reviewer comments.

And Ben already fixed it.

Working on a fix.

For non-functional clean-ups generally llvm doesn't require pre-commit review - I did communicate here so people involved in the original change wouldn't miss the clean-up. I do agree that what commits to pre-review is a fine line, and usually try to err on the side of pre-review; I'll take your feedback into consideration for future changes.

I changed it in 49aca00d63e14df8bc68fc4329e6cbc9c9805eb8.

Generally, why do we need to have that much information? I.e. why do we need to know the exact type of the "requires" keyword?
I do understand we need to know the brace type, but that seems like it would be easier to figure out in the TokenAnnotator (where we already parsed UnwrappedLines).
Do we ever parse UnwrappedLines differently depending on requires clauses/expressions?
If not, we should really do the annotation in TokenAnnotator, where we already have nice parsing bounds from the parsed UnwrappedLines.

Hey ho, sorry for the late comment here, but adding peekNextToken(n) is problematic, as this gets in the way of future changes we want to do to handle macros better.
Usually we want to use X = Tokens->getPosition() and FormatTok = Tokens->setPosition(X) pairs when doing look-ahead.
I did a quick attempt at fixing this, but ran into infinite loops later in the annotator :(

In D88299#3804910, @owenpan wrote:

@sstwcw thanks for pointing it out. See D134329.

In D88299#3660779, @nridge wrote:

Thanks! (I was intrigued by Sam's "solves a whole class of clang-format's biggest problems" comment :-))

In D88299#3660772, @nridge wrote:

Does this patch change the formatting behaviour of clang-format?

If so, are there any test cases that show before/after formatting? The MacroCallReconstructorTest unit test seems like it's testing an internal interface.

Address review comments.

Noticed I should have waiting with the renaming of the file until the review is done :( Sorry for the extra confusion.

Work in review comments.

Thanks for cleaning up after me, and sorry for the mess - do y'all have clang-format set up as a presubmit or do you just remember to format manually?

In D69764#2934686, @aaron.ballman wrote:

In D69764#2934612, @MyDeveloperDay wrote:

I was referring to @rsmith and @aaron.ballman (to clarify), both are maintainers in 'clang', the former of which is the 'superset' maintainer of this format project based on its directory. While Aaron is a peer-maintainer to this project, his contributions are immense, and his points are too-well-reasoned and motivated to be dismissed.

Just so we are clear I'm not dismissing anyone opinions, @arron.ballman and I have been going at this quite a bit both on and off list. I have huge respect for these people, (even if the defence of my review it might not seem so). I can't even think to emulate their contributions.

Ideally I'd like their blessing, but alas I fear that might be beyond my capabilities, but if there are things like the RFC that could even go some way to potentially them seeing a way forward that is mutually beneficial so that the door is even open a jar for this then I'm happy to try.

If ultimately the answer is "no" then I need to understand what can be done if anything, if that means a new tool then I'm ok with that as the compromise. Out right dismissal, I would be very sorry to see.

Here are my thoughts put in one place.

0) I like the idea of being able to format qualifier location, and I think clang-format is the tool I would reach for to perform that work
.33) I wish this was generalized to handle all qualifiers rather than just const. (This is not a blocking wish.)
.66) I wish this used "left" and "right" rather than "east" and "west" for user-facing options and documentation. (This is Very Strong wish but I won't block the patch over it.)

1. In general, I do not like the idea of my code formatting tool having opinions about the input source's existing formatting (I think valid code should remain valid). This is the area of concern causing me to ask for an RFC because I'm operating under the impression that not breaking code is (generally) the status quo for clang-format.
2. If the community consensus is that formatting can break code (blanket permission, case by case basis, whatever), I'll definitely abide by that decision. I just want it to be an explicit decision from a wider audience than people who might be following this very long-running patch.
3. The proposed opt-in nature of the check is a blessing and a curse. It alleviates some of the danger (it's not dangerous by default, you have to manually say you want it). But it doesn't eliminate the danger (code can still be broken) and it does raise the question of how often people opt into this sort of thing and whether it's worth the maintenance costs. I don't think any of us can answer those "do people opt in" questions though, so that's not a concern I think we should hold anything up over unless someone has actual data on usage of opt-in features for clang-format. I think opt-in alleviates enough of the danger that it's a worthwhile approach for this patch (and likely may be a good idea for a policy on future changes of the same kind).

tl;dr: if there's an RFC and the community says "breaking changes aren't that big of a deal to us", it tips me from "against" to "in favor" for this functionality.

In D69764#2934032, @MyDeveloperDay wrote:

In D69764#2932929, @steveire wrote:

@steveire, thanks for your comments, I also agree that a second tool shouldn't be needed, especially as this functionality is off by default (and needs to stay that way). This was my suggestion for a compromise.

Its a tough call, but ultimately no on ever gave this a LGTM so I couldn't proceed anyway. Whilst I think we can cover the other qualifiers with a similar approach I don't want to waste more of my time until the fundamental question has been answered if its ok for clang-format to insert/modify the order of tokens and that is more a collective decision.

It was suggested to me that I write up and RFC on the matter, I'm not a massive fan of those as I don't really see a formal process for handling them (the conversation seems to turn into a series of personal preferences then dwindles and dies without conclusion). But if people think it would solve something I guess I could try.

From my perspective I'm more interested in the views of the major clang-format contributors (@curdeius, @krasimir, @owenpan , @sammccall, @HazardyKnusperkeks and people like yourselves who have put time and effort into blogging about these tools), ultimately it will be us who have to handle the impact of this (and maybe the #clangd people) and I don't want to inconvenience them or make work for them.

So here is what is needed to land this in my view:

1. I'd want at least 2 to 3 of the current reviewers to LGTM.
2. I'd want 1 person (like yourself) a community contributor to also give their approval (I guess I have that conceptually from you!)

or

1. @klimek or @djasper to give us an its ok for "clang-format to insert/modify the order of tokens" in a controlled manner decision
2. 1 LGTM

Whilst I respect the views of those few who have objected, they don't to my knowledge make significant contributions to clang-format (not in the last 5-6 years I've been following it anyway)

I feel this team owns the decision as we are the ones that look after it, warts and all. We'll always have a situation where some don't agree, thats ok, but I feel there is more agreement that this funcationality is ok rather than not.

Until one of those 2 things happens, we are in limbo.

Can you explain the change and the before/after a bit more? Thanks!

Isn't the comment incorrect after this patch?

Adapted based on review comments.

lg

Nice catch, thanks! LG!

In D83296#2299062, @nridge wrote:

What does this change mean for users of clang-format -- better formatting of complicated (e.g. multi-line) macro invocations?

In D84306#2294952, @MyDeveloperDay wrote:

Which bit do you find more complex? adding something to the FormatToken or the use of the is() calls?

FWIW, finding this due to seeing the added complexity. Do you have data on what the difference is on clang-format for either overall memory use or performance? Thanks!

Worked in review comments.

Addressed code review comments.

In D83296#2146970, @sammccall wrote:

In D83296#2146870, @klimek wrote:

Monday-morning ping.

Thanks for the reminder here... however this is taking me a bit to get my head around, and we've got a release branch cut scheduled for a couple of days that we're trying to polish for.
AFAICT there's significant followup work still needed to make use of this - are you wanting this to land in the 11 release? Else i'd probably come back to this after the cut...

Monday-morning ping.

@djasper wrote this iirc, but I doubt he'll have much time to look into this :)

Address review comment.

lg