[clang-format] Add more support for C# 8 nullables

[exv]

Summary

This adds support for the null-coalescing assignment and null-forgiving
operators.

https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/null-coalescing-operator

https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/null-forgiving

[MyDeveloperDay]

Thank you for the patch

clang/lib/Format/TokenAnnotator.cpp

3170

should this honour SpaceBeforeAssignmentOperators ?

[exv]

Remove special whitespace handling for ??=

clang/lib/Format/TokenAnnotator.cpp

3170

Good point. Looking more closely, since I have called setKind(tok::equal) on the operator, it seems that the rest of clang-format reliably understands it as an assignment operator and formats it correctly if I don't intervene. I have updated the test case to test this behavior.

[exv]

Fix incorrect arc usage

[exv]

Fixing arc, hopefully?

[exv]

Hey all, I'm really sorry for the noise, I screwed up my arc command to revise the submission.

[HazardyKnusperkeks]

clang/unittests/Format/FormatTestCSharp.cpp

365

Why do you escape ?? I've never seen that.

[exv]

clang/unittests/Format/FormatTestCSharp.cpp

365

Apparently, ??= is a trigraph for "#" in C, so it must be escaped:

https://en.wikipedia.org/wiki/Digraphs_and_trigraphs#C

To be honest, I'd never heard of this bizarre language "feature" until I wrote this patch.

[curdeius]

Looks very good already. Just some questions and test requests.

Requesting changes so it shows up nicely in the revision queue.

clang/lib/Format/TokenAnnotator.cpp

1578

Does adding false and true here may negatively affect JS in any way? (Just thinking out loud)

3168

Should it be renamed to include C#?

clang/unittests/Format/FormatTestCSharp.cpp

365

Not only in C but in C++ too. Fortunately removed in C++17.
Nobody misses trigraphs...

380

Please test also true! or !true!.

381

Could you test that ! is correctly kept together with the corresponding identifier around the line breaks, please?
You might adjust ColumnLimit for that.

[exv]

• [clang-format] Add more support for C# 8 nullables
• Fix null-coalescing assignment whitespace behavior
• Incorporate feedback from curdeius
• Combine JS and CS types/parsing

clang/lib/Format/TokenAnnotator.cpp

1578

Nothing I can see: https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-0.html#non-null-assertion-operator

3168

Now that you've mentioned it, I do see that JS also defines an almost duplicate set of null-related operators, including null-propagating, null-coalescing and null-coalescing assignment operators, and clang-format handles them with a different code path than C# 😅. They do have slightly different names, but the way they should be parsed and interpreted is the same, and I think it makes sense to use the same logic and token definitions for both of these languages. This resulted in a lot of special-handling for C# tokens that wasn't necessary being removed too.

However, renaming them would introduce a lot of noise, so I think I would prefer to do that in a separate commit.

[curdeius]

LGTM. That's great. Thanks for working on this and for the cleanup.

clang/lib/Format/FormatTokenLexer.cpp

102

Could you explain in what case the operator precedence may be wrong here?

clang/lib/Format/TokenAnnotator.cpp

3168

Fair enough.

[exv]

• Remove obsolete comment

[exv]

According to the wiki, it seems like someone who has commit access must now submit this change.

clang/lib/Format/FormatTokenLexer.cpp

102

It's unlikely this comment is still relevant for this section. I will delete it.

[curdeius]

In D101702#2736886, @exv wrote:

According to the wiki, it seems like someone who has commit access must now submit this change.

Yes, I'll land it. Please provide "Name <mail@address>" for commit attribution if it differs from the info in your Phabricator account.

[exv]

Eliza Velasquez <[e-mail]>

[exv]

I have completed two other formatting fixes related to C# 8 nullables since my last commit. Should I create a new review request for those or update this one?

[curdeius]

Preferably a new one unless it's something really minor.

[curdeius]

Could you please rebase to re-run the CI (that failed for sum unrelated reason)?

[krasimir]

clang/lib/Format/TokenAnnotator.cpp

1570

nit: s/isIdentifier/IsIdentifier/ (LLVM style uses CamelCase for local variables)

clang/unittests/Format/FormatTestCSharp.cpp

365

nit: you could use raw string literals for the ??= test cases (e.g., R"(test ??= ABC;)").

[krasimir]

clang/lib/Format/FormatTokenLexer.cpp

135

nit: use spaces for indentation, no tabs (Phabricator makes it look this line starts with a tab character)

[jbcoe]

Thanks for this patch!

[HazardyKnusperkeks]

clang/lib/Format/FormatTokenLexer.cpp

135

That's not a tab, I fell for that one too once. It just visualizes that there was only whitespace added.

[exv]

• Incorporate krasimir's feedback

[exv]

Rebased and comments addressed!

[exv]

Rebase again in an attempt to fix CI