[clang-format] Treat AttributeMacros more like __attribute__

[jaredgrubb]

Summary

I noticed that clang-format was inserting some strange indentation whenever I used custom "attribute-like macros"
(things like FOO_EXTERN to wrap attribute-visible-default, or macros with parentheses like NS_SWIFT_NAME(...)).

There are two parts to this fix:

• Annotate the paren after an AttributeMacro as an AttributeLParen.
• Treat an AttributeMacro-without-paren the same as one with a paren.

I added a new test-case to differentiate a macro that is or is-not an AttributeMacro; also handled whether ColumnLimit is set to infinite (0) or a finite value, as part of this patch is in ContinuationIndenter.

There may be other places that need to handle TT_AttributeMacro better, but this is at least a step in the right
direction.

Closes https://github.com/llvm/llvm-project/issues/68722.

[HazardyKnusperkeks]

Our tests reside in clang/unittests/Format, you want to look into the TokenAnnotatorTests. In fact I can't read the tests you wrote there.

And could you please provide an example what you want to format and where the misformatting (or apparently misannotation) happens?

[jaredgrubb]

For background, the current clang-format results in the following (-style="{BasedOnStyle: LLVM, ColumnLimit: 0, AttributeMacros: [MACRO]}):

MACRO MACRO(A)
    @interface Foo
@end

MACRO(A)
MACRO
    @interface Foo
@end

This patch improves it (removes the indention and makes both cases have same wrapping):

MACRO MACRO(A)
@interface Foo
@end

MACRO(A) MACRO
@interface Foo
@end

[jaredgrubb]

I wasn't sure about testing (this is my first patch!) and the test-case I did was in clang/test/Format .. I can look at clang/unittests/Format and see how to model something like it.

If I do that, would that be in-addition-to or instead-of the test-case I included already?

[owenpan]

In D145262#4168058, @jaredgrubb wrote:

I wasn't sure about testing (this is my first patch!) and the test-case I did was in clang/test/Format .. I can look at clang/unittests/Format and see how to model something like it.

If I do that, would that be in-addition-to or instead-of the test-case I included already?

Instead of.

[jaredgrubb]

Create unit-tests for the patch (and remove the proposed non-unit test "test").

[HazardyKnusperkeks]

clang/unittests/Format/FormatTestObjC.cpp

1528

So there is most likely a CanBreak not true? That happens also in the TokenAnnotator.

clang/unittests/Format/TokenAnnotatorTest.cpp

1321

Your bug is in regard to Objective-C, right? Maybe add a test for that here too?

[jaredgrubb]

• Fixed an issue in TokenAnnotator about it not breaking between macros properly (it was catching in an ObjC selector-check too early)
• Add more ObjC tests, covering method and property declarations too. There are still some quirks about reflowing multiple attributes, but those quirks exist in C++ too, so I think those are best left for another patch. I added checks for existing behavior so that patch can improve the ObjC version too.

[jaredgrubb]

clang/lib/Format/TokenAnnotator.cpp

5327–5328

I broke this if into two pieces and restored an original return true for this part of the if that was changed in 2020 (via 5a4ddbd69db2b0e09398214510501d0e59a0c30b).

The return !Left... was added to avoid breaking [[ apart when reflowing a function-arg with C++11-style attribute (eg, [[unused]]). As far as I can tell, this check _really_ only matters if the code-path is the second-half of the original if (Right.is(tok::l_square)...).

I couldn't imagine how there Right.is(tok::kw___attribute) needs the Left-check, so I think that patch is better split to differentiate (even though originally it didn't really have any conflicting effect).

Please let me know if my logic here is wrong (or I'm not making sense :) )

[jaredgrubb]

clang/unittests/Format/FormatTestObjC.cpp

1619

I don't love this FIXME, but I was afraid to add more to this patch, as fixing this will require digging into things that have nothing to do with __attribute__ vs AttributeMacros.

For example, suffix macros in C/C++ also are broken in the same way with just plain __attribute__. For example, for ColumnWidth: 50:

int f(double) __attribute__((overloadable))
__attribute__((overloadable));

int ffffffffffffffffffffffffffffff(double)
    __attribute__((overloadable))
    __attribute__((overloadable));

I think fixing reflowing of suffix macros is best done in another PR (which I can take a stab at!)

[owenpan]

clang/lib/Format/TokenAnnotator.cpp

4384

clang/unittests/Format/FormatTestObjC.cpp

1682–1693

clang/unittests/Format/TokenAnnotatorTest.cpp

1303–1309

I would delete it.

1331–1338

Ditto.

1360–1367

Ditto.

1389–1396

Ditto.

[jaredgrubb]

Address review comments from @owenpan

[jaredgrubb]

I have uploaded patch for all the points you called out. (Sorry for delay; I missed the suggestions earlier!)

[MyDeveloperDay]

clang/lib/Format/ContinuationIndenter.h

231 ↗

(On Diff #516211)

If you want this to look like code you need to use \code \endcode

[MyDeveloperDay]

clang/lib/Format/ContinuationIndenter.h

231 ↗

(On Diff #516211)

Ok my bad this is the ContinuationIndenter.h, not Format.h

[jaredgrubb]

Address review feedback about code/endcode. Otherwise the patch is the same and I hope ready for merge? I'd love to get a green check :)

[HazardyKnusperkeks]

clang/lib/Format/TokenAnnotator.cpp

4384–4385

5328

Does changing this return value make no difference? In other words is there no combination of Left.is(TT_AttributeSquare) and Right.is(tok::kw___attribute)?

[jaredgrubb]

clang/lib/Format/TokenAnnotator.cpp

5328

Yes, that combo can happen; example below.

The patch that changed the left-diff line from true to !Left.is(TT_AttributeSquare)
was done _only_ contemplating the [[ case (5a4ddbd69db2b0e09398214510501d0e59a0c30b); tagging @MyDeveloperDay who wrote that patch and can perhaps offer more insight on your question.

My reasoning is to revert that part of the patch just a bit and limit it to that case only.

I couldn't come up with a use-case where you'd want to avoid splitting between TT_AttributeSquare and kw___attribute, but the example below shows that allowing it to break in that combination is preferable to the alternative of breaking in the parens of the attribute:

// Style: "{BasedOnStyle: LLVM, ColumnLimit: 40}"
// Existing Behavior
int ffffffffffffffffffffffffffffff(
    double)
    __attribute__((overloadable))
    [[unused]] __attribute__((
        overloadable));

// With Patch
int ffffffffffffffffffffffffffffff(
    double)
    __attribute__((overloadable))
    [[unused]]
    __attribute__((overloadable));

[HazardyKnusperkeks]

clang/lib/Format/TokenAnnotator.cpp

5328

Thanks for the detailed answer. I'll wait for @MyDeveloperDay. You are right, it's prettier with the patch, but on the other hand it is not desirable to change the formatting (apart from fixing bugs) between versions.

[jaredgrubb]

clang/lib/Format/TokenAnnotator.cpp

5328

This part of the patch isn't necessary so let me revert it so it doesn't slow this patch down.

But, the unit tests don't even fail when I revert it -- which means I really should treat this part as its own change with a unit test that matters.

[jaredgrubb]

Address review comments:

• remove redundant &&
• remove part of patch that was not tested by any test and should be its own patch on its own merits

[jaredgrubb]

Rebased and added a line to Release Notes, no other changes. Have requested this to be merged if all builds are green (since this patch has been approved).

[owenpan]

clang/lib/Format/ContinuationIndenter.h

231 ↗

(On Diff #540477)

Unrelated.

234 ↗

(On Diff #540477)

Ditto.

clang/lib/Format/TokenAnnotator.cpp

5327–5332

I'd delete the comment on line 5519.

clang/unittests/Format/FormatTestObjC.cpp

1619

Half of the test cases passed before this patch but now would fail with this patch. That is, this patch would generate regressions.

[jaredgrubb]

Hopefully this will satisfy and we can merge! I didn't notice the Phabricator timeline and I'm hoping we can finish this before Oct 1.

Rebased.
Removed extraneous doc comment not related to patch (as requested).
Removed comment (as requested).
Added a #if 0 around some cases that should be fixed one day (as requested prior but I missed one of the chunks).

[jaredgrubb]

clang/unittests/Format/FormatTestObjC.cpp

1619

Just saw this comment. Yes, 3 of these did pass, but lots more in this file do NOT pass. I understand the desire to not "regress", but this patch improves so many other examples (as documented in these test cases). I can pick some of the worst if it helps, but otherwise, I'm not sure what I can do to address your comment.

[owenpan]

This looks OK to me. Please rebase after https://github.com/llvm/llvm-project/pull/67518 is merged.

In D145262#4650300, @jaredgrubb wrote:

Hopefully this will satisfy and we can merge! I didn't notice the Phabricator timeline and I'm hoping we can finish this before Oct 1.

The new deadline is Nov 15 according to https://discourse.llvm.org/t/update-on-github-pull-requests/71540/125.

clang/docs/ReleaseNotes.rst

474–475 ↗

(On Diff #557274)

Can you delete this as release notes are usually for new options?

clang/lib/Format/TokenAnnotator.cpp

5327–5329

clang/unittests/Format/FormatTestObjC.cpp

1619

I'm okay with fixing them in another patch.

[owenpan]

In D145262#4651483, @owenpan wrote:

This looks OK to me. Please rebase after https://github.com/llvm/llvm-project/pull/67518 is merged.

I have merged the PR. Commits cef9f40cd4fe and 151d0a4db321 should make this patch less brittle.

[jaredgrubb]

Address review comments, and adjusting the patch to address other merges since this patch was started.

[owenpan]

clang/lib/Format/ContinuationIndenter.cpp

1230–1234

Basically, insert the first two lines and undo the changes to the last three lines.

clang/lib/Format/TokenAnnotator.cpp

4089–4090

Please also add a test case for this.

4383

Please delete this comment, which IMO doesn't quite match the code below.

[owenpan]

Can you add a github issue to show the behavior before this patch?

[jaredgrubb]

Addressed review feedback from Oct2 (@owenpan).
A new issue on Github was opened for this enhancement (https://github.com/llvm/llvm-project/issues/68722)

[owenpan]

LGTM

[owenpan]

@jaredgrubb please provide your authorship for git commit --author so that we can land the patch on your behalf.

[jaredgrubb]

In D145262#4653670, @owenpan wrote:

@jaredgrubb please provide your authorship for git commit --author so that we can land the patch on your behalf.

The best to use would be:

Jared Grubb <[e-mail]>

Thanks so much for your help on this!!

clang/lib/Format/ContinuationIndenter.cpp

unsigned ContinuationIndenter::getNewLineColumn(const LineState &State) {
   }
   if (NextNonComment->isOneOf(TT_CtorInitializerColon, TT_InheritanceColon,
                               TT_InheritanceComma)) {
     return State.FirstIndent + Style.ConstructorInitializerIndentWidth;
   }
   if ((PreviousNonComment &&
        (PreviousNonComment->ClosesTemplateDeclaration ||
         PreviousNonComment->ClosesRequiresClause ||
         PreviousNonComment->isOneOf(
-            TT_AttributeParen, TT_AttributeSquare, TT_FunctionAnnotationRParen,
-            TT_JavaAnnotation, TT_LeadingJavaAnnotation))) ||
+            TT_AttributeParen, TT_AttributeMacro, TT_AttributeSquare,
+            TT_FunctionAnnotationRParen, TT_JavaAnnotation,
+            TT_LeadingJavaAnnotation))) ||
       (!Style.IndentWrappedFunctionNames &&
        NextNonComment->isOneOf(tok::kw_operator, TT_FunctionDeclarationName))) {
     return std::max(CurrentState.LastSpace, CurrentState.Indent);
   }
   if (NextNonComment->is(TT_SelectorName)) {
     if (!CurrentState.ObjCSelectorNameFound) {
       unsigned MinIndent = CurrentState.Indent;
       if (shouldIndentWrappedSelectorName(Style, State.Line->Type)) {
         MinIndent = std::max(MinIndent,

clang/lib/Format/TokenAnnotator.cpp

if (OpeningParen.is(TT_OverloadedOperatorLParen)) {
           OpeningParen.Previous &&
           OpeningParen.Previous->isOneOf(tok::kw_for, tok::kw_catch);
       Contexts.back().IsExpression = !IsForOrCatch;
     }
 
     // Infer the role of the l_paren based on the previous token if we haven't
     // detected one yet.
     if (PrevNonComment && OpeningParen.is(TT_Unknown)) {
-      if (PrevNonComment->is(tok::kw___attribute)) {
+      if (PrevNonComment->isOneOf(tok::kw___attribute, TT_AttributeMacro)) {
         OpeningParen.setType(TT_AttributeParen);
       } else if (PrevNonComment->isOneOf(TT_TypenameMacro, tok::kw_decltype,
                                          tok::kw_typeof,
 #define TRANSFORM_TYPE_TRAIT_DEF(_, Trait) tok::kw___##Trait,
 #include "clang/Basic/TransformTypeTraits.def"
                                          tok::kw__Atomic)) {
         OpeningParen.setType(TT_TypeDeclarationParen);
         // decltype() and typeof() usually contain expressions.
if (Left.is(TT_ModulePartitionColon) &&
         Right.isOneOf(tok::identifier, tok::kw_private)) {
       return false;
     }
     if (Left.is(tok::ellipsis) && Right.is(tok::identifier) &&
         Line.First->is(Keywords.kw_import)) {
       return false;
     }
     // Space in __attribute__((attr)) ::type.
     if (Left.is(TT_AttributeParen) && Right.is(tok::coloncolon))
       return true;
 
     if (Left.is(tok::kw_operator))
       return Right.is(tok::coloncolon);
     if (Right.is(tok::l_brace) && Right.is(BK_BracedInit) &&
         !Left.opensScope() && Style.SpaceBeforeCpp11BracedList) {
       return true;
     }
     if (Left.is(tok::less) && Left.is(TT_OverloadedOperator) &&
if (Style.isCpp()) {
     if (Right.is(tok::l_paren) && Right.is(TT_VerilogStrength))
       return true;
   }
   if (Left.is(TT_ImplicitStringLiteral))
     return Right.hasWhitespaceBefore();
   if (Line.Type == LT_ObjCMethodDecl) {
     if (Left.is(TT_ObjCMethodSpecifier))
       return true;
     if (Left.is(tok::r_paren) && canBeObjCSelectorComponent(Right)) {
       // Don't space between ')' and <id> or ')' and 'new'. 'new' is not a
       // keyword in Objective-C, and '+ (instancetype)new;' is a standard class
       // method declaration.
       return false;
     }
   }
   if (Line.Type == LT_ObjCProperty &&
       (Right.is(tok::equal) || Left.is(tok::equal))) {
     return false;
   }
bool TokenAnnotator::mustBreakBefore(const AnnotatedLine &Line,
   }
 
   if (Left.is(TT_ObjCBlockLBrace) &&
       Style.AllowShortBlocksOnASingleLine == FormatStyle::SBS_Never) {
     return true;
   }
 
   // Ensure wrapping after __attribute__((XX)) and @interface etc.
-  if (Left.is(TT_AttributeParen) && Right.is(TT_ObjCDecl))
+  if (Left.isOneOf(TT_AttributeMacro, TT_AttributeParen) &&
+      Right.is(TT_ObjCDecl)) {
     return true;
+  }
 
   if (Left.is(TT_LambdaLBrace)) {
     if (IsFunctionArgument(Left) &&
         Style.AllowShortLambdasOnASingleLine == FormatStyle::SLS_Inline) {
       return false;
     }
 
     if (Style.AllowShortLambdasOnASingleLine == FormatStyle::SLS_None ||
bool TokenAnnotator::canBreakBefore(const AnnotatedLine &Line,
   // declaration.
   if (Left.is(TT_TrailingAnnotation)) {
     return !Right.isOneOf(tok::l_brace, tok::semi, tok::equal, tok::l_paren,
                           tok::less, tok::coloncolon);
   }
 
   if (Right.is(tok::kw___attribute) ||
       (Right.is(tok::l_square) && Right.is(TT_AttributeSquare))) {
     return !Left.is(TT_AttributeSquare);
   }
 
   if (Left.is(tok::identifier) && Right.is(tok::string_literal))
     return true;
 
   if (Right.is(tok::identifier) && Right.Next && Right.Next->is(TT_DictLiteral))
     return true;
 
   if (Left.is(TT_CtorInitializerColon)) {
     return Style.BreakConstructorInitializers == FormatStyle::BCIS_AfterColon &&
            (!Right.isTrailingComment() || Right.NewlinesBefore > 0);
   }
   if (Right.is(TT_CtorInitializerColon))

clang/unittests/Format/FormatTestObjC.cpp

TEST_F(FormatTestObjC, IfNotUnlikely) {
 
   verifyFormat("if (argc < 5) [[unlikely]]\n"
                "  [obj func:arg];\n"
                "else [[likely]]\n"
                "  [obj func:arg2];");
 }
 
 TEST_F(FormatTestObjC, Attributes) {
+  Style.AttributeMacros.push_back("ATTRIBUTE_MACRO");
+
+  // Check '__attribute__' macro directly.
   verifyFormat("__attribute__((objc_subclassing_restricted))\n"
                "@interface Foo\n"
                "@end");
   verifyFormat("__attribute__((objc_subclassing_restricted))\n"
                "@protocol Foo\n"
                "@end");
   verifyFormat("__attribute__((objc_subclassing_restricted))\n"
                "@implementation Foo\n"
                "@end");
+
+  // Check AttributeMacro gets treated the same, with or without parentheses.
+  verifyFormat("ATTRIBUTE_MACRO\n"
+               "@interface Foo\n"
+               "@end");
+  verifyFormat("ATTRIBUTE_MACRO(X)\n"
+               "@interface Foo\n"
+               "@end");
+
+  // Indenter also needs to understand multiple attribute macros.
+  verifyFormat("ATTRIBUTE_MACRO(X) ATTRIBUTE_MACRO\n"
+               "@interface Foo\n"
+               "@end");
+  verifyFormat("ATTRIBUTE_MACRO ATTRIBUTE_MACRO(X)\n"
+               "@interface Foo\n"
+               "@end");
+
+  Style.ColumnLimit = 30;
+  verifyFormat("ATTRIBUTE_MACRO(X)\n"
+               "ATTRIBUTE_MACRO\n"
+               "@interface Foo\n"
+               "@end");
+  // Note: the following -should- break across multiple lines, but doesn't.
+  // This is added to acknowledge the behavior, but it should be improved.
+  verifyFormat("ATTRIBUTE_MACRO ATTRIBUTE_MACRO(X)\n"
+               "@interface Foo\n"
+               "@end");
+
+  Style.ColumnLimit = 0;
+  verifyFormat("ATTRIBUTE_MACRO(X) ATTRIBUTE_MACRO\n"
+               "@interface Foo\n"
+               "@end");
+  verifyFormat("ATTRIBUTE_MACRO ATTRIBUTE_MACRO(X)\n"
+               "@interface Foo\n"
+               "@end");
 }
 
 } // end namespace
 } // namespace test
 } // end namespace format
 } // end namespace clang

clang/unittests/Format/TokenAnnotatorTest.cpp

 TEST_F(TokenAnnotatorTest, UnderstandsC11GenericSelection) {
   auto Tokens = annotate("_Generic(x, int: 1, default: 0)");
   ASSERT_EQ(Tokens.size(), 13u) << Tokens;
   EXPECT_TOKEN(Tokens[0], tok::kw__Generic, TT_Unknown);
   EXPECT_TOKEN(Tokens[5], tok::colon, TT_GenericSelectionColon);
   EXPECT_TOKEN(Tokens[9], tok::colon, TT_GenericSelectionColon);
 }
 
+TEST_F(TokenAnnotatorTest, UnderstandsAttributeMacros) {
+  // '__attribute__' has special handling.
+  auto Tokens = annotate("__attribute__(X) void Foo(void);");
+  ASSERT_EQ(Tokens.size(), 11u) << Tokens;
+  EXPECT_TOKEN(Tokens[0], tok::kw___attribute, TT_Unknown);
+  EXPECT_TOKEN(Tokens[1], tok::l_paren, TT_AttributeParen);
+  EXPECT_TOKEN(Tokens[3], tok::r_paren, TT_AttributeParen);
+
+  // Generic macro has no special handling in this location.
+  Tokens = annotate("A(X) void Foo(void);");
+  ASSERT_EQ(Tokens.size(), 11u) << Tokens;
+  EXPECT_TOKEN(Tokens[0], tok::identifier, TT_Unknown);
+  EXPECT_TOKEN(Tokens[1], tok::l_paren, TT_Unknown);
+  // 'TT_FunctionAnnotationRParen' doesn't seem right; fix?
+  EXPECT_TOKEN(Tokens[3], tok::r_paren, TT_FunctionAnnotationRParen);
+
+  // Add a custom AttributeMacro. Test that it has the same behavior.
+  FormatStyle Style = getLLVMStyle();
+  Style.AttributeMacros.push_back("A");
+
+  // An "AttributeMacro" gets annotated like '__attribute__'.
+  Tokens = annotate("A(X) void Foo(void);", Style);
+  ASSERT_EQ(Tokens.size(), 11u) << Tokens;
+  EXPECT_TOKEN(Tokens[0], tok::identifier, TT_AttributeMacro);
+  EXPECT_TOKEN(Tokens[1], tok::l_paren, TT_AttributeParen);
+  EXPECT_TOKEN(Tokens[3], tok::r_paren, TT_AttributeParen);
+}
+
 TEST_F(TokenAnnotatorTest, UnderstandsVerilogOperators) {
   auto Annotate = [this](llvm::StringRef Code) {
     return annotate(Code, getLLVMStyle(FormatStyle::LK_Verilog));
   };
   // Test that unary operators get labeled as such and that operators like '++'
   // don't get split.
   tok::TokenKind Unary[] = {tok::plus,  tok::minus,    tok::exclaim,
                             tok::tilde, tok::amp,      tok::pipe,
                             tok::caret, tok::plusplus, tok::minusminus};
   for (auto Kind : Unary) {
     auto Tokens =
         Annotate(std::string("x = ") + tok::getPunctuatorSpelling(Kind) + "x;");
     ASSERT_EQ(Tokens.size(), 6u) << Tokens;
     EXPECT_TOKEN(Tokens[2], Kind, TT_UnaryOperator);
   }
   // Operators formed by joining two operators like '^~'. For some of these
   // joined operators, we don't have a separate type, so we only test for their
   // precedence.
   std::pair<prec::Level, std::string> JoinedBinary[] = {
       {prec::Comma, "<->"},       {prec::Assignment, "+="},
       {prec::Assignment, "-="},   {prec::Assignment, "*="},
       {prec::Assignment, "/="},   {prec::Assignment, "%="},
       {prec::Assignment, "&="},   {prec::Assignment, "^="},
       {prec::Assignment, "<<="},  {prec::Assignment, ">>="},
       {prec::Assignment, "<<<="}, {prec::Assignment, ">>>="},
       {prec::LogicalOr, "||"},    {prec::LogicalAnd, "&&"},
       {prec::Equality, "=="},     {prec::Equality, "!="},
       {prec::Equality, "==="},    {prec::Equality, "!=="},
       {prec::Equality, "==?"},    {prec::Equality, "!=?"},
       {prec::ExclusiveOr, "~^"},  {prec::ExclusiveOr, "^~"},
   };
   for (auto Operator : JoinedBinary) {
     auto Tokens = Annotate(std::string("x = x ") + Operator.second + " x;");
     ASSERT_EQ(Tokens.size(), 7u) << Tokens;
     EXPECT_TOKEN_TYPE(Tokens[3], TT_BinaryOperator);
     EXPECT_TOKEN_PRECEDENCE(Tokens[3], Operator.first);
   }
   // '~^' and '^~' can be unary as well as binary operators.
   auto Tokens = Annotate("x = ~^x;");
   ASSERT_EQ(Tokens.size(), 6u) << Tokens;
   EXPECT_TOKEN_TYPE(Tokens[2], TT_UnaryOperator);
   Tokens = Annotate("x = ^~x;");
   ASSERT_EQ(Tokens.size(), 6u) << Tokens;
   EXPECT_TOKEN_TYPE(Tokens[2], TT_UnaryOperator);
   // The unary operators '~&' and '~|' can only be unary operators. The current
   // implementation treats each of them as separate unary '~' and '&' or '|'
   // operators, which is enough for formatting purposes. In FormatTestVerilog,
   // there is a test that there is no space in between. And even if a new line
   // is inserted between the '~' and '|', the semantic meaning is the same as
   // the joined operator, so the CanBreakBefore property doesn't need to be
   // false for the second operator.
   Tokens = Annotate("x = ~&x;");
   ASSERT_EQ(Tokens.size(), 7u) << Tokens;
   EXPECT_TOKEN(Tokens[2], tok::tilde, TT_UnaryOperator);
   EXPECT_TOKEN(Tokens[3], tok::amp, TT_UnaryOperator);
   Tokens = Annotate("x = ~|x;");
   ASSERT_EQ(Tokens.size(), 7u) << Tokens;
   EXPECT_TOKEN(Tokens[2], tok::tilde, TT_UnaryOperator);
   EXPECT_TOKEN(Tokens[3], tok::pipe, TT_UnaryOperator);
   // Test for block label colons.
   Tokens = Annotate("begin : x\n"
                     "end : x");
   ASSERT_EQ(Tokens.size(), 7u);
   EXPECT_TOKEN(Tokens[1], tok::colon, TT_VerilogBlockLabelColon);
   EXPECT_TOKEN(Tokens[4], tok::colon, TT_VerilogBlockLabelColon);
   // Test that the dimension colon is annotated correctly.
   Tokens = Annotate("var [1 : 0] x;");
   ASSERT_EQ(Tokens.size(), 9u) << Tokens;
   EXPECT_TOKEN(Tokens[3], tok::colon, TT_BitFieldColon);
   Tokens = Annotate("extern function [1 : 0] x;");
   ASSERT_EQ(Tokens.size(), 10u) << Tokens;
   EXPECT_TOKEN(Tokens[4], tok::colon, TT_BitFieldColon);
   // Test case labels and ternary operators.
   Tokens = Annotate("case (x)\n"
                     "  x:\n"
                     "    x;\n"
                     "endcase\n");
   ASSERT_EQ(Tokens.size(), 10u) << Tokens;
   EXPECT_TOKEN(Tokens[5], tok::colon, TT_GotoLabelColon);
   Tokens = Annotate("case (x)\n"
                     "  x ? x : x:\n"
                     "    x;\n"