[clang-format] Handle Verilog preprocessor directives

[sstwcw]

Summary

Verilog uses the backtick instead of the hash. In this revision
backticks are lexed manually and then get labeled as hashes so the logic
for handling C preprocessor stuff don't have to change. Hashes get
labeled as identifiers for Verilog-specific stuff like delays.

[sstwcw]

This revision depends on D124748, but somehow it doesn't show up when I open the parent revision dialog.

[HazardyKnusperkeks]

clang/lib/Format/FormatTokenLexer.cpp

1105

Consider a raw string, for a better reading.

1129

Otherwise I don't see why you change that.

[MyDeveloperDay]

You add significant number of keywords here but I don't see any of them being tested? can you add a unit tests to cover what you are adding

[sstwcw]

• add tests and remove LINE and FILE special cases

[sstwcw]

clang/lib/Format/FormatTokenLexer.cpp

1105

You mean like this?

static const llvm::Regex VerilogToken(R"re(^('|``?|\\(\\)re"
                                      "(\r?\n|\r)|[^[:space:]])*)");

1129

The function is only supposed to be used for Verilog, so && is correct. If you mean why this part is different from D121758, I changed the name from readRawTokenLanguageSpecific to readRawTokenVerilogSpecific as suggested by a reviewer. Then I moved the check for language out of that function like what tryParseJSRegexLiteral does.

[sstwcw]

The two parents of this revision change the same file, so the build bot says patch does not apply. Does that mean I have to submit the parent patches with less context?

[HazardyKnusperkeks]

In D124749#3514224, @sstwcw wrote:

The two parents of this revision change the same file, so the build bot says patch does not apply. Does that mean I have to submit the parent patches with less context?

Maybe just wait until one of the parents has landed. We can make a review without the bot. :)
Or put the parents in relation.

In D124749#3490834, @MyDeveloperDay wrote:

You add significant number of keywords here but I don't see any of them being tested? can you add a unit tests to cover what you are adding

I think this is still open?

clang/lib/Format/FormatTokenLexer.cpp

1105

I'd put it in one line, but basically yeah.

1129

I think I missed the parens.

1131

As demonstrated by me, the parens can go under in a quick read.

[sstwcw]

clang/unittests/Format/FormatTestVerilog.cpp

179

In D124749#3490834, @MyDeveloperDay wrote:

You add significant number of keywords here but I don't see any of them being tested? can you add a unit tests to cover what you are adding

I think this is still open?

The keywords added in this revision are tested here. Most of those added in D123450 don't have tests, but I haven't added stuff to handle them yet.

[sstwcw]

• use raw string for regex
• use default style in test
• remove parentheses
• use seek now that skipOver no longer exists

[HazardyKnusperkeks]

clang/lib/Format/UnwrappedLineParser.cpp

1783

Why did you remove that if?

1800

And now the indentation here is off.

[sstwcw]

clang/lib/Format/UnwrappedLineParser.cpp

1783

Previously the if checked whether the line is comment and an identifier. For Verilog it's comment, a backtick, and an identifier. I found it easier to break if the line doesn't match the pattern.

[sstwcw]

• add comment and format

[HazardyKnusperkeks]

clang/lib/Format/FormatTokenLexer.cpp

1109

Add braces. Maybe check all other added ifs too. You can check with clang-format with InsertBraces and RemoveBracesLLVM.

clang/lib/Format/UnwrappedLineParser.cpp

1785

1805

I find just scopes a bit confusing. Maybe put the check if we want to break into a lambda and call that?

[sstwcw]

• add kw_ prefix to symbols treated as keywords
• use lambda instead of breaking

[sstwcw]

clang/lib/Format/FormatToken.h

1172

Now that I added kw_ to verilogHashHash, four columns can't fit.

[HazardyKnusperkeks]

clang/lib/Format/FormatToken.h

981

It seems a comment before is missing. Should be fixed in a separate patch.

1172

Happens. :)

clang/lib/Format/FormatToken.h

private:
 
   bool HasNestedBracedList;
 };
 
 /// Encapsulates keywords that are context sensitive or for languages not
 /// properly supported by Clang's lexer.
 struct AdditionalKeywords {
   AdditionalKeywords(IdentifierTable &IdentTable) {
+    kw___FILE__ = &IdentTable.get("__FILE__");
+    kw___LINE__ = &IdentTable.get("__LINE__");
+
     kw_final = &IdentTable.get("final");
     kw_override = &IdentTable.get("override");
     kw_in = &IdentTable.get("in");
     kw_of = &IdentTable.get("of");
     kw_CF_CLOSED_ENUM = &IdentTable.get("CF_CLOSED_ENUM");
     kw_CF_ENUM = &IdentTable.get("CF_ENUM");
     kw_CF_OPTIONS = &IdentTable.get("CF_OPTIONS");
     kw_NS_CLOSED_ENUM = &IdentTable.get("NS_CLOSED_ENUM");
AdditionalKeywords(IdentifierTable &IdentTable) {
     kw_uint = &IdentTable.get("uint");
     kw_ulong = &IdentTable.get("ulong");
     kw_unchecked = &IdentTable.get("unchecked");
     kw_unsafe = &IdentTable.get("unsafe");
     kw_ushort = &IdentTable.get("ushort");
     kw_when = &IdentTable.get("when");
     kw_where = &IdentTable.get("where");
 
     kw_always = &IdentTable.get("always");
     kw_always_comb = &IdentTable.get("always_comb");
     kw_always_ff = &IdentTable.get("always_ff");
     kw_always_latch = &IdentTable.get("always_latch");
     kw_assign = &IdentTable.get("assign");
     kw_assume = &IdentTable.get("assume");
     kw_automatic = &IdentTable.get("automatic");
     kw_before = &IdentTable.get("before");
     kw_begin = &IdentTable.get("begin");
+    kw_begin_keywords = &IdentTable.get("begin_keywords");
     kw_bins = &IdentTable.get("bins");
     kw_binsof = &IdentTable.get("binsof");
     kw_casex = &IdentTable.get("casex");
     kw_casez = &IdentTable.get("casez");
     kw_celldefine = &IdentTable.get("celldefine");
     kw_checker = &IdentTable.get("checker");
     kw_clocking = &IdentTable.get("clocking");
     kw_constraint = &IdentTable.get("constraint");
     kw_cover = &IdentTable.get("cover");
     kw_covergroup = &IdentTable.get("covergroup");
     kw_coverpoint = &IdentTable.get("coverpoint");
+    kw_default_decay_time = &IdentTable.get("default_decay_time");
+    kw_default_nettype = &IdentTable.get("default_nettype");
+    kw_default_trireg_strength = &IdentTable.get("default_trireg_strength");
+    kw_delay_mode_distributed = &IdentTable.get("delay_mode_distributed");
+    kw_delay_mode_path = &IdentTable.get("delay_mode_path");
+    kw_delay_mode_unit = &IdentTable.get("delay_mode_unit");
+    kw_delay_mode_zero = &IdentTable.get("delay_mode_zero");
     kw_disable = &IdentTable.get("disable");
     kw_dist = &IdentTable.get("dist");
+    kw_elsif = &IdentTable.get("elsif");
     kw_end = &IdentTable.get("end");
+    kw_end_keywords = &IdentTable.get("end_keywords");
     kw_endcase = &IdentTable.get("endcase");
+    kw_endcelldefine = &IdentTable.get("endcelldefine");
     kw_endchecker = &IdentTable.get("endchecker");
     kw_endclass = &IdentTable.get("endclass");
     kw_endclocking = &IdentTable.get("endclocking");
     kw_endfunction = &IdentTable.get("endfunction");
     kw_endgenerate = &IdentTable.get("endgenerate");
     kw_endgroup = &IdentTable.get("endgroup");
     kw_endinterface = &IdentTable.get("endinterface");
     kw_endmodule = &IdentTable.get("endmodule");
AdditionalKeywords(IdentifierTable &IdentTable) {
     kw_join_any = &IdentTable.get("join_any");
     kw_join_none = &IdentTable.get("join_none");
     kw_large = &IdentTable.get("large");
     kw_local = &IdentTable.get("local");
     kw_localparam = &IdentTable.get("localparam");
     kw_macromodule = &IdentTable.get("macromodule");
     kw_matches = &IdentTable.get("matches");
     kw_medium = &IdentTable.get("medium");
+    kw_nounconnected_drive = &IdentTable.get("nounconnected_drive");
     kw_output = &IdentTable.get("output");
     kw_packed = &IdentTable.get("packed");
     kw_parameter = &IdentTable.get("parameter");
     kw_primitive = &IdentTable.get("primitive");
     kw_priority = &IdentTable.get("priority");
     kw_program = &IdentTable.get("program");
     kw_property = &IdentTable.get("property");
     kw_pull0 = &IdentTable.get("pull0");
     kw_pull1 = &IdentTable.get("pull1");
     kw_pure = &IdentTable.get("pure");
     kw_rand = &IdentTable.get("rand");
     kw_randc = &IdentTable.get("randc");
     kw_randcase = &IdentTable.get("randcase");
     kw_randsequence = &IdentTable.get("randsequence");
     kw_repeat = &IdentTable.get("repeat");
+    kw_resetall = &IdentTable.get("resetall");
     kw_sample = &IdentTable.get("sample");
     kw_scalared = &IdentTable.get("scalared");
     kw_sequence = &IdentTable.get("sequence");
     kw_small = &IdentTable.get("small");
     kw_soft = &IdentTable.get("soft");
     kw_solve = &IdentTable.get("solve");
     kw_specify = &IdentTable.get("specify");
     kw_specparam = &IdentTable.get("specparam");
     kw_strong0 = &IdentTable.get("strong0");
     kw_strong1 = &IdentTable.get("strong1");
     kw_supply0 = &IdentTable.get("supply0");
     kw_supply1 = &IdentTable.get("supply1");
     kw_table = &IdentTable.get("table");
     kw_tagged = &IdentTable.get("tagged");
     kw_task = &IdentTable.get("task");
+    kw_timescale = &IdentTable.get("timescale");
     kw_tri = &IdentTable.get("tri");
     kw_tri0 = &IdentTable.get("tri0");
     kw_tri1 = &IdentTable.get("tri1");
     kw_triand = &IdentTable.get("triand");
     kw_trior = &IdentTable.get("trior");
     kw_trireg = &IdentTable.get("trireg");
+    kw_unconnected_drive = &IdentTable.get("unconnected_drive");
+    kw_undefineall = &IdentTable.get("undefineall");
     kw_unique = &IdentTable.get("unique");
     kw_unique0 = &IdentTable.get("unique0");
     kw_uwire = &IdentTable.get("uwire");
     kw_vectored = &IdentTable.get("vectored");
     kw_wand = &IdentTable.get("wand");
     kw_weak0 = &IdentTable.get("weak0");
     kw_weak1 = &IdentTable.get("weak1");
     kw_wildcard = &IdentTable.get("wildcard");
     kw_wire = &IdentTable.get("wire");
     kw_with = &IdentTable.get("with");
     kw_wor = &IdentTable.get("wor");
 
+    // Symbols that are treated as keywords.
+    verilogHash = &IdentTable.get("#");
+    verilogHashHash = &IdentTable.get("##");
+
     // Keep this at the end of the constructor to make sure everything here
     // is
     // already initialized.
     JsExtraKeywords = std::unordered_set<IdentifierInfo *>(
         {kw_as, kw_async, kw_await, kw_declare, kw_finally, kw_from,
          kw_function, kw_get, kw_import, kw_is, kw_let, kw_module, kw_override,
          kw_readonly, kw_set, kw_type, kw_typeof, kw_var, kw_yield,
          // Keywords from the Java section.
VerilogExtraKeywords = std::unordered_set<IdentifierInfo *>(
          kw_ref,          kw_repeat,       kw_sample,       kw_scalared,
          kw_sequence,     kw_small,        kw_soft,         kw_solve,
          kw_specify,      kw_specparam,    kw_strong0,      kw_strong1,
          kw_supply0,      kw_supply1,      kw_table,        kw_tagged,
          kw_task,         kw_tri,          kw_tri0,         kw_tri1,
          kw_triand,       kw_trior,        kw_trireg,       kw_unique,
          kw_unique0,      kw_uwire,        kw_var,          kw_vectored,
          kw_wand,         kw_weak0,        kw_weak1,        kw_wildcard,
-         kw_wire,         kw_with,         kw_wor});
+         kw_wire,         kw_with,         kw_wor,          verilogHash,
+         verilogHashHash});
   }
 
+  // Preprocessor
+  IdentifierInfo *kw___FILE__;
+  IdentifierInfo *kw___LINE__;
+
   // Context sensitive keywords.
   IdentifierInfo *kw_final;
   IdentifierInfo *kw_override;
   IdentifierInfo *kw_in;
   IdentifierInfo *kw_of;
   IdentifierInfo *kw_CF_CLOSED_ENUM;
   IdentifierInfo *kw_CF_ENUM;
   IdentifierInfo *kw_CF_OPTIONS;
struct AdditionalKeywords {
   IdentifierInfo *kw_always_comb;
   IdentifierInfo *kw_always_ff;
   IdentifierInfo *kw_always_latch;
   IdentifierInfo *kw_assign;
   IdentifierInfo *kw_assume;
   IdentifierInfo *kw_automatic;
   IdentifierInfo *kw_before;
   IdentifierInfo *kw_begin;
+  IdentifierInfo *kw_begin_keywords;
   IdentifierInfo *kw_bins;
   IdentifierInfo *kw_binsof;
   IdentifierInfo *kw_casex;
   IdentifierInfo *kw_casez;
   IdentifierInfo *kw_celldefine;
   IdentifierInfo *kw_checker;
   IdentifierInfo *kw_clocking;
   IdentifierInfo *kw_constraint;
   IdentifierInfo *kw_cover;
   IdentifierInfo *kw_covergroup;
   IdentifierInfo *kw_coverpoint;
+  IdentifierInfo *kw_default_decay_time;
+  IdentifierInfo *kw_default_nettype;
+  IdentifierInfo *kw_default_trireg_strength;
+  IdentifierInfo *kw_delay_mode_distributed;
+  IdentifierInfo *kw_delay_mode_path;
+  IdentifierInfo *kw_delay_mode_unit;
+  IdentifierInfo *kw_delay_mode_zero;
   IdentifierInfo *kw_disable;
   IdentifierInfo *kw_dist;
+  IdentifierInfo *kw_elsif;
   IdentifierInfo *kw_end;
+  IdentifierInfo *kw_end_keywords;
   IdentifierInfo *kw_endcase;
+  IdentifierInfo *kw_endcelldefine;
   IdentifierInfo *kw_endchecker;
   IdentifierInfo *kw_endclass;
   IdentifierInfo *kw_endclocking;
   IdentifierInfo *kw_endfunction;
   IdentifierInfo *kw_endgenerate;
   IdentifierInfo *kw_endgroup;
   IdentifierInfo *kw_endinterface;
   IdentifierInfo *kw_endmodule;
struct AdditionalKeywords {
   IdentifierInfo *kw_join_any;
   IdentifierInfo *kw_join_none;
   IdentifierInfo *kw_large;
   IdentifierInfo *kw_local;
   IdentifierInfo *kw_localparam;
   IdentifierInfo *kw_macromodule;
   IdentifierInfo *kw_matches;
   IdentifierInfo *kw_medium;
+  IdentifierInfo *kw_nounconnected_drive;
   IdentifierInfo *kw_output;
   IdentifierInfo *kw_packed;
   IdentifierInfo *kw_parameter;
   IdentifierInfo *kw_primitive;
   IdentifierInfo *kw_priority;
   IdentifierInfo *kw_program;
   IdentifierInfo *kw_property;
   IdentifierInfo *kw_pull0;
   IdentifierInfo *kw_pull1;
   IdentifierInfo *kw_pure;
   IdentifierInfo *kw_rand;
   IdentifierInfo *kw_randc;
   IdentifierInfo *kw_randcase;
   IdentifierInfo *kw_randsequence;
   IdentifierInfo *kw_repeat;
+  IdentifierInfo *kw_resetall;
   IdentifierInfo *kw_sample;
   IdentifierInfo *kw_scalared;
   IdentifierInfo *kw_sequence;
   IdentifierInfo *kw_small;
   IdentifierInfo *kw_soft;
   IdentifierInfo *kw_solve;
   IdentifierInfo *kw_specify;
   IdentifierInfo *kw_specparam;
   IdentifierInfo *kw_strong0;
   IdentifierInfo *kw_strong1;
   IdentifierInfo *kw_supply0;
   IdentifierInfo *kw_supply1;
   IdentifierInfo *kw_table;
   IdentifierInfo *kw_tagged;
   IdentifierInfo *kw_task;
-  IdentifierInfo *kw_tri;
+  IdentifierInfo *kw_timescale;
   IdentifierInfo *kw_tri0;
   IdentifierInfo *kw_tri1;
+  IdentifierInfo *kw_tri;
   IdentifierInfo *kw_triand;
   IdentifierInfo *kw_trior;
   IdentifierInfo *kw_trireg;
-  IdentifierInfo *kw_unique;
+  IdentifierInfo *kw_unconnected_drive;
+  IdentifierInfo *kw_undefineall;
   IdentifierInfo *kw_unique0;
+  IdentifierInfo *kw_unique;
   IdentifierInfo *kw_uwire;
   IdentifierInfo *kw_vectored;
   IdentifierInfo *kw_wand;
   IdentifierInfo *kw_weak0;
   IdentifierInfo *kw_weak1;
   IdentifierInfo *kw_wildcard;
   IdentifierInfo *kw_wire;
   IdentifierInfo *kw_with;
   IdentifierInfo *kw_wor;
 
+  // Workaround for hashes and backticks in Verilog.
+  IdentifierInfo *verilogHash;
+  IdentifierInfo *verilogHashHash;
+
+  /// Returns \c true if \p Tok is a symbol defined here.
+  bool isAdditionalSymbol(const FormatToken &Tok) const {
+    return Tok.isOneOf(verilogHash, verilogHashHash);
+  }
+
   /// Returns \c true if \p Tok is a true JavaScript identifier, returns
   /// \c false if it is a keyword or a pseudo keyword.
   /// If \c AcceptIdentifierName is true, returns true not only for keywords,
   // but also for IdentifierName tokens (aka pseudo-keywords), such as
   // ``yield``.
   bool IsJavaScriptIdentifier(const FormatToken &Tok,
                               bool AcceptIdentifierName = true) const {
     // Based on the list of JavaScript & TypeScript keywords here:
case tok::identifier:
       return VerilogExtraKeywords.find(Tok.Tok.getIdentifierInfo()) ==
              VerilogExtraKeywords.end();
     default:
       // getIdentifierInfo returns non-null for both identifiers and keywords.
       return Tok.Tok.getIdentifierInfo() != nullptr;
     }
   }
 
+  /// Returns whether \p Tok is a Verilog preprocessor directive.  This is
+  /// needed because macro expansions start with a backtick as well and they
+  /// need to be treated differently.
+  bool isVerilogPPDirective(const FormatToken &Tok) const {
+    auto Info = Tok.Tok.getIdentifierInfo();
+    if (!Info)
+      return false;
+    switch (Info->getPPKeywordID()) {
+    case tok::pp_define:
+    case tok::pp_else:
+    case tok::pp_endif:
+    case tok::pp_ifdef:
+    case tok::pp_ifndef:
+    case tok::pp_include:
+    case tok::pp_line:
+    case tok::pp_pragma:
+    case tok::pp_undef:
+      return true;
+    default:
+      return Tok.isOneOf(
+          kw___FILE__, kw___LINE__, kw_begin_keywords, kw_celldefine,
+          kw_default_decay_time, kw_default_nettype, kw_default_trireg_strength,
+          kw_delay_mode_distributed, kw_delay_mode_path, kw_delay_mode_unit,
+          kw_delay_mode_zero, kw_elsif, kw_end_keywords, kw_endcelldefine,
+          kw_nounconnected_drive, kw_resetall, kw_timescale,
+          kw_unconnected_drive, kw_undefineall);
+    }
+  }
+
   /// Returns whether \p Tok is a Verilog keyword that opens a block.
   bool isVerilogBegin(const FormatToken &Tok) const {
     // `table` is not included since it needs to be treated specially.
     return !Tok.endsSequence(kw_fork, kw_disable) &&
            Tok.isOneOf(kw_begin, kw_fork, kw_generate, kw_specify);
   }
 
   /// Returns whether \p Tok is a Verilog keyword that closes a block.
   bool isVerilogEnd(const FormatToken &Tok) const {
     return !Tok.endsSequence(kw_join, kw_rand) &&
            Tok.isOneOf(TT_MacroBlockEnd, kw_end, kw_endcase, kw_endclass,
                        kw_endclocking, kw_endchecker, kw_endfunction,
                        kw_endgenerate, kw_endgroup, kw_endinterface,
                        kw_endmodule, kw_endpackage, kw_endprimitive,
                        kw_endprogram, kw_endproperty, kw_endsequence,
-                       kw_endspecify, kw_endtable, kw_endtask, kw_join_any,
-                       kw_join_none);
+                       kw_endspecify, kw_endtable, kw_endtask, kw_join,
+                       kw_join_any, kw_join_none);
   }
 
   /// Whether the token begins a block.
   bool isBlockBegin(const FormatToken &Tok, const FormatStyle &Style) const {
     return Tok.is(TT_MacroBlockBegin) ||
            (Style.isVerilog() ? isVerilogBegin(Tok) : Tok.is(tok::l_brace));
   }
 

clang/lib/Format/FormatTokenLexer.h

private:
   bool FormattingDisabled;
 
   llvm::Regex MacroBlockBeginRegex;
   llvm::Regex MacroBlockEndRegex;
 
   // Targets that may appear inside a C# attribute.
   static const llvm::StringSet<> CSharpAttributeTargets;
 
+  /// Handle Verilog-specific tokens.
+  bool readRawTokenVerilogSpecific(Token &Tok);
+
   void readRawToken(FormatToken &Tok);
 
   void resetLexer(unsigned Offset);
 };
 
 } // namespace format
 } // namespace clang
 
 #endif

clang/lib/Format/FormatTokenLexer.cpp

while (BackslashPos != StringRef::npos) {
           FormatTok->TokenText[BackslashPos + 1] == '\n') {
         resizeToken(BackslashPos + 1);
         break;
       }
       BackslashPos = FormatTok->TokenText.find('\\', BackslashPos + 1);
     }
   }
 
+  if (Style.isVerilog()) {
+    // Verilog uses the backtick instead of the hash for preprocessor stuff.
+    // And it uses the hash for delays and parameter lists. In order to continue
+    // using `tok::hash` in other places, the backtick gets marked as the hash
+    // here.  And in order to tell the backtick and hash apart for
+    // Verilog-specific stuff, the hash becomes an identifier.
+    if (FormatTok->isOneOf(tok::hash, tok::hashhash)) {
+      FormatTok->Tok.setKind(tok::raw_identifier);
+    } else if (FormatTok->is(tok::raw_identifier)) {
+      if (FormatTok->TokenText == "`") {
+        FormatTok->Tok.setIdentifierInfo(nullptr);
+        FormatTok->Tok.setKind(tok::hash);
+      } else if (FormatTok->TokenText == "``") {
+        FormatTok->Tok.setIdentifierInfo(nullptr);
+        FormatTok->Tok.setKind(tok::hashhash);
+      }
+    }
+  }
+
   FormatTok->WhitespaceRange = SourceRange(
       WhitespaceStart, WhitespaceStart.getLocWithOffset(WhitespaceLength));
 
   FormatTok->OriginalColumn = Column;
 
   TrailingWhitespace = 0;
   if (FormatTok->is(tok::comment)) {
     // FIXME: Add the trimmed whitespace to Column.
if (!(Tokens.size() > 0 && Tokens.back()->Tok.getIdentifierInfo() &&
       else if (MacroBlockEndRegex.match(Text))
         FormatTok->setType(TT_MacroBlockEnd);
     }
   }
 
   return FormatTok;
 }
 
+bool FormatTokenLexer::readRawTokenVerilogSpecific(Token &Tok) {
+  // In Verilog the quote is not a character literal.
+  //
+  // Make the backtick and double backtick identifiers to match against them
+  // more easily.
+  //
+  // In Verilog an escaped identifier starts with backslash and ends with
+  // whitespace. Unless that whitespace is an escaped newline. A backslash can
+  // also begin an escaped newline outside of an escaped identifier. We check
+  // for that outside of the Regex since we can't use negative lookhead
+  // assertions. Simply changing the '*' to '+' breaks stuff as the escaped
+  // identifier may have a length of 0 according to Section A.9.3.
+  // FIXME: If there is an escaped newline in the middle of an escaped
+  // identifier, allow for pasting the two lines together, But escaped
+  // identifiers usually occur only in generated code anyway.
+  static const llvm::Regex VerilogToken(
+      "^(\'|``?|\\\\(\\\\(\r?\n|\r)|[^[:space:]])*)");
+
+  SmallVector<StringRef, 4> Matches;
+  const char *Start = Lex->getBufferLocation();
+  if (!VerilogToken.match(StringRef(Start, Lex->getBuffer().end() - Start),
+                          &Matches))
+    return false;
+  // There is a null byte at the end of the buffer, so we don't have to check
+  // Start[1] is within the buffer.
+  if (Start[0] == '\\' && (Start[1] == '\r' || Start[1] == '\n'))
+    return false;
+  size_t Len = Matches[0].size();
+
+  Tok.setLength(Len);
+  Tok.setLocation(Lex->getSourceLocation(Start, Len));
+  // The kind has to be an identifier so we can match it against those defined
+  // in Keywords.
+  Tok.setKind(tok::raw_identifier);
+  Tok.setRawIdentifierData(Start);
+  Lex->skipOver(Len);
+  return true;
+}
+
 void FormatTokenLexer::readRawToken(FormatToken &Tok) {
+  if (!(Style.isVerilog() && readRawTokenVerilogSpecific(Tok.Tok)))
     Lex->LexFromRawLexer(Tok.Tok);
   Tok.TokenText = StringRef(SourceMgr.getCharacterData(Tok.Tok.getLocation()),
                             Tok.Tok.getLength());
   // For formatting, treat unterminated string literals like normal string
   // literals.
   if (Tok.is(tok::unknown)) {
     if (!Tok.TokenText.empty() && Tok.TokenText[0] == '"') {
       Tok.Tok.setKind(tok::string_literal);
       Tok.IsUnterminatedLiteral = true;
     } else if (Style.isJavaScript() && Tok.TokenText == "''") {

clang/lib/Format/TokenAnnotator.cpp

LineType parsePreprocessorDirective() {
     if (CurrentToken->is(tok::numeric_constant)) {
       CurrentToken->SpacesRequiredBefore = 1;
       return Type;
     }
     // Hashes in the middle of a line can lead to any strange token
     // sequence.
     if (!CurrentToken->Tok.getIdentifierInfo())
       return Type;
+    // In Verilog macro expansions start with a backtick just like preprocessor
+    // directives. Thus we stop if the word is not a preprocessor directive.
+    if (Style.isVerilog() && !Keywords.isVerilogPPDirective(*CurrentToken))
+      return LT_Invalid;
     switch (CurrentToken->Tok.getIdentifierInfo()->getPPKeywordID()) {
     case tok::pp_include:
     case tok::pp_include_next:
     case tok::pp_import:
       next();
       parseIncludeDirective();
       Type = LT_ImportStatement;
       break;
LineType parsePreprocessorDirective() {
     return Type;
   }
 
 public:
   LineType parseLine() {
     if (!CurrentToken)
       return LT_Invalid;
     NonTemplateLess.clear();
-    if (CurrentToken->is(tok::hash))
-      return parsePreprocessorDirective();
+    if (CurrentToken->is(tok::hash)) {
+      // We were not yet allowed to use C++17 optional when this was being
+      // written. So we used LT_Invalid to mark that the line is not a
+      // preprocessor directive.
+      auto Type = parsePreprocessorDirective();
+      if (Type != LT_Invalid)
+        return Type;
+    }
 
     // Directly allow to 'import <string-literal>' to support protocol buffer
     // definitions (github.com/google/protobuf) or missing "#" (either way we
     // should not break the line).
     IdentifierInfo *Info = CurrentToken->Tok.getIdentifierInfo();
     if ((Style.Language == FormatStyle::LK_Java &&
          CurrentToken->is(Keywords.kw_package)) ||
         (Info && Info->getPPKeywordID() == tok::pp_import &&
bool TokenAnnotator::spaceRequiredBefore(const AnnotatedLine &Line,
                                          const FormatToken &Right) {
   const FormatToken &Left = *Right.Previous;
 
   // If the token is finalized don't touch it (as it could be in a
   // clang-format-off section).
   if (Left.Finalized)
     return Right.hasWhitespaceBefore();
 
-  if (Right.Tok.getIdentifierInfo() && Left.Tok.getIdentifierInfo())
-    return true; // Never ever merge two identifiers.
+  // Never ever merge two words.  getIdentifierInfo() returns non-null for
+  // keywords as well as identifiers.
+  if (Right.Tok.getIdentifierInfo() && !Keywords.isAdditionalSymbol(Right) &&
+      Left.Tok.getIdentifierInfo() && !Keywords.isAdditionalSymbol(Left))
+    return true;
 
   // Leave a space between * and /* to avoid C4138 `comment end` found outside
   // of comment.
   if (Left.is(tok::star) && Right.is(tok::comment))
     return true;
 
   if (Style.isCpp()) {
     // Space between import <iostream>.
if (Left.is(Keywords.kw_synchronized) && Right.is(tok::l_paren))
       return Style.SpaceBeforeParensOptions.AfterControlStatements ||
              spaceRequiredBeforeParens(Right);
     if ((Left.isOneOf(tok::kw_static, tok::kw_public, tok::kw_private,
                       tok::kw_protected) ||
          Left.isOneOf(Keywords.kw_final, Keywords.kw_abstract,
                       Keywords.kw_native)) &&
         Right.is(TT_TemplateOpener))
       return true;
+  } else if (Style.isVerilog()) {
+    // Don't add space within a delay like `#0`.
+    if (!Left.is(TT_BinaryOperator) &&
+        Left.isOneOf(Keywords.verilogHash, Keywords.verilogHashHash))
+      return false;
+    // Add space after a delay.
+    if (!Right.is(tok::semi) &&
+        (Left.endsSequence(tok::numeric_constant, tok::hash) ||
+         Left.endsSequence(tok::numeric_constant, tok::hashhash) ||
+         (Left.is(tok::r_paren) && Left.MatchingParen &&
+          Left.MatchingParen->endsSequence(tok::l_paren, tok::at))))
+      return true;
   }
   if (Left.is(TT_ImplicitStringLiteral))
     return Right.hasWhitespaceBefore();
   if (Line.Type == LT_ObjCMethodDecl) {
     if (Left.is(TT_ObjCMethodSpecifier))
       return true;
     if (Left.is(tok::r_paren) && canBeObjCSelectorComponent(Right))
       // Don't space between ')' and <id> or ')' and 'new'. 'new' is not a

clang/lib/Format/UnwrappedLineParser.cpp

case tok::identifier: {
       nextToken();
 
       // JS doesn't have macros, and within classes colons indicate fields, not
       // labels.
       if (Style.isJavaScript())
         break;
 
       TokenCount = Line->Tokens.size();
-      if (TokenCount == 1 ||
-          (TokenCount == 2 && Line->Tokens.front().Tok->is(tok::comment))) {
+      // Determine whether the line might be a single macro expansion.
+      // In Verilog macro expansions begin with a backtick.
+      {
+        const UnwrappedLineNode *Tok = &Line->Tokens.front(),
+                                *End = Tok + TokenCount;
+        while (Tok != End && Tok->Tok->is(tok::comment))
+          ++Tok;
+        if (Style.Language == FormatStyle::LK_Verilog) {
+          if (Tok != End && Tok->Tok->is(tok::hash))
+            ++Tok;
+          else
+            break;
+        }
+        if (End - Tok != 1)
+          break;
+      }
+
         if (FormatTok->is(tok::colon) && !Line->MustBeDeclaration) {
           Line->Tokens.begin()->Tok->MustBreakBefore = true;
           parseLabel(!Style.IndentGotoLabels);
           if (HasLabel)
             *HasLabel = true;
           return;
         }
         // Recognize function-like macro usages without trailing semicolon as
         // well as free-standing macros like Q_OBJECT.
         bool FunctionLike = FormatTok->is(tok::l_paren);
         if (FunctionLike)
           parseParens();
 
         bool FollowedByNewline =
             CommentsBeforeNextToken.empty()
                 ? FormatTok->NewlinesBefore > 0
                 : CommentsBeforeNextToken.front()->NewlinesBefore > 0;
 
         if (FollowedByNewline && (Text.size() >= 5 || FunctionLike) &&
             tokenCanStartNewLine(*FormatTok) && Text == Text.upper()) {
           PreviousToken->setFinalizedType(TT_FunctionLikeOrFreestandingMacro);
           addUnwrappedLine();
           return;
-        }
-      }
       break;
     }
     case tok::equal:
       if ((Style.isJavaScript() || Style.isCSharp()) &&
           FormatTok->is(TT_FatArrow)) {
         tryToParseChildBlock();
         break;
       }
auto IsFirstNonCommentOnLine = [](bool FirstNonCommentOnLine,
       return IsFirstOnLine(Tok);
     };
 
     FirstNonCommentOnLine = IsFirstNonCommentOnLine(
         FirstNonCommentOnLine, *FormatTok, PreviousWasComment);
     PreviousWasComment = FormatTok->is(tok::comment);
 
     while (!Line->InPPDirective && FormatTok->is(tok::hash) &&
+           (!Style.isVerilog() ||
+            Keywords.isVerilogPPDirective(*Tokens->peekNextToken())) &&
            FirstNonCommentOnLine) {
       distributeComments(Comments, FormatTok);
       Comments.clear();
       // If there is an unfinished unwrapped line, we flush the preprocessor
       // directives only after that unwrapped line was finished later.
       bool SwitchToPreprocessorLines = !Line->Tokens.empty();
       ScopedLineState BlockState(*this, SwitchToPreprocessorLines);
       assert((LevelDifference >= 0 ||

clang/unittests/Format/FormatTestVerilog.cpp

static void verifyFormat(
       llvm::StringRef Code,
       const FormatStyle &Style = getLLVMStyle(FormatStyle::LK_Verilog)) {
     EXPECT_EQ(Code.str(), format(Code, Style)) << "Expected code is not stable";
     EXPECT_EQ(Code.str(),
               format(test::messUp(Code, /*HandleHash=*/false), Style));
   }
 };
 
+TEST_F(FormatTestVerilog, Delay) {
+  // Delay by the default unit.
+  verifyFormat("#0;");
+  verifyFormat("#1;");
+  verifyFormat("#10;");
+  verifyFormat("#1.5;");
+  // Explicit unit.
+  verifyFormat("#1fs;");
+  verifyFormat("#1.5fs;");
+  verifyFormat("#1ns;");
+  verifyFormat("#1.5ns;");
+  verifyFormat("#1us;");
+  verifyFormat("#1.5us;");
+  verifyFormat("#1ms;");
+  verifyFormat("#1.5ms;");
+  verifyFormat("#1s;");
+  verifyFormat("#1.5s;");
+  // The following expression should be on the same line.
+  verifyFormat("#1 x = x;");
+  EXPECT_EQ("#1 x = x;", format("#1\n"
+                                "x = x;",
+                                getLLVMStyle(FormatStyle::LK_Verilog)));
+}
+
 TEST_F(FormatTestVerilog, If) {
   verifyFormat("if (x)\n"
                "  x = x;");
   verifyFormat("if (x)\n"
                "  x = x;\n"
                "x = x;");
 
   // Test else
TEST_F(FormatTestVerilog, If) {
   verifyFormat("if (x)\n"
                "  x = {x};");
   verifyFormat("if (x)\n"
                "  x = {x};\n"
                "else\n"
                "  {x} = {x};");
 }
 
+TEST_F(FormatTestVerilog, Preprocessor) {
+  auto Style = getLLVMStyle(FormatStyle::LK_Verilog);
+  Style.ColumnLimit = 20;
+
+  // Macro definitions.
+  EXPECT_EQ("`define X          \\\n"
+            "  if (x)           \\\n"
+            "    x = x;",
+            format("`define X if(x)x=x;", Style));
+  EXPECT_EQ("`define X(x)       \\\n"
+            "  if (x)           \\\n"
+            "    x = x;",
+            format("`define X(x) if(x)x=x;", Style));
+  EXPECT_EQ("`define X          \\\n"
+            "  x = x;           \\\n"
+            "  x = x;",
+            format("`define X x=x;x=x;", Style));
+  // Macro definitions with invocations inside.
+  EXPECT_EQ("`define LIST       \\\n"
+            "  `ENTRY           \\\n"
+            "  `ENTRY",
+            format("`define LIST \\\n"
+                   "`ENTRY \\\n"
+                   "`ENTRY",
+                   Style));
+  EXPECT_EQ("`define LIST       \\\n"
+            "  `x = `x;         \\\n"
+            "  `x = `x;",
+            format("`define LIST \\\n"
+                   "`x = `x; \\\n"
+                   "`x = `x;",
+                   Style));
+  EXPECT_EQ("`define LIST       \\\n"
+            "  `x = `x;         \\\n"
+            "  `x = `x;",
+            format("`define LIST `x=`x;`x=`x;", Style));
+  // Macro invocations.
+  verifyFormat("`x = (`x1 + `x2 + x);");
+  // Lines starting with a normal macro invocation should be indented as a
+  // normal line.
+  EXPECT_EQ("if (x)\n"
+            "  `x = `x;\n"
+            "`timescale 1ns / 1ps",
+            format("if (x)\n"
+                   "`x = `x;\n"
+                   "`timescale 1ns / 1ps",
+                   Style));
+  EXPECT_EQ("if (x)\n"
+            "`timescale 1ns / 1ps\n"
+            "  `x = `x;",
+            format("if (x)\n"
+                   "`timescale 1ns / 1ps\n"
+                   "`x = `x;",
+                   Style));
+}
+
 } // namespace format
 } // end namespace clang