[docs][clang-format] warn on \code block indentation error

[FederAndInk]

Summary

There is an indentation issue in Format.h causing the html to not render correctly
It's a \code block in the documentation of SpacesInLineComment

So I fixed it, and added some warnings in clang/docs/tools/dump_format_style.py:

• fix intentation of SpacesInLineComment
• warn on indentation error
• also warn on \code \endcode mismatch
• generate precise warnings
• fix some minor and style issues:
  • avoid confusion with the built-in type function
  • fix wrong print on os.sys.stderr (instead of sys.stderr)
  • use with as pattern for files

here is a look at the generated warnings:

> ./dump_format_style.py
../../include/clang/Format/Format.h:3448: warning: code block should be indented:
  3448 | /// before:                                   after:
  3449 | /// Minimum: 1
  3450 | /// //if (b) {                                // if (b) {
  3451 | /// //  return true;                          //   return true;
  3452 | /// //}                                       // }
../../include/clang/Format/Format.h:3454: warning: code block should be indented:
  3454 | /// Maximum: 0
  3455 | /// /// List:                                 ///List:
  3456 | /// ///  - Foo                                /// - Foo
  3457 | /// ///    - Bar                              ///   - Bar

[MyDeveloperDay]

It seems like you are fixing more than just the bug you highlight, i.e looks you might be fixing some autopep8 or pylint issues too.

As this isn't a tool we ship, I don't have too many concerns, but others may have other views. But it looks good to me.

[FederAndInk]

In D112572#3088800, @MyDeveloperDay wrote:

It seems like you are fixing more than just the bug you highlight, i.e looks you might be fixing some autopep8 or pylint issues too.

Yes, I also fixed some minor issues, my bad for not adding it into the summary right away, I'll edit that.

[HazardyKnusperkeks]

Thanks for fixing my mistake. :)

[FederAndInk]

In D112572#3089318, @HazardyKnusperkeks wrote:

Thanks for fixing my mistake. :)

You're welcome ;)

Thanks for your reviews.

Should we add other reviewers?

Also, I don't have commit rights, can one of you commit it on my behalf?

My name: Ludovic Jozeau
My email: [e-mail]

Many thanks

[HazardyKnusperkeks]

In D112572#3089373, @FederAndInk wrote:

In D112572#3089318, @HazardyKnusperkeks wrote:

Thanks for fixing my mistake. :)

You're welcome ;)

Thanks for your reviews.

Should we add other reviewers?

Also, I don't have commit rights, can one of you commit it on my behalf?

My name: Ludovic Jozeau
My email: [e-mail]

Many thanks

You certainly add more reviewers, but I don't think there is a need for that. I will push it for you this weekend, if MyDeveloperDay doesn't beat me to it.

clang/docs/ClangFormatStyleOptions.rst

.. code-block:: js
      true:                                  false:
      var arr = [ 1, 2, 3 ];         vs.     var arr = [1, 2, 3];
      f({a : 1, b : 2, c : 3});              f({a: 1, b: 2, c: 3});
 
 **SpacesInLineCommentPrefix** (``SpacesInLineComment``) :versionbadge:`clang-format 14`
   How many spaces are allowed at the start of a line comment. To disable the
   maximum set it to ``-1``, apart from that the maximum takes precedence
   over the minimum.
-  Minimum = 1 Maximum = -1
+
+  .. code-block:: c++
+
+    Minimum = 1
+    Maximum = -1
     // One space is forced
 
     //  but more spaces are possible
 
     Minimum = 0
     Maximum = 0
     //Forces to start every comment directly after the slashes
 
   Note that in line comment sections the relative indent of the subsequent
   lines is kept, that means the following:
 
   .. code-block:: c++
 
     before:                                   after:
     Minimum: 1
     //if (b) {                                // if (b) {
     //  return true;                          //   return true;
     //}                                       // }
 
     Maximum: 0
     /// List:                                 ///List:
     ///  - Foo                                /// - Foo
     ///    - Bar                              ///   - Bar
 
   Nested configuration flags:
 
 
   * ``unsigned Minimum`` The minimum number of spaces at the start of the comment.
 
   * ``unsigned Maximum`` The maximum number of spaces at the start of the comment.
 

clang/docs/tools/dump_format_style.py

 #!/usr/bin/env python3
 # A tool to parse the FormatStyle struct from Format.h and update the
 # documentation in ../ClangFormatStyleOptions.rst automatically.
 # Run from the directory in which this file is located to update the docs.
 
 import inspect
 import os
 import re
+import sys
+from io import TextIOWrapper
 from typing import Set
 
 CLANG_DIR = os.path.join(os.path.dirname(__file__), '../..')
 FORMAT_STYLE_FILE = os.path.join(CLANG_DIR, 'include/clang/Format/Format.h')
 INCLUDE_STYLE_FILE = os.path.join(CLANG_DIR, 'include/clang/Tooling/Inclusions/IncludeStyle.h')
 DOC_FILE = os.path.join(CLANG_DIR, 'docs/ClangFormatStyleOptions.rst')
 
 PLURALS_FILE = os.path.join(os.path.dirname(__file__), 'plurals.txt')
if plural not in plurals:
 
     plurals.add(plural)
     with open(PLURALS_FILE, 'a') as f:
       f.write(plural + '\n')
     cf = inspect.currentframe()
     lineno = ''
     if cf and cf.f_back:
       lineno = ':' + str(cf.f_back.f_lineno)
-    print(f'{__file__}{lineno} check if plural of {singular} is {plural}', file=os.sys.stderr)
+    print(f'{__file__}{lineno} check if plural of {singular} is {plural}', file=sys.stderr)
   return plural
 
 def pluralize(word: str):
   lword = word.lower()
   if len(lword) >= 2 and lword[-1] == 'y' and lword[-2] not in 'aeiou':
     return register_plural(word, word[:-1] + 'ies')
   elif lword.endswith(('s', 'sh', 'ch', 'x', 'z')):
     return register_plural(word, word[:-1] + 'es')

 
 def doxygen2rst(text):
   text = re.sub(r'<tt>\s*(.*?)\s*<\/tt>', r'``\1``', text)
   text = re.sub(r'\\c ([^ ,;\.]+)', r'``\1``', text)
   text = re.sub(r'\\\w+ ', '', text)
   return text
 
 def indent(text, columns, indent_first_line=True):
-  indent = ' ' * columns
-  s = re.sub(r'\n([^\n])', '\n' + indent + '\\1', text, flags=re.S)
+  indent_str = ' ' * columns
+  s = re.sub(r'\n([^\n])', '\n' + indent_str + '\\1', text, flags=re.S)
   if not indent_first_line or s.startswith('\n'):
     return s
-  return indent + s
+  return indent_str + s
 
 class Option(object):
-  def __init__(self, name, type, comment, version):
+  def __init__(self, name, opt_type, comment, version):
     self.name = name
-    self.type = type
+    self.type = opt_type
     self.comment = comment.strip()
     self.enum = None
     self.nested_struct = None
     self.version = version
 
   def __str__(self):
     if self.version:
       s = '**%s** (``%s``) :versionbadge:`clang-format %s`\n%s' % (self.name, to_yaml_type(self.type), self.version,
def __init__(self, name, enumtype, comment, values):
     self.comment = comment
     self.values = values
     self.type = enumtype
 
   def __str__(self):
     s = '\n* ``%s %s``\n%s' % (to_yaml_type(self.type), self.name,
                                  doxygen2rst(indent(self.comment, 2)))
     s += indent('\nPossible values:\n\n', 2)
-    s += indent('\n'.join(map(str, self.values)),2)
-    return s;
+    s += indent('\n'.join(map(str, self.values)), 2)
+    return s
 
 class EnumValue(object):
   def __init__(self, name, comment, config):
     self.name = name
     self.comment = comment
     self.config = config
 
   def __str__(self):
     return '* ``%s`` (in configuration: ``%s``)\n%s' % (
         self.name,
         re.sub('.*_', '', self.config),
         doxygen2rst(indent(self.comment, 2)))
 
-def clean_comment_line(line):
+
+class OptionsReader:
+  def __init__(self, header: TextIOWrapper):
+    self.header = header
+    self.in_code_block = False
+    self.code_indent = 0
+    self.lineno = 0
+    self.last_err_lineno = -1
+
+  def __file_path(self):
+    return os.path.relpath(self.header.name)
+
+  def __print_line(self, line: str):
+    print(f'{self.lineno:>6} | {line}', file=sys.stderr)
+
+  def __warning(self, msg: str, line: str):
+    print(f'{self.__file_path()}:{self.lineno}: warning: {msg}:', file=sys.stderr)
+    self.__print_line(line)
+
+  def __clean_comment_line(self, line: str):
     match = re.match(r'^/// (?P<indent> +)?\\code(\{.(?P<lang>\w+)\})?$', line)
     if match:
-    indent = match.group('indent')
-    if not indent:
-      indent = ''
+      if self.in_code_block:
+        self.__warning('`\\code` in another `\\code`', line)
+      self.in_code_block = True
+      indent_str = match.group('indent')
+      if not indent_str:
+        indent_str = ''
+      self.code_indent = len(indent_str)
       lang = match.group('lang')
       if not lang:
         lang = 'c++'
-    return '\n%s.. code-block:: %s\n\n' % (indent, lang)
+      return f'\n{indent_str}.. code-block:: {lang}\n\n'
 
     endcode_match = re.match(r'^/// +\\endcode$', line)
     if endcode_match:
+      if not self.in_code_block:
+        self.__warning('no correct `\\code` found before this `\\endcode`', line)
+      self.in_code_block = False
       return ''
 
+    # check code block indentation
+    if (self.in_code_block and not line == '///' and not
+        line.startswith('///  ' + ' ' * self.code_indent)):
+      if self.last_err_lineno == self.lineno - 1:
+        self.__print_line(line)
+      else:
+        self.__warning('code block should be indented', line)
+      self.last_err_lineno = self.lineno
+
     match = re.match(r'^/// \\warning$', line)
     if match:
       return '\n.. warning:: \n\n'
 
     endwarning_match = re.match(r'^/// +\\endwarning$', line)
     if endwarning_match:
       return ''
     return line[4:] + '\n'
 
-def read_options(header):
-  class State(object):
+  def read_options(self):
+    class State:
       BeforeStruct, Finished, InStruct, InNestedStruct, InNestedFieldComment, \
         InFieldComment, InEnum, InEnumMemberComment = range(8)
     state = State.BeforeStruct
 
     options = []
     enums = {}
     nested_structs = {}
     comment = ''
     enum = None
     nested_struct = None
     version = None
 
-  for line in header:
+    for line in self.header:
+      self.lineno += 1
       line = line.strip()
       if state == State.BeforeStruct:
-      if line == 'struct FormatStyle {' or line == 'struct IncludeStyle {':
+        if line in ('struct FormatStyle {', 'struct IncludeStyle {'):
           state = State.InStruct
       elif state == State.InStruct:
         if line.startswith('///'):
           state = State.InFieldComment
-        comment = clean_comment_line(line)
+          comment = self.__clean_comment_line(line)
         elif line == '};':
           state = State.Finished
           break
       elif state == State.InFieldComment:
         if line.startswith(r'/// \version'):
-        match = re.match(r'/// \\version\s*(?P<version>[0-9.]+)*',line)
+          match = re.match(r'/// \\version\s*(?P<version>[0-9.]+)*', line)
           if match:
             version = match.group('version')
         elif line.startswith('///'):
-        comment += clean_comment_line(line)
+          comment += self.__clean_comment_line(line)
         elif line.startswith('enum'):
           state = State.InEnum
           name = re.sub(r'enum\s+(\w+)\s*(:((\s*\w+)+)\s*)?\{', '\\1', line)
           enum = Enum(name, comment)
         elif line.startswith('struct'):
           state = State.InNestedStruct
           name = re.sub(r'struct\s+(\w+)\s*\{', '\\1', line)
           nested_struct = NestedStruct(name, comment)
         elif line.endswith(';'):
           state = State.InStruct
           field_type, field_name = re.match(r'([<>:\w(,\s)]+)\s+(\w+);',
                                             line).groups()
 
           if not version:
-            print('Warning missing version for ', field_name)
+            self.__warning(f'missing version for {field_name}', line)
           option = Option(str(field_name), str(field_type), comment, version)
           options.append(option)
-        version=None
+          version = None
         else:
-        raise Exception('Invalid format, expected comment, field or enum\n'+line)
+          raise Exception('Invalid format, expected comment, field or enum\n' + line)
       elif state == State.InNestedStruct:
         if line.startswith('///'):
           state = State.InNestedFieldComment
-        comment = clean_comment_line(line)
+          comment = self.__clean_comment_line(line)
         elif line == '};':
           state = State.InStruct
           nested_structs[nested_struct.name] = nested_struct
       elif state == State.InNestedFieldComment:
         if line.startswith('///'):
-        comment += clean_comment_line(line)
+          comment += self.__clean_comment_line(line)
         else:
           state = State.InNestedStruct
-        field_type, field_name = re.match(r'([<>:\w(,\s)]+)\s+(\w+);',line).groups()
+          field_type, field_name = re.match(r'([<>:\w(,\s)]+)\s+(\w+);', line).groups()
           if field_type in enums:
-            nested_struct.values.append(NestedEnum(field_name,field_type,comment,enums[field_type].values))
+            nested_struct.values.append(NestedEnum(field_name,
+                                                   field_type,
+                                                   comment,
+                                                   enums[field_type].values))
           else:
             nested_struct.values.append(NestedField(field_type + " " + field_name, comment))
 
       elif state == State.InEnum:
         if line.startswith('///'):
           state = State.InEnumMemberComment
-        comment = clean_comment_line(line)
+          comment = self.__clean_comment_line(line)
         elif line == '};':
           state = State.InStruct
           enums[enum.name] = enum
         else:
           # Enum member without documentation. Must be documented where the enum
           # is used.
           pass
       elif state == State.InEnumMemberComment:
         if line.startswith('///'):
-        comment += clean_comment_line(line)
+          comment += self.__clean_comment_line(line)
         else:
           state = State.InEnum
           val = line.replace(',', '')
           pos = val.find(" // ")
-        if (pos != -1):
-            config = val[pos+4:]
+          if pos != -1:
+            config = val[pos + 4:]
             val = val[:pos]
           else:
-            config = val;
-        enum.values.append(EnumValue(val, comment,config))
+            config = val
+          enum.values.append(EnumValue(val, comment, config))
     if state != State.Finished:
       raise Exception('Not finished by the end of file')
 
     for option in options:
-    if not option.type in ['bool', 'unsigned', 'int', 'std::string',
+      if option.type not in ['bool', 'unsigned', 'int', 'std::string',
                              'std::vector<std::string>',
                              'std::vector<IncludeCategory>',
                              'std::vector<RawStringFormat>']:
         if option.type in enums:
           option.enum = enums[option.type]
         elif option.type in nested_structs:
           option.nested_struct = nested_structs[option.type]
         else:
           raise Exception('Unknown type: %s' % option.type)
     return options
 
-options = read_options(open(FORMAT_STYLE_FILE))
-options += read_options(open(INCLUDE_STYLE_FILE))
 
-options = sorted(options, key=lambda x: x.name)
-options_text = '\n\n'.join(map(str, options))
+with open(FORMAT_STYLE_FILE) as f:
+  opts = OptionsReader(f).read_options()
+with open(INCLUDE_STYLE_FILE) as f:
+  opts += OptionsReader(f).read_options()
+
+opts = sorted(opts, key=lambda x: x.name)
+options_text = '\n\n'.join(map(str, opts))
 
-contents = open(DOC_FILE).read()
+with open(DOC_FILE) as f:
+  contents = f.read()
 
 contents = substitute(contents, 'FORMAT_STYLE_OPTIONS', options_text)
 
 with open(DOC_FILE, 'wb') as output:
   output.write(contents.encode())

clang/include/clang/Format/Format.h

struct SpacesInLineComment {
     unsigned Minimum;
     /// The maximum number of spaces at the start of the comment.
     unsigned Maximum;
   };
 
   /// How many spaces are allowed at the start of a line comment. To disable the
   /// maximum set it to ``-1``, apart from that the maximum takes precedence
   /// over the minimum.
-  /// \code Minimum = 1 Maximum = -1
-  /// // One space is forced
+  /// \code
+  ///   Minimum = 1
+  ///   Maximum = -1
+  ///   // One space is forced
   ///
-  /// //  but more spaces are possible
+  ///   //  but more spaces are possible
   ///
-  /// Minimum = 0
-  /// Maximum = 0
-  /// //Forces to start every comment directly after the slashes
+  ///   Minimum = 0
+  ///   Maximum = 0
+  ///   //Forces to start every comment directly after the slashes
   /// \endcode
   ///
   /// Note that in line comment sections the relative indent of the subsequent
   /// lines is kept, that means the following:
   /// \code
-  /// before:                                   after:
-  /// Minimum: 1
-  /// //if (b) {                                // if (b) {
-  /// //  return true;                          //   return true;
-  /// //}                                       // }
+  ///   before:                                   after:
+  ///   Minimum: 1
+  ///   //if (b) {                                // if (b) {
+  ///   //  return true;                          //   return true;
+  ///   //}                                       // }
   ///
-  /// Maximum: 0
-  /// /// List:                                 ///List:
-  /// ///  - Foo                                /// - Foo
-  /// ///    - Bar                              ///   - Bar
+  ///   Maximum: 0
+  ///   /// List:                                 ///List:
+  ///   ///  - Foo                                /// - Foo
+  ///   ///    - Bar                              ///   - Bar
   /// \endcode
   /// \version 14
   SpacesInLineComment SpacesInLineCommentPrefix;
 
   /// If ``true``, spaces will be inserted after ``(`` and before ``)``.
   /// \code
   ///    true:                                  false:
   ///    t f( Deleted & ) & = delete;   vs.     t f(Deleted &) & = delete;