[libc++] Add format check to CI

[curdeius]

Summary

Note: contrary to what I said previously, I didn't change .clang-format nor utils/generate_feature_test_macro_components.py script.

[curdeius]

FYI, https://clang.llvm.org/docs/ClangFormattedStatus.html shows the status of LLVM parts being clang-formatted.

[miscco]

I really like that we get clang-format to work. Live is simply to short to worry about formatting.

I assume we should simply make one monster commit adding that to the exception list for git blame and be done with it

[ldionne]

In D92229#2420176, @miscco wrote:

I really like that we get clang-format to work. Live is simply to short to worry about formatting.

I assume we should simply make one monster commit adding that to the exception list for git blame and be done with it

That would still make it more difficult to blame stuff if lines move around a lot. Another option would be to mandate that we clang-format everything we touch from now on. Thus, as we make changes to the library, the lines we change (and some amount of surrounding context) would be re-formatted. I'm not sure which approach is best, since I haven't done it before. I'd be curious to hear stories of folks who moved existing codebases to Clang format.

[ldionne]

libcxx/utils/generate_feature_test_macro_components.py

688–690

Shouldn't this be # ifdef (3 spaces) to get the ifdef aligned at 4 columns?

[miscco]

That would be interesting, but I believe that moving /splitting of lines should be common enough that it is already sorted out

[mclow.lists]

Has clang-format been improved enough that it will not "wreck" std::less<void>? Last time I tried it on that code, it pessimized it pretty thoroughly.
Code pasted here for reference:

#if _LIBCPP_STD_VER > 11
template <>
struct _LIBCPP_TEMPLATE_VIS less<void>
{
    template <class _T1, class _T2>
    _LIBCPP_CONSTEXPR_AFTER_CXX11 _LIBCPP_INLINE_VISIBILITY
    auto operator()(_T1&& __t, _T2&& __u) const
    _NOEXCEPT_(noexcept(_VSTD::forward<_T1>(__t) < _VSTD::forward<_T2>(__u)))
    -> decltype        (_VSTD::forward<_T1>(__t) < _VSTD::forward<_T2>(__u))
        { return        _VSTD::forward<_T1>(__t) < _VSTD::forward<_T2>(__u); }
    typedef void is_transparent;
};
#endif

Also, @missco wrote:

I really like that we get clang-format to work. Live is simply to short to worry about formatting.

Writing is once, reading is many, many times.
Formatting that reveals the underlying structure of the code is a time-saver to future you (and others).

[miscco]

In D92229#2420261, @mclow.lists wrote:

Has clang-format been improved enough that it will not "wreck" std::less<void>? Last time I tried it on that code, it pessimized it pretty thoroughly.
Code pasted here for reference:

#if _LIBCPP_STD_VER > 11
template <>
struct _LIBCPP_TEMPLATE_VIS less<void>
{
    template <class _T1, class _T2>
    _LIBCPP_CONSTEXPR_AFTER_CXX11 _LIBCPP_INLINE_VISIBILITY
    auto operator()(_T1&& __t, _T2&& __u) const
    _NOEXCEPT_(noexcept(_VSTD::forward<_T1>(__t) < _VSTD::forward<_T2>(__u)))
    -> decltype        (_VSTD::forward<_T1>(__t) < _VSTD::forward<_T2>(__u))
        { return        _VSTD::forward<_T1>(__t) < _VSTD::forward<_T2>(__u); }
    typedef void is_transparent;
};
#endif

I would like to note that with MSVC STL we simply work around those rare cases via //clang-format off. One main use case are requires clauses that are not yet supported. That said, the vast majority of the code works fine.

Also, @missco wrote:

I really like that we get clang-format to work. Live is simply to short to worry about formatting.

Writing is once, reading is many, many times.
Formatting that reveals the underlying structure of the code is a time-saver to future you (and others).

That is true, but consistency in a code base is incredibly valuable for new contributors. Having to wonder how to indent / format complicated STL code is a really hard hurdle for a first time contributor

Also consistency helps reading a lot. and as said above the few rare cases where it is detrimental one can always simply use // clang-format off

[curdeius]

I agree with @miscco, the advantages of consistent formatting outweigh the inconveniences caused by clang-format's shortcomings.
Comments like "clang-format on/off" are one of the solutions to the problem. Another being to force line breaks with empty trailing comments.

Regarding the strategy of adopting clang-format, the majority of projects I've seen cannot reformat everything at once.
First step is always to enforce git clang-format (but with macro-heavy code care must be taken because the context may accidentally become bigger than intended, at least for all the already released versions of clang-format, there are some improvements in master).
Apart from that, reformatting rarely touched files is another option (there's been a proposal to more or less automatically reformat files that haven't been touched for e.g. 6 months). But that's not a necessity IMO.

[curdeius]

libcxx/utils/generate_feature_test_macro_components.py

688–690

The indentation of preprocessor directives depends on IndentWidth: (currently 2).
I find it a bit inconsistent, as it is:
#, then N * IndentWidth spaces, then directive (so directive is at the N * IndentWidth + 1 column)
against what I used to think and it seems what you think too:
#, N * IndentWidth - 1 spaces, directive (directive being at N * IndentWidth column.

Anyway, I wanted to match how clang-format would format it.
Note that we can keep this script unchanged, it's auto-generated anyway. I'd just keep the addition of clang-format on/off comments.

[ldionne]

I think it's clear that we'll benefit from clang-format.

However, Marshall does have a point that sometimes, you want to format things in a specific way and not have clang-format mess it up. I think it's fine to use // clang-format off for those cases.

@curdeius Could you add a CI job that makes sure that the commit we're testing has been run through git clang-format? You could run it first in the pipeline, and we could even consider adding a wait step in the pipeline so that the rest of the pipeline doesn't run if you haven't clang-formated your changes.

[curdeius]

In D92229#2420375, @ldionne wrote:

@curdeius Could you add a CI job that makes sure that the commit we're testing has been run through git clang-format? You could run it first in the pipeline, and we could even consider adding a wait step in the pipeline so that the rest of the pipeline doesn't run if you haven't clang-formated your changes.

I'll see what I can do!

[Mordante]

Thanks @curdeius for looking into this! The lack of proper clang-format rules for libc++ also bothered me a bit. Ut didn't bother me enough to look into it yet.

I also prefer to use automatic formatting as much as possible. If clang-format has things it doesn't format well, we might want to ask the clang-format devs so they can improve it. Then we can use the automatic formatting after the next release. That would benefit us and the other users of clang-format.

I prefer to have one commit to format everything, then we only need to blacklist one commit in the blame list. I would at least prefer to do all code in include and src in one commit. I don't have a preference whether we should also do the unit tests and benchmarks in one go or fix them when the files are touched.

I don't have a strong preference on which clang-format rules we use. However I think we should decide on the clang-format rules before formatting the code.

[curdeius]

• Break format for testing.
• Add buildkite job for checking formatting.

[curdeius]

Fix bad arc diff base.

[curdeius]

I modified headers in include/ just to test the CI.

[curdeius]

• Wait after Format step.

[curdeius]

• Use tee instead of redirection.

[curdeius]

• Create BUILD_DIR.

[thakis]

If the idea is to reformat everything, I'd recommend just using the normal LLVM style, like the rest of the project.

In D92229#2420240, @ldionne wrote:

In D92229#2420176, @miscco wrote:

I really like that we get clang-format to work. Live is simply to short to worry about formatting.

I assume we should simply make one monster commit adding that to the exception list for git blame and be done with it

That would still make it more difficult to blame stuff if lines move around a lot. Another option would be to mandate that we clang-format everything we touch from now on. Thus, as we make changes to the library, the lines we change (and some amount of surrounding context) would be re-formatted. I'm not sure which approach is best, since I haven't done it before. I'd be curious to hear stories of folks who moved existing codebases to Clang format.

This is a mostly solved problem: Git 2.23 added support for ignoring whitespace commits in blame output. You store hashes that should be blame-transparent in a file called .git-blame-ignore-revs (or any other name), check that file in, and then either pass --ignore-revs-file .git-blame-ignore-revs to git blame or run git config blame.ignoreRevsFile .git-blame-ignore-revs once to make git blame always use that blame file.

We auto-formatted a big chunk of Chromium's code a while ago. We did this before git 2.23 so we wrote a custom tool that does the same thing (https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html), but with git 2.23+ there's even upstream support for this.

[curdeius]

• Bash: set -o pipefail to fail on missing git-clang-format.

[curdeius]

@ldionne, I'll need your help to set this up. Agents need to have git-clang-format installed. I haven't tried but I guess I can't apt install clang-format in the buildkite script and it doesn't sound like a great idea anyway.
And, wouldn't it be possible to reuse the existing machinery in llvm/clang premerge checks that runs git-clang-format? I am afraid it can't be reused because it uses different CI pipeline and different agents...

[curdeius]

• Get only clang-format.patch artifact.

[ldionne]

In D92229#2421298, @thakis wrote:

This is a mostly solved problem: Git 2.23 added support for ignoring whitespace commits in blame output. You store hashes that should be blame-transparent in a file called .git-blame-ignore-revs [...]

I've used that for smaller whitespace-changing patches, however does git do a good job at figuring out the blame correctly when stuff moves around a lot? My concern was that .git-blame-ignore-revs wouldn't work well for such a large reflow, however if it does, then I'd say my concerns are mostly alleviated.

We auto-formatted a big chunk of Chromium's code a while ago. We did this before git 2.23 so we wrote a custom tool that does the same thing (https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html), but with git 2.23+ there's even upstream support for this.

That's large enough to be considered a success story, clearly.

In D92229#2422055, @curdeius wrote:

@ldionne, I'll need your help to set this up. Agents need to have git-clang-format installed. I haven't tried but I guess I can't apt install clang-format in the buildkite script and it doesn't sound like a great idea anyway.

No, indeed. You'd need to update the Dockerfile and install the utility on the macos builders. I'll do it depending on what Mikhail says below -- it's easier for me since they are sitting a meter away :-).

And, wouldn't it be possible to reuse the existing machinery in llvm/clang premerge checks that runs git-clang-format? I am afraid it can't be reused because it uses different CI pipeline and different agents...

I guess it may be possible. @goncharov Mikhail, if we decide to enable clang-format for libc++, would it be possible to re-enable the clang-format checks? IIRC, I think I had requested to remove them cause they were always broken.

libcxx/include/variant

1666–1667

What controls this? I would *love* to move away from declaring the return type of the function above, since it gets lost in our numerous attributes.

libcxx/utils/ci/run-buildbot

111

I think it would be reasonable to set this for the whole script. WDYT?

115

We should also include libcxxabi.

[curdeius]

• Set -o pipefail for the whole script.
• Run git-clang-format also on libcxxabi/.
• Rebase.

[curdeius]

libcxx/include/variant

1666–1667

I guess the hypothetical AlwaysBreak*Before*ReturnType (there is only AlwaysBreakAfterReturnType currently). That shouldn't be something difficult to implement in clang-format I guess.
I agree that the return type can get lost with all the attributes.

[curdeius]

Rebase to trigger CI after buildbot changes in 202df6870ea2ac883ac64fab8e01c836da86978b ("[libc++] Install clang-format on CI nodes").

[curdeius]

Waiting for D93201.

[Mordante]

Do we need to discuss the wanted formatting in this patch or on Slack/dev-ml?

libcxx/.clang-format

5

Do we need to keep this at Cpp03? Changing this to newer versions of C++ will of course break code like 'std::vector<std::pair<int, int>>`. But this seems to break formatting the C++11 string literal prefixes: u8, u, U.

[curdeius]

Concerning the style itself, personally I'd go for a style that is as close to vanilla LLVM style as possible (except for things like disabling comment reflowing for tests, but I'd put this specific .clang-format config file into libcxx/test).
For the moment I'm waiting for the buildbots to be ready so that I can finish the build part.

libcxx/.clang-format

5

I'd love to remove Cpp03... But clang in c++03 mode gives error: error: a space is required between consecutive right angle brackets (use '> >').
BTW, from clang-format documentation: "Cpp03 is a deprecated alias for c++03".
Personally I'd like to use Standard: Latest.
There's also Standard: Auto, that may be a compromise to use > > in c++03 sensitive code and >> elsewhere.

Ref: https://clang.llvm.org/docs/ClangFormatStyleOptions.html

[Mordante]

libcxx/.clang-format

5

True for C++98 the right angle brackets need spaces. AFAIK it's not possible to force that when using a newer C++ version. But C++03 will add a space when using u8"foo" and turn it into u8 "foo". This gives issue with C++11. C++14 code like 1'000'000 will also fail. So maybe we should switch to the newest mode and not format everything, but format per patch basis?

[curdeius]

Rebase to trigger CI after D93201.

[curdeius]

• Use --binary /usr/bin/clang-format-11 as a temporary workaround.

[curdeius]

• Limit folders checked for formatting.

[ldionne]

libcxx/utils/ci/buildkite-pipeline.yml

18

To unblock this and get started even without having entirely settled all formatting questions, I suggest we make this step a soft-fail:

soft_fail:
      - exit_status: 1

(described at https://buildkite.com/docs/pipelines/command-step#soft-fail-attributes)

Then, we'll be able to play around a bit with the configuration. Once we've settled on something, we can remove the soft-fail on the step to enforce formatting for new commits.

[curdeius]

Add soft_fail. Revert .clang-format and generate_feature_test_macro_components.py changes.

[curdeius]

• Remove wait from CI after Format task. Revert changes in <vector> used for testing.

[curdeius]

It would be cool if Buildkite -> Phabricator bridge handled the soft fails.
See previous version https://reviews.llvm.org/D92229?id=320075. It shows CI passed, no warning or whatever, even though Format (soft-) failed, https://buildkite.com/llvm-project/libcxx-ci/builds/1171.
Anyone knows where the code responsible for this is?

[ldionne]

In D92229#2530444, @curdeius wrote:

It would be cool if Buildkite -> Phabricator bridge handled the soft fails.
See previous version https://reviews.llvm.org/D92229?id=320075. It shows CI passed, no warning or whatever, even though Format (soft-) failed, https://buildkite.com/llvm-project/libcxx-ci/builds/1171.
Anyone knows where the code responsible for this is?

@goncharov might know.

This LGTM, I think this is a good start. We can start running that manually before we commit for now, to see what things look like. Once we're comfortable, we can move to the next level and make it mandatory.

libcxx/.clang-format

 BasedOnStyle: LLVM
 
 ---
 Language: Cpp
 Standard: Cpp03
 
+# Lots of code uses indent of 4, but that's not consistent.
+# AccessModifierOffset: -4
+# IndentWidth: 4
+AlignConsecutiveAssignments: true
+AlignTrailingComments: true
+# BreakBeforeBraces: Allman # Usage is not consistent.
+# AlwaysBreakAfterReturnType: All # Do we want it only in templates? We could request and addition of AlwaysBreakBeforeReturnType as well.
+
 AlwaysBreakTemplateDeclarations: true
 PointerAlignment: Left
+IndentPPDirectives: AfterHash
+# SpacesBeforeTrailingComments: 2 # Already inconsistent
 
 # Disable formatting options which may break tests.
 SortIncludes: false
 ReflowComments: false
+# Alternative solution: we can have a different .clang-format in test/. E.g. with:
+# CommentPragmas: '^ expected-' # Alternative to disabling ReflowComments.
+
+# Caveat currently released versions of clang-format mishandle concepts and consider boolean operators as rvalue references, hence using PointerAlignment.
 ---

libcxx/include/variant

struct _LIBCPP_TEMPLATE_VIS hash<
   }
 };
 
 template <>
 struct _LIBCPP_TEMPLATE_VIS hash<monostate> {
   using argument_type = monostate;
   using result_type = size_t;
 
-  inline _LIBCPP_INLINE_VISIBILITY
-  result_type operator()(const argument_type&) const _NOEXCEPT {
+  inline _LIBCPP_INLINE_VISIBILITY result_type
+  operator()(const argument_type&) const _NOEXCEPT {
     return 66740831; // return a fundamentally attractive random value.
   }
 };
 
 #endif  // _LIBCPP_STD_VER > 14
 
 _LIBCPP_END_NAMESPACE_STD
 
 _LIBCPP_POP_MACROS
 
 #endif  // _LIBCPP_VARIANT

libcxx/include/vector

erase(vector<_Tp, _Allocator>& __c, const _Up& __v) {
   return __old_size - __c.size();
 }
 
 template <class _Tp, class _Allocator, class _Predicate>
 inline _LIBCPP_INLINE_VISIBILITY typename vector<_Tp, _Allocator>::size_type
 erase_if(vector<_Tp, _Allocator>& __c, _Predicate __pred) {
   auto __old_size = __c.size();
   __c.erase(_VSTD::remove_if(__c.begin(), __c.end(), __pred), __c.end());
     return __old_size - __c.size();
 }
 #endif
 
 _LIBCPP_END_NAMESPACE_STD
 
 _LIBCPP_POP_MACROS
 
 #endif  // _LIBCPP_VECTOR

libcxx/utils/ci/buildkite-pipeline.yml

 # This file describes the various pre-commit CI bots used to test libc++.
 #
 # This file should never contain logic -- all the logic must be offloaded
 # into scripts. This is critical to being able to reproduce CI issues outside
 # of the CI environment, which is important for debugging.
 #
 
 steps:
+  - label: "Format"
+    command: "libcxx/utils/ci/run-buildbot check-format"
+    artifact_paths:
+      - "**/clang-format.patch"
+    agents:
+      queue: "libcxx-builders"
+    retry:
+      automatic:
+        - exit_status: -1  # Agent was lost
+          limit: 2
+
+  - wait
+
   - label: "C++03"
     command: "libcxx/utils/ci/run-buildbot generic-cxx03"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
     agents:
       queue: "libcxx-builders"
     retry:

libcxx/utils/ci/run-buildbot

 #!/usr/bin/env bash
 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
 set -ex
+set -o pipefail
 
 PROGNAME="$(basename "${0}")"
 
 function usage() {
 cat <<EOF
 Usage:
 ${PROGNAME} [options] <BUILDER>
 

 }
 
 function check-cxx-benchmarks() {
     echo "--- Running the benchmarks"
     ninja -C "${BUILD_DIR}" check-cxx-benchmarks
 }
 
 case "${BUILDER}" in
+check-format)
+    clean
+    echo "+++ Checking formatting"
+    # We need to set --extensions so that clang-format checks extensionless files.
+    mkdir -p ${BUILD_DIR}
+    git-clang-format --diff --extensions ',h,hh,hpp,hxx,c,cc,cxx,cpp' HEAD~1 -- libcxx/ libcxxabi/ | tee ${BUILD_DIR}/clang-format.patch
+    # Check if the diff is empty, fail otherwise.
+    ! grep -q '^--- a' ${BUILD_DIR}/clang-format.patch
+;;
 generic-cxx03)
     export CC=clang
     export CXX=clang++
     clean
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/Generic-cxx03.cmake"
     check-cxx-cxxabi
     check-abi-list
 ;;

libcxx/utils/generate_feature_test_macro_components.py

 """
 
 def produce_macros_definition_for_std(std):
   result = ""
   indent = 56
   for tc in feature_test_macros:
     if std not in tc["values"]:
       continue
-    inner_indent = 1
+    inner_indent = 2
     if 'depends' in tc.keys():
       assert 'internal_depends' in tc.keys()
-      result += "# if %s\n" % tc["internal_depends"]
+      result += "#  if %s\n" % tc["internal_depends"]
       inner_indent += 2
     if get_value_before(tc["values"], std) is not None:
       assert 'depends' not in tc.keys()
-      result += "# undef  %s\n" % tc["name"]
+      result += "#  undef %s\n" % tc["name"]
     line = "#%sdefine %s" % ((" " * inner_indent), tc["name"])
     line += " " * (indent - len(line))
     line += "%sL" % tc["values"][std]
     if 'unimplemented' in tc.keys():
       line = "// " + line
     result += line
     result += "\n"
     if 'depends' in tc.keys():
-      result += "# endif\n"
+      result += "#  endif\n"
   return result
 
 def chunks(l, n):
   """Yield successive n-sized chunks from l."""
   for i in range(0, len(l), n):
     yield l[i:i + n]
 
 def produce_version_synopsis():

 
 {synopsis}
 
 */
 
 #include <__config>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
-#pragma GCC system_header
+#  pragma GCC system_header
 #endif
 
+// clang-format off
+
 #if _LIBCPP_STD_VER > 11
 {cxx14_macros}
 #endif
 
 #if _LIBCPP_STD_VER > 14
 {cxx17_macros}
 #endif
 
 #if _LIBCPP_STD_VER > 17
 {cxx2a_macros}
 #endif
 
+// clang-format on
+
 #endif // _LIBCPP_VERSIONH
 """
 
   version_str = template.format(
       synopsis=produce_version_synopsis().strip(),
       cxx14_macros=produce_macros_definition_for_std('c++14').strip(),
       cxx17_macros=produce_macros_definition_for_std('c++17').strip(),
       cxx2a_macros=produce_macros_definition_for_std('c++2a').strip())
 
   version_header_path = os.path.join(include_path, 'version')
   with open(version_header_path, 'w') as f:
     f.write(version_str)
 
 
 """
     Functions to produce test files
 """
 
 test_types = {
   "undefined": """
-# ifdef {name}
-#   error "{name} should not be defined before {std_first}"
-# endif
+#  ifdef {name}
+#    error "{name} should not be defined before {std_first}"
+#  endif
 """,
 
   "depends": """
-# if {depends}
-#   ifndef {name}
-#     error "{name} should be defined in {std}"
-#   endif
-#   if {name} != {value}
-#     error "{name} should have the value {value} in {std}"
-#   endif
-# else
-#   ifdef {name}
-#     error "{name} should not be defined when {depends} is not defined!"
-#   endif
-# endif
+#  if {depends}
+#    ifndef {name}
+#      error "{name} should be defined in {std}"
+#    endif
+#    if {name} != {value}
+#      error "{name} should have the value {value} in {std}"
+#    endif
+#  else
+#    ifdef {name}
+#      error "{name} should not be defined when {depends} is not defined!"
+#    endif
+#  endif
 """,
 
   "unimplemented": """
-# if !defined(_LIBCPP_VERSION)
-#   ifndef {name}
-#     error "{name} should be defined in {std}"
-#   endif
-#   if {name} != {value}
-#     error "{name} should have the value {value} in {std}"
-#   endif
-# else // _LIBCPP_VERSION
-#   ifdef {name}
-#     error "{name} should not be defined because it is unimplemented in libc++!"
-#   endif
-# endif
+#  if !defined(_LIBCPP_VERSION)
+#    ifndef {name}
+#      error "{name} should be defined in {std}"
+#    endif
+#    if {name} != {value}
+#      error "{name} should have the value {value} in {std}"
+#    endif
+#  else // _LIBCPP_VERSION
+#    ifdef {name}
+#      error "{name} should not be defined because it is unimplemented in libc++!"
+#    endif
+#  endif
 """,
 
   "defined":"""
-# ifndef {name}
-#   error "{name} should be defined in {std}"
-# endif
-# if {name} != {value}
-#   error "{name} should have the value {value} in {std}"
-# endif
+#  ifndef {name}
+#    error "{name} should be defined in {std}"
+#  endif
+#  if {name} != {value}
+#    error "{name} should have the value {value} in {std}"
+#  endif
 """
 }
 
 def generate_std_test(test_list, std):
   result = ""
   for tc in test_list:
     val = get_for_std(tc["values"], std)
     if val is not None:

 
 // Test the feature test macros defined by <{header}>
 
 {synopsis}
 
 #include <{header}>
 #include "test_macros.h"
 
+// clang-format off
+
 #if TEST_STD_VER < 14
 
 {cxx11_tests}
 
 #elif TEST_STD_VER == 14
 
 {cxx14_tests}
 
 #elif TEST_STD_VER == 17
 
 {cxx17_tests}
 
 #elif TEST_STD_VER > 17
 
 {cxx2a_tests}
 
 #endif // TEST_STD_VER > 17
 
+// clang-format on
+
 int main(int, char**) {{ return 0; }}
 """.format(script_name=script_name,
            header=h,
            markup=('\n{}\n'.format(markup) if markup else ''),
            synopsis=generate_synopsis(test_list),
            cxx11_tests=generate_std_test(test_list, 'c++11').strip(),
            cxx14_tests=generate_std_test(test_list, 'c++14').strip(),
            cxx17_tests=generate_std_test(test_list, 'c++17').strip(),