[libc++] improve feature test macro script

[WimLeflere]

Summary

I've been playing a bit with the generate_feature_test_macro_components.py script and replaced some hardcoded values with extra code generation (generate ALL the things).
The output is the same and it makes updating the script less work for the coming 25 C++ standards (until 2 digit number overflow).

Feel free to 'veto' if you think it's overkill.

[• Quuxplusone]

Seems like a generally good direction to me, FWIW.

libcxx/utils/generate_feature_test_macro_components.py

905

I see no obvious reason for the special case here. Sure we currently do it as #if TEST_STD_VER < 14, but would anything really break if we did it as #if TEST_STD_VER == 11 instead? It'd be worth trying to find out.

918

I think this should just be assert not std_dialects[-1][-1].isnumeric(), and then simplify accordingly. (That would technologically enforce my earlier opinion that whoever changes 2a to 20 should go ahead and add 2b at the same time.)

925

FWIW, I would recommend copying the style from circa line 800: use template.format("""...""") instead of f-strings, and use one big """ string with all the arguments collected at the bottom, instead of a bunch of little strings append'ed together. (This also solves your question about whether f-strings are allowed, because you'd no longer need to use f-strings.)

1031

This [1:] smells questionable.

[curdeius]

Concerning Python version, I don't know of any official policy but given that Python 3.5 is already end-of-life, requiring Python 3.6 seems OK.

[ldionne]

The direction looks good to me too.

In D94530#2493796, @curdeius wrote:

Concerning Python version, I don't know of any official policy but given that Python 3.5 is already end-of-life, requiring Python 3.6 seems OK.

Technically, LLVM decided to support Python 2.7 until January 2021 if I remember correctly. So I think we might be good.

But in all cases, this script is only run by libc++ contributors, and I think it's a more-than-reasonable requirement for that.

libcxx/utils/generate_feature_test_macro_components.py

708

I assume nr stands for number? I would find it a lot easier to read if you used number instead of trying to save a few characters (here and below).

[WimLeflere]

• added assert for last std not numeric
• replaced f-strings with template.format

[WimLeflere]

replaced nr with number

[WimLeflere]

libcxx/utils/generate_feature_test_macro_components.py

905

a special case is also needed for #if vs #elif, so I would keep it

1031

If c++11 is not skipped an empty table section will be generated for c++11 as it has not macros

[ldionne]

libcxx/utils/generate_feature_test_macro_components.py

1031

It seems to me like the correct thing to do here is to not include c++11 in the dialects in the first place, then?

[WimLeflere]

removed c++11 from the std dialects and updated the code

[WimLeflere]

Any action required from my side?

[ldionne]

Do you need help committing this? If so, please provide Author Name <email@domain>. Thanks!

[WimLeflere]

In D94530#2505309, @ldionne wrote:

Do you need help committing this? If so, please provide Author Name <email@domain>. Thanks!

Is it an action I can take in the web interface?
Do I need certain access rights?

Info: Wim Leflere [e-mail]

[ldionne]

In D94530#2505331, @WimLeflere wrote:

In D94530#2505309, @ldionne wrote:

Do you need help committing this? If so, please provide Author Name <email@domain>. Thanks!

Is it an action I can take in the web interface?
Do I need certain access rights?

Info: Wim Leflere [e-mail]

The usual workflow for frequent contributors is to commit themselves as explained in https://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access. If you plan on submitting a lot of patches, you should consider getting commit access so that we don't have to commit it for you. For now, it's fine of course -- I'll commit this for you.

Thanks!

libcxx/utils/generate_feature_test_macro_components.py

lit_markup = {
   "ostream": ["UNSUPPORTED: libcpp-has-no-localization"],
   "regex": ["UNSUPPORTED: libcpp-has-no-localization"],
   "semaphore": ["UNSUPPORTED: libcpp-has-no-threads"],
   "shared_mutex": ["UNSUPPORTED: libcpp-has-no-threads"],
   "thread": ["UNSUPPORTED: libcpp-has-no-threads"],
 }
 
 def get_std_dialects():
-  std_dialects = ['c++14', 'c++17', 'c++20', 'c++2b']
+  std_dialects = ['c++11','c++14', 'c++17', 'c++20', 'c++2b']
   return list(std_dialects)
 
 def get_first_std(d):
     for s in get_std_dialects():
         if s in d.keys():
             return s
     return None
 
def get_for_std(d, std):
   # Find the value for the newest C++ dialect between C++14 and std
   std_list = list(std_dialects[0:std_dialects.index(std)+1])
   std_list.reverse()
   for s in std_list:
     if s in d.keys():
       return d[s]
   return None
 
+def get_std_number(std):
+    return std.replace('c++', '')
 
 """
   Functions to produce the <version> header
 """
 
 def produce_macros_definition_for_std(std):
   result = ""
   indent = 56
for tc in feature_test_macros:
     line += " " * (indent - len(line))
     line += "%sL" % tc["values"][std]
     if 'unimplemented' in tc.keys():
       line = "// " + line
     result += line
     result += "\n"
     if 'depends' in tc.keys():
       result += "# endif\n"
-  return result
+  return result.strip()
+
+def produce_macros_definitions():
+  macro_definition_template = """#if _LIBCPP_STD_VER > {previous_std_number}
+{macro_definition}
+#endif"""
+
+  std_dialects = get_std_dialects()
+
+  macros_definitions = []
+  for previous, current in zip(std_dialects, std_dialects[1:]):
+      macros_definitions.append(
+        macro_definition_template.format(previous_std_number=get_std_number(previous),
+                                         macro_definition=produce_macros_definition_for_std(current)))
+
+  return '\n\n'.join(macros_definitions)
 
 def chunks(l, n):
   """Yield successive n-sized chunks from l."""
   for i in range(0, len(l), n):
     yield l[i:i + n]
 
 def produce_version_synopsis():
   indent = 56

 */
 
 #include <__config>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
-#if _LIBCPP_STD_VER > 11
-{cxx14_macros}
-#endif
-
-#if _LIBCPP_STD_VER > 14
-{cxx17_macros}
-#endif
-
-#if _LIBCPP_STD_VER > 17
-{cxx20_macros}
-#endif
-
-#if _LIBCPP_STD_VER > 20
-{cxx2b_macros}
-#endif
+{cxx_macros}
 
 #endif // _LIBCPP_VERSIONH
 """
 
   version_str = template.format(
       synopsis=produce_version_synopsis().strip(),
-      cxx14_macros=produce_macros_definition_for_std('c++14').strip(),
-      cxx17_macros=produce_macros_definition_for_std('c++17').strip(),
-      cxx20_macros=produce_macros_definition_for_std('c++20').strip(),
-      cxx2b_macros=produce_macros_definition_for_std('c++2b').strip())
+      cxx_macros=produce_macros_definitions())
   version_header_path = os.path.join(include_path, 'version')
   with open(version_header_path, 'w', newline='\n') as f:
     f.write(version_str)
 
 
 """
     Functions to produce test files
 """
for tc in test_list:
     if val is None:
       result += test_types["undefined"].format(name=tc["name"], std_first=get_first_std(tc["values"]))
     elif 'unimplemented' in tc.keys():
       result += test_types["unimplemented"].format(name=tc["name"], value=val, std=std)
     elif "depends" in tc.keys():
       result += test_types["depends"].format(name=tc["name"], value=val, std=std, depends=tc["depends"])
     else:
       result +=  test_types["defined"].format(name=tc["name"], value=val, std=std)
-  return result
+  return result.strip()
+
+def generate_std_tests(test_list):
+  std_tests_template = """#if TEST_STD_VER < {second_std_number}
+
+{first_std_test}
+
+{other_std_tests}
+
+#elif TEST_STD_VER > {penultimate_std_number}
+
+{last_std_test}
+
+#endif // TEST_STD_VER > {penultimate_std_number}"""
+
+  std_dialects = get_std_dialects()
+  assert not get_std_number(std_dialects[-1]).isnumeric()
+
+  other_std_tests = []
+  for std in std_dialects[1:-1]:
+    other_std_tests.append('#elif TEST_STD_VER == ' + get_std_number(std))
+    other_std_tests.append(generate_std_test(test_list, std))
+
+  std_tests = std_tests_template.format(second_std_number=get_std_number(std_dialects[1]),
+                                        first_std_test=generate_std_test(test_list, std_dialects[0]),
+                                        other_std_tests='\n\n'.join(other_std_tests),
+                                        penultimate_std_number=get_std_number(std_dialects[-2]),
+                                        last_std_test=generate_std_test(test_list, std_dialects[-1]))
+
+  return std_tests
 
 def generate_synopsis(test_list):
     max_name_len = max([len(tc["name"]) for tc in test_list])
     indent = max_name_len + 8
     def mk_line(prefix, suffix):
         return "{prefix: <{max_len}}{suffix}\n".format(prefix=prefix, suffix=suffix,
         max_len=indent)
     result = ""

 
 // Test the feature test macros defined by <{header}>
 
 {synopsis}
 
 #include <{header}>
 #include "test_macros.h"
 
-#if TEST_STD_VER < 14
-
-{cxx11_tests}
-
-#elif TEST_STD_VER == 14
-
-{cxx14_tests}
-
-#elif TEST_STD_VER == 17
-
-{cxx17_tests}
-
-#elif TEST_STD_VER == 20
-
-{cxx20_tests}
-
-#elif TEST_STD_VER > 20
-
-{cxx2b_tests}
-
-#endif // TEST_STD_VER > 20
+{cxx_tests}
 
 int main(int, char**) {{ return 0; }}
 """.format(script_name=script_name,
            header=h,
            markup=('\n{}\n'.format(markup) if markup else ''),
            synopsis=generate_synopsis(test_list),
-           cxx11_tests=generate_std_test(test_list, 'c++11').strip(),
-           cxx14_tests=generate_std_test(test_list, 'c++14').strip(),
-           cxx17_tests=generate_std_test(test_list, 'c++17').strip(),
-           cxx20_tests=generate_std_test(test_list, 'c++20').strip(),
-           cxx2b_tests=generate_std_test(test_list, 'c++2b').strip())
+           cxx_tests=generate_std_tests(test_list))
     test_name = "{header}.version.pass.cpp".format(header=h)
     out_path = os.path.join(macro_test_path, test_name)
     with open(out_path, 'w', newline='\n') as f:
       f.write(test_body)
 
 """
     Produce documentation for the feature test macros
 """

 
 def pad_cell(s, length, left_align=True):
   padding = ((length - len(s)) * ' ')
   return s + padding
 
 
 def get_status_table():
   table = [["Macro Name", "Value"]]
-  for std in get_std_dialects():
+  for std in get_std_dialects()[1:]:
     table += [["**" + std.replace("c++", "C++ ") + "**", ""]]
     for tc in feature_test_macros:
       if std not in tc["values"].keys():
         continue
       value = "``%sL``" % tc["values"][std]
       if 'unimplemented' in tc.keys():
         value = '*unimplemented*'
       table += [["``%s``" % tc["name"], value]]