[clang][utils] Modify make-ast-dump-check.sh to generate AST serialization dump tests

[riccibruno]

Summary

An AST serialization dump test is a test which compares the output of -ast-dump on the source and of -ast-dump-all on a PCH generated from the source. Modulo a few differences the outputs should match.

This patch to make-ast-dump-check.sh enables automatically generating these tests.

[riccibruno]

Example of a test:

ast-dump-lambda.cpp24 KBDownload

[lebedev.ri]

+1, thank you for looking into this!
I suspect there's a few issues to be discovered with this :)

[riccibruno]

In D81786#2091387, @lebedev.ri wrote:

+1, thank you for looking into this!
I suspect there's a few issues to be discovered with this :)

Indeed; I already got one in D81787.

[riccibruno]

Summarising a quick discussion with @lebedev.ri on IRC yesterday: this is not necessarily the best way to test the serialization of AST nodes, in that we could instead perform some kind of structural equivalence test. However this approach has the huge advantage of being simply a matter of 1. Adding a few run lines to each ast-dump-*, and 2. running this script on the test.
(see rG6a79f5aa5dbc2528444b4dfb92bb68039c5a32e9 for an example of what the output looks like)

What I'd like to do (and get approval for, since I don't think it is worthwhile to individually put each changed test for review) is go over each AST dump tests (ie: each ast-dump-*) and do the above (possibly splitting some tests intro multiple files if they are too big).

Edit: Excluding AST dump tests which are about the serialization itself, or modules.

[aaron.ballman]

In D81786#2091781, @riccibruno wrote:

Summarising a quick discussion with @lebedev.ri on IRC yesterday: this is not necessarily the best way to test the serialization of AST nodes, in that we could instead perform some kind of structural equivalence test. However this approach has the huge advantage of being simply a matter of 1. Adding a few run lines to each ast-dump-*, and 2. running this script on the test.
(see rG6a79f5aa5dbc2528444b4dfb92bb68039c5a32e9 for an example of what the output looks like)

What I'd like to do (and get approval for, since I don't think it is worthwhile to individually put each changed test for review) is go over each AST dump tests (ie: each ast-dump-*) and do the above (possibly splitting some tests intro multiple files if they are too big).

Edit: Excluding AST dump tests which are about the serialization itself, or modules.

This seems reasonable to me. LGTM

[lebedev.ri]

SGTM, thank you!

clang/utils/make-ast-dump-check.sh

 #! /bin/bash
 
 # This script is intended as a FileCheck replacement to update the test
 # expectations in a -ast-dump test.
 #
-# Usage:
+# Usage (to generate normal AST dump tests):
 #
 # $ lit -DFileCheck=$PWD/utils/make-ast-dump-check.sh test/AST/ast-dump-openmp-*
+#
+# Usage (to generate serialization AST dump tests):
+#
+# $ lit -DFileCheck="generate_serialization_test=1 $PWD/utils/make-ast-dump-check.sh"
+#     test/AST/ast-dump-openmp-*
 
 prefix=CHECK
 
+if [ -z ${generate_serialization_test+x} ];
+  then generate_serialization_test=0;
+fi
+
 while [[ "$#" -ne 0 ]]; do
   case "$1" in
   --check-prefix)
     shift
     prefix="$1"
     ;;
   --implicit-check-not)
     shift

 /TranslationUnitDecl/ {
   skipping_builtins = 1
 }
 
 {
   s = \$0
   gsub("0x[0-9a-fA-F]+", "{{.*}}", s)
   gsub("$testdir/", "{{.*}}", s)
+  if ($generate_serialization_test == 1) {
+    gsub(" imported", "{{( imported)?}}", s)
+    gsub(" <undeserialized declarations>", "{{( <undeserialized declarations>)?}}", s)
+  }
 }
 
 matched_last_line == 0 {
   print "// ${prefix}: " s
 }
 
 matched_last_line == 1 {
   print "// ${prefix}-NEXT: " s