[LNT] Python 3 support: don't assume order of cmake args

[thopre]

Summary

runtest/test_suite-cache.shtest test assumes the order of cmake
arguments but some of these are stored in the same dictionary which does
not guarantee any specific ordering. This causes the test to fail on
Python 3.

In particular, CMAKE_C_COMPILER and CMAKE_CXX_COMPILER must come first
but can be in any particular order between each other, then cmake cache
option is expected to come next followed by other option in any
particular order. Since there is no good reason to have arguments in a
given group of option sorted, this commit adapts the testcase to accept
any order by testing each group of argument in a separate FileCheck
invocation.

[rogfer01]

Thanks for the patch @thopre ! I've got a comment above.

tests/runtest/test_suite-cache.shtest

18

My reading from this

for key in ['CMAKE_C_COMPILER:FILEPATH',                          
            'CMAKE_CXX_COMPILER:FILEPATH']:                       
    value = defs.pop(key, None)                                   
    if value is not None:                                         
        early_defs[key] = value                                   
                                                                  
cmake_cmd = ([cmake_cmd] +                                        
             ['-D%s=%s' % (k, v) for k, v in early_defs.items()] +
             cmake_flags + [self._test_suite_dir()] +             
             ['-D%s=%s' % (k, v) for k, v in defs.items()])

is that we may still want to check that the compilers and the cache appear in the specific order (right after the cmake).

The other ones, whose order is unpredictable, (-DFOO=BAR, -DBAR=BAZ) we do want to check them individually,

Does this make sense?

[thopre]

Check that CMAKE_C(XX)?_COMPILER come first on cmake invocation

[thopre]

tests/runtest/test_suite-cache.shtest

18

It does, good catch! How about now?

[rogfer01]

tests/runtest/test_suite-cache.shtest

18

Would it be possible to ensure the -C flag does appear right after the compilers?

It occurs to me that either we make Python 2 deterministic here using an OrderedDict for early_defs (and then we can write a simples check: first C, then CXX, then the cache file) or we write a check like the one below (untested!) where we allow any order and then the cache file

# We allow compilers in either order for Python 2 compat
# CHECK-CACHE1: Execute: {{.*}}cmake {{(-DCMAKE_CXX_COMPILER:FILEPATH=.*/FakeCompilers/clang\+\+-r154331 -DCMAKE_C_COMPILER:FILEPATH=.*/FakeCompilers/clang-r154331 )|(-DCMAKE_C_COMPILER:FILEPATH=.*/FakeCompilers/clang-r154331 -DCMAKE_CXX_COMPILER:FILEPATH=.*/FakeCompilers/clang\+\+-r154331 )}} -C {{.*}}/Release.cmake

(cmake documentation does not seem clear to me regarding the positional behaviour of -C and -D but I imagine early_defs exists for a reason)

[thopre]

Restrict ordering of options further according to what the code does.

[thopre]

tests/runtest/test_suite-cache.shtest

18

Oh I see, early_defs, then cmake_flags (which includes -C option), then test_suite_dir then macro definitions.

[rogfer01]

LGTM.

Thanks again @thopre !

[thopre]

Committed as d52e617d81aa62182f45d5d8e1f153f7c0c43a42

tests/runtest/test_suite-cache.shtest

 # Check a run of test-suite using a cmake cache
 # Also make sure we get: compiler defines, cache, other defines on the cmake
 # commandline.
 # RUN: rm -rf %t.SANDBOX
 # RUN: lnt runtest test-suite \
 # RUN:     --sandbox %t.SANDBOX \
 # RUN:     --no-timestamp \
 # RUN:     --test-suite %S/Inputs/test-suite-cmake \
 # RUN:     --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
 # RUN:     --use-cmake %S/Inputs/test-suite-cmake/fake-cmake \
 # RUN:     --use-make %S/Inputs/test-suite-cmake/fake-make \
 # RUN:     --use-lit %S/Inputs/test-suite-cmake/fake-lit \
 # RUN:     --cmake-cache Release \
 # RUN:     --cmake-define FOO=BAR \
 # RUN:     -D BAR=BAZ \
 # RUN:     &> %t.cmake-cache.log
-# RUN: FileCheck  --check-prefix CHECK-CACHE < %t.cmake-cache.log %s
-# CHECK-CACHE: Execute: {{.*}}cmake -DCMAKE_CXX_COMPILER:FILEPATH={{.*}}/FakeCompilers/clang++-r154331 -DCMAKE_C_COMPILER:FILEPATH={{.*}}FakeCompilers/clang-r154331 -C {{.*}}/Release.cmake {{.*}}-DFOO=BAR{{.*}}-DBAR=BAZ
+
+# Check CMAKE_C_COMPILER and CMAKE_CXX_COMPILER comes first, followed by cmake
+# options. This is verified by checking that:
+# 1) there's 2 cmake compiler definition followed by the -C option
+# 2) each of the cmake compiler definition only accepts the other cmake
+#    compiler definition between cmake and itself
+# RUN: FileCheck  --check-prefix CHECK-CACHE1 < %t.cmake-cache.log %s
+# CHECK-CACHE1: Execute: {{.*}}cmake -DCMAKE_C{{(XX)?}}_COMPILER:FILEPATH={{.*}}/FakeCompilers/clang{{(\+\+)?}}-r154331 -DCMAKE_C{{(XX)?}}_COMPILER:FILEPATH={{.*}}/FakeCompilers/clang{{(\+\+)?}}-r154331 -C {{.*}}/Release.cmake
+# RUN: FileCheck  --check-prefix CHECK-CACHE2 < %t.cmake-cache.log %s
+# CHECK-CACHE2: Execute: {{.*}}cmake {{(-DCMAKE_CXX_COMPILER:FILEPATH=.*/FakeCompilers/clang\+\+-r154331 )?}}-DCMAKE_C_COMPILER:FILEPATH={{.*}}/FakeCompilers/clang-r154331
+# RUN: FileCheck  --check-prefix CHECK-CACHE3 < %t.cmake-cache.log %s
+# CHECK-CACHE3: Execute: {{.*}}cmake {{(-DCMAKE_C_COMPILER:FILEPATH=.*/FakeCompilers/clang-r154331 )?}}-DCMAKE_CXX_COMPILER:FILEPATH={{.*}}/FakeCompilers/clang++-r154331
+
+# RUN: FileCheck  --check-prefix CHECK-CACHE4 < %t.cmake-cache.log %s
+# CHECK-CACHE4: Execute: {{.*}}cmake {{(.+ )?}}-DFOO=BAR
+# RUN: FileCheck  --check-prefix CHECK-CACHE5 < %t.cmake-cache.log %s
+# CHECK-CACHE5: Execute: {{.*}}cmake {{(.+ )?}}-DBAR=BAZ
 
 # RUN: rm -rf %t.SANDBOX
 # Check a run of test-suite using a invalid cmake cache
 # RUN: lnt runtest test-suite \
 # RUN:     --sandbox %t.SANDBOX \
 # RUN:     --no-timestamp \
 # RUN:     --test-suite %S/Inputs/test-suite-cmake \
 # RUN:     --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
 # RUN:     --use-cmake %S/Inputs/test-suite-cmake/fake-cmake \
 # RUN:     --use-make %S/Inputs/test-suite-cmake/fake-make \
 # RUN:     --use-lit %S/Inputs/test-suite-cmake/fake-lit \
 # RUN:     --cmake-cache Debug \
 # RUN:     &> %t.cmake-cache2.err || true
-# RUN: FileCheck  --check-prefix CHECK-CACHE2 < %t.cmake-cache2.err %s
-# CHECK-CACHE2: Could not find CMake cache file
+# RUN: FileCheck  --check-prefix CHECK-CACHE6 < %t.cmake-cache2.err %s
+# CHECK-CACHE6: Could not find CMake cache file