[test-suite] Sort the SPEC file outputs to force a deterministic result

[jdoerfert]

Summary

The wildcard used to reference all files generated by a SPEC benchmark
does not guarantee an ordered result. Probably depending on the default
shell e.g., with zsh 5.2, some SPEC benchmarks* did mistakenly fail.

[MatzeB]

LGTM

BTW: I really recommend using the cmake/lit variant to run SPEC, the Makefiles only run something resembling the SPEC train inputs and are not necessary representativ of the actual performance measured in SPEC.

[jdoerfert]

Thanks for the tip!

I'm (currently almost) only interested in the compilation of SPEC and I have a setup that works with the old "nt" driver. I will switch at some point soon though!

test-suite/trunk/External/SPEC/Makefile.spec

 ##===- Makefile.spec ---------------------------------------*- Makefile -*-===##
 #
 # This makefile is a template for building SPEC as an external test. It is
 # included by Makefile.spec2006, Makefile.spec2000, and Makefile.spec95.
 #
 ##===----------------------------------------------------------------------===##
 
 include $(LEVEL)/MultiSource/Makefile.multisrc
 
 CPPFLAGS += -I $(SPEC_BENCH_DIR)/src/
 SPEC_SANDBOX := $(PROGDIR)/External/SPEC/Sandbox.sh
 
 # Information about testing the program...
 REF_IN_DIR  := $(SPEC_BENCH_DIR)/data/$(RUN_TYPE)/input/
 REF_OUT_DIR := $(SPEC_BENCH_DIR)/data/$(RUN_TYPE)/output/
-LOCAL_OUTPUTS = $(notdir $(wildcard $(REF_OUT_DIR)/*))
+LOCAL_OUTPUTS = $(sort $(notdir $(wildcard $(REF_OUT_DIR)/*)))
 
 # Sometimes a test will output its program name or some other gunk that
 # we don't want to diff against. Tests can override this definition to
 # provide a filter for the output files.
 SPEC_OUTPUT_FILE_FILTER = cat
 
 
 # Specify how to generate output from the SPEC programs.  Basically we just run