[sanitizers] Controlling the build width and targets for PowerPC sanitizers

[kamaub]

Summary

The cmake sanitizer buildbots use the nproc value for the build
width by default, this means the PowerPC sanitizer buildbots were
building and testing at the max parallelism their host machines could
handle.
This patch adds a build and test width specification for the PPC
sanitizers and sets the build targets to only build for PowerPC
with the intension of reducing the load the sanitizers have on their
host while improving build times. It fixes up cmake argument passing
to facilitate the change.

[vitalybuka]

buildbot/osuosl/master/config/builders.py

1753

-DLLVM_CCACHE_BUILD=ON is not needed as it's enabled by buildbot_cmake.sh:31

also it should not go into CMAKE_COMMON_OPTIONS as only the first stage can benefit of that

1754–1755

why the first one 80 and this one 256?

zorg/buildbot/builders/sanitizers/buildbot_cmake.sh

23

maybe better to detect arch(or even bot name) and make adjustments in buildbot_cmake.sh

for UnifiedTreeBuilder it make sense to keep them in builders.py, but SanitizerBuilder most configuration happening sh files already, and it easier to see when I edit the file.

also sh applies immediately and does not need to wait staging->buildbot release/reboot

[kamaub]

Thank you for the review, I am testing these changes at the moment, will post the update afterwards.

buildbot/osuosl/master/config/builders.py

1753

Could point, I was originally just adapting the code since passing the argument the previous way just get's it ignored, but I will remove this in the next update.

1753

Good point*

1754–1755

These values are based on the host machine and takes into consideration the load of the other buildbots running on them; our Big Endian host has a smaller capacity than our Little Endian host. We're okay with make and lit starting these many jobs but not with using the full nproc.

zorg/buildbot/builders/sanitizers/buildbot_cmake.sh

23

No worries, thank you for maintaining the sanitizers, I will move the cmake arguments to the target specific section of buildbot_cmake.sh and control for endianness.

[kamaub]

Addressing review comments

[kamaub]

ping

[amyk]

zorg/buildbot/builders/sanitizers/buildbot_cmake.sh

103

It looks like there is an extra ' at the end of the -v value. Is this intended?

106

It looks like there is an extra ' at the end of the -v value. Is this intended?

[lei]

zorg/buildbot/builders/sanitizers/buildbot_cmake.sh

100

I don't think we need ' around PowerPC

[kamaub]

Addressing review comments

[kamaub]

Thank you for the review, I have removed the necessary quotations.

[amyk]

Thanks for addressing my comments. I think unless if anyone else has any concerns, this LGTM.

buildbot/osuosl/master/config/builders.py

# with large memory usage, so assign them to different workers.
     ],
     'builddir': "sanitizer-aarch64-linux-fuzzer",
     'factory' : SanitizerBuilder.getSanitizerBuildFactory()},
 
     {'name' : "sanitizer-ppc64be-linux",
     'tags'  : ["sanitizer", "ppc"],
     'workernames' : ["ppc64be-sanitizer"],
     'builddir': "sanitizer-ppc64be",
-    'factory' : SanitizerBuilder.getSanitizerBuildFactory(
-                    timeout=1800,
-                    extra_configure_args=["-DLLVM_CCACHE_BUILD=ON"])},
+    'factory' : SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)},
 
     {'name' : "sanitizer-ppc64le-linux",
     'tags'  : ["sanitizer", "ppc", "ppc64le"],
     'workernames' : ["ppc64le-sanitizer"],
     'builddir': "sanitizer-ppc64le",
-    'factory' : SanitizerBuilder.getSanitizerBuildFactory(
-                    timeout=1800,
-                    extra_configure_args=[
-                       "-CMAKE_ARGS='-DLLVM_LIT_ARGS=-v -j256'",
-                       "-DLLVM_CCACHE_BUILD=ON"])},
+    'factory' : SanitizerBuilder.getSanitizerBuildFactory(timeout=1800)},
 
     {'name' : "sanitizer-windows",
     'tags'  : ["sanitizer"],
     'workernames' : ["sanitizer-windows"],
     'builddir': "sanitizer-windows",
     'factory' : AnnotatedBuilder.getAnnotatedBuildFactory(
                     script="sanitizer-windows.py",
                     depends_on_projects=["llvm", "clang", "lld", "compiler-rt"])},

zorg/buildbot/builders/sanitizers/buildbot_cmake.sh

 
 CMAKE_ARGS=""
 for arg in "$@"
 do
     case $arg in
         --CMAKE_ARGS=*)
         CMAKE_ARGS="${arg#*=}"
     esac
 done
 
 USE_CCACHE=
 if ccache -s ; then
   USE_CCACHE="-DLLVM_CCACHE_BUILD=ON"
   if ccache -p | grep "default.*max_size" ; then
     ccache -M 20G
     ccache -o compression=true
   fi
mips64*)
     CHECK_LSAN=1
     CHECK_DFSAN=1
   ;;
   ppc64*)
     CHECK_ASAN=1
     CHECK_UBSAN=1
     CHECK_MSAN=1
     CHECK_TSAN=1
+    CMAKE_COMMON_OPTIONS+=" -DLLVM_TARGETS_TO_BUILD=PowerPC"
+    if [[ "$ARCH" == "ppc64le" ]]; then
+      MAKE_JOBS=256
+      CMAKE_COMMON_OPTIONS+=" -DLLVM_LIT_ARGS=-vj256"
+    else
+      MAKE_JOBS=80
+      CMAKE_COMMON_OPTIONS+=" -DLLVM_LIT_ARGS=-vj80"
+    fi
   ;;
   i*86)
     CHECK_UBSAN=1
     CHECK_ASAN=1
   ;;
   mips*)
     CHECK_UBSAN=1
     CHECK_ASAN=1