workaround for failing GPU

[kuhnel]

Summary

Implemented workaround when GPU is not ready when deploying the container. It will wait for 30 secs and then restart the container. This should be enough for the GPU to be initialized.

[kuhnel]

cleaned up git history

[kuhnel]

Bot has been stable over the weekend, moving this config to production

buildbot/google/docker/buildbot-mlir-nvidia/VERSION

-3
+6

buildbot/google/docker/buildbot-mlir-nvidia/run.sh

(
   echo "number of cores: $(nproc)" ;\
   nvidia-smi -L | cut -d "(" -f 1 ;\
   lsb_release -d | cut -f 2- ; \
   clang --version | head -n1 ; \
   ld.lld-8 --version ; \
   cmake --version | head -n1 
 ) > ${WORKER_NAME}/info/host 
 
-# FIXME(kuhnel):
 # It looks like GKE sometimes deploys the container before the NVIDIA drivers 
 # are loaded on the host. In this case the GPU is not available during the 
 # entire lifecycle of the container. Not sure how to fix this properly. 
-# Maybe the above entry is enough as it depends on a working `nvidia-smi`. 
-# If not a workaround might be to check for the graphics card in this script and 
-# exit immediately if it's not available.
+
+RETURN_CODE=$(nvidia-smi > /dev/null ; echo $?)
+if [[ "$RETURN_CODE" != "0" ]] ; then
+  echo "ERROR: Failed to access NVIDIA graphics card."
+  echo "Exiting in 30 secs..."
+  sleep 30
+  exit 1
+fi
 
 # create the folder structure
 buildslave create-slave --keepalive=200 "${WORKER_NAME}" \
   lab.llvm.org:9994 "${WORKER_NAME}" "${WORKER_PASSWORD}"
 
 # start the daemon, this command return immetiately
 buildslave start "${WORKER_NAME}"
 
 # To keep the container running and produce log outputs: dump the worker
 # log to stdout
 tail -f ${WORKER_NAME}/twistd.log