Install MLIR Python requirements on buildbot workers

[ftynse]

Summary

MLIR Python interface has its requirements listed in the repository, but
they are not necessarily available on the buildbot workers. Add an extra
step of installing these requirements using pip. Wrap the entire process
into a new MLIR build factory that adds the step, if requested, after
checking out the source and before running cmake.

[mehdi_amini]

Nice!

I'll let Galina do the review, I'm never totally sure about the buildbot code.

[mehdi_amini]

Galina: does it look good to you?

[gkistanova]

Thanks for the patch, Alex!

zorg/buildbot/builders/MLIRBuilder.py

11

This would build you a path according to the rules of the OS buildbot master is running on. But you need the path for the worker, which could run a different OS. It is unlikely that we would run the main buildbot on Windows, but still.

Having it as mlir/python/requirements.txt is fine here.

18

Did you forget to import getCmakeWithNinjaBuildFactory? Or, if I guessed right, you wanted to use getLLVMBuildFactoryAndSourcecodeSteps here instead?

[makslevental]

I'm happy to get this across the finish line in order to get mlir-python builders back. @gkistanova Is there a brief description somewhere for how to test this code locally?

[gkistanova]

Hi Maksim,

We can stage this patch after you will address my comments. That would be the easiest way to the result.

buildbot/osuosl/master/config/builders.py

 from zorg.buildbot.builders import UnifiedTreeBuilder
 from zorg.buildbot.builders import AOSPBuilder
 from zorg.buildbot.builders import AnnotatedBuilder
 from zorg.buildbot.builders import LLDPerformanceTestsuite
 from zorg.buildbot.builders import FuchsiaBuilder
 from zorg.buildbot.builders import XToolchainBuilder
 from zorg.buildbot.builders import TestSuiteBuilder
 from zorg.buildbot.builders import BOLTBuilder
+from zorg.buildbot.builders import MLIRBuilder
 
 from zorg.buildbot.builders import HtmlDocsBuilder
 from zorg.buildbot.builders import DoxygenDocsBuilder
 
 reload(HtmlDocsBuilder)
 reload(DoxygenDocsBuilder)
 
 
'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                         '-DMLIR_RUN_AMX_TESTS=ON',
                         '-DINTEL_SDE_EXECUTABLE=/intel_sde/sde64'
                     ])},
 
     {'name' : "mlir-nvidia",
     'tags'  : ["mlir"],
     'workernames' : ["mlir-nvidia"],
     'builddir': "mlir-nvidia",
-    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+    'factory' : MLIRBuilder.getMLIRCmakeBuildFactory(
                     llvm_srcdir="llvm.src",
                     obj_dir="llvm.obj",
                     clean=True,
                     targets = ['check-mlir-build-only'],
                     checks = ['check-mlir'],
                     depends_on_projects=['llvm','mlir'],
+                    install_pip_requirements=True,
                     extra_configure_args=[
                         '-DLLVM_BUILD_EXAMPLES=ON',
                         '-DLLVM_TARGETS_TO_BUILD=host;NVPTX',
                         '-DLLVM_ENABLE_PROJECTS=mlir',
                         '-DMLIR_ENABLE_CUDA_RUNNER=1',
                         '-DMLIR_INCLUDE_INTEGRATION_TESTS=ON',
                         '-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc',
                         '-DMLIR_ENABLE_VULKAN_RUNNER=1',

zorg/buildbot/builders/MLIRBuilder.py

+from buildbot.plugins import steps
+from zorg.buildbot.builders.UnifiedTreeBuilder import getLLVMBuildFactoryAndSourcecodeSteps, addCmakeSteps, addNinjaSteps
+from os import path
+
+def addInstallPipRequirementsStep(f):
+    f.addStep(
+        steps.ShellCommand(
+            name="Install Python requirements",
+            haltOnFailure=True,
+            command=["pip", "install", "-r",
+                     path.join("mlir", "python", "requirements.txt")],
+            workdir=f.llvm_srcdir))
+
+def getMLIRCmakeBuildFactory(
+        clean = False,
+        install_pip_requirements = False,
+        **kwars):
+    f = getCmakeWithNinjaBuildFactory(**kwargs)
+
+    if install_pip_requirements:
+      addInstallPipRequirementsStep(f)
+
+    cleanBuildRequested = lambda step: step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) or clean
+    addCmakeSteps(f,
+                  cleanBuildRequested=cleanBuildRequested,
+                  obj_dir=f.obj_dir,
+                  **kwargs)
+    addNinjaSteps(f, **kwargs)
+
+    return f
+
+