Add Clang dependency for new Flang driver in out-of-tree builder.

[stevanradakovic]

Summary

When building out-of-tree we need to use CLANG_DIR to find
installed Clang components.

[rovka]

This seems fine to me, but maybe @awarzynski can also have a look, since he knows the details of the new driver build better :)

[awarzynski]

@stevanradakovic , are LLVM, MLIR and Clang _installed_ or just _built_? Basically, CLANG_DIRneeds to point to a directory that contains AddClang.cmake. AFAIK, that script is not copied into the build directory (but it will be installed in CMAKE_INSTALL_PREFIX). Could you double check?

[stevanradakovic]

@awarzynski you're right it's not there. I have a local builder and it failed for this exact reason: http://ex40-01.tcwglab.linaro.org:8019/#/builders/16/builds/112
so what is the default value for CMAKE_INSTALL_PREFIX? is that where clang_dir should point to?

[awarzynski]

In D98787#2634457, @stevanradakovic wrote:

@awarzynski you're right it's not there. I have a local builder and it failed for this exact reason: http://ex40-01.tcwglab.linaro.org:8019/#/builders/16/builds/112
so what is the default value for CMAKE_INSTALL_PREFIX? is that where clang_dir should point to?

Yes, but CMAKE_INSTALL_PREFIX will be empty unless the install target is run, e.g. ninja install (or make install). Alternatively, AddClang.cmake could be copied into the build dir (similarly to AddLLVM.cmake and AddClang.cmake). Either should work. Previous effort in this direction: https://reviews.llvm.org/D94533.

[stevanradakovic]

@awarzynski Hm even after running ninja install it's still not found here. Even though in step 7 I can clearly see AddClang.cmake (line 2629). I thought the CLANG_DIR path was ambiguous but it's not.

[rovka]

I think the issue might be here: https://github.com/llvm/llvm-project/blob/7515e81e8c58ca07ac5fede7149634c0dfacae8a/flang/CMakeLists.txt#L52
We seem to be using CLANG_DIR directly, but for LLVM_DIR and MLIR_DIR we build absolute paths if needed.
@awarzynski, does this sound correct to you?

[awarzynski]

In D98787#2641033, @rovka wrote:

I think the issue might be here: https://github.com/llvm/llvm-project/blob/7515e81e8c58ca07ac5fede7149634c0dfacae8a/flang/CMakeLists.txt#L52
We seem to be using CLANG_DIR directly, but for LLVM_DIR and MLIR_DIR we build absolute paths if needed.
@awarzynski, does this sound correct to you?

Thanks for looking into this! It sounds very plausible. Have you been able to test it?

I've also noticed that you're installing into the build dir (as opposed to some other, dedicated directory). I think that that should work fine (though I haven't checked), but it's possible that LLVM_DIR, MLIR_DIR and CLANG_DIR refer to build locations (which will work for LLVM and MLIR, but not for Clang). It's something that's worth double-checking.

[stevanradakovic]

@awarzynski no dice. The result is the same if I put the arbitrary install path http://ex40-01.tcwglab.linaro.org:8019/#/builders/16/builds/224

[awarzynski]

In D98787#2641509, @stevanradakovic wrote:

@awarzynski no dice. The result is the same if I put the arbitrary install path http://ex40-01.tcwglab.linaro.org:8019/#/builders/16/builds/224

Noted, thanks for checking! @rovka's suggestion implemented here: https://reviews.llvm.org/D99088.

[stevanradakovic]

Add install step for Clang.

[stevanradakovic]

Great work everyone. Thanks to @rovka's changes, the build is now properly running

[awarzynski]

In D98787#2644525, @stevanradakovic wrote:

Great work everyone. Thanks to @rovka's changes, the build is now properly running

Great to see this progressing!

So is the CMake change not required then? Since https://reviews.llvm.org/D99088 hasn't been merged yet, and this already works.

[stevanradakovic]

No no, I hacked it manually to include that change. It is certainly still necessary.

[awarzynski]

In D98787#2644681, @stevanradakovic wrote:

No no, I hacked it manually to include that change. It is certainly still necessary.

Ah, makes sense :) Thanks for confirming. Great effort, really pleased to see this one working!

LGTM

PS Please could you double-check the latest version of https://reviews.llvm.org/D99088, it should be OK!

zorg/buildbot/builders/FlangBuilder.py

 from zorg.buildbot.commands.CmakeCommand import CmakeCommand
 from zorg.buildbot.process.factory import LLVMBuildFactory
 from zorg.buildbot.builders.UnifiedTreeBuilder import addNinjaSteps, getCmakeWithNinjaBuildFactory
 
 def getFlangOutOfTreeBuildFactory(
            checks = None,
            clean = False,
            llvm_extra_configure_args = None,
            flang_extra_configure_args = None,
            env = None,
            **kwargs):
 
     if env is None:
         env = dict()
 
     f = getCmakeWithNinjaBuildFactory(
-            depends_on_projects=['llvm','mlir'],
+            depends_on_projects=['llvm','clang','mlir'],
             obj_dir="build_llvm",
             checks=[],
+            install_dir="install",
             clean=clean,
             extra_configure_args=llvm_extra_configure_args,
             env=env,
             **kwargs)
 
     if checks is None:
         checks = ['check-all']
 
CmakeCommand.applyRequiredOptions(flang_cmake_args, [
         ])
 
     flang_obj_dir = "build_flang"
     flang_src_dir = "{}/flang".format(f.monorepo_dir)
 
     # Add LLVM_DIR and MLIR_DIR to the CMake invocation.
     llvm_dir = "{}/lib/cmake/llvm".format(f.obj_dir)
     mlir_dir = "{}/lib/cmake/mlir".format(f.obj_dir)
+    clang_dir = "{}/lib/cmake/clang".format(f.install_dir)
     CmakeCommand.applyRequiredOptions(flang_cmake_args, [
         # We actually need the paths to be relative to the source directory,
         # otherwise find_package can't locate the config files.
         ('-DLLVM_DIR:PATH=',
             LLVMBuildFactory.pathRelativeTo(llvm_dir, flang_src_dir)),
         ('-DMLIR_DIR:PATH=',
             LLVMBuildFactory.pathRelativeTo(mlir_dir, flang_src_dir)),
+        ('-DCLANG_DIR:PATH=',
+            LLVMBuildFactory.pathRelativeTo(clang_dir, flang_src_dir)),
         ])
 
     # We can't use addCmakeSteps as that would use the path in f.llvm_srcdir.
     f.addStep(CmakeCommand(name="cmake-configure-flang",
                            haltOnFailure=True,
                            description=["CMake", "configure", "flang"],
                            options=flang_cmake_args,
                            path=LLVMBuildFactory.pathRelativeTo(flang_src_dir,