[flang] Add builder/worker for debug build with reverse iteration

[DavidSpickett]

Summary

For https://discourse.llvm.org/t/reverse-iteration-bots/72224.

This does not warrant a whole new builder, so I'm adding reverse
iteration to the debug configuration.

I thought of a few other places but:

• The release builder tests what we will later ship, so I don't want to change the configuration.
• Release with asserts is a very common developer configuration so it helps to know if problems occur with that only.
• The rest are build types, out of tree, shared libs, etc. which are niche enough I don't want reversed iterator problems polluting them.

Initially I'm not removing the existing worker, so that the two
can run in parallel with the reverse iterator bot on silent.

Once it's been setup and is stable, it'll move to normal and
I'll remove the old worker and builder.

[omjavaid]

Thanks David for adding this builder. Re debug builder do you plan to remove the flang-aarch64-debug after the new builder is up and running. I guess if we add a new builder with DLLVM_REVERSE_ITERATION=ON we ll loose the ccache advantage given many files will get built twice for the same set of changes.

buildbot/osuosl/master/config/builders.py

2072

Debug builds sometime face bottleneck on memory. Should we consider adding -DLLVM_PARALLEL_LINK_JOBS to a reasonable number?

[DavidSpickett]

Add the "rev_it" tag used by the existing reverse iteration bot.

[omjavaid]

Never mind got the answer from the summary.

[DavidSpickett]

Re debug builder do you plan to remove the flang-aarch64-debug after the new builder is up and running.

• Initially existing builder runs on normal, new builder on silent. (this needs to be done anyway to follow https://llvm.org/docs/HowToAddABuilder.html#steps-to-add-builder-to-llvm-buildbot)
• When that's stable, I'll remove the normal builder.
• Then the new builder moves to normal to replace the old one.

I guess if we add a new builder with DLLVM_REVERSE_ITERATION=ON we ll loose the ccache advantage given many files will get built twice for the same set of changes.

Yes, but that wasn't my motivation in (eventually) replacing the debug builder. The motivation is to keep the number of bots down and not add too many specialty configurations that report mostly the same results.
(and I'm renaming just so people don't have to look into the cmake)

In this case, I think the problems caused by debug mode and those with reverse iteration will likely be easy to tell apart.

Although, @MaskRay can you confirm that there's no conflict between reverse iteration and debug builds? It's still useful coverage as I understand it.

buildbot/osuosl/master/config/builders.py

2072

They do but initially I'll run the worker on a different machine where memory isn't a concern. In the end it will replace the existing worker, which has not had any memory problems recently on its current host.

In general only in favour of adding these sort of machine specific details if there's no other way to handle it.

[DavidSpickett]

In case it's useful, this is what Linaro runs currently for flang:

clang-aarch64-full-2stage       
clang-arm64-windows-msvc        
clang-arm64-windows-msvc-2stage 
clang-aarch64-sve-vla           
clang-aarch64-sve-vla-2stage    
clang-aarch64-sve-vls           
clang-aarch64-sve-vls-2stage    
flang-aarch64-dylib             
flang-aarch64-sharedlibs        
flang-aarch64-out-of-tree       
flang-aarch64-release           
flang-aarch64-debug             
flang-aarch64-latest-clang      
flang-aarch64-rel-assert        
flang-aarch64-latest-gcc

To give you an idea why I'd like to replace an existing configuration rather than adding a new one on top of all that. Feel free to disagree of course, or suggest another home for reverse iteration.

[omjavaid]

In D157274#4565313, @DavidSpickett wrote:

In case it's useful, this is what Linaro runs currently for flang:

clang-aarch64-full-2stage       
clang-arm64-windows-msvc        
clang-arm64-windows-msvc-2stage 
clang-aarch64-sve-vla           
clang-aarch64-sve-vla-2stage    
clang-aarch64-sve-vls           
clang-aarch64-sve-vls-2stage    
flang-aarch64-dylib             
flang-aarch64-sharedlibs        
flang-aarch64-out-of-tree       
flang-aarch64-release           
flang-aarch64-debug             
flang-aarch64-latest-clang      
flang-aarch64-rel-assert        
flang-aarch64-latest-gcc

To give you an idea why I'd like to replace an existing configuration rather than adding a new one on top of all that. Feel free to disagree of course, or suggest another home for reverse iteration.

I believe this list should slowly shrink over time but given flang is still not mature enough we might not see that happening anytime soon.

[DavidSpickett]

Very true.

And if having debug and reverse iteration on the same worker did prove to be confusing in practice, I'd of course split them up.

[MaskRay]

Thanks for adding a builder!

In D157274#4565303, @DavidSpickett wrote:

Re debug builder do you plan to remove the flang-aarch64-debug after the new builder is up and running.

• Initially existing builder runs on normal, new builder on silent. (this needs to be done anyway to follow https://llvm.org/docs/HowToAddABuilder.html#steps-to-add-builder-to-llvm-buildbot)
• When that's stable, I'll remove the normal builder.
• Then the new builder moves to normal to replace the old one.

I guess if we add a new builder with DLLVM_REVERSE_ITERATION=ON we ll loose the ccache advantage given many files will get built twice for the same set of changes.

Yes, but that wasn't my motivation in (eventually) replacing the debug builder. The motivation is to keep the number of bots down and not add too many specialty configurations that report mostly the same results.
(and I'm renaming just so people don't have to look into the cmake)

In this case, I think the problems caused by debug mode and those with reverse iteration will likely be easy to tell apart.

Although, @MaskRay can you confirm that there's no conflict between reverse iteration and debug builds? It's still useful coverage as I understand it.

LLVM_ENABLE_REVERSE_ITERATION=on (currently, LLVM_REVERSE_ITERATION=on achieves the same effect) is orthogonal to CMAKE_BUILD_TYPE=Debug, LLVM_ENABLE_ASSERTIONS=on.

To give you an idea why I'd like to replace an existing configuration rather than adding a new one on top of all that. Feel free to disagree of course, or suggest another home for reverse iteration.

And if having debug and reverse iteration on the same worker did prove to be confusing in practice, I'd of course split them up.

My gut feeling is that combining some configurations will make more effective uses of the resources, but you maintainers make the call:)

[gkistanova]

LGTM.

[DavidSpickett]

@omjavaid / @luporl / @sscalpone If you have any further comments please make them before Tuesday next week. If not, I'll go ahead with this as is.

[DavidSpickett]

https://github.com/llvm/llvm-zorg/commit/809115cf668cb1dde6e12aa00e2569b775629ebe

buildbot/osuosl/master/config/builders.py

'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     depends_on_projects=['llvm','mlir','clang','flang'],
                     extra_configure_args=[
                         "-DLLVM_TARGETS_TO_BUILD=AArch64",
                         "-DCMAKE_BUILD_TYPE=Debug",
                         "-DCMAKE_CXX_STANDARD=17",
                         "-DLLVM_USE_LINKER=lld",
                     ])},
 
+    {'name' : "flang-aarch64-debug-reverse-iteration",
+    'tags'  : ["flang", "rev_iter"],
+    'workernames' : ["linaro-flang-aarch64-debug-reverse-iteration"],
+    'builddir': "flang-aarch64-debug-reverse-iteration",
+    'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
+                    clean=True,
+                    checks=['check-flang'],
+                    depends_on_projects=['llvm','mlir','clang','flang'],
+                    extra_configure_args=[
+                        "-DLLVM_TARGETS_TO_BUILD=AArch64",
+                        "-DCMAKE_BUILD_TYPE=Debug",
+                        "-DCMAKE_CXX_STANDARD=17",
+                        "-DLLVM_USE_LINKER=lld",
+                        "-DLLVM_REVERSE_ITERATION:BOOL=ON",
+                    ])},
+
     {'name' : "flang-aarch64-latest-clang",
     'tags'  : ['flang'],
     'workernames' : ["linaro-flang-aarch64-latest-clang"],
     'builddir': "flang-aarch64-latest-clang",
     'factory' : UnifiedTreeBuilder.getCmakeWithNinjaBuildFactory(
                     clean=True,
                     checks=['check-flang'],
                     depends_on_projects=['llvm','mlir','clang','flang'],

buildbot/osuosl/master/config/workers.py

return [
         create_worker("linaro-clang-aarch64-lld-2stage", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-clang-aarch64-global-isel", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-clang-aarch64-full-2stage", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-lldb-aarch64-ubuntu", properties={'jobs': 16}, max_builds=1),
         create_worker("linaro-flang-aarch64-dylib", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-sharedlibs", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-out-of-tree", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-debug", properties={'jobs' : 32}, max_builds=1),
+        create_worker("linaro-flang-aarch64-debug-reverse-iteration", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-latest-clang", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-release", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-rel-assert", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-flang-aarch64-latest-gcc", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-g3-01", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-g3-02", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-g3-03", properties={'jobs' : 32}, max_builds=1),
         create_worker("linaro-g3-04", properties={'jobs' : 32}, max_builds=1),