[zorg] Build and test LLVM with GN

[vitalybuka]

remove unused depot_tools

[thakis]

Cool!

Having a bot for this is great, under these conditions:

• It doesn't send email to regular contributors when it goes red (it can email you and me and others who'd like to opt in)
• It doesn't show up on http://lab.llvm.org:8011/console

Basically, it needs to be consistent with the 1st paragraph at https://github.com/llvm/llvm-project/blob/master/llvm/utils/gn/README.rst

From what I understand, this doesn't add a bot yet, so lgtm.

zorg/buildbot/builders/sanitizers/buildbot_gn.sh

30

What's the motivation for doing a stage 1 build with cmake first? (Not saying it's wrong and you can keep it if you want, just trying to understand the motivation.)

If you just want some clang binary as host compiler, you could alternatively git clone https://chromium.googlesource.com/chromium/src/tools/clang and then run update.py -- this will download a prebuilt clang and lld, and will be much faster than building it from scratch.

If you want to test bootstrap builds, I think there's a way to hold the GN build so that it does a bootstrap build in a single build dir.

39

Does the binary downloaded by llvm/utils/gn/get.py not work on this bot?

56

You don't need to pass use_lld=true is_optimized=true is_debug=false -- use_lld defaults to true if clang_base_path is set, and optimized release builds (with asserts) is the default .

[vitalybuka]

In D62020#1505233, @thakis wrote:

Cool!

Having a bot for this is great, under these conditions:

• It doesn't send email to regular contributors when it goes red (it can email you and me and others who'd like to opt in)

already done in D61965, but you need to opt-in. I can do it for you if you don'e have zorg checkout.

• It doesn't show up on http://lab.llvm.org:8011/console

not done yet, as-is it will probably will show in sanitizers section. I'll check how to avoid that.

Basically, it needs to be consistent with the 1st paragraph at https://github.com/llvm/llvm-project/blob/master/llvm/utils/gn/README.rst

From what I understand, this doesn't add a bot yet, so lgtm.

Empty bot is added in D61965 but it will show up only after master restart. My plan is to land this patch only after empty one is alive, as .sh scripts do not need master restart and will be used by bots immediately.

[vitalybuka]

Don't hardcode default GN options

zorg/buildbot/builders/sanitizers/buildbot_gn.sh

30

That's for historical reasons. We need working clang on the host which is newer than provided by distro. Also current setup let us quickly switch to the just commited into upstream fix or feature if needed. But I don't remember we used that.

So we have other bots which use the same build_stage1_clang_at_revison functions, which is on cmake and we even can switch to GN, but it does not realy matter. We just need some fresh clang. build_stage1_clang_at_revison use fixed revision which we update manually couple times a year.

We can switch to prebuilt binaries e.g. from chrome, but this build happens only once per bot restart which is usually just a couple times a day. So at average it does not bother us yet.

But we open to switching to prebuild clang.

39

It does not. depo_tools version does not work as well.
I have not figured out how to get previous build, so decided to compile it.
Also probably we don't want "latest" GN to avoid regressions like this. On buildbot will will be hard to distinguish from LLVM issues.

zorg/buildbot/builders/sanitizers/buildbot_functions.sh

function buildbot_update {
     if [ -d svn_checkout ]; then
       for D in svn_checkout/*; do
         cp -rflP $D/* .
       done
       rm -rf svn_checkout
     fi
 }
 
+function buildbot_update_git {
+  if [[ -d "$BUILDBOT_MONO_REPO_PATH" ]]; then
+    LLVM=$BUILDBOT_MONO_REPO_PATH/llvm
+  else
+    (
+      [[ -d llvm-project ]] || git clone https://github.com/llvm/llvm-project.git
+      cd llvm-project
+      git fetch
+      git clean -fd
+      if [[ "$BUILDBOT_REVISION" == "" ]] ; then
+        REV=origin/master
+      else
+        REV=$(git log --format="%H" -n1 --grep "^llvm-svn: ${BUILDBOT_REVISION}$" origin/master)
+        [[ "$REV" != "" ]] || exit 1
+      fi
+      git checkout $REV
+      git status
+      git log -n1 --oneline
+    ) || { echo @@@STEP_EXCEPTION@@@ ; exit 1 ; }
+    LLVM=$ROOT/llvm-project/llvm
+  fi
+}
+
 function set_chrome_suid_sandbox {
   export CHROME_DEVEL_SANDBOX=/usr/local/sbin/chrome-devel-sandbox
 }
 
 function fetch_depot_tools {
   ROOT=$1
   (
     cd $ROOT

zorg/buildbot/builders/sanitizers/buildbot_gn.sh

 #!/usr/bin/env bash
 
 set -exu
 
 HERE="$(cd $(dirname $0) && pwd)"
 . ${HERE}/buildbot_functions.sh
 
 ROOT=`pwd`
 PLATFORM=`uname`
-export PATH="/usr/local/bin:$PATH"
 
 CHECK_LIBCXX=${CHECK_LIBCXX:-0}
-CHECK_LLD=${CHECK_LLD:-0}
-LLVM=$ROOT/llvm
+CHECK_LLD=${CHECK_LLD:-1}
+STAGE1_DIR=llvm_build0
+STAGE1_CLOBBER=
+STAGE2_DIR=llvm_build
 
 if [ "$BUILDBOT_CLOBBER" != "" ]; then
   echo @@@BUILD_STEP clobber@@@
   rm -rf llvm
+  rm -rf llvm-project
+  rm -rf gn
+  rm -rf ${STAGE1_DIR}
+  rm -rf ${STAGE2_DIR}
 fi
 
+(
+  LLVM=$ROOT/llvm
+  CMAKE_COMMON_OPTIONS="-GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_PARALLEL_LINK_JOBS=20"
+  build_stage1_clang_at_revison
+)
+
+echo @@@BUILD_STEP get depot_tools@@@
+[[ -d depot_tools ]] || git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
+ls $ROOT/depot_tools/
+
+
+echo @@@BUILD_STEP build GN@@@
+[[ -d gn ]] || git clone https://gn.googlesource.com/gn
+(
+  export PATH="$PATH:$(readlink -f $STAGE1_DIR)/bin"
+  cd gn
+  # Does not work with 0d038c2e0a32a528713d3dfaf1f1e0cdfe87fd46
+  git checkout 0d038c2e0a32a528713d3dfaf1f1e0cdfe87fd46^
+  python build/gen.py
+  ninja -C out
+) || { echo @@@STEP_EXCEPTION@@@ ; exit 1 ; }
+
 echo @@@BUILD_STEP update@@@
-buildbot_update
+buildbot_update_git
 
+(
+  echo @@@BUILD_STEP configure@@@
+  export PATH=$(readlink -f gn/out/):$PATH
+  $LLVM/utils/gn/gn.py gen ${STAGE2_DIR} \
+    --list --short --overrides-only \
+    --args="clang_base_path=\"$(readlink -f $STAGE1_DIR)\" use_lld=true is_optimized=true is_debug=false"
+) || echo @@@STEP_FAILURE@@@
+
+(
+  cd $STAGE2_DIR
+  echo @@@BUILD_STEP build all@@@
+
+  for TARGET in "" $(ninja -t targets | grep -o "^check-[^:]*") ; do
+    echo @@@BUILD_STEP $TARGET@@@
+    ninja $TARGET || echo @@@STEP_FAILURE@@@
+  done
+)