[Test-Suite] Support Cross-Compilation and Cross-execution targeting arm64-linux-android

[ziangwan]

Summary

This patch allows users to cross-compile and cross-execute LLVM test-suite on an Android device for testing and benchmark purposes.

Usage:

1. Obtain llvm-test-suite and Android NDK for your host machine. The test-suite also supports SPEC CPU2006 benchmark. (SPEC CPU2017 requires too much system memory and is not suitable for an actual android device.)

2. Get the build template inside cmake/caches/target-arm64-android-template.sh and create your own build script build.sh. Put build.sh inside the build directory you desire.

3. Run sh build.sh to setup the build. Then do ninja to build.

4. Run ninja push_android to push the tests onto an Android device, whose serial number is specified inside build.sh.

5. Run llvm-lit -j1 -o result.json ./build to cross-execute.

Notes

1. I manually exclude several tests and benchmarks for Android build by checking against TEST_SUITE_REMOTE_CLIENT STREQUAL "adb" in corresponding CMakeLists.txt files. All of them are excluded due to build failures. A list of excluded tests and benchmarks with concise build failure messages:
   • test-suite/External/SPEC/CINT2006/CMakeLists.txt:
     • Exclude cpu2006_subdir(400.perlbench)
       • incomplete definition of type 'struct __sFILE'
       • Many other errors
     • Exclude cpu2006_subdir(464.h264ref)
       • sys/timeb.h not found <- this header file is deprecated and no longer exists inside Android NDK.
     • Exclude cpu2006_subdir(483.xalancbmk)
       • undefined reference to `xercesc_2_5::XMLPlatformUtils::openFile(unsigned short const*, xercesc_2_5::MemoryManager*)'
       • Many other errors
   • test-suite/External/SPEC/CINT2006/462.libquantum/CMakeLists.txt
     • Use diff -w instead of diff --strip-trailing-cr
       • --strip-trailing-cr is not supported in Android's diff
   • test-suite/External/SPEC/CFP2006/CMakeLists.txt
     • Exclude cpu2006_subdir(446.dealII)
       • error: no template named 'pair' in namespace 'std'
   • test-suite/Multisource/Applications/CMakeLists.txt
     • Exclude add_subdirectory(JM)
     • Exclude add_subdirectory(hexxagon)
     • Exclude out add_subdirectory(SPASS)
       • sys/timeb.h not found
     • Exclude add_subdirectory(ClamAV)
       • undefined reference to `endprotoent'
     • Exclude add_subdirectory(siod)
       • undefined reference to `getpwent'
     • Exclude add_subdirectory(Burg)
       • error: conflicting types for 'atoi'
     • Exclude add_subdirectory(spiff)
       • parse.c:85: undefined reference to `index'
   • test-suite/Multisource/Benchmarks/MiBench/CMakeLists.txt
     • Exclude add_subdirectory(network-patricia)
       • undefined reference to `bcopy'
     • Exclude add_subdirectory(office-ispell)
       • fatal error: 'sgtty.h' file not found
   • test-suite/Multisource/Benchmarks/Prolangs-C/CMakeLists.txt
     • Exclude add_subdirectory(bison)
       • conflicts.c:266: undefined reference to `bcopy'
   • test-suite/Multisource/Benchmarks/DOE-ProxyApps-C++/CMakeLists.txt
     • Exclude add_subdirectory(CLAMR)
       • fatal error: 'sys/sysctl.h' file not found
   • test-suite/Multisource/Benchmarks/CMakeLists.txt
     • Exclude add_subdirectory(7zip)
       • 7zip/CPP/7zip/UI/Console/UserInputUtils.cpp:81:25: error: use of undeclared identifier 'getpass'
       • Many other errors
   • test-suite/SingleSource/UnitTests/
     • Exclude 2005-05-11-Popcount-ffs-fls.c
       • 110: undefined reference to `ffsl'
   • test-suite/Microbenchmarks
     • Exclude the whole folder because of unsupported test-format when cross-exec
       • An issue on the llvm-lit side
   • test-suite/MultiSource/Benchmarks/Prolangs-C/CMakeLists.txt
     • Exclude add_subdirectory(unix-small)
       • can't create /tmp/rmJKvjCz.
   • test-suite/MultiSource/Applications/CMakeLists.txt
     • Exclude out add_subdirectory(obsequi)
       • Unable to find library -lsupc++

2. Initially, I just want to use the test-suite to benchmark a compiler feature. Therefore, I only care about the benchmarks in the test-suite.

The patch works locally for me. If you have any suggestions / comments / encounter bugs when targeting Android, please let me know.

[kristof.beyls]

Thank you very much Ziang for sharing this work!
I'd love to more easily benchmark on Android devices - and this seems to be a good step in the right direction.

Please find some of my immediate thoughts on the patch below.

MultiSource/Benchmarks/CMakeLists.txt

32

I'm wondering if (NOT "${TARGET_OS}" STREQUAL "Android") wouldn't be a better test?
Isn't the reason many of these programs fail to build that some header file isn't provided on the Android platform?
Also, that would make the test more similar to other tests already in here that remove specific benchmarks for specific architectures or OSs.

cmake/caches/target-arm64-android-template.sh

1–47

I wonder if you're also aiming to use LNT as an extra layer on top of this test-suite?
I tend to use LNT to build/run tests/benchmarks. I wonder if support was added in LNT, would there still be a lot of value in having this custom shell script?
I would love to avoid having to go through different benchmarking flows depending on which OS I'm running benchmarks on.

31

Shouldn't the SPEC_CPU_LINUX define get added somehow as part of the External/SPEC cmake files rather than added here on the command line?

lit.site.cfg.in

10

My understanding is that adding quotes here will make the profile_generate feature break in some circumstances.

[ziangwan]

Update diff:

1. Have a workaround inside top-level CMakeLists.txt to set TARGET_OS to Android.
2. Test against TARGET_OS instead of TEST_SUITE_REMOTE_CLIENT in the build system.
3. lit.site.cfg.in: remove the double quote around @TEST_SUITE_PROFILE_GENERATE@.

[ziangwan]

MultiSource/Benchmarks/CMakeLists.txt

32

Most of the build failures are due to missing header files in the Android platform. There is an issue with the data format of Microbenchmark. I didn't attempt to fix that since I didn't care much about Microbenchmarks. I simply disabled them. There are a few tests/benchmarks I disabled because of bugs that prevent compilation. I didn't try to fix them either.

cmake/caches/target-arm64-android-template.sh

1–47

I currently don't have the plan to add Android-support into LNT. My goal is to run test-suite and SPEC 2006 on Android to obtain data.

[fhahn]

cmake/caches/target-arm64-android-template.sh

2

I think cmake/caches should only contain cmake cache files (a set of pre-configured settings), that can be used with cmake -C, not bash scripts.

IIUC the key thing to do here is to locate the android ndk and set the paths accordingly. Is there a standard way to locate the ndk, similar to how it is done for the Xcode SDKs ? (like https://github.com/llvm/llvm-test-suite/blob/master/cmake/caches/util/xcode_sdk.cmake)

7

This setting and the ones below seems very system/setup specific, and related to general test-suite setup, not android related.

43

Can this be handled by the general sync logic, that copies over the other artifacts? Also, is using a libcxx built alongside clang supported?

[ziangwan]

cmake/caches/target-arm64-android-template.sh

7

Exactly, there is a requirement on host system setup for llvm-test-suite to correctly execute:

Special Requirement for build directory: Due to the current state of the build system, the tests are generated using the absolute path on the host machine. For example, if you build the test-suite at /aa/bb/cc on the host machine, all the build files should also be stored at /aa/bb/cc on the target android device for the tests to correctly execute. Therefore, we need to create a build directory, whose absolute path is also a writable location on the target Android device, on the host machine. For example, if you want the tests to run on /data/local/tmp/build on the Android device, you should build the test-suite inside /data/local/tmp/build as well. You specifically cannot use arbitrary path on the host machine since Android's SeLinux prevents creating directories at sysroot.

[danalbert]

CMakeLists.txt

187

Got a link to a bug? CMAKE_SYSTEM_NAME should be Android for Android.

External/SPEC/CFP2006/CMakeLists.txt

10

Everything that's excluded should have a comment with a link to a bug explaining what we need to fix, and should probably also include message(WARNING "Skipping <test name> on Android: <bug number>")

External/SPEC/CINT2006/462.libquantum/CMakeLists.txt

11–24

Have you filed a bug? That sounds trivial to add to toybox.

MultiSource/Benchmarks/CMakeLists.txt

32

Missing platform stuff, or missing third-party stuff? i.e. is libc missing something, or is the problem that Android doesn't include the 7zip development package? The former needs to be fixed and/or have bugs filed.

cmake/caches/target-arm64-android-template.sh

7

Probably worth a comment referencing whatever doc you're quoting.

9

(why is this "doc" space?)

Should probably keep everything under a common subdirectory to avoid any collisions. /data/local/tmp/llvm-test-suite/...

Is it actually necessary to push the NDK to the device? None of the toolchains are built to run on Android. I can't even think of what use the sysroot would be.

11

Same. Why are we running the compiler on the device?

33

This is needed because the compiler being used isn't the NDK toolchain, right? It's just the bare compiler without the sysroot and all that? If so, you *might* be able to use -gcc-toolchain $ANDROID_NDK_LOCATION/toolchains/llvm/prebuilt/$HOST_SYSTEM instead of specifying the internal location of libgcc. (If not, fixing the driver to make that work would probably be good)

43

Also, is using a libcxx built alongside clang supported?

Not really. I haven't had the time to upstream all of our patches. It might work if this only needs to support new devices, but we have a number of things we have to do to build libc++ for Jelly Bean.

44

nit: indent continuation lines

litsupport/modules/android.py

2

Nit: docstring convention is:

"""Short summary sentence.

Additional description if needed.
"""

(also, is LLVM Python 3 yet?)

5

nit: stdlib imports come first and are separated by a blank line

import logging
import os
import subprocess

from litsupport import testplan

23

prepend

25

It's doing more than that. Maybe make_adb_command?

utils/rsync_android.sh

2

This isn't running rsync, so don't call it that. Just sync_android.sh or push_android.sh.

12

+jmgao: should adb push have an --unsymlink option?

32

I'm guessing it's slower to run a lot of pushes instead of a single push command... but idk if jmgao likes the idea of adb push --filter $PATTERN

[pcc]

FWIW, I did something similar: https://github.com/pcc/llvm-test-suite/tree/android
but never got time to clean it up and send it for review.

Here are a few differences between your changes and mine:

• I statically linked libc++ (avoids needing to mess around with LD_LIBRARY_PATH and pushing libc++_shared.so).
• I fixed all of the tests that fail to compile/link on Android instead of disabling them. That should probably be easy to send out as a separate review, so I'll try to do that soon.
• I wrote cmake cache files for Android instead of using a shell script.

cmake/caches/target-arm64-android-template.sh

7

I addressed this in my implementation by basically search-and-replacing any commands sent to the device to change host-side paths into device-side paths (which is basically /data/local/tmp/llvm-test-suite/staging plus the source-relative path).

33

What I usually do is: -B${ANDROID_NDK_PATH}/toolchains/llvm/prebuilt/linux-x86_64.

Would it be worth considering moving libgcc.a into the sysroot in a future release of the NDK? Then I believe only --sysroot should be required.

[pcc]

utils/rsync_android.sh

32

In my implementation I used the regular rsync command to copy the files to a staging directory and then adb pushed the staging directory.

[danalbert]

cmake/caches/target-arm64-android-template.sh

33

Maybe. I was somewhat trying to keep toolchain components separate from the platform APIs, but that's not really that important in this case since that's a fairly blurry line anyway. On the other hand, we're hopefully switching to libclangrt_builtins soon anyway, so maybe not worth the effort for either.

[ziangwan]

External/SPEC/CINT2006/462.libquantum/CMakeLists.txt

11–24

I've filed a bug.

cmake/caches/target-arm64-android-template.sh

9

"docspace" is an arbitrary name I use for myself. You don't have to use "docspace".

Yes, you can use /data/local/tmp/llvm-test-suite/.. as your path. This shell script is just a template.

I am not pushing the whole NDK to the device. It is only used during compilation.

11

We are not running compilers on the device. This path is specified if you have a custom build of clang. In my case, I do.

litsupport/modules/android.py

2

This file can be executed using python2.

[danalbert]

cmake/caches/target-arm64-android-template.sh

9

Then why is it a /data path? That's part of the Android file system. Mirroring that on the host requires root.

litsupport/modules/android.py

2

I mean can it be *not* Python 2. Python 2 reaches EOL at the end of the year, so if using python 3 is an option in LLVM we should be using Python 3. Specifically, if we can assume Python 3.6+, there are some cleanups that can be made here.

[ziangwan]

litsupport/modules/android.py

2

Yeah, we can use f-string instead of .format. However, the majority of the time spent is execution time. Replacing .format with f-strings is not a big issue.

utils/rsync_android.sh

32

It is not that slow to use many adb push command. I takes around 3 min to do the whole transfer.

[ziangwan]

cmake/caches/target-arm64-android-template.sh

9

I will follow Peter's advice. I will do a string replace of host-side path with device-side path.

[srhines]

cmake/caches/target-arm64-android-template.sh

33

Unfortunately, libgcc.a is here to stay until the Android unwinder can be used as part of the builtins. I don't think that is considered a high priority though. I would prefer that libgcc.a exist somewhere better in the NFL, considering we could drop the rest of the GCC dirs if we do that.

33

Lol autocorrect. NDK, not NFL.

[ziangwan]

Update diff.

1. Create a new templates folder inside cmake to store the targeting android template shell script (subject to change)
2. Rename and improve the file transfer script push_android.py.
3. Use strreplace in the android module to support arbitrary build directory on the host system. In that way, we don't need root privilege to build the test cases anymore.

[ziangwan]

CMakeLists.txt

187

Resolved by using NDK's android.toolchain.cmake file.

CMakeLists.txt

 if(TEST_SUITE_REMOTE_HOST)
   add_custom_target(rsync
     COMMAND ${PROJECT_SOURCE_DIR}/utils/rsync.sh
                 ${TEST_SUITE_REMOTE_HOST} ${PROJECT_BINARY_DIR}
     USES_TERMINAL
   )
 endif()
 
+if(TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
+  add_custom_target(rsync_android
+    COMMAND ${PROJECT_SOURCE_DIR}/utils/rsync_android.sh
+                ${TEST_SUITE_REMOTE_HOST} ${PROJECT_BINARY_DIR}
+    USES_TERMINAL
+  )
+endif()
+
 # Run Under configuration for RunSafely.sh (will be set in lit.site.cfg)
 set(TEST_SUITE_RUN_UNDER "" CACHE STRING "RunSafely.sh run-under (-u) parameter")
 
 # run type/benchmark size configuration (mostly for SPEC at the moment)
 set(TEST_SUITE_RUN_TYPE "train" CACHE STRING
     "Type of benchmark inputs (may be test,train or ref)")
 
 get_filename_component(CMAKE_C_COMPILER_DIRECTORY ${CMAKE_C_COMPILER} DIRECTORY)

 endif()
 
 include(TestSuite)
 include(SingleMultiSource)
 find_package(TCL)
 
 if(NOT DEFINED TARGET_OS)
   message(STATUS "Check target operating system - ${CMAKE_SYSTEM_NAME}")
   set(TARGET_OS ${CMAKE_SYSTEM_NAME})
 endif()
 if(NOT DEFINED ARCH)
   include(DetectArchitecture)
   detect_architecture(ARCH)
 endif()
 if(NOT DEFINED X86CPU_ARCH AND ARCH STREQUAL "x86")
   include(DetectArchitecture)
   detect_x86_cpu_architecture(X86CPU_ARCH)

External/SPEC/CFP2006/CMakeLists.txt

 include(${CMAKE_SOURCE_DIR}/External/SPEC/SpecCPU2006.cmake)
 if(TEST_SUITE_SPEC2006_ROOT)
   cpu2006_subdir(433.milc)
   cpu2006_subdir(444.namd)
-  cpu2006_subdir(447.dealII)
   cpu2006_subdir(450.soplex)
   cpu2006_subdir(453.povray)
   cpu2006_subdir(470.lbm)
   cpu2006_subdir(482.sphinx3)
+
+  if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
+    cpu2006_subdir(447.dealII)
+  endif()
 endif()

External/SPEC/CINT2006/462.libquantum/CMakeLists.txt

 list(APPEND LDFLAGS -lm)
 
 macro(test_input run_type)
   llvm_test_run(RUN_TYPE ${run_type} WORKDIR ${CMAKE_CURRENT_BINARY_DIR}
     ${ARGN} > ${run_type}.out
   )
   # Reference outputs for test+train have CRLF line endings so we have to use
   # diff --strip-trailing-cr
   # (This flag is a gnu extension, maybe we should rather extend fpcmp or
   #  somehow copy+fix the reference outputs?)
+  # diff in Android doesn't have --strip-trailing-cr, use -w instead
+  if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
     llvm_test_verify(RUN_TYPE ${run_type} WORKDIR ${CMAKE_CURRENT_BINARY_DIR}
       diff --strip-trailing-cr
       data/${run_type}/output/${run_type}.out
       ${run_type}.out
     )
+  else()
+    llvm_test_verify(RUN_TYPE ${run_type} WORKDIR ${CMAKE_CURRENT_BINARY_DIR}
+      diff -w
+      data/${run_type}/output/${run_type}.out
+      ${run_type}.out
+    )
+  endif()
 endmacro()
 
 test_input(test 33 5)
 test_input(train 143 25)
 test_input(ref 1397 8)
 
 llvm_test_executable(462.libquantum ${Source})
 llvm_test_data_spec_default(462.libquantum)

External/SPEC/CINT2006/CMakeLists.txt

 include(${CMAKE_SOURCE_DIR}/External/SPEC/SpecCPU2006.cmake)
 if(TEST_SUITE_SPEC2006_ROOT)
-  cpu2006_subdir(400.perlbench)
   cpu2006_subdir(401.bzip2)
   cpu2006_subdir(403.gcc)
   cpu2006_subdir(429.mcf)
   cpu2006_subdir(445.gobmk)
   cpu2006_subdir(456.hmmer)
   cpu2006_subdir(458.sjeng)
   cpu2006_subdir(462.libquantum)
-  cpu2006_subdir(464.h264ref)
   cpu2006_subdir(471.omnetpp)
   cpu2006_subdir(473.astar)
+  
+  if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
+    cpu2006_subdir(400.perlbench)
+    cpu2006_subdir(464.h264ref)
     cpu2006_subdir(483.xalancbmk)
   endif()
+endif()

MicroBenchmarks/CMakeLists.txt

 file(COPY lit.local.cfg DESTINATION ${CMAKE_CURRENT_BINARY_DIR})
 
+if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
   add_subdirectory(libs)
   add_subdirectory(XRay)
   add_subdirectory(LCALS)
   add_subdirectory(harris)
   add_subdirectory(ImageProcessing)
   add_subdirectory(LoopInterchange)
+endif()

MultiSource/Applications/CMakeLists.txt

+if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
   add_subdirectory(JM)
+endif()
 add_subdirectory(SIBsim4)
 add_subdirectory(aha)
 add_subdirectory(d)
 if(NOT ARCH STREQUAL "PowerPC")
 # This test has problems running on powerpc starting with r295538 and should
 # be restored when the issue is corrected.
   add_subdirectory(oggenc)
 endif()
 add_subdirectory(sgefa)
+if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
   add_subdirectory(spiff)
+endif()
 add_subdirectory(viterbi)
 
 add_subdirectory(ALAC)
 add_subdirectory(hbd)
 add_subdirectory(lambda-0.1.3)
 add_subdirectory(minisat)
-if(NOT TARGET_OS STREQUAL "SunOS")
+if(NOT TARGET_OS STREQUAL "SunOS"
+  AND (NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb"))
   add_subdirectory(hexxagon)
 endif()
 if(NOT DEFINED SMALL_PROBLEM_SIZE)
   add_subdirectory(lua)
 endif()
-if(TARGET_OS STREQUAL "Linux" AND (NOT ARCH STREQUAL "XCore"))
+if(TARGET_OS STREQUAL "Linux" AND (NOT ARCH STREQUAL "XCore")
+  AND (NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb"))
   add_subdirectory(obsequi)
 endif()
 if(NOT TARGET_OS STREQUAL "SunOS")
   add_subdirectory(kimwitu++)
 endif()
-if(NOT TARGET_OS STREQUAL "SunOS")
+if(NOT TARGET_OS STREQUAL "SunOS"
+  AND (NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb"))
   add_subdirectory(SPASS)
 endif()
 if(NOT ARCH STREQUAL "XCore")
-  add_subdirectory(ClamAV)
   add_subdirectory(lemon)
+  if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
+    add_subdirectory(ClamAV)
     add_subdirectory(siod)
   endif()
+endif()
 if((NOT ARCH STREQUAL "PowerPC") AND (NOT ARCH STREQUAL "XCore"))
   add_subdirectory(sqlite3)
 endif()
 if(NOT TEST_SUITE_BENCHMARKING_ONLY)
+  if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
     add_subdirectory(Burg)
+  endif()
   add_subdirectory(treecc)
 endif()

MultiSource/Benchmarks/CMakeLists.txt

 if((NOT "${TARGET_OS}" STREQUAL "Darwin") OR (NOT "${ARCH}" STREQUAL "ARM"))
   add_subdirectory(TSVC)
 endif()
 add_subdirectory(Prolangs-C++)
 add_subdirectory(Bullet)
 add_subdirectory(tramp3d-v4)
 add_subdirectory(DOE-ProxyApps-C++)
 if(NOT "${ARCH}" STREQUAL "XCore")
+  if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
     add_subdirectory(7zip)
+  endif()
   add_subdirectory(PAQ8p)
 endif()
 if(NOT DEFINED SMALL_PROBLEM_SIZE)
   add_subdirectory(mafft)
 endif()

MultiSource/Benchmarks/DOE-ProxyApps-C++/CMakeLists.txt

 add_subdirectory(HPCCG)
 add_subdirectory(PENNANT)
 add_subdirectory(miniFE)
+if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
   add_subdirectory(CLAMR)
+endif()
 add_subdirectory(HACCKernels)

MultiSource/Benchmarks/MiBench/CMakeLists.txt

 add_subdirectory(automotive-basicmath)
 add_subdirectory(automotive-bitcount)
 add_subdirectory(automotive-susan)
 add_subdirectory(consumer-jpeg)
 add_subdirectory(consumer-typeset)
 add_subdirectory(network-dijkstra)
+if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
   add_subdirectory(network-patricia)
+endif()
 add_subdirectory(security-rijndael)
 add_subdirectory(security-sha)
 add_subdirectory(telecomm-CRC32)
 add_subdirectory(telecomm-FFT)
 add_subdirectory(telecomm-gsm)
 if(NOT "${ARCH}" STREQUAL "XCore")
   add_subdirectory(consumer-lame)
 endif()
 if(NOT TEST_SUITE_BENCHMARKING_ONLY)
   add_subdirectory(office-stringsearch)
   add_subdirectory(security-blowfish)
   add_subdirectory(telecomm-adpcm)
-  if((NOT "${ARCH}" STREQUAL "XCore") AND (NOT "${ARCH}" STREQUAL "ARM"))
+  if((NOT "${ARCH}" STREQUAL "XCore") AND (NOT "${ARCH}" STREQUAL "ARM")
+    AND (NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb"))
     add_subdirectory(office-ispell)
   endif()
 endif()

MultiSource/Benchmarks/Prolangs-C/CMakeLists.txt

 add_subdirectory(agrep)
 add_subdirectory(gnugo)
-if((NOT "${ARCH}" STREQUAL "Sparc") AND (NOT "${ARCH}" STREQUAL "Alpha"))
+if((NOT "${ARCH}" STREQUAL "Sparc") AND (NOT "${ARCH}" STREQUAL "Alpha")
+  AND (NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb"))
   add_subdirectory(bison)
 endif()
 if(NOT TEST_SUITE_BENCHMARKING_ONLY)
   add_subdirectory(TimberWolfMC)
   add_subdirectory(allroots)
   add_subdirectory(assembler)
   add_subdirectory(cdecl)
   add_subdirectory(compiler)
   add_subdirectory(fixoutput)
   add_subdirectory(football)
   add_subdirectory(loader)
   add_subdirectory(plot2fig)
   add_subdirectory(simulator)
   add_subdirectory(unix-tbl)
   if(NOT "${ARCH}" STREQUAL "XCore")
     add_subdirectory(archie-client)
+    if(NOT TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
       add_subdirectory(unix-smail)
     endif()
   endif()
+endif()

SingleSource/UnitTests/CMakeLists.txt

list(REMOVE_ITEM Source
     ms_struct-bitfield.c
     ms_struct-bitfield-1.c
     ms_struct-bitfield-init.c
     ms_struct-bitfield-init-1.c
     ms_struct_pack_layout.c
     ms_struct_pack_layout-1.c
   )
 endif()
+if(TEST_SUITE_REMOTE_CLIENT STREQUAL "adb")
+  list(REMOVE_ITEM Source 2005-05-11-Popcount-ffs-fls.c)
+endif()
 llvm_singlesource()

cmake/caches/target-arm64-android-template.sh

+#!/bin/bash
+rm -r CMakeCache.txt
+
+
+# path to the test-suite source file
+TEST_SUITE_LOCATION="/data/local/tmp/devspace/test-suite"
+# path to android ndk that contains necessary header files and libraries
+ANDROID_NDK_LOCATION="/data/local/tmp/docspace/android-ndk-r20"
+# path to llvm build bin
+COMPILER_BIN="/data/local/tmp/docspace/llvm-build/bin"
+# OPTIONA: path to SPEC CPU2006 folder. Leave it empty if there is no SPEC CPU2006.
+SPEC2006_LOCATION="/data/local/tmp/docspace/cpu2006"
+# host system name
+# look inside $ANDROID_NDK_LOCATION/toolchains/llvm/prebuilt to figure out
+HOST_SYSTEM="linux-x86_64"
+# target android device serial number
+DEVICE_SERIAL="872Y00ETB"
+# the location on the device to store shared c++ library
+EXTRA_LIB_LOCATION="/data/local/tmp/docspace"
+# flag used during compilation
+COMPILE_FLAGS="--target=aarch64-linux-androideabi29 -fuse-ld=lld -L \
+$ANDROID_NDK_LOCATION/toolchains/llvm/prebuilt/$HOST_SYSTEM/lib/gcc/aarch64-linux-android/4.9.x/"
+
+
+# push the shared library onto the device
+adb -s $DEVICE_SERIAL push $ANDROID_NDK_LOCATION/toolchains/llvm/prebuilt/\
+$HOST_SYSTEM/sysroot/usr/lib/aarch64-linux-android/libc++_shared.so \
+$EXTRA_LIB_LOCATION/libc++_shared.so
+
+cmake \
+  -D CMAKE_C_FLAGS="$COMPILE_FLAGS -D BIONIC_IOCTL_NO_SIGNEDNESS_OVERLOAD -D SPEC_CPU_LINUX" \
+  -D CMAKE_CXX_FLAGS="$COMPILE_FLAGS" \
+  -D CMAKE_C_COMPILER=$COMPILER_BIN/clang \
+  -D CMAKE_CXX_COMPILER=$COMPILER_BIN/clang++ \
+  -D CMAKE_STRIP=$COMPILER_BIN/llvm-strip \
+  -D CMAKE_SYSROOT=$ANDROID_NDK_LOCATION/toolchains/llvm/prebuilt/$HOST_SYSTEM/sysroot \
+  -D TEST_SUITE_REMOTE_CLIENT="adb" \
+  -D TEST_SUITE_REMOTE_HOST=$DEVICE_SERIAL \
+  -D TEST_SUITE_EXTRA_LIB_LOCATION=$EXTRA_LIB_LOCATION \
+  -D TEST_SUITE_BENCHMARKING_ONLY=true \
+  -D TEST_SUITE_COLLECT_CODE_SIZE=OFF \
+  -D TEST_SUITE_SPEC2006_ROOT=$SPEC2006_LOCATION \
+  -D TEST_SUITE_RUN_TYPE=ref \
+  -G "Ninja" \
+  -C $TEST_SUITE_LOCATION/cmake/caches/O3.cmake \
+  $TEST_SUITE_LOCATION
 No newline at end of file

lit.cfg

 config.single_source = False
 if 'SSH_AUTH_SOCK' in os.environ:
     config.environment['SSH_AUTH_SOCK'] = os.environ['SSH_AUTH_SOCK']
 if not hasattr(config, 'remote_host'):
     config.remote_host = ""
 config.remote_host = lit_config.params.get('remote_host', config.remote_host)
 if config.remote_host:
     config.test_modules.append('remote')
+if config.remote_client == "adb":
+  config.test_modules.remove("remote")
+  config.test_modules.append("android")
 
 # Load previous test results so we can skip tests that did not change.
 previous_results_file = lit_config.params.get('previous', None)
 if previous_results_file:
     import json
     config.previous_results = json.load(open(previous_results_file))
 else:
     config.previous_results = None

lit.site.cfg.in

 import sys
 
 config.test_source_root = "@CMAKE_BINARY_DIR@"
 config.test_exec_root = "@CMAKE_BINARY_DIR@"
 config.remote_client = "@TEST_SUITE_REMOTE_CLIENT@"
 config.remote_host = "@TEST_SUITE_REMOTE_HOST@"
 config.run_under = "@TEST_SUITE_RUN_UNDER@"
+config.extra_lib_loc = "@TEST_SUITE_EXTRA_LIB_LOCATION@"
 config.strip_tool = "@CMAKE_STRIP@"
-config.profile_generate = @TEST_SUITE_PROFILE_GENERATE@
+config.profile_generate = "@TEST_SUITE_PROFILE_GENERATE@"
 config.llvm_profdata = "@TEST_SUITE_LLVM_PROFDATA@"
 config.llvm_size = "@TEST_SUITE_LLVM_SIZE@"
 config.test_modules = "@LIT_MODULES@".split(';')
 
 lit_config.load_config(config, "@CMAKE_SOURCE_DIR@/lit.cfg")

litsupport/modules/android.py

+"""Test module to execute a benchmark through adb on a connected Android
+device. This assumes all relevant directories and files are present on the remote
+device."""
+from litsupport import testplan
+import logging
+import os
+import subprocess
+
+
+def mutatePlan(context, plan):
+    # serial number of the device
+    if hasattr(context.config, "remote_host") and context.config.remote_host != "":
+        remote_host = "-s " + context.config.remote_host
+    else:
+        remote_host = ""
+    # the path to load libc++_shared.so
+    if hasattr(context.config, "extra_lib_loc") and context.config.extra_lib_loc != "":
+        extra_lib_loc = "LD_LIBRARY_PATH=" + context.config.extra_lib_loc
+    else:
+        extra_lib_loc = ""
+    
+    # append `adb [-s remote_host] shell` in front of each command
+    # append LD_LIBRARY_PATH to resolve libc++_shared.so
+    def append_adb_shell(s_in):
+        s_out = "adb {} shell \"{} {}\"".format(
+            remote_host, extra_lib_loc, s_in)
+        return s_out
+    
+    plan.preparescript = [append_adb_shell(script)
+        for script in plan.preparescript]
+    plan.runscript = [append_adb_shell(script)
+        for script in plan.runscript]
+    plan.verifyscript = [append_adb_shell(script)
+        for script in plan.verifyscript]
+
+    # create output directory if nonexist
+    plan.preparescript.append("adb {} shell mkdir -p {}".format(
+        remote_host, context.tmpDir))
+
+    # then adb pull timeit time file
+    if hasattr(context, "timefiles"):
+        for timefile in context.timefiles:
+            plan.profilescript.append("adb {} pull {} {}".format(
+                remote_host, timefile, timefile))

utils/rsync_android.sh

+#!/bin/bash
+
+# the target id
+TARGET=$1
+
+# build source path
+TEST_SUITE_BUILD=$2
+
+# convert the sym-linked directories
+# into actual directories by making a copy
+find $TEST_SUITE_BUILD -type l -exec bash -c 'cp -r $(readlink -m "$0") "$0.new"; rm "$0"; mv "$0.new" "$0"' {} \;
+
+# exclude all the files that won't be needed in the run
+# transfer the rest onto the device
+for file in $(find $TEST_SUITE_BUILD -type f \
+    ! -name "*.o" \
+    ! -name "*.a" \
+    ! -name "*.cmake" \
+    ! -name "*.test" \
+    ! -name "*.time" \
+    ! -name "*lit.local.cfg" \
+    ! -name ".ninja_deps" \
+    ! -name ".ninja_log" \
+    ! -name "build.ninja" \
+    ! -name "rules.ninja" \
+    ! -name "CMakeFiles")
+do
+    if [[ -z $TARGET ]]; then
+        adb push $file $file
+    else
+        adb -s $TARGET push $file $file
+    fi
+done