gn build: Add support for building scudo and its unit tests.

[pcc]

Summary

Depends on D71077

Depends on D71078

Depends on D71079

[merge_guards_bot]

Build result: pass - 60520 tests passed, 0 failed and 726 were skipped.

Log files: console-log.txt, CMakeCache.txt

[pcc]

• gn format

[merge_guards_bot]

Build result: FAILURE - Could not check out parent git hash "93bae253ee57aa94241382011f12488eff7d31e6". It was not found in the repository. Did you configure the "Parent Revision" in Phabricator properly? Trying to apply the patch to the master branch instead...

ERROR: arc patch failed with error code 1. Check build log for details.
Log files: console-log.txt, CMakeCache.txt

[pcc]

• Silence sync_source_lists_from_cmake.py

[merge_guards_bot]

Build result: FAILURE - Could not check out parent git hash "93bae253ee57aa94241382011f12488eff7d31e6". It was not found in the repository. Did you configure the "Parent Revision" in Phabricator properly? Trying to apply the patch to the master branch instead...

ERROR: arc patch failed with error code 1. Check build log for details.
Log files: console-log.txt, CMakeCache.txt

[phosek]

LGTM

[thakis]

This breaks building on Mac: http://45.33.8.238/mac/3673/step_4.txt

[thakis]

I think the problem there is that on mac, libc++ headers are supposed to go with the compiler, and the stage1 clang in out/gn doesn't have libc++ headers installed next to it. The stage2_unix toolchain should depend on copying libc++ headers to the build output dir on macOS.

For now I'll disable this on mac, but every other stage2_unix target that uses C++ headers will have the same problem. So hopefully I'll find time to fix this right soon.

[thakis]

In D71081#1772598, @thakis wrote:

I think the problem there is that on mac, libc++ headers are supposed to go with the compiler, and the stage1 clang in out/gn doesn't have libc++ headers installed next to it. The stage2_unix toolchain should depend on copying libc++ headers to the build output dir on macOS.

For now I'll disable this on mac, but every other stage2_unix target that uses C++ headers will have the same problem. So hopefully I'll find time to fix this right soon.

I looked a this some. It's easy to add a dep on libc++ [1]. However, scudo still fails to build with errors like ../../compiler-rt/lib/scudo/standalone/common.h:155:11: error: unknown type name 'MapPlatformData', and nothing seems to define that type on non-Linux. Is scudo supposed to build on mac in the cmake build?

1:

% git diff
diff --git a/llvm/utils/gn/build/toolchain/BUILD.gn b/llvm/utils/gn/build/toolchain/BUILD.gn
index b8e46034cca..9ebbe001d0b 100644
--- a/llvm/utils/gn/build/toolchain/BUILD.gn
+++ b/llvm/utils/gn/build/toolchain/BUILD.gn
@@ -183,7 +183,9 @@ template("stage2_unix_toolchain") {
       "//:clang($host_toolchain)",
       "//:lld($host_toolchain)",
     ]
-    if (current_os != "mac") {
+    if (current_os == "mac") {
+      deps += [ "//libcxx/include" ]
+    } else {
       deps += [ "//:llvm-ar($host_toolchain)" ]
     }
   }
diff --git a/llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn b/llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn
index a95613c5bcd..8c658837cd2 100644
--- a/llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn
+++ b/llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn
@@ -1,11 +1,7 @@
 import("//llvm/utils/gn/build/toolchain/compiler.gni")
 
 supported_toolchains = []
-# FIXME: On macOS, stage2_unix currently doesn't copy libc++ headers to
-# the out dir, but clang relies on them on mac to compile code that uses C++
-# standard library headers. scudo needs C++ standard library headers, so disable
-# this on mac until stage2_unix correctly copies libc++ headers.
-if (target_os != "win" && target_os != "mac")  {
+if (target_os != "win")  {
   supported_toolchains += [ "//llvm/utils/gn/build/toolchain:stage2_unix" ]
 }
 if (android_ndk_path != "") {

[pcc]

Yes, looking through the code as well as the CMake [1], scudo only supports building on Linux, Android and Fuchsia. Most of the code in linux.cpp looks like it would work on non-Linux POSIX, except for the mutex implementation.

It looks like the non-standalone allocator had some level of support for Windows [2] but that wasn't carried over to the standalone allocator.

So we should probably change the blacklist in the supported platforms to a whitelist for now. I'll go ahead and do that.

[1] http://llvm-cs.pcc.me.uk/projects/compiler-rt/cmake/config-ix.cmake#695
[2] http://llvm-cs.pcc.me.uk/projects/compiler-rt/lib/scudo/scudo_allocator_secondary.h#110

[pcc]

In D71081#1773044, @pcc wrote:

Yes, looking through the code as well as the CMake [1], scudo only supports building on Linux, Android and Fuchsia. Most of the code in linux.cpp looks like it would work on non-Linux POSIX, except for the mutex implementation.

It looks like the non-standalone allocator had some level of support for Windows [2] but that wasn't carried over to the standalone allocator.

So we should probably change the blacklist in the supported platforms to a whitelist for now. I'll go ahead and do that.

[1] http://llvm-cs.pcc.me.uk/projects/compiler-rt/cmake/config-ix.cmake#695
[2] http://llvm-cs.pcc.me.uk/projects/compiler-rt/lib/scudo/scudo_allocator_secondary.h#110

D71131

llvm/utils/gn/secondary/BUILD.gn

 import("//clang/lib/ARCMigrate/enable.gni")
 import("//clang/lib/StaticAnalyzer/Frontend/enable.gni")
 import("//llvm/utils/gn/build/toolchain/compiler.gni")
 
 group("default") {
   deps = [
     "//clang-tools-extra/clangd/test",
     "//clang-tools-extra/test",
     "//clang/test",
+    "//compiler-rt/lib/scudo",
     "//lld/test",
     "//llvm/test",
   ]
   if (current_os == "linux" || current_os == "mac") {
     deps += [ "//compiler-rt" ]
   }
   if (current_os == "linux") {
     deps += [

llvm/utils/gn/secondary/compiler-rt/lib/scudo/BUILD.gn

+import("//llvm/utils/gn/build/toolchain/compiler.gni")
+
+supported_toolchains = []
+if (target_os != "win") {
+  supported_toolchains += [ "//llvm/utils/gn/build/toolchain:stage2_unix" ]
+}
+if (android_ndk_path != "") {
+  supported_toolchains += [
+    "//llvm/utils/gn/build/toolchain:stage2_android_aarch64",
+    "//llvm/utils/gn/build/toolchain:stage2_android_arm",
+  ]
+}
+
+group("scudo") {
+  deps = []
+  foreach(toolchain, supported_toolchains) {
+    deps += [
+      "standalone/tests:ScudoCUnitTest($toolchain)",
+      "standalone/tests:ScudoCxxUnitTest($toolchain)",
+      "standalone/tests:ScudoUnitTest($toolchain)",
+    ]
+  }
+  testonly = true
+}
+
+# This target is unused, it only exists to satisfy
+# sync_source_lists_from_cmake.py.
+source_set("sources") {
+  sources = [
+    "scudo_allocator.cpp",
+    "scudo_allocator.h",
+    "scudo_allocator_combined.h",
+    "scudo_allocator_secondary.h",
+    "scudo_crc32.cpp",
+    "scudo_crc32.h",
+    "scudo_errors.cpp",
+    "scudo_errors.h",
+    "scudo_flags.cpp",
+    "scudo_flags.h",
+    "scudo_interface_internal.h",
+    "scudo_malloc.cpp",
+    "scudo_platform.h",
+    "scudo_termination.cpp",
+    "scudo_tsd.h",
+    "scudo_tsd_exclusive.cpp",
+    "scudo_tsd_shared.cpp",
+  ]
+}

llvm/utils/gn/secondary/compiler-rt/lib/scudo/standalone/BUILD.gn

+import("//compiler-rt/target.gni")
+
+source_set("sources") {
+  configs -= [ "//llvm/utils/gn/build:llvm_code" ]
+  configs += [ "//llvm/utils/gn/build:crt_code" ]
+  sources = [
+    "allocator_config.h",
+    "atomic_helpers.h",
+    "bytemap.h",
+    "checksum.cpp",
+    "checksum.h",
+    "chunk.h",
+    "combined.h",
+    "common.cpp",
+    "crc32_hw.cpp",
+    "flags.cpp",
+    "flags.h",
+    "flags_parser.cpp",
+    "flags_parser.h",
+    "fuchsia.cpp",
+    "fuchsia.h",
+    "interface.h",
+    "internal_defs.h",
+    "linux.cpp",
+    "linux.h",
+    "list.h",
+    "local_cache.h",
+    "mutex.h",
+    "platform.h",
+    "primary32.h",
+    "primary64.h",
+    "quarantine.h",
+    "release.h",
+    "report.cpp",
+    "report.h",
+    "secondary.h",
+    "size_class_map.h",
+    "stats.h",
+    "string_utils.cpp",
+    "string_utils.h",
+    "tsd.h",
+    "tsd_exclusive.h",
+    "tsd_shared.h",
+    "vector.h",
+    "wrappers_c.h",
+    "wrappers_c_checks.h",
+  ]
+
+  if (current_cpu == "arm" || current_cpu == "arm64") {
+    cflags = [ "-mcrc" ]
+  }
+  if (current_cpu == "x64") {
+    cflags = [ "-msse4.2" ]
+  }
+
+  public_configs = [ ":scudo_config" ]
+}
+
+source_set("c_wrapper_sources") {
+  configs -= [ "//llvm/utils/gn/build:llvm_code" ]
+  configs += [ "//llvm/utils/gn/build:crt_code" ]
+  sources = [
+    "wrappers_c.cpp",
+  ]
+
+  public_configs = [ ":scudo_config" ]
+}
+
+source_set("cxx_wrapper_sources") {
+  configs -= [ "//llvm/utils/gn/build:llvm_code" ]
+  configs += [ "//llvm/utils/gn/build:crt_code" ]
+  sources = [
+    "wrappers_cpp.cpp",
+  ]
+
+  public_configs = [ ":scudo_config" ]
+}
+
+config("scudo_config") {
+  include_dirs = [ "//compiler-rt/lib/scudo/standalone" ]
+  if (current_os == "android") {
+    cflags = [ "-fno-emulated-tls" ]
+  }
+}

llvm/utils/gn/secondary/compiler-rt/lib/scudo/standalone/tests/BUILD.gn

+import("//llvm/utils/unittest/unittest.gni")
+
+unittest("ScudoUnitTest") {
+  configs += [ "//llvm/utils/gn/build:crt_code" ]
+  deps = [
+    "//compiler-rt/lib/scudo/standalone:sources",
+  ]
+  sources = [
+    "atomic_test.cpp",
+    "bytemap_test.cpp",
+    "checksum_test.cpp",
+    "chunk_test.cpp",
+    "combined_test.cpp",
+    "flags_test.cpp",
+    "list_test.cpp",
+    "map_test.cpp",
+    "mutex_test.cpp",
+    "primary_test.cpp",
+    "quarantine_test.cpp",
+    "release_test.cpp",
+    "report_test.cpp",
+    "scudo_unit_test_main.cpp",
+    "secondary_test.cpp",
+    "size_class_map_test.cpp",
+    "stats_test.cpp",
+    "strings_test.cpp",
+    "tsd_test.cpp",
+    "vector_test.cpp",
+  ]
+  has_custom_main = true
+}
+
+unittest("ScudoCUnitTest") {
+  configs += [ "//llvm/utils/gn/build:crt_code" ]
+  deps = [
+    "//compiler-rt/lib/scudo/standalone:c_wrapper_sources",
+    "//compiler-rt/lib/scudo/standalone:sources",
+  ]
+  sources = [
+    "scudo_unit_test_main.cpp",
+    "wrappers_c_test.cpp",
+  ]
+  has_custom_main = true
+}
+
+unittest("ScudoCxxUnitTest") {
+  configs += [ "//llvm/utils/gn/build:crt_code" ]
+  deps = [
+    "//compiler-rt/lib/scudo/standalone:c_wrapper_sources",
+    "//compiler-rt/lib/scudo/standalone:cxx_wrapper_sources",
+    "//compiler-rt/lib/scudo/standalone:sources",
+  ]
+  sources = [
+    "scudo_unit_test_main.cpp",
+    "wrappers_cpp_test.cpp",
+  ]
+  has_custom_main = true
+}