This is an archive of the discontinued LLVM Phabricator instance.

[scudo][standalone] Introduce platform specific code & mutexes
ClosedPublic

Authored by cryptoad on Feb 13 2019, 7:39 AM.

Details

Summary

This CL adds the platform specific code for Fuchsia, Linux & Android,
as well as some tests related to those (more tests to come later).
While some of it is pretty much a straight port of the existing scudo &
sanitizer_common code, the memory mapping functions have been reworked
a bit to fit the limited usage scenario that Scudo has for them.

For Fuchsia, I can now track the Vmar/Vmo pair for memory mappings if
there is an intent to grow or decommit some mapping (that will be
useful for the Primary).

Event Timeline

cryptoad created this revision.Feb 13 2019, 7:39 AM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptFeb 13 2019, 7:39 AM
Herald added subscribers: Restricted Project, jdoerfert, jfb and 3 others. · View Herald Transcript

I only really looked at the Fuchsia code.

lib/scudo/standalone/common.h
141

Parens?

150

Is it really useful to have MAP_ERROR instead of just using nullptr here?
In the traditional mmap interface, MAP_FAILED is not 0 because it's sometimes possible to actually mmap address 0.
But that's not relevant here, so why not just stick the standard C++?

lib/scudo/standalone/fuchsia.cc
32

We mostly just use __builtin_trap() directly.

51

Use _zx_vmar_allocate, the new API. Parameters are in a different order, and flags are spelled ZX_VM_* rather than ZX_VM_FLAG_*.

67

You're only using VMARs you created yourself, so why use a syscall to get the bounds rather than keeping track locally?

177

Why doesn't this just call libc's getenv?

186

Use _zx_futex_wait. It just needs an extra ZX_HANDLE_INVALID parameter (second to last).

200

_zx

203

Where does the magic 256 constant come from?
If it's part of the getRandom API then it should be a macro or constexpr in the OS-independent headers.

212

There no need for locking around this.

lib/scudo/standalone/internal_defs.h
38

Why not just use C++ standard alignas?

128

Why not use <cstdint>?

cryptoad updated this revision to Diff 186723.Feb 13 2019, 12:55 PM
cryptoad marked 10 inline comments as done.

Addressing some of Roland's review comments.

cryptoad added inline comments.Feb 13 2019, 12:55 PM
lib/scudo/standalone/common.h
150

Personal preference here. I thought that sanitizer_common was bad at mixing null & (void *)-1 for errors, and it did try to map 0 in some places.
So I went with all MAP_FAILED type error for map functions.
Though I am not opposed to all nullptr.

lib/scudo/standalone/fuchsia.cc
67

It was mostly for storage reasons.
The vmar+vmo fits in a u64 which works well. Keeping the base & size would require 2 extra uptr, which to be fair is not the end of the world.
The extra 16 bytes would be for every secondary backed allocation & for every region of the primary.
I am open to doing either.

177

I went the way it was done in sanitizer_common.
Part of the reason being that on some platform (Android), getenv would not work when called either too early or from the init process for example.
If this is not case for Fuchsia, happy to switch.

203

256 is a magic Linux related constant.
getrandom doesn't get interrupted for sizes < 256.
And when reading /dev/urandom, reads of up to 256 bytes will return as many bytes as are requested and will not be interrupted by a signal handler.
Since we don't use large random streams of bytes, I enforced 256.
I will put that in the common header.

lib/scudo/standalone/internal_defs.h
128

Right, it's mostly a sanitizer_common artifact, I haven't revisited it.
cstdint might not work with -nostdinc++ but stdint.h might.

cryptoad updated this revision to Diff 186724.Feb 13 2019, 12:58 PM

clang-format the changes.

cryptoad marked 2 inline comments as not done.Feb 13 2019, 12:59 PM
mcgrathr added inline comments.Feb 13 2019, 1:20 PM
lib/scudo/standalone/common.h
150

I'd prefer we completely avoid casting bogons to pointer types if we can.

lib/scudo/standalone/fuchsia.cc
67

Give it some comments about the trade-offs of storage cost vs syscall cost and we can optimize later.

177

There should be no problem on Fuchsia and if something crops up we can figure it out.

203

OK, then I think a static_assert(NewConstantForMaxSize <= ZX_CPRNG_DRAW_MAX_LEN) is what we want here.

lib/scudo/standalone/internal_defs.h
128

<stdint.h> is already used here, so just use that.

cryptoad updated this revision to Diff 186744.Feb 13 2019, 2:19 PM
cryptoad marked 12 inline comments as done.

Address additional comments by Roland:

  • map() now returns nullptr on error;
  • New constant for max random length & associated static_assert;
  • use getenv() on Fuchsia;
  • Comments on storage vs syscall for vmar ranges;
  • remove integer constants, replaced by stdint.h
cryptoad updated this revision to Diff 186749.Feb 13 2019, 2:26 PM

Correct comment typo.

dmmoore415 added inline comments.Feb 14 2019, 9:30 AM
lib/scudo/standalone/fuchsia.cc
63

sp: alternive

132

Shouldn't you at least check and log a bad status here?

150

while there's not much we can do if these things fail, maybe we should at least log them so that leaks can be discovered. There are several other cases like this. Is it Fuchsia style to ignore failures on things like this?

cryptoad updated this revision to Diff 186864.Feb 14 2019, 9:37 AM
cryptoad marked 2 inline comments as done.

Correcting spelling mistake in comment.

cryptoad marked an inline comment as not done.Feb 14 2019, 9:37 AM
cryptoad added inline comments.
lib/scudo/standalone/fuchsia.cc
132

I am going to have to defer to people who know more about the internals than I do: in which situations can closing a handle (or destroying a vmar) fail? Is it recoverable? Most of the code I see in musl for example doesn't check for that.

mcgrathr added inline comments.Feb 14 2019, 10:11 AM
lib/scudo/standalone/fuchsia.cc
132

They should never fail. If they do it's probably a bug somewhere so having a panic wouldn't hurt. I think the only expected error cases for those are for a bad handle, which would indicate a bug in this code or some catastrophic clobberation.

cryptoad updated this revision to Diff 186867.Feb 14 2019, 10:20 AM
cryptoad marked 4 inline comments as done.

CHECK'ing the success (ZX_OK) of close & destroy operations.

cryptoad updated this revision to Diff 187034.Feb 15 2019, 9:54 AM

Rebasing.

morehouse added inline comments.Feb 15 2019, 4:28 PM
lib/scudo/standalone/common.h
90

What is this used for?

123

If we will always get the page size through this function call (and not the global variable), this could be simplified to:

static PageSizeCached = getPageSizeSlow();
return PageSizeCached;
172

What's this here for?

lib/scudo/standalone/linux.cc
50

Since we will be mapping PROT_NONE in the scenario, I believe MAP_NORESERVE is unnecessary. I.e., the OS won't reserve swap space anyway since the memory cannot be accessed.

54

Does Scudo actually do MAP_FIXED? If so, why?

59

Considering we just did a system call, this path optimization doesn't provide any substantial benefit.

72

Another unnecessary path optimization.

78

Should we check errno for EAGAIN here?

85

reinterpret_cast? Here and below

104

Should this be cached, as it is in sanitizer_common?

119

static_cast? Same for below.

lib/scudo/standalone/linux.h
56

Why is this 8 when sanitizer_common uses 6?

lib/scudo/standalone/mutex.h
40

With a u8, we might wraparound and do 10 more "quick yields" on a long lock. Should we bump this up to u32 to reduce chance of this happening?

58

Are these constructors ever used? If not, maybe just use SpinMutex(const SpinMutex &) = delete.

101

Maybe make these deleted too.

cryptoad marked 6 inline comments as done.Feb 15 2019, 11:27 PM

Thanks Matt! Textual answers only for now, code changes will follow.

lib/scudo/standalone/common.h
90

shuffle will be used for randomizing arrays in the Primary & Quarantine.
Both haven't been checked in yet but will be in a few check-ins (once all the "common" files make it in).
A similar function can be found in sanitizer_common, use will be very similar if not identical.

123

AFAIR those constructs end up in an initialization function being created and run with the global ctors which I am generally trying to avoid.
I will give it a try.

172

It's a common function used by the Primary & Secondary (not checked in yet).
It probably could find a home somewhere else, I am going to remove it for now.

lib/scudo/standalone/linux.cc
50

I will revisit that.

54

We MAP_FIXED inside memory that was reserved with PROT_NONE.
The two usage scenario are:

  • mapping on demand in the Primary, basically extending regions;
  • mapping secondary allocations with guard pages on either side;
59

I will change that.

72

I will change that.

78

Good point, I didn't think of it. It is indeed probably useful to retry in that scenario rather than wait for another opportunity.

85

I will change that.

104

In the current state of things it's only called once for the initialization of the Shared TSD registry.
I figured caching was not needed (I don't think sanitizer_common use it more than once either).

119

I will change that.

lib/scudo/standalone/linux.h
56

It changed somewhat recently (https://android-review.googlesource.com/c/platform/bionic/+/847554).
Right now I am testing on platforms that still have 8, but it will have to be change to 6 for Q.

lib/scudo/standalone/mutex.h
40

I will change that.

58

I will change that.

101

I will change that.

mcgrathr added inline comments.Feb 16 2019, 1:16 AM
lib/scudo/standalone/common.h
123

A static local ctor is different from a global ctor (and harder to notice from examining the object files).
Each is problematic in its own different way.
I think both should be completely ruled out for the scudo internals.
That's a hard requirement for Fuchsia anyway, and it seems like a good thing to do universally.
Of course any kind of constexpr initialization (i.e. "linker-initialized") is OK.

cryptoad updated this revision to Diff 187146.Feb 16 2019, 11:43 AM
cryptoad marked 21 inline comments as done.

Addressing various comments from Matt's review.

cryptoad marked 2 inline comments as not done.Feb 16 2019, 11:47 AM
morehouse added inline comments.Feb 19 2019, 1:00 PM
lib/scudo/standalone/mutex.h
58

I think operator= needs to be deleted as well to avoid copies. Below as well.

lib/scudo/standalone/tests/map_test.cc
52

This boilerplate would be a good candidate to put in a test setup class.

65

Maybe we should combine these with the above tests since they will only cost ~1 line each then.

also this patch includes to many unrelated changes

lib/scudo/standalone/internal_defs.h
126

Why not to use static_assert directly
Same for INLINE

lib/scudo/standalone/linux.cc
42

sysconf can be intercepted, e.g. in Google internal code

107

I assume this comment is for "#if defined"
could you move it there?

119

do you need static_cast here?

lib/scudo/standalone/linux.h
56

please run clang-format

lib/scudo/standalone/mutex.h
46

this if does not look atomic
atomic_compare_exchange?

cryptoad marked an inline comment as done.Feb 21 2019, 11:46 AM
cryptoad added subscribers: dvyukov, kcc.

Currently OOO so addressing mostly the easy to deal with comments.

lib/scudo/standalone/internal_defs.h
126

Personal preference here: I like the consistency.
For inline, it's so that the function attributes all are cased identically.
For the static assert, it's more because of habit with the current sanitizer code (since it used a macro). The macro isn't great with Werror due to unused typedefs, but static_assert is supported widely enough.
I am not opposed to changing to changing static_assert everywhere, but I would like to keep INLINE for consistency.
Let me know.

lib/scudo/standalone/linux.cc
42

Will that code not work if the code is intercepted? Current sanitizer code returns EXEC_PAGESIZE for Linux, should I just use that? If so it feel like it could go in the headers instead and that would optimize a bunch of code.

107

Well, my comment was more about the UNUSED attribute for Blocking in the function prototype, hence the position here.
Technically it only matters for the #if defined block indeed. Let me know what you think makes more sense.

119

Yes, the ssize_t is signed and the compiler complains about signed vs unsigned comparison otherwise.

lib/scudo/standalone/linux.h
56

It is currently clang-format'd but with -style=llvm which doesn't enforce the 2 space prior to comment if that's the thing you are referring to.

lib/scudo/standalone/mutex.h
46

Good point. So this is current how it's done in sanitizer_common and I didn't put additional thought into it.
Maybe the atomicity was too costly given that the state would eventually settle to something consistent?
cc: @kcc @dvyukov

lib/scudo/standalone/tests/map_test.cc
65

I'll do that. I initially went for full separation of Death Tests but it's not worth it.

cryptoad updated this revision to Diff 187836.Feb 21 2019, 11:48 AM

Deleting the operator= for the mutex & lock classes.

cryptoad updated this revision to Diff 188197.Feb 25 2019, 9:11 AM
cryptoad marked 7 inline comments as done.

Handle EAGAIN for madvice.
Reorganize the map tests to lower the code cost of death tests (~1 line).

morehouse accepted this revision.Feb 25 2019, 10:41 AM

Linux-specific and common code LGTM. Will let Fuschia people comment on that part.

This revision is now accepted and ready to land.Feb 25 2019, 10:41 AM
vitalybuka accepted this revision.Feb 25 2019, 1:47 PM

LGTM

lib/scudo/standalone/internal_defs.h
126

I would prefer standard keyword when possible.
static_assert was probably not available at that time.
Not sure why they decided to use INLINE.

I don't have strong opinion on that. Feel free to keep either way.

lib/scudo/standalone/linux.cc
42

It depends.
You are likely going to use this in initialization, and interceptor may try to allocate memory using SCUDO. I guess it's fine to keep it as-is and resolve if we hit this problem.

107

either way is OK

lib/scudo/standalone/linux.h
56

Yep, spaces.
Then it's my mistake, please keep is as it's set by clang-format.

lib/scudo/standalone/mutex.h
46

If you are going to submit as-is maybe some TODO or other comment here?

kcc added inline comments.Feb 25 2019, 6:23 PM
lib/scudo/standalone/mutex.h
46

This is dvyukov's original code, but it looks just fine as is.
First load is a quick check: is the lock most likely unlocked now?
The second exchange is the actual atomic part.

cryptoad marked 22 inline comments as done.Feb 26 2019, 8:35 AM
cryptoad updated this revision to Diff 188387.Feb 26 2019, 8:36 AM

Rebasing.

This revision was automatically updated to reflect the committed changes.

Hello @cryptoad , I am seeing compiler build failures starting with the commit for this patch (sha 41aba567d991c2bd551c ).

There are actually two breaks I've seen. The first is a linker error when attempting to add i386 and x86_64 objects together on an Ubuntu 18.04 LTS host:

[1840/4478] : && /usr/bin/clang++-6.0 -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-string
s -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-con
version -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++11 -Wno-unused-parameter -O3  -Wl,-z,defs -Wl,-z,nodelete -shared -Wl,-soname,libRTScudoStandalone.test.i386.so -o
 lib/libRTScudoStandalone.test.i386.so projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/common.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoS
tandalone.i386.dir/fuchsia.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/linux.cc.o   && :
FAILED: lib/libRTScudoStandalone.test.i386.so
: && /usr/bin/clang++-6.0 -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++11 -Wno-unused-parameter -O3  -Wl,-z,defs -Wl,-z,nodelete -shared -Wl,-soname,libRTScudoStandalone.test.i386.so -o lib/libRTScudoStandalone.test.i386.so projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/common.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/fuchsia.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/linux.cc.o   && :
/usr/bin/ld: i386 architecture of input file `projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/common.cc.o' is incompatible with i386:x86-64 output
/usr/bin/ld: i386 architecture of input file `projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/fuchsia.cc.o' is incompatible with i386:x86-64 output
/usr/bin/ld: i386 architecture of input file `projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/linux.cc.o' is incompatible with i386:x86-64 output
/usr/bin/ld: projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/common.cc.o: file class ELFCLASS32 incompatible with ELFCLASS64
/usr/bin/ld: final link failed: File in wrong format
clang: error: linker command failed with exit code 1 (use -v to see invocation)

which I suspect are partly caused by CMake detecting i386 and x86_64 support for Compiler-RT:

-- Looking for __i386__
-- Looking for __i386__ - found
-- Compiler-RT supported architectures: x86_64;i386
-- Performing Test COMPILER_RT_HAS_STD_C11_FLAG
-- Performing Test COMPILER_RT_HAS_STD_C11_FLAG - Success
-- Performing Test COMPILER_RT_HAS_VISIBILITY_HIDDEN_FLAG
-- Performing Test COMPILER_RT_HAS_VISIBILITY_HIDDEN_FLAG - Success
-- Performing Test COMPILER_RT_HAS_OMIT_FRAME_POINTER_FLAG
-- Performing Test COMPILER_RT_HAS_OMIT_FRAME_POINTER_FLAG - Success
-- Performing Test COMPILER_RT_HAS_FREESTANDING_FLAG
-- Performing Test COMPILER_RT_HAS_FREESTANDING_FLAG - Success
-- Performing Test COMPILER_RT_HAS_XRAY_COMPILER_FLAG
-- Performing Test COMPILER_RT_HAS_XRAY_COMPILER_FLAG - Success
-- Performing Test COMPILER_RT_HAS_ATOMIC_KEYWORD
-- Performing Test COMPILER_RT_HAS_ATOMIC_KEYWORD - Success
-- Builtin supported architectures: i386;x86_64

The second compiler failure is due to clock_gettime undefined on RHEL 6.7 hosts:

[2428/3976] : && /tools/build/gcc-8.2.0/bin/g++ -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wc
ast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment
 -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++11 -Wno-unused-parameter -O3  -Wl,-z,defs -Wl,-z,nodelete -shared -Wl,-soname,libRTScudoStandalone.test.x86_64.so -o lib/
libRTScudoStandalone.test.x86_64.so projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.x86_64.dir/common.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoSt
andalone.x86_64.dir/fuchsia.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.x86_64.dir/linux.cc.o   && :
FAILED: lib/libRTScudoStandalone.test.x86_64.so
: && /tools/build/gcc-8.2.0/bin/g++ -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++11 -Wno-unused-parameter -O3  -Wl,-z,defs -Wl,-z,nodelete -shared -Wl,-soname,libRTScudoStandalone.test.x86_64.so -o lib/libRTScudoStandalone.test.x86_64.so projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.x86_64.dir/common.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.x86_64.dir/fuchsia.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.x86_64.dir/linux.cc.o   && :
projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.x86_64.dir/linux.cc.o: In function `scudo::getMonotonicTime()':
linux.cc:(.text._ZN5scudo16getMonotonicTimeEv+0xd): undefined reference to `clock_gettime'
collect2: error: ld returned 1 exit status

I suspect this is due to glibc being very old :

$ rpm -qa | grep ^glibc-2
glibc-2.12-1.166.el6_7.7.i686
glibc-2.12-1.166.el6_7.7.x86_64

The glibc maintainers moved clock_*() from librt to libc in version 2.17. This means some builds require -lrt for linking while others do not. And yes, this is ancient code but the folks at RedHat move at glacial speed when upgrading core packages in RHEL. They just barely moved to glibc 2.17 in RHEL 7.

I don't have patches for these yet but I wanted to inform you in case you may be aware of quick fixes/workarounds for these issues.

Ack, will look into it. I do not have the setups to repro those locally so it might take a little while.
Feel free to submit something if you come up with it quicker than I can.

I assume the rt issue can be fixed with:
append_list_if(COMPILER_RT_HAS_LIBRT -lrt LINK_FLAGS)
in the standalone tests CMakeLists.txt.
I am unclear as to what the solution is for the i386 vs x64 issue, as I also have both and running test for both without issue.

@cryptoad the following (terribly hacky) patch successfully builds for me on Ubuntu 18.04 LTS:

diff --git a/compiler-rt/cmake/base-config-ix.cmake b/compiler-rt/cmake/base-config-ix.cmake
index ee9426b715d..8cc1e922f50 100644
--- a/compiler-rt/cmake/base-config-ix.cmake
+++ b/compiler-rt/cmake/base-config-ix.cmake
@@ -161,7 +161,7 @@ macro(test_targets)
           endif()
         else()
           test_target_arch(x86_64 "" "-m64")
-          test_target_arch(i386 __i386__ "-m32")
+          #test_target_arch(i386 __i386__ "-m32")
         endif()
       else()
         if (CMAKE_SIZEOF_VOID_P EQUAL 4)

I'm about to test it on RHEL 6.7 to see if the two are orthogonal or related (I suspect the former).

It's interesting you have no problems building i386. For reference here is my cmake configuration line:

cmake -G Ninja  -D CMAKE_C_COMPILER=/usr/bin/clang-6.0 -D CMAKE_CXX_COMPILER=/usr/bin/clang++-6.0 -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX=/work/b.rzycki/upstream/install -D LLVM_TOOL_CLANG_BUILD=ON -D LIBCXX_INCLUDE_BENCHMARKS=ON -D BUILD_SHARED_LIBS=ON -D LLVM_ENABLE_ASSERTIONS=ON  -D LLVM_TARGETS_TO_BUILD=X86 /work/b.rzycki/upstream/llvm-project/llvm

I'm about to test it on RHEL 6.7 to see if the two are orthogonal or related (I suspect the former).

As expected, it did not fix RHEL 6.7. I am now trying your append_list_if(COMPILER_RT_HAS_LIBRT -lrt LINK_FLAGS) suggestion.

append_list_if(COMPILER_RT_HAS_LIBRT -lrt LINK_FLAGS) did not work, nor did list(APPEND LINK_FLAGS -lrt) for RHEL 6.7.

I noticed this code failure path is a spawned cmake instance within the Ninja build. It starts by trying to build libfuzzer support for libcxx:

[1225/3976] cd /tmp/tmp.DxqFrKPwdj/build/llvm/projects/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64-bins && /tools/build/cmake-3.13.4/bin/cmake -DCMAKE_SHARED_LINKER_FLAGS=
-DCMAKE_MODULE_LINKER_FLAGS= -DCMAKE_EXE_LINKER_FLAGS= -DCMAKE_INSTALL_PREFIX=/tmp/bmr/4 -DCMAKE_MAKE_PROGRAM=/tools/build/ninja-1.9.0/ninja -DCMAKE_LINKER=/usr/bin/ld -DCM
AKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_NM=/usr/bin/nm -DCMAKE_OBJCOPY=/usr/bin/objcopy -DCMAKE_OBJDUMP=/usr/bin/objdump -DCMAKE_STRIP=/usr/bin/strip -DCMAKE_C_COMPILER=/sarc
-c/compiler_tmp/tools/build/gcc-8.2.0/bin/gcc -DCMAKE_CXX_COMPILER=/tools/build/gcc-8.2.0/bin/g++ "-DCMAKE_C_FLAGS=-m64 " "-DCMAKE_CXX_FLAGS=-m64 " -DCMAKE_BUILD_TYPE=Relea
se -DLLVM_PATH=/tmp/tmp.DxqFrKPwdj/src/llvm -DLLVM_BINARY_DIR=/tmp/tmp.DxqFrKPwdj/build/llvm/projects/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64 -DLLVM_LIBRARY_OUTPUT_INTDIR=/tmp/tmp.DxqFrKPw
dj/build/llvm/projects/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/lib -DCOMPILER_RT_LIBCXX_PATH=/tmp/tmp.DxqFrKPwdj/src/llvm/projects/libcxx -DCOMPILER_RT_LIBCXXABI_PATH=/tmp/tmp.DxqFrKPwdj/s
rc/llvm/projects/libcxxabi -DCMAKE_CXX_COMPILER_WORKS=ON -DLIBCXX_ABI_NAMESPACE=Fuzzer -GNinja /tmp/tmp.DxqFrKPwdj/src/llvm/projects/compiler-rt/cmake/Modules/CustomLibcxx && /sarc-c/compiler_
tmp/tools/build/cmake-3.13.4/bin/cmake -E touch /tmp/tmp.DxqFrKPwdj/build/llvm/projects/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64-stamps//libcxx_fuzzer_x86_64-configure
Re-run cmake no build system arguments
-- The C compiler identification is GNU 8.2.0
-- The CXX compiler identification is GNU 8.2.0
...

I am creating symlinks at the start of the build:

cd "$BT_SRCD/llvm/tools"
ln -s ../../polly
ln -s ../../clang

cd "$BT_SRCD/llvm/projects"
ln -s "../../libcxx"
ln -s "../../libcxxabi"
ln -s "../../compiler-rt"

Are you possibly not building with libcxx and libcxxabi?

I managed to repro the issue:
/usr/bin/ld: projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/common.cc.o: file class ELFCLASS32 incompatible with ELFCLASS64
with your cmake parameters.
Trying to figure out what's up.

AFAICT, the compilation invocation that is failing should have a -m32 for it to succeed which is not there. The following succeeds (with the extra -m32):

.../llvm-build/gcc/bin/clang++ -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -Wall -std=c++11 -Wno-unused-parameter -g  -Wl,-z,defs -Wl,-z,nodelete -shared -Wl,-soname,libRTScudoStandalone.test.i386.so -o lib/libRTScudoStandalone.test.i386.so projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/common.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/fuchsia.cc.o projects/compiler-rt/lib/scudo/standalone/CMakeFiles/RTScudoStandalone.i386.dir/linux.cc.o  -m32

The -m32 seems to be added to the CMAKE_C_FLAGS variable but is not.
The different between my build that succeeds and this one boils down to -DCMAKE_C_COMPILER=$HOME/llvm-build/gcc/bin/clang vs CC=$HOME/llvm-build/gcc/bin/clang for me.
I am still not fully sure how to get around that.

I seemed to have fixed it locally:

add_library("RTScudoStandalone.test.${arch}" STATIC
  $<TARGET_OBJECTS:RTScudoStandalone.${arch}>)

The STATIC was missing. Let me know if that works for you.

I seemed to have fixed it locally:

add_library("RTScudoStandalone.test.${arch}" STATIC
  $<TARGET_OBJECTS:RTScudoStandalone.${arch}>)

The STATIC was missing. Let me know if that works for you.

Yes, that worked for me as well on Ubuntu. Thank you! I was deep in the bowels of recursive CMake macros when I noticed your update. I will also try this on RHEL 6.7 to see if it fixes it for clock_gettime(). I know you don't have that environment so I'll have to figure that one out if it doesn't work.

Yes, that worked for me as well on Ubuntu. Thank you! I was deep in the bowels of recursive CMake macros when I noticed your update. I will also try this on RHEL 6.7 to see if it fixes it for clock_gettime(). I know you don't have that environment so I'll have to figure that one out if it doesn't work.

Cool. Feel free to send it out if you want, or I'll do it when I am done with my current CL.
For the RHEL I thought linking against librt would do, but I don't have a way to debug it locally fast.

Cool. Feel free to send it out if you want, or I'll do it when I am done with my current CL.
For the RHEL I thought linking against librt would do, but I don't have a way to debug it locally fast.

Good news: the patch fixes the RHEL 6.7 build issues as well.

If you could push the patch I'd appreciate it. I can't get to it today or tomorrow, the soonest would probably be Monday. If that's the case for you too I can do so Monday AM.

lib/scudo/standalone/common.cc