This is an archive of the discontinued LLVM Phabricator instance.

[TSAN] add support for riscv64
ClosedPublic

Authored by alexfanqi on Mar 2 2023, 8:54 PM.

Details

Summary

Implements for sv39 and sv48 VMA layout.

Userspace only has access to the bottom half of vma range. The top half is used by kernel.
There is no dedicated vsyscall or heap segment.
PIE program is allocated to start at TASK_SIZE/3*2. Maximum ASLR is ARCH_MMAP_RND_BITS_MAX+PAGE_SHIFT=24+12=36
Loader, vdso and other libraries are allocated below stack from the top.

Also change RestoreAddr to use 4 bits to accommodate MappingRiscv64_48

Diff Detail

Event Timeline

alexfanqi created this revision.Mar 2 2023, 8:54 PM
Herald added a project: Restricted Project. · View Herald TranscriptMar 2 2023, 8:54 PM
alexfanqi updated this revision to Diff 502351.Mar 3 2023, 10:33 PM

fix tsan_rtl_riscv64.S and test failures

remaining test failure:
ThreadSanitizer-riscv64 :: custom_mutex5.cpp
The debuginfo line reported is slightly off.
custom_mutex5.cpp:14 -> 15
custom_mutex5.cpp:22 -> 23

ThreadSanitizer-riscv64 :: mmap_lots.cpp
flaky, the last 1000 sized mmap can get allocated to 0x0000-0x1000

alexfanqi retitled this revision from [TSAN] add RISCV support to [TSAN] add support for riscv64.Mar 3 2023, 10:54 PM
alexfanqi edited the summary of this revision. (Show Details)
alexfanqi added reviewers: MaskRay, dvyukov, asb, StephenFan.
alexfanqi published this revision for review.Mar 3 2023, 11:16 PM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptMar 3 2023, 11:16 PM
Herald added subscribers: Restricted Project, cfe-commits, pcwang-thead, eopXD. · View Herald Transcript
jrtc27 added inline comments.Mar 3 2023, 11:24 PM
compiler-rt/CMakeLists.txt
529 ↗(On Diff #502351)

This is only to work around a long-standing bug in GCC's atomics implementation for RISC-V. If you build compiler-rt with Clang you don't need this, and in theory future versions of GCC (possibly trunk already fixes this, I'm not sure of the status of it).

compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp
60 ↗(On Diff #502351)

?

compiler-rt/lib/tsan/rtl/tsan_platform.h
971

?

compiler-rt/lib/tsan/rtl/tsan_platform_linux.cpp
271–272

Why don't other architectures need the +1?

compiler-rt/lib/tsan/tests/CMakeLists.txt
59 ↗(On Diff #502351)

Same as before

compiler-rt/test/tsan/test.h
80

What's this the length of? RVC is optional.

alexfanqi updated this revision to Diff 502397.Mar 4 2023, 7:01 PM

remove libatomic link

alexfanqi added inline comments.Mar 4 2023, 7:04 PM
compiler-rt/lib/tsan/rtl/tsan_platform.h
971

This is changed for sv48. Tsan compresses the address to 44 bits and uses the top 3 bits (42-44) to uncompress it by comparing with the corresponding bits in the mapping. So the 42-44th bits of each app mapping range must be all different from each other. But 3 bits are not enough to distinguish mappings in this scheme of MappingRiscv64_48, so I increment it to 4 bits. An alternative of this would be reducing app mapping size by half.

compiler-rt/lib/tsan/rtl/tsan_platform_linux.cpp
271–272

I am not too sure about other architectures. Perhaps, because riscv kernel divides vma range further by 2. https://elixir.bootlin.com/linux/latest/source/arch/riscv/include/asm/pgtable.h#L785
By https://docs.kernel.org/riscv/vm-layout.html, botton half is exposed for userspace, top half is for kernel.
The available userspace virtual address size for arm64 with same bit size is exactly double of riscv. For example, 39bit -> 0x4000000000(riscv) vs 0x8000000000(arm64)

compiler-rt/test/tsan/test.h
80

The same length as in GetPreviousInstructionPc. Without it, I got java_race_pc, race_range_pc test failure.

alexfanqi edited the summary of this revision. (Show Details)Mar 4 2023, 7:07 PM

friendly ping?

pirama added a subscriber: pirama.Jun 7 2023, 11:17 PM

@alexfanqi are you still actively working on this? maybe we can move it to github then.

@alexfanqi are you still actively working on this? maybe we can move it to github then.

Yeah, but the patch is already complete and tsan functions well with tests passing on my local qemu system emulation. I am waiting for reviewers to have more feedback or accept this patch.

evandro removed a subscriber: evandro.Sep 15 2023, 4:19 PM

ping: @jrtc27 @asb do you have any pending feedback for this patch?

hiraditya added inline comments.Sep 19 2023, 1:30 PM
compiler-rt/lib/tsan/rtl/CMakeLists.txt
224

I'm getting error with this line for some reason. Unknown cmake command "add_asm_sources" see: D152102

hiraditya added inline comments.Sep 20 2023, 2:17 PM
compiler-rt/lib/tsan/rtl/CMakeLists.txt
224

replace add_asm_sources with set and it will work.

hiraditya accepted this revision.EditedSep 22 2023, 2:26 PM

I tested it on a licheepi board (riscv64 debian) and all the tests modified in this patch have passed. LGTM.

There are other tsan failures but i think it is okay if they can be fixed in subsequent patches.

This revision is now accepted and ready to land.Sep 22 2023, 2:26 PM
alexfanqi added a comment.EditedSep 26 2023, 7:22 AM

I tested it on a licheepi board (riscv64 debian) and all the tests modified in this patch have passed. LGTM.

Can you help me land this patch? I used to have commit access, but seems not working atm.

There are other tsan failures but i think it is okay if they can be fixed in subsequent patches.

Do you have a list of these failures? I will try to reproduce on my machine and fix in later patch.

I tested it on a licheepi board (riscv64 debian) and all the tests modified in this patch have passed. LGTM.

Can you help me land this patch? I used to have commit access, but seems not working atm.

There are other tsan failures but i think it is okay if they can be fixed in subsequent patches.

Do you have a list of these failures? I will try to reproduce on my machine and fix in later patch.

While testing compiler-rt/test/tsan/RISCV64Config only compiler-rt/test/tsan/mmap_lots.cpp failed. I couldn't run all the compiler-rt tests on Licheepi because it is very slow.

While testing compiler-rt/test/tsan/RISCV64Config only compiler-rt/test/tsan/mmap_lots.cpp failed. I couldn't run all the compiler-rt tests on Licheepi because it is very slow

I had this only test failure before. When the upper memory space is fully mapped out, the kernel will try to use the lower memory range starting from 0x00, but kLoAppMem leaves out the initial 0x00-0x1000. I saw other archs did the same, and maybe it won't hurt.

MaskRay added inline comments.Sep 26 2023, 3:20 PM
compiler-rt/lib/tsan/rtl/tsan_platform.h
971
vitalybuka added 1 blocking reviewer(s): dvyukov.Sep 26 2023, 3:50 PM
This revision now requires review to proceed.Sep 26 2023, 3:50 PM
vitalybuka accepted this revision.Sep 26 2023, 3:52 PM
vitalybuka added a subscriber: vitalybuka.
vitalybuka added inline comments.
compiler-rt/lib/tsan/rtl/CMakeLists.txt
5

Maybe this one is not needed after b31bd6d8046d01a66aa92993bacb56b115a67fc5

MaskRay accepted this revision.Sep 26 2023, 3:59 PM

Other than -Wframe-larger-than=656 and const uptr indicator = 0x0f0000000000ull; changes, others looks good to me.
I agree that this should be reviewed by @dvyukov :)

Another test that failed:
compiler-rt/test/tsan/signal_thread.cpp

dvyukov accepted this revision.Sep 28 2023, 9:26 PM
dvyukov added inline comments.
compiler-rt/lib/tsan/rtl/CMakeLists.txt
5

Yes, is this needed? What function does have larger frame?
If we increase the limit, other arches will slowly slinetly degrade too.

This revision is now accepted and ready to land.Sep 28 2023, 9:26 PM
hiraditya added inline comments.Oct 2 2023, 1:26 PM
compiler-rt/lib/tsan/rtl/CMakeLists.txt
5

yeah probably not needed anymore. but if we need this we can just change this only for RISC-V?

jrtc27 added inline comments.Oct 2 2023, 2:22 PM
compiler-rt/lib/tsan/rtl/CMakeLists.txt
5

I would rather diagnose and fix whatever makes RISC-V code generation less efficient

hiraditya added inline comments.Oct 6 2023, 12:30 PM
compiler-rt/lib/tsan/rtl/CMakeLists.txt
5

without this change we have one less failure. so i'll discard the changes to -Wframe-larger-than=656 and commit the patch.

haowei added a subscriber: haowei.Oct 6 2023, 2:24 PM

Hi,

I believe you forgot to add tsan_rtl_riscv64.S to your patch when you land it. We are seeing build errors on our bots with following messages:

-- Configuring done (12.3s)
CMake Error at /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/cmake/Modules/AddCompilerRT.cmake:357 (add_library):
  Cannot find source file:

    tsan_rtl_riscv64.S
Call Stack (most recent call first):
  /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/lib/tsan/rtl/CMakeLists.txt:244 (add_compiler_rt_runtime)


CMake Error at /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/cmake/Modules/AddCompilerRT.cmake:357 (add_library):
  No SOURCES given to target: clang_rt.tsan-riscv64
Call Stack (most recent call first):
  /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/lib/tsan/rtl/CMakeLists.txt:244 (add_compiler_rt_runtime)


CMake Error at /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/cmake/Modules/AddCompilerRT.cmake:357 (add_library):
  No SOURCES given to target: clang_rt.tsan-dynamic-riscv64
Call Stack (most recent call first):
  /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/lib/tsan/rtl/CMakeLists.txt:268 (add_compiler_rt_runtime)


CMake Generate step failed.  Build files cannot be regenerated correctly.
[5508/5672](1) Completed 'runtimes-i386-unknown-linux-gnu'
FAILED: runtimes/runtimes-riscv64-unknown-linux-gnu-stamps/runtimes-riscv64-unknown-linux-gnu-configure /b/s/w/ir/x/w/llvm_build/runtimes/runtimes-riscv64-unknown-linux-gnu-stamps/runtimes-riscv64-unknown-linux-gnu-configure 
cd /b/s/w/ir/x/w/llvm_build/runtimes/runtimes-riscv64-unknown-linux-gnu-bins && /b/s/w/ir/x/w/cipd_tool/fuchsia/third_party/cmake/72c014f0086372a2bab7ed771e2ed942d6122d94331d19c95566bb25d2f962ff/bin/cmake --no-warn-unused-cli -DCMAKE_C_COMPILER=/b/s/w/ir/x/w/llvm_build/./bin/clang -DCMAKE_CXX_COMPILER=/b/s/w/ir/x/w/llvm_build/./bin/clang++ -DCMAKE_ASM_COMPILER=/b/s/w/ir/x/w/llvm_build/./bin/clang -DCMAKE_LINKER=/b/s/w/ir/x/w/llvm_build/./bin/ld.lld -DCMAKE_AR=/b/s/w/ir/x/w/llvm_build/./bin/llvm-ar -DCMAKE_RANLIB=/b/s/w/ir/x/w/llvm_build/./bin/llvm-ranlib -DCMAKE_NM=/b/s/w/ir/x/w/llvm_build/./bin/llvm-nm -DCMAKE_OBJDUMP=/b/s/w/ir/x/w/llvm_build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/b/s/w/ir/x/w/llvm_build/./bin/llvm-objcopy -DCMAKE_STRIP=/b/s/w/ir/x/w/llvm_build/./bin/llvm-strip -DCMAKE_READELF=/b/s/w/ir/x/w/llvm_build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=riscv64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=riscv64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=riscv64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX= -DCMAKE_SYSROOT=/b/s/w/ir/x/w/cipd/linux -DLLVM_BINARY_DIR=/b/s/w/ir/x/w/llvm_build -DLLVM_CONFIG_PATH=/b/s/w/ir/x/w/llvm_build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=ON "-DLLVM_LIT_ARGS=--resultdb-output=r.j -v" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=18.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/b/s/w/ir/x/w/cipd_tool/fuchsia/third_party/ninja/476febc43233af3534d14a045651a4f365106e894b0998df28fa241d68733c8d/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=OFF -DLLVM_INCLUDE_TESTS=ON -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON -DLLVM_RUNTIMES_TARGET=riscv64-unknown-linux-gnu -DHAVE_LLVM_LIT=ON -DLLVM_DEFAULT_TARGET_TRIPLE=riscv64-unknown-linux-gnu "-DLLVM_ENABLE_RUNTIMES=compiler-rt;libcxx;libcxxabi;libunwind" -DLLVM_USE_LINKER=lld -DCMAKE_ASM_FLAGS=--target=riscv64-unknown-linux-gnu -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_FLAGS=--target=riscv64-unknown-linux-gnu -DCMAKE_C_FLAGS=--target=riscv64-unknown-linux-gnu -DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=lld -DCMAKE_MODULE_LINKER_FLAGS=-fuse-ld=lld -DCMAKE_SHARED_LINKER_FLAGS=-fuse-ld=lld -DCMAKE_SYSROOT=/b/s/w/ir/x/w/cipd/focal -DCMAKE_SYSTEM_NAME=Linux -DCOMPILER_RT_BUILD_STANDALONE_LIBATOMIC=ON -DCOMPILER_RT_CAN_EXECUTE_TESTS=ON -DCOMPILER_RT_CXX_LIBRARY=libcxx -DCOMPILER_RT_USE_BUILTINS_LIBRARY=ON -DCOMPILER_RT_USE_LLVM_UNWINDER=ON -DLIBCXXABI_ENABLE_SHARED=OFF -DLIBCXXABI_INSTALL_LIBRARY=OFF -DLIBCXXABI_USE_COMPILER_RT=ON -DLIBCXXABI_USE_LLVM_UNWINDER=ON -DLIBCXX_ABI_VERSION=2 -DLIBCXX_ENABLE_SHARED=OFF -DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON -DLIBCXX_USE_COMPILER_RT=ON -DLIBUNWIND_ENABLE_SHARED=OFF -DLIBUNWIND_USE_COMPILER_RT=ON -DLLVM_ENABLE_ASSERTIONS=OFF "-DLLVM_ENABLE_RUNTIMES=compiler-rt;libcxx;libcxxabi;libunwind" -DLLVM_TOOLS_DIR=/b/s/w/ir/x/w/llvm_build/bin -DSANITIZER_CXX_ABI=libc++ -DSANITIZER_CXX_ABI_INTREE=ON -DSANITIZER_TEST_CXX=libc++ -DSANITIZER_TEST_CXX_INTREE=ON -GNinja -S /b/s/w/ir/x/w/llvm-llvm-project/llvm/runtimes/../../runtimes -B /b/s/w/ir/x/w/llvm_build/runtimes/runtimes-riscv64-unknown-linux-gnu-bins && /b/s/w/ir/x/w/cipd_tool/fuchsia/third_party/cmake/72c014f0086372a2bab7ed771e2ed942d6122d94331d19c95566bb25d2f962ff/bin/cmake -E touch /b/s/w/ir/x/w/llvm_build/runtimes/runtimes-riscv64-unknown-linux-gnu-stamps/runtimes-riscv64-unknown-linux-gnu-configure
ninja: build stopped: subcommand failed.

Failed build task: https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64-rbe/b8767953001421733361/overview
Full stdout/err: https://logs.chromium.org/logs/fuchsia/buildbucket/cr-buildbucket/8767953001421733361/+/u/clang/build/stdout

Could you either add this file or revert your change please. Thanks.

@alexfanqi can you please upload the file (tsan_rtl_riscv64.S) with the patch?

nvm i found the file in one of the previous commits. https://github.com/llvm/llvm-project/pull/68735 for review.