This is an archive of the discontinued LLVM Phabricator instance.

[LLDB][RISCV]Add initial support for lldb-server.
AbandonedPublic

Authored by Emmmer on Jun 21 2022, 1:19 AM.

Details

Summary

I added some initial support for lldb-server on riscv64 platform, now it can be built on riscv64.
But when I use lldb for debugging, it reports:

error: 'A' packet returned an error: -1

I tried to fix this error with gdb, but the debugging process was too slow, so I'm sorry for not being able to provide a test patch for verification of my commit.

Diff Detail

Event Timeline

Emmmer created this revision.Jun 21 2022, 1:19 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 21 2022, 1:19 AM
Emmmer requested review of this revision.Jun 21 2022, 1:19 AM

I'll give this a more thorough read later just one comment for now.

In the meantime, what's your plan for testing this config going forward? This patch would be fine as the first of a series but I'd like to see the follow ups that make it usable.

Will there be a build bot? You could at least enable building lldb on an existing riscv bot, while you iron out all the inevitable test suite issues.

lldb/source/Plugins/Process/Linux/NativeRegisterContextLinux_riscv64.h
10

Should this be &&? I'm assuming that __riscv is also defined for riscv32.

I'll give this a more thorough read later just one comment for now.

In the meantime, what's your plan for testing this config going forward? This patch would be fine as the first of a series but I'd like to see the follow ups that make it usable.

Will there be a build bot? You could at least enable building lldb on an existing riscv bot, while you iron out all the inevitable test suite issues.

To use gdbserver by lldb requires the participation of lldb-server, so my current plan is to allow lldb to co-op with gdbserver, then let lldb be able to use lldb-server, pass the unit tests, and finally add riscv32 support.
We use Open Build Server as the riscv build bot.

Overall this looks fine. Without the ability to test it I don't think we would accept it yet, so I didn't look into the details too much.

To use gdbserver by lldb requires the participation of lldb-server, so my current plan is to allow lldb to co-op with gdbserver, then let lldb be able to use lldb-server, pass the unit tests, and finally add riscv32 support.

Sounds good. If those lldb changes can be tested by existing bots then they can go in first. For anything else it's probably better to build a tree of your changes and when you've got something that ends in a testable lldb-server then put them into review.

I appreciate it's difficult to test a lot of the enabling code e.g. the breakpoint code in this change. That'll be exercised by the test suite in general once more things are working. So it would be fine to say here is a series that ends in an lldb/lldb-server that can run a large amount of existing tests.

If you find anything non-riscv specific e.g. some gdb protocol difference that can go into review whenever. Anything to improve lldb-gdb compatibility is welcome.

We use Open Build Server as the riscv build bot.

I'm not familiar with this infrastructure. In a future where your changes are in llvm main will developers be notified when they break the riscv lldb build?

lldb/source/Plugins/Process/Linux/NativeRegisterContextLinux_riscv64.cpp
250
return GetRegisterInfo().GetRegisterSetFromRegisterIndex(reg) == RegisterInfoPOSIX_riscv64::GPRegSet;
257

Seems like this should do a lookup?

324

Minor thing but if you call this in the constructor it's one less place to = false all the things.

342

This also a placeholder I assume (I'm not sure what this does for other targets tbh).

346

This is a placeholder?

lldb/source/Plugins/Process/Utility/NativeRegisterContextDBReg_riscv64.cpp
19 ↗(On Diff #438581)

and PAC means? (please add it to the comment)

34 ↗(On Diff #438581)

I don't see an implementation of this yet, not one that actually reads anything. Intentional? It'll just see 0 breakpoints I guess.

lldb/source/Plugins/Process/Utility/RegisterInfoPOSIX_riscv64.cpp
52

So the way lldb-server works, at least at the moment, is that one binary does one architecture. So for example the AArch64 linux build doesn't handle AArch32 mode, you'd need to use an AArch32 built binary for that.

Maybe you can make this work with riscv but I'd start with just looking for riscv64 here.

lldb/source/Utility/RISCV64_DWARF_Registers.h
55

No DWARF numbers for the FPU regs?

I tried to fix this error with gdb, but the debugging process was too slow, so I'm sorry for not being able to provide a test patch for verification of my commit.

Oh and if you want help with this please ask on discord/discourse. This stuff is a bit tedious with so many exchanges on first connection.

Emmmer marked 2 inline comments as done.Jun 24 2022, 10:23 PM
Emmmer added inline comments.
lldb/source/Plugins/Process/Linux/NativeRegisterContextLinux_riscv64.h
10

WIP

Emmmer updated this revision to Diff 439955.Jun 24 2022, 10:52 PM
Emmmer marked an inline comment as done.

This patch change:

  • add lldb/source/Plugins/Architecture/RISCV64
  • add lldb/source/Plugins/Architecture/RISCV32
  • update lldb/source/Utility/ArchSpec.cpp
  • remove lldb/source/Plugins/Process/Utility/NativeRegisterContextDBReg_riscv64

No DWARF numbers for the FPU regs?

I would like to add support for GPR first, after making sure they work well, FPU regs are a simple copy-paste.

AArch64 linux build doesn't handle AArch32 mode, you'd need to use an AArch32 built binary for that.

Thanks for your suggestion, I split them out.

and PAC means? (please add it to the comment)
I don't see an implementation of this yet, not one that actually reads anything. Intentional? It'll just see 0 breakpoints I guess.

At first, I referenced the AArch64 code to add riscv64 support because their arch is similar, but I overlooked that riscv64 doesn't have a Debug reg yet, so it was a mistake on my part.

I try to build lldb with the patch, but fail:

The environment:

cat /proc/cpuinfo 
processor       : 0
hart            : 2
isa             : rv64imafdc
mmu             : sv39
uarch           : sifive,u74-mc

llvm commit: d1b098fc

Command: cmake -DLLVM_TARGETS_TO_BUILD="RISCV" -DLLVM_ENABLE_PROJECTS="lldb" -DCMAKE_BUILD_TYPE="Release" -G Ninja ../llvm

Error:

CMake Error at /home/liaochunyu/llvm-project/lldb/cmake/modules/LLDBConfig.cmake:289 (message):
  Expected directory for clang-resource headers not found:
Call Stack (most recent call first):
  /home/liaochunyu/llvm-project/lldb/CMakeLists.txt:28 (include)


-- Configuring incomplete, errors occurred!

I try to build lldb with the patch, but fail:

The environment:

cat /proc/cpuinfo 
processor       : 0
hart            : 2
isa             : rv64imafdc
mmu             : sv39
uarch           : sifive,u74-mc

llvm commit: d1b098fc

Command: cmake -DLLVM_TARGETS_TO_BUILD="RISCV" -DLLVM_ENABLE_PROJECTS="lldb" -DCMAKE_BUILD_TYPE="Release" -G Ninja ../llvm

Error:

CMake Error at /home/liaochunyu/llvm-project/lldb/cmake/modules/LLDBConfig.cmake:289 (message):
  Expected directory for clang-resource headers not found:
Call Stack (most recent call first):
  /home/liaochunyu/llvm-project/lldb/CMakeLists.txt:28 (include)


-- Configuring incomplete, errors occurred!

The most convenient way to handle clang dependencies is adding clang to LLVM_ENABLE_PROJECTS

-DLLVM_ENABLE_PROJECTS="lldb;clang"

I try to build lldb with the patch, but fail:

The environment:

cat /proc/cpuinfo 
processor       : 0
hart            : 2
isa             : rv64imafdc
mmu             : sv39
uarch           : sifive,u74-mc

llvm commit: d1b098fc

Command: cmake -DLLVM_TARGETS_TO_BUILD="RISCV" -DLLVM_ENABLE_PROJECTS="lldb" -DCMAKE_BUILD_TYPE="Release" -G Ninja ../llvm

Error:

CMake Error at /home/liaochunyu/llvm-project/lldb/cmake/modules/LLDBConfig.cmake:289 (message):
  Expected directory for clang-resource headers not found:
Call Stack (most recent call first):
  /home/liaochunyu/llvm-project/lldb/CMakeLists.txt:28 (include)


-- Configuring incomplete, errors occurred!

The most convenient way to handle clang dependencies is adding clang to LLVM_ENABLE_PROJECTS

-DLLVM_ENABLE_PROJECTS="lldb;clang"

without the patch, no error

FWIW I do get the same error in a clean directory without this patch applied:

-- LLDB version: 15.0.0git
CMake Error at /home/david.spickett/llvm-project/lldb/cmake/modules/LLDBConfig.cmake:289 (message):
  Expected directory for clang-resource headers not found:
Call Stack (most recent call first):
  /home/david.spickett/llvm-project/lldb/CMakeLists.txt:28 (include)

There's probably a better way to tell the user what to do but that's how it works at the moment. The "standard" set of projects to enable is lld, clang and lldb if you want to build and test lldb.
(there is a standalone build https://lldb.llvm.org/resources/build.html#standalone-builds but it works by having clang already built, so same thing with extra steps)

Thanks all.

Command: -DLLVM_ENABLE_PROJECTS="clang;lldb;lld" , it works.

labath added a subscriber: labath.Jun 30 2022, 3:08 AM

I'll give this a more thorough read later just one comment for now.

In the meantime, what's your plan for testing this config going forward? This patch would be fine as the first of a series but I'd like to see the follow ups that make it usable.

Will there be a build bot? You could at least enable building lldb on an existing riscv bot, while you iron out all the inevitable test suite issues.

To use gdbserver by lldb requires the participation of lldb-server, so my current plan is to allow lldb to co-op with gdbserver, then let lldb be able to use lldb-server, pass the unit tests, and finally add riscv32 support.

We do have a separate test suite just for lldb-server, so you might be able to use that to gauge your progress without worrying about the lldb side too much.

I wouldn't say that the presence of a buildbot is an absolute requirement to get this off the ground, but I would strongly recommend setting up one, even if it's just building for now. We don't e.g. have a bot for SystemZ, and the overall effect of that is that nobody knows whether SystemZ debugging actually works.

lldb/source/Plugins/Architecture/RISCV64/ArchitectureRISCV64.cpp
2

same here

lldb/source/Plugins/Architecture/RISCV64/ArchitectureRISCV64.h
2

Please use the standard llvm header with licence info and stuff.

Emmmer updated this revision to Diff 442006.Jul 3 2022, 11:31 PM
  • Format the standard llvm header.
  • add RegisterContextPOSIX_riscv64.cpp and RegisterContextPOSIX_riscv64.h
  • add Floating-Point Register support.

I wouldn't say that the presence of a buildbot is an absolute requirement to get this off the ground, but I would strongly recommend setting up one, even if it's just building for now.

I currently don't quite understand how to configure the build bot, and how to set the test configuration, I went through this article but didn't find useful information for me, could someone help me.

Emmmer marked 8 inline comments as done.Jul 3 2022, 11:33 PM

Minor thing but if you call this in the constructor it's one less place to = false all the things.

It is a virtual function so I decided not to call it from the constructor.

I currently don't quite understand how to configure the build bot, and how to set the test configuration, I went through this article but didn't find useful information for me, could someone help me.

Start with https://llvm.org/docs/HowToAddABuilder.html instead. Though it may be better to propose adding the lldb-server build to an existing riscv buildbot, which would be a case of reaching out to the owner and/or proposing a change on https://github.com/llvm/llvm-zorg to add the build target to the bot's config.

Emmmer updated this revision to Diff 443482.Jul 10 2022, 12:58 AM

This patch change:

  • Add the recognition of architecture riscv64 in HostInfoBase.cpp
  • Add the recognition of architecture riscv64 and riscv32 in ObjectFilePECOFF.cpp
  • Add riscv's ebreak command to Platform.cpp

Now lldb can debug with simple executables on qemu-system-riscv64 and be able to attach to a gdbserver.


TODO: some unittest failed

bash
[ RUN      ] TestBase.LaunchModePreservesEnvironment
/home/emmmer/git/llvm-project/lldb/unittests/tools/lldb-server/tests/LLGSTest.cpp:29: Failure
Value of: llvm::detail::TakeExpected(ClientOr)
Expected: succeeded
  Actual: failed  (Unable to parse qRegisterInfo: generic)
[  FAILED  ] TestBase.LaunchModePreservesEnvironment (662 ms)


[ RUN      ] TestBase.vAttachRichError
Connection established.
Launched '/home/emmmer/git/llvm-project/build-cross/tools/lldb/unittests/tools/lldb-server/./environment_check' as process 1553...
lldb-server-local_build
Connection established.
Launched '/home/emmmer/git/llvm-project/build-cross/tools/lldb/unittests/tools/lldb-server/./environment_check' as process 1556...
lldb-server-local_build
/home/emmmer/git/llvm-project/lldb/unittests/tools/lldb-server/tests/LLGSTest.cpp:60: Failure
Value of: llvm::detail::TakeExpected(ClientOr)
Expected: succeeded
  Actual: failed  (Unable to parse qRegisterInfo: generic)
[  FAILED  ] TestBase.vAttachRichError (364 ms)

In riscv, the user-mode process cannot directly take the pc register but must obtain the pc state through auipc, while the pc register is required to exist in the test, so it failed.


bash
[ RUN      ] DumpDataExtractorTest.Formats
/home/emmmer/git/llvm-project/lldb/unittests/Core/DumpDataExtractorTest.cpp:90: Failure
Expected equality of these values:
  expected
    Which is: "{-nan -nan nan nan}"
  result.GetString()
    Which is: "{nan nan nan nan}"
[  FAILED  ] DumpDataExtractorTest.Formats (25 ms)

The reason is currently unknown, and further verification is required


About buildbot: Unfortunately, as an individual developer, I may not have the ability to maintain a 7*24-hour compile server or even a cluster, but I will do my best to provide some test reports.

Emmmer updated this revision to Diff 445699.Jul 18 2022, 11:11 PM

This commit updates:

The NaN problem has been solved by D129750

At this point, we can pass all LLDBUnitTest.

It may be challenging to review and merge at one time for such a large patch. I would like to ask if it‘s necessary to split this patch and merge them in turn.

DavidSpickett added inline comments.Jul 25 2022, 7:57 AM
lldb/source/Plugins/Architecture/RISCV64/ArchitectureRISCV64.cpp
23

Just noticed this in passing, should this and the 32 bit one be named like "RISCV-64"/"RISCV-32" instead?

tzb99 added a subscriber: tzb99.Jul 25 2022, 9:34 AM

This patch change:

  • Add the recognition of architecture riscv64 in HostInfoBase.cpp
  • Add the recognition of architecture riscv64 and riscv32 in ObjectFilePECOFF.cpp
  • Add riscv's ebreak command to Platform.cpp

Now lldb can debug with simple executables on qemu-system-riscv64 and be able to attach to a gdbserver.


TODO: some unittest failed

bash
[ RUN      ] TestBase.LaunchModePreservesEnvironment
/home/emmmer/git/llvm-project/lldb/unittests/tools/lldb-server/tests/LLGSTest.cpp:29: Failure
Value of: llvm::detail::TakeExpected(ClientOr)
Expected: succeeded
  Actual: failed  (Unable to parse qRegisterInfo: generic)
[  FAILED  ] TestBase.LaunchModePreservesEnvironment (662 ms)


[ RUN      ] TestBase.vAttachRichError
Connection established.
Launched '/home/emmmer/git/llvm-project/build-cross/tools/lldb/unittests/tools/lldb-server/./environment_check' as process 1553...
lldb-server-local_build
Connection established.
Launched '/home/emmmer/git/llvm-project/build-cross/tools/lldb/unittests/tools/lldb-server/./environment_check' as process 1556...
lldb-server-local_build
/home/emmmer/git/llvm-project/lldb/unittests/tools/lldb-server/tests/LLGSTest.cpp:60: Failure
Value of: llvm::detail::TakeExpected(ClientOr)
Expected: succeeded
  Actual: failed  (Unable to parse qRegisterInfo: generic)
[  FAILED  ] TestBase.vAttachRichError (364 ms)

In riscv, the user-mode process cannot directly take the pc register but must obtain the pc state through auipc, while the pc register is required to exist in the test, so it failed.


bash
[ RUN      ] DumpDataExtractorTest.Formats
/home/emmmer/git/llvm-project/lldb/unittests/Core/DumpDataExtractorTest.cpp:90: Failure
Expected equality of these values:
  expected
    Which is: "{-nan -nan nan nan}"
  result.GetString()
    Which is: "{nan nan nan nan}"
[  FAILED  ] DumpDataExtractorTest.Formats (25 ms)

The reason is currently unknown, and further verification is required


About buildbot: Unfortunately, as an individual developer, I may not have the ability to maintain a 7*24-hour compile server or even a cluster, but I will do my best to provide some test reports.

Hello:

I implemented the diff into my local llvm project and cross-compiled the project using in-tree build with enable projects as:
-DLLVM_ENABLE_PROJECTS="clang;lld;lldb"

The project can be compiled using the "Release" mode. The lldb-server can be initiated and connected to the host machine from the qemu environment. It can ran the riscv binary using process continue / thread continue, but the compiled lldb-server cannot get any thread information and cannot perform thread step-in functionality. Then I performed the Debug mode to build the project. Error occurred so the build command cannot be finished. The error shows like:

[3759/4081] Linking CXX shared library lib/libclang-cpp.so.15git
FAILED: lib/libclang-cpp.so.15git
: && riscv-gnu-toolchain/bin/riscv64-unknown-linux-gnu-g++ -

Can your lldb-server work properly on the riscv qemu environment? My question might be, what is the proper recipe for cross-building the lldb-server? Or, should the diff be changed to enable getting the thread instruction info of the lldb-server?

Thank you very much!

tzb99 added a comment.Jul 25 2022, 2:13 PM

This patch change:

  • Add the recognition of architecture riscv64 in HostInfoBase.cpp
  • Add the recognition of architecture riscv64 and riscv32 in ObjectFilePECOFF.cpp
  • Add riscv's ebreak command to Platform.cpp

Now lldb can debug with simple executables on qemu-system-riscv64 and be able to attach to a gdbserver.


TODO: some unittest failed

bash
[ RUN      ] TestBase.LaunchModePreservesEnvironment
/home/emmmer/git/llvm-project/lldb/unittests/tools/lldb-server/tests/LLGSTest.cpp:29: Failure
Value of: llvm::detail::TakeExpected(ClientOr)
Expected: succeeded
  Actual: failed  (Unable to parse qRegisterInfo: generic)
[  FAILED  ] TestBase.LaunchModePreservesEnvironment (662 ms)


[ RUN      ] TestBase.vAttachRichError
Connection established.
Launched '/home/emmmer/git/llvm-project/build-cross/tools/lldb/unittests/tools/lldb-server/./environment_check' as process 1553...
lldb-server-local_build
Connection established.
Launched '/home/emmmer/git/llvm-project/build-cross/tools/lldb/unittests/tools/lldb-server/./environment_check' as process 1556...
lldb-server-local_build
/home/emmmer/git/llvm-project/lldb/unittests/tools/lldb-server/tests/LLGSTest.cpp:60: Failure
Value of: llvm::detail::TakeExpected(ClientOr)
Expected: succeeded
  Actual: failed  (Unable to parse qRegisterInfo: generic)
[  FAILED  ] TestBase.vAttachRichError (364 ms)

In riscv, the user-mode process cannot directly take the pc register but must obtain the pc state through auipc, while the pc register is required to exist in the test, so it failed.


bash
[ RUN      ] DumpDataExtractorTest.Formats
/home/emmmer/git/llvm-project/lldb/unittests/Core/DumpDataExtractorTest.cpp:90: Failure
Expected equality of these values:
  expected
    Which is: "{-nan -nan nan nan}"
  result.GetString()
    Which is: "{nan nan nan nan}"
[  FAILED  ] DumpDataExtractorTest.Formats (25 ms)

The reason is currently unknown, and further verification is required


About buildbot: Unfortunately, as an individual developer, I may not have the ability to maintain a 7*24-hour compile server or even a cluster, but I will do my best to provide some test reports.

Hello:

I implemented the diff into my local llvm project and cross-compiled the project using in-tree build with enable projects as:
-DLLVM_ENABLE_PROJECTS="clang;lld;lldb"

The project can be compiled using the "Release" mode. The lldb-server can be initiated and connected to the host machine from the qemu environment. It can ran the riscv binary using process continue / thread continue, but the compiled lldb-server cannot get any thread information and cannot perform thread step-in functionality. Then I performed the Debug mode to build the project. Error occurred so the build command cannot be finished. The error shows like:

[3759/4081] Linking CXX shared library lib/libclang-cpp.so.15git
FAILED: lib/libclang-cpp.so.15git
: && riscv-gnu-toolchain/bin/riscv64-unknown-linux-gnu-g++ -

Can your lldb-server work properly on the riscv qemu environment? My question might be, what is the proper recipe for cross-building the lldb-server? Or, should the diff be changed to enable getting the thread instruction info of the lldb-server?

Thank you very much!

One more detail about the ThreadInfo: This is the strace of my lldb-server runing on the qemu:

recvfrom(9, "$jThreadsInfo#c1", 8192, 0, NULL, NULL) = 16
gettid() = 353
openat(AT_FDCWD, "/proc/354/task/354/comm", O_RDONLY|O_CLOEXEC) = 8
read(8, "riscvv\n", 16384) = 7
read(8, "", 16384) = 0
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [HUP CHLD], 8) = 0
close(8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP CHLD], NULL, 8) = 0
sendto(9, "$[{\"name\":\"riscvv\",\"reason\":\"sig"..., 64, 0, NULL, 0) = 64
gettid() = 353
pselect6(10, [6 9], NULL, NULL, {tv_sec=0, tv_nsec=0}, NULL) = 1 (in [9], left {tv_sec=0, tv_nsec=0})
recvfrom(9, "$jThreadExtendedInfo:#b9", 8192, 0, NULL, NULL) = 24
gettid() = 353

tzb99 added a comment.Aug 3 2022, 10:02 PM

This commit updates:

The NaN problem has been solved by D129750

At this point, we can pass all LLDBUnitTest.

It may be challenging to review and merge at one time for such a large patch. I would like to ask if it‘s necessary to split this patch and merge them in turn.

Hello:

Thank you so much for sharing the patch files. One thing I am still curious is what the ABISysv file you are using. Would you mind also sharing the remaining patches added for the lldb-server support?

Hello:

Thank you so much for sharing the patch files. One thing I am still curious is what the ABISysv file you are using. Would you mind also sharing the remaining patches added for the lldb-server support?

I didn't add any ABISysV code before uploading this patch, and now I'm struggling with the breakpoints not hit problem, and it may be related to ABISysV.
After I solve this problem, I would love to share all patches added for the lldb-server support.

tzb99 added a comment.Aug 4 2022, 9:59 AM

Hello:

Thank you so much for sharing the patch files. One thing I am still curious is what the ABISysv file you are using. Would you mind also sharing the remaining patches added for the lldb-server support?

I didn't add any ABISysV code before uploading this patch, and now I'm struggling with the breakpoints not hit problem, and it may be related to ABISysV.
After I solve this problem, I would love to share all patches added for the lldb-server support.

Cool! There is one thread talking about the RISCV ABI support: https://reviews.llvm.org/D62732. I tried that ABI in the RISCV file but it seems like the same issue happened as yours, the breakpoint cannot be stopped, and the lldb-server cannot show the reasonable stack frame pointer address. So I think the ABI support should be modified.

Emmmer added a comment.Aug 9 2022, 7:38 AM

Cool! There is one thread talking about the RISCV ABI support: https://reviews.llvm.org/D62732. I tried that ABI in the RISCV file but it seems like the same issue happened as yours, the breakpoint cannot be stopped, and the lldb-server cannot show the reasonable stack frame pointer address. So I think the ABI support should be modified.

The ABI support you mentioned does not relate to breakpoint mechanism in lldb-server (but it may be required by other features). lldb-server failed to add breakpoints due to the lack of trap code implementation in NativeProcessProtocol::EnableSoftwareBreakpoint. You can checkout the new revision in D130342 which successfully set the breakpoint on a statically linked executable.

Emmmer abandoned this revision.Aug 10 2022, 11:38 PM

Done in D130342 and D131566