[Android] Use -l:libunwind.a with --rtlib=compiler-rt

[rprichard]

Summary

On Android, the unwinder isn't part of the C++ STL and isn't (in older
versions) exported from libc.so. Instead, the driver links the static
unwinder archive implicitly. Currently, the Android NDK implicitly
links libgcc.a to provide both builtins and the unwinder.

To support switching to compiler-rt builtins and libunwind, make
--rtlib=compiler-rt behave the same way on Android, and implicitly pass
-l:libunwind.a to the linker.

Adjust the -ldl logic. For the Android NDK, the unwinder (whether
libgcc.a or libunwind.a) is linked statically and calls a function in
the dynamic loader for finding unwind tables (e.g. dl_iterate_phdr).
On Android, this function is in libc.a for static executables and
libdl.so otherwise, so -ldl is needed. (glibc doesn't need -ldl because
its libc.so exports dl_iterate_phdr.)

[rprichard]

For reference, I know of four ways that LLVM's libunwind is currently linked into programs:

• Fuchsia/Windows (i.e. Clang's default behavior): it's part of libc++, e.g.:
  • Fuchsia:
    • fuchsia/prebuilt/third_party/clang/linux-x64/lib/x86_64-unknown-fuchsia/c++/libc++.so is INPUT(libc++.so.2 -lunwind -lc++abi).
    • fuchsia/prebuilt/third_party/clang/linux-x64/lib/x86_64-unknown-fuchsia/c++/libc++.a is an archive that contains the libunwind object files.
  • In https://github.com/mstorsjo/llvm-mingw/releases/tag/20201020:
    • llvm-mingw\x86_64-w64-mingw32\bin\libc++.dll uses _Unwind_Resume from libunwind.dll
    • llvm-mingw\x86_64-w64-mingw32\lib\libc++.dll.a also seems to import _Unwind_Resume from libunwind.dll
    • llvm-mingw\x86_64-w64-mingw32\lib\libc++.a has the libunwind object files in it
• FreeBSD: libclang_rt.builtins and libunwind are repackaged into libraries that look like libgcc, linked implicitly.
• Darwin: it's part of libSystem, linked implicitly.
• crosstool in google3: I think this build system is passing --unwindlib=libunwind to the driver.

The Android team's LLVM build is used to target several OS's: host/glibc Linux, Windows, Darwin, and Android. At some point, we'd also like Android NDK to rely on the unwinder exported from libc.so (when the API level is high enough), which prevents us from using Clang's CLANG_DEFAULT_UNWINDLIB config setting.

Even for C programs, the unwinder has at least a couple of uses I know of:

• _Unwind_Backtrace
• __attribute__((cleanup)) with -fexceptions

I don't really know how important these use cases are, or whether there are other important use cases I'm not thinking of. I'd prefer to keep linking the unwinder implicitly.

See D57128 and D59109.

I think the Android team is happy with this driver change, though, so I expect I'll push it next week.

[smeenai]

With NDK r22, I only see libunwind.a for armv7. Will it be provided for other architectures in future NDK versions?

[rprichard]

In D96403#2638883, @smeenai wrote:

With NDK r22, I only see libunwind.a for armv7. Will it be provided for other architectures in future NDK versions?

NDK r23 should have a libunwind.a for all architectures. This change isn't in the r23 beta 2 that was just released, but it is in the current NDK canary build at https://ci.android.com/builds/branches/aosp-master-ndk/grid. (e.g. build 7220482).

[smeenai]

In D96403#2638932, @rprichard wrote:

In D96403#2638883, @smeenai wrote:

With NDK r22, I only see libunwind.a for armv7. Will it be provided for other architectures in future NDK versions?

NDK r23 should have a libunwind.a for all architectures. This change isn't in the r23 beta 2 that was just released, but it is in the current NDK canary build at https://ci.android.com/builds/branches/aosp-master-ndk/grid. (e.g. build 7220482).

Got it, thanks.

Is there a public repository for viewing the libunwind source state that the NDK r23 libunwind is built from? I have the AOSP source tree checked out (following the instructions in https://source.android.com/setup/build/downloading) on the llvm-toolchain manifest branch. toolchain/llvm-project inside that checkout (https://android.googlesource.com/toolchain/llvm-project) has a branch for ndk-release-r23, but the merge base of that branch and upstream LLVM main is from April 29 2020, and I don't see many newer changes to libunwind after that, so I wasn't sure I was looking at the right place. I know you've done a bunch more libunwind work since then (e.g. D87750, D87881, and D90898), so I was curious if those were gonna be part of r23's libunwind.

[srhines]

In D96403#2639137, @smeenai wrote:

In D96403#2638932, @rprichard wrote:

In D96403#2638883, @smeenai wrote:

With NDK r22, I only see libunwind.a for armv7. Will it be provided for other architectures in future NDK versions?

NDK r23 should have a libunwind.a for all architectures. This change isn't in the r23 beta 2 that was just released, but it is in the current NDK canary build at https://ci.android.com/builds/branches/aosp-master-ndk/grid. (e.g. build 7220482).

Got it, thanks.

Is there a public repository for viewing the libunwind source state that the NDK r23 libunwind is built from? I have the AOSP source tree checked out (following the instructions in https://source.android.com/setup/build/downloading) on the llvm-toolchain manifest branch. toolchain/llvm-project inside that checkout (https://android.googlesource.com/toolchain/llvm-project) has a branch for ndk-release-r23, but the merge base of that branch and upstream LLVM main is from April 29 2020, and I don't see many newer changes to libunwind after that, so I wasn't sure I was looking at the right place. I know you've done a bunch more libunwind work since then (e.g. D87750, D87881, and D90898), so I was curious if those were gonna be part of r23's libunwind.

So libunwind is now produced as a prebuilt with the rest of the toolchain build, and not built from that copy of toolchain/llvm_project from the NDK packaging/release branch. Thus if you want to rebuild/examine the sources, you need to look at the toolchain that was included for NDK r23, which is clang-r399163b (from https://android.googlesource.com/platform/ndk/+/refs/tags/ndk-r23-beta2/ndk/toolchains.py#25). Given that, you can get the manifest for the toolchain directly here. You can use that manifest to check out the exact sources that were used to build everything (following instructions from here).

^ PROCESS is correct, but I picked the wrong version (not beta2). I'll update and post a better reply in a few minutes.

[srhines]

In D96403#2639180, @srhines wrote:

In D96403#2639137, @smeenai wrote:

In D96403#2638932, @rprichard wrote:

In D96403#2638883, @smeenai wrote:

With NDK r22, I only see libunwind.a for armv7. Will it be provided for other architectures in future NDK versions?

NDK r23 should have a libunwind.a for all architectures. This change isn't in the r23 beta 2 that was just released, but it is in the current NDK canary build at https://ci.android.com/builds/branches/aosp-master-ndk/grid. (e.g. build 7220482).

Got it, thanks.

Is there a public repository for viewing the libunwind source state that the NDK r23 libunwind is built from? I have the AOSP source tree checked out (following the instructions in https://source.android.com/setup/build/downloading) on the llvm-toolchain manifest branch. toolchain/llvm-project inside that checkout (https://android.googlesource.com/toolchain/llvm-project) has a branch for ndk-release-r23, but the merge base of that branch and upstream LLVM main is from April 29 2020, and I don't see many newer changes to libunwind after that, so I wasn't sure I was looking at the right place. I know you've done a bunch more libunwind work since then (e.g. D87750, D87881, and D90898), so I was curious if those were gonna be part of r23's libunwind.

So libunwind is now produced as a prebuilt with the rest of the toolchain build, and not built from that copy of toolchain/llvm_project from the NDK packaging/release branch. Thus if you want to rebuild/examine the sources, you need to look at the toolchain that was included for NDK r23, which is clang-r399163b (from https://android.googlesource.com/platform/ndk/+/refs/tags/ndk-r23-beta2/ndk/toolchains.py#25). Given that, you can get the manifest for the toolchain directly here. You can use that manifest to check out the exact sources that were used to build everything (following instructions from here).

^ PROCESS is correct, but I picked the wrong version (not beta2). I'll update and post a better reply in a few minutes.

It is actually using clang-r416183 from toolchains.py. From there, you can go to the manifest for building those prebuilts. This line shows you the SHA for the toolchain/llvm-project used to build the prebuilts. Finally, you can look here to see what was merged. Note that there's a typo for the revision number in the commit message, but the upstream LLVM SHA mentioned there is correct. Here is where this toolchain was built in case that is useful/interesting.

[smeenai]

Got it, thanks for all the info!

[rprichard]

In D96403#2639210, @srhines wrote:

It is actually using clang-r416183 from toolchains.py. From there, you can go to the manifest for building those prebuilts. This line shows you the SHA for the toolchain/llvm-project used to build the prebuilts. Finally, you can look here to see what was merged. Note that there's a typo for the revision number in the commit message, but the upstream LLVM SHA mentioned there is correct. Here is where this toolchain was built in case that is useful/interesting.

Also: that manifest also links to the toolchain/llvm_android git repository, which has a Python-based build system and a series of patches (mostly cherry-picks) applied to the baseline llvm-project source.

clang/lib/Driver/ToolChain.cpp

ToolChain::UnwindLibType ToolChain::GetUnwindLibType(
 
   const Arg *A = Args.getLastArg(options::OPT_unwindlib_EQ);
   StringRef LibName = A ? A->getValue() : CLANG_DEFAULT_UNWINDLIB;
 
   if (LibName == "none")
     unwindLibType = ToolChain::UNW_None;
   else if (LibName == "platform" || LibName == "") {
     ToolChain::RuntimeLibType RtLibType = GetRuntimeLibType(Args);
-    if (RtLibType == ToolChain::RLT_CompilerRT)
+    if (RtLibType == ToolChain::RLT_CompilerRT) {
+      if (getTriple().isAndroid())
+        unwindLibType = ToolChain::UNW_CompilerRT;
+      else
         unwindLibType = ToolChain::UNW_None;
-    else if (RtLibType == ToolChain::RLT_Libgcc)
+    } else if (RtLibType == ToolChain::RLT_Libgcc)
       unwindLibType = ToolChain::UNW_Libgcc;
   } else if (LibName == "libunwind") {
     if (GetRuntimeLibType(Args) == RLT_Libgcc)
       getDriver().Diag(diag::err_drv_incompatible_unwindlib);
     unwindLibType = ToolChain::UNW_CompilerRT;
   } else if (LibName == "libgcc")
     unwindLibType = ToolChain::UNW_Libgcc;
   else {

clang/lib/Driver/ToolChains/CommonArgs.cpp

 /// Determine whether Objective-C automated reference counting is
 /// enabled.
 bool tools::isObjCAutoRefCount(const ArgList &Args) {
   return Args.hasFlag(options::OPT_fobjc_arc, options::OPT_fno_objc_arc, false);
 }
 
 enum class LibGccType { UnspecifiedLibGcc, StaticLibGcc, SharedLibGcc };
 
-static LibGccType getLibGccType(const Driver &D, const ArgList &Args) {
+static LibGccType getLibGccType(const ToolChain &TC, const Driver &D,
+                                const ArgList &Args) {
   if (Args.hasArg(options::OPT_static_libgcc) ||
       Args.hasArg(options::OPT_static) || Args.hasArg(options::OPT_static_pie))
     return LibGccType::StaticLibGcc;
-  if (Args.hasArg(options::OPT_shared_libgcc) || D.CCCIsCXX())
+  if (Args.hasArg(options::OPT_shared_libgcc))
+    return LibGccType::SharedLibGcc;
+  // The Android NDK only provides libunwind.a, not libunwind.so.
+  if (TC.getTriple().isAndroid())
+    return LibGccType::StaticLibGcc;
+  if (D.CCCIsCXX())
     return LibGccType::SharedLibGcc;
   return LibGccType::UnspecifiedLibGcc;
 }
 
 // Gcc adds libgcc arguments in various ways:
 //
 // gcc <none>:     -lgcc --as-needed -lgcc_s --no-as-needed
 // g++ <none>:                       -lgcc_s               -lgcc
 // gcc shared:                       -lgcc_s               -lgcc
 // g++ shared:                       -lgcc_s               -lgcc
 // gcc static:     -lgcc             -lgcc_eh
 // g++ static:     -lgcc             -lgcc_eh
 // gcc static-pie: -lgcc             -lgcc_eh
 // g++ static-pie: -lgcc             -lgcc_eh
 //
 // Also, certain targets need additional adjustments.
 
 static void AddUnwindLibrary(const ToolChain &TC, const Driver &D,
                              ArgStringList &CmdArgs, const ArgList &Args) {
   ToolChain::UnwindLibType UNW = TC.GetUnwindLibType(Args);
   // Targets that don't use unwind libraries.
-  if (TC.getTriple().isAndroid() || TC.getTriple().isOSIAMCU() ||
-      TC.getTriple().isOSBinFormatWasm() ||
+  if ((TC.getTriple().isAndroid() && UNW == ToolChain::UNW_Libgcc) ||
+      TC.getTriple().isOSIAMCU() || TC.getTriple().isOSBinFormatWasm() ||
       UNW == ToolChain::UNW_None)
     return;
 
-  LibGccType LGT = getLibGccType(D, Args);
+  LibGccType LGT = getLibGccType(TC, D, Args);
   bool AsNeeded = LGT == LibGccType::UnspecifiedLibGcc &&
                   !TC.getTriple().isAndroid() && !TC.getTriple().isOSCygMing();
   if (AsNeeded)
     CmdArgs.push_back(getAsNeededOption(TC, true));
 
   switch (UNW) {
   case ToolChain::UNW_None:
     return;
static void AddUnwindLibrary(const ToolChain &TC, const Driver &D,
   }
 
   if (AsNeeded)
     CmdArgs.push_back(getAsNeededOption(TC, false));
 }
 
 static void AddLibgcc(const ToolChain &TC, const Driver &D,
                       ArgStringList &CmdArgs, const ArgList &Args) {
-  LibGccType LGT = getLibGccType(D, Args);
+  LibGccType LGT = getLibGccType(TC, D, Args);
   if (LGT != LibGccType::SharedLibGcc)
     CmdArgs.push_back("-lgcc");
   AddUnwindLibrary(TC, D, CmdArgs, Args);
   if (LGT == LibGccType::SharedLibGcc)
     CmdArgs.push_back("-lgcc");
-
-  // According to Android ABI, we have to link with libdl if we are
-  // linking with non-static libgcc.
-  //
-  // NOTE: This fixes a link error on Android MIPS as well.  The non-static
-  // libgcc for MIPS relies on _Unwind_Find_FDE and dl_iterate_phdr from libdl.
-  if (TC.getTriple().isAndroid() && LGT != LibGccType::StaticLibGcc)
-    CmdArgs.push_back("-ldl");
 }
 
 void tools::AddRunTimeLibs(const ToolChain &TC, const Driver &D,
                            ArgStringList &CmdArgs, const ArgList &Args) {
   // Make use of compiler-rt if --rtlib option is used
   ToolChain::RuntimeLibType RLT = TC.GetRuntimeLibType(Args);
 
   switch (RLT) {
if (TC.getTriple().isKnownWindowsMSVCEnvironment()) {
       if (Args.hasArg(options::OPT_rtlib_EQ)) {
         TC.getDriver().Diag(diag::err_drv_unsupported_rtlib_for_platform)
             << Args.getLastArg(options::OPT_rtlib_EQ)->getValue() << "MSVC";
       }
     } else
       AddLibgcc(TC, D, CmdArgs, Args);
     break;
   }
+
+  // On Android, the unwinder uses dl_iterate_phdr (or one of
+  // dl_unwind_find_exidx/__gnu_Unwind_Find_exidx on arm32) from libdl.so. For
+  // statically-linked executables, these functions come from libc.a instead.
+  if (TC.getTriple().isAndroid() && !Args.hasArg(options::OPT_static) &&
+      !Args.hasArg(options::OPT_static_pie))
+    CmdArgs.push_back("-ldl");
 }
 
 SmallString<128> tools::getStatsFileName(const llvm::opt::ArgList &Args,
                                          const InputInfo &Output,
                                          const InputInfo &Input,
                                          const Driver &D) {
   const Arg *A = Args.getLastArg(options::OPT_save_stats_EQ);
   if (!A)