[WebAssembly] Support multilibs for wasm32 and add a wasm OS that uses it

[sunfish]

Summary

This adds support for multilib paths for wasm32 targets, following [Debian's Multiarch conventions], and also adds an experimental OS name in order to test it. It also introduce a wasm OS name that uses this support.

[Debian's Multiarch conventions]: https://wiki.debian.org/Multiarch/

[sunfish]

From a survey and some testing, Emscripten doesn't appear to rely on these paths, so this just affects things like wasmception and the waterfall, which should be straightforward to update.

[dschuff]

LGTM but is there really no test for this? I'd expect to see this on linker command lines or something, right? If not we should add one.

[sbc100]

My understanding is that names like "lib32" and "lib64", at least for x86/amd64 are the old fashioned (legact) way of doing multi-arch. I things a better approach it to either provide and single-arch sysroot (where /usr/lib contains what you want) or a multi-arch setup (where you would use something like /usr/lib/x86_64-linux-gnu/). Since other parts of a sysroot need to be arch specific I suggest we stick with the former option and assume a separate sysroot for each architecture and sub-architecture.

So, while I don't support this change, I would support something like adding "/lib/wasm32-unknown-unknown" before "/lib" in the default library path.

[sunfish]

Per @sbc100's comment, this update switches to full multilib paths.

[sbc100]

Ooo - a new OS is born!

Perhaps create a separate change to introduce the new OS? .. which will then make this CL tiny and specific to what the CL title says it is doing.

tools/clang/lib/Driver/ToolChains/WebAssembly.cpp

95

For include paths you have multiarch + default but for library paths you have multiarch *or* default. Is that deliberate?

[dschuff]

One difference between the multilib style and multi-arch style is whether any headers can be shared between the 2 arches. Historically at least, with bitness-flavored variants of the same architecture there were common headers used by both, and it was just the library directories that needed to be different. I've heard anecdotally that this is not the case for ARM though.

Would we expect that users might want to use a single sysroot to generate code for both 32-and 64-bit output?

[sunfish]

In D56553#1356467, @sbc100 wrote:

Ooo - a new OS is born!

Perhaps create a separate change to introduce the new OS? .. which will then make this CL tiny and specific to what the CL title says it is doing.

Without the new OS I wouldn't be able to fully test the new functionality, as we have no other wasm OS's in tree. Is the size of this patch an issue?

tools/clang/lib/Driver/ToolChains/WebAssembly.cpp

95

Yes; I followed https://wiki.debian.org/Multiarch/Implementation . "include" can contain platform-independent headers, while libraries are inherently platform-dependent.

[sunfish]

In D56553#1356482, @dschuff wrote:

One difference between the multilib style and multi-arch style is whether any headers can be shared between the 2 arches. Historically at least, with bitness-flavored variants of the same architecture there were common headers used by both, and it was just the library directories that needed to be different. I've heard anecdotally that this is not the case for ARM though.

Would we expect that users might want to use a single sysroot to generate code for both 32-and 64-bit output?

Perhaps when we add support for wasm64, we could also add an additional include path "include/wasm-$OS", separate from "include/wasm32-$OS" and "include/wasm64-$OS", which could be shared by both wasm32 and wasm64, but still specific to wasm.

[sunfish]

Or, here's another idea: What if we *just* add "include/wasm-$OS" and don't add "wasm32-$OS" or "wasm64-$OS"-specific include directories? Most headers for wasm will probably be shared between the two, and for anything that really can't be, we always have the __wasm32__ and __wasm64__ predefined macros.

[sbc100]

In D56553#1356654, @sunfish wrote:

In D56553#1356467, @sbc100 wrote:

Ooo - a new OS is born!

Perhaps create a separate change to introduce the new OS? .. which will then make this CL tiny and specific to what the CL title says it is doing.

Without the new OS I wouldn't be able to fully test the new functionality, as we have no other wasm OS's in tree. Is the size of this patch an issue?

Not so much the size of the patch, but more that atomicity. Introducing the new OS seems like an naturally separate change to me.

If you don't want to split it at least update the CL title and description.

[sbc100]

Looking that the behaviour for x86 clang it looks like -L/lib -L/usr/lib are always appended for both -m32 and -m64 with the arch specific directories coming first.

[sbc100]

Can you make change to llvm and clang at the same time? I thought that was not possible..

[sunfish]

Title updated.

In D56553#1356705, @sbc100 wrote:

Looking that the behaviour for x86 clang it looks like -L/lib -L/usr/lib are always appended for both -m32 and -m64 with the arch specific directories coming first.

My reading of https://wiki.debian.org/Multiarch/Implementation is that /lib and /usr/lib don't exist a full multi-arch world. I assume they're still around on x86 because the transition to multiarch is still ongoing and some packages still expect to install their libs there. I'd prefer to avoid adding it since it could cause trouble if anyone actually created a multilib sysroot with non-wasm libs in /lib.

[sunfish]

In D56553#1356720, @sbc100 wrote:

Can you make change to llvm and clang at the same time? I thought that was not possible..

That's right, I'll commit the two parts separately.

[dschuff]

include/llvm/ADT/Triple.h

191

The fact that COWS comes right after Hurd makes me way happier than it should.

include/llvm/ADT/Triple.h

     AMDPAL,     // AMD PAL Runtime
     HermitCore, // HermitCore Unikernel/Multikernel
     Hurd,       // GNU/Hurd
-    LastOSType = Hurd
+    COWS,       // Capability-Oriented WebAssembly Syscalls
+    LastOSType = COWS
   };
   enum EnvironmentType {
     UnknownEnvironment,

     return getOS() == Triple::Hurd;
   }
 
+  /// Tests whether the OS is COWS.
+  bool isOSCOWS() const {
+    return getOS() == Triple::COWS;
+  }
+
   /// Tests whether the OS uses glibc.
   bool isOSGlibc() const {
     return (getOS() == Triple::Linux || getOS() == Triple::KFreeBSD ||

lib/Support/Triple.cpp

   case AMDPAL: return "amdpal";
   case HermitCore: return "hermit";
   case Hurd: return "hurd";
+  case COWS: return "cows";
   }
 
   llvm_unreachable("Invalid OSType");

     .StartsWith("amdpal", Triple::AMDPAL)
     .StartsWith("hermit", Triple::HermitCore)
     .StartsWith("hurd", Triple::Hurd)
+    .StartsWith("cows", Triple::COWS)
     .Default(Triple::UnknownOS);
 }
 

tools/clang/lib/Basic/Targets.cpp

   case llvm::Triple::wasm32:
     if (Triple.getSubArch() != llvm::Triple::NoSubArch ||
         Triple.getVendor() != llvm::Triple::UnknownVendor ||
-        Triple.getOS() != llvm::Triple::UnknownOS ||
-        Triple.getEnvironment() != llvm::Triple::UnknownEnvironment ||
-        !(Triple.isOSBinFormatELF() || Triple.isOSBinFormatWasm()))
+        !Triple.isOSBinFormatWasm())
       return nullptr;
+    if (Triple.getOS() != llvm::Triple::UnknownOS &&
+        Triple.getOS() != llvm::Triple::COWS)
+      return nullptr;
     return new WebAssemblyOSTargetInfo<WebAssembly32TargetInfo>(Triple, Opts);
   case llvm::Triple::wasm64:
     if (Triple.getSubArch() != llvm::Triple::NoSubArch ||
         Triple.getVendor() != llvm::Triple::UnknownVendor ||
-        Triple.getOS() != llvm::Triple::UnknownOS ||
-        Triple.getEnvironment() != llvm::Triple::UnknownEnvironment ||
-        !(Triple.isOSBinFormatELF() || Triple.isOSBinFormatWasm()))
+        !Triple.isOSBinFormatWasm())
       return nullptr;
+    if (Triple.getOS() != llvm::Triple::UnknownOS &&
+        Triple.getOS() != llvm::Triple::COWS)
+      return nullptr;
     return new WebAssemblyOSTargetInfo<WebAssembly64TargetInfo>(Triple, Opts);
 
   case llvm::Triple::renderscript32:

tools/clang/lib/Driver/ToolChains/WebAssembly.cpp

 wasm::Linker::Linker(const ToolChain &TC)
     : GnuTool("wasm::Linker", "lld", TC) {}
 
+/// Following the conventions in https://wiki.debian.org/Multiarch/Tuples,
+/// we remove the vendor field to form the multiarch triple.
+static std::string getMultiarchTriple(const Driver &D,
+                                      const llvm::Triple &TargetTriple,
+                                      StringRef SysRoot) {
+    return TargetTriple.getArchName() + "-" +
+           TargetTriple.getOSAndEnvironmentName();
+}
+
 bool wasm::Linker::isLinkJob() const { return true; }
 
 bool wasm::Linker::hasIntegratedCPP() const { return false; }

 
   getProgramPaths().push_back(getDriver().getInstalledDir());
 
-  getFilePaths().push_back(getDriver().SysRoot + "/lib");
+  if (Triple.getOS() == Triple::UnknownOS) {
+    // Theoretically an "unknown" OS should mean no standard libraries, however
+    // it could also mean that a custom set of libraries is in use, so just add
+    // /lib to the search path. Disable multiarch in this case, to discourage
+    // paths containing "unknown" from acquiring meanings.
+    getFilePaths().push_back(getDriver().SysRoot + "/lib");
+  } else {
+    const std::string MultiarchTriple = getMultiarchTriple(D, Triple, SysRoot);
+    getFilePaths().push_back(getDriver().SysRoot + "/lib/" + MultiarchTriple);
+  }
 }
 
 bool WebAssembly::IsMathErrnoDefault() const { return false; }

 
 void WebAssembly::AddClangSystemIncludeArgs(const ArgList &DriverArgs,
                                             ArgStringList &CC1Args) const {
-  if (!DriverArgs.hasArg(options::OPT_nostdinc))
+  if (!DriverArgs.hasArg(options::OPT_nostdinc)) {
+    if (Triple.getOS() != Triple::UnknownOS) {
+      const std::string MultiarchTriple = getMultiarchTriple(D, Triple, SysRoot);
+      addSystemInclude(DriverArgs, CC1Args, getDriver().SysRoot + "/include/" + MultiarchTriple);
+    }
     addSystemInclude(DriverArgs, CC1Args, getDriver().SysRoot + "/include");
+  }
 }
 
 void WebAssembly::AddClangCXXStdlibIncludeArgs(const ArgList &DriverArgs,
                                                ArgStringList &CC1Args) const {
   if (!DriverArgs.hasArg(options::OPT_nostdlibinc) &&
-      !DriverArgs.hasArg(options::OPT_nostdincxx))
+      !DriverArgs.hasArg(options::OPT_nostdincxx)) {
+    if (Triple.getOS() != Triple::UnknownOS) {
+      const std::string MultiarchTriple = getMultiarchTriple(D, Triple, SysRoot);
+      addSystemInclude(DriverArgs, CC1Args,
+                       getDriver().SysRoot + "/include/" + MultiarchTriple + "/c++/v1");
+    }
     addSystemInclude(DriverArgs, CC1Args,
                      getDriver().SysRoot + "/include/c++/v1");
+  }
 }
 
 void WebAssembly::AddCXXStdlibLibArgs(const llvm::opt::ArgList &Args,

tools/clang/test/Driver/wasm-toolchain.c

 // RUN: %clang %s -### -target wasm32-unknown-unknown -fvisibility=default 2>&1 | FileCheck -check-prefix=FVISIBILITY_DEFAULT %s
 // FVISIBILITY_DEFAULT-NOT: hidden
 
-// A basic C link command-line.
+// A basic C link command-line with unknown OS.
 
 // RUN: %clang -### -no-canonical-prefixes -target wasm32-unknown-unknown --sysroot=/foo -fuse-ld=wasm-ld %s 2>&1 | FileCheck -check-prefix=LINK %s
 // LINK: clang{{.*}}" "-cc1" {{.*}} "-o" "[[temp:[^"]*]]"
 // LINK: wasm-ld{{.*}}" "-L/foo/lib" "crt1.o" "[[temp]]" "-lc" "{{.*[/\\]}}libclang_rt.builtins-wasm32.a" "-o" "a.out"
 
-// A basic C link command-line with optimization.
+// A basic C link command-line with optimization with unknown OS.
 
 // RUN: %clang -### -O2 -no-canonical-prefixes -target wasm32-unknown-unknown --sysroot=/foo -fuse-ld=wasm-ld %s 2>&1 | FileCheck -check-prefix=LINK_OPT %s
 // LINK_OPT: clang{{.*}}" "-cc1" {{.*}} "-o" "[[temp:[^"]*]]"
 // LINK_OPT: wasm-ld{{.*}}" "-L/foo/lib" "crt1.o" "[[temp]]" "-lc" "{{.*[/\\]}}libclang_rt.builtins-wasm32.a" "-o" "a.out"
+
+// A basic C link command-line with known OS.
+
+// RUN: %clang -### -no-canonical-prefixes -target wasm32-unknown-cows-musl --sysroot=/foo -fuse-ld=wasm-ld %s 2>&1 | FileCheck -check-prefix=LINK %s
+// LINK: clang{{.*}}" "-cc1" {{.*}} "-o" "[[temp:[^"]*]]"
+// LINK: wasm-ld{{.*}}" "-L/foo/lib/wasm32-cows-musl" "crt1.o" "[[temp]]" "-lc" "{{.*[/\\]}}libclang_rt.builtins-wasm32.a" "-o" "a.out"
+
+// A basic C link command-line with optimization with known OS.
+
+// RUN: %clang -### -O2 -no-canonical-prefixes -target wasm32-unknown-cows-musl --sysroot=/foo -fuse-ld=wasm-ld %s 2>&1 | FileCheck -check-prefix=LINK_OPT %s
+// LINK_OPT: clang{{.*}}" "-cc1" {{.*}} "-o" "[[temp:[^"]*]]"
+// LINK_OPT: wasm-ld{{.*}}" "-L/foo/lib/wasm32-cows-musl" "crt1.o" "[[temp]]" "-lc" "{{.*[/\\]}}libclang_rt.builtins-wasm32.a" "-o" "a.out"
+
+// A basic C compile command-line with known OS.
+
+// RUN: %clang -### -no-canonical-prefixes -target wasm32-unknown-cows-musl --sysroot=/foo %s 2>&1 | FileCheck -check-prefix=COMPILE %s
+// COMPILE: clang{{.*}}" "-cc1" {{.*}} -I /foo/wasm32-cows-musl/include -I /include

tools/clang/test/Driver/wasm-toolchain.cpp

 // RUN: %clangxx %s -### -target wasm32-unknown-unknown -fvisibility=default 2>&1 | FileCheck -check-prefix=FVISIBILITY_DEFAULT %s
 // FVISIBILITY_DEFAULT-NOT: hidden
 
-// A basic C++ link command-line.
+// A basic C++ link command-line with unknown OS.
 
-// RUN: %clangxx -### -no-canonical-prefixes -target wasm32-unknown-unknown --sysroot=/foo --stdlib=c++ -fuse-ld=wasm-ld %s 2>&1 | FileCheck -check-prefix=LINK %s
+// RUN: %clangxx -### -no-canonical-prefixes -target wasm32-unknown-unknown --sysroot=/foo -fuse-ld=wasm-ld --stdlib=c++ %s 2>&1 | FileCheck -check-prefix=LINK %s
 // LINK: clang{{.*}}" "-cc1" {{.*}} "-o" "[[temp:[^"]*]]"
 // LINK: wasm-ld{{.*}}" "-L/foo/lib" "crt1.o" "[[temp]]" "-lc++" "-lc++abi" "-lc" "{{.*[/\\]}}libclang_rt.builtins-wasm32.a" "-o" "a.out"
 
-// A basic C++ link command-line with optimization.
+// A basic C++ link command-line with optimization with unknown OS.
 
 // RUN: %clangxx -### -O2 -no-canonical-prefixes -target wasm32-unknown-unknown --sysroot=/foo %s --stdlib=c++ -fuse-ld=wasm-ld 2>&1 | FileCheck -check-prefix=LINK_OPT %s
 // LINK_OPT: clang{{.*}}" "-cc1" {{.*}} "-o" "[[temp:[^"]*]]"
 // LINK_OPT: wasm-ld{{.*}}" "-L/foo/lib" "crt1.o" "[[temp]]" "-lc++" "-lc++abi" "-lc" "{{.*[/\\]}}libclang_rt.builtins-wasm32.a" "-o" "a.out"
+
+// A basic C++ link command-line with known OS.
+
+// RUN: %clangxx -### -no-canonical-prefixes -target wasm32-unknown-cows-musl --sysroot=/foo -fuse-ld=wasm-ld --stdlib=c++ %s 2>&1 | FileCheck -check-prefix=LINK %s
+// LINK: clang{{.*}}" "-cc1" {{.*}} "-o" "[[temp:[^"]*]]"
+// LINK: wasm-ld{{.*}}" "-L/foo/lib/wasm32-cows-musl" "crt1.o" "[[temp]]" "-lc++" "-lc++abi" "-lc" "{{.*[/\\]}}libclang_rt.builtins-wasm32.a" "-o" "a.out"
+
+// A basic C++ link command-line with optimization with known OS.
+
+// RUN: %clangxx -### -O2 -no-canonical-prefixes -target wasm32-unknown-cows-musl --sysroot=/foo %s --stdlib=c++ -fuse-ld=wasm-ld 2>&1 | FileCheck -check-prefix=LINK_OPT %s
+// LINK_OPT: clang{{.*}}" "-cc1" {{.*}} "-o" "[[temp:[^"]*]]"
+// LINK_OPT: wasm-ld{{.*}}" "-L/foo/lib/wasm32-cows-musl" "crt1.o" "[[temp]]" "-lc++" "-lc++abi" "-lc" "{{.*[/\\]}}libclang_rt.builtins-wasm32.a" "-o" "a.out"
+
+// A basic C++ compile command-line with known OS.
+
+// RUN: %clangxx -### -no-canonical-prefixes -target wasm32-unknown-cows-musl --sysroot=/foo --stdlib=c++ %s 2>&1 | FileCheck -check-prefix=COMPILE %s
+// COMPILE: clang{{.*}}" "-cc1" {{.*}} -I /foo/wasm32-cows-musl/include -I /include -I /foo/wasm32-cows-musl/include/c++/v1 -I /foo/wasm32/include/c++/v1

unittests/ADT/TripleTest.cpp

   EXPECT_EQ(Triple::UnknownOS, T.getOS());
   EXPECT_EQ(Triple::UnknownEnvironment, T.getEnvironment());
 
+  T = Triple("wasm32-unknown-cows-musl");
+  EXPECT_EQ(Triple::wasm32, T.getArch());
+  EXPECT_EQ(Triple::UnknownVendor, T.getVendor());
+  EXPECT_EQ(Triple::COWS, T.getOS());
+  EXPECT_EQ(Triple::Musl, T.getEnvironment());
+
   T = Triple("wasm64-unknown-unknown");
   EXPECT_EQ(Triple::wasm64, T.getArch());
   EXPECT_EQ(Triple::UnknownVendor, T.getVendor());

   EXPECT_EQ(Triple::UnknownOS, T.getOS());
   EXPECT_EQ(Triple::UnknownEnvironment, T.getEnvironment());
 
+  T = Triple("wasm64-unknown-cows-musl");
+  EXPECT_EQ(Triple::wasm64, T.getArch());
+  EXPECT_EQ(Triple::UnknownVendor, T.getVendor());
+  EXPECT_EQ(Triple::COWS, T.getOS());
+  EXPECT_EQ(Triple::Musl, T.getEnvironment());
+
   T = Triple("avr-unknown-unknown");
   EXPECT_EQ(Triple::avr, T.getArch());
   EXPECT_EQ(Triple::UnknownVendor, T.getVendor());

 
   EXPECT_EQ(Triple::Wasm, Triple("wasm32-unknown-unknown").getObjectFormat());
   EXPECT_EQ(Triple::Wasm, Triple("wasm64-unknown-unknown").getObjectFormat());
+  EXPECT_EQ(Triple::Wasm, Triple("wasm32-unknown-cows-musl").getObjectFormat());
+  EXPECT_EQ(Triple::Wasm, Triple("wasm64-unknown-cows-musl").getObjectFormat());
 
   EXPECT_EQ(Triple::Wasm,
             Triple("wasm32-unknown-unknown-wasm").getObjectFormat());
   EXPECT_EQ(Triple::Wasm,
             Triple("wasm64-unknown-unknown-wasm").getObjectFormat());
+  EXPECT_EQ(Triple::Wasm,
+            Triple("wasm32-unknown-cows-musl-wasm").getObjectFormat());
+  EXPECT_EQ(Triple::Wasm,
+            Triple("wasm64-unknown-cows-musl-wasm").getObjectFormat());
 
   Triple MSVCNormalized(Triple::normalize("i686-pc-windows-msvc-elf"));
   EXPECT_EQ(Triple::ELF, MSVCNormalized.getObjectFormat());