[llvm-lib] Add support for ARM64EC libraries.

[jacek]

Summary

ARM64EC allows having both pure ARM64 objects and ARM64EC in the same archive. This allows using single static library for linking pure ARM64, pure ARM64EC or mixed modules (what MS calls ARM64X: a single module that may be used in both modes). To achieve that, such static libraries need two separated symbol maps. The usual map contains only pure ARM64 symbols, while a new /<ECSYMBOLS>/ section contains EC symbols. EC symbols map has very similar format to the usual map, except it doesn't contain object offsets and uses offsets from regular map instead. This is true even for pure ARM64EC static library: it will simply have 0 symbols in the symbol map.

[efriedma]

I'll try to find time to review in more detail soon... unless someone else wants to review. (I'm not really that familiar with this code, but I doubt anyone has looked at it in years anyway.) Leaving a couple quick comments from my first glance.

If you're going to number your patches, please use some sort of consistent numbering... but you don't need to number a "series" like this. Just stacking them in Phabricator is enough. Maybe change the refactoring patches to explicitly put [NFC] in the title.

llvm/lib/Object/ArchiveWriter.cpp

574

T.isWindowsArm64EC()

llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp

262

Do we need to mess with this inference code to make mixed archives work correctly?

[jacek]

Thanks! The new version implements better logic for inferred machine type. I'm not sure if there are any recommendations about compatibility: I checked with lib.exe and it simply always requires explicit machine type for ARM64EC. If I try to create pure ARM64EC library with no -machine: argument, it just fails. I guess it's fine to do better than that, but I may follow that if preferred.

[jacek]

The new version also allows including x86_64 object files in ARM64EC libraries. I originally avoided expressing target in writeArchive call, but it now doesn't have enough info to decide what kind of symbol map to produce (if only x86_64 binaries are passed, it can't deduce if it's meant to be x86_64 or ARM64EC library).

I also added a few more tests and I will add another patch to llvm-nm for printing EC symbol map, which allows for better tests.

[jacek]

I reconsidered support for inferring ARM64EC target from object files and followed MSVC lib, which requires an explicit -machine argument. This makes the patch a bit smaller. Allowing inferring machine type makes type checks much weaker and could potentially unintentionally allow users to mix x86_64 and aarch64 objects. I don't have strong opinion on this, I will restore that code if it's preferred. I also rebased the patch.

[jacek]

llvm-lib will need a bit more changes for ARM64X support, they are part of D148517.

[efriedma]

LGTM

[TWeaver]

Hello and good afternoon from the UK,

I believe this is 1 of 3 patches that may have caused the following build bot to begin failing, are you able to take a look?

https://lab.llvm.org/buildbot/#/builders/216/builds/20269

Warm regards,
Tom W

[jacek]

Yes, sorry about that. I created https://reviews.llvm.org/D148922 with missing REQUIRES statements.

llvm/lib/Object/ArchiveWriter.cpp

 #else
 #include <io.h>
 #endif
 
 using namespace llvm;
 
 struct SymMap {
   std::map<std::string, uint16_t> Map;
+  std::map<std::string, uint16_t> ECMap;
 };
 
 NewArchiveMember::NewArchiveMember(MemoryBufferRef BufRef)
     : Buf(MemoryBuffer::getMemBuffer(BufRef, false)),
       MemberName(BufRef.getBufferIdentifier()) {}
 
 object::Archive::Kind NewArchiveMember::detectKindFromObject() const {
   auto MemBufferRef = this->Buf->getMemBufferRef();
static uint64_t computeSymbolMapSize(uint64_t NumObj, SymMap &SymMap,
 
   uint32_t Pad = offsetToAlignment(Size, Align(2));
   Size += Pad;
   if (Padding)
     *Padding = Pad;
   return Size;
 }
 
+static uint64_t computeECSymbolsSize(SymMap &SymMap,
+                                     uint32_t *Padding = nullptr) {
+  uint64_t Size = sizeof(uint32_t); // Number of symbols
+
+  for (auto S : SymMap.ECMap)
+    Size += sizeof(uint16_t) + S.first.length() + 1;
+
+  uint32_t Pad = offsetToAlignment(Size, Align(2));
+  Size += Pad;
+  if (Padding)
+    *Padding = Pad;
+  return Size;
+}
+
 static void writeSymbolTableHeader(raw_ostream &Out, object::Archive::Kind Kind,
                                    bool Deterministic, uint64_t Size,
                                    uint64_t PrevMemberOffset = 0) {
   if (isBSDLike(Kind)) {
     const char *Name = is64BitKind(Kind) ? "__.SYMDEF_64" : "__.SYMDEF";
     printBSDMemberHeader(Out, Out.tell(), Name, now(Deterministic), 0, 0, 0,
                          Size);
   } else if (isAIXBigArchive(Kind)) {
auto computeSymbolTableHeaderSize = [=] {
     writeSymbolTableHeader(Tmp, Kind, true, SymtabSize);
     return TmpBuf.size();
   };
   uint32_t HeaderSize = computeSymbolTableHeaderSize();
 
   uint64_t Size = sizeof("!<arch>\n") - 1;
   Size += HeaderSize + SymtabSize;
 
-  if (SymMap)
+  if (SymMap) {
     Size += HeaderSize + computeSymbolMapSize(NumMembers, *SymMap);
+    if (SymMap->ECMap.size())
+      Size += HeaderSize + computeECSymbolsSize(*SymMap);
+  }
 
   return Size + StringMemberSize;
 }
 
 static void writeSymbolTable(raw_ostream &Out, object::Archive::Kind Kind,
                              bool Deterministic, ArrayRef<MemberData> Members,
                              StringRef StringTable, uint64_t MembersOffset,
                              uint64_t PrevMemberOffset = 0) {
for (auto S : SymMap.Map)
     printLE(Out, S.second);
   for (auto S : SymMap.Map)
     Out << S.first << '\0';
 
   while (Pad--)
     Out.write(uint8_t(0));
 }
 
+static void writeECSymbols(raw_ostream &Out, object::Archive::Kind Kind,
+                           bool Deterministic, ArrayRef<MemberData> Members,
+                           SymMap &SymMap) {
+  uint32_t Pad;
+  uint64_t Size = computeECSymbolsSize(SymMap, &Pad);
+  printGNUSmallMemberHeader(Out, "/<ECSYMBOLS>", now(Deterministic), 0, 0, 0,
+                            Size);
+
+  printLE<uint32_t>(Out, SymMap.ECMap.size());
+
+  for (auto S : SymMap.ECMap)
+    printLE(Out, S.second);
+  for (auto S : SymMap.ECMap)
+    Out << S.first << '\0';
+  while (Pad--)
+    Out.write(uint8_t(0));
+}
+
+static bool isECObject(object::SymbolicFile &Obj) {
+  if (Obj.isCOFF())
+    return cast<llvm::object::COFFObjectFile>(&Obj)->getMachine() ==
+           COFF::IMAGE_FILE_MACHINE_ARM64EC;
+
+  if (Obj.isIR()) {
+    Expected<std::string> TripleStr =
+        getBitcodeTargetTriple(Obj.getMemoryBufferRef());
+    if (!TripleStr)
+      return false;
+    Triple T(*TripleStr);
+    return T.getArch() == Triple::aarch64 &&
+           T.getSubArch() == Triple::AArch64SubArch_arm64ec;
+  }
+
+  return false;
+}
+
 static Expected<std::vector<unsigned>>
 getSymbols(MemoryBufferRef Buf, uint16_t Index, raw_ostream &SymNames,
            SymMap *SymMap, bool &HasObject) {
   std::vector<unsigned> Ret;
 
   // In the scenario when LLVMContext is populated SymbolicFile will contain a
   // reference to it, thus SymbolicFile should be destroyed first.
   LLVMContext Context;
if (Type == file_magic::bitcode) {
     Obj = std::move(*ObjOrErr);
   } else {
     auto ObjOrErr = object::SymbolicFile::createSymbolicFile(Buf);
     if (!ObjOrErr)
       return ObjOrErr.takeError();
     Obj = std::move(*ObjOrErr);
   }
 
+  std::map<std::string, uint16_t> *Map = nullptr;
+  if (SymMap)
+    Map = isECObject(*Obj) ? &SymMap->ECMap : &SymMap->Map;
   HasObject = true;
   for (const object::BasicSymbolRef &S : Obj->symbols()) {
     if (!isArchiveSymbol(S))
       continue;
-    if (SymMap) {
+    if (Map) {
       std::string Name;
       raw_string_ostream NameStream(Name);
       if (Error E = S.printName(NameStream))
         return std::move(E);
-      if (SymMap->Map.find(Name) != SymMap->Map.end())
+      if (Map->find(Name) != Map->end())
         continue; // ignore duplicated symbol
-      SymMap->Map[Name] = Index;
+      (*Map)[Name] = Index;
+      if (Map == &SymMap->Map) {
         Ret.push_back(SymNames.tell());
         SymNames << Name << '\0';
+      }
     } else {
       Ret.push_back(SymNames.tell());
       if (Error E = S.printName(SymNames))
         return std::move(E);
       SymNames << '\0';
     }
   }
   return Ret;
if (WriteSymtab) {
       if (isCOFFArchive(Kind))
         writeSymbolMap(Out, Kind, Deterministic, Data, SymMap, HeadersSize);
     }
 
     if (StringTableSize)
       Out << StringTableMember.Header << StringTableMember.Data
           << StringTableMember.Padding;
 
+    if (WriteSymtab && SymMap.ECMap.size())
+      writeECSymbols(Out, Kind, Deterministic, Data, SymMap);
+
     for (const MemberData &M : Data)
       Out << M.Header << M.Data << M.Padding;
   } else {
     // For the big archive (AIX) format, compute a table of member names and
     // offsets, used in the member table.
     uint64_t MemberTableNameStrTblSize = 0;
     std::vector<size_t> MemberOffsets;
     std::vector<StringRef> MemberNames;

llvm/lib/ToolDrivers/llvm-lib/LibDriver.cpp

static Expected<COFF::MachineTypes> getCOFFFileMachine(MemoryBufferRef MB) {
   auto Obj = object::COFFObjectFile::create(MB);
   if (!Obj)
     return Obj.takeError();
 
   uint16_t Machine = (*Obj)->getMachine();
   if (Machine != COFF::IMAGE_FILE_MACHINE_I386 &&
       Machine != COFF::IMAGE_FILE_MACHINE_AMD64 &&
       Machine != COFF::IMAGE_FILE_MACHINE_ARMNT &&
-      Machine != COFF::IMAGE_FILE_MACHINE_ARM64) {
+      Machine != COFF::IMAGE_FILE_MACHINE_ARM64 &&
+      Machine != COFF::IMAGE_FILE_MACHINE_ARM64EC) {
     return createStringError(inconvertibleErrorCode(),
                              "unknown machine: " + std::to_string(Machine));
   }
 
   return static_cast<COFF::MachineTypes>(Machine);
 }
 
 static Expected<COFF::MachineTypes> getBitcodeFileMachine(MemoryBufferRef MB) {
static Expected<COFF::MachineTypes> getBitcodeFileMachine(MemoryBufferRef MB) {
   case Triple::aarch64:
     return COFF::IMAGE_FILE_MACHINE_ARM64;
   default:
     return createStringError(inconvertibleErrorCode(),
                              "unknown arch in target triple: " + *TripleStr);
   }
 }
 
+static bool machineMatches(COFF::MachineTypes LibMachine,
+                           COFF::MachineTypes FileMachine) {
+  if (LibMachine == FileMachine)
+    return true;
+  // ARM64EC mode allows both pure ARM64 and ARM64EC objects to be mixed in the
+  // archive.
+  return LibMachine == COFF::IMAGE_FILE_MACHINE_ARM64EC &&
+         FileMachine == COFF::IMAGE_FILE_MACHINE_ARM64;
+}
+
 static void appendFile(std::vector<NewArchiveMember> &Members,
                        COFF::MachineTypes &LibMachine,
                        std::string &LibMachineSource, MemoryBufferRef MB) {
   file_magic Magic = identify_magic(MB.getBuffer());
 
   if (Magic != file_magic::coff_object && Magic != file_magic::bitcode &&
       Magic != file_magic::archive && Magic != file_magic::windows_resource &&
       Magic != file_magic::coff_import_library) {
if (Magic == file_magic::coff_object || Magic == file_magic::bitcode) {
     }
     COFF::MachineTypes FileMachine = *MaybeFileMachine;
 
     // FIXME: Once lld-link rejects multiple resource .obj files:
     // Call convertResToCOFF() on .res files and add the resulting
     // COFF file to the .lib output instead of adding the .res file, and remove
     // this check. See PR42180.
     if (FileMachine != COFF::IMAGE_FILE_MACHINE_UNKNOWN) {
       if (LibMachine == COFF::IMAGE_FILE_MACHINE_UNKNOWN) {
         LibMachine = FileMachine;
         LibMachineSource =
             (" (inferred from earlier file '" + MB.getBufferIdentifier() + "')")
                 .str();
-      } else if (LibMachine != FileMachine) {
+      } else if (!machineMatches(LibMachine, FileMachine)) {
         llvm::errs() << MB.getBufferIdentifier() << ": file machine type "
                      << machineToStr(FileMachine)
                      << " conflicts with library machine type "
                      << machineToStr(LibMachine) << LibMachineSource << '\n';
         exit(1);
       }
     }
   }

llvm/test/tools/llvm-lib/ecsymbols.test

 Members:
   - Name:            '/'
     Size:            '0'
   - Name:            '/'
     Size:            '0'
   - Name:            '/<ECSYMBOLS>/'
     Size:            '0'
 ...
+
+# Check that llvm-lib produces /<ECSYMBOLS>/ members for ARM64EC libraries.
+RUN: rm -rf %t
+RUN: mkdir -p %t
+RUN: llvm-mc -triple=arm64ec-pc-windows-msvc -filetype=obj -o %t/foo.o %S/Inputs/a.s
+RUN: llvm-lib -machine:arm64ec -out:%t/foo.lib %t/foo.o
+RUN: grep -q '/<ECSYMBOLS>/' %t/foo.lib