Arm64EC entry/exit thunks, consolidated.

[efriedma]

Summary

Arm64EC entry/exit thunks, consolidated.

This combines the previously posted patches with some additional work I've done to more closely match MSVC output.

Most of the important logic here is implemented in AArch64Arm64ECCallLowering. The purpose of the AArch64Arm64ECCallLowering is to take "normal" IR we'd generate for other targets, and generate most of the Arm64EC-specific bits: generating thunks, mangling symbols, generating aliases, and generating the .hybmp$x table. This is all done late for a few reasons: to consolidate the logic as much as possible, and to ensure the IR exposed to optimization passes doesn't contain complex arm64ec-specific constructs.

The other changes are supporting changes, to handle the new constructs generated by that pass.

There's a global llvm.arm64ec.symbolmap representing the .hybmp$x entries for the thunks. This gets handled directly by the AsmPrinter because it needs symbol indexes that aren't available before that.

There are two new calling conventions used to represent calls to and from thunks: ARM64EC_Thunk_X64 and ARM64EC_Thunk_Native. There are a few changes to handle the associated exception-handling info, SEH_SaveAnyRegQP and SEH_SaveAnyRegQPX.

I've intentionally left out handling for structs with small non-power-of-two sizes, because that's easily separated out. The rest of my current work is here. I squashed my current patches because they were split in ways that didn't really make sense. Maybe I could split out some bits, but it's hard to meaningfully test most of the parts independently.

I'm planning to write more IR tests for test coverage.

[jacek]

llvm/lib/Target/AArch64/AArch64MCInstLower.cpp

54–57

I think that mangled weak symbol should link to another kind exit thunk, not to unmangled symbol directly.

When an extern code symbol is resolved by an unmangled name, it means that we have ARM64EC->X64 call and we need to use an exit thunk. Linker doesn't need a special logic for that: on MSVC it seems to be achieved by pointing the antidependency symbol to yet another exit thunk. That other exit thunk loads the target exit thunk and X64 (unmangled) symbol into x10, x11 and uses __os_arm64x_dispatch_icall to call emulator. I guess we may worry about that later, but in that case maybe it's better not to emit mangled antidependency symbol at all for now?

[efriedma]

llvm/lib/Target/AArch64/AArch64MCInstLower.cpp

54–57

From what I recall, I wrote the code here primarily to deal with issues trying to take the address of a function; even if the function is defined in arm64ec code, the address is the unmangled symbol. So there's some weirdness there involving symbol lookup even before there's any actual x64 code involved.

If you have a better idea of how external symbol references are supposed to be emitted, I'd appreciate a brief writeup, since all the information I have comes from trying to read dumpbin dumps of MSVC output.

[jacek]

llvm/lib/Target/AArch64/AArch64MCInstLower.cpp

54–57

Yes, when EC code takes an address of a function, it needs to use unmangled symbols, because mangled symbols may point to a thunk (in cases when the implementation is in X64 or __declspec(hybrid_patchable) is used) and we still want an address of the real implementation. Here is an example how MSVC sets it up:

$ cat test.c
extern int otherfunc(void);
int myfunc(void) { return otherfunc(); }
$ llvm-readobj --symbols test.obj
...

Symbol {
  Name: #otherfunc$exit_thunk
  Value: 0
  Section: .wowthk$aa (4)
  BaseType: Null (0x0)
  ComplexType: Function (0x2)
  StorageClass: External (0x2)
  AuxSymbolCount: 0
}

...

Symbol {
  Name: #otherfunc
  Value: 0
  Section: IMAGE_SYM_UNDEFINED (0)
  BaseType: Null (0x0)
  ComplexType: Function (0x2)
  StorageClass: WeakExternal (0x69)
  AuxSymbolCount: 1
  AuxWeakExternal {
    Linked: #otherfunc$exit_thunk (14)
    Search: AntiDependency (0x4)
  }
}
Symbol {
  Name: #myfunc
  Value: 0
  Section: .text$mn (3)
  BaseType: Null (0x0)
  ComplexType: Function (0x2)
  StorageClass: External (0x2)
  AuxSymbolCount: 0
}
Symbol {
  Name: otherfunc
  Value: 0
  Section: IMAGE_SYM_UNDEFINED (0)
  BaseType: Null (0x0)
  ComplexType: Function (0x2)
  StorageClass: WeakExternal (0x69)
  AuxSymbolCount: 1
  AuxWeakExternal {
    Linked: #otherfunc (19)
    Search: AntiDependency (0x4)
  }
}
Symbol {
  Name: myfunc
  Value: 0
  Section: IMAGE_SYM_UNDEFINED (0)
  BaseType: Null (0x0)
  ComplexType: Function (0x2)
  StorageClass: WeakExternal (0x69)
  AuxSymbolCount: 1
  AuxWeakExternal {
    Linked: #myfunc (21)
    Search: AntiDependency (0x4)
  }
}

In this example myfunc links to #myfunc (and similarly otherfunc->#otherfunc). That's enough because even if we pass a pointer of #myfunc to actual X64 code which will try to call this address, emulator will have a chance to figure it out and use entry thunk as needed.

However, #otherfunc points to #otherfunc$exit_thunk (which then references otherfunc and $iexit_thunk$cdecl$i8$v in its implementation). If otherfunc is implemented in ARM64EC, #otherfunc symbol will be replaced by the object file implementing it. If it will not be replaced, it means that symbol is resolved to X64 implementation and the linked thunk will be used to call it.

My information are based on experimentation with MSVC as well, so those are only my best guesses. I mostly experimented with it in a context of linker support and I have those aspects of linker working in my tree. I will work on more comprehensive description of those things.

[dpaoliello]

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

2735–2738

To be clear: you're seeing MSVC emit the same entry twice, but you opted to emit the entry only once?

llvm/lib/Target/AArch64/AArch64Arm64ECCallLowering.cpp

333–334

Having a look through the available parameter attributes:

• Do we need byval and byref, or are they irrelevant to the thunk?
• Should we copy the alignment attributes to keep them the same?
• Might be interesting to copy some of the optimization related ones (unless we're too late in the compilation) like readnone, readonly, immarg, returned, etc.

609

Are you asking if we should be handling functions with weak linkage, or is this a TODO to implement support somewhere else?

642–646

Can you provide some more details about this? What extra stub are you seeing?

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

1644

Full list is:

#ceil
#floor
#memcmp
#memcpy
#memmove
#memset

llvm/lib/Target/AArch64/AArch64InstrInfo.td

4667–4668

Should these be added to AArch64InstrInfo::isSEHInstruction?

[dpaoliello]

llvm/lib/Target/AArch64/AArch64CallingConvention.td

555–557

I've been hitting asserts in computeCalleeSaveRegisterPairs due to this: LLVM doesn't support gaps when saving registers for Windows (~line 2744 of AArch64FrameLowering.cpp), so placing the smaller registers before the larger ones causes issues if those smaller ones aren't paired (the assert(OffsetPre % Scale == 0); fires since OffsetPre will be a multiple of 8 but Scale will be 16).

[efriedma]

llvm/lib/Target/AArch64/AArch64CallingConvention.td

555–557

That looks right; I thought I had that fix in here, but I guess I mixed up my patches and pulled in a different version.

(I'll reply to the other review comments in more detail soon.)

[dpaoliello]

Another issue: looks like the names in AArch64Subtarget.h need to be fully decorated, including the leading # - so #__chkstk_arm64ec and #__security_check_cookie_arm64ec.

[efriedma]

Update to address issues found so far.

[jacek]

I published an initial version of my attempt at documenting this stuff here: https://wiki.winehq.org/ARM64ECToolchain.

It's not yet complete, but I hope it's already useful. I plan to extend it in the future. If some clarification or more information would be useful, please let me know and I will try to improve it or fill the gaps.

[bylaws]

llvm/lib/Target/AArch64/AArch64MCInstLower.cpp

62

This mangling logic is also necessary for dllimported functions, as is visible if you compile:

__declspec(dllimport) long UnhandledExceptionFilter (void *ExceptionInfo);

void test() {

UnhandledExceptionFilter(0);

}

With cl.exe /c and run dumpbin, without this link.exe will fail

[dpaoliello]

llvm/lib/Target/AArch64/AArch64MCInstLower.cpp

37–38

We now also need to call this for constants in case the function is only referenced from a variable (e.g., a VTable) and is never called elsewhere.

I split this function into a new function GetGlobalValueSymbol:

MCSymbol *
AArch64MCInstLower::GetGlobalAddressSymbol(const MachineOperand &MO) const {
  return GetGlobalValueSymbol(MO.getGlobal(), MO.getTargetFlags());
}

MCSymbol *
AArch64MCInstLower::GetGlobalValueSymbol(const GlobalValue *GV, unsigned TargetFlags) const { ... }

And then called it from AArch64AsmPrinter:

const MCExpr *AArch64AsmPrinter::lowerConstant(const Constant *CV) {
  if (const GlobalValue *GV = dyn_cast<GlobalValue>(CV))
    return MCSymbolRefExpr::create(MCInstLowering.GetGlobalValueSymbol(GV, 0), OutContext);

  return AsmPrinter::lowerConstant(CV);
}

[bylaws]

llvm/lib/Target/AArch64/AArch64MCInstLower.cpp

62

Looking into it further, it probably makes sense to drop MO_DLLIMPORT from the hasDLLImportStorageClass case in AArch64Subtarget::classifyGlobalFunctionReference, then emit the __imp symbol for that special case here

[efriedma]

Updated with my attempt at guest exit thunks.

Seems to be working for simple cases, but I'm not sure this is actually working properly (I'm still seeing LNK1000 crashes).

[jacek]

In D157547#4650643, @efriedma wrote:

Seems to be working for simple cases, but I'm not sure this is actually working properly (I'm still seeing LNK1000 crashes).

I don't know if it's related to crashes that you're seeing, but those anti-dependency symbols look different than what MSVC produces. With the patch, I get:

$ cat extfunc.c
extern void extfunc(void);
void func(void) { extfunc(); }
$ clang -target arm64ec-windows -c extcall.c -o extcall.o
$ llvm-readobj --symbols extcall.o
...

Symbol {
  Name: #extfunc
  Value: 0
  Section: IMAGE_SYM_UNDEFINED (0)
  BaseType: Null (0x0)
  ComplexType: Null (0x0)
  StorageClass: WeakExternal (0x69)
  AuxSymbolCount: 1
  AuxWeakExternal {
    Linked: .weak.#extfunc.default.#func (40)
    Search: AntiDependency (0x4)
  }
}

...

Symbol {
  Name: extfunc
  Value: 0
  Section: IMAGE_SYM_UNDEFINED (0)
  BaseType: Null (0x0)
  ComplexType: Null (0x0)
  StorageClass: WeakExternal (0x69)
  AuxSymbolCount: 1
  AuxWeakExternal {
    Linked: #extfunc$exit_thunk (43)
    Search: AntiDependency (0x4)
  }
}

While an equivalent object file produced by MSVC has:

Symbol {
  Name: #extfunc
  Value: 0
  Section: IMAGE_SYM_UNDEFINED (0)
  BaseType: Null (0x0)
  ComplexType: Function (0x2)
  StorageClass: WeakExternal (0x69)
  AuxSymbolCount: 1
  AuxWeakExternal {
    Linked: #extfunc$exit_thunk (14)
    Search: AntiDependency (0x4)
  }
}
Symbol {
  Name: extfunc
  Value: 0
  Section: IMAGE_SYM_UNDEFINED (0)
  BaseType: Null (0x0)
  ComplexType: Function (0x2)
  StorageClass: WeakExternal (0x69)
  AuxSymbolCount: 1
  AuxWeakExternal {
    Linked: #extfunc (19)
    Search: AntiDependency (0x4)
  }
}

It's the mangled symbol that's linked to guest exit thunk, not unmangled one. Also there is no .weak.*.default.* involved.

BTW, I hope to get my WIP lld-link branch into usable state very soon and plan to publish it then. I'm able to link real-world code now, but it still requires a few hacks that I need to fix to make it work out of the box. Hopefully it will make testing and debugging such problems easier.

[efriedma]

Revised so guest exit thunks actually work. Thanks for the tip about the symbols; I forgot to look at the object file in addition to the generated assembly.

[dpaoliello]

@efriedma the patch is failing to apply - can you please update your baseline, or move this to a GitHub PR?

[efriedma]

Rebased

[dpaoliello]

llvm/lib/Target/AArch64/AArch64CallingConvention.td

248–257

These seem wrong, shouldn't the shadows be [X0, X1, X2, X3] (instead of X2 twice)

[efriedma]

Updated with feedback from @dpaoliello and additional internal testing.

[jacek]

It works nice with my testing so far, thanks!

I noticed one problem with code that defines symbols in assembly. It conflicts with anti-dependency symbols emitted by codegen and hits an asserts in assembly printer. This commit fixes it for me:
https://github.com/cjacek/llvm-project/commit/879b1b6bf99b1e506ae04b8db0b44eb1b2205f19

Here is a test case:

extern void asmfunc(void);
__asm__( ".globl \"#asmfunc\"\n"
         "\t.section .text,\"xr\",discard,\"#asmfunc\"\n"
         "\"#asmfunc\":\n\t"
         "nop; ret\n\t" );
void test(void) { asmfunc(); }

[efriedma]

For assembly, I'm not really comfortable with the fix you're proposing; it relies on the order in which functions are defined in assembly. I think LLVM usually ends up emitting module-level inline assembly before generated code, but it seems fragile to depend on that ordering. And the way the check is written doesn't account for other differences in the way we normally generate code for declarations vs. definitions, which I'm afraid might make the linker unhappy.

How important is that particular pattern? I think most patterns involving assembly should be covered by some combination of naked functions, and functions defined in separate assembly files, both of which avoid weird questions about what the compiler should do when assembly tries to define a function symbol.

[efriedma]

Fix the calling convention for functions returning C++ classes.

[efriedma]

For assembly, I'm not really comfortable with the fix you're proposing; it relies on the order in which functions are defined in assembly. I think LLVM usually ends up emitting module-level inline assembly before generated code, but it seems fragile to depend on that ordering. And the way the check is written doesn't account for other differences in the way we normally generate code for declarations vs. definitions, which I'm afraid might make the linker unhappy.

Also, checking isUndefined() completely explodes if you use -fno-integrated-as, although I guess that's not common for Windows targets.

[jacek]

In D157547#4653477, @efriedma wrote:

How important is that particular pattern? I think most patterns involving assembly should be covered by some combination of naked functions, and functions defined in separate assembly files, both of which avoid weird questions about what the compiler should do when assembly tries to define a function symbol.

Thank for suggestions. We can indeed change that to naked functions (or separated assembly file when that's not suitable). We use full assembly declaration on other targets for compatibility reasons (which may be worth revisiting if that's still a problem and it's definitely not a concern in Arm64EC case).

It may be nice to output some form of error instead of assert crash in those cases, but I guess that's a secondary concern at this point.

[dpaoliello]

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

2717–2719

Comment for what this table is?

llvm/lib/Target/AArch64/AArch64Arm64ECCallLowering.cpp

321

Can you please add comments for each of these buildXXX functions to explain what each of these thunks are used for (especially exit thunk vs guest exit thunk)

519

Redundant assert (already asserted on line 511)

clang/lib/CodeGen/CGCXX.cpp

bool CodeGenModule::TryEmitBaseDestructorAsAlias(const CXXDestructorDecl *D) {
   if (!getCodeGenOpts().CXXCtorDtorAliases)
     return true;
 
   // Producing an alias to a base class ctor/dtor can degrade debug quality
   // as the debugger cannot tell them apart.
   if (getCodeGenOpts().OptimizationLevel == 0)
     return true;
 
+  // Disable this optimization for ARM64EC.  FIXME: This probably should work,
+  // but getting the symbol table correct is complicated.
+  if (getTarget().getTriple().isWindowsArm64EC())
+    return true;
+
   // If sanitizing memory to check for use-after-dtor, do not emit as
   //  an alias, unless this class owns no members.
   if (getCodeGenOpts().SanitizeMemoryUseAfterDtor &&
       !D->getParent()->field_empty())
     return true;
 
   // If the destructor doesn't have a trivial body, we have to emit it
   // separately.

llvm/include/llvm/IR/CallingConv.h

enum {
     /// Used on AMDGPUs to give the middle-end more control over argument
     /// placement.
     AMDGPU_CS_Chain = 104,
 
     /// Used on AMDGPUs to give the middle-end more control over argument
     /// placement. Preserves active lane values for input VGPRs.
     AMDGPU_CS_ChainPreserve = 105,
 
+    /// Calling convention used in the ARM64EC ABI to implement calls between
+    /// x64 code and thunks. This is basically the x64 calling convention using
+    /// ARM64 register names. The first parameter is mapped to x9.
+    ARM64EC_Thunk_X64 = 106,
+
+    /// Calling convention used in the ARM64EC ABI to implement calls between
+    /// ARM64 code and thunks. This is just the ARM64 calling convention,
+    /// except that the first parameter is mapped to x9.
+    ARM64EC_Thunk_Native = 107,
+
     /// The highest possible ID. Must be some 2^k - 1.
     MaxID = 1023
   };
 
 } // end namespace CallingConv
 
 } // end namespace llvm
 
 #endif // LLVM_IR_CALLINGCONV_H

llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

if (GV->getName() == "llvm.used") {
     return true;
   }
 
   // Ignore debug and non-emitted data.  This handles llvm.compiler.used.
   if (GV->getSection() == "llvm.metadata" ||
       GV->hasAvailableExternallyLinkage())
     return true;
 
+  if (GV->getName() == "llvm.arm64ec.symbolmap") {
+    OutStreamer->switchSection(OutContext.getCOFFSection(
+        ".hybmp$x", COFF::IMAGE_SCN_LNK_INFO, SectionKind::getMetadata()));
+    auto *Arr = cast<ConstantArray>(GV->getInitializer());
+    for (auto &U : Arr->operands()) {
+      auto *C = cast<Constant>(U);
+      auto *Src = cast<Function>(C->getOperand(0)->stripPointerCasts());
+      auto *Dst = cast<Function>(C->getOperand(1)->stripPointerCasts());
+      int Kind = cast<ConstantInt>(C->getOperand(2))->getZExtValue();
+
+      if (Src->hasDLLImportStorageClass()) {
+        // For now, we assume dllimport functions aren't directly called.
+        // (We might change this later to match MSVC.)
+        OutStreamer->emitCOFFSymbolIndex(
+            OutContext.getOrCreateSymbol("__imp_" + Src->getName()));
+        OutStreamer->emitCOFFSymbolIndex(getSymbol(Dst));
+        OutStreamer->emitInt32(Kind);
+      } else {
+        // FIXME: For non-dllimport functions, MSVC emits the same entry
+        // twice, for reasons I don't understand.  I have to assume the linker
+        // ignores the redundant entry; there aren't any reasonable semantics
+        // to attach to it.
+        OutStreamer->emitCOFFSymbolIndex(getSymbol(Src));
+        OutStreamer->emitCOFFSymbolIndex(getSymbol(Dst));
+        OutStreamer->emitInt32(Kind);
+      }
+    }
+    return true;
+  }
+
   if (!GV->hasAppendingLinkage()) return false;
 
   assert(GV->hasInitializer() && "Not a special LLVM global!");
 
   if (GV->getName() == "llvm.global_ctors") {
     emitXXStructorList(GV->getParent()->getDataLayout(), GV->getInitializer(),
                        /* isCtor */ true);
 

llvm/lib/Target/AArch64/AArch64.h

 FunctionPass *createAArch64O0PreLegalizerCombiner();
 FunctionPass *createAArch64PreLegalizerCombiner();
 FunctionPass *createAArch64PostLegalizerCombiner(bool IsOptNone);
 FunctionPass *createAArch64PostLegalizerLowering();
 FunctionPass *createAArch64PostSelectOptimize();
 FunctionPass *createAArch64StackTaggingPass(bool IsOptNone);
 FunctionPass *createAArch64StackTaggingPreRAPass();
 ModulePass *createAArch64GlobalsTaggingPass();
+ModulePass *createAArch64Arm64ECCallLoweringPass();
 
 void initializeAArch64A53Fix835769Pass(PassRegistry&);
 void initializeAArch64A57FPLoadBalancingPass(PassRegistry&);
 void initializeAArch64AdvSIMDScalarPass(PassRegistry&);
 void initializeAArch64PointerAuthPass(PassRegistry&);
 void initializeAArch64BranchTargetsPass(PassRegistry&);
 void initializeAArch64CFIFixupPass(PassRegistry&);
 void initializeAArch64CollectLOHPass(PassRegistry &);

 void initializeAArch64StackTaggingPass(PassRegistry &);
 void initializeAArch64StackTaggingPreRAPass(PassRegistry &);
 void initializeAArch64StorePairSuppressPass(PassRegistry&);
 void initializeFalkorHWPFFixPass(PassRegistry&);
 void initializeFalkorMarkStridedAccessesLegacyPass(PassRegistry&);
 void initializeLDTLSCleanupPass(PassRegistry&);
 void initializeSMEABIPass(PassRegistry &);
 void initializeSVEIntrinsicOptsPass(PassRegistry &);
+void initializeAArch64Arm64ECCallLoweringPass(PassRegistry &);
 } // end namespace llvm
 
 #endif

llvm/lib/Target/AArch64/AArch64Arm64ECCallLowering.cpp

+//===-- AArch64Arm64ECCallLowering.cpp - Lower Arm64EC calls ----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file contains the IR transform to lower external or indirect calls for
+/// the ARM64EC calling convention. Such calls must go through the runtime, so
+/// we can translate the calling convention for calls into the emulator.
+///
+/// This subsumes Control Flow Guard handling.
+///
+//===----------------------------------------------------------------------===//
+
+#include "AArch64.h"
+#include "llvm/ADT/SetVector.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/IR/CallingConv.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/Instruction.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/TargetParser/Triple.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+
+using namespace llvm;
+
+using OperandBundleDef = OperandBundleDefT<Value *>;
+
+#define DEBUG_TYPE "arm64eccalllowering"
+
+STATISTIC(Arm64ECCallsLowered, "Number of Arm64EC calls lowered");
+
+static cl::opt<bool> LowerDirectToIndirect(
+    "arm64ec-lower-direct-to-indirect", cl::Hidden, cl::init(true));
+static cl::opt<bool> GenerateThunks(
+    "arm64ec-generate-thunks", cl::Hidden, cl::init(true));
+
+namespace {
+
+class AArch64Arm64ECCallLowering : public ModulePass {
+public:
+  static char ID;
+  AArch64Arm64ECCallLowering() : ModulePass(ID) {
+    initializeAArch64Arm64ECCallLoweringPass(*PassRegistry::getPassRegistry());
+  }
+
+  Function *buildExitThunk(FunctionType *FnTy, AttributeList Attrs);
+  Function *buildEntryThunk(Function *F);
+  void lowerCall(CallBase *CB);
+  Function *buildGuestExitThunk(Function *F);
+  bool processFunction(Function &F,
+                       SetVector<Function *> &DirectCalledFns);
+  bool runOnModule(Module &M) override;
+
+private:
+  int cfguard_module_flag = 0;
+  FunctionType *GuardFnType = nullptr;
+  PointerType *GuardFnPtrType = nullptr;
+  Constant *GuardFnCFGlobal = nullptr;
+  Constant *GuardFnGlobal = nullptr;
+  Module *M = nullptr;
+
+  Type *I8PtrTy;
+  Type *I64Ty;
+  Type *VoidTy;
+
+  void getThunkType(FunctionType *FT, AttributeList AttrList, bool EntryThunk,
+                    raw_ostream &Out, FunctionType *&Arm64Ty,
+                    FunctionType *&X64Ty);
+  void getThunkRetType(FunctionType *FT, AttributeList AttrList,
+                       raw_ostream &Out, Type *&Arm64RetTy,
+                       Type *&X64RetTy, SmallVectorImpl<Type *> &Arm64ArgTypes,
+                       SmallVectorImpl<Type *> &X64ArgTypes);
+  void getThunkArgTypes(FunctionType *FT, AttributeList AttrList,
+                        raw_ostream &Out,
+                        SmallVectorImpl<Type *> &Arm64ArgTypes,
+                        SmallVectorImpl<Type *> &X64ArgTypes);
+  void canonicalizeThunkType(Type *T, Align Alignment,
+                             bool Ret, uint64_t ArgSizeBytes, raw_ostream &Out,
+                             Type *&Arm64Ty, Type *&X64Ty);
+};
+
+} // end anonymous namespace
+
+void AArch64Arm64ECCallLowering::getThunkType(FunctionType *FT,
+                                              AttributeList AttrList,
+                                              bool EntryThunk, raw_ostream &Out,
+                                              FunctionType *&Arm64Ty,
+                                              FunctionType *&X64Ty) {
+  Out << (EntryThunk ? "$ientry_thunk$cdecl$" : "$iexit_thunk$cdecl$");
+
+  Type *Arm64RetTy;
+  Type *X64RetTy;
+
+  SmallVector<Type *> Arm64ArgTypes;
+  SmallVector<Type *> X64ArgTypes;
+
+  // The first argument to a thunk is the called function, stored in x9.
+  // For exit thunks, we pass the called function down to the emulator;
+  // for entry thunks, we just call the Arm64 function directly.
+  if (!EntryThunk)
+    Arm64ArgTypes.push_back(I8PtrTy);
+  X64ArgTypes.push_back(I8PtrTy);
+
+  getThunkRetType(FT, AttrList, Out, Arm64RetTy, X64RetTy,
+                  Arm64ArgTypes, X64ArgTypes);
+
+  getThunkArgTypes(FT, AttrList, Out, Arm64ArgTypes, X64ArgTypes);
+
+  Arm64Ty = FunctionType::get(Arm64RetTy, Arm64ArgTypes, false);
+  X64Ty = FunctionType::get(X64RetTy, X64ArgTypes, false);
+}
+
+void AArch64Arm64ECCallLowering::getThunkArgTypes(
+    FunctionType *FT, AttributeList AttrList, raw_ostream &Out,
+    SmallVectorImpl<Type *> &Arm64ArgTypes,
+    SmallVectorImpl<Type *> &X64ArgTypes) {
+  bool HasSretPtr = Arm64ArgTypes.size() > 1;
+
+  Out << "$";
+  if (FT->isVarArg()) {
+    // We treat the variadic function's thunk as a normal function
+    // with the following type on the ARM side:
+    //   rettype exitthunk(
+    //     ptr x9, ptr x0, i64 x1, i64 x2, i64 x3, ptr x4, i64 x5)
+    //
+    // that can coverage all types of variadic function.
+    // x9 is similar to normal exit thunk, store the called function.
+    // x0-x3 is the arguments be stored in registers.
+    // x4 is the address of the arguments on the stack.
+    // x5 is the size of the arguments on the stack.
+    //
+    // On the x64 side, it's the same except that x5 isn't set.
+    //
+    // If both the ARM and X64 sides are sret, there are only three
+    // arguments in registers.
+    //
+    // If the X64 side is sret, but the ARM side isn't, we pass an extra value
+    // to/from the X64 side, and let SelectionDAG transform it into a memory
+    // location.
+    Out << "varargs";
+
+    // x0-x3
+    for (int i = HasSretPtr ? 1 : 0; i < 4; i++) {
+      Arm64ArgTypes.push_back(I64Ty);
+      X64ArgTypes.push_back(I64Ty);
+    }
+
+    // x4
+    Arm64ArgTypes.push_back(I8PtrTy);
+    X64ArgTypes.push_back(I8PtrTy);
+    // x5
+    Arm64ArgTypes.push_back(I64Ty);
+    // FIXME: x5 isn't actually passed/used by the x64 side; revisit once we
+    // have proper isel for varargs
+    X64ArgTypes.push_back(I64Ty);
+    return;
+  }
+
+  unsigned I = 0;
+  if (HasSretPtr)
+    I++;
+
+  if (I == FT->getNumParams()) {
+    Out << "v";
+    return;
+  }
+
+  for (unsigned E = FT->getNumParams(); I != E; ++I) {
+    Align ParamAlign = AttrList.getParamAlignment(I).valueOrOne();
+#if 0
+    // FIXME: Need more information about argument size; see
+    // https://reviews.llvm.org/D132926
+    uint64_t ArgSizeBytes = AttrList.getParamArm64ECArgSizeBytes(I);
+#else
+    uint64_t ArgSizeBytes = 0;
+#endif
+    Type *Arm64Ty, *X64Ty;
+    canonicalizeThunkType(FT->getParamType(I), ParamAlign,
+                          /*Ret*/ false, ArgSizeBytes, Out, Arm64Ty, X64Ty);
+    Arm64ArgTypes.push_back(Arm64Ty);
+    X64ArgTypes.push_back(X64Ty);
+  }
+}
+
+void AArch64Arm64ECCallLowering::getThunkRetType(
+    FunctionType *FT, AttributeList AttrList, raw_ostream &Out,
+    Type *&Arm64RetTy, Type *&X64RetTy, SmallVectorImpl<Type *> &Arm64ArgTypes,
+    SmallVectorImpl<Type *> &X64ArgTypes) {
+  Type *T = FT->getReturnType();
+#if 0
+  // FIXME: Need more information about argument size; see
+  // https://reviews.llvm.org/D132926
+  uint64_t ArgSizeBytes = AttrList.getRetArm64ECArgSizeBytes();
+#else
+  int64_t ArgSizeBytes = 0;
+#endif
+  if (T->isVoidTy()) {
+    if (FT->getNumParams()) {
+      auto SRetAttr = AttrList.getParamAttr(0, Attribute::StructRet);
+      auto InRegAttr = AttrList.getParamAttr(0, Attribute::InReg);
+      if (SRetAttr.isValid() && InRegAttr.isValid()) {
+        // sret+inreg indicates a call that returns a C++ class value. This is
+        // actually equivalent to just passing and returning a void* pointer
+        // as the first argument. Translate it that way, instead of trying
+        // to model "inreg" in the thunk's calling convention, to simplify
+        // the rest of the code.
+        Out << "i8";
+        Arm64RetTy = I64Ty;
+        X64RetTy = I64Ty;
+        return;
+      }
+      if (SRetAttr.isValid()) {
+        Type *SRetType = SRetAttr.getValueAsType();
+        Align SRetAlign = AttrList.getParamAlignment(0).valueOrOne();
+        Type *Arm64Ty, *X64Ty;
+        canonicalizeThunkType(SRetType, SRetAlign, /*Ret*/ true,
+                              ArgSizeBytes, Out, Arm64Ty, X64Ty);
+        Arm64RetTy = VoidTy;
+        X64RetTy = VoidTy;
+        Arm64ArgTypes.push_back(FT->getParamType(0));
+        X64ArgTypes.push_back(FT->getParamType(0));
+        return;
+      }
+    }
+
+    Out << "v";
+    Arm64RetTy = VoidTy;
+    X64RetTy = VoidTy;
+    return;
+  }
+
+  canonicalizeThunkType(T, Align(), /*Ret*/ true, ArgSizeBytes, Out,
+                        Arm64RetTy, X64RetTy);
+  if (X64RetTy->isPointerTy()) {
+    // If the X64 type is canonicalized to a pointer, that means it's
+    // passed/returned indirectly. For a return value, that means it's an
+    // sret pointer.
+    X64ArgTypes.push_back(X64RetTy);
+    X64RetTy = VoidTy;
+  }
+}
+
+void AArch64Arm64ECCallLowering::canonicalizeThunkType(
+    Type *T, Align Alignment, bool Ret, uint64_t ArgSizeBytes,
+    raw_ostream &Out, Type *&Arm64Ty, Type *&X64Ty) {
+  if (T->isFloatTy()) {
+    Out << "f";
+    Arm64Ty = T;
+    X64Ty = T;
+    return;
+  }
+
+  if (T->isDoubleTy()) {
+    Out << "d";
+    Arm64Ty = T;
+    X64Ty = T;
+    return;
+  }
+
+  auto &DL = M->getDataLayout();
+
+  if (auto *StructTy = dyn_cast<StructType>(T))
+    if (StructTy->getNumElements() == 1)
+      T = StructTy->getElementType(0);
+
+  if (T->isArrayTy()) {
+    Type *ElementTy = T->getArrayElementType();
+    uint64_t ElementCnt = T->getArrayNumElements();
+    uint64_t ElementSizePerBytes = DL.getTypeSizeInBits(ElementTy) / 8;
+    uint64_t TotalSizeBytes = ElementCnt * ElementSizePerBytes;
+    if (ElementTy->isFloatTy() || ElementTy->isDoubleTy()) {
+      Out << (ElementTy->isFloatTy() ? "F" : "D") << TotalSizeBytes;
+      if (Alignment.value() >= 8 && !T->isPointerTy())
+        Out << "a" << Alignment.value();
+      Arm64Ty = T;
+      if (TotalSizeBytes <= 8) {
+        // Arm64 returns small structs of float/double in float registers;
+        // X64 uses RAX.
+        X64Ty = llvm::Type::getIntNTy(M->getContext(), TotalSizeBytes * 8);
+      } else {
+        // Struct is passed directly on Arm64, but indirectly on X64.
+        X64Ty = Arm64Ty->getPointerTo(0);
+      }
+      return;
+    }
+  }
+
+  if ((T->isIntegerTy() || T->isPointerTy()) && DL.getTypeSizeInBits(T) <= 64) {
+    Out << "i8";
+    Arm64Ty = I64Ty;
+    X64Ty = I64Ty;
+    return;
+  }
+
+  unsigned TypeSize = ArgSizeBytes;
+  if (TypeSize == 0)
+    TypeSize = DL.getTypeSizeInBits(T) / 8;
+  Out << "m";
+  if (TypeSize != 4)
+    Out << TypeSize;
+  if (Alignment.value() >= 8 && !T->isPointerTy())
+    Out << "a" << Alignment.value();
+  // FIXME: Try to canonicalize Arm64Ty more thoroughly?
+  Arm64Ty = T;
+  if (TypeSize == 1 || TypeSize == 2 || TypeSize == 4 || TypeSize == 8) {
+    // Pass directly in an integer register
+    X64Ty = llvm::Type::getIntNTy(M->getContext(), TypeSize * 8);
+  } else {
+    // Passed directly on Arm64, but indirectly on X64.
+    X64Ty = Arm64Ty->getPointerTo(0);
+  }
+}
+
+Function *AArch64Arm64ECCallLowering::buildExitThunk(FunctionType *FT,
+                                                     AttributeList Attrs) {
+  SmallString<256> ExitThunkName;
+  llvm::raw_svector_ostream ExitThunkStream(ExitThunkName);
+  FunctionType *Arm64Ty, *X64Ty;
+  getThunkType(FT, Attrs, /*EntryThunk*/ false, ExitThunkStream, Arm64Ty,
+               X64Ty);
+  if (Function *F = M->getFunction(ExitThunkName))
+    return F;
+
+  Function *F = Function::Create(Arm64Ty, GlobalValue::LinkOnceODRLinkage, 0,
+                                 ExitThunkName, M);
+  F->setCallingConv(CallingConv::ARM64EC_Thunk_Native);
+  F->setSection(".wowthk$aa");
+  F->setComdat(M->getOrInsertComdat(ExitThunkName));
+  // Copy MSVC, and always set up a frame pointer. (Maybe this isn't necessary.)
+  F->addFnAttr("frame-pointer", "all");
+  // Only copy sret from the first argument. For C++ instance methods, clang can
+  // stick an sret marking on a later argument, but it doesn't actually affect
+  // the ABI, so we can omit it. This avoids triggering a verifier assertion.
+  if (FT->getNumParams()) {
+    auto SRet = Attrs.getParamAttr(0, Attribute::StructRet);
+    auto InReg = Attrs.getParamAttr(0, Attribute::InReg);
+    if (SRet.isValid() && !InReg.isValid())
+      F->addParamAttr(1, SRet);
+  }
+  // FIXME: Copy anything other than sret?  Shouldn't be necessary for normal
+  // C ABI, but might show up in other cases.
+  BasicBlock *BB = BasicBlock::Create(M->getContext(), "", F);
+  IRBuilder<> IRB(BB);
+  PointerType *DispatchPtrTy =
+      FunctionType::get(IRB.getVoidTy(), false)->getPointerTo(0);
+  Value *CalleePtr = M->getOrInsertGlobal(
+      "__os_arm64x_dispatch_call_no_redirect", DispatchPtrTy);
+  Value *Callee = IRB.CreateLoad(DispatchPtrTy, CalleePtr);
+  auto &DL = M->getDataLayout();
+  SmallVector<Value *> Args;
+
+  // Pass the called function in x9.
+  Args.push_back(F->arg_begin());
+
+  Type *RetTy = Arm64Ty->getReturnType();
+  if (RetTy != X64Ty->getReturnType()) {
+    // If the return type is an array or struct, translate it. Values of size
+    // 8 or less go into RAX; bigger values go into memory, and we pass a
+    // pointer.
+    if (DL.getTypeStoreSize(RetTy) > 8) {
+      Args.push_back(IRB.CreateAlloca(RetTy));
+    }
+  }
+
+  for (auto &Arg : make_range(F->arg_begin() + 1, F->arg_end())) {
+    // Translate arguments from AArch64 calling convention to x86 calling
+    // convention.
+    //
+    // For simple types, we don't need to do any translation: they're
+    // represented the same way. (Implicit sign extension is not part of
+    // either convention.)
+    //
+    // The big thing we have to worry about is struct types... but
+    // fortunately AArch64 clang is pretty friendly here: the cases that need
+    // translation are always passed as a struct or array. (If we run into
+    // some cases where this doesn't work, we can teach clang to mark it up
+    // with an attribute.)
+    //
+    // The first argument is the called function, stored in x9.
+    if (Arg.getType()->isArrayTy() || Arg.getType()->isStructTy() ||
+        DL.getTypeStoreSize(Arg.getType()) > 8) {
+      Value *Mem = IRB.CreateAlloca(Arg.getType());
+      IRB.CreateStore(&Arg, Mem);
+      if (DL.getTypeStoreSize(Arg.getType()) <= 8) {
+        Type *IntTy = IRB.getIntNTy(DL.getTypeStoreSizeInBits(Arg.getType()));
+        Args.push_back(IRB.CreateLoad(
+            IntTy, IRB.CreateBitCast(Mem, IntTy->getPointerTo(0))));
+      } else
+        Args.push_back(Mem);
+    } else {
+      Args.push_back(&Arg);
+    }
+  }
+  // FIXME: Transfer necessary attributes? sret? anything else?
+
+  Callee = IRB.CreateBitCast(Callee, X64Ty->getPointerTo(0));
+  CallInst *Call = IRB.CreateCall(X64Ty, Callee, Args);
+  Call->setCallingConv(CallingConv::ARM64EC_Thunk_X64);
+
+  Value *RetVal = Call;
+  if (RetTy != X64Ty->getReturnType()) {
+    // If we rewrote the return type earlier, convert the return value to
+    // the proper type.
+    if (DL.getTypeStoreSize(RetTy) > 8) {
+      RetVal = IRB.CreateLoad(RetTy, Args[1]);
+    } else {
+      Value *CastAlloca = IRB.CreateAlloca(RetTy);
+      IRB.CreateStore(Call, IRB.CreateBitCast(
+                                CastAlloca, Call->getType()->getPointerTo(0)));
+      RetVal = IRB.CreateLoad(RetTy, CastAlloca);
+    }
+  }
+
+  if (RetTy->isVoidTy())
+    IRB.CreateRetVoid();
+  else
+    IRB.CreateRet(RetVal);
+  return F;
+}
+
+Function *AArch64Arm64ECCallLowering::buildEntryThunk(Function *F) {
+  SmallString<256> EntryThunkName;
+  llvm::raw_svector_ostream EntryThunkStream(EntryThunkName);
+  FunctionType *Arm64Ty, *X64Ty;
+  getThunkType(F->getFunctionType(), F->getAttributes(), /*EntryThunk*/ true,
+               EntryThunkStream, Arm64Ty, X64Ty);
+  if (Function *F = M->getFunction(EntryThunkName))
+    return F;
+
+  Function *Thunk = Function::Create(X64Ty, GlobalValue::LinkOnceODRLinkage, 0,
+                                     EntryThunkName, M);
+  Thunk->setCallingConv(CallingConv::ARM64EC_Thunk_X64);
+  Thunk->setSection(".wowthk$aa");
+  Thunk->setComdat(M->getOrInsertComdat(EntryThunkName));
+  // Copy MSVC, and always set up a frame pointer. (Maybe this isn't necessary.)
+  Thunk->addFnAttr("frame-pointer", "all");
+
+  auto &DL = M->getDataLayout();
+  BasicBlock *BB = BasicBlock::Create(M->getContext(), "", Thunk);
+  IRBuilder<> IRB(BB);
+
+  Type *RetTy = Arm64Ty->getReturnType();
+  Type *X64RetType = X64Ty->getReturnType();
+
+  bool TransformDirectToSRet = X64RetType->isVoidTy() && !RetTy->isVoidTy();
+  unsigned ThunkArgOffset = TransformDirectToSRet ? 2 : 1;
+
+  // Translate arguments to call.
+  SmallVector<Value *> Args;
+  for (unsigned i = ThunkArgOffset, e = Thunk->arg_size(); i != e; ++i) {
+    Value *Arg = Thunk->getArg(i);
+    Type *ArgTy = Arm64Ty->getParamType(i - ThunkArgOffset);
+    if (ArgTy->isArrayTy() || ArgTy->isStructTy() ||
+        DL.getTypeStoreSize(ArgTy) > 8) {
+      // Translate array/struct arguments to the expected type.
+      if (DL.getTypeStoreSize(ArgTy) <= 8) {
+        Value *CastAlloca = IRB.CreateAlloca(ArgTy);
+        IRB.CreateStore(Arg, IRB.CreateBitCast(
+                                 CastAlloca, Arg->getType()->getPointerTo(0)));
+        Arg = IRB.CreateLoad(ArgTy, CastAlloca);
+      } else {
+        Arg = IRB.CreateLoad(ArgTy,
+                             IRB.CreateBitCast(Arg, ArgTy->getPointerTo(0)));
+      }
+    }
+    Args.push_back(Arg);
+  }
+
+  // Call the function passed to the thunk.
+  Value *Callee = Thunk->getArg(0);
+  Callee = IRB.CreateBitCast(Callee, Arm64Ty->getPointerTo(0));
+  Value *Call = IRB.CreateCall(Arm64Ty, Callee, Args);
+
+  Value *RetVal = Call;
+  if (TransformDirectToSRet) {
+    IRB.CreateStore(RetVal,
+                    IRB.CreateBitCast(Thunk->getArg(1),
+                                      RetVal->getType()->getPointerTo(0)));
+  } else if (X64RetType != RetTy) {
+    Value *CastAlloca = IRB.CreateAlloca(X64RetType);
+    IRB.CreateStore(
+        Call, IRB.CreateBitCast(CastAlloca, Call->getType()->getPointerTo(0)));
+    RetVal = IRB.CreateLoad(X64RetType, CastAlloca);
+  }
+
+  // Return to the caller.  Note that the isel has code to translate this
+  // "ret" to a tail call to __os_arm64x_dispatch_ret.  (Alternatively, we
+  // could emit a tail call here, but that would require a dedicated calling
+  // convention, which seems more complicated overall.)
+  if (X64RetType->isVoidTy())
+    IRB.CreateRetVoid();
+  else
+    IRB.CreateRet(RetVal);
+
+  return Thunk;
+}
+
+Function *AArch64Arm64ECCallLowering::buildGuestExitThunk(Function *F) {
+  llvm::raw_null_ostream NullThunkName;
+  FunctionType *Arm64Ty, *X64Ty;
+  getThunkType(F->getFunctionType(), F->getAttributes(), /*EntryThunk*/ true,
+               NullThunkName, Arm64Ty, X64Ty);
+  auto MangledName = getArm64ECMangledFunctionName(F->getName().str());
+  assert(MangledName &&
+         "Can't guest exit to function that's already native");
+  std::string ThunkName = *MangledName;
+  if (ThunkName[0] == '?' && ThunkName.find("@") != std::string::npos) {
+    ThunkName.insert(ThunkName.find("@"), "$exit_thunk");
+  } else {
+    ThunkName.append("$exit_thunk");
+  }
+  assert(MangledName && "Can't guest exit to function that's already native");
+  Function *GuestExit = Function::Create(Arm64Ty, GlobalValue::WeakODRLinkage, 0,
+                                         ThunkName, M);
+  GuestExit->setComdat(M->getOrInsertComdat(ThunkName));
+  GuestExit->setSection(".wowthk$aa");
+  GuestExit->setMetadata("arm64ec_unmangled_name", MDNode::get(M->getContext(), MDString::get(M->getContext(), F->getName())));
+  GuestExit->setMetadata("arm64ec_ecmangled_name", MDNode::get(M->getContext(), MDString::get(M->getContext(), *MangledName)));
+  F->setMetadata("arm64ec_hasguestexit", MDNode::get(M->getContext(), {}));
+  BasicBlock *BB = BasicBlock::Create(M->getContext(), "", GuestExit);
+  IRBuilder<> B(BB);
+
+  // Load the global symbol as a pointer to the check function.
+  Value *GuardFn;
+  if (cfguard_module_flag == 2 && !F->hasFnAttribute("guard_nocf"))
+    GuardFn = GuardFnCFGlobal;
+  else
+    GuardFn = GuardFnGlobal;
+  LoadInst *GuardCheckLoad = B.CreateLoad(GuardFnPtrType, GuardFn);
+
+  // Create new call instruction. The CFGuard check should always be a call,
+  // even if the original CallBase is an Invoke or CallBr instruction.
+  Function *Thunk = buildExitThunk(F->getFunctionType(), F->getAttributes());
+  CallInst *GuardCheck =
+      B.CreateCall(GuardFnType, GuardCheckLoad,
+                   {B.CreateBitCast(F, B.getInt8PtrTy()),
+                    B.CreateBitCast(Thunk, B.getInt8PtrTy())});
+
+  // Ensure that the first argument is passed in the correct register
+  // (e.g. ECX on 32-bit X86 targets).
+  GuardCheck->setCallingConv(CallingConv::CFGuard_Check);
+
+  Value *GuardRetVal = B.CreateBitCast(GuardCheck, Arm64Ty->getPointerTo(0));
+  SmallVector<Value *> Args;
+  for (Argument &Arg : GuestExit->args())
+    Args.push_back(&Arg);
+  CallInst *Call = B.CreateCall(Arm64Ty, GuardRetVal, Args);
+  Call->setTailCallKind(llvm::CallInst::TCK_MustTail);
+
+  if (Call->getType()->isVoidTy())
+    B.CreateRetVoid();
+  else
+    B.CreateRet(Call);
+
+  auto SRetAttr = F->getAttributes().getParamAttr(0, Attribute::StructRet);
+  auto InRegAttr = F->getAttributes().getParamAttr(0, Attribute::InReg);
+  if (SRetAttr.isValid() && !InRegAttr.isValid()) {
+    GuestExit->addParamAttr(0, SRetAttr);
+    Call->addParamAttr(0, SRetAttr);
+  }
+
+  return GuestExit;
+}
+
+void AArch64Arm64ECCallLowering::lowerCall(CallBase *CB) {
+  assert(Triple(CB->getModule()->getTargetTriple()).isOSWindows() &&
+         "Only applicable for Windows targets");
+
+  IRBuilder<> B(CB);
+  Value *CalledOperand = CB->getCalledOperand();
+
+  // If the indirect call is called within catchpad or cleanuppad,
+  // we need to copy "funclet" bundle of the call.
+  SmallVector<llvm::OperandBundleDef, 1> Bundles;
+  if (auto Bundle = CB->getOperandBundle(LLVMContext::OB_funclet))
+    Bundles.push_back(OperandBundleDef(*Bundle));
+
+  // Load the global symbol as a pointer to the check function.
+  Value *GuardFn;
+  if (cfguard_module_flag == 2 && !CB->hasFnAttr("guard_nocf"))
+    GuardFn = GuardFnCFGlobal;
+  else
+    GuardFn = GuardFnGlobal;
+  LoadInst *GuardCheckLoad = B.CreateLoad(GuardFnPtrType, GuardFn);
+
+  // Create new call instruction. The CFGuard check should always be a call,
+  // even if the original CallBase is an Invoke or CallBr instruction.
+  Function *Thunk = buildExitThunk(CB->getFunctionType(), CB->getAttributes());
+  CallInst *GuardCheck =
+      B.CreateCall(GuardFnType, GuardCheckLoad,
+                   {B.CreateBitCast(CalledOperand, B.getInt8PtrTy()),
+                    B.CreateBitCast(Thunk, B.getInt8PtrTy())},
+                   Bundles);
+
+  // Ensure that the first argument is passed in the correct register
+  // (e.g. ECX on 32-bit X86 targets).
+  GuardCheck->setCallingConv(CallingConv::CFGuard_Check);
+
+  Value *GuardRetVal = B.CreateBitCast(GuardCheck, CalledOperand->getType());
+  CB->setCalledOperand(GuardRetVal);
+}
+
+bool AArch64Arm64ECCallLowering::runOnModule(Module &Mod) {
+  if (!GenerateThunks)
+    return false;
+
+  M = &Mod;
+
+  // Check if this module has the cfguard flag and read its value.
+  if (auto *MD =
+          mdconst::extract_or_null<ConstantInt>(M->getModuleFlag("cfguard")))
+    cfguard_module_flag = MD->getZExtValue();
+
+  I8PtrTy = Type::getInt8PtrTy(M->getContext());
+  I64Ty = Type::getInt64Ty(M->getContext());
+  VoidTy = Type::getVoidTy(M->getContext());
+
+  GuardFnType = FunctionType::get(I8PtrTy, {I8PtrTy, I8PtrTy}, false);
+  GuardFnPtrType = PointerType::get(GuardFnType, 0);
+  GuardFnCFGlobal =
+      M->getOrInsertGlobal("__os_arm64x_check_icall_cfg", GuardFnPtrType);
+  GuardFnGlobal =
+      M->getOrInsertGlobal("__os_arm64x_check_icall", GuardFnPtrType);
+
+  SetVector<Function *> DirectCalledFns;
+  for (Function &F : Mod)
+    if (!F.isDeclaration() &&
+        F.getCallingConv() != CallingConv::ARM64EC_Thunk_Native &&
+        F.getCallingConv() != CallingConv::ARM64EC_Thunk_X64)
+      processFunction(F, DirectCalledFns);
+
+  struct ThunkInfo {
+    Constant *Src;
+    Constant *Dst;
+    unsigned Kind;
+  };
+  SmallVector<ThunkInfo> ThunkMapping;
+  for (Function &F : Mod) {
+    if (!F.isDeclaration() && (!F.hasLocalLinkage() || F.hasAddressTaken()) &&
+        F.getCallingConv() != CallingConv::ARM64EC_Thunk_Native &&
+        F.getCallingConv() != CallingConv::ARM64EC_Thunk_X64) {
+      if (!F.hasComdat())
+        F.setComdat(Mod.getOrInsertComdat(F.getName()));
+      ThunkMapping.push_back({&F, buildEntryThunk(&F), 1});
+    }
+  }
+  for (Function *F : DirectCalledFns) {
+    ThunkMapping.push_back(
+        {F, buildExitThunk(F->getFunctionType(), F->getAttributes()), 4});
+    if (!F->hasDLLImportStorageClass())
+      ThunkMapping.push_back({buildGuestExitThunk(F), F, 0});
+  }
+
+  if (!ThunkMapping.empty()) {
+    Type *VoidPtr = Type::getInt8PtrTy(M->getContext());
+    SmallVector<Constant *> ThunkMappingArrayElems;
+    for (ThunkInfo &Thunk : ThunkMapping) {
+      ThunkMappingArrayElems.push_back(ConstantStruct::getAnon(
+          {ConstantExpr::getBitCast(Thunk.Src, VoidPtr),
+           ConstantExpr::getBitCast(Thunk.Dst, VoidPtr),
+           ConstantInt::get(M->getContext(), APInt(32, Thunk.Kind))}));
+    }
+    Constant *ThunkMappingArray = ConstantArray::get(
+        llvm::ArrayType::get(ThunkMappingArrayElems[0]->getType(),
+                             ThunkMappingArrayElems.size()),
+        ThunkMappingArrayElems);
+    new GlobalVariable(Mod, ThunkMappingArray->getType(), /*isConstant*/ false,
+                       GlobalValue::ExternalLinkage, ThunkMappingArray,
+                       "llvm.arm64ec.symbolmap");
+  }
+
+  return true;
+}
+
+bool AArch64Arm64ECCallLowering::processFunction(
+    Function &F, SetVector<Function *> &DirectCalledFns) {
+  SmallVector<CallBase *, 8> IndirectCalls;
+
+  // For ARM64EC targets, a function definition's name is mangled differently
+  // from the normal symbol. We currently have no representation of this sort
+  // of symbol in IR, so we change the name to the mangled name, then store
+  // the unmangled name as metadata.  Later passes that need the unmangled
+  // name (emitting the definition) can grab it from the metadata.
+  //
+  // FIXME: Handle functions with weak linkage?
+  if (F.hasExternalLinkage() || F.hasWeakLinkage() || F.hasLinkOnceLinkage()) {
+    if (std::optional<std::string> MangledName =
+            getArm64ECMangledFunctionName(F.getName().str())) {
+      F.setMetadata("arm64ec_unmangled_name", MDNode::get(M->getContext(), MDString::get(M->getContext(), F.getName())));
+      if (F.hasComdat() && F.getComdat()->getName() == F.getName()) {
+        Comdat *MangledComdat = M->getOrInsertComdat(MangledName.value());
+        SmallVector<GlobalObject*> ComdatUsers = to_vector(F.getComdat()->getUsers());
+        for (GlobalObject *User : ComdatUsers)
+          User->setComdat(MangledComdat);
+      }
+      F.setName(MangledName.value());
+    }
+  }
+
+  // Iterate over the instructions to find all indirect call/invoke/callbr
+  // instructions. Make a separate list of pointers to indirect
+  // call/invoke/callbr instructions because the original instructions will be
+  // deleted as the checks are added.
+  for (BasicBlock &BB : F) {
+    for (Instruction &I : BB) {
+      auto *CB = dyn_cast<CallBase>(&I);
+      if (!CB || CB->getCallingConv() == CallingConv::ARM64EC_Thunk_X64 ||
+          CB->isInlineAsm())
+        continue;
+
+      // We need to instrument any call that isn't directly calling an
+      // ARM64 function.
+      //
+      // FIXME: getCalledFunction() fails if there's a bitcast (e.g.
+      // unprototyped functions in C)
+      if (Function *F = CB->getCalledFunction()) {
+        if (!LowerDirectToIndirect || F->hasLocalLinkage() || F->isIntrinsic() ||
+            !F->isDeclaration())
+          continue;
+
+        DirectCalledFns.insert(F);
+        continue;
+      }
+
+      IndirectCalls.push_back(CB);
+      ++Arm64ECCallsLowered;
+    }
+  }
+
+  if (IndirectCalls.empty())
+    return false;
+
+  for (CallBase *CB : IndirectCalls)
+    lowerCall(CB);
+
+  return true;
+}
+
+char AArch64Arm64ECCallLowering::ID = 0;
+INITIALIZE_PASS(AArch64Arm64ECCallLowering, "Arm64ECCallLowering",
+                "AArch64Arm64ECCallLowering", false, false)
+
+ModulePass *llvm::createAArch64Arm64ECCallLoweringPass() {
+  return new AArch64Arm64ECCallLowering;
+}

llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp

bool runOnMachineFunction(MachineFunction &MF) override {
 
     // Emit the XRay table for this function.
     emitXRayTable();
 
     // We didn't modify anything.
     return false;
   }
 
+  const MCExpr *lowerConstant(const Constant *CV) override;
+
 private:
   void printOperand(const MachineInstr *MI, unsigned OpNum, raw_ostream &O);
   bool printAsmMRegister(const MachineOperand &MO, char Mode, raw_ostream &O);
   bool printAsmRegInClass(const MachineOperand &MO,
                           const TargetRegisterClass *RC, unsigned AltName,
                           raw_ostream &O);
 
   bool PrintAsmOperand(const MachineInstr *MI, unsigned OpNum,
if (MF->getFunction().getCallingConv() == CallingConv::AArch64_VectorCall ||
       MF->getFunction().getCallingConv() ==
           CallingConv::AArch64_SVE_VectorCall ||
       MF->getInfo<AArch64FunctionInfo>()->isSVECC()) {
     auto *TS =
         static_cast<AArch64TargetStreamer *>(OutStreamer->getTargetStreamer());
     TS->emitDirectiveVariantPCS(CurrentFnSym);
   }
 
+  if (TM.getTargetTriple().isWindowsArm64EC()) {
+    // For ARM64EC targets, a function definition's name is mangled differently
+    // from the normal symbol. We emit the alias from the unmangled symbol to
+    // mangled symbol name here.
+    if (MDNode *Unmangled = MF->getFunction().getMetadata("arm64ec_unmangled_name")) {
+      AsmPrinter::emitFunctionEntryLabel();
+
+      if (MDNode *ECMangled = MF->getFunction().getMetadata("arm64ec_ecmangled_name")) {
+        StringRef UnmangledStr =
+            cast<MDString>(Unmangled->getOperand(0))->getString();
+        MCSymbol *UnmangledSym =
+            MMI->getContext().getOrCreateSymbol(UnmangledStr);
+        StringRef ECMangledStr =
+            cast<MDString>(ECMangled->getOperand(0))->getString();
+        MCSymbol *ECMangledSym =
+            MMI->getContext().getOrCreateSymbol(ECMangledStr);
+        OutStreamer->emitSymbolAttribute(UnmangledSym, MCSA_WeakAntiDep);
+        OutStreamer->emitAssignment(
+            UnmangledSym,
+            MCSymbolRefExpr::create(ECMangledSym, MCSymbolRefExpr::VK_WEAKREF,
+                                    MMI->getContext()));
+        OutStreamer->emitSymbolAttribute(ECMangledSym, MCSA_WeakAntiDep);
+        OutStreamer->emitAssignment(
+            ECMangledSym,
+            MCSymbolRefExpr::create(CurrentFnSym, MCSymbolRefExpr::VK_WEAKREF,
+                                    MMI->getContext()));
+        return;
+      } else {
+        StringRef UnmangledStr =
+            cast<MDString>(Unmangled->getOperand(0))->getString();
+        MCSymbol *UnmangledSym =
+            MMI->getContext().getOrCreateSymbol(UnmangledStr);
+        OutStreamer->emitSymbolAttribute(UnmangledSym, MCSA_WeakAntiDep);
+        OutStreamer->emitAssignment(
+            UnmangledSym,
+            MCSymbolRefExpr::create(CurrentFnSym, MCSymbolRefExpr::VK_WEAKREF,
+                                    MMI->getContext()));
+        return;
+      }
+    }
+  }
+
   return AsmPrinter::emitFunctionEntryLabel();
 }
 
 /// Small jump tables contain an unsigned byte or half, representing the offset
 /// from the lowest-addressed possible destination to the desired basic
 /// block. Since all instructions are 4-byte aligned, this is further compressed
 /// by counting in instructions rather than bytes (i.e. divided by 4). So, to
 /// materialize the correct destination we need:
void AArch64AsmPrinter::emitInstruction(const MachineInstr *MI) {
 
   case AArch64::SEH_EpilogEnd:
     TS->emitARM64WinCFIEpilogEnd();
     return;
 
   case AArch64::SEH_PACSignLR:
     TS->emitARM64WinCFIPACSignLR();
     return;
+
+  case AArch64::SEH_SaveAnyRegQP:
+    assert(MI->getOperand(1).getImm() - MI->getOperand(0).getImm() == 1 &&
+           "Non-consecutive registers not allowed for save_any_reg");
+    assert(MI->getOperand(2).getImm() >= 0 &&
+           "SaveAnyRegQP SEH opcode offset must be non-negative");
+    assert(MI->getOperand(2).getImm() <= 1008 &&
+           "SaveAnyRegQP SEH opcode offset must fit into 6 bits");
+    TS->emitARM64WinCFISaveAnyRegQP(MI->getOperand(0).getImm(),
+                                    MI->getOperand(2).getImm());
+    return;
+
+  case AArch64::SEH_SaveAnyRegQPX:
+    assert(MI->getOperand(1).getImm() - MI->getOperand(0).getImm() == 1 &&
+           "Non-consecutive registers not allowed for save_any_reg");
+    assert(MI->getOperand(2).getImm() < 0 &&
+           "SaveAnyRegQPX SEH opcode offset must be negative");
+    assert(MI->getOperand(2).getImm() >= -1008 &&
+           "SaveAnyRegQPX SEH opcode offset must fit into 6 bits");
+    TS->emitARM64WinCFISaveAnyRegQPX(MI->getOperand(0).getImm(),
+                                     -MI->getOperand(2).getImm());
+    return;
   }
 
   // Finally, do the automated lowerings for everything else.
   MCInst TmpInst;
   MCInstLowering.Lower(MI, TmpInst);
   EmitToStreamer(*OutStreamer, TmpInst);
 }
 
+const MCExpr *AArch64AsmPrinter::lowerConstant(const Constant *CV) {
+  if (const GlobalValue *GV = dyn_cast<GlobalValue>(CV)) {
+    return MCSymbolRefExpr::create(MCInstLowering.GetGlobalValueSymbol(GV, 0),
+                                   OutContext);
+  }
+
+  return AsmPrinter::lowerConstant(CV);
+}
+
 // Force static initialization.
 extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAArch64AsmPrinter() {
   RegisterAsmPrinter<AArch64AsmPrinter> X(getTheAArch64leTarget());
   RegisterAsmPrinter<AArch64AsmPrinter> Y(getTheAArch64beTarget());
   RegisterAsmPrinter<AArch64AsmPrinter> Z(getTheARM64Target());
   RegisterAsmPrinter<AArch64AsmPrinter> W(getTheARM64_32Target());
   RegisterAsmPrinter<AArch64AsmPrinter> V(getTheAArch64_32Target());
 }

llvm/lib/Target/AArch64/AArch64CallingConvention.h

 
 namespace llvm {
 bool CC_AArch64_AAPCS(unsigned ValNo, MVT ValVT, MVT LocVT,
                       CCValAssign::LocInfo LocInfo, ISD::ArgFlagsTy ArgFlags,
                       CCState &State);
 bool CC_AArch64_Arm64EC_VarArg(unsigned ValNo, MVT ValVT, MVT LocVT,
                                CCValAssign::LocInfo LocInfo,
                                ISD::ArgFlagsTy ArgFlags, CCState &State);
+bool CC_AArch64_Arm64EC_Thunk(unsigned ValNo, MVT ValVT, MVT LocVT,
+                              CCValAssign::LocInfo LocInfo,
+                              ISD::ArgFlagsTy ArgFlags, CCState &State);
+bool CC_AArch64_Arm64EC_Thunk_Native(unsigned ValNo, MVT ValVT, MVT LocVT,
+                                     CCValAssign::LocInfo LocInfo,
+                                     ISD::ArgFlagsTy ArgFlags, CCState &State);
 bool CC_AArch64_DarwinPCS_VarArg(unsigned ValNo, MVT ValVT, MVT LocVT,
                                  CCValAssign::LocInfo LocInfo,
                                  ISD::ArgFlagsTy ArgFlags, CCState &State);
 bool CC_AArch64_DarwinPCS(unsigned ValNo, MVT ValVT, MVT LocVT,
                           CCValAssign::LocInfo LocInfo,
                           ISD::ArgFlagsTy ArgFlags, CCState &State);
 bool CC_AArch64_DarwinPCS_ILP32_VarArg(unsigned ValNo, MVT ValVT, MVT LocVT,
                           CCValAssign::LocInfo LocInfo,
                           ISD::ArgFlagsTy ArgFlags, CCState &State);
 bool CC_AArch64_Win64_VarArg(unsigned ValNo, MVT ValVT, MVT LocVT,
                              CCValAssign::LocInfo LocInfo,
                              ISD::ArgFlagsTy ArgFlags, CCState &State);
 bool CC_AArch64_Win64_CFGuard_Check(unsigned ValNo, MVT ValVT, MVT LocVT,
                                     CCValAssign::LocInfo LocInfo,
                                     ISD::ArgFlagsTy ArgFlags, CCState &State);
+bool CC_AArch64_Arm64EC_CFGuard_Check(unsigned ValNo, MVT ValVT, MVT LocVT,
+                                      CCValAssign::LocInfo LocInfo,
+                                      ISD::ArgFlagsTy ArgFlags, CCState &State);
 bool CC_AArch64_WebKit_JS(unsigned ValNo, MVT ValVT, MVT LocVT,
                           CCValAssign::LocInfo LocInfo,
                           ISD::ArgFlagsTy ArgFlags, CCState &State);
 bool CC_AArch64_GHC(unsigned ValNo, MVT ValVT, MVT LocVT,
                     CCValAssign::LocInfo LocInfo, ISD::ArgFlagsTy ArgFlags,
                     CCState &State);
 bool RetCC_AArch64_AAPCS(unsigned ValNo, MVT ValVT, MVT LocVT,
                          CCValAssign::LocInfo LocInfo, ISD::ArgFlagsTy ArgFlags,
                          CCState &State);
+bool RetCC_AArch64_Arm64EC_Thunk(unsigned ValNo, MVT ValVT, MVT LocVT,
+                                 CCValAssign::LocInfo LocInfo,
+                                 ISD::ArgFlagsTy ArgFlags, CCState &State);
+bool RetCC_AArch64_Arm64EC_CFGuard_Check(unsigned ValNo, MVT ValVT, MVT LocVT,
+                                         CCValAssign::LocInfo LocInfo,
+                                         ISD::ArgFlagsTy ArgFlags,
+                                         CCState &State);
 bool RetCC_AArch64_WebKit_JS(unsigned ValNo, MVT ValVT, MVT LocVT,
                              CCValAssign::LocInfo LocInfo,
                              ISD::ArgFlagsTy ArgFlags, CCState &State);
 } // namespace llvm
 
 #endif

llvm/lib/Target/AArch64/AArch64CallingConvention.td

def CC_AArch64_Arm64EC_VarArg : CallingConv<[
   // Pass first four arguments in x0-x3.
   CCIfType<[i32], CCAssignToReg<[W0, W1, W2, W3]>>,
   CCIfType<[i64], CCAssignToReg<[X0, X1, X2, X3]>>,
 
   // Put remaining arguments on stack.
   CCIfType<[i32, i64], CCAssignToStack<8, 8>>,
 ]>;
 
+// Arm64EC thunks use a calling convention that's precisely the x64 calling
+// convention, except that the registers have different names, and the callee
+// address is passed in X9.
+let Entry = 1 in
+def CC_AArch64_Arm64EC_Thunk : CallingConv<[
+  // Byval aggregates are passed by pointer
+  CCIfByVal<CCPassIndirect<i64>>,
+
+  // ARM64EC-specific: promote small integers to i32. (x86 only promotes i1,
+  // but that would confuse ARM64 lowering code.)
+  CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
+
+  // The 'nest' parameter, if any, is passed in R10 (X4).
+  CCIfNest<CCAssignToReg<[X4]>>,
+
+  // A SwiftError is passed in R12 (X19).
+  CCIfSwiftError<CCIfType<[i64], CCAssignToReg<[X19]>>>,
+
+  // Pass SwiftSelf in R13 (X20).
+  CCIfSwiftSelf<CCIfType<[i64], CCAssignToReg<[X20]>>>,
+
+  // Pass SwiftAsync in an otherwise callee saved register so that calls to
+  // normal functions don't need to save it somewhere.
+  CCIfSwiftAsync<CCIfType<[i64], CCAssignToReg<[X21]>>>,
+
+  // The 'CFGuardTarget' parameter, if any, is passed in RAX (R8).
+  CCIfCFGuardTarget<CCAssignToReg<[X8]>>,
+
+  // 128 bit vectors are passed by pointer
+  CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64], CCPassIndirect<i64>>,
+
+  // 256 bit vectors are passed by pointer
+  CCIfType<[v32i8, v16i16, v8i32, v4i64, v16f16, v8f32, v4f64], CCPassIndirect<i64>>,
+
+  // 512 bit vectors are passed by pointer
+  CCIfType<[v64i8, v32i16, v16i32, v32f16, v16f32, v8f64, v8i64], CCPassIndirect<i64>>,
+
+  // Long doubles are passed by pointer
+  CCIfType<[f80], CCPassIndirect<i64>>,
+
+  // The first 4 MMX vector arguments are passed in GPRs.
+  CCIfType<[x86mmx], CCBitConvertToType<i64>>,
+
+  // The first 4 FP/Vector arguments are passed in XMM registers.
+  CCIfType<[f16],
+           CCAssignToRegWithShadow<[H0, H1, H2, H3],
+                                   [X0, X1, X2, X3]>>,
+  CCIfType<[f32],
+           CCAssignToRegWithShadow<[S0, S1, S2, S3],
+                                   [X0, X1, X2, X3]>>,
+  CCIfType<[f64],
+           CCAssignToRegWithShadow<[D0, D1, D2, D3],
+                                   [X0, X1, X2, X3]>>,
+
+  // The first 4 integer arguments are passed in integer registers.
+  CCIfType<[i32], CCAssignToRegWithShadow<[W0, W1, W2, W3],
+                                          [Q0, Q1, Q2, Q3]>>,
+
+  // Arm64EC thunks: the first argument is always a pointer to the destination
+  // address, stored in x9.
+  CCIfType<[i64], CCAssignToReg<[X9]>>,
+
+  CCIfType<[i64], CCAssignToRegWithShadow<[X0, X1, X2, X3],
+                                          [Q0, Q1, Q2, Q3]>>,
+
+  // Integer/FP values get stored in stack slots that are 8 bytes in size and
+  // 8-byte aligned if there are no more registers to hold them.
+  CCIfType<[i8, i16, i32, i64, f16, f32, f64], CCAssignToStack<8, 8>>
+]>;
+
+// The native side of ARM64EC thunks
+let Entry = 1 in
+def CC_AArch64_Arm64EC_Thunk_Native : CallingConv<[
+  CCIfType<[i64], CCAssignToReg<[X9]>>,
+  CCDelegateTo<CC_AArch64_AAPCS>
+]>;
+
+let Entry = 1 in
+def RetCC_AArch64_Arm64EC_Thunk : CallingConv<[
+  // The X86-Win64 calling convention always returns __m64 values in RAX.
+  CCIfType<[x86mmx], CCBitConvertToType<i64>>,
+
+  // Otherwise, everything is the same as 'normal' X86-64 C CC.
+
+  // The X86-64 calling convention always returns FP values in XMM0.
+  CCIfType<[f16], CCAssignToReg<[H0, H1]>>,
+  CCIfType<[f32], CCAssignToReg<[S0, S1]>>,
+  CCIfType<[f64], CCAssignToReg<[D0, D1]>>,
+  CCIfType<[f128], CCAssignToReg<[Q0, Q1]>>,
+
+  CCIfSwiftError<CCIfType<[i64], CCAssignToReg<[X19]>>>,
+
+  // Scalar values are returned in AX first, then DX.  For i8, the ABI
+  // requires the values to be in AL and AH, however this code uses AL and DL
+  // instead. This is because using AH for the second register conflicts with
+  // the way LLVM does multiple return values -- a return of {i16,i8} would end
+  // up in AX and AH, which overlap. Front-ends wishing to conform to the ABI
+  // for functions that return two i8 values are currently expected to pack the
+  // values into an i16 (which uses AX, and thus AL:AH).
+  //
+  // For code that doesn't care about the ABI, we allow returning more than two
+  // integer values in registers.
+  CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
+  CCIfType<[i32], CCAssignToReg<[W8, W1, W0]>>,
+  CCIfType<[i64], CCAssignToReg<[X8, X1, X0]>>,
+
+  // Vector types are returned in XMM0 and XMM1, when they fit.  XMM2 and XMM3
+  // can only be used by ABI non-compliant code. If the target doesn't have XMM
+  // registers, it won't have vector types.
+  CCIfType<[v16i8, v8i16, v4i32, v2i64, v8f16, v4f32, v2f64],
+            CCAssignToReg<[Q0, Q1, Q2, Q3]>>
+]>;
+
 // Windows Control Flow Guard checks take a single argument (the target function
 // address) and have no return value.
 let Entry = 1 in
 def CC_AArch64_Win64_CFGuard_Check : CallingConv<[
   CCIfType<[i64], CCAssignToReg<[X15]>>
 ]>;
 
+let Entry = 1 in
+def CC_AArch64_Arm64EC_CFGuard_Check : CallingConv<[
+  CCIfType<[i64], CCAssignToReg<[X11, X10]>>
+]>;
+
+let Entry = 1 in
+def RetCC_AArch64_Arm64EC_CFGuard_Check : CallingConv<[
+  CCIfType<[i64], CCAssignToReg<[X11]>>
+]>;
+
 
 // Darwin uses a calling convention which differs in only two ways
 // from the standard one at this level:
 //     + i128s (i.e. split i64s) don't need even registers.
 //     + Stack slots are sized as needed rather than being at least 64-bit.
 let Entry = 1 in
 def CC_AArch64_DarwinPCS : CallingConv<[
   CCIfType<[iPTR], CCBitConvertToType<i64>>,
def CSR_Win_AArch64_AAPCS_SwiftTail
     : CalleeSavedRegs<(sub CSR_Win_AArch64_AAPCS, X20, X22)>;
 
 // The Control Flow Guard check call uses a custom calling convention that also
 // preserves X0-X8 and Q0-Q7.
 def CSR_Win_AArch64_CFGuard_Check : CalleeSavedRegs<(add CSR_Win_AArch64_AAPCS,
                                                (sequence "X%u", 0, 8),
                                                (sequence "Q%u", 0, 7))>;
 
+// To match the x64 calling convention, Arm64EC thunks preserve q6-q15.
+def CSR_Win_AArch64_Arm64EC_Thunk : CalleeSavedRegs<(add (sequence "Q%u", 6, 15),
+                                                         X19, X20, X21, X22, X23, X24,
+                                                         X25, X26, X27, X28, FP, LR)>;
+
 // AArch64 PCS for vector functions (VPCS)
 // must (additionally) preserve full Q8-Q23 registers
 def CSR_AArch64_AAVPCS : CalleeSavedRegs<(add X19, X20, X21, X22, X23, X24,
                                           X25, X26, X27, X28, LR, FP,
                                           (sequence "Q%u", 8, 23))>;
 
 // Functions taking SVE arguments or returning an SVE type
 // must (additionally) preserve full Z8-Z23 and predicate registers P4-P15

llvm/lib/Target/AArch64/AArch64FastISel.cpp

bool AArch64FastISel::fastLowerCall(CallLoweringInfo &CLI) {
   // FIXME: Add large code model support for ELF.
   if (CM == CodeModel::Large && !Subtarget->isTargetMachO())
     return false;
 
   // Let SDISel handle vararg functions.
   if (IsVarArg)
     return false;
 
+  if (Subtarget->isWindowsArm64EC())
+    return false;
+
   for (auto Flag : CLI.OutFlags)
     if (Flag.isInReg() || Flag.isSRet() || Flag.isNest() || Flag.isByVal() ||
         Flag.isSwiftSelf() || Flag.isSwiftAsync() || Flag.isSwiftError())
       return false;
 
   // Set up the argument vectors.
   SmallVector<MVT, 16> OutVTs;
   OutVTs.reserve(CLI.OutVals.size());

llvm/lib/Target/AArch64/AArch64FrameLowering.cpp

static MachineBasicBlock::iterator InsertSEH(MachineBasicBlock::iterator MBBI,
   case AArch64::LDRDui: {
     unsigned Reg = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
     MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFReg))
               .addImm(Reg)
               .addImm(Imm * 8)
               .setMIFlag(Flag);
     break;
   }
+  case AArch64::STPQi:
+  case AArch64::LDPQi: {
+    unsigned Reg0 = RegInfo->getSEHRegNum(MBBI->getOperand(0).getReg());
+    unsigned Reg1 = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
+    MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveAnyRegQP))
+              .addImm(Reg0)
+              .addImm(Reg1)
+              .addImm(Imm * 16)
+              .setMIFlag(Flag);
+    break;
+  }
+  case AArch64::LDPQpost:
+    Imm = -Imm;
+    LLVM_FALLTHROUGH;
+  case AArch64::STPQpre: {
+    unsigned Reg0 = RegInfo->getSEHRegNum(MBBI->getOperand(1).getReg());
+    unsigned Reg1 = RegInfo->getSEHRegNum(MBBI->getOperand(2).getReg());
+    MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveAnyRegQPX))
+              .addImm(Reg0)
+              .addImm(Reg1)
+              .addImm(Imm * 16)
+              .setMIFlag(Flag);
+    break;
+  }
   }
   auto I = MBB->insertAfter(MBBI, MIB);
   return I;
 }
 
 // Fix up the SEH opcode associated with the save/restore instruction.
 static void fixupSEHOpcode(MachineBasicBlock::iterator MBBI,
                            unsigned LocalStackSize) {
   MachineOperand *ImmOpnd = nullptr;
   unsigned ImmIdx = MBBI->getNumOperands() - 1;
   switch (MBBI->getOpcode()) {
   default:
     llvm_unreachable("Fix the offset in the SEH instruction");
   case AArch64::SEH_SaveFPLR:
   case AArch64::SEH_SaveRegP:
   case AArch64::SEH_SaveReg:
   case AArch64::SEH_SaveFRegP:
   case AArch64::SEH_SaveFReg:
+  case AArch64::SEH_SaveAnyRegQP:
+  case AArch64::SEH_SaveAnyRegQPX:
     ImmOpnd = &MBBI->getOperand(ImmIdx);
     break;
   }
   if (ImmOpnd)
     ImmOpnd->setImm(ImmOpnd->getImm() + LocalStackSize);
 }
 
 // Convert callee-save register save/restore instruction to do stack pointer

llvm/lib/Target/AArch64/AArch64ISelLowering.h

enum NodeType : unsigned {
   CALL_RVMARKER,
 
   CALL_BTI, // Function call followed by a BTI instruction.
 
   SMSTART,
   SMSTOP,
   RESTORE_ZA,
 
+  // A call with the callee in x16, i.e. "blr x16".
+  CALL_ARM64EC_TO_X64,
+
   // Produces the full sequence of instructions for getting the thread pointer
   // offset of a variable into X0, using the TLSDesc model.
   TLSDESC_CALLSEQ,
   ADRP,     // Page address of a TargetGlobalAddress operand.
   ADR,      // ADR
   ADDlow,   // Add the low 12 bits of a TargetGlobalAddress operand.
   LOADgot,  // Load from automatically generated descriptor (e.g. Global
             // Offset Table, TLS record).
private:
   SDValue getTargetNode(GlobalAddressSDNode *N, EVT Ty, SelectionDAG &DAG,
                         unsigned Flag) const;
   SDValue getTargetNode(JumpTableSDNode *N, EVT Ty, SelectionDAG &DAG,
                         unsigned Flag) const;
   SDValue getTargetNode(ConstantPoolSDNode *N, EVT Ty, SelectionDAG &DAG,
                         unsigned Flag) const;
   SDValue getTargetNode(BlockAddressSDNode *N, EVT Ty, SelectionDAG &DAG,
                         unsigned Flag) const;
+  SDValue getTargetNode(ExternalSymbolSDNode *N, EVT Ty, SelectionDAG &DAG,
+                        unsigned Flag) const;
   template <class NodeTy>
   SDValue getGOT(NodeTy *N, SelectionDAG &DAG, unsigned Flags = 0) const;
   template <class NodeTy>
   SDValue getAddrLarge(NodeTy *N, SelectionDAG &DAG, unsigned Flags = 0) const;
   template <class NodeTy>
   SDValue getAddr(NodeTy *N, SelectionDAG &DAG, unsigned Flags = 0) const;
   template <class NodeTy>
   SDValue getAddrTiny(NodeTy *N, SelectionDAG &DAG, unsigned Flags = 0) const;

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Subtarget->hasMOPS() && Subtarget->hasMTE()) {
     setOperationAction(ISD::INTRINSIC_W_CHAIN, MVT::i8, Custom);
   }
 
   setOperationAction(ISD::INTRINSIC_VOID, MVT::Other, Custom);
 
   PredictableSelectIsExpensive = Subtarget->predictableSelectIsExpensive();
 
   IsStrictFPEnabled = true;
+
+  if (Subtarget->isWindowsArm64EC()) {
+    // FIXME: are there other intrinsics we need to add here?
+    setLibcallName(RTLIB::MEMCPY, "#memcpy");
+    setLibcallName(RTLIB::MEMSET, "#memset");
+    setLibcallName(RTLIB::MEMMOVE, "#memmove");
+    setLibcallName(RTLIB::REM_F32, "#fmodf");
+    setLibcallName(RTLIB::REM_F64, "#fmod");
+    setLibcallName(RTLIB::FMA_F32, "#fmaf");
+    setLibcallName(RTLIB::FMA_F64, "#fma");
+    setLibcallName(RTLIB::SQRT_F32, "#sqrtf");
+    setLibcallName(RTLIB::SQRT_F64, "#sqrt");
+    setLibcallName(RTLIB::CBRT_F32, "#cbrtf");
+    setLibcallName(RTLIB::CBRT_F64, "#cbrt");
+    setLibcallName(RTLIB::LOG_F32, "#logf");
+    setLibcallName(RTLIB::LOG_F64, "#log");
+    setLibcallName(RTLIB::LOG2_F32, "#log2f");
+    setLibcallName(RTLIB::LOG2_F64, "#log2");
+    setLibcallName(RTLIB::LOG10_F32, "#log10f");
+    setLibcallName(RTLIB::LOG10_F64, "#log10");
+    setLibcallName(RTLIB::EXP_F32, "#expf");
+    setLibcallName(RTLIB::EXP_F64, "#exp");
+    setLibcallName(RTLIB::EXP2_F32, "#exp2f");
+    setLibcallName(RTLIB::EXP2_F64, "#exp2");
+    setLibcallName(RTLIB::EXP10_F32, "#exp10f");
+    setLibcallName(RTLIB::EXP10_F64, "#exp10");
+    setLibcallName(RTLIB::SIN_F32, "#sinf");
+    setLibcallName(RTLIB::SIN_F64, "#sin");
+    setLibcallName(RTLIB::COS_F32, "#cosf");
+    setLibcallName(RTLIB::COS_F64, "#cos");
+    setLibcallName(RTLIB::POW_F32, "#powf");
+    setLibcallName(RTLIB::POW_F64, "#pow");
+    setLibcallName(RTLIB::LDEXP_F32, "#ldexpf");
+    setLibcallName(RTLIB::LDEXP_F64, "#ldexp");
+    setLibcallName(RTLIB::FREXP_F32, "#frexpf");
+    setLibcallName(RTLIB::FREXP_F64, "#frexp");
+
+  }
 }
 
 void AArch64TargetLowering::addTypeForNEON(MVT VT) {
   assert(VT.isVector() && "VT should be a vector type");
 
   if (VT.isFloatingPoint()) {
     MVT PromoteTo = EVT(VT).changeVectorElementTypeToInteger().getSimpleVT();
     setOperationPromotedToType(ISD::LOAD, VT, PromoteTo);
case AArch64ISD::FIRST_NUMBER:
     MAKE_CASE(AArch64ISD::MOPS_MEMSET)
     MAKE_CASE(AArch64ISD::MOPS_MEMSET_TAGGING)
     MAKE_CASE(AArch64ISD::MOPS_MEMCOPY)
     MAKE_CASE(AArch64ISD::MOPS_MEMMOVE)
     MAKE_CASE(AArch64ISD::CALL_BTI)
     MAKE_CASE(AArch64ISD::MRRS)
     MAKE_CASE(AArch64ISD::MSRR)
     MAKE_CASE(AArch64ISD::RSHRNB_I)
+    MAKE_CASE(AArch64ISD::CALL_ARM64EC_TO_X64)
   }
 #undef MAKE_CASE
   return nullptr;
 }
 
 MachineBasicBlock *
 AArch64TargetLowering::EmitF128CSEL(MachineInstr &MI,
                                     MachineBasicBlock *MBB) const {
CCAssignFn *AArch64TargetLowering::CCAssignFnForCall(CallingConv::ID CC,
    case CallingConv::Win64:
      if (IsVarArg) {
        if (Subtarget->isWindowsArm64EC())
          return CC_AArch64_Arm64EC_VarArg;
        return CC_AArch64_Win64_VarArg;
      }
      return CC_AArch64_AAPCS;
    case CallingConv::CFGuard_Check:
+     if (Subtarget->isWindowsArm64EC())
+       return CC_AArch64_Arm64EC_CFGuard_Check;
      return CC_AArch64_Win64_CFGuard_Check;
    case CallingConv::AArch64_VectorCall:
    case CallingConv::AArch64_SVE_VectorCall:
    case CallingConv::AArch64_SME_ABI_Support_Routines_PreserveMost_From_X0:
    case CallingConv::AArch64_SME_ABI_Support_Routines_PreserveMost_From_X2:
      return CC_AArch64_AAPCS;
+  case CallingConv::ARM64EC_Thunk_X64:
+    return CC_AArch64_Arm64EC_Thunk;
+  case CallingConv::ARM64EC_Thunk_Native:
+    return CC_AArch64_Arm64EC_Thunk_Native;
   }
 }
 
 CCAssignFn *
 AArch64TargetLowering::CCAssignFnForReturn(CallingConv::ID CC) const {
-  return CC == CallingConv::WebKit_JS ? RetCC_AArch64_WebKit_JS
-                                      : RetCC_AArch64_AAPCS;
+  switch (CC) {
+  default:
+    return RetCC_AArch64_AAPCS;
+  case CallingConv::WebKit_JS:
+    return RetCC_AArch64_WebKit_JS;
+  case CallingConv::ARM64EC_Thunk_X64:
+    return RetCC_AArch64_Arm64EC_Thunk;
+  case CallingConv::CFGuard_Check:
+    if (Subtarget->isWindowsArm64EC())
+      return RetCC_AArch64_Arm64EC_CFGuard_Check;
+    return RetCC_AArch64_AAPCS;
+  }
 }
 
 
 unsigned
 AArch64TargetLowering::allocateLazySaveBuffer(SDValue &Chain, const SDLoc &DL,
                                               SelectionDAG &DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
   MachineFrameInfo &MFI = MF.getFrameInfo();

 SDValue AArch64TargetLowering::LowerFormalArguments(
     SDValue Chain, CallingConv::ID CallConv, bool isVarArg,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &DL,
     SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals) const {
   MachineFunction &MF = DAG.getMachineFunction();
   const Function &F = MF.getFunction();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   bool IsWin64 = Subtarget->isCallingConvWin64(F.getCallingConv());
+  bool StackViaX4 = CallConv == CallingConv::ARM64EC_Thunk_X64 ||
+                    (isVarArg && Subtarget->isWindowsArm64EC());
   AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
 
   SmallVector<ISD::OutputArg, 4> Outs;
   GetReturnInfo(CallConv, F.getReturnType(), F.getAttributes(), Outs,
                 DAG.getTargetLoweringInfo(), MF.getDataLayout());
   if (any_of(Outs, [](ISD::OutputArg &Out){ return Out.VT.isScalableVector(); }))
     FuncInfo->setIsSVECC(true);
 
if (VA.isRegLoc()) {
 
       uint32_t BEAlign = 0;
       if (!Subtarget->isLittleEndian() && ArgSize < 8 &&
           !Ins[i].Flags.isInConsecutiveRegs())
         BEAlign = 8 - ArgSize;
 
       SDValue FIN;
       MachinePointerInfo PtrInfo;
-      if (isVarArg && Subtarget->isWindowsArm64EC()) {
-        // In the ARM64EC varargs convention, fixed arguments on the stack are
-        // accessed relative to x4, not sp.
+      if (StackViaX4) {
+        // In both the ARM64EC varargs convention and the thunk convention,
+        // arguments on the stack are accessed relative to x4, not sp. In
+        // the thunk convention, there's an additional offset of 32 bytes
+        // to account for the shadow store.
         unsigned ObjOffset = ArgOffset + BEAlign;
+        if (CallConv == CallingConv::ARM64EC_Thunk_X64)
+          ObjOffset += 32;
         Register VReg = MF.addLiveIn(AArch64::X4, &AArch64::GPR64RegClass);
         SDValue Val = DAG.getCopyFromReg(Chain, DL, VReg, MVT::i64);
         FIN = DAG.getNode(ISD::ADD, DL, MVT::i64, Val,
                           DAG.getConstant(ObjOffset, DL, MVT::i64));
         PtrInfo = MachinePointerInfo::getUnknownStack(MF);
       } else {
         int FI = MFI.CreateFixedObject(ArgSize, ArgOffset + BEAlign, true);
 
if (MFI.hasMustTailInVarArgFunc()) {
         Forwards.push_back(ForwardedRegister(X8VReg, AArch64::X8, MVT::i64));
       }
     }
   }
 
   // On Windows, InReg pointers must be returned, so record the pointer in a
   // virtual register at the start of the function so it can be returned in the
   // epilogue.
-  if (IsWin64) {
+  if (IsWin64 || F.getCallingConv() == CallingConv::ARM64EC_Thunk_X64) {
     for (unsigned I = 0, E = Ins.size(); I != E; ++I) {
-      if (Ins[I].Flags.isInReg() && Ins[I].Flags.isSRet()) {
+      if ((F.getCallingConv() == CallingConv::ARM64EC_Thunk_X64 ||
+           Ins[I].Flags.isInReg()) &&
+          Ins[I].Flags.isSRet()) {
         assert(!FuncInfo->getSRetReturnReg());
 
         MVT PtrTy = getPointerTy(DAG.getDataLayout());
         Register Reg =
             MF.getRegInfo().createVirtualRegister(getRegClassFor(PtrTy));
         FuncInfo->setSRetReturnReg(Reg);
 
         SDValue Copy = DAG.getCopyToReg(DAG.getEntryNode(), DL, Reg, InVals[I]);
static void analyzeCallOperands(const AArch64TargetLowering &TLI,
                                 const TargetLowering::CallLoweringInfo &CLI,
                                 CCState &CCInfo) {
   const SelectionDAG &DAG = CLI.DAG;
   CallingConv::ID CalleeCC = CLI.CallConv;
   bool IsVarArg = CLI.IsVarArg;
   const SmallVector<ISD::OutputArg, 32> &Outs = CLI.Outs;
   bool IsCalleeWin64 = Subtarget->isCallingConvWin64(CalleeCC);
 
+  // For Arm64EC thunks, allocate 32 extra bytes at the bottom of the stack
+  // for the shadow store.
+  if (CalleeCC == CallingConv::ARM64EC_Thunk_X64)
+    CCInfo.AllocateStack(32, Align(16));
+
   unsigned NumArgs = Outs.size();
   for (unsigned i = 0; i != NumArgs; ++i) {
     MVT ArgVT = Outs[i].VT;
     ISD::ArgFlagsTy ArgFlags = Outs[i].Flags;
 
     bool UseVarArgCC = false;
     if (IsVarArg) {
       // On Windows, the fixed arguments in a vararg call are passed in GPRs
if (auto *G = dyn_cast<GlobalAddressSDNode>(Callee)) {
     auto GV = G->getGlobal();
     unsigned OpFlags =
         Subtarget->classifyGlobalFunctionReference(GV, getTargetMachine());
     if (OpFlags & AArch64II::MO_GOT) {
       Callee = DAG.getTargetGlobalAddress(GV, DL, PtrVT, 0, OpFlags);
       Callee = DAG.getNode(AArch64ISD::LOADgot, DL, PtrVT, Callee);
     } else {
       const GlobalValue *GV = G->getGlobal();
-      Callee = DAG.getTargetGlobalAddress(GV, DL, PtrVT, 0, 0);
+      Callee = DAG.getTargetGlobalAddress(GV, DL, PtrVT, 0, OpFlags);
     }
   } else if (auto *S = dyn_cast<ExternalSymbolSDNode>(Callee)) {
     if (getTargetMachine().getCodeModel() == CodeModel::Large &&
         Subtarget->isTargetMachO()) {
       const char *Sym = S->getSymbol();
       Callee = DAG.getTargetExternalSymbol(Sym, PtrVT, AArch64II::MO_GOT);
       Callee = DAG.getNode(AArch64ISD::LOADgot, DL, PtrVT, Callee);
     } else {
assert(!IsTailCall &&
            "tail calls cannot be marked with clang.arc.attachedcall");
     CallOpc = AArch64ISD::CALL_RVMARKER;
 
     // Add a target global address for the retainRV/claimRV runtime function
     // just before the call target.
     Function *ARCFn = *objcarc::getAttachedARCFunction(CLI.CB);
     auto GA = DAG.getTargetGlobalAddress(ARCFn, DL, PtrVT);
     Ops.insert(Ops.begin() + 1, GA);
-  } else if (GuardWithBTI)
+  } else if (CallConv == CallingConv::ARM64EC_Thunk_X64) {
+    CallOpc = AArch64ISD::CALL_ARM64EC_TO_X64;
+  } else if (GuardWithBTI) {
     CallOpc = AArch64ISD::CALL_BTI;
+  }
 
   // Returns a chain and a flag for retval copy to use.
   Chain = DAG.getNode(CallOpc, DL, NodeTys, Ops);
 
   if (IsCFICall)
     Chain.getNode()->setCFIType(CLI.CFIType->getZExtValue());
 
   DAG.addNoMergeSiteInfo(Chain.getNode(), CLI.NoMerge);
AArch64TargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
   // the sret argument into X0 for the return.
   // We saved the argument into a virtual register in the entry block,
   // so now we copy the value out and into X0.
   if (unsigned SRetReg = FuncInfo->getSRetReturnReg()) {
     SDValue Val = DAG.getCopyFromReg(RetOps[0], DL, SRetReg,
                                      getPointerTy(MF.getDataLayout()));
 
     unsigned RetValReg = AArch64::X0;
+    if (CallConv == CallingConv::ARM64EC_Thunk_X64)
+      RetValReg = AArch64::X8;
     Chain = DAG.getCopyToReg(Chain, DL, RetValReg, Val, Glue);
     Glue = Chain.getValue(1);
 
     RetOps.push_back(
       DAG.getRegister(RetValReg, getPointerTy(DAG.getDataLayout())));
   }
 
   const MCPhysReg *I = TRI->getCalleeSavedRegsViaCopy(&MF);
AArch64TargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
   }
 
   RetOps[0] = Chain; // Update chain.
 
   // Add the glue if we have it.
   if (Glue.getNode())
     RetOps.push_back(Glue);
 
+  if (CallConv == CallingConv::ARM64EC_Thunk_X64) {
+    // ARM64EC entry thunks use a special return sequence: instead of a regular
+    // "ret" instruction, they need to explicitly call the emulator.
+    EVT PtrVT = getPointerTy(DAG.getDataLayout());
+    SDValue Arm64ECRetDest =
+        DAG.getExternalSymbol("__os_arm64x_dispatch_ret", PtrVT);
+    Arm64ECRetDest =
+        getAddr(cast<ExternalSymbolSDNode>(Arm64ECRetDest), DAG, 0);
+    Arm64ECRetDest = DAG.getLoad(PtrVT, DL, DAG.getEntryNode(), Arm64ECRetDest,
+                                 MachinePointerInfo());
+    RetOps.insert(RetOps.begin() + 1, Arm64ECRetDest);
+    RetOps.insert(RetOps.begin() + 2, DAG.getTargetConstant(0, DL, MVT::i32));
+    return DAG.getNode(AArch64ISD::TC_RETURN, DL, MVT::Other, RetOps);
+  }
+
   return DAG.getNode(AArch64ISD::RET_GLUE, DL, MVT::Other, RetOps);
 }
 
 //===----------------------------------------------------------------------===//
 //  Other Lowering Code
 //===----------------------------------------------------------------------===//
 
 SDValue AArch64TargetLowering::getTargetNode(GlobalAddressSDNode *N, EVT Ty,

 }
 
 SDValue AArch64TargetLowering::getTargetNode(BlockAddressSDNode* N, EVT Ty,
                                              SelectionDAG &DAG,
                                              unsigned Flag) const {
   return DAG.getTargetBlockAddress(N->getBlockAddress(), Ty, 0, Flag);
 }
 
+SDValue AArch64TargetLowering::getTargetNode(ExternalSymbolSDNode *N, EVT Ty,
+                                             SelectionDAG &DAG,
+                                             unsigned Flag) const {
+  return DAG.getTargetExternalSymbol(N->getSymbol(), Ty, Flag);
+}
+
 // (loadGOT sym)
 template <class NodeTy>
 SDValue AArch64TargetLowering::getGOT(NodeTy *N, SelectionDAG &DAG,
                                       unsigned Flags) const {
   LLVM_DEBUG(dbgs() << "AArch64TargetLowering::getGOT\n");
   SDLoc DL(N);
   EVT Ty = getPointerTy(DAG.getDataLayout());
   SDValue GotAddr = getTargetNode(N, Ty, DAG, AArch64II::MO_GOT | Flags);
if (getTargetMachine().getCodeModel() == CodeModel::Large) {
     Result = getAddrLarge(GN, DAG, OpFlags);
   } else if (getTargetMachine().getCodeModel() == CodeModel::Tiny) {
     Result = getAddrTiny(GN, DAG, OpFlags);
   } else {
     Result = getAddr(GN, DAG, OpFlags);
   }
   EVT PtrVT = getPointerTy(DAG.getDataLayout());
   SDLoc DL(GN);
-  if (OpFlags & (AArch64II::MO_DLLIMPORT | AArch64II::MO_DLLIMPORTAUX |
-                 AArch64II::MO_COFFSTUB))
+  if (OpFlags & (AArch64II::MO_DLLIMPORT | AArch64II::MO_COFFSTUB))
     Result = DAG.getLoad(PtrVT, DL, DAG.getEntryNode(), Result,
                          MachinePointerInfo::getGOT(DAG.getMachineFunction()));
   return Result;
 }
 
 /// Convert a TLS address reference into the correct sequence of loads
 /// and calls to compute the variable's address (for Darwin, currently) and
 /// return an SDValue containing the final node.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

switch (Opc) {
     case AArch64::SEH_SaveFRegP_X:
     case AArch64::SEH_SetFP:
     case AArch64::SEH_AddFP:
     case AArch64::SEH_Nop:
     case AArch64::SEH_PrologEnd:
     case AArch64::SEH_EpilogStart:
     case AArch64::SEH_EpilogEnd:
     case AArch64::SEH_PACSignLR:
+    case AArch64::SEH_SaveAnyRegQP:
+    case AArch64::SEH_SaveAnyRegQPX:
       return true;
   }
 }
 
 bool AArch64InstrInfo::isCoalescableExtInstr(const MachineInstr &MI,
                                              Register &SrcReg, Register &DstReg,
                                              unsigned &SubIdx) const {
   switch (MI.getOpcode()) {
AArch64InstrInfo::getSerializableBitmaskMachineOperandTargetFlags() const {
 
   static const std::pair<unsigned, const char *> TargetFlags[] = {
       {MO_COFFSTUB, "aarch64-coffstub"},
       {MO_GOT, "aarch64-got"},
       {MO_NC, "aarch64-nc"},
       {MO_S, "aarch64-s"},
       {MO_TLS, "aarch64-tls"},
       {MO_DLLIMPORT, "aarch64-dllimport"},
-      {MO_DLLIMPORTAUX, "aarch64-dllimportaux"},
       {MO_PREL, "aarch64-prel"},
-      {MO_TAGGED, "aarch64-tagged"}};
+      {MO_TAGGED, "aarch64-tagged"},
+      {MO_ARM64EC_CALLMANGLE, "aarch64-arm64ec-callmangle"},
+  };
   return ArrayRef(TargetFlags);
 }
 
 ArrayRef<std::pair<MachineMemOperand::Flags, const char *>>
 AArch64InstrInfo::getSerializableMachineMemOperandTargetFlags() const {
   static const std::pair<MachineMemOperand::Flags, const char *> TargetFlags[] =
       {{MOSuppressPair, "aarch64-suppress-pair"},
        {MOStridedAccess, "aarch64-strided-access"}};

llvm/lib/Target/AArch64/AArch64InstrInfo.td

def AArch64call_bti      : SDNode<"AArch64ISD::CALL_BTI",
                                 [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
                                  SDNPVariadic]>;
 
 def AArch64call_rvmarker: SDNode<"AArch64ISD::CALL_RVMARKER",
                              SDTypeProfile<0, -1, [SDTCisPtrTy<0>]>,
                              [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
                               SDNPVariadic]>;
 
+def AArch64call_arm64ec_to_x64 : SDNode<"AArch64ISD::CALL_ARM64EC_TO_X64",
+                                      SDTypeProfile<0, -1, [SDTCisPtrTy<0>]>,
+                                      [SDNPHasChain, SDNPOptInGlue, SDNPOutGlue,
+                                       SDNPVariadic]>;
+
 def AArch64brcond        : SDNode<"AArch64ISD::BRCOND", SDT_AArch64Brcond,
                                 [SDNPHasChain]>;
 def AArch64cbz           : SDNode<"AArch64ISD::CBZ", SDT_AArch64cbz,
                                 [SDNPHasChain]>;
 def AArch64cbnz           : SDNode<"AArch64ISD::CBNZ", SDT_AArch64cbz,
                                 [SDNPHasChain]>;
 def AArch64tbz           : SDNode<"AArch64ISD::TBZ", SDT_AArch64tbz,
                                 [SDNPHasChain]>;
let isCall = 1, Defs = [LR], Uses = [SP] in {
   def BLR : BranchReg<0b0001, "blr", []>;
   def BLRNoIP : Pseudo<(outs), (ins GPR64noip:$Rn), []>,
                 Sched<[WriteBrReg]>,
                 PseudoInstExpansion<(BLR GPR64:$Rn)>;
   def BLR_RVMARKER : Pseudo<(outs), (ins variable_ops), []>,
                      Sched<[WriteBrReg]>;
   def BLR_BTI : Pseudo<(outs), (ins variable_ops), []>,
                 Sched<[WriteBrReg]>;
+  let Uses = [X16, SP] in
+  def BLR_X16 : Pseudo<(outs), (ins), [(AArch64call_arm64ec_to_x64 X16)]>,
+                Sched<[WriteBrReg]>,
+                PseudoInstExpansion<(BLR X16)>;
 } // isCall
 
 def : Pat<(AArch64call GPR64:$Rn),
           (BLR GPR64:$Rn)>,
       Requires<[NoSLSBLRMitigation]>;
 def : Pat<(AArch64call GPR64noip:$Rn),
           (BLRNoIP GPR64noip:$Rn)>,
       Requires<[SLSBLRMitigation]>;
let isPseudo = 1 in {
   def SEH_SaveFRegP_X : Pseudo<(outs), (ins i32imm:$reg0, i32imm:$reg1, i32imm:$offs), []>, Sched<[]>;
   def SEH_SetFP : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_AddFP : Pseudo<(outs), (ins i32imm:$offs), []>, Sched<[]>;
   def SEH_Nop : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_PrologEnd : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_EpilogStart : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_EpilogEnd : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_PACSignLR : Pseudo<(outs), (ins), []>, Sched<[]>;
+  def SEH_SaveAnyRegQP : Pseudo<(outs), (ins i32imm:$reg0, i32imm:$reg1, i32imm:$offs), []>, Sched<[]>;
+  def SEH_SaveAnyRegQPX : Pseudo<(outs), (ins i32imm:$reg0, i32imm:$reg1, i32imm:$offs), []>, Sched<[]>;
 }
 
 // Pseudo instructions for Windows EH
 //===----------------------------------------------------------------------===//
 let isTerminator = 1, hasSideEffects = 1, isBarrier = 1, hasCtrlDep = 1,
     isCodeGenOnly = 1, isReturn = 1, isEHScopeReturn = 1, isPseudo = 1 in {
    def CLEANUPRET : Pseudo<(outs), (ins), [(cleanupret)]>, Sched<[]>;
    let usesCustomInserter = 1 in

llvm/lib/Target/AArch64/AArch64MCInstLower.h

 //===-- AArch64MCInstLower.h - Lower MachineInstr to MCInst ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_AARCH64_AARCH64MCINSTLOWER_H
 #define LLVM_LIB_TARGET_AARCH64_AARCH64MCINSTLOWER_H
 
+#include "llvm/IR/GlobalValue.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/TargetParser/Triple.h"
 
 namespace llvm {
 class AsmPrinter;
 class MCContext;
 class MCInst;
 class MCOperand;
public:
   MCOperand lowerSymbolOperandMachO(const MachineOperand &MO,
                                     MCSymbol *Sym) const;
   MCOperand lowerSymbolOperandELF(const MachineOperand &MO,
                                   MCSymbol *Sym) const;
   MCOperand lowerSymbolOperandCOFF(const MachineOperand &MO,
                                    MCSymbol *Sym) const;
   MCOperand LowerSymbolOperand(const MachineOperand &MO, MCSymbol *Sym) const;
 
+  MCSymbol *GetGlobalValueSymbol(const GlobalValue *GV,
+                                 unsigned TargetFlags) const;
   MCSymbol *GetGlobalAddressSymbol(const MachineOperand &MO) const;
   MCSymbol *GetExternalSymbolSymbol(const MachineOperand &MO) const;
 };
 }
 
 #endif

llvm/lib/Target/AArch64/AArch64MCInstLower.cpp

 #include "llvm/Target/TargetMachine.h"
 using namespace llvm;
 
 extern cl::opt<bool> EnableAArch64ELFLocalDynamicTLSGeneration;
 
 AArch64MCInstLower::AArch64MCInstLower(MCContext &ctx, AsmPrinter &printer)
     : Ctx(ctx), Printer(printer) {}
 
 MCSymbol *
 AArch64MCInstLower::GetGlobalAddressSymbol(const MachineOperand &MO) const {
-  const GlobalValue *GV = MO.getGlobal();
-  unsigned TargetFlags = MO.getTargetFlags();
+  return GetGlobalValueSymbol(MO.getGlobal(), MO.getTargetFlags());
+}
+
+MCSymbol *AArch64MCInstLower::GetGlobalValueSymbol(const GlobalValue *GV,
+                                                   unsigned TargetFlags) const {
   const Triple &TheTriple = Printer.TM.getTargetTriple();
   if (!TheTriple.isOSBinFormatCOFF())
     return Printer.getSymbolPreferLocal(*GV);
 
   assert(TheTriple.isOSWindows() &&
          "Windows is the only supported COFF target");
 
   bool IsIndirect =
-      (TargetFlags & (AArch64II::MO_DLLIMPORT | AArch64II::MO_DLLIMPORTAUX |
-                      AArch64II::MO_COFFSTUB));
-  if (!IsIndirect)
+      (TargetFlags & (AArch64II::MO_DLLIMPORT | AArch64II::MO_COFFSTUB));
+  if (!IsIndirect) {
+    // For ARM64EC, symbol lookup in the MSVC linker has limited awareness
+    // of ARM64EC mangling ("#"/"$$h"). So object files need to refer to both
+    // the mangled and unmangled names of ARM64EC symbols, even if they aren't
+    // actually used by any relocations. Emit the necessary references here.
+    if (!TheTriple.isWindowsArm64EC() || !isa<Function>(GV) ||
+        !GV->hasExternalLinkage())
+      return Printer.getSymbol(GV);
+
+    StringRef Name = Printer.getSymbol(GV)->getName();
+    // Don't mangle ARM64EC runtime functions.
+    static constexpr StringLiteral ExcludedFns[] = {
+        "__os_arm64x_check_icall_cfg", "__os_arm64x_dispatch_call_no_redirect",
+        "__os_arm64x_check_icall"};
+    if (is_contained(ExcludedFns, Name))
+      return Printer.getSymbol(GV);
+
+    if (std::optional<std::string> MangledName =
+            getArm64ECMangledFunctionName(Name.str())) {
+      MCSymbol *MangledSym = Ctx.getOrCreateSymbol(MangledName.value());
+      if (!cast<Function>(GV)->hasMetadata("arm64ec_hasguestexit")) {
+        Printer.OutStreamer->emitSymbolAttribute(Printer.getSymbol(GV),
+                                                 MCSA_WeakAntiDep);
+        Printer.OutStreamer->emitAssignment(
+            Printer.getSymbol(GV),
+            MCSymbolRefExpr::create(MangledSym, MCSymbolRefExpr::VK_WEAKREF,
+                                    Ctx));
+        Printer.OutStreamer->emitSymbolAttribute(MangledSym, MCSA_WeakAntiDep);
+        Printer.OutStreamer->emitAssignment(
+            MangledSym,
+            MCSymbolRefExpr::create(Printer.getSymbol(GV),
+                                    MCSymbolRefExpr::VK_WEAKREF, Ctx));
+      }
+
+      if (TargetFlags & AArch64II::MO_ARM64EC_CALLMANGLE)
+        return MangledSym;
+    }
+
     return Printer.getSymbol(GV);
+  }
 
   SmallString<128> Name;
 
-  if (TargetFlags & AArch64II::MO_DLLIMPORTAUX) {
+  if ((TargetFlags & AArch64II::MO_DLLIMPORT) &&
+      TheTriple.isWindowsArm64EC() &&
+      !(TargetFlags & AArch64II::MO_ARM64EC_CALLMANGLE) &&
+      isa<Function>(GV)) {
     // __imp_aux is specific to arm64EC; it represents the actual address of
     // an imported function without any thunks.
     //
     // If we see a reference to an "aux" symbol, also emit a reference to the
     // corresponding non-aux symbol.  Otherwise, the Microsoft linker behaves
     // strangely when linking against x64 import libararies.
     //
     // emitSymbolAttribute() doesn't have any real effect here; it just

llvm/lib/Target/AArch64/AArch64RegisterInfo.cpp

AArch64RegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
 
   if (MF->getFunction().getCallingConv() == CallingConv::GHC)
     // GHC set of callee saved regs is empty as all those regs are
     // used for passing STG regs around
     return CSR_AArch64_NoRegs_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::AnyReg)
     return CSR_AArch64_AllRegs_SaveList;
 
+  if (MF->getFunction().getCallingConv() == CallingConv::ARM64EC_Thunk_X64)
+    return CSR_Win_AArch64_Arm64EC_Thunk_SaveList;
+
   // Darwin has its own CSR_AArch64_AAPCS_SaveList, which means most CSR save
   // lists depending on that will need to have their Darwin variant as well.
   if (MF->getSubtarget<AArch64Subtarget>().isTargetDarwin())
     return getDarwinCalleeSavedRegs(MF);
 
   if (MF->getFunction().getCallingConv() == CallingConv::CFGuard_Check)
     return CSR_Win_AArch64_CFGuard_Check_SaveList;
   if (MF->getSubtarget<AArch64Subtarget>().isTargetWindows()) {

llvm/lib/Target/AArch64/AArch64Subtarget.h

#include "AArch64GenSubtargetInfo.inc"
   unsigned getVScaleForTuning() const { return VScaleForTuning; }
 
   TailFoldingOpts getSVETailFoldingDefaultOpts() const {
     return DefaultSVETFOpts;
   }
 
   const char* getChkStkName() const {
     if (isWindowsArm64EC())
-      return "__chkstk_arm64ec";
+      return "#__chkstk_arm64ec";
     return "__chkstk";
   }
 
   const char* getSecurityCheckCookieName() const {
     if (isWindowsArm64EC())
-      return "__security_check_cookie_arm64ec";
+      return "#__security_check_cookie_arm64ec";
     return "__security_check_cookie";
   }
 };
 } // End llvm namespace
 
 #endif

llvm/lib/Target/AArch64/AArch64Subtarget.cpp

AArch64Subtarget::ClassifyGlobalReference(const GlobalValue *GV,
   // synthesized. This is done by having the loader stash the tag in the GOT
   // entry. Force all tagged globals (even ones with internal linkage) through
   // the GOT.
   if (GV->isTagged())
     return AArch64II::MO_GOT;
 
   if (!TM.shouldAssumeDSOLocal(*GV->getParent(), GV)) {
     if (GV->hasDLLImportStorageClass()) {
-      if (isWindowsArm64EC() && GV->getValueType()->isFunctionTy())
-        return AArch64II::MO_GOT | AArch64II::MO_DLLIMPORTAUX;
       return AArch64II::MO_GOT | AArch64II::MO_DLLIMPORT;
     }
     if (getTargetTriple().isOSWindows())
       return AArch64II::MO_GOT | AArch64II::MO_COFFSTUB;
     return AArch64II::MO_GOT;
   }
 
   // The small code model's direct accesses use ADRP, which cannot
unsigned AArch64Subtarget::classifyGlobalFunctionReference(
 
   // NonLazyBind goes via GOT unless we know it's available locally.
   auto *F = dyn_cast<Function>(GV);
   if (UseNonLazyBind && F && F->hasFnAttribute(Attribute::NonLazyBind) &&
       !TM.shouldAssumeDSOLocal(*GV->getParent(), GV))
     return AArch64II::MO_GOT;
 
   if (getTargetTriple().isOSWindows()) {
-    if (isWindowsArm64EC() && GV->getValueType()->isFunctionTy() &&
-        GV->hasDLLImportStorageClass()) {
-      // On Arm64EC, if we're calling a function directly, use MO_DLLIMPORT,
-      // not MO_DLLIMPORTAUX.
-      return AArch64II::MO_GOT | AArch64II::MO_DLLIMPORT;
+    if (isWindowsArm64EC() && GV->getValueType()->isFunctionTy()) {
+      if (GV->hasDLLImportStorageClass()) {
+        // On Arm64EC, if we're calling a symbol from the import table
+        // directly, use MO_ARM64EC_CALLMANGLE.
+        return AArch64II::MO_GOT | AArch64II::MO_DLLIMPORT |
+               AArch64II::MO_ARM64EC_CALLMANGLE;
+      }
+      if (GV->hasExternalLinkage()) {
+        // If we're calling a symbol directly, use the mangled form in the
+        // call instruction.
+        return AArch64II::MO_ARM64EC_CALLMANGLE;
+      }
     }
 
     // Use ClassifyGlobalReference for setting MO_DLLIMPORT/MO_COFFSTUB.
     return ClassifyGlobalReference(GV, TM);
   }
 
   return AArch64II::MO_NO_FLAG;
 }

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

void AArch64PassConfig::addIRPasses() {
   }
 
   // Expand any functions marked with SME attributes which require special
   // changes for the calling convention or that require the lazy-saving
   // mechanism specified in the SME ABI.
   addPass(createSMEABIPass());
 
   // Add Control Flow Guard checks.
-  if (TM->getTargetTriple().isOSWindows())
+  if (TM->getTargetTriple().isOSWindows()) {
+    if (TM->getTargetTriple().isWindowsArm64EC())
+      addPass(createAArch64Arm64ECCallLoweringPass());
+    else
       addPass(createCFGuardCheckPass());
+  }
 
   if (TM->Options.JMCInstrument)
     addPass(createJMCInstrumenterPass());
 }
 
 // Pass Pipeline Configuration
 bool AArch64PassConfig::addPreISel() {
   // Run promote constant before global merge, so that the promoted constants

llvm/lib/Target/AArch64/CMakeLists.txt

add_llvm_target(AArch64CodeGen
   GISel/AArch64O0PreLegalizerCombiner.cpp
   GISel/AArch64PreLegalizerCombiner.cpp
   GISel/AArch64PostLegalizerCombiner.cpp
   GISel/AArch64PostLegalizerLowering.cpp
   GISel/AArch64PostSelectOptimize.cpp
   GISel/AArch64RegisterBankInfo.cpp
   AArch64A57FPLoadBalancing.cpp
   AArch64AdvSIMDScalarPass.cpp
+  AArch64Arm64ECCallLowering.cpp
   AArch64AsmPrinter.cpp
   AArch64BranchTargets.cpp
   AArch64CallingConvention.cpp
   AArch64CleanupLocalDynamicTLSPass.cpp
   AArch64CollectLOH.cpp
   AArch64CondBrTuning.cpp
   AArch64ConditionalCompares.cpp
   AArch64DeadRegisterDefinitionsPass.cpp

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

 bool AArch64CallLowering::lowerFormalArguments(
     MachineIRBuilder &MIRBuilder, const Function &F,
     ArrayRef<ArrayRef<Register>> VRegs, FunctionLoweringInfo &FLI) const {
   MachineFunction &MF = MIRBuilder.getMF();
   MachineBasicBlock &MBB = MIRBuilder.getMBB();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   auto &DL = F.getParent()->getDataLayout();
   auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
-  // TODO: Support Arm64EC
+
+  // Arm64EC has extra requirements for varargs calls which are only implemented
+  // in SelectionDAG; bail out for now.
+  if (F.isVarArg() && Subtarget.isWindowsArm64EC())
+    return false;
+
+  // Arm64EC thunks have a special calling convention which is only implemented
+  // in SelectionDAG; bail out for now.
+  if (F.getCallingConv() == CallingConv::ARM64EC_Thunk_Native ||
+      F.getCallingConv() == CallingConv::ARM64EC_Thunk_X64)
+    return false;
+
   bool IsWin64 = Subtarget.isCallingConvWin64(F.getCallingConv()) && !Subtarget.isWindowsArm64EC();
 
   SmallVector<ArgInfo, 8> SplitArgs;
   SmallVector<std::pair<Register, Register>> BoolArgs;
 
   // Insert the hidden sret parameter if the return value won't fit in the
   // return registers.
   if (!FLI.CanLowerReturn)
bool AArch64CallLowering::lowerCall(MachineIRBuilder &MIRBuilder,
   MachineFunction &MF = MIRBuilder.getMF();
   const Function &F = MF.getFunction();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   auto &DL = F.getParent()->getDataLayout();
   const AArch64TargetLowering &TLI = *getTLI<AArch64TargetLowering>();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
 
   // Arm64EC has extra requirements for varargs calls; bail out for now.
-  if (Info.IsVarArg && Subtarget.isWindowsArm64EC())
+  //
+  // Arm64EC has special mangling rules for calls; bail out on all calls for
+  // now.
+  if (Subtarget.isWindowsArm64EC())
+    return false;
+
+  // Arm64EC thunks have a special calling convention which is only implemented
+  // in SelectionDAG; bail out for now.
+  if (Info.CallConv == CallingConv::ARM64EC_Thunk_Native ||
+      Info.CallConv == CallingConv::ARM64EC_Thunk_X64)
     return false;
 
   SmallVector<ArgInfo, 8> OutArgs;
   for (auto &OrigArg : Info.OrigArgs) {
     splitToValueTypes(OrigArg, OutArgs, DL, Info.CallConv);
     // AAPCS requires that we zero-extend i1 to 8 bits by the caller.
     auto &Flags = OrigArg.Flags[0];
     if (OrigArg.Ty->isIntegerTy(1) && !Flags.isSExt() && !Flags.isZExt()) {

llvm/lib/Target/AArch64/Utils/AArch64BaseInfo.h

static inline bool atomicBarrierDroppedOnZero(unsigned Opcode) {
   case AArch64::SWPAW:     case AArch64::SWPAX:
   case AArch64::SWPALB:    case AArch64::SWPALH:
   case AArch64::SWPALW:    case AArch64::SWPALX:
     return true;
   }
   return false;
 }
 
+static inline std::optional<std::string>
+getArm64ECMangledFunctionName(std::string Name) {
+  bool IsCppFn = Name[0] == '?';
+  if (IsCppFn && Name.find("$$h") != std::string::npos)
+    return std::nullopt;
+  if (!IsCppFn && Name[0] == '#')
+    return std::nullopt;
+
+  StringRef Prefix = "$$h";
+  size_t InsertIdx = 0;
+  if (IsCppFn) {
+    InsertIdx = Name.find("@@");
+    size_t ThreeAtSignsIdx = Name.find("@@@");
+    if (InsertIdx != std::string::npos && InsertIdx != ThreeAtSignsIdx) {
+      InsertIdx += 2;
+    } else {
+      InsertIdx = Name.find("@");
+      if (InsertIdx != std::string::npos)
+        InsertIdx++;
+    }
+  } else {
+    Prefix = "#";
+  }
+
+  Name.insert(Name.begin() + InsertIdx, Prefix.begin(), Prefix.end());
+  return std::optional<std::string>(Name);
+}
+
 namespace AArch64CC {
 
 // The CondCodes constants map directly to the 4-bit encoding of the condition
 // field for predicated instructions.
 enum CondCode {  // Meaning (integer)          Meaning (floating-point)
   EQ = 0x0,      // Equal                      Equal
   NE = 0x1,      // Not equal                  Not equal, or unordered
   HS = 0x2,      // Unsigned higher or same    >, ==, or unordered
enum TOF {
     /// MO_TAGGED - With MO_PAGE, indicates that the page includes a memory tag
     /// in bits 56-63.
     /// On a FrameIndex operand, indicates that the underlying memory is tagged
     /// with an unknown tag value (MTE); this needs to be lowered either to an
     /// SP-relative load or store instruction (which do not check tags), or to
     /// an LDG instruction to obtain the tag value.
     MO_TAGGED = 0x400,
 
-    /// MO_DLLIMPORTAUX - Symbol refers to "auxilliary" import stub. On
-    /// Arm64EC, there are two kinds of import stubs used for DLL import of
-    /// functions: MO_DLLIMPORT refers to natively callable Arm64 code, and
-    /// MO_DLLIMPORTAUX refers to the original address which can be compared
-    /// for equality.
-    MO_DLLIMPORTAUX = 0x800,
+    /// MO_ARM64EC_CALLMANGLE - Operand refers to the Arm64EC-mangled version
+    /// of a symbol, not the original. For dllimport symbols, this means it
+    /// uses "__imp_aux".  For other symbols, this means it uses the mangled
+    /// ("#" prefix for C) name.
+    MO_ARM64EC_CALLMANGLE = 0x800,
   };
 } // end namespace AArch64II
 
 //===----------------------------------------------------------------------===//
 // v8.3a Pointer Authentication
 //
 
 namespace AArch64PACKey {

llvm/test/CodeGen/AArch64/arm64ec-dllimport.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=arm64ec-pc-windows-msvc < %s | FileCheck %s
+; RUN: llc -mtriple=arm64ec-pc-windows-msvc -arm64ec-generate-thunks=false < %s | FileCheck %s
 
 @a = external dllimport global i32
 declare dllimport void @b()
 
 define ptr @dllimport_var() nounwind {
 ; CHECK-LABEL: dllimport_var:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    adrp x0, __imp_a

llvm/test/CodeGen/AArch64/arm64ec-reservedregs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=arm64ec-pc-windows-msvc < %s | FileCheck %s
+; RUN: llc -mtriple=arm64ec-pc-windows-msvc -arm64ec-generate-thunks=false < %s | FileCheck %s
 
 ; Make sure we're reserving all the registers that are supposed to be
 ; reserved. Integer regs x13, x15, x23, x24, x28. Float regs v16-v31.
 ; We confirm this by ensuring that we spill if and only if none of the
 ; unreserved registers are available.
 
 define i32 @no_int_regs(i32 %x) nounwind {
 ; CHECK-LABEL: no_int_regs:

llvm/test/CodeGen/AArch64/arm64ec-varargs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=arm64ec-pc-windows-msvc < %s | FileCheck %s
-; RUN: llc -mtriple=arm64ec-pc-windows-msvc < %s -global-isel=1 -global-isel-abort=0 | FileCheck %s
+; RUN: llc -mtriple=arm64ec-pc-windows-msvc -arm64ec-generate-thunks=false < %s | FileCheck %s
+; RUN: llc -mtriple=arm64ec-pc-windows-msvc -arm64ec-generate-thunks=false < %s -global-isel=1 -global-isel-abort=0 | FileCheck %s
 
 define void @varargs_callee(double %x, ...) nounwind {
 ; CHECK-LABEL: varargs_callee:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #48
 ; CHECK-NEXT:    stp x1, x2, [x4, #-24]!
 ; CHECK-NEXT:    str x3, [x4, #16]
 ; CHECK-NEXT:    str x4, [sp, #8]

 ; CHECK-NEXT:    mov x0, #4607182418800017408 // =0x3ff0000000000000
 ; CHECK-NEXT:    mov w1, #2 // =0x2
 ; CHECK-NEXT:    mov x2, #4613937818241073152 // =0x4008000000000000
 ; CHECK-NEXT:    mov w3, #4 // =0x4
 ; CHECK-NEXT:    mov w5, #16 // =0x10
 ; CHECK-NEXT:    stp xzr, x30, [sp, #24] // 8-byte Folded Spill
 ; CHECK-NEXT:    stp x9, x8, [sp]
 ; CHECK-NEXT:    str xzr, [sp, #16]
-; CHECK-NEXT:    bl varargs_callee
+; CHECK-NEXT:    .weak_anti_dep varargs_callee
+; CHECK-NEXT:  .set varargs_callee, "#varargs_callee"@WEAKREF
+; CHECK-NEXT:    .weak_anti_dep "#varargs_callee"
+; CHECK-NEXT:  .set "#varargs_callee", varargs_callee@WEAKREF
+; CHECK-NEXT:    bl "#varargs_callee"
 ; CHECK-NEXT:    ldr x30, [sp, #32] // 8-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #48
 ; CHECK-NEXT:    ret
   call void (double, ...) @varargs_callee(double 1.0, i32 2, double 3.0, i32 4, double 5.0, <2 x double> <double 0.0, double 0.0>)
   ret void
 }
 
 define <2 x double> @varargs_many_argscallee(double %a, double %b, double %c,

 ; CHECK-NEXT:    mov x0, #4607182418800017408 // =0x3ff0000000000000
 ; CHECK-NEXT:    mov x1, #4611686018427387904 // =0x4000000000000000
 ; CHECK-NEXT:    mov x2, #4613937818241073152 // =0x4008000000000000
 ; CHECK-NEXT:    mov x4, sp
 ; CHECK-NEXT:    mov w5, #16 // =0x10
 ; CHECK-NEXT:    str x30, [sp, #48] // 8-byte Folded Spill
 ; CHECK-NEXT:    stp x9, x8, [sp]
 ; CHECK-NEXT:    stp q0, q0, [sp, #16]
-; CHECK-NEXT:    bl varargs_many_argscallee
+; CHECK-NEXT:    .weak_anti_dep varargs_many_argscallee
+; CHECK-NEXT:  .set varargs_many_argscallee, "#varargs_many_argscallee"@WEAKREF
+; CHECK-NEXT:    .weak_anti_dep "#varargs_many_argscallee"
+; CHECK-NEXT:  .set "#varargs_many_argscallee", varargs_many_argscallee@WEAKREF
+; CHECK-NEXT:    bl "#varargs_many_argscallee"
 ; CHECK-NEXT:    ldr x30, [sp, #48] // 8-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #64
 ; CHECK-NEXT:    ret
   call <2 x double> (double, double, double, <2 x double>, <2 x double>, ...)
       @varargs_many_argscallee(double 1., double 2., double 3.,
                                <2 x double> zeroinitializer,
                                <2 x double> zeroinitializer, double 6.)
   ret void
 }
 
 
 declare void @llvm.va_start(ptr)

llvm/test/CodeGen/AArch64/stack-protector-target.ll

 ; WINDOWS-AARCH64: str x8, [sp, #8]
 ; WINDOWS-AARCH64: bl  _Z7CapturePi
 ; WINDOWS-AARCH64: ldr x0, [sp, #8]
 ; WINDOWS-AARCH64: bl  __security_check_cookie
 
 ; WINDOWS-ARM64EC: adrp x8, __security_cookie
 ; WINDOWS-ARM64EC: ldr x8, [x8, :lo12:__security_cookie]
 ; WINDOWS-ARM64EC: str x8, [sp, #8]
-; WINDOWS-ARM64EC: bl  _Z7CapturePi
+; WINDOWS-ARM64EC: bl "#_Z7CapturePi"
 ; WINDOWS-ARM64EC: ldr x0, [sp, #8]
-; WINDOWS-ARM64EC: bl  __security_check_cookie_arm64ec
+; WINDOWS-ARM64EC: bl "#__security_check_cookie_arm64ec"

llvm/test/CodeGen/AArch64/win-alloca.ll

 ; call to __chkstk.
 
 ; CHECK: add [[REG1:x[0-9]+]], x0, #15
 ; CHECK-OPT: lsr x15, [[REG1]], #4
 ; CHECK: bl __chkstk
 ; CHECK-OPT: sub [[REG3:x[0-9]+]], sp, x15, lsl #4
 ; CHECK-OPT: mov sp, [[REG3]]
 ; CHECK: bl func2
-; CHECK-ARM64EC: bl __chkstk_arm64ec
+; CHECK-ARM64EC: bl "#__chkstk_arm64ec"