[cfi] Implement cfi-icall using inline assembly.

[eugenis]

Summary

The current implementation is emitting a global constant that happens
to evaluate to the same bytes + relocation as a jump instruction on
X86. This does not work for PIE executables and shared libraries
though, because we end up with a wrong relocation type. And it has no
chance of working on ARM/AArch64 which use different relocation types
for jump instructions (R_ARM_JUMP24) that is never generated for
data.

This change replaces the constant with module-level inline assembly
followed by a hidden declaration of the jump table. Works fine for
ARM/AArch64, but has some drawbacks.

• Extra symbols are added to the static symbol table, which inflate

the size of the unstripped binary a little. Stripped binaries are not
affected. This happens because jump table declarations must be
external (because their body is in the inline asm).

• Original functions that were anonymous are now named

<original name>.cfi, and it affects symbolization sometimes. This is
necessary because the only user of these functions is the (inline
asm) jump table, so they had to be added to @llvm.used, which does
not allow unnamed functions.

[pcc]

lib/Transforms/IPO/LowerTypeTests.cpp

669

Could be simpler to have a single AsmOS that everyone writes to, then a single appendModuleInlineAsm at the end?

687

Don't you want to copy the linkage from the original function here?

736

http://llvm-cs.pcc.me.uk/?q=setSection.*llvm.metadata

We have code like this all over the place. May I ask you to add a general utility function for this?

829

This could just be Int8Ty since the type isn't important any more.

[eugenis]

lib/Transforms/IPO/LowerTypeTests.cpp

736

D25965

[eugenis]

lib/Transforms/IPO/LowerTypeTests.cpp

687

done, I think

829

The type matters for the GEP at line 830

[pcc]

lib/Transforms/IPO/LowerTypeTests.cpp

684

I wonder whether we want to escape these names? This appears to be the escaping we'd need to use:
http://llvm-cs.pcc.me.uk/lib/MC/MCSymbol.cpp#53

Maybe it's not necessary though since we pretty much expect all names we see to satisfy MCAsmInfo::isValidUnquotedName.

687

Also weak?

829

Sorry, yes, thanks.

[pcc]

lib/Transforms/IPO/LowerTypeTests.cpp

218

You no longer need this variable.

[eugenis]

lib/Transforms/IPO/LowerTypeTests.cpp

684

Looks like we should escape the names.
It's easy to call a function "\n" with asm in the declaration, and that breaks the jumptable code.

[eugenis]

PTAL

lib/Transforms/IPO/LowerTypeTests.cpp

684

It appears that neither llvm-mc nor gnu as support quoted identifier properly.
Gnu as allows a quoted string in, for example, .globl directive. Llvm-mc does too, but it forgets to unescape it, so, say, "\n" turns into a 2-character string when saves as textual asm and read back.

Neither support quoted strings in expressions we need to declare aliases, like

"\n" = .jumptable + 4.

I think I'll just make it a fatal error to apply cfi-icall to any name that requires escaping.

[pcc]

lib/Transforms/IPO/LowerTypeTests.cpp

684

Makes sense, I suppose. Though I think you'll also need to apply IR mangling (Mangler::getNameWithPrefix).

736

Maybe isValidAsmUnquotedName or something?

738

This function should have a FIXME to replace the inline asm with some better representation.

test/Transforms/LowerTypeTests/function.ll

1–5

Maybe use check-prefixes here to reduce duplication below?

[eugenis]

test/Transforms/LowerTypeTests/function.ll

1–5

I'd rather not. It will make an unreadable mess of x86, arm and common lines

[pcc]

lib/Transforms/IPO/LowerTypeTests.cpp

671

Although I think it doesn't matter in this case, it would probably be better to use the overload that takes a GlobalValue.

[eugenis]

lib/Transforms/IPO/LowerTypeTests.cpp

671

That one is not static. It can assign temporary labels for unnamed globals, but to use that we would need to access the instance in TargetLoweringObjectFile, not create our own.

[echristo]

lib/Transforms/IPO/LowerTypeTests.cpp

671

This is why I was mentioning wanting TLOF for this in general. Effectively you can get two different sets of labels if you try the "calling Mangler myself" path.

I'd rather initialize/split TLOF early on the TM rather than have lots of different mechanisms to get symbol names.

[pcc]

lib/Transforms/IPO/LowerTypeTests.cpp

671

But you're renaming unnamed globals elsewhere? With that just a temporary instance should be enough.

[eugenis]

lib/Transforms/IPO/LowerTypeTests.cpp

671

Actually, the current code does not handle unnamed functions. I'll fix it.

Eric, could you elaborate on the "initialize/split TLOF early on the TM"?

[eugenis]

lib/Transforms/IPO/LowerTypeTests.cpp

671

Renaming unnamed globals now.
Switched to non-static Mangler methods - they seem to do something special on Windows; we probably want that.

[pcc]

LGTM if Eric is ok with this.

Eric: in the long term do we want the mangler exposed to Transforms? Ideally Transforms should just be dealing with GlobalValues, correct? If that's the case, it doesn't seem worth threading TLOF through to here.

[eugenis]

Added hidden visibility to CFI aliases to internal functions.
I'm not sure why aarch64 gold does not relax GOT relocations against local symbols into a simple PC-relative offset, but that's how it is.

[eugenis]

Eric, are you OK with this?

[eugenis]

Switched from .p2align to .balign and reduced arm/aarch64 jump table alignment to 4.

[echristo]

In D25927#581379, @pcc wrote:

LGTM if Eric is ok with this.

Eric: in the long term do we want the mangler exposed to Transforms? Ideally Transforms should just be dealing with GlobalValues, correct? If that's the case, it doesn't seem worth threading TLOF through to here.

Long term it's interesting - I think if you're creating named symbols in a Transform we're going to need to plug this through, and TLOF is a better option for doing that than everyone calling various (different) versions of the Mangler/TLOF/TM current API that I'm cleaning up.

-eric

[pcc]

In D25927#584092, @echristo wrote:

In D25927#581379, @pcc wrote:

LGTM if Eric is ok with this.

Eric: in the long term do we want the mangler exposed to Transforms? Ideally Transforms should just be dealing with GlobalValues, correct? If that's the case, it doesn't seem worth threading TLOF through to here.

Long term it's interesting - I think if you're creating named symbols in a Transform we're going to need to plug this through, and TLOF is a better option for doing that than everyone calling various (different) versions of the Mangler/TLOF/TM current API that I'm cleaning up.

My point is that eventually we will come to some consensus on an IR-level representation for jump tables, so we would not need to mangle symbols here.

[rjmccall]

lib/Transforms/IPO/LowerTypeTests.cpp

880–889

This variable is dead now.

[echristo]

Since my Mangler/TLOF work isn't finished yet, for now go ahead with this. A couple of requests: a) add a comment/assert that global symbols aren't allowed (we don't want to collide with others) and b) that we should be using the one off of TLOF if possible.

Thanks!

-eric

[eugenis]

added a FIXME and an assert for the mangler vs TLOF issue.

[eugenis]

r286611, thanks

lib/Transforms/IPO/LowerTypeTests.cpp

880–889

and it's gone

lib/Transforms/IPO/LowerTypeTests.cpp

 #include "llvm/IR/Constant.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalObject.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/Mangler.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Operator.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
+#include "llvm/Transforms/Utils/ModuleUtils.h"
 
 using namespace llvm;
 using namespace lowertypetests;
 
 #define DEBUG_TYPE "lowertypetests"
 
 STATISTIC(ByteArraySizeBits, "Byte array size in bits");
 STATISTIC(ByteArraySizeBytes, "Byte array size in bytes");
class LowerTypeTestsModule {
 
   bool LinkerSubsectionsViaSymbols;
   Triple::ArchType Arch;
   Triple::ObjectFormatType ObjectFormat;
 
   IntegerType *Int1Ty = Type::getInt1Ty(M.getContext());
   IntegerType *Int8Ty = Type::getInt8Ty(M.getContext());
   IntegerType *Int32Ty = Type::getInt32Ty(M.getContext());
   PointerType *Int32PtrTy = PointerType::getUnqual(Int32Ty);
   IntegerType *Int64Ty = Type::getInt64Ty(M.getContext());
   IntegerType *IntPtrTy = M.getDataLayout().getIntPtrType(M.getContext(), 0);
 
   // Indirect function call index assignment counter for WebAssembly
   uint64_t IndirectIndex = 1;
 
   // Mapping from type identifiers to the call sites that test them.
   DenseMap<Metadata *, std::vector<CallInst *>> TypeTestCallSites;
 
   std::vector<ByteArrayInfo> ByteArrayInfos;
 
+  Mangler Mang;
+
   BitSetInfo
   buildBitSet(Metadata *TypeId,
               const DenseMap<GlobalObject *, uint64_t> &GlobalLayout);
   ByteArrayInfo *createByteArray(BitSetInfo &BSI);
   void allocateByteArrays();
   Value *createBitSetTest(IRBuilder<> &B, BitSetInfo &BSI, ByteArrayInfo *&BAI,
                           Value *BitOffset);
   void
   lowerTypeTestCalls(ArrayRef<Metadata *> TypeIds, Constant *CombinedGlobalAddr,
                      const DenseMap<GlobalObject *, uint64_t> &GlobalLayout);
   Value *
   lowerBitSetCall(CallInst *CI, BitSetInfo &BSI, ByteArrayInfo *&BAI,
                   Constant *CombinedGlobal,
                   const DenseMap<GlobalObject *, uint64_t> &GlobalLayout);
   void buildBitSetsFromGlobalVariables(ArrayRef<Metadata *> TypeIds,
                                        ArrayRef<GlobalVariable *> Globals);
   unsigned getJumpTableEntrySize();
   Type *getJumpTableEntryType();
-  Constant *createJumpTableEntry(GlobalObject *Src, Function *Dest,
-                                 unsigned Distance);
+  void createJumpTableEntry(raw_ostream &OS, Function *Dest, unsigned Distance);
+  void createJumpTableAlias(raw_ostream &OS, Function *Dest,
+                            GlobalVariable *JumpTable, unsigned Distance);
   void verifyTypeMDNode(GlobalObject *GO, MDNode *Type);
   void buildBitSetsFromFunctions(ArrayRef<Metadata *> TypeIds,
                                  ArrayRef<Function *> Functions);
-  void buildBitSetsFromFunctionsX86(ArrayRef<Metadata *> TypeIds,
+  void buildBitSetsFromFunctionsNative(ArrayRef<Metadata *> TypeIds,
                                     ArrayRef<Function *> Functions);
   void buildBitSetsFromFunctionsWASM(ArrayRef<Metadata *> TypeIds,
                                      ArrayRef<Function *> Functions);
   void buildBitSetsFromDisjointSet(ArrayRef<Metadata *> TypeIds,
                                    ArrayRef<GlobalObject *> Globals);
 
 public:
   LowerTypeTestsModule(Module &M);
void LowerTypeTestsModule::verifyTypeMDNode(GlobalObject *GO, MDNode *Type) {
   if (!OffsetConstMD)
     report_fatal_error("Type offset must be a constant");
   auto OffsetInt = dyn_cast<ConstantInt>(OffsetConstMD->getValue());
   if (!OffsetInt)
     report_fatal_error("Type offset must be an integer constant");
 }
 
 static const unsigned kX86JumpTableEntrySize = 8;
+static const unsigned kARMJumpTableEntrySize = 4;
 
 unsigned LowerTypeTestsModule::getJumpTableEntrySize() {
+  switch (Arch) {
+    case Triple::x86:
+    case Triple::x86_64:
       return kX86JumpTableEntrySize;
+    case Triple::arm:
+    case Triple::aarch64:
+      return kARMJumpTableEntrySize;
+    default:
+      report_fatal_error("Unsupported architecture for jump tables");
+  }
+}
+
+static bool isValidAsmUnquotedName(StringRef Name) {
+  if (Name.empty())
+    return false;
+
+  for (char C : Name) {
+    if (!((C >= 'a' && C <= 'z') || (C >= 'A' && C <= 'Z') ||
+          (C >= '0' && C <= '9') || C == '_' || C == '$' || C == '.' ||
+          C == '@'))
+      return false;
+  }
+
+  return true;
 }
 
 // Create a constant representing a jump table entry for the target. This
 // consists of an instruction sequence containing a relative branch to Dest. The
 // constant will be laid out at address Src+(Len*Distance) where Len is the
 // target-specific jump table entry size.
-Constant *LowerTypeTestsModule::createJumpTableEntry(GlobalObject *Src,
-                                                     Function *Dest,
-                                                     unsigned Distance) {
-  const unsigned kJmpPCRel32Code = 0xe9;
-  const unsigned kInt3Code = 0xcc;
-
-  ConstantInt *Jmp = ConstantInt::get(Int8Ty, kJmpPCRel32Code);
+void LowerTypeTestsModule::createJumpTableEntry(raw_ostream &OS, Function *Dest,
+  // FIXME: replace IR Mangler with TargetLoweringObjectFile interface.
+  // A private instance of Mangler we use here can not deal with unnamed
+  // symbols, as it may create colliding labels. Thankfully(?), the use of
+  // inline asm requires us to give names to all affected functions anyway.
+  assert(Dest->hasName() && "jumptable targets can not be anonymous");
+  SmallString<16> Name;
+  Mang.getNameWithPrefix(Name, Dest, /* CannotUsePrivateLabel */ false);
+
+  if (!isValidAsmUnquotedName(Name)) {
+    // We are going to emit a function call as textual asm. Escaped strings
+    // in such expressions are not well supported.
+    report_fatal_error(
+        "CFI-ICall does not allow special characters in a function name.");
+  }
 
-  // Build a constant representing the displacement between the constant's
-  // address and Dest. This will resolve to a PC32 relocation referring to Dest.
-  Constant *DestInt = ConstantExpr::getPtrToInt(Dest, IntPtrTy);
-  Constant *SrcInt = ConstantExpr::getPtrToInt(Src, IntPtrTy);
-  Constant *Disp = ConstantExpr::getSub(DestInt, SrcInt);
-  ConstantInt *DispOffset =
-      ConstantInt::get(IntPtrTy, Distance * kX86JumpTableEntrySize + 5);
-  Constant *OffsetedDisp = ConstantExpr::getSub(Disp, DispOffset);
-  OffsetedDisp = ConstantExpr::getTruncOrBitCast(OffsetedDisp, Int32Ty);
+  if (Arch == Triple::x86 || Arch == Triple::x86_64) {
+    OS << "jmp " << Name << "@plt\n";
+    OS << "int3\nint3\nint3\n";
+  } else if (Arch == Triple::arm || Arch == Triple::aarch64) {
+    OS << "b " << Name << "\n";
+  } else {
+    report_fatal_error("Unsupported architecture for jump tables");
+  }
+}
 
-  ConstantInt *Int3 = ConstantInt::get(Int8Ty, kInt3Code);
+void LowerTypeTestsModule::createJumpTableAlias(raw_ostream &OS, Function *Dest,
+                                                GlobalVariable *JumpTable,
+                                                unsigned Distance) {
+  assert(Dest->hasName() && "jumptable targets can not be anonymous");
+  SmallString<16> Name;
+  Mang.getNameWithPrefix(Name, Dest, /* CannotUsePrivateLabel */ false);
+
+  if (!isValidAsmUnquotedName(Name)) {
+    // We are going to emit a function alias as textual asm. Escaped strings
+    // in such expressions are not well supported.
+    report_fatal_error(
+        "CFI-ICall does not allow special characters in a function name.");
+  }
 
-  Constant *Fields[] = {
-      Jmp, OffsetedDisp, Int3, Int3, Int3,
-  };
-  return ConstantStruct::getAnon(Fields, /*Packed=*/true);
+  if (Dest->isWeakForLinker())
+    OS << ".weak " << Name << "\n";
+  else if (!Dest->hasLocalLinkage())
+    OS << ".globl " << Name << "\n";
+  OS << ".type " << Name << ", function\n";
+  OS << Name << " = " << JumpTable->getName() << " + "
+     << (getJumpTableEntrySize() * Distance) << "\n";
+  OS << ".size " << Name << ", " << getJumpTableEntrySize() << "\n";
 }
 
 Type *LowerTypeTestsModule::getJumpTableEntryType() {
-  return StructType::get(M.getContext(),
-                         {Int8Ty, Int32Ty, Int8Ty, Int8Ty, Int8Ty},
-                         /*Packed=*/true);
+  return ArrayType::get(Int8Ty, getJumpTableEntrySize());
 }
 
 /// Given a disjoint set of type identifiers and functions, build the bit sets
 /// and lower the llvm.type.test calls, architecture dependently.
 void LowerTypeTestsModule::buildBitSetsFromFunctions(
     ArrayRef<Metadata *> TypeIds, ArrayRef<Function *> Functions) {
-  if (Arch == Triple::x86 || Arch == Triple::x86_64)
-    buildBitSetsFromFunctionsX86(TypeIds, Functions);
+  if (Arch == Triple::x86 || Arch == Triple::x86_64 || Arch == Triple::arm ||
+      Arch == Triple::aarch64)
+    buildBitSetsFromFunctionsNative(TypeIds, Functions);
   else if (Arch == Triple::wasm32 || Arch == Triple::wasm64)
     buildBitSetsFromFunctionsWASM(TypeIds, Functions);
   else
     report_fatal_error("Unsupported architecture for jump tables");
 }
 
 /// Given a disjoint set of type identifiers and functions, build a jump table
 /// for the functions, build the bit sets and lower the llvm.type.test calls.
-void LowerTypeTestsModule::buildBitSetsFromFunctionsX86(
+void LowerTypeTestsModule::buildBitSetsFromFunctionsNative(
     ArrayRef<Metadata *> TypeIds, ArrayRef<Function *> Functions) {
   // Unlike the global bitset builder, the function bitset builder cannot
   // re-arrange functions in a particular order and base its calculations on the
   // layout of the functions' entry points, as we have no idea how large a
   // particular function will end up being (the size could even depend on what
   // this pass does!) Instead, we build a jump table, which is a block of code
   // consisting of one branch instruction for each of the functions in the bit
   // set that branches to the target function, and redirect any taken function
void LowerTypeTestsModule::buildBitSetsFromFunctionsNative(
   // ret
   //
   // foo:
   // mov f, %eax
   // mov g, %edx
   // mov h, %ecx
   // ret
   //
-  // To create a jump table for these functions, we instruct the LLVM code
-  // generator to output a jump table in the .text section. This is done by
-  // representing the instructions in the jump table as an LLVM constant and
-  // placing them in a global variable in the .text section. The end result will
-  // (conceptually) look like this:
+  // We output the jump table as module-level inline asm string. The end result
+  // will (conceptually) look like this:
   //
-  // f:
-  // jmp .Ltmp0 ; 5 bytes
+  // f = .cfi.jumptable
+  // g = .cfi.jumptable + 4
+  // h = .cfi.jumptable + 8
+  // .cfi.jumptable:
+  // jmp f.cfi  ; 5 bytes
   // int3       ; 1 byte
   // int3       ; 1 byte
   // int3       ; 1 byte
-  //
-  // g:
-  // jmp .Ltmp1 ; 5 bytes
+  // jmp g.cfi  ; 5 bytes
   // int3       ; 1 byte
   // int3       ; 1 byte
   // int3       ; 1 byte
-  //
-  // h:
-  // jmp .Ltmp2 ; 5 bytes
+  // jmp h.cfi  ; 5 bytes
   // int3       ; 1 byte
   // int3       ; 1 byte
   // int3       ; 1 byte
   //
-  // .Ltmp0:
+  // f.cfi:
   // mov 0, %eax
   // ret
   //
-  // .Ltmp1:
+  // g.cfi:
   // mov 1, %eax
   // ret
   //
-  // .Ltmp2:
+  // h.cfi:
   // mov 2, %eax
   // ret
   //
   // foo:
   // mov f, %eax
   // mov g, %edx
   // mov h, %ecx
   // ret
   //
   // Because the addresses of f, g, h are evenly spaced at a power of 2, in the
   // normal case the check can be carried out using the same kind of simple
   // arithmetic that we normally use for globals.
 
+  // FIXME: find a better way to represent the jumptable in the IR.
+
   assert(!Functions.empty());
 
   // Build a simple layout based on the regular layout of jump tables.
   DenseMap<GlobalObject *, uint64_t> GlobalLayout;
   unsigned EntrySize = getJumpTableEntrySize();
   for (unsigned I = 0; I != Functions.size(); ++I)
     GlobalLayout[Functions[I]] = I * EntrySize;
 
   // Create a constant to hold the jump table.
   ArrayType *JumpTableType =
       ArrayType::get(getJumpTableEntryType(), Functions.size());
-  auto JumpTable = new GlobalVariable(M, JumpTableType,
-                                      /*isConstant=*/true,
-                                      GlobalValue::PrivateLinkage, nullptr);
-  JumpTable->setSection(ObjectFormat == Triple::MachO
-                            ? "__TEXT,__text,regular,pure_instructions"
-                            : ".text");
+  auto JumpTable =
+      new GlobalVariable(M, JumpTableType,
+                         /*isConstant=*/true, GlobalValue::ExternalLinkage,
+                         nullptr, ".cfi.jumptable");
+  JumpTable->setVisibility(GlobalValue::HiddenVisibility);
   lowerTypeTestCalls(TypeIds, JumpTable, GlobalLayout);
 
+  std::string AsmStr;
+  raw_string_ostream AsmOS(AsmStr);
+
   // Build aliases pointing to offsets into the jump table, and replace
   // references to the original functions with references to the aliases.
   for (unsigned I = 0; I != Functions.size(); ++I) {
+    // Need a name for the asm label. Normally, unnamed functions get temporary
+    // asm labels in TargetLoweringObjectFile but we don't have access to that
+    // here.
+    if (!Functions[I]->hasName())
+      Functions[I]->setName("unnamed");
+    if (LinkerSubsectionsViaSymbols || Functions[I]->isDeclarationForLinker()) {
       Constant *CombinedGlobalElemPtr = ConstantExpr::getBitCast(
           ConstantExpr::getGetElementPtr(
               JumpTableType, JumpTable,
               ArrayRef<Constant *>{ConstantInt::get(IntPtrTy, 0),
                                    ConstantInt::get(IntPtrTy, I)}),
           Functions[I]->getType());
-    if (LinkerSubsectionsViaSymbols || Functions[I]->isDeclarationForLinker()) {
       Functions[I]->replaceAllUsesWith(CombinedGlobalElemPtr);
+
+      if (Functions[I]->isWeakForLinker())
+        AsmOS << ".weak " << Functions[I]->getName() << "\n";
     } else {
       assert(Functions[I]->getType()->getAddressSpace() == 0);
-      GlobalAlias *GAlias = GlobalAlias::create(Functions[I]->getValueType(), 0,
-                                                Functions[I]->getLinkage(), "",
-                                                CombinedGlobalElemPtr, &M);
-      GAlias->setVisibility(Functions[I]->getVisibility());
-      GAlias->takeName(Functions[I]);
-      Functions[I]->replaceAllUsesWith(GAlias);
+
+      createJumpTableAlias(AsmOS, Functions[I], JumpTable, I);
+
+      Function *DeclAlias =
+          Function::Create(cast<FunctionType>(Functions[I]->getValueType()),
+                           GlobalValue::ExternalLinkage, "", &M);
+      // Since the alias (DeclAlias) is actually a declaration, it can not have
+      // internal linkage. Compensate for that by giving it hidden visibility.
+      // With this we end up with a GOT relocation against a local symbol.
+      DeclAlias->setVisibility(Functions[I]->hasLocalLinkage()
+                                   ? GlobalValue::HiddenVisibility
+                                   : Functions[I]->getVisibility());
+      DeclAlias->takeName(Functions[I]);
+      // Unnamed functions can not be added to llvm.used.
+      Functions[I]->setName(DeclAlias->getName() + ".cfi");
+      Functions[I]->replaceAllUsesWith(DeclAlias);
     }
     if (!Functions[I]->isDeclarationForLinker())
-      Functions[I]->setLinkage(GlobalValue::PrivateLinkage);
+      Functions[I]->setLinkage(GlobalValue::InternalLinkage);
   }
 
-  // Build and set the jump table's initializer.
-  std::vector<Constant *> JumpTableEntries;
+  // Try to emit the jump table at the end of the text segment.
+  // Jump table must come after __cfi_check in the cross-dso mode.
+  // FIXME: this magic section name seems to do the trick.
+  AsmOS << ".section " << (ObjectFormat == Triple::MachO
+                               ? "__TEXT,__text,regular,pure_instructions"
+                               : ".text.cfi, \"ax\", @progbits")
+        << "\n";
+  // Align the whole table by entry size.
+  AsmOS << ".balign " << EntrySize << "\n";
+  AsmOS << JumpTable->getName() << ":\n";
   for (unsigned I = 0; I != Functions.size(); ++I)
-    JumpTableEntries.push_back(
-        createJumpTableEntry(JumpTable, Functions[I], I));
-  JumpTable->setInitializer(
-      ConstantArray::get(JumpTableType, JumpTableEntries));
+    createJumpTableEntry(AsmOS, Functions[I], I);
+
+  M.appendModuleInlineAsm(AsmOS.str());
+
+  SmallVector<GlobalValue *, 16> Used;
+  Used.reserve(Functions.size());
+  for (auto *F : Functions)
+    Used.push_back(F);
+  appendToUsed(M, Used);
 }
 
 /// Assign a dummy layout using an incrementing counter, tag each function
 /// with its index represented as metadata, and lower each type test to an
 /// integer range comparison. During generation of the indirect function call
 /// table in the backend, it will assign the given indexes.
 /// Note: Dynamic linking is not supported, as the WebAssembly ABI has not yet
 /// been finalized.

test/Transforms/LowerTypeTests/function-disjoint.ll

 ; RUN: opt -S -lowertypetests -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck --check-prefix=X64 %s
 ; RUN: opt -S -lowertypetests -mtriple=wasm32-unknown-unknown < %s | FileCheck --check-prefix=WASM32 %s
 
 ; Tests that we correctly handle bitsets with disjoint call target sets.
 
 target datalayout = "e-p:64:64"
 
-; X64: @[[JT0:.*]] = private constant [1 x <{ i8, i32, i8, i8, i8 }>] [<{ i8, i32, i8, i8, i8 }> <{ i8 -23, i32 trunc (i64 sub (i64 sub (i64 ptrtoint (void ()* @[[FNAME:.*]] to i64), i64 ptrtoint ([1 x <{ i8, i32, i8, i8, i8 }>]* @[[JT0]] to i64)), i64 5) to i32), i8 -52, i8 -52, i8 -52 }>], section ".text"
-; X64: @[[JT1:.*]] = private constant [1 x <{ i8, i32, i8, i8, i8 }>] [<{ i8, i32, i8, i8, i8 }> <{ i8 -23, i32 trunc (i64 sub (i64 sub (i64 ptrtoint (void ()* @[[GNAME:.*]] to i64), i64 ptrtoint ([1 x <{ i8, i32, i8, i8, i8 }>]* @[[JT1]] to i64)), i64 5) to i32), i8 -52, i8 -52, i8 -52 }>], section ".text"
+; X64:      module asm "f = .cfi.jumptable + 0"
+
+; X64:      module asm ".cfi.jumptable:"
+; X64-NEXT: module asm "jmp f.cfi@plt"
+; X64-NEXT: module asm "int3"
+; X64-NEXT: module asm "int3"
+; X64-NEXT: module asm "int3"
+
+; X64:      module asm "g = .cfi.jumptable.1 + 0"
+
+; X64:      module asm ".cfi.jumptable.1:"
+; X64-NEXT: module asm "jmp g.cfi@plt"
+; X64-NEXT: module asm "int3"
+; X64-NEXT: module asm "int3"
+; X64-NEXT: module asm "int3"
+
+
+; X64: @.cfi.jumptable = external hidden constant [1 x [8 x i8]]
+; X64: @.cfi.jumptable.1 = external hidden constant [1 x [8 x i8]]
+
 ; WASM32: private constant [0 x i8] zeroinitializer
 @0 = private unnamed_addr constant [2 x void ()*] [void ()* @f, void ()* @g], align 16
 
-; X64: @f = alias void (), bitcast ([1 x <{ i8, i32, i8, i8, i8 }>]* @[[JT0]] to void ()*)
-; X64: @g = alias void (), bitcast ([1 x <{ i8, i32, i8, i8, i8 }>]* @[[JT1]] to void ()*)
-
-; X64: define private void @[[FNAME]]()
+; X64: define internal void @f.cfi()
 ; WASM32: define void @f() !type !{{[0-9]+}} !wasm.index ![[I0:[0-9]+]]
 define void @f() !type !0 {
   ret void
 }
 
-; X64: define private void @[[GNAME]]()
+; X64: define internal void @g.cfi()
 ; WASM32: define void @g() !type !{{[0-9]+}} !wasm.index ![[I1:[0-9]+]]
 define void @g() !type !1 {
   ret void
 }
 
 !0 = !{i32 0, !"typeid1"}
 !1 = !{i32 0, !"typeid2"}
 
 declare i1 @llvm.type.test(i8* %ptr, metadata %bitset) nounwind readnone
 
 define i1 @foo(i8* %p) {
-  ; X64: icmp eq i64 {{.*}}, ptrtoint ([1 x <{ i8, i32, i8, i8, i8 }>]* @[[JT0]] to i64)
+  ; X64: icmp eq i64 {{.*}}, ptrtoint ([1 x [8 x i8]]* @.cfi.jumptable to i64)
   ; WASM32: icmp eq i64 {{.*}}, 1
   %x = call i1 @llvm.type.test(i8* %p, metadata !"typeid1")
-  ; X64: icmp eq i64 {{.*}}, ptrtoint ([1 x <{ i8, i32, i8, i8, i8 }>]* @[[JT1]] to i64)
+  ; X64: icmp eq i64 {{.*}}, ptrtoint ([1 x [8 x i8]]* @.cfi.jumptable.1 to i64)
   ; WASM32: icmp eq i64 {{.*}}, 2
   %y = call i1 @llvm.type.test(i8* %p, metadata !"typeid2")
   %z = add i1 %x, %y
   ret i1 %z
 }
 
+; X64: declare void @f()
+; X64: declare void @g()
+
 ; WASM32: ![[I0]] = !{i64 1}
-; WASM32: ![[I1]] = !{i64 2}
+; WASM32: ![[I1]] = !{i64 2}
 No newline at end of file

test/Transforms/LowerTypeTests/function-ext.ll

 ; RUN: opt -S -lowertypetests -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck --check-prefix=X64 %s
 ; RUN: opt -S -lowertypetests -mtriple=wasm32-unknown-unknown < %s | FileCheck --check-prefix=WASM32 %s
 
 ; Tests that we correctly handle external references, including the case where
 ; all functions in a bitset are external references.
 
-; X64: @[[JT:.*]] = private constant [1 x <{ i8, i32, i8, i8, i8 }>] [<{ i8, i32, i8, i8, i8 }> <{ i8 -23, i32 trunc (i64 sub (i64 sub (i64 ptrtoint (void ()* @foo to i64), i64 ptrtoint ([1 x <{ i8, i32, i8, i8, i8 }>]* @[[JT]] to i64)), i64 5) to i32), i8 -52, i8 -52, i8 -52 }>], section ".text"
+; X64:      module asm ".cfi.jumptable:"
+; X64-NEXT: module asm "jmp foo@plt"
+; X64-NOT:  module asm "jmp {{.*}}@plt"
+
+; X64: @.cfi.jumptable = external hidden constant [1 x [8 x i8]]
 ; WASM32: private constant [0 x i8] zeroinitializer
 
 ; WASM32: declare !type !{{[0-9]+}} void @foo()
 declare !type !0 void @foo()
 
 define i1 @bar(i8* %ptr) {
-  ; X64: icmp eq i64 {{.*}}, ptrtoint ([1 x <{ i8, i32, i8, i8, i8 }>]* @[[JT]] to i64)
+  ; X64: icmp eq i64 {{.*}}, ptrtoint ([1 x [8 x i8]]* @.cfi.jumptable to i64)
   ; WASM32: sub i64 {{.*}}, 0
   ; WASM32: icmp ult i64 {{.*}}, 1
   %p = call i1 @llvm.type.test(i8* %ptr, metadata !"void")
   ret i1 %p
 }
 
 declare i1 @llvm.type.test(i8* %ptr, metadata %bitset) nounwind readnone
 
 !0 = !{i64 0, !"void"}
 ; WASM-NOT: !{i64 0}
 ; WASM-NOT: !{i64 1}

test/Transforms/LowerTypeTests/function.ll

-; RUN: opt -S -lowertypetests -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck --check-prefix=X64 %s
+; RUN: opt -S -lowertypetests -mtriple=i686-unknown-linux-gnu < %s | FileCheck --check-prefix=X86 %s
+; RUN: opt -S -lowertypetests -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck --check-prefix=X86 %s
+; RUN: opt -S -lowertypetests -mtriple=arm-unknown-linux-gnu < %s | FileCheck --check-prefix=ARM %s
+; RUN: opt -S -lowertypetests -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck --check-prefix=ARM %s
 ; RUN: opt -S -lowertypetests -mtriple=wasm32-unknown-unknown < %s | FileCheck --check-prefix=WASM32 %s
 
 ; Tests that we correctly handle bitsets containing 2 or more functions.
 
 target datalayout = "e-p:64:64"
 
-; X64: @[[JT:.*]] = private constant [2 x <{ i8, i32, i8, i8, i8 }>] [<{ i8, i32, i8, i8, i8 }> <{ i8 -23, i32 trunc (i64 sub (i64 sub (i64 ptrtoint (void ()* @[[FNAME:.*]] to i64), i64 ptrtoint ([2 x <{ i8, i32, i8, i8, i8 }>]* @[[JT]] to i64)), i64 5) to i32), i8 -52, i8 -52, i8 -52 }>, <{ i8, i32, i8, i8, i8 }> <{ i8 -23, i32 trunc (i64 sub (i64 sub (i64 ptrtoint (void ()* @[[GNAME:.*]] to i64), i64 ptrtoint ([2 x <{ i8, i32, i8, i8, i8 }>]* @[[JT]] to i64)), i64 13) to i32), i8 -52, i8 -52, i8 -52 }>], section ".text"
+; X86:      module asm ".globl f"
+; X86-NEXT: module asm ".type f, function"
+; X86-NEXT: module asm "f = .cfi.jumptable + 0"
+; X86-NEXT: module asm ".size f, 8"
+; X86-NEXT: module asm ".type g, function"
+; X86-NEXT: module asm "g = .cfi.jumptable + 8"
+; X86-NEXT: module asm ".size g, 8"
+; X86-NEXT: module asm ".section .text.cfi, \22ax\22, @progbits"
+; X86-NEXT: module asm ".balign 8"
+; X86-NEXT: module asm ".cfi.jumptable:"
+; X86-NEXT: module asm "jmp f.cfi@plt"
+; X86-NEXT: module asm "int3"
+; X86-NEXT: module asm "int3"
+; X86-NEXT: module asm "int3"
+; X86-NEXT: module asm "jmp g.cfi@plt"
+; X86-NEXT: module asm "int3"
+; X86-NEXT: module asm "int3"
+; X86-NEXT: module asm "int3"
+
+; ARM:      module asm ".globl f"
+; ARM-NEXT: module asm ".type f, function"
+; ARM-NEXT: module asm "f = .cfi.jumptable + 0"
+; ARM-NEXT: module asm ".size f, 4"
+; ARM-NEXT: module asm ".type g, function"
+; ARM-NEXT: module asm "g = .cfi.jumptable + 4"
+; ARM-NEXT: module asm ".size g, 4"
+; ARM-NEXT: module asm ".section .text.cfi, \22ax\22, @progbits"
+; ARM-NEXT: module asm ".balign 4"
+; ARM-NEXT: module asm ".cfi.jumptable:"
+; ARM-NEXT: module asm "b f.cfi"
+; ARM-NEXT: module asm "b g.cfi"
+
+; X86: @.cfi.jumptable = external hidden constant [2 x [8 x i8]]
+; ARM: @.cfi.jumptable = external hidden constant [2 x [4 x i8]]
+
 ; WASM32: private constant [0 x i8] zeroinitializer
 @0 = private unnamed_addr constant [2 x void (...)*] [void (...)* bitcast (void ()* @f to void (...)*), void (...)* bitcast (void ()* @g to void (...)*)], align 16
 
-; X64: @f = alias void (), bitcast ([2 x <{ i8, i32, i8, i8, i8 }>]* @[[JT]] to void ()*)
-; X64: @g = alias void (), bitcast (<{ i8, i32, i8, i8, i8 }>* getelementptr inbounds ([2 x <{ i8, i32, i8, i8, i8 }>], [2 x <{ i8, i32, i8, i8, i8 }>]* @[[JT]], i64 0, i64 1) to void ()*)
+; X86: @llvm.used = appending global [2 x i8*] [i8* bitcast (void ()* @f.cfi to i8*), i8* bitcast (void ()* @g.cfi to i8*)], section "llvm.metadata"
+; ARM: @llvm.used = appending global [2 x i8*] [i8* bitcast (void ()* @f.cfi to i8*), i8* bitcast (void ()* @g.cfi to i8*)], section "llvm.metadata"
 
-; X64: define private void @[[FNAME]]()
+; X86: define internal void @f.cfi()
+; ARM: define internal void @f.cfi()
 ; WASM32: define void @f() !type !{{[0-9]+}} !wasm.index ![[I0:[0-9]+]]
 define void @f() !type !0 {
   ret void
 }
 
-; X64: define private void @[[GNAME]]()
-; WASM32: define void @g() !type !{{[0-9]+}} !wasm.index ![[I1:[0-9]+]]
-define void @g() !type !0 {
+; X86: define internal void @g.cfi()
+; ARM: define internal void @g.cfi()
+; WASM32: define internal void @g() !type !{{[0-9]+}} !wasm.index ![[I1:[0-9]+]]
+define internal void @g() !type !0 {
   ret void
 }
 
 !0 = !{i32 0, !"typeid1"}
 
 declare i1 @llvm.type.test(i8* %ptr, metadata %bitset) nounwind readnone
 
 define i1 @foo(i8* %p) {
-  ; X64: sub i64 {{.*}}, ptrtoint ([2 x <{ i8, i32, i8, i8, i8 }>]* @[[JT]] to i64)
+  ; X86: sub i64 {{.*}}, ptrtoint ([2 x [8 x i8]]* @.cfi.jumptable to i64)
+  ; ARM: sub i64 {{.*}}, ptrtoint ([2 x [4 x i8]]* @.cfi.jumptable to i64)
   ; WASM32: sub i64 {{.*}}, 1
   ; WASM32: icmp ult i64 {{.*}}, 2
   %x = call i1 @llvm.type.test(i8* %p, metadata !"typeid1")
   ret i1 %x
 }
 
+; X86: declare void @f()
+; ARM: declare void @f()
+; X86: declare hidden void @g()
+; ARM: declare hidden void @g()
+
 ; WASM32: ![[I0]] = !{i64 1}
 ; WASM32: ![[I1]] = !{i64 2}

test/Transforms/LowerTypeTests/section.ll

 ; Test that functions with "section" attribute are accepted, and jumptables are
 ; emitted in ".text".
 
 ; RUN: opt -S -lowertypetests < %s | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: @[[A:.*]] = private constant {{.*}} section ".text"
-; CHECK: @f = alias void (), bitcast ({{.*}}* @[[A]] to void ()*)
-; CHECK: define private void {{.*}} section "xxx"
+; CHECK:      module asm ".section .text.cfi,
+; CHECK:      module asm ".cfi.jumptable:"
+; CHECK-NEXT: module asm "jmp f.cfi@plt"
+
+; CHECK: define internal void @f.cfi() section "xxx"
 
 define void @f() section "xxx" !type !0 {
 entry:
   ret void
 }
 
 define i1 @g() {
 entry:
   %0 = call i1 @llvm.type.test(i8* bitcast (void ()* @f to i8*), metadata !"_ZTSFvE")
   ret i1 %0
 }
 
 declare i1 @llvm.type.test(i8*, metadata) nounwind readnone
 
 !0 = !{i64 0, !"_ZTSFvE"}