[X86] Support ``-mindirect-branch-cs-prefix`` for call and jmp to indirect thunk

[pengfei]

Summary

This is to address feature request from https://github.com/ClangBuiltLinux/linux/issues/1665

[nickdesaulniers]

LGTM; thanks for the patch!

[MaskRay]

When a module with "indirect_branch_cs_prefix" and another without the module flag are merged, what the result should be? If 0, we should use Min instead of Override.

[MaskRay]

clang/lib/Driver/ToolChains/Clang.cpp

6362

Check CC1Option uses in clang/include/clang/Driver/Options.td

[pengfei]

Add CC1 option test.

When a module with "indirect_branch_cs_prefix" and another without the module flag are merged, what the result should be? If 0, we should use Min instead of Override.

I think Override is correct. This option is used for Linux Kernel build. When merged, all should be set to 1.

[MaskRay]

clang/lib/Driver/ToolChains/Clang.cpp

6362

This is not needed with the TableGen CC1Option change.

[pengfei]

clang/lib/Driver/ToolChains/Clang.cpp

6362

Do you have an example how to change the CC1Option? Removing the code directly won't generate the expected module flag anymore.

[MaskRay]

clang/lib/Driver/ToolChains/Clang.cpp

6362

OK, sorry. A statement is needed: Args.AddLastArg(CmdArgs, options::OPT_mindirect_branch_cs_prefix);

clang/test/Driver/x86_features.c

15

Use --target=i386 instead of legacy -target

[pengfei]

Thanks for review!

clang/lib/Driver/ToolChains/Clang.cpp

6362

Good to know such usage. Thank you!

[nikic]

This change caused a significant compile-time regression for O0 builds (about 1%): http://llvm-compile-time-tracker.com/compare.php?from=45bae1be90472c696f6ba3bb4f8fabee76040fa9&to=6f867f9102838ebe314c1f3661fdf95700386e5a&stat=instructions

At a guess, fetching a module flag for every single instruction is slow?

[pengfei]

In D130754#3701837, @nikic wrote:

This change caused a significant compile-time regression for O0 builds (about 1%): http://llvm-compile-time-tracker.com/compare.php?from=45bae1be90472c696f6ba3bb4f8fabee76040fa9&to=6f867f9102838ebe314c1f3661fdf95700386e5a&stat=instructions

At a guess, fetching a module flag for every single instruction is slow?

Thanks for reporting it. I guess so. I'll try to move it outside :)

[pengfei]

In D130754#3701858, @pengfei wrote:

In D130754#3701837, @nikic wrote:

This change caused a significant compile-time regression for O0 builds (about 1%): http://llvm-compile-time-tracker.com/compare.php?from=45bae1be90472c696f6ba3bb4f8fabee76040fa9&to=6f867f9102838ebe314c1f3661fdf95700386e5a&stat=instructions

At a guess, fetching a module flag for every single instruction is slow?

Thanks for reporting it. I guess so. I'll try to move it outside :)

D131245 should help with it.

clang/docs/ReleaseNotes.rst

 CUDA Support in Clang
 ---------------------
 
 - ...
 
 X86 Support in Clang
 --------------------
 
+- Support ``-mindirect-branch-cs-prefix`` for call and jmp to indirect thunk.
+
 DWARF Support in Clang
 ----------------------
 
 Arm and AArch64 Support in Clang
 --------------------------------
 
 Floating Point Support in Clang
 -------------------------------

clang/include/clang/Basic/CodeGenOptions.def

 CODEGENOPT(InstrumentFunctionEntryBare , 1, 0) ///< Set when
                                ///< -finstrument-function-entry-bare is enabled.
 CODEGENOPT(CFProtectionReturn , 1, 0) ///< if -fcf-protection is
                                       ///< set to full or return.
 CODEGENOPT(CFProtectionBranch , 1, 0) ///< if -fcf-protection is
                                       ///< set to full or branch.
 CODEGENOPT(IBTSeal, 1, 0)             ///< set to optimize CFProtectionBranch.
 CODEGENOPT(FunctionReturnThunks, 1, 0) ///< -mfunction-return={keep|thunk-extern}
+CODEGENOPT(IndirectBranchCSPrefix, 1, 0) ///< if -mindirect-branch-cs-prefix
+                                         ///< is set.
 
 CODEGENOPT(XRayInstrumentFunctions , 1, 0) ///< Set when -fxray-instrument is
                                            ///< enabled.
 CODEGENOPT(StackSizeSection  , 1, 0) ///< Set when -fstack-size-section is enabled.
 CODEGENOPT(ForceDwarfFrameSection , 1, 0) ///< Set when -fforce-dwarf-frame is
                                           ///< enabled.
 
 ///< Set when -femit-dwarf-unwind is passed.

clang/include/clang/Driver/Options.td

def mibt_seal : Flag<["-"], "mibt-seal">, Group<m_Group>, Flags<[CoreOption, CC1Option]>,
   HelpText<"Optimize fcf-protection=branch/full (requires LTO).">;
 def mfunction_return_EQ : Joined<["-"], "mfunction-return=">,
   Group<m_Group>, Flags<[CoreOption, CC1Option]>,
   HelpText<"Replace returns with jumps to ``__x86_return_thunk`` (x86 only, error otherwise)">,
   Values<"keep,thunk-extern">,
   NormalizedValues<["Keep", "Extern"]>,
   NormalizedValuesScope<"llvm::FunctionReturnThunksKind">,
   MarshallingInfoEnum<CodeGenOpts<"FunctionReturnThunks">, "Keep">;
+def mindirect_branch_cs_prefix : Flag<["-"], "mindirect-branch-cs-prefix">,
+  Group<m_Group>, Flags<[CoreOption, CC1Option]>,
+  HelpText<"Add cs prefix to call and jmp to indirect thunk">,
+  MarshallingInfoFlag<CodeGenOpts<"IndirectBranchCSPrefix">>;
 
 defm xray_instrument : BoolFOption<"xray-instrument",
   LangOpts<"XRayInstrument">, DefaultFalse,
   PosFlag<SetTrue, [CC1Option], "Generate XRay instrumentation sleds on function entry and exit">,
   NegFlag<SetFalse>>;
 
 def fxray_instruction_threshold_EQ :
   Joined<["-"], "fxray-instruction-threshold=">,

clang/lib/CodeGen/CodeGenModule.cpp

void CodeGenModule::Release() {
   }
 
   if (CodeGenOpts.IBTSeal)
     getModule().addModuleFlag(llvm::Module::Min, "ibt-seal", 1);
 
   if (CodeGenOpts.FunctionReturnThunks)
     getModule().addModuleFlag(llvm::Module::Override, "function_return_thunk_extern", 1);
 
+  if (CodeGenOpts.IndirectBranchCSPrefix)
+    getModule().addModuleFlag(llvm::Module::Override, "indirect_branch_cs_prefix", 1);
+
   // Add module metadata for return address signing (ignoring
   // non-leaf/all) and stack tagging. These are actually turned on by function
   // attributes, but we use module metadata to emit build attributes. This is
   // needed for LTO, where the function attributes are inside bitcode
   // serialised into a global variable by the time build attributes are
   // emitted, so we can't access them. LTO objects could be compiled with
   // different flags therefore module flags are set to "Min" behavior to achieve
   // the same end result of the normal build where e.g BTI is off if any object

clang/lib/Driver/ToolChains/Clang.cpp

 
   if (IsUsingLTO)
     Args.AddLastArg(CmdArgs, options::OPT_mibt_seal);
 
   if (Arg *A = Args.getLastArg(options::OPT_mfunction_return_EQ))
     CmdArgs.push_back(
         Args.MakeArgString(Twine("-mfunction-return=") + A->getValue()));
 
+  Args.AddLastArg(CmdArgs, options::OPT_mindirect_branch_cs_prefix);
+
   // Forward -f options with positive and negative forms; we translate these by
   // hand.  Do not propagate PGO options to the GPU-side compilations as the
   // profile info is for the host-side compilation only.
   if (!(IsCudaDevice || IsHIPDevice)) {
     if (Arg *A = getLastProfileSampleUseArg(Args)) {
       auto *PGOArg = Args.getLastArg(
           options::OPT_fprofile_generate, options::OPT_fprofile_generate_EQ,
           options::OPT_fcs_profile_generate,

clang/test/CodeGen/X86/indirect-branch-cs-prefix.c

+// RUN: %clang -target i386-unknown-unknown -o - -emit-llvm -S -mindirect-branch-cs-prefix %s | FileCheck %s
+
+// CHECK: !{i32 4, !"indirect_branch_cs_prefix", i32 1}
+void foo() {}

clang/test/Driver/x86_features.c

 // RUN: %clang -target i386-unknown-unknown -### -S %s -msse -msse4 -mno-sse -mno-mmx -msse  2>&1 | FileCheck %s
 // CHECK: "pentium4" "-target-feature" "+sse4.2" "-target-feature" "-mmx" "-target-feature" "+sse"
 // Note that we filter out all but the last -m(no)sse.
 
 // Test that we don't produce an error with -mieee-fp.
 // RUN: %clang -### %s -mieee-fp -S 2>&1 | FileCheck --check-prefix=IEEE %s
 // IEEE-NOT: error: unknown argument
 
-// RUN: %clang -target i386-unknown-unknown -### %s -mskip-rax-setup -S 2>&1 | FileCheck --check-prefix=SRS %s
+// RUN: %clang --target=i386 -### %s -mskip-rax-setup -S 2>&1 | FileCheck --check-prefix=SRS %s
 // SRS: "-mskip-rax-setup"
 
-// RUN: %clang -target i386-unknown-unknown -### %s -mno-skip-rax-setup -S 2>&1 | FileCheck --check-prefix=NO-SRS %s
+// RUN: %clang --target=i386 -### %s -mno-skip-rax-setup -S 2>&1 | FileCheck --check-prefix=NO-SRS %s
 // NO-SRS-NOT: "-mskip-rax-setup"
+
+// RUN: %clang --target=i386 -### %s -mindirect-branch-cs-prefix -S 2>&1 | FileCheck --check-prefix=IND-CS %s
+// IND-CS: "-mindirect-branch-cs-prefix"

llvm/lib/Target/X86/X86MCInstLower.cpp

if (TM.Options.MCOptions.ShowMCEncoding) {
     if (MI->getAsmPrinterFlags() & X86::AC_EVEX_2_VEX)
       OutStreamer->AddComment("EVEX TO VEX Compression ", false);
   }
 
   // Add comments for values loaded from constant pool.
   if (OutStreamer->isVerboseAsm())
     addConstantComments(MI, *OutStreamer);
 
+  bool IndCS =
+      MF->getMMI().getModule()->getModuleFlag("indirect_branch_cs_prefix");
+
   switch (MI->getOpcode()) {
   case TargetOpcode::DBG_VALUE:
     llvm_unreachable("Should be handled target independently");
 
   // Emit nothing here but a comment if we can.
   case X86::Int_MemBarrier:
     OutStreamer->emitRawComment("MEMBARRIER");
     return;
if (CurrentPatchableFunctionEntrySym &&
       EmitAndCountInstruction(Inst);
       CurrentPatchableFunctionEntrySym = createTempSymbol("patch");
       OutStreamer->emitLabel(CurrentPatchableFunctionEntrySym);
       return;
     }
     break;
   }
 
+  case X86::TAILJMPd64:
+    if (IndCS && MI->hasRegisterImplicitUseOperand(X86::R11))
+      EmitAndCountInstruction(MCInstBuilder(X86::CS_PREFIX));
+    LLVM_FALLTHROUGH;
   case X86::TAILJMPr:
   case X86::TAILJMPm:
   case X86::TAILJMPd:
   case X86::TAILJMPd_CC:
   case X86::TAILJMPr64:
   case X86::TAILJMPm64:
-  case X86::TAILJMPd64:
   case X86::TAILJMPd64_CC:
   case X86::TAILJMPr64_REX:
   case X86::TAILJMPm64_REX:
     // Lower these as normal, but add some comments.
     OutStreamer->AddComment("TAILCALL");
     break;
 
   case X86::TLS_addr32:
case X86::UBSAN_UD1:
     EmitAndCountInstruction(MCInstBuilder(X86::UD1Lm)
                                 .addReg(X86::EAX)
                                 .addReg(X86::EAX)
                                 .addImm(1)
                                 .addReg(X86::NoRegister)
                                 .addImm(MI->getOperand(0).getImm())
                                 .addReg(X86::NoRegister));
     return;
+  case X86::CALL64pcrel32:
+    if (IndCS && MI->hasRegisterImplicitUseOperand(X86::R11))
+      EmitAndCountInstruction(MCInstBuilder(X86::CS_PREFIX));
+    break;
   }
 
   MCInst TmpInst;
   MCInstLowering.Lower(MI, TmpInst);
 
   // Stackmap shadows cannot include branch targets, so we can count the bytes
   // in a call towards the shadow, but must ensure that the no thread returns
   // in to the stackmap shadow.  The only way to achieve this is if the call

llvm/lib/Target/X86/X86ReturnThunks.cpp

 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/MC/MCInstrDesc.h"
 #include "llvm/Support/Debug.h"
 
 using namespace llvm;
 
 #define PASS_KEY "x86-return-thunks"
 #define DEBUG_TYPE PASS_KEY
 
bool X86ReturnThunks::runOnMachineFunction(MachineFunction &MF) {
   const unsigned RetOpc = Is64Bit ? X86::RET64 : X86::RET32;
   SmallVector<MachineInstr *, 16> Rets;
 
   for (MachineBasicBlock &MBB : MF)
     for (MachineInstr &Term : MBB.terminators())
       if (Term.getOpcode() == RetOpc)
         Rets.push_back(&Term);
 
+  bool IndCS =
+      MF.getMMI().getModule()->getModuleFlag("indirect_branch_cs_prefix");
+  const MCInstrDesc &CS = ST.getInstrInfo()->get(X86::CS_PREFIX);
   const MCInstrDesc &JMP = ST.getInstrInfo()->get(X86::TAILJMPd);
 
   for (MachineInstr *Ret : Rets) {
+    if (IndCS)
+      BuildMI(Ret->getParent(), Ret->getDebugLoc(), CS);
     BuildMI(Ret->getParent(), Ret->getDebugLoc(), JMP)
         .addExternalSymbol(ThunkName.data());
     Ret->eraseFromParent();
     Modified = true;
   }
 
   return Modified;
 }
 
 INITIALIZE_PASS(X86ReturnThunks, PASS_KEY, "X86 Return Thunks", false, false)
 
 FunctionPass *llvm::createX86ReturnThunksPass() {
   return new X86ReturnThunks();
 }

llvm/test/CodeGen/X86/attr-function-return.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc --mtriple=i386-linux-gnu %s -o - -verify-machineinstrs \
 ; RUN:   | FileCheck %s
 ; RUN: llc --mtriple=x86_64-linux-gnu %s -o - -verify-machineinstrs \
 ; RUN:   | FileCheck %s
 define void @x() fn_ret_thunk_extern {
 ; CHECK-LABEL: x:
 ; CHECK:       # %bb.0:
+; CHECK-NEXT:    cs
 ; CHECK-NEXT:    jmp __x86_return_thunk
   ret void
 }
+
+!llvm.module.flags = !{!0}
+
+!0 = !{i32 4, !"indirect_branch_cs_prefix", i32 1}

llvm/test/CodeGen/X86/lvi-hardening-indirectbr.ll

 
 ; X64-LABEL: icall_reg:
 ; X64-DAG:   movq %rdi, %[[fp:[^ ]*]]
 ; X64-DAG:   movl %esi, %[[x:[^ ]*]]
 ; X64:       movl %esi, %edi
 ; X64:       callq bar
 ; X64-DAG:   movl %[[x]], %edi
 ; X64-DAG:   movq %[[fp]], %r11
-; X64:       callq __llvm_lvi_thunk_r11
+; X64:       cs
+; X64-NEXT:  callq __llvm_lvi_thunk_r11
 ; X64:       movl %[[x]], %edi
 ; X64:       callq bar
 ; X64-DAG:   movl %[[x]], %edi
 ; X64-DAG:   movq %[[fp]], %r11
-; X64:       jmp __llvm_lvi_thunk_r11 # TAILCALL
+; X64:       cs
+; X64-NEXT:  jmp __llvm_lvi_thunk_r11 # TAILCALL
 
 ; X64FAST-LABEL: icall_reg:
 ; X64FAST:       callq bar
-; X64FAST:       callq __llvm_lvi_thunk_r11
+; X64FAST:       cs
+; X64FAST-NEXT:  callq __llvm_lvi_thunk_r11
 ; X64FAST:       callq bar
-; X64FAST:       jmp __llvm_lvi_thunk_r11 # TAILCALL
+; X64FAST:       cs
+; X64FAST-NEXT:  jmp __llvm_lvi_thunk_r11 # TAILCALL
 
 
 @global_fp = external dso_local global ptr
 
 ; Test an indirect call through a global variable.
 define void @icall_global_fp(i32 %x, ptr %fpp) #0 {
   %fp1 = load ptr, ptr @global_fp
   call void %fp1(i32 %x)
   %fp2 = load ptr, ptr @global_fp
   tail call void %fp2(i32 %x)
   ret void
 }
 
 ; X64-LABEL: icall_global_fp:
 ; X64-DAG:   movl %edi, %[[x:[^ ]*]]
 ; X64-DAG:   movq global_fp(%rip), %r11
-; X64:       callq __llvm_lvi_thunk_r11
+; X64:       cs
+; X64-NEXT:  callq __llvm_lvi_thunk_r11
 ; X64-DAG:   movl %[[x]], %edi
 ; X64-DAG:   movq global_fp(%rip), %r11
-; X64:       jmp __llvm_lvi_thunk_r11 # TAILCALL
+; X64:       cs
+; X64-NEXT:  jmp __llvm_lvi_thunk_r11 # TAILCALL
 
 ; X64FAST-LABEL: icall_global_fp:
 ; X64FAST:       movq global_fp(%rip), %r11
-; X64FAST:       callq __llvm_lvi_thunk_r11
+; X64FAST:       cs
+; X64FAST-NEXT:  callq __llvm_lvi_thunk_r11
 ; X64FAST:       movq global_fp(%rip), %r11
-; X64FAST:       jmp __llvm_lvi_thunk_r11 # TAILCALL
+; X64FAST:       cs
+; X64FAST-NEXT:  jmp __llvm_lvi_thunk_r11 # TAILCALL
 
 
 %struct.Foo = type { ptr }
 
 ; Test an indirect call through a vtable.
 define void @vcall(ptr %obj) #0 {
   %vptr = load ptr, ptr %obj
   %vslot = getelementptr ptr, ptr %vptr, i32 1
   %fp = load ptr, ptr %vslot
   tail call void %fp(ptr %obj)
   tail call void %fp(ptr %obj)
   ret void
 }
 
 ; X64-LABEL: vcall:
 ; X64:       movq %rdi, %[[obj:[^ ]*]]
 ; X64:       movq (%rdi), %[[vptr:[^ ]*]]
 ; X64:       movq 8(%[[vptr]]), %[[fp:[^ ]*]]
 ; X64:       movq %[[fp]], %r11
-; X64:       callq __llvm_lvi_thunk_r11
+; X64:       cs
+; X64-NEXT:  callq __llvm_lvi_thunk_r11
 ; X64-DAG:   movq %[[obj]], %rdi
 ; X64-DAG:   movq %[[fp]], %r11
-; X64:       jmp __llvm_lvi_thunk_r11 # TAILCALL
+; X64:       cs
+; X64-NEXT:  jmp __llvm_lvi_thunk_r11 # TAILCALL
 
 ; X64FAST-LABEL: vcall:
-; X64FAST:       callq __llvm_lvi_thunk_r11
-; X64FAST:       jmp __llvm_lvi_thunk_r11 # TAILCALL
+; X64FAST:       cs
+; X64FAST-NEXT:  callq __llvm_lvi_thunk_r11
+; X64FAST:       cs
+; X64FAST-NEXT:  jmp __llvm_lvi_thunk_r11 # TAILCALL
 
 
 declare dso_local void @direct_callee()
 
 define void @direct_tail() #0 {
   tail call void @direct_callee()
   ret void
 }
define void @nonlazybind_caller() #0 {
   call void @nonlazybind_callee()
   tail call void @nonlazybind_callee()
   ret void
 }
 
 ; X64-LABEL: nonlazybind_caller:
 ; X64:       movq nonlazybind_callee@GOTPCREL(%rip), %[[REG:.*]]
 ; X64:       movq %[[REG]], %r11
-; X64:       callq __llvm_lvi_thunk_r11
+; X64:       cs
+; X64-NEXT:  callq __llvm_lvi_thunk_r11
 ; X64:       movq %[[REG]], %r11
-; X64:       jmp __llvm_lvi_thunk_r11 # TAILCALL
+; X64:       cs
+; X64-NEXT:  jmp __llvm_lvi_thunk_r11 # TAILCALL
 ; X64FAST-LABEL: nonlazybind_caller:
 ; X64FAST:   movq nonlazybind_callee@GOTPCREL(%rip), %r11
-; X64FAST:   callq __llvm_lvi_thunk_r11
+; X64FAST:   cs
+; X64FAST-NEXT:  callq __llvm_lvi_thunk_r11
 ; X64FAST:   movq nonlazybind_callee@GOTPCREL(%rip), %r11
-; X64FAST:   jmp __llvm_lvi_thunk_r11 # TAILCALL
+; X64FAST:   cs
+; X64FAST-NEXT:  jmp __llvm_lvi_thunk_r11 # TAILCALL
 
 
 ; Check that a switch gets lowered using a jump table
 define void @switch_jumptable(ptr %ptr, ptr %sink) #0 {
 ; X64-LABEL: switch_jumptable:
 ; X64-NOT:      jmpq *
 entry:
   br label %header

 ; X64-NEXT:          .hidden __llvm_lvi_thunk_r11
 ; X64-NEXT:          .weak   __llvm_lvi_thunk_r11
 ; X64:       __llvm_lvi_thunk_r11:
 ; X64-NEXT:  # {{.*}}                                # %entry
 ; X64-NEXT:          lfence
 ; X64-NEXT:          jmpq     *%r11
 
 attributes #1 = { nonlazybind }
+
+!llvm.module.flags = !{!0}
+
+!0 = !{i32 4, !"indirect_branch_cs_prefix", i32 1}