Added control flow architecture protection Flag

[oren_ben_simhon]

Summary

Instrument control flow is target independent flag that instructs the back-end to instrument control flow mechanisms.
Specifically in X86 this flag will be used to instrument Indirect Branch Tracking instructions.

[craig.topper]

lib/Driver/ToolChains/Clang.cpp

3979

This should probably have a blank line above it. Right now it kinda looks like it goes with the OpenCL code.

[oren_ben_simhon]

Implemented comments posted until 11/29 (Thanks Craig)

[pcc]

include/clang/Driver/Options.td

1040

My comments about the attribute name in D40482 also apply to this flag name. GCC appears to have chosen a better name for this flag, -mcet. Can we use that name instead?

[oren_ben_simhon]

include/clang/Driver/Options.td

1040

Again, I am sorry for the late response, I was off for about a week.
GCC is only using -mcet as a super set for -mibt and -mshstk. I am planning to add it in a different review.
-mcet/-mibt/-mshstk only enables the HW technology (ISA/Intrinsics).
GCC is using -cf-protection for actually instrumenting the control flow.
I think it will be best to stay coherent with GCC (and ICC and soon MS) and use -cf-protection.
What do you think?

[pcc]

include/clang/Driver/Options.td

1040

I see. As far as I can tell, there are no released versions of GCC and ICC with the new flags, so I think there is still a possibility to change them. (MS is a different story, their flags are incompatible anyway.)

I took a closer look at what GCC does, and here are my thoughts:

• As far as I can tell, IBT does not introduce any new instructions other than ENDBRANCH, and it doesn't seem to make sense for ENDBRANCH to be exposed via intrinsics (but rather via attributes, as you have done). That means that -mibt basically does nothing except for allowing the use of -fcf-protection.
• SHSTK introduces ISA extensions that seem reasonable to expose via intrinsics, so it would probably be fine to have a -mshstk flag that enables the intrinsics.
• In GCC, -fcf-protection means "emit code that is compatible with CET" and requires that the user pass -mibt or -mshstk. But the code that it emits is also compatible with non-CET processors, so it doesn't make sense to me to require the use of flags such as -mshstk which imply that SHSTK is required.

As for the name of the flag: there is nothing about the name -fcf-protection that implies hardware-based protection, so the presence of this flag could confuse users who want software-based protection. The hardware-based protection is sort of implied by the requirement to pass -mibt or -mshstk, but that is a little awkward, and I don't think it makes sense either for the reasons I mentioned. What we want is a clear way to say "I want to generate code that is compatible with (but does not require) this hardware feature", but there doesn't seem to be any precedent for how to express that. The most straightforward way to express it is to mention the name of the hardware feature in the flag.

So here are the changes that I would propose across compilers:

• Remove the -mibt flag.
• Change the flag that emits ENDBRANCH to something like -mcet-compatible and stop requiring the use of other flags.

Please let me know what you think.

[oren_ben_simhon]

include/clang/Driver/Options.td

1040

It is true that -mibt doesn't enable intrinsics however it is required for the case of inline asm.
The code that -cf-protection produce is not always compatible with non-CET processors.
For example, consider the required fix to the shadow stack in the case of setJmp/longJmp.
In that case, we need to use SHSTK instructions (like incssp) that are not compatible.
Would the flag -fcf-arch-protection make more sense (as it implies on HW support)?

[pcc]

include/clang/Driver/Options.td

1040

It is true that -mibt doesn't enable intrinsics however it is required for the case of inline asm.

For enabling ENDBRANCH in inline asm? Would there be any harm in allowing ENDBRANCH unconditionally?

For example, consider the required fix to the shadow stack in the case of setJmp/longJmp.
In that case, we need to use SHSTK instructions (like incssp) that are not compatible.

I don't think that is true. I looked at the code that GCC emits for setjmp and longjmp, and it appears to quite cleverly avoid the need to execute SHSTK-required instructions on older processors by taking advantage of the fact that RDSSP is a NOP on older processors.

With this program:

#include <setjmp.h>

jmp_buf env;
void f() {
  __builtin_setjmp(env);
}

void g() {
  __builtin_longjmp(env, 1);
}

I get this asm for saving the SSP:

	movl	$0, %eax
	rdsspq	%rax
	movq	%rax, env+24(%rip)

and this asm for restoring it:

	movl	$0, %eax
	rdsspq	%rax
	subq	env+24(%rip), %rax
	je	.L5
	negq	%rax
	shrq	$3, %rax
	cmpq	$255, %rax
	jbe	.L6
.L7:
	incsspq	%rax
	subq	$255, %rax
	cmpq	$255, %rax
	ja	.L7
.L6:
	incsspq	%rax
.L5:

So on a non-SHSTK processor, we will store 0 to the slot reserved for SSP during setjmp, and then compare 0 to 0 during longjmp and bypass the entire loop.

Would the flag -fcf-arch-protection make more sense (as it implies on HW support)?

It isn't ideal to me, but it does at least seem better than -fcf-protection.

[oren_ben_simhon]

Implemented some of the comments posted until 12/12 (Thanks Peter)

[oren_ben_simhon]

include/clang/Driver/Options.td

1040

You are correct, -mshstk flag should be used only for enabling the intrinsics that are not rdssp.
It shouldn't be a requirement for generating shadow stack fixes. Is that what you meant?

[craig.topper]

-mibt isn't required for inline assembly, There's no AssemblePredicate defined with HasIBT in X86InstrInfo.td. We don't normally do fine grained assembler feature enabling so I believe we should always support it.

[pcc]

include/clang/Driver/Options.td

1040

Yes, exactly.

[craig.topper]

include/clang/Driver/Options.td

1040

I think the rdssp is a nop on older Intel processors, but its hard to say for sure. Older instruction manuals(I checked a 2001 version I had on my bookshelf) from Intel don't list anything in the opcode map for those instruction encodings. There are also other vendors that have made X86 CPUs that may not have implemented them as NOPs since they were undocumented. The "cleverness" is really there for CPUs that support CET instructions, but do not have OS support enabled via CR4.

[oren_ben_simhon]

Implemented all comments posted until 12/14 (Thanks Peter)

[oren_ben_simhon]

-mibt is currently in discussions with other compilers, any change will be uploaded to a different review.
If you have more comments i will appreciate it.

[craig.topper]

lib/CodeGen/CGCall.cpp

1737 ↗

(On Diff #127008)

If the command line option is intended to be target independent, shouldn't these generically named?

lib/CodeGen/CodeGenFunction.cpp

876–877

Why this change?

[oren_ben_simhon]

lib/CodeGen/CodeGenFunction.cpp

876–877

Apparently, some compilers do strange optimizations on bit structures (like CodeGenOptions) that cause this code to be compiled wrongly.
I encountered this issue when i added a new Code Gen Option and compiled clang in windows.
In order to overcome this issue, i changed a bit the code and the wrong optimization didn't occur.

[oren_ben_simhon]

Implemented comments posted until 12/19 (Thanks Craig)

[oren_ben_simhon]

ping

[craig.topper]

Are we sure we want a different command line option name from gcc? From our internal conversations with the gcc folks I thought they were suggesting that -fcf-protection could imply a software mechanism if a hardware mechanism was not available thorugh -mibt or -march?

Should we emit an error to the user if -mibt isn't available? We should be able to add virtual methods on TargetInfo that X86 can customize to check for ibt and shstk.

Can you provide more information about the miscompile on MSVC? I think we should do more to understand that, this sounds like it could be a time bomb waiting to fail somewhere else.

[oren_ben_simhon]

In D40478#962348, @craig.topper wrote:

Are we sure we want a different command line option name from gcc? From our internal conversations with the gcc folks I thought they were suggesting that -fcf-protection could imply a software mechanism if a hardware mechanism was not available thorugh -mibt or -march?

Should we emit an error to the user if -mibt isn't available? We should be able to add virtual methods on TargetInfo that X86 can customize to check for ibt and shstk.

Can you provide more information about the miscompile on MSVC? I think we should do more to understand that, this sounds like it could be a time bomb waiting to fail somewhere else.

LLVM already has a flag for SW mechanisms "-sanitize=*". Anyway I believe that GCC and LLVM should agree first before i change it. I restored cf-protection and will create a new patch review after an agreement with GCC will be made.

I added an error message.

I currently don't have more information. After seeing the issue I workaround it. I prefer not to open MSVC miscompilation issue in this code review and investigate it on a different thread.

[oren_ben_simhon]

Implemented comments posted until 12/24 (Thanks Craig)
Moved cf-protection attributes from function attributes to module attributes.

[oren_ben_simhon]

Reverted clang-format whitespaces updates

[craig.topper]

lib/CodeGen/CodeGenModule.cpp

501

Should we still be adding the module flag if the target says its not supported?

[oren_ben_simhon]

Implemented comments posted until 01/04 (Thanks Craig)

[craig.topper]

LGTM

[alexfh]

test/CodeGen/x86-cf-protection.c

2

Any reason this test runs clang with "-S" and "-emit-llvm"? Neither of those seems to be needed for the actual checks being made below.

[oren_ben_simhon]

test/CodeGen/x86-cf-protection.c

2

I agree that -emit-llvm is redundant for the test checks. Without -S the command has no output on stdout or stderr. So the run fails and doesn't execute.

[alexfh]

test/CodeGen/x86-cf-protection.c

2

The specific problem I've stumbled upon is that the test fails when the source file is on a read-only partition:

[3250/3251] Running the Clang regression tests    
llvm-lit: /src/utils/lit/lit/llvm/config.py:334: note: using clang: /build/bin/clang
FAIL: Clang :: CodeGen/x86-cf-protection.c (2485 of 11862)
******************** TEST 'Clang :: CodeGen/x86-cf-protection.c' FAILED ********************
Script:                                           
--                                                
not /build/bin/clang -cc1 -internal-isystem /build/lib/clang/7.0.0/include -nostdsysteminc -fsyntax-only -S -emit-llvm -triple i386-unknown-unknown -fcf-protection=return /src/tools/clang/test/CodeGen/x86-cf-protection.c 2>&1 | /build/bin/FileCheck /src/tools/clang/test/CodeGen/x86
-cf-protection.c --check-prefix=RETURN            
not /build/bin/clang -cc1 -internal-isystem /build/lib/clang/7.0.0/include -nostdsysteminc -fsyntax-only -S -emit-llvm -triple i386-unknown-unknown -fcf-protection=branch /src/tools/clang/test/CodeGen/x86-cf-protection.c 2>&1 | /build/bin/FileCheck /src/tools/clang/test/CodeGen/x86
-cf-protection.c --check-prefix=BRANCH            
--                                                
Exit Code: 1                                      
                                                  
Command Output (stderr):                          
--                                                
/src/tools/clang/test/CodeGen/x86-cf-protection.c:4:12: error: expected string not found in input
// RETURN: error: option 'cf-protection=return' cannot be specified without '-mshstk'
           ^                                      
<stdin>:1:1: note: scanning from here             
error: unable to open output file '': 'Read-only file system'
^                                                 
                                                  
--

Adding -o %t solves the problem (r322082).

[oren_ben_simhon]

test/CodeGen/x86-cf-protection.c

2

I see. Thanks for the fix.

include/clang/Basic/DiagnosticCommonKinds.td

 def err_target_unsupported_fpmath : Error<
     "the '%0' unit is not supported with this instruction set">;
 def err_target_unsupported_unaligned : Error<
   "the %0 sub-architecture does not support unaligned accesses">;
 def err_target_unsupported_execute_only : Error<
   "execute only is not supported for the %0 sub-architecture">;
 def err_opt_not_valid_with_opt : Error<
   "option '%0' cannot be specified with '%1'">;
+def err_opt_not_valid_without_opt : Error<
+  "option '%0' cannot be specified without '%1'">;
 
 // Source manager
 def err_cannot_open_file : Error<"cannot open file '%0': %1">, DefaultFatal;
 def err_file_modified : Error<
   "file '%0' modified since it was first processed">, DefaultFatal;
 def err_unsupported_bom : Error<"%0 byte order mark detected in '%1', but "
   "encoding is not supported">, DefaultFatal;
 def err_unable_to_rename_temp : Error<

include/clang/Basic/TargetInfo.h

public:
   }
 
   /// Controls if __builtin_longjmp / __builtin_setjmp can be lowered to
   /// llvm.eh.sjlj.longjmp / llvm.eh.sjlj.setjmp.
   virtual bool hasSjLjLowering() const {
     return false;
   }
 
+  /// Check if the target supports CFProtection branch.
+  virtual void
+  checkCFProtectionBranchSupported(DiagnosticsEngine &Diags) const {}
+
+  /// Check if the target supports CFProtection branch.
+  virtual void
+  checkCFProtectionReturnSupported(DiagnosticsEngine &Diags) const {}
+
   /// \brief Whether target allows to overalign ABI-specified preferred alignment
   virtual bool allowsLargerPreferedTypeAlignment() const { return true; }
 
   /// \brief Set supported OpenCL extensions and optional core features.
   virtual void setSupportedOpenCLOpts() {}
 
   /// \brief Set supported OpenCL extensions as written on command line
   virtual void setOpenCLExtensionOpts() {

include/clang/Driver/Options.td

 def finput_charset_EQ : Joined<["-"], "finput-charset=">, Group<f_Group>;
 def fexec_charset_EQ : Joined<["-"], "fexec-charset=">, Group<f_Group>;
 def finstrument_functions : Flag<["-"], "finstrument-functions">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Generate calls to instrument function entry and exit">;
 def finstrument_functions_after_inlining : Flag<["-"], "finstrument-functions-after-inlining">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Like -finstrument-functions, but insert the calls after inlining">;
 def finstrument_function_entry_bare : Flag<["-"], "finstrument-function-entry-bare">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Instrument function entry only, after inlining, without arguments to the instrumentation call">;
+def fcf_protection_EQ : Joined<["-"], "fcf-protection=">, Flags<[CoreOption, CC1Option]>, Group<f_Group>,
+  HelpText<"Instrument control-flow architecture protection. Options: return, branch, full, none.">, Values<"return,branch,full,none">;
+def fcf_protection : Flag<["-"], "fcf-protection">, Group<f_Group>, Flags<[CoreOption, CC1Option]>,
+  Alias<fcf_protection_EQ>, AliasArgs<["full"]>,
+  HelpText<"Enable cf-protection in 'full' mode">;
 
 def fxray_instrument : Flag<["-"], "fxray-instrument">, Group<f_Group>,
   Flags<[CC1Option]>,
   HelpText<"Generate XRay instrumentation sleds on function entry and exit">;
 def fnoxray_instrument : Flag<["-"], "fno-xray-instrument">, Group<f_Group>,
   Flags<[CC1Option]>;
 
 def fxray_instruction_threshold_EQ :
   JoinedOrSeparate<["-"], "fxray-instruction-threshold=">,

include/clang/Frontend/CodeGenOptions.def

CODEGENOPT(ForbidGuardVariables , 1, 0) ///< Issue errors if C++ guard variables
                                         ///< are required.
 CODEGENOPT(FunctionSections  , 1, 0) ///< Set when -ffunction-sections is enabled.
 CODEGENOPT(InstrumentFunctions , 1, 0) ///< Set when -finstrument-functions is
                                        ///< enabled.
 CODEGENOPT(InstrumentFunctionsAfterInlining , 1, 0) ///< Set when
                           ///< -finstrument-functions-after-inlining is enabled.
 CODEGENOPT(InstrumentFunctionEntryBare , 1, 0) ///< Set when
                                ///< -finstrument-function-entry-bare is enabled.
-
+CODEGENOPT(CFProtectionReturn , 1, 0) ///< if -fcf-protection is
+                                      ///< set to full or return.
+CODEGENOPT(CFProtectionBranch , 1, 0) ///< if -fcf-protection is
+                                      ///< set to full or branch.
 CODEGENOPT(XRayInstrumentFunctions , 1, 0) ///< Set when -fxray-instrument is
                                            ///< enabled.
 
 ///< Set the minimum number of instructions in a function to determine selective
 ///< XRay instrumentation.
 VALUE_CODEGENOPT(XRayInstructionThreshold , 32, 200)
 
 CODEGENOPT(InstrumentForProfiling , 1, 0) ///< Set when -pg is enabled.

lib/Basic/Targets/X86.h

bool validateGlobalRegisterVariable(StringRef RegName, unsigned RegSize,
 
     return false;
   }
 
   bool validateOutputSize(StringRef Constraint, unsigned Size) const override;
 
   bool validateInputSize(StringRef Constraint, unsigned Size) const override;
 
+  virtual void
+  checkCFProtectionReturnSupported(DiagnosticsEngine &Diags) const override;
+
+  virtual void
+  checkCFProtectionBranchSupported(DiagnosticsEngine &Diags) const override;
+
   virtual bool validateOperandSize(StringRef Constraint, unsigned Size) const;
 
   std::string convertConstraint(const char *&Constraint) const override;
   const char *getClobbers() const override {
     return "~{dirflag},~{fpsr},~{flags}";
   }
 
   StringRef getConstraintRegister(const StringRef &Constraint,

lib/Basic/Targets/X86.cpp

bool X86TargetInfo::setFPMath(StringRef Name) {
   }
   if (Name == "sse") {
     FPMath = FP_SSE;
     return true;
   }
   return false;
 }
 
+void X86TargetInfo::checkCFProtectionReturnSupported(
+    DiagnosticsEngine &Diags) const {
+  if (!HasSHSTK)
+    Diags.Report(diag::err_opt_not_valid_without_opt) << "cf-protection=return"
+                                                      << "-mshstk";
+}
+
+void X86TargetInfo::checkCFProtectionBranchSupported(
+    DiagnosticsEngine &Diags) const {
+  if (!HasIBT)
+    Diags.Report(diag::err_opt_not_valid_without_opt) << "cf-protection=branch"
+                                                      << "-mibt";
+}
+
 bool X86TargetInfo::initFeatureMap(
     llvm::StringMap<bool> &Features, DiagnosticsEngine &Diags, StringRef CPU,
     const std::vector<std::string> &FeaturesVec) const {
   // FIXME: This *really* should not be here.
   // X86_64 always has SSE2.
   if (getTriple().getArch() == llvm::Triple::x86_64)
     setFeatureEnabledImpl(Features, "sse2", true);
 

lib/CodeGen/CodeGenFunction.cpp

#undef SANITIZER
   // from void* to T* before object initialization completes. Don't match on the
   // namespace because not all allocators are in std::
   if (D && SanOpts.has(SanitizerKind::CFIUnrelatedCast)) {
     if (matchesStlAllocatorFn(D, getContext()))
       SanOpts.Mask &= ~SanitizerKind::CFIUnrelatedCast;
   }
 
   // Apply xray attributes to the function (as a string, for now)
-  if (D && ShouldXRayInstrumentFunction()) {
+  bool InstrumentXray = ShouldXRayInstrumentFunction();
+  if (D && InstrumentXray) {
     if (const auto *XRayAttr = D->getAttr<XRayInstrumentAttr>()) {
       if (XRayAttr->alwaysXRayInstrument())
         Fn->addFnAttr("function-instrument", "xray-always");
       if (XRayAttr->neverXRayInstrument())
         Fn->addFnAttr("function-instrument", "xray-never");
       if (const auto *LogArgs = D->getAttr<XRayLogArgsAttr>()) {
         Fn->addFnAttr("xray-log-args",
                       llvm::utostr(LogArgs->getArgumentCount()));

lib/CodeGen/CodeGenModule.cpp

if (   Arch == llvm::Triple::arm
     getModule().addModuleFlag(llvm::Module::Error, "min_enum_size", EnumWidth);
   }
 
   if (CodeGenOpts.SanitizeCfiCrossDso) {
     // Indicate that we want cross-DSO control flow integrity checks.
     getModule().addModuleFlag(llvm::Module::Override, "Cross-DSO CFI", 1);
   }
 
+  if (CodeGenOpts.CFProtectionReturn) {
+    Target.checkCFProtectionReturnSupported(getDiags());
+    // Indicate that we want to instrument return control flow protection.
+    getModule().addModuleFlag(llvm::Module::Override, "cf-protection-return",
+                              1);
+  }
+
+  if (CodeGenOpts.CFProtectionBranch) {
+    Target.checkCFProtectionBranchSupported(getDiags());
+    // Indicate that we want to instrument branch control flow protection.
+    getModule().addModuleFlag(llvm::Module::Override, "cf-protection-branch",
+                              1);
+  }
+
   if (LangOpts.CUDAIsDevice && getTriple().isNVPTX()) {
     // Indicate whether __nvvm_reflect should be configured to flush denormal
     // floating point values to 0.  (This corresponds to its "__CUDA_FTZ"
     // property.)
     getModule().addModuleFlag(llvm::Module::Override, "nvvm-reflect-ftz",
                               LangOpts.CUDADeviceFlushDenormalsToZero ? 1 : 0);
   }
 

lib/Driver/ToolChains/Clang.cpp

if (A->getOption().matches(options::OPT_mrestrict_it)) {
               Triple.getArch() == llvm::Triple::thumb)) {
     // Windows on ARM expects restricted IT blocks
     CmdArgs.push_back("-backend-option");
     CmdArgs.push_back("-arm-restrict-it");
   }
 
   // Forward -cl options to -cc1
   RenderOpenCLOptions(Args, CmdArgs);
 
+  if (Arg *A = Args.getLastArg(options::OPT_fcf_protection_EQ)) {
+    CmdArgs.push_back(
+        Args.MakeArgString(Twine("-fcf-protection=") + A->getValue()));
+  }
+
   // Forward -f options with positive and negative forms; we translate
   // these by hand.
   if (Arg *A = getLastProfileSampleUseArg(Args)) {
     StringRef fname = A->getValue();
     if (!llvm::sys::fs::exists(fname))
       D.Diag(diag::err_drv_no_such_file) << fname;
     else
       A->render(Args, CmdArgs);

lib/Frontend/CompilerInvocation.cpp

Opts.InstrumentFunctionEntryBare =
       Args.hasArg(OPT_finstrument_function_entry_bare);
   Opts.XRayInstrumentFunctions = Args.hasArg(OPT_fxray_instrument);
   Opts.XRayInstructionThreshold =
       getLastArgIntValue(Args, OPT_fxray_instruction_threshold_EQ, 200, Diags);
   Opts.InstrumentForProfiling = Args.hasArg(OPT_pg);
   Opts.CallFEntry = Args.hasArg(OPT_mfentry);
   Opts.EmitOpenCLArgMetadata = Args.hasArg(OPT_cl_kernel_arg_info);
 
+  if (const Arg *A = Args.getLastArg(OPT_fcf_protection_EQ)) {
+    StringRef Name = A->getValue();
+    if (Name == "full") {
+      Opts.CFProtectionReturn = 1;
+      Opts.CFProtectionBranch = 1;
+    } else if (Name == "return")
+      Opts.CFProtectionReturn = 1;
+    else if (Name == "branch")
+      Opts.CFProtectionBranch = 1;
+    else if (Name != "none") {
+      Diags.Report(diag::err_drv_invalid_value) << A->getAsString(Args) << Name;
+      Success = false;
+    }
+  }
+
   if (const Arg *A = Args.getLastArg(OPT_compress_debug_sections,
                                      OPT_compress_debug_sections_EQ)) {
     if (A->getOption().getID() == OPT_compress_debug_sections) {
       // TODO: be more clever about the compression type auto-detection
       Opts.setCompressDebugSections(llvm::DebugCompressionType::GNU);
     } else {
       auto DCT = llvm::StringSwitch<llvm::DebugCompressionType>(A->getValue())
                      .Case("none", llvm::DebugCompressionType::None)

test/CodeGen/complex-builtins.c

 // NO__ERRNO: declare { double, double } @ctanh(double, double) [[READNONE]]
 // NO__ERRNO: declare <2 x float> @ctanhf(<2 x float>) [[READNONE]]
 // NO__ERRNO: declare { x86_fp80, x86_fp80 } @ctanhl({ x86_fp80, x86_fp80 }* byval align 16) [[NOT_READNONE]]
 // HAS_ERRNO: declare { double, double } @ctanh(double, double) [[NOT_READNONE]]
 // HAS_ERRNO: declare <2 x float> @ctanhf(<2 x float>) [[NOT_READNONE]]
 // HAS_ERRNO: declare { x86_fp80, x86_fp80 } @ctanhl({ x86_fp80, x86_fp80 }* byval align 16) [[NOT_READNONE]]
 };
 
-
 // NO__ERRNO: attributes [[READNONE]] = { {{.*}}readnone{{.*}} }
 // NO__ERRNO: attributes [[NOT_READNONE]] = { nounwind "correctly{{.*}} }
 
 // HAS_ERRNO: attributes [[NOT_READNONE]] = { nounwind "correctly{{.*}} }
 // HAS_ERRNO: attributes [[READNONE]] = { {{.*}}readnone{{.*}} }
-

test/CodeGen/complex-libcalls.c

 // NO__ERRNO: declare { double, double } @ctanh(double, double) [[READNONE]]
 // NO__ERRNO: declare <2 x float> @ctanhf(<2 x float>) [[READNONE]]
 // NO__ERRNO: declare { x86_fp80, x86_fp80 } @ctanhl({ x86_fp80, x86_fp80 }* byval align 16) [[NOT_READNONE]]
 // HAS_ERRNO: declare { double, double } @ctanh(double, double) [[NOT_READNONE]]
 // HAS_ERRNO: declare <2 x float> @ctanhf(<2 x float>) [[NOT_READNONE]]
 // HAS_ERRNO: declare { x86_fp80, x86_fp80 } @ctanhl({ x86_fp80, x86_fp80 }* byval align 16) [[NOT_READNONE]]
 };
 
-
 // NO__ERRNO: attributes [[READNONE]] = { {{.*}}readnone{{.*}} }
 // NO__ERRNO: attributes [[NOT_READNONE]] = { nounwind "correctly{{.*}} }
 
 // HAS_ERRNO: attributes [[NOT_READNONE]] = { nounwind "correctly{{.*}} }
 // HAS_ERRNO: attributes [[READNONE]] = { {{.*}}readnone{{.*}} }
-

test/CodeGen/math-builtins.c

 // NO__ERRNO: declare double @llvm.trunc.f64(double) [[READNONE_INTRINSIC]]
 // NO__ERRNO: declare float @llvm.trunc.f32(float) [[READNONE_INTRINSIC]]
 // NO__ERRNO: declare x86_fp80 @llvm.trunc.f80(x86_fp80) [[READNONE_INTRINSIC]]
 // HAS_ERRNO: declare double @llvm.trunc.f64(double) [[READNONE_INTRINSIC]]
 // HAS_ERRNO: declare float @llvm.trunc.f32(float) [[READNONE_INTRINSIC]]
 // HAS_ERRNO: declare x86_fp80 @llvm.trunc.f80(x86_fp80) [[READNONE_INTRINSIC]]
 };
 
-
 // NO__ERRNO: attributes [[READNONE]] = { {{.*}}readnone{{.*}} }
 // NO__ERRNO: attributes [[READNONE_INTRINSIC]] = { {{.*}}readnone{{.*}} }
 // NO__ERRNO: attributes [[NOT_READNONE]] = { nounwind "correctly{{.*}} }
 
 // HAS_ERRNO: attributes [[NOT_READNONE]] = { nounwind "correctly{{.*}} }
 // HAS_ERRNO: attributes [[READNONE_INTRINSIC]] = { {{.*}}readnone{{.*}} }
 // HAS_ERRNO: attributes [[READNONE]] = { {{.*}}readnone{{.*}} }
 
 // HAS_ERRNO_GNU: attributes [[READNONE_INTRINSIC]] = { {{.*}}readnone{{.*}} }
 // HAS_ERRNO_WIN: attributes [[READNONE_INTRINSIC]] = { {{.*}}readnone{{.*}} }
 

test/CodeGen/math-libcalls.c

 // NO__ERRNO: declare double @trunc(double) [[READNONE]]
 // NO__ERRNO: declare float @truncf(float) [[READNONE]]
 // NO__ERRNO: declare x86_fp80 @truncl(x86_fp80) [[READNONE]]
 // HAS_ERRNO: declare double @trunc(double) [[READNONE]]
 // HAS_ERRNO: declare float @truncf(float) [[READNONE]]
 // HAS_ERRNO: declare x86_fp80 @truncl(x86_fp80) [[READNONE]]
 };
 
-
 // NO__ERRNO: attributes [[READNONE]] = { {{.*}}readnone{{.*}} }
 // NO__ERRNO: attributes [[NOT_READNONE]] = { nounwind "correctly{{.*}} }
 // NO__ERRNO: attributes [[READONLY]] = { {{.*}}readonly{{.*}} }
 // NO__ERRNO: attributes [[READNONE_INTRINSIC]] = { {{.*}}readnone{{.*}} }
 
 // HAS_ERRNO: attributes [[NOT_READNONE]] = { nounwind "correctly{{.*}} }
 // HAS_ERRNO: attributes [[READNONE]] = { {{.*}}readnone{{.*}} }
 // HAS_ERRNO: attributes [[READONLY]] = { {{.*}}readonly{{.*}} }
 
 // HAS_ERRNO_GNU: attributes [[READNONE_INTRINSIC]] = { {{.*}}readnone{{.*}} }
 // HAS_ERRNO_WIN: attributes [[READNONE_INTRINSIC]] = { {{.*}}readnone{{.*}} }

test/CodeGen/x86-cf-protection.c

+// RUN: not %clang_cc1 -fsyntax-only -S -emit-llvm -triple i386-unknown-unknown -fcf-protection=return %s 2>&1 | FileCheck %s --check-prefix=RETURN
+// RUN: not %clang_cc1 -fsyntax-only -S -emit-llvm -triple i386-unknown-unknown -fcf-protection=branch %s 2>&1 | FileCheck %s --check-prefix=BRANCH
+
+// RETURN: error: option 'cf-protection=return' cannot be specified without '-mshstk'
+// BRANCH: error: option 'cf-protection=branch' cannot be specified without '-mibt'
+void foo() {}

test/Driver/clang_f_opts.c

 // RUN: %clang -### -S -fno-allow-editor-placeholders %s 2>&1 | FileCheck -check-prefix=CHECK-NO-ALLOW-PLACEHOLDERS %s
 // CHECK-ALLOW-PLACEHOLDERS: -fallow-editor-placeholders
 // CHECK-NO-ALLOW-PLACEHOLDERS-NOT: -fallow-editor-placeholders
 
 // RUN: %clang -### -target x86_64-unknown-windows-msvc -fno-short-wchar %s 2>&1 | FileCheck -check-prefix CHECK-WINDOWS-ISO10646 %s
 // CHECK-WINDOWS-ISO10646: "-fwchar-type=int"
 // CHECK-WINDOWS-ISO10646: "-fsigned-wchar"
 
+// RUN: %clang -### -S -fcf-protection %s 2>&1 | FileCheck -check-prefix=CHECK-CF-PROTECTION-FULL %s
+// RUN: %clang -### -S %s 2>&1 | FileCheck -check-prefix=CHECK-NO-CF-PROTECTION-FULL %s
+// RUN: %clang -### -S -fcf-protection=full %s 2>&1 | FileCheck -check-prefix=CHECK-CF-PROTECTION-FULL %s
+// RUN: %clang -### -S %s 2>&1 | FileCheck -check-prefix=CHECK-NO-CF-PROTECTION-FULL %s
+// CHECK-CF-PROTECTION-FULL: -fcf-protection=full
+// CHECK-NO-CF-PROTECTION-FULL-NOT: -fcf-protection=full
+// RUN: %clang -### -S -fcf-protection=return %s 2>&1 | FileCheck -check-prefix=CHECK-CF-PROTECTION-RETURN %s
+// RUN: %clang -### -S %s 2>&1 | FileCheck -check-prefix=CHECK-NO-CF-PROTECTION-RETURN %s
+// CHECK-CF-PROTECTION-RETURN: -fcf-protection=return
+// CHECK-NO-CF-PROTECTION-RETURN-NOT: -fcf-protection=return
+// RUN: %clang -### -S -fcf-protection=branch %s 2>&1 | FileCheck -check-prefix=CHECK-CF-PROTECTION-BRANCH %s
+// RUN: %clang -### -S %s 2>&1 | FileCheck -check-prefix=CHECK-NO-CF-PROTECTION-BRANCH %s
+// CHECK-CF-PROTECTION-BRANCH: -fcf-protection=branch
+// CHECK-NO-CF-PROTECTION-BRANCH-NOT: -fcf-protection=branch