Use-after-return sanitizer binary metadata

[dvyukov]

Summary

Currently per-function metadata consists of:
(start-pc, size, features)

This adds a new UAR feature and if it's set an additional element:
(start-pc, size, features, stack-args-size)

[dvyukov]

WIP

[dvyukov]

WIP

[dvyukov]

WIP

[dvyukov]

Marco, does this look reasonable? Any high-level comments?

There are lots of plumbing to add a new flag and integrate a new pass, so you can look only at:
llvm/lib/Transforms/Instrumentation/SanitizerBinaryMetadata.cpp
llvm/lib/CodeGen/SanitizerMetadata.cpp

As far as I understand the stack layout is only known in machine passes via MachineFrameInfo,
so I had to add a machine pass.

[dvyukov]

Unrelated, but looking at the metadata format more closely, I think this can benefit from LEB128-like varlen encoding.
Function size is small.
Features are very small.
Stack args size is small.
Function start (if we encode it from the end of the previous one) is very small as well.
So potentially it can reduce the entry size from 16 bytes to just 4.
I am just thinking if we use 10x more metadata in future, the size can grow from percents of binary to tens of percents.

[dvyukov]

take args alignment into account

[melver]

In D136078#3867627, @dvyukov wrote:

Unrelated, but looking at the metadata format more closely, I think this can benefit from LEB128-like varlen encoding.
Function size is small.
Features are very small.
Stack args size is small.
Function start (if we encode it from the end of the previous one) is very small as well.

Function start is currently encoded as a relative relocation (see AsmPrinter). Anything else, including attempting to use a previous function does not work because of section merging, potential discarding of functions by linker (ffunction-sections), etc.

We could implement compression in the linker, which would be the most reliable. Otherwise we can only play with encoding of the aux constants.

So potentially it can reduce the entry size from 16 bytes to just 4.
I am just thinking if we use 10x more metadata in future, the size can grow from percents of binary to tens of percents.

If that's the case, I think the only option is to implement compression in the linker.

llvm/include/llvm/Transforms/Instrumentation.h

153

inline constexpr

But overall, I'd prefer if these wouldn't need to be defined here, but instead in SanitizerBinaryMetadata.h.

Also, re naming "SanitizerMetadata" is already overloaded and we need to stick to SanitizerBinaryMetadata :-/

llvm/lib/CodeGen/SanitizerMetadata.cpp

1 ↗

(On Diff #468876)

SanitizerMetadata is already overloaded for other purposes (see clang/lib/CodeGen/SanitizerMetadata.cpp).

(Which is why I had to choose SanitizerBinaryMetadata)

47 ↗

(On Diff #468876)

I think you also need to check the target section name matches (in operand 0). Because having !pcsections does not imply SanitizerMetadata enabled.

Perhaps at this point it might make sense to move MetadataInfo to SanitizerBinaryMetadata.h (as SanitizerBinaryMetadataInfo perhaps ... long I know :-/ .. unfortunately "SanitizerMetadata" is already overloaded elsewhere -- SanitizerBinaryMDInfo?).

49 ↗

(On Diff #468876)

APInt

llvm/lib/IR/MDBuilder.cpp

176 ↗

(On Diff #468876)

This is not great. We still want MDNode interning to work for these.

Doesn't replaceOperandWith() work where you want to replace the operand?

[dvyukov]

rebase and adressed comments

[dvyukov]

PTAL

I've named everything consistently as "SanitizeBinaryMetadata",
fixed interning of metadata, and added a test.

[dvyukov]

llvm/lib/CodeGen/SanitizerBinaryMetadata.cpp

80

Should this be a new method on MDBuilder?

[melver]

llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h

38

inline constexpr char[]

llvm/lib/CodeGen/SanitizerBinaryMetadata.cpp

80

As discussed, this could just use the MDBuilder, and then extract the operand out of the new MDNode without using the full MDNode.

llvm/lib/Transforms/Instrumentation/SanitizerBinaryMetadata.cpp

244

I think this is no longer PerInstrFeatureMask, but should instead become FeatureMask.

247–250

Comment why also || Options.UAR (because PerInstrFeatureMask setting of UAR bit is deferred?).

257

What if Options.Covered==true?

Will it still emit some UAR metadata or will it emit something it shouldn't?

Should the F.isVarArg() check be done above in if (PerInstrFeatureMask || (Options.UAR && !F.isVarArg()) ? Then you wouldn't need the PerInstrFeatureMask && RequiresCovered change below and it could still just check RequiresCovered as before.

[melver]

llvm/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp

1 ↗

(On Diff #478168)

For LLVM IR tests, consider if utils/update_test_checks.py and for MIR utils/update_mir_test_checks.py could help.

Ideally, there'd be no .cpp runtime test, but I also think writing the .ll and .mir tests manually is impossible.

[dvyukov]

addressed comments

[dvyukov]

PTAL

llvm/lib/Transforms/Instrumentation/SanitizerBinaryMetadata.cpp

257

What if Options.Covered==true?
Will it still emit some UAR metadata or will it emit something it shouldn't?

If Options.Covered==true && F.isVarArg(), we emit covered metadata with features=0. This looks WAI.

Should the F.isVarArg() check be done above in if (PerInstrFeatureMask || (Options.UAR && !F.isVarArg()) ? Then you wouldn't need the PerInstrFeatureMask && RequiresCovered change below and it could still just check RequiresCovered as before.

It's tricky.
I forgot why I structured the code this way, but I added a new test for all permutations of covered/atomics/uar and it shows the following breakage with your proposed change:

- CHECK-AU: ellipsis: features=1 stack_args=0
+ CHECK-AU: ellipsis: features=3 stack_args=0

llvm/test/Instrumentation/SanitizerBinaryMetadata/covered.cpp

1 ↗

(On Diff #478569)

This is the new test.

[melver]

LGTM

llvm/include/llvm/CodeGen/MachinePassRegistry.def

205

I called the LLVM IR pass just "sanmd-module".
So this could just be "machine-sanmd".

llvm/test/Instrumentation/SanitizerBinaryMetadata/common.h

54 ↗

(On Diff #478569)

Could make this have its real signature, i.e. add version,start,end args. Then could also check in del that start,end matches, like above.

llvm/test/Instrumentation/SanitizerBinaryMetadata/covered.cpp

1 ↗

(On Diff #478569)

Nice!

[melver]

llvm/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp

19 ↗

(On Diff #478569)

Is this comment out of place?

[dvyukov]

addressed comments

[dvyukov]

llvm/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp

19 ↗

(On Diff #478569)

If a function is completely empty, it won't be marked with UAR feature metadata.
This is super-primitive approximation of the actual analysis we should do (the function has addr-taken/escaped locals/arguments).
So I needed at least somebody for the function.

[melver]

llvm/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp

19 ↗

(On Diff #478569)

Ack

[melver]

llvm/lib/CodeGen/SanitizerBinaryMetadata.cpp

58

Probably should be getNumOperands()?

https://reviews.llvm.org/rGdbb11309665fdf7d411d25cc8f2e6a0c8f658143

-DLLVM_ENABLE_ASSERTIONS=On will catch it.

[dvyukov]

fixed the debug build

[dvyukov]

Kazu, thanks for taking care of the revert.

I've fixed the assert and tested with -DLLVM_ENABLE_ASSERTIONS=On.

[dvyukov]

FTR ./llvm/test/CodeGen/AMDGPU/llc-pipeline.ll also failed as it hardcodes all passes:
https://lab.llvm.org/buildbot/#/builders/123/builds/14397

[dvyukov]

In D136078#3959643, @dvyukov wrote:

FTR ./llvm/test/CodeGen/AMDGPU/llc-pipeline.ll also failed as it hardcodes all passes:
https://lab.llvm.org/buildbot/#/builders/123/builds/14397

Also these:

LLVM :: CodeGen/X86/O0-pipeline.ll
LLVM :: CodeGen/X86/opt-pipeline.ll

need to grep for something.

[dvyukov]

moved *.cpp tests to clang/test/
updated tests that hardcode pipeline passes
fixed metadata update

[dvyukov]

PTAL

moved *.cpp tests to clang/test/
updated tests that hardcode pipeline passes
fixed metadata update

This did not work, it also changed all shared instances:

const auto *NewMD = MDB.createPCSections(
    {{Section.getString(), {Features, IRB.getInt32(Size)}}});
MD->replaceOperandWith(1, NewMD->getOperand(1));

so I changed it to:

F.setMetadata(LLVMContext::MD_pcsections,
              MDB.createPCSections(
                  {{Section.getString(), {Features, IRB.getInt32(Size)}}}));

[melver]

AFAIK Windows bot will break: https://buildkite.com/llvm-project/premerge-checks/builds/124003#0184c872-af5b-47c0-938c-b31ee8808241

Need to restrict test to Linux only.

[dvyukov]

restrict tests to linux

[mysterymath]

Hi, it looks like this change is causing Fuchsia's linux-x64 clang builders to fail: https://luci-milo.appspot.com/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8796062278266465473/overview

Would you mind taking a look and fixing forward, or that ends up being difficult, reverting until this can be fixed?

[aeubanks]

clang/test/Instrumentation/SanitizerBinaryMetadata/common.h

2–4

tests in clang/test cannot assume that standard headers are available, the tests should be hermetic. could you revert and reland with a proper test fix?

[wolfgangp]

covered.cpp and uar.cpp seem to fail on ubuntu bots here.

[dvyukov]

@melver Re this failure:

******************** TEST 'Clang :: Instrumentation/SanitizerBinaryMetadata/uar.cpp' FAILED ********************
Script:
--
: 'RUN: at line 4';   /home/buildbot/as-worker-91/clang-with-lto-ubuntu/build/stage1/bin/clang --driver-mode=g++ /home/buildbot/as-worker-91/clang-with-lto-ubuntu/llvm-project/clang/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp -o /home/buildbot/as-worker-91/clang-with-lto-ubuntu/build/stage1/tools/clang/test/Instrumentation/SanitizerBinaryMetadata/Output/uar.cpp.tmp -fexperimental-sanitize-metadata=covered,uar && /home/buildbot/as-worker-91/clang-with-lto-ubuntu/build/stage1/tools/clang/test/Instrumentation/SanitizerBinaryMetadata/Output/uar.cpp.tmp | /home/buildbot/as-worker-91/clang-with-lto-ubuntu/build/stage1/bin/FileCheck /home/buildbot/as-worker-91/clang-with-lto-ubuntu/llvm-project/clang/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp
--
Exit Code: 1
Command Output (stderr):
--
/usr/local/bin/ld: sanmd_covered has both ordered [`sanmd_covered[_Z7consumeIjET_RPKcS2_]' in /tmp/lit-tmp-2jisr97l/uar-d5d8d4.o] and unordered [`sanmd_covered[__dummy_sanmd_covered]' in /tmp/lit-tmp-2jisr97l/uar-d5d8d4.o] sections
/usr/local/bin/ld: final link failed: Bad value
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)

https://lab.llvm.org/buildbot/#/builders/124/builds/5759/steps/7/logs/stdio

This looks like a latent issue which is just exposed by the test.

As far as I understand this happens because this dummy variable somehow ends up in an "unordered" section, while other "real" metadata objects end up in "ordered" sections:

void SanitizerBinaryMetadata::createZeroSizedObjectInSection(
    Type *Ty, StringRef SectionSuffix) {
  auto *DummyInit = ConstantAggregateZero::get(ArrayType::get(Ty, 0));
  auto *DummyEntry = new GlobalVariable(Mod, DummyInit->getType(), true,
                                        GlobalVariable::ExternalLinkage,
                                        DummyInit, "__dummy_" + SectionSuffix);
  DummyEntry->setSection(getSectionName(SectionSuffix));
  DummyEntry->setVisibility(GlobalValue::HiddenVisibility);
  if (TargetTriple.supportsCOMDAT())
    DummyEntry->setComdat(Mod.getOrInsertComdat(DummyEntry->getName()));
  // Make sure the section isn't discarded by gc-sections.
  appendToUsed(Mod, DummyEntry);
}

I don't see any method on GlobalVariable to set the section to "ordered".

The only idea I have is to print dummy object in AsmPrinter the same way we print all other metadata objects (then I assume it will have the same properties as other objects).

Do you have any other ideas on how to dead with this?

[dvyukov]

Re this failure:

project/clang/test/Instrumentation/SanitizerBinaryMetadata/covered.cpp
--
Exit Code: 2

Command Output (stderr):
--
covered.cpp.tmp: /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Instrumentation/SanitizerBinaryMetadata/common.h:22: T consume(const char *&, const char *) [T = unsigned int]: Assertion `pos <= end' failed.
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /b/s/w/ir/x/w/staging/llvm_build/bin/FileCheck -check-prefix=CHECK-A /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Instrumentation/SanitizerBinaryMetadata/covered.cpp

https://logs.chromium.org/logs/fuchsia/buildbucket/cr-buildbucket/8796062278266465473/+/u/clang/test/stdout

Not sure why it run on fuchsia at all and the strange failure mode.
But I hope this will go away when we move tests to compiler-rt.

[melver]

In D136078#3964100, @dvyukov wrote:

@melver Re this failure:

******************** TEST 'Clang :: Instrumentation/SanitizerBinaryMetadata/uar.cpp' FAILED ********************
Script:
--
: 'RUN: at line 4';   /home/buildbot/as-worker-91/clang-with-lto-ubuntu/build/stage1/bin/clang --driver-mode=g++ /home/buildbot/as-worker-91/clang-with-lto-ubuntu/llvm-project/clang/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp -o /home/buildbot/as-worker-91/clang-with-lto-ubuntu/build/stage1/tools/clang/test/Instrumentation/SanitizerBinaryMetadata/Output/uar.cpp.tmp -fexperimental-sanitize-metadata=covered,uar && /home/buildbot/as-worker-91/clang-with-lto-ubuntu/build/stage1/tools/clang/test/Instrumentation/SanitizerBinaryMetadata/Output/uar.cpp.tmp | /home/buildbot/as-worker-91/clang-with-lto-ubuntu/build/stage1/bin/FileCheck /home/buildbot/as-worker-91/clang-with-lto-ubuntu/llvm-project/clang/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp
--
Exit Code: 1
Command Output (stderr):
--
/usr/local/bin/ld: sanmd_covered has both ordered [`sanmd_covered[_Z7consumeIjET_RPKcS2_]' in /tmp/lit-tmp-2jisr97l/uar-d5d8d4.o] and unordered [`sanmd_covered[__dummy_sanmd_covered]' in /tmp/lit-tmp-2jisr97l/uar-d5d8d4.o] sections
/usr/local/bin/ld: final link failed: Bad value
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)

https://lab.llvm.org/buildbot/#/builders/124/builds/5759/steps/7/logs/stdio

This looks like a latent issue which is just exposed by the test.

As far as I understand this happens because this dummy variable somehow ends up in an "unordered" section, while other "real" metadata objects end up in "ordered" sections:

void SanitizerBinaryMetadata::createZeroSizedObjectInSection(
    Type *Ty, StringRef SectionSuffix) {
  auto *DummyInit = ConstantAggregateZero::get(ArrayType::get(Ty, 0));
  auto *DummyEntry = new GlobalVariable(Mod, DummyInit->getType(), true,
                                        GlobalVariable::ExternalLinkage,
                                        DummyInit, "__dummy_" + SectionSuffix);
  DummyEntry->setSection(getSectionName(SectionSuffix));
  DummyEntry->setVisibility(GlobalValue::HiddenVisibility);
  if (TargetTriple.supportsCOMDAT())
    DummyEntry->setComdat(Mod.getOrInsertComdat(DummyEntry->getName()));
  // Make sure the section isn't discarded by gc-sections.
  appendToUsed(Mod, DummyEntry);
}

I don't see any method on GlobalVariable to set the section to "ordered".

The only idea I have is to print dummy object in AsmPrinter the same way we print all other metadata objects (then I assume it will have the same properties as other objects).

Do you have any other ideas on how to dead with this?

It's strange we've not encountered this elsewhere - this must be some special (old?) linker. I'll investigate...

[melver]

Do you have any other ideas on how to dead with this?

It's strange we've not encountered this elsewhere - this must be some special (old?) linker. I'll investigate...

GNU ld <= 2.35 does not support mixed SHF_LINK_ORDER and non-SHF_LINK_ORDER sections.

This is about the relative order of the section w.r.t. other sections it references (we link it to the function section, see MCObjectFileInfo::getPCSection) when the linker merges the sections. From what I can find, we need to retain SHF_LINK_ORDER to support gc-sections properly.

This is a good summary: https://maskray.me/blog/2021-01-31-metadata-sections-comdat-and-shf-link-order
@MaskRay

I'm going to try and see how I can add the SHF_LINK_ORDER attribute to the section output for the dummy variable.

[melver]

In D136078#3966187, @melver wrote:

Do you have any other ideas on how to dead with this?

It's strange we've not encountered this elsewhere - this must be some special (old?) linker. I'll investigate...

GNU ld <= 2.35 does not support mixed SHF_LINK_ORDER and non-SHF_LINK_ORDER sections.

This is about the relative order of the section w.r.t. other sections it references (we link it to the function section, see MCObjectFileInfo::getPCSection) when the linker merges the sections. From what I can find, we need to retain SHF_LINK_ORDER to support gc-sections properly.

This is a good summary: https://maskray.me/blog/2021-01-31-metadata-sections-comdat-and-shf-link-order
@MaskRay

I'm going to try and see how I can add the SHF_LINK_ORDER attribute to the section output for the dummy variable.

This should fix it: https://reviews.llvm.org/D139224 - at least whatever I was able to reproduce with the linker version I found (2.31) seems to work.

[MaskRay]

SanitizerBinaryMetadata::createZeroSizedObjectInSection creates __dummy_*, but why is it needed? (There is no comment.)

[melver]

In D136078#3969061, @MaskRay wrote:

SanitizerBinaryMetadata::createZeroSizedObjectInSection creates __dummy_*, but why is it needed? (There is no comment.)

// Create a 0-sized object in a section, so that the section is not discarded
// if all inputs have been discarded.

We need the __begin and __end markers even if there ends up being no binary metadata.

[MaskRay]

In D136078#3969091, @melver wrote:

In D136078#3969061, @MaskRay wrote:

SanitizerBinaryMetadata::createZeroSizedObjectInSection creates __dummy_*, but why is it needed? (There is no comment.)

// Create a 0-sized object in a section, so that the section is not discarded
// if all inputs have been discarded.

We need the __begin and __end markers even if there ends up being no binary metadata.

Thanks! I saw the comment associated with createZeroSizedObjectInSection but wasn't so sure if it was only for section based garbage collection.
If that's the case, D139276 ExternalWeakLinkage would be a more size efficient fix.

[dvyukov]

Moved tests to compiler-rt and rebased to HEAD.

[dvyukov]

PTAL

[thakis]

This seems to break tests: http://45.33.8.238/linux/93224/step_12.txt

Can you take a look?

[dvyukov]

In D136078#3970792, @thakis wrote:

This seems to break tests: http://45.33.8.238/linux/93224/step_12.txt

Can you take a look?

Looking.

[dvyukov]

Fix for:

-- Supported architectures for crt: aarch64
CMake Error at compiler-rt/cmake/config-ix.cmake:244 (message):
  Unsupported architecture: x86_64
Call Stack (most recent call first):
  compiler-rt/cmake/config-ix.cmake:280 (get_target_flags_for_arch)
  compiler-rt/test/metadata/CMakeLists.txt:7 (get_test_cc_for_arch)
-- Configuring incomplete, errors occurred!

https://reviews.llvm.org/D139323

[dvyukov]

In D136078#3970792, @thakis wrote:

This seems to break tests: http://45.33.8.238/linux/93224/step_12.txt

Can you take a look?

Sent a fix: https://reviews.llvm.org/D139325

[dvyukov]

FTR 2 breakage fixes merged:
https://github.com/llvm/llvm-project/commit/2a05bd212e3e8aaed818ee23464f4d1fe0b0596d
https://github.com/llvm/llvm-project/commit/08742e72a34e835e6fc3c696eabe6045c78d6289

clang/include/clang/Basic/CodeGenOptions.h

#include "clang/Basic/CodeGenOptions.def"
   bool hasSanitizeCoverage() const {
     return SanitizeCoverageType || SanitizeCoverageIndirectCalls ||
            SanitizeCoverageTraceCmp || SanitizeCoverageTraceLoads ||
            SanitizeCoverageTraceStores || SanitizeCoverageControlFlow;
   }
 
   // Check if any one of SanitizeBinaryMetadata* is enabled.
   bool hasSanitizeBinaryMetadata() const {
-    return SanitizeBinaryMetadataCovered || SanitizeBinaryMetadataAtomics;
+    return SanitizeBinaryMetadataCovered || SanitizeBinaryMetadataAtomics ||
+           SanitizeBinaryMetadataUAR;
   }
 };
 
 }  // end namespace clang
 
 #endif

clang/include/clang/Basic/CodeGenOptions.def

 CODEGENOPT(SanitizeCoveragePCTable, 1, 0) ///< Create a PC Table.
 CODEGENOPT(SanitizeCoverageControlFlow, 1, 0) ///< Collect control flow
 CODEGENOPT(SanitizeCoverageNoPrune, 1, 0) ///< Disable coverage pruning.
 CODEGENOPT(SanitizeCoverageStackDepth, 1, 0) ///< Enable max stack depth tracing
 CODEGENOPT(SanitizeCoverageTraceLoads, 1, 0) ///< Enable tracing of loads.
 CODEGENOPT(SanitizeCoverageTraceStores, 1, 0) ///< Enable tracing of stores.
 CODEGENOPT(SanitizeBinaryMetadataCovered, 1, 0) ///< Emit PCs for covered functions.
 CODEGENOPT(SanitizeBinaryMetadataAtomics, 1, 0) ///< Emit PCs for atomic operations.
+CODEGENOPT(SanitizeBinaryMetadataUAR, 1, 0) ///< Emit PCs for start of functions
+                                            ///< that are subject for use-after-return checking.
 CODEGENOPT(SanitizeStats     , 1, 0) ///< Collect statistics for sanitizers.
 CODEGENOPT(SimplifyLibCalls  , 1, 1) ///< Set when -fbuiltin is enabled.
 CODEGENOPT(SoftFloat         , 1, 0) ///< -soft-float.
 CODEGENOPT(SpeculativeLoadHardening, 1, 0) ///< Enable speculative load hardening.
 CODEGENOPT(FineGrainedBitfieldAccesses, 1, 0) ///< Enable fine-grained bitfield accesses.
 CODEGENOPT(StrictEnums       , 1, 0) ///< Optimize based on strict enum definition.
 CODEGENOPT(StrictVTablePointers, 1, 0) ///< Optimize based on the strict vtable pointers
 CODEGENOPT(TimePasses        , 1, 0) ///< Set when -ftime-report or -ftime-report= is enabled.

clang/include/clang/Driver/Options.td

 def fexperimental_sanitize_metadata_EQ_covered
     : Flag<["-"], "fexperimental-sanitize-metadata=covered">,
       HelpText<"Emit PCs for code covered with binary analysis sanitizers">,
       MarshallingInfoFlag<CodeGenOpts<"SanitizeBinaryMetadataCovered">>;
 def fexperimental_sanitize_metadata_EQ_atomics
     : Flag<["-"], "fexperimental-sanitize-metadata=atomics">,
       HelpText<"Emit PCs for atomic operations used by binary analysis sanitizers">,
       MarshallingInfoFlag<CodeGenOpts<"SanitizeBinaryMetadataAtomics">>;
+def fexperimental_sanitize_metadata_EQ_uar
+    : Flag<["-"], "fexperimental-sanitize-metadata=uar">,
+      HelpText<"Emit PCs for start of functions that are subject for use-after-return checking.">,
+      MarshallingInfoFlag<CodeGenOpts<"SanitizeBinaryMetadataUAR">>;
 def fpatchable_function_entry_offset_EQ
     : Joined<["-"], "fpatchable-function-entry-offset=">, MetaVarName<"<M>">,
       HelpText<"Generate M NOPs before function entry">,
       MarshallingInfoInt<CodeGenOpts<"PatchableFunctionEntryOffset">>;
 def fprofile_instrument_EQ : Joined<["-"], "fprofile-instrument=">,
     HelpText<"Enable PGO instrumentation">, Values<"none,clang,llvm,csllvm">,
     NormalizedValuesScope<"CodeGenOptions">,
     NormalizedValues<["ProfileNone", "ProfileClangInstr", "ProfileIRInstr", "ProfileCSIRInstr"]>,

clang/lib/CodeGen/BackendUtil.cpp

getSancovOptsFromCGOpts(const CodeGenOptions &CGOpts) {
   return Opts;
 }
 
 static SanitizerBinaryMetadataOptions
 getSanitizerBinaryMetadataOptions(const CodeGenOptions &CGOpts) {
   SanitizerBinaryMetadataOptions Opts;
   Opts.Covered = CGOpts.SanitizeBinaryMetadataCovered;
   Opts.Atomics = CGOpts.SanitizeBinaryMetadataAtomics;
+  Opts.UAR = CGOpts.SanitizeBinaryMetadataUAR;
   return Opts;
 }
 
 // Check if ASan should use GC-friendly instrumentation for globals.
 // First of all, there is no point if -fdata-sections is off (expect for MachO,
 // where this is not a factor). Also, on ELF this feature requires an assembler
 // extension that only works with -integrated-as at the moment.
 static bool asanUseGlobalsGC(const Triple &T, const CodeGenOptions &CGOpts) {

clang/lib/Driver/SanitizerArgs.cpp

enum CoverageFeature {
   CoverageTraceLoads = 1 << 16,
   CoverageTraceStores = 1 << 17,
   CoverageControlFlow = 1 << 18,
 };
 
 enum BinaryMetadataFeature {
   BinaryMetadataCovered = 1 << 0,
   BinaryMetadataAtomics = 1 << 1,
+  BinaryMetadataUAR = 1 << 2,
 };
 
 /// Parse a -fsanitize= or -fno-sanitize= argument's values, diagnosing any
 /// invalid components. Returns a SanitizerMask.
 static SanitizerMask parseArgValues(const Driver &D, const llvm::opt::Arg *A,
                                     bool DiagnoseErrors);
 
 /// Parse -f(no-)?sanitize-coverage= flag values, diagnosing any invalid
addSpecialCaseListOpt(
       Args, CmdArgs, "-fsanitize-coverage-allowlist=", CoverageAllowlistFiles);
   addSpecialCaseListOpt(Args, CmdArgs, "-fsanitize-coverage-ignorelist=",
                         CoverageIgnorelistFiles);
 
   // Translate available BinaryMetadataFeatures to corresponding clang-cc1
   // flags. Does not depend on any other sanitizers.
   const std::pair<int, std::string> BinaryMetadataFlags[] = {
       std::make_pair(BinaryMetadataCovered, "covered"),
-      std::make_pair(BinaryMetadataAtomics, "atomics")};
+      std::make_pair(BinaryMetadataAtomics, "atomics"),
+      std::make_pair(BinaryMetadataUAR, "uar")};
   for (const auto &F : BinaryMetadataFlags) {
     if (BinaryMetadataFeatures & F.first)
       CmdArgs.push_back(
           Args.MakeArgString("-fexperimental-sanitize-metadata=" + F.second));
   }
 
   if (TC.getTriple().isOSWindows() && needsUbsanRt()) {
     // Instruct the code generator to embed linker directives in the object file
assert(
       A->getOption().matches(
           options::OPT_fno_experimental_sanitize_metadata_EQ));
   int Features = 0;
   for (int i = 0, n = A->getNumValues(); i != n; ++i) {
     const char *Value = A->getValue(i);
     int F = llvm::StringSwitch<int>(Value)
                 .Case("covered", BinaryMetadataCovered)
                 .Case("atomics", BinaryMetadataAtomics)
+                .Case("uar", BinaryMetadataUAR)
                 .Case("all", ~0)
                 .Default(0);
     if (F == 0 && DiagnoseErrors)
       D.Diag(clang::diag::err_drv_unsupported_option_argument)
           << A->getSpelling() << Value;
     Features |= F;
   }
   return Features;

clang/test/Instrumentation/SanitizerBinaryMetadata/common.h

+#include <stdint.h>
+#include <stdio.h>
+#include <assert.h>
+
+int main() {
+  printf("main\n");
+}
+
+typedef unsigned long uptr;
+
+#define FN(X) if (pc == reinterpret_cast<uptr>(X)) return #X
+
+const char* symbolize(uptr pc) {
+  FUNCTIONS;
+  return nullptr;
+}
+
+template<typename T>
+T consume(const char*& pos, const char* end) {
+  T v = *reinterpret_cast<const T*>(pos);
+  pos += sizeof(T);
+  assert(pos <= end);
+  return v;
+}
+
+uint32_t meta_version;
+const char* meta_start;
+const char* meta_end;
+
+extern "C" {
+void __sanitizer_metadata_covered_add(uint32_t version, const char* start, const char* end) {
+  printf("metadata add version %u\n", version);
+  for (const char* pos = start; pos < end;) {
+    const uptr base = reinterpret_cast<uptr>(pos);
+    const long offset = (version & (1 << 16)) ?
+        consume<long>(pos, end) : consume<int>(pos, end);
+    const uint32_t size = consume<uint32_t>(pos, end);
+    const uint32_t features = consume<uint32_t>(pos, end);
+    uint32_t stack_args = 0;
+    if (features & (1 << 1))
+      stack_args = consume<uint32_t>(pos, end);
+    if (const char* name = symbolize(base + offset))
+      printf("%s: features=%x stack_args=%u\n", name, features, stack_args);
+  }
+  meta_version = version;
+  meta_start = start;
+  meta_end = end;
+}
+
+void __sanitizer_metadata_covered_del(uint32_t version, const char* start, const char* end) {
+  assert(version == meta_version);
+  assert(start == meta_start);
+  assert(end == meta_end);
+}
+
+const char* atomics_start;
+const char* atomics_end;
+
+void __sanitizer_metadata_atomics_add(uint32_t version, const char* start, const char* end) {
+  assert(version == meta_version);
+  assert(start);
+  assert(end >= end);
+  atomics_start = start;
+  atomics_end = end;
+}
+
+void __sanitizer_metadata_atomics_del(uint32_t version, const char* start, const char* end) {
+  assert(version == meta_version);
+  assert(atomics_start == start);
+  assert(atomics_end == end);
+}
+}

clang/test/Instrumentation/SanitizerBinaryMetadata/covered.cpp

+// REQUIRES: native && x86_64-linux
+// RUN: %clangxx %s -o %t && %t | FileCheck %s
+// RUN: %clangxx %s -o %t -fexperimental-sanitize-metadata=covered && %t | FileCheck -check-prefix=CHECK-C %s
+// RUN: %clangxx %s -o %t -fexperimental-sanitize-metadata=atomics && %t | FileCheck -check-prefix=CHECK-A %s
+// RUN: %clangxx %s -o %t -fexperimental-sanitize-metadata=uar && %t | FileCheck -check-prefix=CHECK-U %s
+// RUN: %clangxx %s -o %t -fexperimental-sanitize-metadata=covered,atomics && %t | FileCheck -check-prefix=CHECK-CA %s
+// RUN: %clangxx %s -o %t -fexperimental-sanitize-metadata=covered,uar && %t | FileCheck -check-prefix=CHECK-CU %s
+// RUN: %clangxx %s -o %t -fexperimental-sanitize-metadata=atomics,uar && %t | FileCheck -check-prefix=CHECK-AU %s
+// RUN: %clangxx %s -o %t -fexperimental-sanitize-metadata=covered,atomics,uar && %t | FileCheck -check-prefix=CHECK-CAU %s
+
+// CHECK-NOT: metadata add
+// CHECK: main
+// CHECK-NOT: metadata del
+
+// CHECK-C:      empty: features=0
+// CHECK-A-NOT:  empty:
+// CHECK-U-NOT:  empty:
+// CHECK-CA:     empty: features=1
+// CHECK-CU:     empty: features=0
+// CHECK-AU-NOT: empty:
+// CHECK-CAU:    empty: features=1
+void empty() {
+}
+
+// CHECK-C:  normal: features=0
+// CHECK-A:  normal: features=1
+// CHECK-U:  normal: features=2
+// CHECK-CA: normal: features=1
+// CHECK-CU: normal: features=2
+// CHECK-AU: normal: features=3
+// CHECK-CAU:normal: features=3
+void normal() {
+  volatile int x;
+  x = 0;
+}
+
+// CHECK-C:   with_atomic: features=0
+// CHECK-A:   with_atomic: features=1
+// CHECK-U:   with_atomic: features=2
+// CHECK-CA:  with_atomic: features=1
+// CHECK-CU:  with_atomic: features=2
+// CHECK-AU:  with_atomic: features=3
+// CHECK-CAU: with_atomic: features=3
+int with_atomic(int* p) {
+  return __atomic_load_n(p, __ATOMIC_RELAXED);
+}
+
+// CHECK-C:     ellipsis: features=0
+// CHECK-A:     ellipsis: features=1
+// CHECK-U-NOT: ellipsis:
+// CHECK-CA:    ellipsis: features=1
+// CHECK-CU:    ellipsis: features=0
+// CHECK-AU:    ellipsis: features=1
+// CHECK-CAU:   ellipsis: features=1
+void ellipsis(int* p, ...) {
+  volatile int x;
+  x = 0;
+}
+
+// CHECK-C:     ellipsis_with_atomic: features=0
+// CHECK-A:     ellipsis_with_atomic: features=1
+// CHECK-U-NOT: ellipsis_with_atomic:
+// CHECK-CA:    ellipsis_with_atomic: features=1
+// CHECK-CU:    ellipsis_with_atomic: features=0
+// CHECK-AU:    ellipsis_with_atomic: features=1
+// CHECK-CAU:   ellipsis_with_atomic: features=1
+int ellipsis_with_atomic(int* p, ...) {
+  return __atomic_load_n(p, __ATOMIC_RELAXED);
+}
+
+#define FUNCTIONS \
+  FN(empty); \
+  FN(normal); \
+  FN(with_atomic); \
+  FN(ellipsis); \
+  FN(ellipsis_with_atomic); \
+/**/
+
+#include "common.h"

clang/test/Instrumentation/SanitizerBinaryMetadata/uar.cpp

+// We run the compiled binary + sizes of stack arguments depend on the arch
+// + binary metadata is currently emited only for ELF binaries.
+// REQUIRES: native && x86_64-linux
+// RUN: %clangxx %s -o %t -fexperimental-sanitize-metadata=covered,uar && %t | FileCheck %s
+
+// CHECK: metadata add version 1
+
+// CHECK: empty: features=0 stack_args=0
+void empty() {
+}
+
+// CHECK: ellipsis: features=0 stack_args=0
+void ellipsis(const char* fmt, ...) {
+  volatile int x;
+  x = 1;
+}
+
+// CHECK: non_empty_function: features=2 stack_args=0
+void non_empty_function() {
+  // Completely empty functions don't get uar metadata.
+  volatile int x;
+  x = 1;
+}
+
+// CHECK: no_stack_args: features=2 stack_args=0
+void no_stack_args(long a0, long a1, long a2, long a3, long a4, long a5) {
+  volatile int x;
+  x = 1;
+}
+
+// CHECK: stack_args: features=2 stack_args=16
+void stack_args(long a0, long a1, long a2, long a3, long a4, long a5, long a6) {
+  volatile int x;
+  x = 1;
+}
+
+// CHECK: more_stack_args: features=2 stack_args=32
+void more_stack_args(long a0, long a1, long a2, long a3, long a4, long a5, long a6, long a7, long a8) {
+  volatile int x;
+  x = 1;
+}
+
+// CHECK: struct_stack_args: features=2 stack_args=144
+struct large { char x[131]; };
+void struct_stack_args(large a) {
+  volatile int x;
+  x = 1;
+}
+
+#define FUNCTIONS \
+  FN(empty); \
+  FN(ellipsis); \
+  FN(non_empty_function); \
+  FN(no_stack_args); \
+  FN(stack_args); \
+  FN(more_stack_args); \
+  FN(struct_stack_args); \
+/**/
+
+#include "common.h"

llvm/include/llvm/CodeGen/CodeGenPassBuilder.h

if (TM.Options.EnableIPRA)
     // Collect register usage information and produce a register mask of
     // clobbered registers, to be used to optimize call sites.
     addPass(RegUsageInfoCollectorPass());
 
   addPass(FuncletLayoutPass());
 
   addPass(StackMapLivenessPass());
   addPass(LiveDebugValuesPass());
+  addPass(MachineSanitizerBinaryMetadata());
 
   if (TM.Options.EnableMachineOutliner && getOptLevel() != CodeGenOpt::None &&
       Opt.EnableMachineOutliner != RunOutliner::NeverOutline) {
     bool RunOnAllFunctions =
         (Opt.EnableMachineOutliner == RunOutliner::AlwaysOutline);
     bool AddOutliner = RunOnAllFunctions || TM.Options.SupportsDefaultOutlining;
     if (AddOutliner)
       addPass(MachineOutlinerPass(RunOnAllFunctions));

llvm/include/llvm/CodeGen/MachinePassRegistry.def

 DUMMY_MACHINE_FUNCTION_PASS("ra-pbqp", RAPBQPPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("legalizer", LegalizerPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("irtranslator", IRTranslatorPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("regbankselect", RegBankSelectPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("instruction-select", InstructionSelectPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("reset-machine-function", ResetMachineFunctionPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("machineverifier", MachineVerifierPass, ())
 DUMMY_MACHINE_FUNCTION_PASS("print-machine-cycles", MachineCycleInfoPrinterPass, ())
+DUMMY_MACHINE_FUNCTION_PASS("machine-sanmd", MachineSanitizerBinaryMetadata, ())
 #undef DUMMY_MACHINE_FUNCTION_PASS

llvm/include/llvm/CodeGen/Passes.h

namespace llvm {
   /// bundles (created earlier, e.g. during pre-RA scheduling).
   extern char &FinalizeMachineBundlesID;
 
   /// StackMapLiveness - This pass analyses the register live-out set of
   /// stackmap/patchpoint intrinsics and attaches the calculated information to
   /// the intrinsic for later emission to the StackMap.
   extern char &StackMapLivenessID;
 
+  // MachineSanitizerBinaryMetadata - appends/finalizes sanitizer binary
+  // metadata after llvm SanitizerBinaryMetadata pass.
+  extern char &MachineSanitizerBinaryMetadataID;
+
   /// RemoveRedundantDebugValues pass.
   extern char &RemoveRedundantDebugValuesID;
 
   /// MachineCFGPrinter pass.
   extern char &MachineCFGPrinterID;
 
   /// LiveDebugValues pass
   extern char &LiveDebugValuesID;

llvm/include/llvm/InitializePasses.h

 void initializeMachineLICMPass(PassRegistry&);
 void initializeMachineLoopInfoPass(PassRegistry&);
 void initializeMachineModuleInfoWrapperPassPass(PassRegistry &);
 void initializeMachineOptimizationRemarkEmitterPassPass(PassRegistry&);
 void initializeMachineOutlinerPass(PassRegistry&);
 void initializeMachinePipelinerPass(PassRegistry&);
 void initializeMachinePostDominatorTreePass(PassRegistry&);
 void initializeMachineRegionInfoPassPass(PassRegistry&);
+void initializeMachineSanitizerBinaryMetadataPass(PassRegistry &);
 void initializeMachineSchedulerPass(PassRegistry&);
 void initializeMachineSinkingPass(PassRegistry&);
 void initializeMachineTraceMetricsPass(PassRegistry&);
 void initializeMachineVerifierPassPass(PassRegistry&);
 void initializeMemCpyOptLegacyPassPass(PassRegistry&);
 void initializeMemDepPrinterPass(PassRegistry&);
 void initializeMemDerefPrinterPass(PassRegistry&);
 void initializeMemoryDependenceWrapperPassPass(PassRegistry&);

llvm/include/llvm/Transforms/Instrumentation.h

struct SanitizerCoverageOptions {
   bool StackDepth = false;
   bool TraceLoads = false;
   bool TraceStores = false;
   bool CollectControlFlow = false;
 
   SanitizerCoverageOptions() = default;
 };
 
-/// Options for SanitizerBinaryMetadata.
-struct SanitizerBinaryMetadataOptions {
-  bool Covered = false;
-  bool Atomics = false;
-  SanitizerBinaryMetadataOptions() = default;
-};
-
 /// Calculate what to divide by to scale counts.
 ///
 /// Given the maximum count, calculate a divisor that will scale all the
 /// weights to strictly less than std::numeric_limits<uint32_t>::max().
 static inline uint64_t calculateCountScale(uint64_t MaxCount) {
   return MaxCount < std::numeric_limits<uint32_t>::max()
              ? 1
              : MaxCount / std::numeric_limits<uint32_t>::max() + 1;
 }

llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h

 
 #include "llvm/IR/Function.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Transforms/Instrumentation.h"
 
 namespace llvm {
 
+struct SanitizerBinaryMetadataOptions {
+  bool Covered = false;
+  bool Atomics = false;
+  bool UAR = false;
+  SanitizerBinaryMetadataOptions() = default;
+};
+
+inline constexpr int kSanitizerBinaryMetadataAtomicsBit = 0;
+inline constexpr int kSanitizerBinaryMetadataUARBit = 1;
+
+inline constexpr uint32_t kSanitizerBinaryMetadataNone = 0;
+inline constexpr uint32_t kSanitizerBinaryMetadataAtomics =
+    1 << kSanitizerBinaryMetadataAtomicsBit;
+inline constexpr uint32_t kSanitizerBinaryMetadataUAR =
+    1 << kSanitizerBinaryMetadataUARBit;
+
+inline constexpr char kSanitizerBinaryMetadataCoveredSection[] =
+    "sanmd_covered";
+inline constexpr char kSanitizerBinaryMetadataAtomicsSection[] =
+    "sanmd_atomics";
+
 /// Public interface to the SanitizerBinaryMetadata module pass for emitting
 /// metadata for binary analysis sanitizers.
 //
 /// The pass should be inserted after optimizations.
 class SanitizerBinaryMetadataPass
     : public PassInfoMixin<SanitizerBinaryMetadataPass> {
 public:
   explicit SanitizerBinaryMetadataPass(

llvm/lib/CodeGen/CMakeLists.txt

add_llvm_component_library(LLVMCodeGen
   RegUsageInfoCollector.cpp
   RegUsageInfoPropagate.cpp
   ReplaceWithVeclib.cpp
   ResetMachineFunctionPass.cpp
   RegisterBank.cpp
   RegisterBankInfo.cpp
   SafeStack.cpp
   SafeStackLayout.cpp
+  SanitizerBinaryMetadata.cpp
   ScheduleDAG.cpp
   ScheduleDAGInstrs.cpp
   ScheduleDAGPrinter.cpp
   ScoreboardHazardRecognizer.cpp
   SelectOptimize.cpp
   ShadowStackGCLowering.cpp
   ShrinkWrap.cpp
   SjLjEHPrepare.cpp

llvm/lib/CodeGen/CodeGen.cpp

void llvm::initializeCodeGen(PassRegistry &Registry) {
   initializeMachineDominatorTreePass(Registry);
   initializeMachineFunctionPrinterPassPass(Registry);
   initializeMachineLICMPass(Registry);
   initializeMachineLoopInfoPass(Registry);
   initializeMachineModuleInfoWrapperPassPass(Registry);
   initializeMachineOptimizationRemarkEmitterPassPass(Registry);
   initializeMachineOutlinerPass(Registry);
   initializeMachinePipelinerPass(Registry);
+  initializeMachineSanitizerBinaryMetadataPass(Registry);
   initializeModuloScheduleTestPass(Registry);
   initializeMachinePostDominatorTreePass(Registry);
   initializeMachineRegionInfoPassPass(Registry);
   initializeMachineSchedulerPass(Registry);
   initializeMachineSinkingPass(Registry);
   initializeMachineVerifierPassPass(Registry);
   initializeObjCARCContractLegacyPassPass(Registry);
   initializeOptimizePHIsPass(Registry);

llvm/lib/CodeGen/SanitizerBinaryMetadata.cpp

+//===- SanitizerBinaryMetadata.cpp
+//----------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a part of SanitizerBinaryMetadata.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h"
+#include "llvm/CodeGen/MachineFrameInfo.h"
+#include "llvm/CodeGen/MachineFunction.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/MDBuilder.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include <algorithm>
+
+using namespace llvm;
+
+class MachineSanitizerBinaryMetadata : public MachineFunctionPass {
+public:
+  static char ID;
+
+  MachineSanitizerBinaryMetadata();
+  bool runOnMachineFunction(MachineFunction &F) override;
+};
+
+INITIALIZE_PASS(MachineSanitizerBinaryMetadata, "machine-sanmd",
+                "Machine Sanitizer Binary Metadata", false, false)
+
+char MachineSanitizerBinaryMetadata::ID = 0;
+char &llvm::MachineSanitizerBinaryMetadataID =
+    MachineSanitizerBinaryMetadata::ID;
+
+MachineSanitizerBinaryMetadata::MachineSanitizerBinaryMetadata()
+    : MachineFunctionPass(ID) {
+  initializeMachineSanitizerBinaryMetadataPass(
+      *PassRegistry::getPassRegistry());
+}
+
+bool MachineSanitizerBinaryMetadata::runOnMachineFunction(MachineFunction &MF) {
+  MDNode *MD = MF.getFunction().getMetadata(LLVMContext::MD_pcsections);
+  if (!MD)
+    return false;
+  const auto &Section = *cast<MDString>(MD->getOperand(0));
+  if (!Section.getString().equals(kSanitizerBinaryMetadataCoveredSection))
+    return false;
+  auto &AuxMDs = *cast<MDTuple>(MD->getOperand(1));
+  // Assume it currently only has features.
+  assert(AuxMDs.getNumOperands() == 1);
+  auto *Features = cast<ConstantAsMetadata>(AuxMDs.getOperand(0))->getValue();
+  if (!Features->getUniqueInteger()[kSanitizerBinaryMetadataUARBit])
+    return false;
+  // Calculate size of stack args for the function.
+  int64_t Size = 0;
+  uint64_t Align = 0;
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  for (int i = -1; i >= (int)-MFI.getNumFixedObjects(); --i) {
+    Size = std::max(Size, MFI.getObjectOffset(i) + MFI.getObjectSize(i));
+    Align = std::max(Align, MFI.getObjectAlign(i).value());
+  }
+  Size = (Size + Align - 1) & ~(Align - 1);
+  auto &F = MF.getFunction();
+  IRBuilder<> IRB(F.getContext());
+  MDBuilder MDB(F.getContext());
+  // Keep the features and append size of stack args to the metadata.
+  F.setMetadata(LLVMContext::MD_pcsections,
+                MDB.createPCSections(
+                    {{Section.getString(), {Features, IRB.getInt32(Size)}}}));
+  return false;
+}

llvm/lib/CodeGen/TargetPassConfig.cpp

if (TM->Options.EnableIPRA)
     addPass(createRegUsageInfoCollector());
 
   // FIXME: Some backends are incompatible with running the verifier after
   // addPreEmitPass.  Maybe only pass "false" here for those targets?
   addPass(&FuncletLayoutID);
 
   addPass(&StackMapLivenessID);
   addPass(&LiveDebugValuesID);
+  addPass(&MachineSanitizerBinaryMetadataID);
 
   if (TM->Options.EnableMachineOutliner && getOptLevel() != CodeGenOpt::None &&
       EnableMachineOutliner != RunOutliner::NeverOutline) {
     bool RunOnAllFunctions =
         (EnableMachineOutliner == RunOutliner::AlwaysOutline);
     bool AddOutliner =
         RunOnAllFunctions || TM->Options.SupportsDefaultOutlining;
     if (AddOutliner)

llvm/lib/Transforms/Instrumentation/SanitizerBinaryMetadata.cpp

public:
   const uint32_t FeatureMask;
 
   static const MetadataInfo Covered;
   static const MetadataInfo Atomics;
 
 private:
   // Forbid construction elsewhere.
   explicit constexpr MetadataInfo(StringRef FunctionPrefix,
-                                  StringRef SectionSuffix, int Feature)
+                                  StringRef SectionSuffix, uint32_t Feature)
       : FunctionPrefix(FunctionPrefix), SectionSuffix(SectionSuffix),
-        FeatureMask(Feature != -1 ? (1u << Feature) : 0) {}
+        FeatureMask(Feature) {}
 };
 const MetadataInfo MetadataInfo::Covered{"__sanitizer_metadata_covered",
-                                         "sanmd_covered", -1};
+                                         kSanitizerBinaryMetadataCoveredSection,
+                                         kSanitizerBinaryMetadataNone};
 const MetadataInfo MetadataInfo::Atomics{"__sanitizer_metadata_atomics",
-                                         "sanmd_atomics", 0};
+                                         kSanitizerBinaryMetadataAtomicsSection,
+                                         kSanitizerBinaryMetadataAtomics};
 
 // The only instances of MetadataInfo are the constants above, so a set of
 // them may simply store pointers to them. To deterministically generate code,
 // we need to use a set with stable iteration order, such as SetVector.
 using MetadataInfoSet = SetVector<const MetadataInfo *>;
 
 //===--- Command-line options ---------------------------------------------===//
 
 cl::opt<bool> ClEmitCovered("sanitizer-metadata-covered",
                             cl::desc("Emit PCs for covered functions."),
                             cl::Hidden, cl::init(false));
 cl::opt<bool> ClEmitAtomics("sanitizer-metadata-atomics",
                             cl::desc("Emit PCs for atomic operations."),
                             cl::Hidden, cl::init(false));
+cl::opt<bool> ClEmitUAR("sanitizer-metadata-uar",
+                        cl::desc("Emit PCs for start of functions that are "
+                                 "subject for use-after-return checking"),
+                        cl::Hidden, cl::init(false));
 
 //===--- Statistics -------------------------------------------------------===//
 
 STATISTIC(NumMetadataCovered, "Metadata attached to covered functions");
 STATISTIC(NumMetadataAtomics, "Metadata attached to atomics");
+STATISTIC(NumMetadataUAR, "Metadata attached to UAR functions");
 
 //===----------------------------------------------------------------------===//
 
 // Apply opt overrides.
 SanitizerBinaryMetadataOptions &&
 transformOptionsFromCl(SanitizerBinaryMetadataOptions &&Opts) {
   Opts.Covered |= ClEmitCovered;
   Opts.Atomics |= ClEmitAtomics;
+  Opts.UAR |= ClEmitUAR;
   return std::move(Opts);
 }
 
 class SanitizerBinaryMetadata {
 public:
   SanitizerBinaryMetadata(Module &M, SanitizerBinaryMetadataOptions Opts)
       : Mod(M), Options(transformOptionsFromCl(std::move(Opts))),
         TargetTriple(M.getTargetTriple()), IRB(M.getContext()) {
private:
 
   // Determines which set of metadata to collect for this instruction.
   //
   // Returns true if covered metadata is required to unambiguously interpret
   // other metadata. For example, if we are interested in atomics metadata, any
   // function with memory operations (atomic or not) requires covered metadata
   // to determine if a memory operation is atomic or not in modules compiled
   // with SanitizerBinaryMetadata.
-  bool runOn(Instruction &I, MetadataInfoSet &MIS, MDBuilder &MDB);
+  bool runOn(Instruction &I, MetadataInfoSet &MIS, MDBuilder &MDB,
+             uint32_t &FeatureMask);
 
   // Get start/end section marker pointer.
   GlobalVariable *getSectionMarker(const Twine &MarkerName, Type *Ty);
 
   // Create a 0-sized object in a section, so that the section is not discarded
   // if all inputs have been discarded.
   void createZeroSizedObjectInSection(Type *Ty, StringRef SectionSuffix);
 
void SanitizerBinaryMetadata::runOn(Function &F, MetadataInfoSet &MIS) {
   // Don't touch available_externally functions, their actual body is elsewhere.
   if (F.getLinkage() == GlobalValue::AvailableExternallyLinkage)
     return;
 
   MDBuilder MDB(F.getContext());
 
   // The metadata features enabled for this function, stored along covered
   // metadata (if enabled).
-  uint32_t PerInstrFeatureMask = getEnabledPerInstructionFeature();
+  uint32_t FeatureMask = getEnabledPerInstructionFeature();
   // Don't emit unnecessary covered metadata for all functions to save space.
   bool RequiresCovered = false;
-  if (PerInstrFeatureMask) {
+  // We can only understand if we need to set UAR feature after looking
+  // at the instructions. So we need to check instructions even if FeatureMask
+  // is empty.
+  if (FeatureMask || Options.UAR) {
     for (BasicBlock &BB : F)
       for (Instruction &I : BB)
-        RequiresCovered |= runOn(I, MIS, MDB);
+        RequiresCovered |= runOn(I, MIS, MDB, FeatureMask);
   }
 
+  if (F.isVarArg())
+    FeatureMask &= ~kSanitizerBinaryMetadataUAR;
+  if (FeatureMask & kSanitizerBinaryMetadataUAR)
+    NumMetadataUAR++;
+
   // Covered metadata is always emitted if explicitly requested, otherwise only
   // if some other metadata requires it to unambiguously interpret it for
   // modules compiled with SanitizerBinaryMetadata.
-  if (Options.Covered || RequiresCovered) {
+  if (Options.Covered || (FeatureMask && RequiresCovered)) {
     NumMetadataCovered++;
     const auto *MI = &MetadataInfo::Covered;
     MIS.insert(MI);
     const StringRef Section = getSectionName(MI->SectionSuffix);
     // The feature mask will be placed after the size (32 bit) of the function,
     // so in total one covered entry will use `sizeof(void*) + 4 + 4`.
-    Constant *CFM = IRB.getInt32(PerInstrFeatureMask);
+    Constant *CFM = IRB.getInt32(FeatureMask);
     F.setMetadata(LLVMContext::MD_pcsections,
                   MDB.createPCSections({{Section, {CFM}}}));
   }
 }
 
 bool SanitizerBinaryMetadata::runOn(Instruction &I, MetadataInfoSet &MIS,
-                                    MDBuilder &MDB) {
+                                    MDBuilder &MDB, uint32_t &FeatureMask) {
   SmallVector<const MetadataInfo *, 1> InstMetadata;
   bool RequiresCovered = false;
 
+  if (Options.UAR) {
+    for (unsigned i = 0; i < I.getNumOperands(); ++i) {
+      const Value *V = I.getOperand(i);
+      // TODO(dvyukov): check if V is an address of alloca/function arg.
+      // See isSafeAndProfitableToSinkLoad for addr-taken allocas
+      // and DeadArgumentEliminationPass::removeDeadStuffFromFunction
+      // for iteration over function args.
+      if (V) {
+        RequiresCovered = true;
+        FeatureMask |= kSanitizerBinaryMetadataUAR;
+      }
+    }
+  }
+
   if (Options.Atomics && I.mayReadOrWriteMemory()) {
     auto SSID = getAtomicSyncScopeID(&I);
     if (SSID.has_value() && SSID.value() != SyncScope::SingleThread) {
       NumMetadataAtomics++;
       InstMetadata.push_back(&MetadataInfo::Atomics);
     }
     RequiresCovered = true;
   }

llvm/test/CodeGen/AArch64/O0-pipeline.ll

 ; CHECK-NEXT:       Insert XRay ops
 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       Workaround A53 erratum 835769 pass
 ; CHECK-NEXT:       AArch64 Branch Targets
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
+; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       Insert CFI remember/restore state instructions
 ; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       AArch64 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 define void @f() {
   ret void
 }

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       Machine Copy Propagation Pass
 ; CHECK-NEXT:       Workaround A53 erratum 835769 pass
 ; CHECK-NEXT:       AArch64 Branch Targets
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       AArch64 Compress Jump Tables
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
+; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:     Machine Outliner
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Insert CFI remember/restore state instructions
 ; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       AArch64 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

 ; GCN-O0-NEXT:        MachinePostDominator Tree Construction
 ; GCN-O0-NEXT:        SI insert wait instructions
 ; GCN-O0-NEXT:        Insert required mode register values
 ; GCN-O0-NEXT:        SI Final Branch Preparation
 ; GCN-O0-NEXT:        Post RA hazard recognizer
 ; GCN-O0-NEXT:        Branch relaxation pass
 ; GCN-O0-NEXT:        Register Usage Information Collector Pass
 ; GCN-O0-NEXT:        Live DEBUG_VALUE analysis
+; GCN-O0-NEXT:        Machine Sanitizer Binary Metadata
 ; GCN-O0-NEXT:    Function register usage analysis
 ; GCN-O0-NEXT:    FunctionPass Manager
 ; GCN-O0-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O0-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O0-NEXT:      AMDGPU Assembly Printer
 ; GCN-O0-NEXT:      Free MachineFunction
 ; GCN-O0-NEXT:Pass Arguments:  -domtree
 ; GCN-O0-NEXT:  FunctionPass Manager

 ; GCN-O1-NEXT:        SI Insert Hard Clauses
 ; GCN-O1-NEXT:        SI Final Branch Preparation
 ; GCN-O1-NEXT:        SI peephole optimizations
 ; GCN-O1-NEXT:        Post RA hazard recognizer
 ; GCN-O1-NEXT:        AMDGPU Insert Delay ALU
 ; GCN-O1-NEXT:        Branch relaxation pass
 ; GCN-O1-NEXT:        Register Usage Information Collector Pass
 ; GCN-O1-NEXT:        Live DEBUG_VALUE analysis
+; GCN-O1-NEXT:        Machine Sanitizer Binary Metadata
 ; GCN-O1-NEXT:    Function register usage analysis
 ; GCN-O1-NEXT:    FunctionPass Manager
 ; GCN-O1-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O1-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O1-NEXT:      AMDGPU Assembly Printer
 ; GCN-O1-NEXT:      Free MachineFunction
 ; GCN-O1-NEXT:Pass Arguments:  -domtree
 ; GCN-O1-NEXT:  FunctionPass Manager

 ; GCN-O1-OPTS-NEXT:        SI Insert Hard Clauses
 ; GCN-O1-OPTS-NEXT:        SI Final Branch Preparation
 ; GCN-O1-OPTS-NEXT:        SI peephole optimizations
 ; GCN-O1-OPTS-NEXT:        Post RA hazard recognizer
 ; GCN-O1-OPTS-NEXT:        AMDGPU Insert Delay ALU
 ; GCN-O1-OPTS-NEXT:        Branch relaxation pass
 ; GCN-O1-OPTS-NEXT:        Register Usage Information Collector Pass
 ; GCN-O1-OPTS-NEXT:        Live DEBUG_VALUE analysis
+; GCN-O1-OPTS-NEXT:        Machine Sanitizer Binary Metadata
 ; GCN-O1-OPTS-NEXT:    Function register usage analysis
 ; GCN-O1-OPTS-NEXT:    FunctionPass Manager
 ; GCN-O1-OPTS-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O1-OPTS-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O1-OPTS-NEXT:      AMDGPU Assembly Printer
 ; GCN-O1-OPTS-NEXT:      Free MachineFunction
 ; GCN-O1-OPTS-NEXT:Pass Arguments:  -domtree
 ; GCN-O1-OPTS-NEXT:  FunctionPass Manager

 ; GCN-O2-NEXT:        SI Final Branch Preparation
 ; GCN-O2-NEXT:        SI peephole optimizations
 ; GCN-O2-NEXT:        Post RA hazard recognizer
 ; GCN-O2-NEXT:        Release VGPRs
 ; GCN-O2-NEXT:        AMDGPU Insert Delay ALU
 ; GCN-O2-NEXT:        Branch relaxation pass
 ; GCN-O2-NEXT:        Register Usage Information Collector Pass
 ; GCN-O2-NEXT:        Live DEBUG_VALUE analysis
+; GCN-O2-NEXT:        Machine Sanitizer Binary Metadata
 ; GCN-O2-NEXT:    Function register usage analysis
 ; GCN-O2-NEXT:    FunctionPass Manager
 ; GCN-O2-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O2-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O2-NEXT:      AMDGPU Assembly Printer
 ; GCN-O2-NEXT:      Free MachineFunction
 ; GCN-O2-NEXT:Pass Arguments:  -domtree
 ; GCN-O2-NEXT:  FunctionPass Manager

 ; GCN-O3-NEXT:        SI Final Branch Preparation
 ; GCN-O3-NEXT:        SI peephole optimizations
 ; GCN-O3-NEXT:        Post RA hazard recognizer
 ; GCN-O3-NEXT:        Release VGPRs
 ; GCN-O3-NEXT:        AMDGPU Insert Delay ALU
 ; GCN-O3-NEXT:        Branch relaxation pass
 ; GCN-O3-NEXT:        Register Usage Information Collector Pass
 ; GCN-O3-NEXT:        Live DEBUG_VALUE analysis
+; GCN-O3-NEXT:        Machine Sanitizer Binary Metadata
 ; GCN-O3-NEXT:    Function register usage analysis
 ; GCN-O3-NEXT:    FunctionPass Manager
 ; GCN-O3-NEXT:      Lazy Machine Block Frequency Analysis
 ; GCN-O3-NEXT:      Machine Optimization Remark Emitter
 ; GCN-O3-NEXT:      AMDGPU Assembly Printer
 ; GCN-O3-NEXT:      Free MachineFunction
 ; GCN-O3-NEXT:Pass Arguments:  -domtree
 ; GCN-O3-NEXT:  FunctionPass Manager
 ; GCN-O3-NEXT:    Dominator Tree Construction
 
 define void @empty() {
   ret void
 }

llvm/test/CodeGen/ARM/O3-pipeline.ll

 ; CHECK-NEXT:      Unpack machine instruction bundles
 ; CHECK-NEXT:      MachineDominator Tree Construction
 ; CHECK-NEXT:      Machine Natural Loop Construction
 ; CHECK-NEXT:      ARM block placement
 ; CHECK-NEXT:      optimise barriers pass
 ; CHECK-NEXT:      Contiguously Lay Out Funclets
 ; CHECK-NEXT:      StackMap Liveness Analysis
 ; CHECK-NEXT:      Live DEBUG_VALUE analysis
+; CHECK-NEXT:      Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:    Machine Outliner
 ; CHECK-NEXT:    FunctionPass Manager
 ; CHECK-NEXT:      ReachingDefAnalysis
 ; CHECK-NEXT:      ARM fix for Cortex-A57 AES Erratum 1742098
 ; CHECK-NEXT:      ARM Branch Targets
 ; CHECK-NEXT:      MachineDominator Tree Construction
 ; CHECK-NEXT:      ARM constant island placement and branch shortening pass
 ; CHECK-NEXT:      MachineDominator Tree Construction
 ; CHECK-NEXT:      Machine Natural Loop Construction
 ; CHECK-NEXT:      ReachingDefAnalysis
 ; CHECK-NEXT:      ARM Low Overhead Loops pass
 ; CHECK-NEXT:      Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:      Machine Optimization Remark Emitter
 ; CHECK-NEXT:      ARM Assembly Printer
 ; CHECK-NEXT:      Free MachineFunction

llvm/test/CodeGen/M68k/pipeline.ll

 ; CHECK-NEXT:      Branch Probability Basic Block Placement
 ; CHECK-NEXT:      Insert fentry calls
 ; CHECK-NEXT:      Insert XRay ops
 ; CHECK-NEXT:      Implement the 'patchable-function' attribute
 ; CHECK-NEXT:      M68k MOVEM collapser pass
 ; CHECK-NEXT:      Contiguously Lay Out Funclets
 ; CHECK-NEXT:      StackMap Liveness Analysis
 ; CHECK-NEXT:      Live DEBUG_VALUE analysis
+; CHECK-NEXT:      Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:      Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:      Machine Optimization Remark Emitter
 ; CHECK-NEXT:      M68k Assembly Printer
 ; CHECK-NEXT:      Free MachineFunction

llvm/test/CodeGen/PowerPC/O3-pipeline.ll

 ; CHECK-NEXT:       Insert XRay ops
 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       PowerPC Pre-Emit Peephole
 ; CHECK-NEXT:       PowerPC Expand ISEL Generation
 ; CHECK-NEXT:       PowerPC Early-Return Creation
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
+; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       PowerPC Expand Atomic
 ; CHECK-NEXT:       PowerPC Branch Selector
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Linux PPC Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 define void @f() {
   ret void
 }

llvm/test/CodeGen/RISCV/O0-pipeline.ll

 ; CHECK-NEXT:       Insert fentry calls
 ; CHECK-NEXT:       Insert XRay ops
 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       RISCV Make Compressible
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
+; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       RISCV pseudo instruction expansion pass
 ; CHECK-NEXT:       RISCV atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISCV Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/RISCV/O3-pipeline.ll

 ; CHECK-NEXT:       Insert fentry calls
 ; CHECK-NEXT:       Insert XRay ops
 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       RISCV Make Compressible
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
+; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:     Machine Outliner
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       RISCV pseudo instruction expansion pass
 ; CHECK-NEXT:       RISCV atomic pseudo instruction expansion pass
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       RISCV Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction

llvm/test/CodeGen/X86/O0-pipeline.ll

 ; CHECK-NEXT:       X86 vzeroupper inserter
 ; CHECK-NEXT:       Compressing EVEX instrs to VEX encoding when possibl
 ; CHECK-NEXT:       X86 Discriminate Memory Operands
 ; CHECK-NEXT:       X86 Insert Cache Prefetches
 ; CHECK-NEXT:       X86 insert wait instruction
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
+; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       X86 Speculative Execution Side Effect Suppression
 ; CHECK-NEXT:       X86 Indirect Thunks
 ; CHECK-NEXT:       X86 Return Thunks
 ; CHECK-NEXT:       Check CFA info and insert CFI instructions if needed
 ; CHECK-NEXT:       X86 Load Value Injection (LVI) Ret-Hardening
 ; CHECK-NEXT:       Pseudo Probe Inserter
 ; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       X86 Assembly Printer
 ; CHECK-NEXT:       Free MachineFunction
 
 define void @f() {
   ret void
 }

llvm/test/CodeGen/X86/opt-pipeline.ll

 ; CHECK-NEXT:       X86 LEA Fixup
 ; CHECK-NEXT:       Compressing EVEX instrs to VEX encoding when possible
 ; CHECK-NEXT:       X86 Discriminate Memory Operands
 ; CHECK-NEXT:       X86 Insert Cache Prefetches
 ; CHECK-NEXT:       X86 insert wait instruction
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
+; CHECK-NEXT:       Machine Sanitizer Binary Metadata
 ; CHECK-NEXT:       X86 Speculative Execution Side Effect Suppression
 ; CHECK-NEXT:       X86 Indirect Thunks
 ; CHECK-NEXT:       X86 Return Thunks
 ; CHECK-NEXT:       Check CFA info and insert CFI instructions if needed
 ; CHECK-NEXT:       X86 Load Value Injection (LVI) Ret-Hardening
 ; CHECK-NEXT:       Pseudo Probe Inserter
 ; CHECK-NEXT:       Unpack machine instruction bundles
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis