[Clang][ExtendLifetimes][3/4] Add -fextend-lifetimes flag to Clang to extend the lifetime of local variables

[StephenTozer]

Summary

This patch is part of a set of patches that add an -fextend-lifetimes flag to clang, which extends the lifetimes of local variables and parameters for improved debuggability. In addition to that flag, the patch series adds a new function attribute to disable the post-RA scheduler (which is applied by -fextend-lifetimes), a pragma to selectively disable -fextend-lifetimes, and an -fextend-this-ptr flag which functions as -fextend-lifetimes for this pointers only. All changes and tests in these patches were written by @wolfgangp, though I will be managing these reviews and addressing any comments raised. Discussion on the approach of this patch series as a whole should be directed to this patch, in order to simplify review.

---

This particular patch implements the clang side of the -fextend-lifetimes and -fextend-this-ptr flags, including adding the command line flags, applying the disable-post-ra attribute to functions having lifetimes extended (see the first patch in this sequence), and the emission of llvm.fake.use intrinsics for variables at the end of their scopes. The main bulk of the patch concerns emitting fake.use intrinsics, which are treated as noop cleanups since they should be emitted at all points where the variable falls out of scope.

In our (Sony's) own internal measurements (presented at EuroLLVM2023), using this flag at -O1 gave a 2.4% performance cost for a 22.1% increase in step and variable availability compared to just -O1. From these results, it appears that using this flag gives an efficient increase in debuggability of programs for its performance cost, and should be very useful for development builds.

[jmorse]

The code changes here look good, but it's time to be annoying about the tests -- most of them appear to run the optimisation pipelines, which is testing all of LLVM as well as clang in these tests. Possibly a simple fix to that is to put these in cross-project-tests. Otherwise, I believe these should be only testing the plain codegen output without optimisations at all, i.e. --disable-llvm-passes.

clang/lib/CodeGen/CGCall.cpp

3492–3499

It's a bit ugly to switch away from having a range-based loop, am I right in thinking this is because you need to skip the load-to-fake-use? Annoying but necessary I guess. Could I request an explicit "II = std::next(II);" though to really ram home to the reader that we're skipping an extra instruction, using the pre-increment is liable to be missed, and we're not trying to be super-concise here IMO.

The addition of the "I" variable looks like it might be un-necessary, that's only _used_ by GetStoreIfValid which has a domination assignment above it no?

clang/lib/CodeGen/CGDecl.cpp

1729–1730

(This could be even more concise; no strong opinion though).

[StephenTozer]

clang/lib/CodeGen/CGCall.cpp

3492–3499

Annoying but necessary I guess.

Probably not necessary actually - the addition of an extra variable could cover this case without removing the for-range.

clang/include/clang/Basic/Attr.td

 }
 
 def ObjCBoxable : Attr {
   let Spellings = [Clang<"objc_boxable">];
   let Subjects = SubjectList<[Record], ErrorDiag>;
   let Documentation = [ObjCBoxableDocs];
 }
 
+def ExtendLifetimesDisable : InheritableAttr {
+  let Spellings = [];
+  let Subjects = SubjectList<[Function]>;
+  let Documentation = [ExtendLifetimesDisableDocs];
+}
+
 def OptimizeNone : InheritableAttr {
   let Spellings = [Clang<"optnone">];
   let Subjects = SubjectList<[Function, ObjCMethod]>;
   let Documentation = [OptnoneDocs];
 }
 
 def Overloadable : Attr {
   let Spellings = [Clang<"overloadable">];

clang/include/clang/Basic/CodeGenOptions.def

 ENUM_CODEGENOPT(VecLib, VectorLibrary, 3, NoLibrary)
 
 /// The default TLS model to use.
 ENUM_CODEGENOPT(DefaultTLSModel, TLSModel, 2, GeneralDynamicTLSModel)
 
 /// Bit size of immediate TLS offsets (0 == use the default).
 VALUE_CODEGENOPT(TLSSize, 8, 0)
 
+/// Whether to extend the live range of the `this` pointer.
+CODEGENOPT(ExtendThisPtr, 1, 0)
+
+/// Whether to extend the live ranges of all local variables.
+CODEGENOPT(ExtendLifetimes, 1, 0)
+
 /// The default stack protector guard offset to use.
 VALUE_CODEGENOPT(StackProtectorGuardOffset, 32, INT_MAX)
 
 /// Number of path components to strip when emitting checks. (0 == full
 /// filename)
 VALUE_CODEGENOPT(EmitCheckPathComponentsToStrip, 32, 0)
 
 /// Whether to report the hotness of the code region for optimization remarks.

clang/include/clang/Driver/Options.td

defm stack_size_section : BoolFOption<"stack-size-section",
   PosFlag<SetTrue, [CC1Option], "Emit section containing metadata on function stack sizes">,
   NegFlag<SetFalse>>;
 def fstack_usage : Flag<["-"], "fstack-usage">, Group<f_Group>,
   HelpText<"Emit .su file containing information on function stack sizes">;
 def stack_usage_file : Separate<["-"], "stack-usage-file">,
   Flags<[CC1Option, NoDriverOption]>,
   HelpText<"Filename (or -) to write stack usage output to">,
   MarshallingInfoString<CodeGenOpts<"StackUsageOutput">>;
+def fextend_this_ptr : Flag <["-"], "fextend-this-ptr">, Group<f_Group>,
+  HelpText<"Extend the lifetime of the 'this' pointer to improve visibility "
+           "in optimized debugging">, Flags<[CC1Option]>,
+  MarshallingInfoFlag<CodeGenOpts<"ExtendThisPtr">>;
+def fextend_lifetimes : Flag <["-"], "fextend-lifetimes">, Group<f_Group>,
+  HelpText<"Extend the lifetimes of local variables and parameters to improve "
+           "visibility in optimized debugging">, Flags<[CC1Option]>,
+  MarshallingInfoFlag<CodeGenOpts<"ExtendLifetimes">>;
 
 defm unique_basic_block_section_names : BoolFOption<"unique-basic-block-section-names",
   CodeGenOpts<"UniqueBasicBlockSectionNames">, DefaultFalse,
   PosFlag<SetTrue, [CC1Option], "Use unique names for basic block sections (ELF Only)">,
   NegFlag<SetFalse>>;
 defm unique_internal_linkage_names : BoolFOption<"unique-internal-linkage-names",
   CodeGenOpts<"UniqueInternalLinkageNames">, DefaultFalse,
   PosFlag<SetTrue, [CC1Option], "Uniqueify Internal Linkage Symbol Names by appending"

clang/lib/CodeGen/CGCall.cpp

auto shouldDisableTailCalls = [&] {
             return true;
       }
 
       return false;
     };
     if (shouldDisableTailCalls())
       FuncAttrs.addAttribute("disable-tail-calls", "true");
 
+    // Suppress the machine instruction scheduler when -fextend-lifetimes is on.
+    if (CodeGenOpts.ExtendLifetimes)
+      FuncAttrs.addAttribute("disable-post-ra");
+
     // CPU/feature overrides.  addDefaultFunctionDefinitionAttributes
     // handles these separately to set them based on the global defaults.
     GetCPUAndFeaturesAttributes(CalleeInfo.getCalleeDecl(), FuncAttrs);
   }
 
   // Collect attributes from arguments and return values.
   ClangToLLVMArgMapping IRFunctionArgs(getContext(), FI);
 
static llvm::StoreInst *findDominatingStoreToReturnValue(CodeGenFunction &CGF) {
   // happen with how we generate implicit-returns; it can also happen
   // with noreturn cleanups.
   if (!CGF.ReturnValue.getPointer()->hasOneUse()) {
     llvm::BasicBlock *IP = CGF.Builder.GetInsertBlock();
     if (IP->empty()) return nullptr;
 
     // Look at directly preceding instruction, skipping bitcasts and lifetime
     // markers.
-    for (llvm::Instruction &I : make_range(IP->rbegin(), IP->rend())) {
-      if (isa<llvm::BitCastInst>(&I))
+    llvm::Instruction *I = &IP->back();
+    for (llvm::BasicBlock::reverse_iterator II = IP->rbegin(), IE = IP->rend();
+         II != IE; ++II) {
+      if (isa<llvm::BitCastInst>(&*II))
         continue;
-      if (auto *II = dyn_cast<llvm::IntrinsicInst>(&I))
-        if (II->getIntrinsicID() == llvm::Intrinsic::lifetime_end)
+      if (auto *Intrinsic = dyn_cast<llvm::IntrinsicInst>(&*II)) {
+        if (Intrinsic->getIntrinsicID() == llvm::Intrinsic::lifetime_end)
           continue;
 
-      return GetStoreIfValid(&I);
+        // Ignore fake uses and the instructions that load their
+        // operands.
+        if (Intrinsic->getIntrinsicID() == llvm::Intrinsic::fake_use) {
+          const llvm::Value *FakeUseArg = Intrinsic->getArgOperand(0);
+          if (++II == IE || &*II != FakeUseArg)
+            break;
+          continue;
+        }
+      }
+      I = &*II;
+      return GetStoreIfValid(I);
     }
     return nullptr;
   }
 
   llvm::StoreInst *store =
       GetStoreIfValid(CGF.ReturnValue.getPointer()->user_back());
   if (!store) return nullptr;
 

clang/lib/CodeGen/CGCleanup.h

class CleanupBitFields {
     unsigned IsEHCleanup : 1;
 
     /// Whether this cleanup is currently active.
     unsigned IsActive : 1;
 
     /// Whether this cleanup is a lifetime marker
     unsigned IsLifetimeMarker : 1;
 
+    /// Whether this cleanup is a fake use
+    unsigned IsFakeUse : 1;
+
     /// Whether the normal cleanup should test the activation flag.
     unsigned TestFlagInNormalCleanup : 1;
 
     /// Whether the EH cleanup should test the activation flag.
     unsigned TestFlagInEHCleanup : 1;
 
     /// The amount of extra storage needed by the Cleanup.
     /// Always a multiple of the scope-stack alignment.
EHCleanupScope(bool isNormal, bool isEH, unsigned cleanupSize,
       : EHScope(EHScope::Cleanup, enclosingEH),
         EnclosingNormal(enclosingNormal), NormalBlock(nullptr),
         ActiveFlag(Address::invalid()), ExtInfo(nullptr),
         FixupDepth(fixupDepth) {
     CleanupBits.IsNormalCleanup = isNormal;
     CleanupBits.IsEHCleanup = isEH;
     CleanupBits.IsActive = true;
     CleanupBits.IsLifetimeMarker = false;
+    CleanupBits.IsFakeUse = false;
     CleanupBits.TestFlagInNormalCleanup = false;
     CleanupBits.TestFlagInEHCleanup = false;
     CleanupBits.CleanupSize = cleanupSize;
 
     assert(CleanupBits.CleanupSize == cleanupSize && "cleanup size overflow");
   }
 
   void Destroy() {
public:
   bool isEHCleanup() const { return CleanupBits.IsEHCleanup; }
 
   bool isActive() const { return CleanupBits.IsActive; }
   void setActive(bool A) { CleanupBits.IsActive = A; }
 
   bool isLifetimeMarker() const { return CleanupBits.IsLifetimeMarker; }
   void setLifetimeMarker() { CleanupBits.IsLifetimeMarker = true; }
 
+  bool isFakeUse() const { return CleanupBits.IsFakeUse; }
+  void setFakeUse() { CleanupBits.IsFakeUse = true; }
+
   bool hasActiveFlag() const { return ActiveFlag.isValid(); }
   Address getActiveFlag() const {
     return ActiveFlag;
   }
   void setActiveFlag(Address Var) {
     assert(Var.getAlignment().isOne());
     ActiveFlag = Var;
   }

clang/lib/CodeGen/CGCleanup.cpp

char *EHScopeStack::allocate(size_t Size) {
   StartOfData -= Size;
   return StartOfData;
 }
 
 void EHScopeStack::deallocate(size_t Size) {
   StartOfData += llvm::alignTo(Size, ScopeStackAlignment);
 }
 
-bool EHScopeStack::containsOnlyLifetimeMarkers(
+bool EHScopeStack::containsOnlyNoopCleanups(
     EHScopeStack::stable_iterator Old) const {
   for (EHScopeStack::iterator it = begin(); stabilize(it) != Old; it++) {
     EHCleanupScope *cleanup = dyn_cast<EHCleanupScope>(&*it);
-    if (!cleanup || !cleanup->isLifetimeMarker())
+    if (!cleanup || !(cleanup->isLifetimeMarker() || cleanup->isFakeUse()))
       return false;
   }
 
   return true;
 }
 
 bool EHScopeStack::requiresLandingPad() const {
   for (stable_iterator si = getInnermostEHScope(); si != stable_end(); ) {

 }
 
 
 void *EHScopeStack::pushCleanup(CleanupKind Kind, size_t Size) {
   char *Buffer = allocate(EHCleanupScope::getSizeForCleanupSize(Size));
   bool IsNormalCleanup = Kind & NormalCleanup;
   bool IsEHCleanup = Kind & EHCleanup;
   bool IsLifetimeMarker = Kind & LifetimeMarker;
+  bool IsFakeUse = Kind & FakeUse;
 
   // Per C++ [except.terminate], it is implementation-defined whether none,
   // some, or all cleanups are called before std::terminate. Thus, when
   // terminate is the current EH scope, we may skip adding any EH cleanup
   // scopes.
   if (InnermostEHScope != stable_end() &&
       find(InnermostEHScope)->getKind() == EHScope::Terminate)
     IsEHCleanup = false;
 
   EHCleanupScope *Scope =
     new (Buffer) EHCleanupScope(IsNormalCleanup,
                                 IsEHCleanup,
                                 Size,
                                 BranchFixups.size(),
                                 InnermostNormalCleanup,
                                 InnermostEHScope);
   if (IsNormalCleanup)
     InnermostNormalCleanup = stable_begin();
   if (IsEHCleanup)
     InnermostEHScope = stable_begin();
   if (IsLifetimeMarker)
     Scope->setLifetimeMarker();
+  if (IsFakeUse)
+    Scope->setFakeUse();
 
   // With Windows -EHa, Invoke llvm.seh.scope.begin() for EHCleanup
   if (CGF->getLangOpts().EHAsynch && IsEHCleanup && !IsLifetimeMarker &&
       CGF->getTarget().getCXXABI().isMicrosoft())
     CGF->EmitSehCppScopeBegin();
 
   return Scope->getCleanupBuffer();
 }

clang/lib/CodeGen/CGDecl.cpp

assert(Addr->getType()->getPointerAddressSpace() ==
              CGM.getDataLayout().getAllocaAddrSpace() &&
          "Pointer should be in alloca address space");
   Addr = Builder.CreateBitCast(Addr, AllocaInt8PtrTy);
   llvm::CallInst *C =
       Builder.CreateCall(CGM.getLLVMLifetimeEndFn(), {Size, Addr});
   C->setDoesNotThrow();
 }
 
+void CodeGenFunction::EmitFakeUse(Address Addr) {
+  auto NL = ApplyDebugLocation::CreateEmpty(*this);
+  llvm::Value *V = Builder.CreateLoad(Addr, "fake.use");
+  llvm::CallInst *C = Builder.CreateCall(CGM.getLLVMFakeUseFn(), {V});
+  C->setDoesNotThrow();
+  C->setTailCallKind(llvm::CallInst::TCK_NoTail);
+}
+
 void CodeGenFunction::EmitAndRegisterVariableArrayDimensions(
     CGDebugInfo *DI, const VarDecl &D, bool EmitDebugInfo) {
   // For each dimension stores its QualType and corresponding
   // size-expression Value.
   SmallVector<CodeGenFunction::VlaSizePair, 4> Dimensions;
   SmallVector<IdentifierInfo *, 4> VLAExprNames;
 
   // Break down the array into individual dimensions.
else {
       MD = DI->EmitDeclareOfAutoVariable(ArtificialDecl, VlaSize.NumElts,
                                          Builder);
     }
     assert(MD && "No Size expression debug node created");
     DI->registerVLASizeExpression(VlaSize.Type, MD);
   }
 }
 
+/// Return the maximum size of an aggregate for which we generate a fake use
+/// intrinsic when -fextend-lifetimes is in effect.
+static uint64_t maxFakeUseAggregateSize(const ASTContext &C) {
+  return 4 * C.getTypeSize(C.UnsignedIntTy);
+}
+
+// Helper function to determine whether a variable's or parameter's lifetime
+// should be extended.
+static bool extendLifetime(const ASTContext &Context, const Decl *FuncDecl,
+                           const VarDecl &D,
+                           ImplicitParamDecl *CXXABIThisDecl) {
+  // When we're not inside a valid function it is unlikely that any
+  // lifetime extension is useful.
+  if (!FuncDecl)
+    return false;
+  if (FuncDecl->isImplicit())
+    return false;
+  // Do not extend compiler-created variables except for the this pointer.
+  if (D.isImplicit() && &D != CXXABIThisDecl)
+    return false;
+  QualType Ty = D.getType();
+  // No need to extend volatiles, they have a memory location.
+  if (Ty.isVolatileQualified())
+    return false;
+  // Don't extend variables that exceed a certain size.
+  if (Context.getTypeSize(Ty) > maxFakeUseAggregateSize(Context))
+    return false;
+  // Do not extend variables in nodebug functions.
+  if (FuncDecl->hasAttr<NoDebugAttr>())
+    return false;
+  return true;
+}
+
 /// EmitAutoVarAlloca - Emit the alloca and debug information for a
 /// local variable.  Does not emit initialization or destruction.
 CodeGenFunction::AutoVarEmission
 CodeGenFunction::EmitAutoVarAlloca(const VarDecl &D) {
   QualType Ty = D.getType();
   assert(
       Ty.getAddressSpace() == LangAS::Default ||
       (Ty.getAddressSpace() == LangAS::opencl_private && getLangOpts().OpenCL));
if (D.hasAttr<AnnotateAttr>() && HaveInsertPoint())
     EmitVarAnnotations(&D, address.getPointer());
 
   // Make sure we call @llvm.lifetime.end.
   if (emission.useLifetimeMarkers())
     EHStack.pushCleanup<CallLifetimeEnd>(NormalEHLifetimeMarker,
                                          emission.getOriginalAllocatedAddress(),
                                          emission.getSizeForLifetimeMarkers());
 
+  // Analogous to lifetime markers, we use a 'cleanup' to emit fake.use
+  // calls for local variables. We are exempting volatile variables and
+  // non-scalars larger than 4 times the size of an unsigned int (32 bytes).
+  // Larger non-scalars are often allocated in memory and may create unnecessary
+  // overhead.
+  if (CGM.getCodeGenOpts().ExtendLifetimes) {
+    if (extendLifetime(getContext(), CurCodeDecl, D, CXXABIThisDecl))
+      EHStack.pushCleanup<FakeUse>(NormalFakeUse,
+                                   emission.getAllocatedAddress());
+  }
+
   return emission;
 }
 
 static bool isCapturedBy(const VarDecl &, const Expr *);
 
 /// Determines whether the given __block variable is potentially
 /// captured by the given statement.
 static bool isCapturedBy(const VarDecl &Var, const Stmt *S) {

 llvm::Function *CodeGenModule::getLLVMLifetimeEndFn() {
   if (LifetimeEndFn)
     return LifetimeEndFn;
   LifetimeEndFn = llvm::Intrinsic::getDeclaration(&getModule(),
     llvm::Intrinsic::lifetime_end, AllocaInt8PtrTy);
   return LifetimeEndFn;
 }
 
+/// Lazily declare the @llvm.fake.use intrinsic.
+llvm::Function *CodeGenModule::getLLVMFakeUseFn() {
+  if (FakeUseFn)
+    return FakeUseFn;
+  FakeUseFn =
+      llvm::Intrinsic::getDeclaration(&getModule(), llvm::Intrinsic::fake_use);
+  return FakeUseFn;
+}
+
 namespace {
   /// A cleanup to perform a release of an object at the end of a
   /// function.  This is used to balance out the incoming +1 of a
   /// ns_consumed argument when we can't reasonably do that just by
   /// not doing the initial retain for a __block argument.
   struct ConsumeARCParameter final : EHScopeStack::Cleanup {
     ConsumeARCParameter(llvm::Value *param,
                         ARCPreciseLifetime_t precise)
void CodeGenFunction::EmitParmDecl(const VarDecl &D, ParamValue Arg,
   }
 
   // Store the initial value into the alloca.
   if (DoStore)
     EmitStoreOfScalar(ArgVal, lv, /* isInitialization */ true);
 
   setAddrOfLocalVar(&D, DeclPtr);
 
+  // Push a FakeUse 'cleanup' object onto the EHStack for the parameter,
+  // which may be the 'this' pointer. This causes the emission of a fake.use
+  // call with the parameter as argument at the end of the function.
+  if (CGM.getCodeGenOpts().ExtendLifetimes ||
+      (CGM.getCodeGenOpts().ExtendThisPtr && &D == CXXABIThisDecl)) {
+    if (extendLifetime(getContext(), CurCodeDecl, D, CXXABIThisDecl))
+      EHStack.pushCleanup<FakeUse>(NormalFakeUse, DeclPtr);
+  }
+
   // Emit debug info for param declarations in non-thunk functions.
   if (CGDebugInfo *DI = getDebugInfo()) {
     if (CGM.getCodeGenOpts().hasReducedDebugInfo() && !CurFuncIsThunk &&
         !NoDebugInfo) {
       llvm::DILocalVariable *DILocalVar = DI->EmitDeclareOfArgVariable(
           &D, AllocaPtr.getPointer(), ArgNo, Builder, UseIndirectDebugAddress);
       if (const auto *Var = dyn_cast_or_null<ParmVarDecl>(&D))
         DI->getParamDbgMappings().insert({Var, DILocalVar});

clang/lib/CodeGen/CodeGenFunction.h

public:
     CallLifetimeEnd(Address addr, llvm::Value *size)
         : Addr(addr.getPointer()), Size(size) {}
 
     void Emit(CodeGenFunction &CGF, Flags flags) override {
       CGF.EmitLifetimeEnd(Size, Addr);
     }
   };
 
+  // We are using objects of this 'cleanup' class to emit fake.use calls
+  // for -fextend-lifetimes and -fextend-this-ptr. They are placed at the end of
+  // a variable's scope analogous to lifetime markers.
+  class FakeUse final : public EHScopeStack::Cleanup {
+    Address Addr;
+
+  public:
+    FakeUse(Address addr) : Addr(addr) {}
+
+    void Emit(CodeGenFunction &CGF, Flags flags) override {
+      CGF.EmitFakeUse(Addr);
+    }
+  };
+
   /// Header for data within LifetimeExtendedCleanupStack.
   struct LifetimeExtendedCleanupHeader {
     /// The size of the following cleanup object.
     unsigned Size;
     /// The kind of cleanup to push: a value from the CleanupKind enumeration.
     unsigned Kind : 31;
     /// Whether this is a conditional cleanup.
     unsigned IsConditional : 1;
public:
   void EmitCXXConstructExpr(const CXXConstructExpr *E, AggValueSlot Dest);
 
   void EmitSynthesizedCXXCopyCtor(Address Dest, Address Src, const Expr *Exp);
 
   void EmitCXXThrowExpr(const CXXThrowExpr *E, bool KeepInsertionPoint = true);
 
   RValue EmitAtomicExpr(AtomicExpr *E);
 
+  void EmitFakeUse(Address Addr);
+
   //===--------------------------------------------------------------------===//
   //                         Annotations Emission
   //===--------------------------------------------------------------------===//
 
   /// Emit an annotation call (intrinsic).
   llvm::Value *EmitAnnotationCall(llvm::Function *AnnotationFn,
                                   llvm::Value *AnnotatedVal,
                                   StringRef AnnotationStr,

clang/lib/CodeGen/CodeGenFunction.cpp

else
       DI->EmitLocation(Builder, EndLoc);
   }
 
   // Pop any cleanups that might have been associated with the
   // parameters.  Do this in whatever block we're currently in; it's
   // important to do this before we enter the return block or return
   // edges will be *really* confused.
   bool HasCleanups = EHStack.stable_begin() != PrologueCleanupDepth;
-  bool HasOnlyLifetimeMarkers =
-      HasCleanups && EHStack.containsOnlyLifetimeMarkers(PrologueCleanupDepth);
-  bool EmitRetDbgLoc = !HasCleanups || HasOnlyLifetimeMarkers;
+  bool HasOnlyNoopCleanups =
+      HasCleanups && EHStack.containsOnlyNoopCleanups(PrologueCleanupDepth);
+  bool EmitRetDbgLoc = !HasCleanups || HasOnlyNoopCleanups;
 
   std::optional<ApplyDebugLocation> OAL;
   if (HasCleanups) {
     // Make sure the line table doesn't jump back into the body for
     // the ret after it's been at EndLoc.
     if (CGDebugInfo *DI = getDebugInfo()) {
       if (OnlySimpleReturnStmts)
         DI->EmitLocation(Builder, EndLoc);

clang/lib/CodeGen/CodeGenModule.h

private:
   /// @}
 
   /// void @llvm.lifetime.start(i64 %size, i8* nocapture <ptr>)
   llvm::Function *LifetimeStartFn = nullptr;
 
   /// void @llvm.lifetime.end(i64 %size, i8* nocapture <ptr>)
   llvm::Function *LifetimeEndFn = nullptr;
 
+  /// void @llvm.fake.use(i8* nocapture <ptr>)
+  llvm::Function *FakeUseFn = nullptr;
+
   std::unique_ptr<SanitizerMetadata> SanitizerMD;
 
   llvm::MapVector<const Decl *, bool> DeferredEmptyCoverageMappingDecls;
 
   std::unique_ptr<CoverageMappingModuleGen> CoverageMapping;
 
   /// Mapping from canonical types to their metadata identifiers. We need to
   /// maintain this mapping because identifiers may be formed from distinct
public:
   llvm::Constant *getNSConcreteStackBlock();
   llvm::FunctionCallee getBlockObjectAssign();
   llvm::FunctionCallee getBlockObjectDispose();
 
   ///@}
 
   llvm::Function *getLLVMLifetimeStartFn();
   llvm::Function *getLLVMLifetimeEndFn();
+  llvm::Function *getLLVMFakeUseFn();
 
   // Make sure that this type is translated.
   void UpdateCompletedType(const TagDecl *TD);
 
   llvm::Constant *getMemberPointerConstant(const UnaryOperator *e);
 
   /// Emit type info if type of an expression is a variably modified
   /// type. Also emit proper debug info for cast types.

clang/lib/CodeGen/EHScopeStack.h

enum CleanupKind : unsigned {
   /// Denotes a cleanup that should run when a scope is exited using normal
   /// control flow (falling off the end of the scope, return, goto, ...).
   NormalCleanup = 0x2,
 
   NormalAndEHCleanup = EHCleanup | NormalCleanup,
 
   LifetimeMarker = 0x8,
   NormalEHLifetimeMarker = LifetimeMarker | NormalAndEHCleanup,
+
+  // FakeUse needs to be recognized as a special cleanup similar to lifetime
+  // markers chiefly to be ignored in most contexts.
+  FakeUse = 0x10,
+  NormalFakeUse = FakeUse | NormalCleanup,
 };
 
 /// A stack of scopes which respond to exceptions, including cleanups
 /// and catch blocks.
 class EHScopeStack {
 public:
   /* Should switch to alignof(uint64_t) instead of 8, when EHCleanupScope can */
   enum { ScopeStackAlignment = 8 };
public:
 
   /// Push a terminate handler on the stack.
   void pushTerminate();
 
   /// Pops a terminate handler off the stack.
   void popTerminate();
 
   // Returns true iff the current scope is either empty or contains only
-  // lifetime markers, i.e. no real cleanup code
-  bool containsOnlyLifetimeMarkers(stable_iterator Old) const;
+  // noop cleanups, i.e. lifetime markers and fake uses.
+  bool containsOnlyNoopCleanups(stable_iterator Old) const;
 
   /// Determines whether the exception-scopes stack is empty.
   bool empty() const { return StartOfData == EndOfBuffer; }
 
   bool requiresLandingPad() const;
 
   /// Determines whether there are any normal cleanups on the stack.
   bool hasNormalCleanups() const {

clang/lib/Driver/ToolChains/Clang.cpp

   if (Arg *A = Args.getLastArg(options::OPT__serialize_diags)) {
     CmdArgs.push_back("-serialize-diagnostic-file");
     CmdArgs.push_back(Args.MakeArgString(A->getValue()));
   }
 
   if (Args.hasArg(options::OPT_fretain_comments_from_system_headers))
     CmdArgs.push_back("-fretain-comments-from-system-headers");
 
+  if (Args.hasArg(options::OPT_fextend_this_ptr))
+    CmdArgs.push_back("-fextend-this-ptr");
+  if (Args.hasArg(options::OPT_fextend_lifetimes))
+    CmdArgs.push_back("-fextend-lifetimes");
+
   // Forward -fcomment-block-commands to -cc1.
   Args.AddAllArgs(CmdArgs, options::OPT_fcomment_block_commands);
   // Forward -fparse-all-comments to -cc1.
   Args.AddAllArgs(CmdArgs, options::OPT_fparse_all_comments);
 
   // Turn -fplugin=name.so into -load name.so
   for (const Arg *A : Args.filtered(options::OPT_fplugin_EQ)) {
     CmdArgs.push_back("-load");

clang/lib/Frontend/CompilerInvocation.cpp

#undef CODEGEN_OPTION_WITH_MARSHALLING
   // FIXME: Report unrecoverable sanitizers incorrectly specified here.
   parseSanitizerKinds("-fsanitize-recover=",
                       Args.getAllArgValues(OPT_fsanitize_recover_EQ), Diags,
                       Opts.SanitizeRecover);
   parseSanitizerKinds("-fsanitize-trap=",
                       Args.getAllArgValues(OPT_fsanitize_trap_EQ), Diags,
                       Opts.SanitizeTrap);
 
+  Opts.ExtendThisPtr =
+      Opts.OptimizationLevel > 0 && Args.hasArg(OPT_fextend_this_ptr);
+  Opts.ExtendLifetimes =
+      Opts.OptimizationLevel > 0 && Args.hasArg(OPT_fextend_lifetimes);
+
   Opts.EmitVersionIdentMetadata = Args.hasFlag(OPT_Qy, OPT_Qn, true);
 
   if (Args.hasArg(options::OPT_ffinite_loops))
     Opts.FiniteLoops = CodeGenOptions::FiniteLoopsKind::Always;
   else if (Args.hasArg(options::OPT_fno_finite_loops))
     Opts.FiniteLoops = CodeGenOptions::FiniteLoopsKind::Never;
 
   Opts.EmitIEEENaNCompliantInsts = Args.hasFlag(

clang/test/CodeGen/extend-liveness1.c

+// RUN: %clang_cc1 %s -O2 -emit-llvm -fextend-lifetimes -o - | FileCheck %s
+// Check that fake use calls are emitted at the correct locations, i.e.
+// at the end of lexical blocks and at the end of the function.
+
+extern int use(int);
+int glob1;
+int glob2;
+float globf;
+
+int foo(int i) {
+  // CHECK: define{{.*}}foo
+  if (i < 4) {
+    int j = i * 3;
+    if (glob1 > 3) {
+      float f = globf;
+      // CHECK: [[SSAVAL:%[a-z0-9]*]] = load float{{.*}}globf
+      j = f;
+      glob2 = j;
+      // CHECK: store{{.*}}glob2
+      // CHECK-NEXT: call void (...) @llvm.fake.use(float [[SSAVAL]])
+    }
+    glob1 = j;
+    // CHECK: store{{.*}}glob1
+    // CHECK-NEXT: call void (...) @llvm.fake.use(i32 %j.
+  }
+  // CHECK: call void (...) @llvm.fake.use(i32 %i)
+  // CHECK-NEXT: ret
+  return 4;
+}

clang/test/CodeGen/extend-liveness2.cpp

+// RUN: %clang_cc1 %s -O2 -emit-llvm -fextend-lifetimes -fcxx-exceptions -fexceptions -o - | FileCheck %s
+// REQUIRES: x86-registered-target
+// This test checks that the fake_use concept works with exception handling and that we
+// can handle the __int128 data type.
+
+class A {
+public:
+  A(int i) : m_i(i) {}
+  void func(__int128 i128);
+
+  int m_i;
+};
+
+extern int bar();
+extern void foo();
+int glob;
+
+void A::func(__int128 i128) {
+  int j = 4;
+  try {
+    int k = bar();
+    foo();
+    // CHECK: [[SSAVAL:%[a-z0-9]*]] = invoke{{.*}}bar
+    glob = 0;
+    // CHECK: store{{.*}}glob
+    // CHECK-NEXT: call void (...) @llvm.fake.use(i32 [[SSAVAL]])
+  } catch (...) {
+    foo();
+  }
+  // CHECK-LABEL: try.cont:
+  // CHECK-DAG: call void (...) @llvm.fake.use({{.*%this}})
+  // CHECK-DAG: call void (...) @llvm.fake.use(i128 %i128)
+  // CHECK: ret void
+}

clang/test/CodeGen/fake-use-determinism.c

+// RUN: %clang -S -O2 -emit-llvm -fextend-lifetimes %s -o - | FileCheck %s
+// REQUIRES: asserts
+//
+// We are checking that the fake.use calls for i, j and k appear
+// in a particular order. It is not the order itself that is important
+// but that it remains the same between different test runs.
+
+// CHECK:       call {{.*}}void (...) @llvm.fake.use(i32 %i)
+// CHECK-NEXT:  call {{.*}}void (...) @llvm.fake.use(i32 %j)
+// CHECK-NEXT:  call {{.*}}void (...) @llvm.fake.use(i32 %k)
+
+extern void bar();
+void foo(int i, int j, int k)
+{
+   for (int l = 0; l < i; l++) {
+      bar();
+   }
+}

clang/test/CodeGen/fake-use-lambda.cpp

+// RUN: %clang_cc1 %s -triple=%itanium_abi_triple -O1 -emit-llvm -fextend-lifetimes -o - | FileCheck %s
+// Make sure we don't crash compiling a lambda that is not nested in a function.
+// We also check that fake uses are properly issued in lambdas.
+
+int glob;
+
+extern int foo();
+
+struct S {
+  static const int a;
+};
+
+const int S::a = [](int b) __attribute__((noinline)) {
+  return b * foo();
+}
+(glob);
+
+int func(int param) {
+  return ([=](int lambdaparm) __attribute__((noinline))->int {
+    int lambdalocal = lambdaparm * 2;
+    return lambdalocal;
+  }(glob));
+}
+
+// We are looking for the first lambda's call operator, which should contain
+// 2 fake uses, one for 'b' and one for its 'this' pointer (in that order).
+// The mangled function name contains a $_0, followed by 'cl'.
+// This lambda is an orphaned lambda, i.e. one without lexical parent.
+//
+// CHECK:     define internal {{.+\"_Z.+\$_0.*cl.*\"}}
+// CHECK-NOT: ret
+// CHECK:     fake.use(i32
+// CHECK-NOT: ret
+// CHECK:     fake.use(ptr
+
+// The second lambda. We are looking for 3 fake uses.
+// CHECK:     define internal {{.+\"_Z.+\$_0.*cl.*\"}}
+// CHECK-NOT: ret
+// CHECK:     fake.use(i32
+// CHECK-NOT: ret
+// CHECK:     fake.use(i32
+// CHECK-NOT: ret
+// CHECK:     fake.use(ptr

clang/test/CodeGen/fake-use-landingpad.c

+// RUN: %clang_cc1 %s -O3 -emit-llvm -fextend-lifetimes -fexceptions -o - | FileCheck %s
+
+// Check that fake uses do not mistakenly cause a landing pad to be generated when
+// exceptions are enabled.
+
+extern void bar(int);
+void foo(int p) {
+  int a = 17;
+  bar(a);
+}
+
+// CHECK:      define {{.*}} @foo
+// CHECK-NOT:  personality
+// CHECK:      entry:
+// CHECK-NOT:  landingpad

clang/test/CodeGen/fake-use-noreturn.c

+// RUN: %clang_cc1 %s -S -emit-llvm -fextend-lifetimes -o %t.ll
+//
+// Check we don't assert when we have a return in a nested conditional and
+// there is no code at the end of the function.
+
+// CHECK: define{{.*}}main
+// CHECK: call{{.*}}llvm.fake.use
+
+void foo(int i) {
+   while (0)
+     if (1)
+       return;
+}

clang/test/CodeGen/fake-use-return-line.c

+// RUN: %clang_cc1 -S -emit-llvm -O2 -debug-info-kind=limited -fextend-lifetimes -o - %s | FileCheck %s
+int main()
+{
+  volatile int a = 1;
+  int b = a + 2;
+  return b;
+}
+// CHECK: define{{.*}}@main
+// CHECK: ret i32{{.*}}!dbg ![[MDINDEX:[0-9]*]]
+// CHECK: ![[MDINDEX]] = !DILocation(line: 6

clang/test/CodeGen/fake-use-sanitizer.cpp

+// RUN: %clang_cc1 %s -O1 -emit-llvm -fextend-lifetimes -fsanitize=null -fsanitize-trap=null -o - | FileCheck --check-prefix TRAP %s
+// RUN: %clang_cc1 %s -O1 -emit-llvm -fextend-lifetimes -o - | FileCheck --check-prefix FAKEUSE %s
+
+// With -fextend-lifetimes the compiler generated a fake.use of a
+// reference variable at the end of the function, in the cleanup block. This prompted the 
+// address sanitizer to verify that the variable has been allocated properly, even when 
+// the function returns early.
+// We check that sanitizers are disabled for code generated for the benefit of fake.use 
+// intrinsics, as well as that the fake.use is no longer generated in the cleanup block.
+// It should be generated in the block preceding the cleanup block instead.
+
+struct A { short s1, s2; };
+extern A& getA();
+
+void foo()
+{
+  auto& va = getA();
+  short s = va.s1 & ~va.s2;
+  if (s == 0)
+    return;
+
+  auto& vb = getA();
+}
+
+// TRAP:         define{{.*}}foo
+// TRAP:         [[COMPARE:%[^\s]*]] = icmp eq
+// TRAP-NOT:     br i1 [[COMPARE]]{{.*}}trap
+// TRAP:         br i1 [[COMPARE]]{{.*}}%if.end
+// TRAP-NOT:     trap:
+// TRAP:         if.end:
+// TRAP-NOT:     call{{.*}}llvm.trap
+
+// FAKEUSE:      if.end:
+// FAKEUSE-NEXT: [[CALL:%[^\s]*]] = {{.*}}call{{.*}}getA
+// FAKEUSE-NOT:  br{{.*}}cleanup
+// FAKEUSE:      call{{.*}}fake.use({{.*}}[[CALL]])
+// FAKEUSE:      cleanup:

clang/test/CodeGen/fake-use-scalar.c

+// RUN: %clang_cc1 %s -O2 -emit-llvm -fextend-lifetimes -o - | FileCheck %s
+// Make sure we don't generate fake.use for non-scalar variables.
+// Make sure we don't generate fake.use for volatile variables
+// and parameters even when they are scalar.
+
+struct A {
+  unsigned long t;
+  char c[1024];
+  unsigned char r[32];
+};
+
+
+int foo(volatile int param)
+{
+  struct A s;
+  volatile int vloc;
+  struct A v[128];
+  char c[33];
+  return 0;
+}
+
+// CHECK-NOT:  fake.use

clang/test/CodeGen/fake-use-small-aggs.c

+// RUN: %clang_cc1 %s -O2 -emit-llvm -fextend-lifetimes -disable-llvm-passes -o - | FileCheck %s
+// Check that we generate a fake_use call for small aggregate types.
+
+// CHECK-DAG:  %[[FAKEUSE1:[^ ]+]] = load{{.*}} %loc,
+// CHECK-DAG:  call{{.*}}llvm.fake.use({{.*}}%[[FAKEUSE1]])
+// CHECK-DAG:  %[[FAKEUSE2:[^ ]+]] = load{{.*}} %arr,
+// CHECK-DAG:  call{{.*}}llvm.fake.use({{.*}}%[[FAKEUSE2]])
+// CHECK-DAG:  %[[FAKEUSE3:[^ ]+]] = load{{.*}} %S,
+// CHECK-DAG:  call{{.*}}llvm.fake.use({{.*}}%[[FAKEUSE3]])
+
+struct s {
+  int i;
+  int j;
+};
+
+extern void inita(int *);
+extern struct s inits();
+void foo(struct s S)
+{
+   int arr[4];
+   inita (arr);
+   struct s loc = inits();
+}
+

clang/test/CodeGen/fake-use-while.c

+// RUN: %clang_cc1 %s -S -emit-llvm -fextend-lifetimes -o %t.ll
+//
+// Check we don't assert when there is no more code after a while statement
+// and the body of the while statement ends in a return, i.e. no insertion point
+// is available.
+
+// CHECK: define{{.*}}main
+// CHECK: call{{.*}}llvm.fake.use
+
+void foo() {
+  {
+    while (1) {
+      int ret;
+      if (1)
+        return;
+    }
+  }
+}

clang/test/CodeGen/fake-use.cpp

+// RUN: %clang_cc1 %s -O2 -emit-llvm -fextend-this-ptr -o - | FileCheck %s
+// Check that we generate a fake_use call with the 'this' pointer as argument.
+// The call should appear after the call to bar().
+
+extern void bar();
+
+class v
+{
+public:
+    int x;
+    int y;
+    int z;
+    int w;
+
+    v(int a, int b, int c, int d) : x(a), y(b), z(c), w(d) {}
+};
+
+class w
+{
+public:
+    v test(int, int, int, int, int, int, int, int, int, int);
+    w(int in): a(in), b(1234) {}
+
+private:
+    int a;
+    int b;
+};
+
+v w::test(int q, int w, int e, int r, int t, int y, int u, int i, int o, int p)
+{
+// CHECK: define{{.*}}test
+    int res = q*w + e - r*t + y*u*i*o*p;
+    int res2 = (w + e + r + t + y + o)*(p*q);
+    int res3 = res + res2;
+    int res4 = q*e + t*y*i + p*e*w * 6 * 4 * 3;
+
+    v V(res, res2, res3, res4);
+
+    bar();
+// CHECK: call{{.*}}bar
+// CHECK: call void (...) @llvm.fake.use(ptr nonnull %this)
+    return V;
+// CHECK: ret
+}

clang/test/CodeGen/no-fake-use-O0.cpp

+// RUN: %clang_cc1 %s -O0 -emit-llvm -fextend-this-ptr -o -  | FileCheck %s \
+// RUN:      --implicit-check-not="call void (...) @llvm.fake.use"
+// RUN: %clang_cc1 %s -O0 -emit-llvm -fextend-lifetimes -o - | FileCheck %s \
+// RUN:      --implicit-check-not="call void (...) @llvm.fake.use"
+// RUN: %clang_cc1 %s -O1 -emit-llvm -fextend-this-ptr -o -  | FileCheck -check-prefix=OPT %s
+// RUN: %clang_cc1 %s -O1 -emit-llvm -fextend-lifetimes -o - | FileCheck -check-prefix=OPT %s
+// RUN: %clang_cc1 %s -Os -emit-llvm -fextend-this-ptr -o -  | FileCheck -check-prefix=OPT %s
+// RUN: %clang_cc1 %s -Os -emit-llvm -fextend-lifetimes -o - | FileCheck -check-prefix=OPT %s
+// RUN: %clang_cc1 %s -Oz -emit-llvm -fextend-this-ptr -o -  | FileCheck -check-prefix=OPT %s
+// RUN: %clang_cc1 %s -Oz -emit-llvm -fextend-lifetimes -o - | FileCheck -check-prefix=OPT %s
+// Check that we do not generate a fake_use call when we are not optimizing. 
+
+extern void bar();
+
+class v
+{
+public:
+    int x;
+    int y;
+    int z;
+    int w;
+
+    v(int a, int b, int c, int d) : x(a), y(b), z(c), w(d) {}
+};
+
+class w
+{
+public:
+    v test(int, int, int, int, int, int, int, int, int, int);
+    w(int in): a(in), b(1234) {}
+
+private:
+    int a;
+    int b;
+};
+
+v w::test(int q, int w, int e, int r, int t, int y, int u, int i, int o, int p)
+{
+// CHECK: define{{.*}}test
+    int res = q*w + e - r*t + y*u*i*o*p;
+    int res2 = (w + e + r + t + y + o)*(p*q);
+    int res3 = res + res2;
+    int res4 = q*e + t*y*i + p*e*w * 6 * 4 * 3;
+
+    v V(res, res2, res3, res4);
+
+    bar();
+// OPT:       call void (...) @llvm.fake.use
+    return V;
+}

clang/test/CodeGen/suppress-post-ra.c

+// RUN: %clang_cc1 %s -S -emit-llvm -O2 -fextend-lifetimes -o - | FileCheck %s
+
+// Emit the function attribute disable-post-ra when
+// -fextend-lifetimes is on.
+
+// CHECK: attributes #0 = {{{.*}}"disable-post-ra"
+
+void foo() {}