Pretend NRVO variables are references so they can be found by debug info

[akhuang]

Summary

Previously variables optimized by NRVO showed up as "optimized away" in CodeView and Dwarf debug info because their debug location was the return register.
This change stores a reference to the NRVO variable and uses that for the debug value.

Related to https://bugs.chromium.org/p/chromium/issues/detail?id=860398#c16

[rnk]

+@aprantl @dblaikie

The basic idea of this change is that clang should store implicit sret pointer arguments into a static alloca, and the dbg.declare should reference the static alloca instead of referring to the LLVM argument directly. The idea is that that will be more resilient. The current consensus seems to be that that's the best way to ensure we have maximally correct debug info at -O0. Let us know if you have any feedback on how to do this.

clang/lib/CodeGen/CGDebugInfo.cpp

3944–3945

I wouldn't mention references here since those are codeview specific. This should instead say something like: "Clang stores the sret pointer provided by the caller in a static alloca. Use DW_OP_deref to tell the debugger to load that pointer and treat it as the address of the variable.".

clang/lib/CodeGen/CGDecl.cpp

1567–1577

I think this alloca is something that we should set up in the prologue. Take a look at CodeGenFunction::StartFunction for where the ReturnValue member variable is initialized. You can create the alloca there, save it, and reuse it here for every NRVO variable, since there could be more than one, and we they can all use the same static alloca.

Initially I thought perhaps we could change the meaning of ReturnValue to simply *be* this alloca, but I don't think it will be so easy. I'd recommend adding a second member next to it that's null unless sret is in use.

Finally, I notice we need to do something if inalloca is involved. (Yuck.) You can see how its handled in StartFunction. You could use the result of the Builder.CreateStructGEP call here as the value to save to pass to dbg.declare when NRVO fires:
https://github.com/llvm/llvm-project/blob/14059d2a13667409ccf2860ff7d1c56f6c8c70a6/clang/lib/CodeGen/CodeGenFunction.cpp#L904

llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp

1145–1154

I see. I thought this code was using the stuff I added in DbgVariableLocation::extractFromMachineInstruction. Hm.

In practice, do you actually see DIExpression offsets here? I think maybe we should just special case a single DW_OP_deref expression, since that's what we get at O0.

[akhuang]

• fix test case

[aprantl]

clang/lib/CodeGen/CGDecl.cpp

1573

We can't change IR generation based on whether debug info is enabled or not. The alloca should be emitted unconditionally.

It's a strong goal of LLVM that enabling/disabling debug info should have no effect on the contents of the .text section.

[akhuang]

Now creates a pointer to the return location in the function prolog, whenever sret is being used.

Also addressed some other comments.

llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp

1145–1154

That should work

[akhuang]

• add semicolon

[rnk]

This needs two unit tests:

• A clang test at clang/test/CodeGenCXX/debug-info-nrvo.cpp similar to other debug-info-* tests there. This test should have a second RUN line and extra checks for -fno-elide-constructors.
• An LLVM test at llvm/test/DebugInfo/COFF/nrvo.ll to show that we handle dbg.declare + deref

Truly, we should also test the inalloca case.

clang/lib/CodeGen/CGDebugInfo.cpp

3947

I think we should check for getLangOpts().ElideConstructors here, and check that the debug info is correct (no deref) in that mode.

clang/lib/CodeGen/CodeGenFunction.cpp

910–911

What I had in mind was to use this GEP as the ReturnValuePointer here. The inalloca parameter is also a pointer to stack memory, and a GEP is an offset, so it should end up being handled like a static alloca.

[aprantl]

Thanks, this addresses my concern!

[akhuang]

clang/lib/CodeGen/CGDebugInfo.cpp

3947

Makes sense, though now I realized there is also an issue because CGDecl checks whether ReturnValuePointer is valid before changing the debug value, but CGDebugInfo doesn't check this when adding DW_OP_deref.. so maybe it makes sense to pass something through to EmitDeclare?

[rnk]

clang/lib/CodeGen/CGDebugInfo.cpp

3947

I guess adding a boolean to EmitDeclareOfAutoVariable would be the way to go, and then to thread that through here.

[akhuang]

• Add clang and llvm tests, and windows debuginfo test
• Use GEP as returnValuePointer in inalloca case
• Add bool parameter for EmitDeclare when pointer is being used

[akhuang]

clang/lib/CodeGen/CodeGenFunction.cpp

910–911

Oh, ok. I changed it, but not sure how to test debug info for the inalloca case

[rnk]

I see we don't have any tests for inalloca to model this on, so I think we should skip that for this change. I'll add one later that handles arguments as well, since those are interesting.

clang/lib/CodeGen/CGDecl.cpp

1564

Nit: Since the boolean is really the condition of the if, I'd suggest writing it like this:

bool UsePointerValue = NRVO && ReturnValuePointer.isValid();
if (UsePointerValue)
  DebugAddr = ReturnValuePointer;
...

It's more functional, less stateful, and is one less line in the end.

clang/lib/CodeGen/CodeGenFunction.cpp

910–911

I don't think this is the correct alignment. The sret slot in the inalloca has a pointer type, so I think you want pointer alignment here. I think you can substitute Int8PtrTy for RetTy here.

[akhuang]

• fix alignment of pointer in inalloca case
• make existing tests stop failing by changing some and adding a check for existing return value alloca (I think?) before adding the ReturnValuePointer alloca

[akhuang]

clang/lib/CodeGen/CodeGenFunction.cpp

910–911

I suppose getPointerAlign() should work?

[rnk]

lgtm, thanks!

clang/lib/CodeGen/CodeGenFunction.cpp

910–911

Looks right to me. :)

clang/lib/CodeGen/CGDebugInfo.h

public:
   /// Emit metadata to indicate the end of a new lexical block and pop
   /// the current block.
   void EmitLexicalBlockEnd(CGBuilderTy &Builder, SourceLocation Loc);
 
   /// Emit call to \c llvm.dbg.declare for an automatic variable
   /// declaration.
   /// Returns a pointer to the DILocalVariable associated with the
   /// llvm.dbg.declare, or nullptr otherwise.
-  llvm::DILocalVariable *EmitDeclareOfAutoVariable(const VarDecl *Decl,
-                                                   llvm::Value *AI,
-                                                   CGBuilderTy &Builder);
+  llvm::DILocalVariable *
+  EmitDeclareOfAutoVariable(const VarDecl *Decl, llvm::Value *AI,
+                            CGBuilderTy &Builder,
+                            const bool UsePointerValue = false);
 
   /// Emit call to \c llvm.dbg.label for an label.
   void EmitLabel(const LabelDecl *D, CGBuilderTy &Builder);
 
   /// Emit call to \c llvm.dbg.declare for an imported variable
   /// declaration in a block.
   void EmitDeclareOfBlockDeclRefVariable(
       const VarDecl *variable, llvm::Value *storage, CGBuilderTy &Builder,
llvm::DIMacroFile *CreateTempMacroFile(llvm::DIMacroFile *Parent,
                                          SourceLocation FileLoc);
 
 private:
   /// Emit call to llvm.dbg.declare for a variable declaration.
   /// Returns a pointer to the DILocalVariable associated with the
   /// llvm.dbg.declare, or nullptr otherwise.
   llvm::DILocalVariable *EmitDeclare(const VarDecl *decl, llvm::Value *AI,
                                      llvm::Optional<unsigned> ArgNo,
-                                     CGBuilderTy &Builder);
+                                     CGBuilderTy &Builder,
+                                     const bool UsePointerValue = false);
 
   struct BlockByRefType {
     /// The wrapper struct used inside the __block_literal struct.
     llvm::DIType *BlockByRefWrapper;
     /// The type as it appears in the source code.
     llvm::DIType *WrappedType;
   };
 

clang/lib/CodeGen/CGDebugInfo.cpp

CGDebugInfo::EmitTypeForVarWithBlocksAttr(const VarDecl *VD,
   return {DBuilder.createStructType(Unit, "", Unit, 0, FieldOffset, 0,
                                     llvm::DINode::FlagZero, nullptr, Elements),
           WrappedTy};
 }
 
 llvm::DILocalVariable *CGDebugInfo::EmitDeclare(const VarDecl *VD,
                                                 llvm::Value *Storage,
                                                 llvm::Optional<unsigned> ArgNo,
-                                                CGBuilderTy &Builder) {
+                                                CGBuilderTy &Builder,
+                                                const bool UsePointerValue) {
   assert(DebugKind >= codegenoptions::LimitedDebugInfo);
   assert(!LexicalBlockStack.empty() && "Region stack mismatch, stack empty!");
   if (VD->hasAttr<NoDebugAttr>())
     return nullptr;
 
   bool Unwritten =
       VD->isImplicit() || (isa<Decl>(VD->getDeclContext()) &&
                            cast<Decl>(VD->getDeclContext())->isImplicit());
if (RD->isUnion() && RD->isAnonymousStructOrUnion()) {
         DBuilder.insertDeclare(
             Storage, D, DBuilder.createExpression(Expr),
             llvm::DebugLoc::get(Line, Column, Scope, CurInlinedAt),
             Builder.GetInsertBlock());
       }
     }
   }
 
+  // Clang stores the sret pointer provided by the caller in a static alloca.
+  // Use DW_OP_deref to tell the debugger to load the pointer and treat it as
+  // the address of the variable.
+  if (UsePointerValue) {
+    assert(std::find(Expr.begin(), Expr.end(), llvm::dwarf::DW_OP_deref) ==
+               Expr.end() &&
+           "Debug info already contains DW_OP_deref.");
+    Expr.push_back(llvm::dwarf::DW_OP_deref);
+  }
+
   // Create the descriptor for the variable.
   auto *D = ArgNo ? DBuilder.createParameterVariable(
                         Scope, Name, *ArgNo, Unit, Line, Ty,
                         CGM.getLangOpts().Optimize, Flags)
                   : DBuilder.createAutoVariable(Scope, Name, Unit, Line, Ty,
                                                 CGM.getLangOpts().Optimize,
                                                 Flags, Align);
 
   // Insert an llvm.dbg.declare into the current block.
   DBuilder.insertDeclare(Storage, D, DBuilder.createExpression(Expr),
                          llvm::DebugLoc::get(Line, Column, Scope, CurInlinedAt),
                          Builder.GetInsertBlock());
 
   return D;
 }
 
 llvm::DILocalVariable *
 CGDebugInfo::EmitDeclareOfAutoVariable(const VarDecl *VD, llvm::Value *Storage,
-                                       CGBuilderTy &Builder) {
+                                       CGBuilderTy &Builder,
+                                       const bool UsePointerValue) {
   assert(DebugKind >= codegenoptions::LimitedDebugInfo);
-  return EmitDeclare(VD, Storage, llvm::None, Builder);
+  return EmitDeclare(VD, Storage, llvm::None, Builder, UsePointerValue);
 }
 
 void CGDebugInfo::EmitLabel(const LabelDecl *D, CGBuilderTy &Builder) {
   assert(DebugKind >= codegenoptions::LimitedDebugInfo);
   assert(!LexicalBlockStack.empty() && "Region stack mismatch, stack empty!");
 
   if (D->hasAttr<NoDebugAttr>())
     return;

clang/lib/CodeGen/CGDecl.cpp

bool EmitDebugInfo = DI && CGM.getCodeGenOpts().getDebugInfo() >=
                                  codegenoptions::LimitedDebugInfo;
 
   Address address = Address::invalid();
   Address AllocaAddr = Address::invalid();
   Address OpenMPLocalAddr =
       getLangOpts().OpenMP
           ? CGM.getOpenMPRuntime().getAddressOfLocalVariable(*this, &D)
           : Address::invalid();
+  bool NRVO = getLangOpts().ElideConstructors && D.isNRVOVariable();
+
   if (getLangOpts().OpenMP && OpenMPLocalAddr.isValid()) {
     address = OpenMPLocalAddr;
   } else if (Ty->isConstantSizeType()) {
-    bool NRVO = getLangOpts().ElideConstructors &&
-      D.isNRVOVariable();
-
     // If this value is an array or struct with a statically determinable
     // constant initializer, there are optimizations we can do.
     //
     // TODO: We should constant-evaluate the initializer of any variable,
     // as long as it is initialized by a constant expression. Currently,
     // isConstantInitializer produces wrong answers for structs with
     // reference or bitfield members, and a few other cases, and checking
     // for POD-ness protects us from some of these.
CodeGenFunction::EmitAutoVarAlloca(const VarDecl &D) {
   }
 
   setAddrOfLocalVar(&D, address);
   emission.Addr = address;
   emission.AllocaAddr = AllocaAddr;
 
   // Emit debug info for local var declaration.
   if (EmitDebugInfo && HaveInsertPoint()) {
+    Address DebugAddr = address;
+    bool UsePointerValue = NRVO && ReturnValuePointer.isValid();
     DI->setLocation(D.getLocation());
-    (void)DI->EmitDeclareOfAutoVariable(&D, address.getPointer(), Builder);
+
+    // If NRVO, use a pointer to the return address.
+    if (UsePointerValue)
+      DebugAddr = ReturnValuePointer;
+
+    (void)DI->EmitDeclareOfAutoVariable(&D, DebugAddr.getPointer(), Builder,
+                                        UsePointerValue);
   }
 
   if (D.hasAttr<AnnotateAttr>() && HaveInsertPoint())
     EmitVarAnnotations(&D, address.getPointer());
 
   // Make sure we call @llvm.lifetime.end.
   if (emission.useLifetimeMarkers())
     EHStack.pushCleanup<CallLifetimeEnd>(NormalEHLifetimeMarker,
                                          emission.getOriginalAllocatedAddress(),
                                          emission.getSizeForLifetimeMarkers());
 
   return emission;
 }

clang/lib/CodeGen/CodeGenFunction.h

public:
 
   /// ReturnBlock - Unified return block.
   JumpDest ReturnBlock;
 
   /// ReturnValue - The temporary alloca to hold the return
   /// value. This is invalid iff the function has no return value.
   Address ReturnValue = Address::invalid();
 
+  /// ReturnValuePointer - The temporary alloca to hold a pointer to sret.
+  /// This is invalid if sret is not in use.
+  Address ReturnValuePointer = Address::invalid();
+
   /// Return true if a label was seen in the current scope.
   bool hasLabelBeenSeenInCurrentScope() const {
     if (CurLexicalScope)
       return CurLexicalScope->hasLabels();
     return !LabelMap.empty();
   }
 
   /// AllocaInsertPoint - This is an instruction in the entry block before which

clang/lib/CodeGen/CodeGenFunction.cpp

if (!endsWithReturn(D))
       ++NumReturnExprs;
   } else if (CurFnInfo->getReturnInfo().getKind() == ABIArgInfo::Indirect) {
     // Indirect return; emit returned value directly into sret slot.
     // This reduces code size, and affects correctness in C++.
     auto AI = CurFn->arg_begin();
     if (CurFnInfo->getReturnInfo().isSRetAfterThis())
       ++AI;
     ReturnValue = Address(&*AI, CurFnInfo->getReturnInfo().getIndirectAlign());
+    if (!CurFnInfo->getReturnInfo().getIndirectByVal()) {
+      ReturnValuePointer = CreateDefaultAlignTempAlloca(Int8PtrTy, "result.ptr");
+      Builder.CreateStore(Builder.CreatePointerBitCastOrAddrSpaceCast(
+                              ReturnValue.getPointer(), Int8PtrTy),
+                          ReturnValuePointer);
+    }
   } else if (CurFnInfo->getReturnInfo().getKind() == ABIArgInfo::InAlloca &&
              !hasScalarEvaluationKind(CurFnInfo->getReturnType())) {
     // Load the sret pointer from the argument struct and return into that.
     unsigned Idx = CurFnInfo->getReturnInfo().getInAllocaFieldIndex();
     llvm::Function::arg_iterator EI = CurFn->arg_end();
     --EI;
     llvm::Value *Addr = Builder.CreateStructGEP(nullptr, &*EI, Idx);
+    ReturnValuePointer = Address(Addr, getPointerAlign());
     Addr = Builder.CreateAlignedLoad(Addr, getPointerAlign(), "agg.result");
     ReturnValue = Address(Addr, getNaturalTypeAlignment(RetTy));
   } else {
     ReturnValue = CreateIRTemp(RetTy, "retval");
 
     // Tell the epilog emitter to autorelease the result.  We do this
     // now so that various specialized functions can suppress it
     // during their IR-generation.

clang/test/CodeGen/arm64-microsoft-arguments.cpp

 S3 f3() {
   S3 x;
   return func3(x);
 }
 
 // Pass and return aggregate (of size < 16 bytes) with non-trivial destructor.
 // Passed directly but returned indirectly.
 // CHECK: define {{.*}} void {{.*}}f4{{.*}}(%struct.S4* inreg noalias sret %agg.result)
-// CHECK: call void {{.*}}func4{{.*}}(%struct.S4* inreg sret %agg.result, [2 x i64] %4)
+// CHECK: call void {{.*}}func4{{.*}}(%struct.S4* inreg sret %agg.result, [2 x i64] %5)
 struct S4 {
   int a[3];
   ~S4();
 };
 
 S4 func4(S4 x);
 S4 f4() {
   S4 x;
   return func4(x);
 }
 
 // Pass and return from instance method called from instance method.
 // CHECK: define {{.*}} void @{{.*}}bar@Q1{{.*}}(%class.Q1* %this, %class.P1* inreg noalias sret %agg.result)
-// CHECK: call void {{.*}}foo@P1{{.*}}(%class.P1* %ref.tmp, %class.P1* inreg sret %agg.result, i8 %0)
+// CHECK: call void {{.*}}foo@P1{{.*}}(%class.P1* %ref.tmp, %class.P1* inreg sret %agg.result, i8 %1)
 
 class P1 {
 public:
   P1 foo(P1 x);
 };
 
 class Q1 {
 public:

clang/test/CodeGenCXX/conditional-gnu-ext.cpp

struct B {
     ~B();
     operator bool();
     operator A();
   };
 
   B test0(B &x) {
     // CHECK-LABEL:    define void @_ZN5test35test0ERNS_1BE(
     // CHECK:      [[X:%.*]] = alloca [[B:%.*]]*,
-    // CHECK-NEXT: store [[B]]* {{%.*}}, [[B]]** [[X]]
+    // CHECK:      store [[B]]* {{%.*}}, [[B]]** [[X]]
     // CHECK-NEXT: [[T0:%.*]] = load [[B]]*, [[B]]** [[X]]
     // CHECK-NEXT: [[BOOL:%.*]] = call zeroext i1 @_ZN5test31BcvbEv([[B]]* [[T0]])
     // CHECK-NEXT: br i1 [[BOOL]]
     // CHECK:      call void @_ZN5test31BC1ERKS0_([[B]]* [[RESULT:%.*]], [[B]]* dereferenceable({{[0-9]+}}) [[T0]])
     // CHECK-NEXT: br label
     // CHECK:      call void @_ZN5test31BC1Ev([[B]]* [[RESULT]])
     // CHECK-NEXT: br label
     // CHECK:      ret void
     return x ?: B();
   }
 
   B test1() {
     // CHECK-LABEL:    define void @_ZN5test35test1Ev(
     // CHECK:      [[TEMP:%.*]] = alloca [[B]],
-    // CHECK-NEXT: call  void @_ZN5test312test1_helperEv([[B]]* sret [[TEMP]])
+    // CHECK:      call  void @_ZN5test312test1_helperEv([[B]]* sret [[TEMP]])
     // CHECK-NEXT: [[BOOL:%.*]] = call zeroext i1 @_ZN5test31BcvbEv([[B]]* [[TEMP]])
     // CHECK-NEXT: br i1 [[BOOL]]
     // CHECK:      call void @_ZN5test31BC1ERKS0_([[B]]* [[RESULT:%.*]], [[B]]* dereferenceable({{[0-9]+}}) [[TEMP]])
     // CHECK-NEXT: br label
     // CHECK:      call void @_ZN5test31BC1Ev([[B]]* [[RESULT]])
     // CHECK-NEXT: br label
     // CHECK:      call void @_ZN5test31BD1Ev([[B]]* [[TEMP]])
     // CHECK-NEXT: ret void
     extern B test1_helper();
     return test1_helper() ?: B();
   }
 
 
   A test2(B &x) {
     // CHECK-LABEL:    define void @_ZN5test35test2ERNS_1BE(
     // CHECK:      [[X:%.*]] = alloca [[B]]*,
-    // CHECK-NEXT: store [[B]]* {{%.*}}, [[B]]** [[X]]
+    // CHECK:      store [[B]]* {{%.*}}, [[B]]** [[X]]
     // CHECK-NEXT: [[T0:%.*]] = load [[B]]*, [[B]]** [[X]]
     // CHECK-NEXT: [[BOOL:%.*]] = call zeroext i1 @_ZN5test31BcvbEv([[B]]* [[T0]])
     // CHECK-NEXT: br i1 [[BOOL]]
     // CHECK:      call void @_ZN5test31BcvNS_1AEEv([[A:%.*]]* sret [[RESULT:%.*]], [[B]]* [[T0]])
     // CHECK-NEXT: br label
     // CHECK:      call void @_ZN5test31AC1Ev([[A]]* [[RESULT]])
     // CHECK-NEXT: br label
     // CHECK:      ret void
     return x ?: A();
   }
 
   A test3() {
     // CHECK-LABEL:    define void @_ZN5test35test3Ev(
     // CHECK:      [[TEMP:%.*]] = alloca [[B]],
-    // CHECK-NEXT: call  void @_ZN5test312test3_helperEv([[B]]* sret [[TEMP]])
+    // CHECK:      call  void @_ZN5test312test3_helperEv([[B]]* sret [[TEMP]])
     // CHECK-NEXT: [[BOOL:%.*]] = call zeroext i1 @_ZN5test31BcvbEv([[B]]* [[TEMP]])
     // CHECK-NEXT: br i1 [[BOOL]]
     // CHECK:      call void @_ZN5test31BcvNS_1AEEv([[A]]* sret [[RESULT:%.*]], [[B]]* [[TEMP]])
     // CHECK-NEXT: br label
     // CHECK:      call void @_ZN5test31AC1Ev([[A]]* [[RESULT]])
     // CHECK-NEXT: br label
     // CHECK:      call void @_ZN5test31BD1Ev([[B]]* [[TEMP]])
     // CHECK-NEXT: ret void

clang/test/CodeGenCXX/debug-info-nrvo.cpp

+// RUN: %clangxx -target x86_64-unknown-unknown -g %s -emit-llvm -S -o - | FileCheck %s
+// RUN: %clangxx -target x86_64-unknown-unknown -g -fno-elide-constructors %s -emit-llvm -S -o - | FileCheck %s -check-prefix=NOELIDE
+struct Foo {
+  Foo() = default;
+  Foo(Foo &&other) { x = other.x; }
+  int x;
+};
+void some_function(int);
+Foo getFoo() {
+  Foo foo;
+  foo.x = 41;
+  some_function(foo.x);
+  return foo;
+}
+
+int main() {
+  Foo bar = getFoo();
+  return bar.x;
+}
+
+// Check that NRVO variables are stored as a pointer with deref if they are
+// stored in the return register.
+
+// CHECK: %result.ptr = alloca i8*, align 8
+// CHECK: call void @llvm.dbg.declare(metadata i8** %result.ptr,
+// CHECK-SAME: metadata !DIExpression(DW_OP_deref)
+// NOELIDE: call void @llvm.dbg.declare(metadata %struct.Foo* %foo,
+// NOELIDE-SAME:                        metadata !DIExpression()

clang/test/CodeGenCXX/lambda-expressions.cpp

 // CHECK: store i32
 // CHECK-NEXT: store i32
 // CHECK-NEXT: load i32, i32*
 // CHECK-NEXT: load i32, i32*
 // CHECK-NEXT: call i32 @"_ZZ1fvENK3$_6clEii"
 // CHECK-NEXT: ret i32
 
 // CHECK-LABEL: define internal void @"_ZZ1hvEN4$_118__invokeEv"(%struct.A* noalias sret %agg.result) {{.*}} {
-// CHECK-NOT: =
 // CHECK: call void @"_ZZ1hvENK4$_11clEv"(%struct.A* sret %agg.result,
 // CHECK-NEXT: ret void
 struct A { ~A(); };
 void h() {
   A (*h)() = [] { return A(); };
 }
 
 // <rdar://problem/12778708>

clang/test/CodeGenObjC/objc-non-trivial-struct-nrvo.m

Trivial testTrivial(void) {
   Trivial a;
   func0(&a);
   return a;
 }
 
 void func1(TrivialBig *);
 
 // CHECK: define void @testTrivialBig(%[[STRUCT_TRIVIALBIG]]* noalias sret %[[AGG_RESULT:.*]])
-// CHECK-NOT: alloca
 // CHECK: call void @func1(%[[STRUCT_TRIVIALBIG]]* %[[AGG_RESULT]])
 // CHECK-NEXT: ret void
 
 TrivialBig testTrivialBig(void) {
   TrivialBig a;
   func1(&a);
   return a;
 }

debuginfo-tests/nrvo-string.cpp

struct string {
   string() {}
   string(int i) : i(i) {}
   ~string() {}
   int i = 0;
 };
 string get_string() {
   string unused;
   string result = 3;
   // DEBUGGER: break 21
   return result;
 }
-int main() { get_string(); }
+void some_function(int) {}
+struct string2 {
+  string2() = default;
+  string2(string2 &&other) { i = other.i; }
+  int i;
+};
+string2 get_string2() {
+  string2 result;
+  result.i = 5;
+  some_function(result.i);
+  // Test that the debugger can get the value of result after another
+  // function is called.
+  // DEBUGGER: break 35
+  return result;
+}
+int main() {
+  get_string();
+  get_string2();
+}
 
 // DEBUGGER: r
 // DEBUGGER: print result.i
 // CHECK:  = 3
+// DEBUGGER: c
+// DEBUGGER: print result.i
+// CHECK:  = 5

debuginfo-tests/win_cdb/nrvo.cpp

+// This ensures that DW_OP_deref is inserted when necessary, such as when NRVO
+// of a string object occurs in C++.
+//
+// RUN: %clang_cl %s -o %t.exe -fuse-ld=lld -Z7
+// RUN: grep DE[B]UGGER: %s | sed -e 's/.*DE[B]UGGER: //' > %t.script
+// RUN: %cdb -cf %t.script %t.exe | FileCheck %s --check-prefixes=DEBUGGER,CHECK
+//
+
+struct string {
+  string() {}
+  string(int i) : i(i) {}
+  ~string() {}
+  int i = 0;
+};
+string get_string() {
+  string unused;
+  string result = 3;
+  __debugbreak();
+  return result;
+}
+void some_function(int) {}
+struct string2 {
+  string2() = default;
+  string2(string2 &&other) { i = other.i; }
+  int i;
+};
+string2 get_string2() {
+  string2 result;
+  result.i = 5;
+  some_function(result.i);
+  // Test that the debugger can get the value of result after another
+  // function is called.
+  __debugbreak();
+  return result;
+}
+int main() {
+  get_string();
+  get_string2();
+}
+
+// DEBUGGER: g
+// DEBUGGER: ?? result
+// CHECK: struct string *
+// CHECK:    +0x000 i : 0n3
+// DEBUGGER: g
+// DEBUGGER: ?? result
+// CHECK: struct string2 *
+// CHECK:    +0x000 i : 0n5
+// DEBUGGER: q

llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp

for (const MachineFunction::VariableDbgInfo &VI : MF.getVariableDbgInfo()) {
 
     // If variable scope is not found then skip this variable.
     if (!Scope)
       continue;
 
     // If the variable has an attached offset expression, extract it.
     // FIXME: Try to handle DW_OP_deref as well.
     int64_t ExprOffset = 0;
-    if (VI.Expr)
-      if (!VI.Expr->extractIfOffset(ExprOffset))
+    bool Deref = false;
+    if (VI.Expr) {
+      // If there is one DW_OP_deref element, use offset of 0 and keep going.
+      if (VI.Expr->getNumElements() == 1 &&
+          VI.Expr->getElement(0) == llvm::dwarf::DW_OP_deref)
+        Deref = true;
+      else if (!VI.Expr->extractIfOffset(ExprOffset))
         continue;
+    }
 
     // Get the frame register used and the offset.
     unsigned FrameReg = 0;
     int FrameOffset = TFI->getFrameIndexReference(*Asm->MF, VI.Slot, FrameReg);
     uint16_t CVReg = TRI->getCodeViewRegNum(FrameReg);
 
     // Calculate the label ranges.
     LocalVarDefRange DefRange =
         createDefRangeMem(CVReg, FrameOffset + ExprOffset);
+
     for (const InsnRange &Range : Scope->getRanges()) {
       const MCSymbol *Begin = getLabelBeforeInsn(Range.first);
       const MCSymbol *End = getLabelAfterInsn(Range.second);
       End = End ? End : Asm->getFunctionEnd();
       DefRange.Ranges.emplace_back(Begin, End);
     }
 
     LocalVariable Var;
     Var.DIVar = VI.Var;
     Var.DefRanges.emplace_back(std::move(DefRange));
+    if (Deref)
+      Var.UseReferenceType = true;
+
     recordLocalVariable(std::move(Var), Scope);
   }
 }
 
 static bool canUseReferenceType(const DbgVariableLocation &Loc) {
   return !Loc.LoadChain.empty() && Loc.LoadChain.back() == 0;
 }
 

llvm/test/DebugInfo/COFF/nrvo.ll

+; RUN: llc < %s | FileCheck %s --check-prefix=ASM
+; RUN: llc < %s -filetype=obj | llvm-readobj - --codeview | FileCheck %s --check-prefix=OBJ
+
+; C++ source to regenerate:
+; struct Foo {
+;   Foo() = default;
+;   Foo(Foo &&other) { x = other.x; }
+;   int x;
+; };
+; void some_function(int);
+; Foo getFoo() {
+;   Foo foo;
+;   foo.x = 41;
+;   some_function(foo.x);
+;   return foo;
+; }
+;
+; int main() {
+;   Foo bar = getFoo();
+;   return bar.x;
+; }
+; $ clang t.cpp -S -emit-llvm -g -o t.ll
+
+; ASM-LABEL:  .long  241                      # Symbol subsection for GetFoo 
+; ASM:        .short 4414                     # Record kind: S_LOCAL
+; ASM-NEXT:   .long 4113                      # TypeIndex
+; ASM-NEXT:   .short 0                        # Flags
+; ASM-NEXT:   .asciz "foo"
+; ASM-NEXT:   .p2align 2
+; ASM-NEXT: .Ltmp
+; ASM:        .cv_def_range  .Ltmp{{.*}} .Ltmp{{.*}}, "B\021(\000\000\000"
+
+; OBJ: Subsection [
+; OBJ:   SubSectionType: Symbols (0xF1)
+; OBJ:   LocalSym {
+; OBJ:     Kind: S_LOCAL (0x113E)
+; OBJ:     Type: Foo& (0x1011)
+; OBJ:     Flags [ (0x0)
+; OBJ:     ]
+; OBJ:     VarName: foo
+; OBJ:   }
+; OBJ:   DefRangeFramePointerRelSym {
+; OBJ:     Kind: S_DEFRANGE_FRAMEPOINTER_REL (0x1142)
+; OBJ:     Offset: 40
+; OBJ:     LocalVariableAddrRange {
+; OBJ:       OffsetStart: .text+0x1D
+; OBJ:       ISectStart: 0x0
+; OBJ:       Range: 0x16
+; OBJ:   }
+
+; ModuleID = 't.cpp'
+source_filename = "t.cpp"
+target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc19.16.27030"
+
+%struct.Foo = type { i32 }
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @"?some_function@@YAXH@Z"(i32) #0 !dbg !8 {
+entry:
+  %.addr = alloca i32, align 4
+  store i32 %0, i32* %.addr, align 4
+  call void @llvm.dbg.declare(metadata i32* %.addr, metadata !12, metadata !DIExpression()), !dbg !13
+  ret void, !dbg !13
+}
+
+; Function Attrs: nounwind readnone speculatable
+declare void @llvm.dbg.declare(metadata, metadata, metadata) #1
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @"?GetFoo@@YA?AUFoo@@XZ"(%struct.Foo* noalias sret %agg.result) #0 !dbg !14 {
+entry:
+  %result.ptr = alloca i8*, align 8
+  %0 = bitcast %struct.Foo* %agg.result to i8*
+  store i8* %0, i8** %result.ptr, align 8
+  call void @llvm.dbg.declare(metadata i8** %result.ptr, metadata !28, metadata !DIExpression(DW_OP_deref)), !dbg !29
+  %x = getelementptr inbounds %struct.Foo, %struct.Foo* %agg.result, i32 0, i32 0, !dbg !30
+  store i32 41, i32* %x, align 4, !dbg !30
+  %x1 = getelementptr inbounds %struct.Foo, %struct.Foo* %agg.result, i32 0, i32 0, !dbg !31
+  %1 = load i32, i32* %x1, align 4, !dbg !31
+  call void @"?some_function@@YAXH@Z"(i32 %1), !dbg !31
+  ret void, !dbg !32
+}
+
+; Function Attrs: noinline norecurse nounwind optnone uwtable
+define dso_local i32 @main() #2 !dbg !33 {
+entry:
+  %retval = alloca i32, align 4
+  %bar = alloca %struct.Foo, align 4
+  store i32 0, i32* %retval, align 4
+  call void @llvm.dbg.declare(metadata %struct.Foo* %bar, metadata !36, metadata !DIExpression()), !dbg !37
+  call void @"?GetFoo@@YA?AUFoo@@XZ"(%struct.Foo* sret %bar), !dbg !37
+  %x = getelementptr inbounds %struct.Foo, %struct.Foo* %bar, i32 0, i32 0, !dbg !38
+  %0 = load i32, i32* %x, align 4, !dbg !38
+  ret i32 %0, !dbg !38
+}
+
+attributes #0 = { noinline nounwind optnone uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #1 = { nounwind readnone speculatable }
+attributes #2 = { noinline norecurse nounwind optnone uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4, !5, !6}
+!llvm.ident = !{!7}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 9.0.0 (https://github.com/llvm/llvm-project.git c19ebebac4bf853e77a69c74abe9f7fce98c1d17)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, nameTableKind: None)
+!1 = !DIFile(filename: "t.cpp", directory: "C:\5Csrc\5Ctesting\5Cnrvo", checksumkind: CSK_MD5, checksum: "52a5a20c02c102dfd255d5615680a8bd")
+!2 = !{}
+!3 = !{i32 2, !"CodeView", i32 1}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = !{i32 1, !"wchar_size", i32 2}
+!6 = !{i32 7, !"PIC Level", i32 2}
+!7 = !{!"clang version 9.0.0 (https://github.com/llvm/llvm-project.git c19ebebac4bf853e77a69c74abe9f7fce98c1d17)"}
+!8 = distinct !DISubprogram(name: "some_function", linkageName: "?some_function@@YAXH@Z", scope: !1, file: !1, line: 13, type: !9, scopeLine: 13, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
+!9 = !DISubroutineType(types: !10)
+!10 = !{null, !11}
+!11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!12 = !DILocalVariable(arg: 1, scope: !8, file: !1, line: 13, type: !11)
+!13 = !DILocation(line: 13, scope: !8)
+!14 = distinct !DISubprogram(name: "GetFoo", linkageName: "?GetFoo@@YA?AUFoo@@XZ", scope: !1, file: !1, line: 15, type: !15, scopeLine: 15, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
+!15 = !DISubroutineType(types: !16)
+!16 = !{!17}
+!17 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "Foo", file: !1, line: 1, size: 32, flags: DIFlagTypePassByReference | DIFlagNonTrivial, elements: !18, identifier: ".?AUFoo@@")
+!18 = !{!19, !20, !24}
+!19 = !DIDerivedType(tag: DW_TAG_member, name: "x", scope: !17, file: !1, line: 4, baseType: !11, size: 32)
+!20 = !DISubprogram(name: "Foo", scope: !17, file: !1, line: 2, type: !21, scopeLine: 2, flags: DIFlagPrototyped, spFlags: 0)
+!21 = !DISubroutineType(types: !22)
+!22 = !{null, !23}
+!23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !17, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
+!24 = !DISubprogram(name: "Foo", scope: !17, file: !1, line: 3, type: !25, scopeLine: 3, flags: DIFlagPrototyped, spFlags: 0)
+!25 = !DISubroutineType(types: !26)
+!26 = !{null, !23, !27}
+!27 = !DIDerivedType(tag: DW_TAG_rvalue_reference_type, baseType: !17, size: 64)
+!28 = !DILocalVariable(name: "foo", scope: !14, file: !1, line: 17, type: !17)
+!29 = !DILocation(line: 17, scope: !14)
+!30 = !DILocation(line: 18, scope: !14)
+!31 = !DILocation(line: 19, scope: !14)
+!32 = !DILocation(line: 21, scope: !14)
+!33 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 23, type: !34, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
+!34 = !DISubroutineType(types: !35)
+!35 = !{!11}
+!36 = !DILocalVariable(name: "bar", scope: !33, file: !1, line: 24, type: !17)
+!37 = !DILocation(line: 24, scope: !33)
+!38 = !DILocation(line: 25, scope: !33)