[CloneFunction][DebugInfo] Clone DISubprogram's local types

[dzhidzhoev]

Summary

When DISubprogram is being cloned, types belonging to its scope
must be cloned too. Otherwise, function-local types of the original
function are used in the cloned one, which may cause conflicts.

It helps to tackle the crash encountered after committing
https://reviews.llvm.org/D144006.

[dblaikie]

Additionally, when tracking function-local types in the DICompileUnit (via retainedTypes or enums fields), it seems reasonable to track the clones as well.

Hmm, when do we have function-local types in the DICompileUnit list? I thought we always put those on the subprogram now?

llvm/lib/Transforms/Utils/CloneFunction.cpp

299–307

maybe turn this into a lambda or helper function - since it's 10 lines that's repeated twice, by the looks of it? (can pass in the list to push back to so one pushes back to the enums, and the other pushes back to the types)

311–318

Maybe this could be streamlined a bit too - maybe doesn't saves lines, but makes it easier to read?

[dzhidzhoev]

In D155818#4529566, @dblaikie wrote:

Additionally, when tracking function-local types in the DICompileUnit (via retainedTypes or enums fields), it seems reasonable to track the clones as well.

Hmm, when do we have function-local types in the DICompileUnit list? I thought we always put those on the subprogram now?

The commit with this change https://reviews.llvm.org/D144006 was reverted because of the crash I'm trying to fix here. Since I will commit this before https://reviews.llvm.org/D144006, if it gets accepted xD, we technically still have function-local types in the DICompileUnit list.

[dblaikie]

In D155818#4531577, @dzhidzhoev wrote:

In D155818#4529566, @dblaikie wrote:

Additionally, when tracking function-local types in the DICompileUnit (via retainedTypes or enums fields), it seems reasonable to track the clones as well.

Hmm, when do we have function-local types in the DICompileUnit list? I thought we always put those on the subprogram now?

The commit with this change https://reviews.llvm.org/D144006 was reverted because of the crash I'm trying to fix here. Since I will commit this before https://reviews.llvm.org/D144006, if it gets accepted xD, we technically still have function-local types in the DICompileUnit list.

So this code would be removed once D144006 goes in? Perhaps we could review them separately but assuming they go in together (they could be squashed together before committing) if it's causing this much extra code that'd only exist for 1-2 commits anyway?

[dzhidzhoev]

Addressed @dblaikie comments. This PR is supposed to be commited
along with https://reviews.llvm.org/D144006.

[dzhidzhoev]

Throw away excessive dump() calls from test.

[dblaikie]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

I guess this comes up, though I'd love it if it doesn't...

When does this situation arise? I would've hoped that function-local types wouldn't appear in the CU's retained types list. It'd simplify the model if they were only retained by either the CU or the subprogram, and not both.

[dzhidzhoev]

This comment was removed by dzhidzhoev.

[dzhidzhoev]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

enum my_enum { AA, AB, AC } e;
int main() {
  enum my_local_enum { A, B, C };
  return 0;
}

This example can be compiled with the flag -fno-eliminate-unused-debug-types :

bin/clang local-retained.c -c -fno-eliminate-unused-debug-types -O0 -g -emit-llvm

That's what metadata looks like:

!2 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, retainedTypes: !11 ...
!5 = !DICompositeType(tag: DW_TAG_enumeration_type, name: "my_enum", ...
!11 = !{!5, !12}
!12 = !DICompositeType(tag: DW_TAG_enumeration_type, name: "my_local_enum", scope: !13, ...
!13 = distinct !DISubprogram(name: "main", retainedNodes: !17, ...
!17 = !{!12}

Here my_local_enum, which is local to the subprogram, is referenced by both CU and Subprogram, via retainedTypes and retainedNodes correspondingly. I haven't encountered such a situation in real-world metadata since I haven't got much experience dealing with DebugInfo.

[dzhidzhoev]

Rebased.

[dexonsmith]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

I agree with @dblaikie that local types seem like a property of subprograms.

To replicate the current IRGen retention policy, we could make -fno-eliminate-unused-debug-types retain any subprogram that has a local type (dropping the explicit reference to the local types from compile-unit) and get the same effect.

But maybe -fno-eliminate-unused-debug-types is overreaching here? It's not obvious to me that it implies retaining function-local types. Feels like -fno-eliminate-unused-debug-subprograms (or similar) would be a better fit for that behaviour.

Either way, then we'd end up with a clean model here and this could wouldn't be necessary.

[dblaikie]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

Yeah - @nickdesaulniers do you care about local types for optimized-out functions in -fno-eliminate-unused-debug-types? Looks like GCC is maybe slightly inconsistent. For a totally uncalled inline function with a local type, it doesn't generate any DWARF, but if it's called and then optimized away, it still manages to emit a DW_TAG_inlined_subroutine (though it seems to do this even without the local type - so the local type isn't what's causing the inlined subroutine description to be emitted)

I think it's fine to, yeah, change -fno-eliminate-unused-debug-types to attach local types to the subprogram - and if at some point we improve inlining (I'm not sure it's an improvement to include a DW_TAG_inlined_subroutine that describes a zero length range of instructions, though... ) then we'll keep these types too.

[nickdesaulniers]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

do you care about local types for optimized-out functions in -fno-eliminate-unused-debug-types?

I assume the answer is "yes" but let me check with the consumers of this data (in Android in the UK).

[nickdesaulniers]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

Direct feedback:

I'm not sure what GCC's intent is. When I experimented with this (and the vaguely related -fstandalone-debug) it looked like that instead of doing some reachability analysis it was just emitting everything.
In general we are completely uninterested in types that are defined wholly within functions (at any level of nesting if allowing nested functions) as the language semantics imply they cannot leak out.
There is the edge case of types defined as part of function prototypes. Whatever their actual usefulness, it makes sense for them to be considered "used" if the function itself is.
The only bug we've seen with Clang is when there is a DIE outside a function referring to a type defined in a function.
From the perspective of knowing "all types', I'd say it makes sense to emit them, if there's a DWARF context that is meaningful for them (like an unused out of line function definition or something).
From ABI perspective, I don't think we care.

[dexonsmith]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

@aprantl, do your users use -fno-eliminate-unused-debug-types? If so, is the proposed change fine for them too?

[aprantl]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

On Darwin, the flag is exposed via the -gfull option, however, I am not aware that this feature is being (widely) used at all.

[dzhidzhoev]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

I think it's fine to, yeah, change -fno-eliminate-unused-debug-types to attach local types to the subprogram - and if at some point we improve inlining (I'm not sure it's an improvement to include a DW_TAG_inlined_subroutine that describes a zero length range of instructions, though... ) then we'll keep these types too.

Are there any obstacles to committing that change together with https://reviews.llvm.org/D144006? Since I'm not sure about the consistency of changing this behavior while having other function-local types tracked by DICompileUnit's enum field.

[dzhidzhoev]

llvm/lib/Transforms/Utils/CloneFunction.cpp

281–301

I think it's fine to, yeah, change -fno-eliminate-unused-debug-types to attach local types to the subprogram - and if at some point we improve inlining (I'm not sure it's an improvement to include a DW_TAG_inlined_subroutine that describes a zero length range of instructions, though... ) then we'll keep these types too.

Is it meant to look like this https://reviews.llvm.org/D158730?

[dzhidzhoev]

Friendly ping

[dzhidzhoev]

Rebased & dropped redundant code for cloning local types referenced by DICompileUnit.

[dzhidzhoev]

Gentle ping

[dblaikie]

Is this comment in the patch description (and any associated actual functionality in the patch) still relevant, now we're planning to/have moved local types out of the retained types list?

Additionally, when tracking function-local types in the DICompileUnit (via retainedTypes fields), it seems reasonable to track the clones as well.

[dzhidzhoev]

In D155818#4647515, @dblaikie wrote:

Is this comment in the patch description (and any associated actual functionality in the patch) still relevant, now we're planning to/have moved local types out of the retained types list?

Additionally, when tracking function-local types in the DICompileUnit (via retainedTypes fields), it seems reasonable to track the clones as well.

Thanks for the catch! No, it's not relevant anymore. The code processing retainedTypes of DICompileUnit has been dropped.

[dzhidzhoev]

In D155818#4647515, @dblaikie wrote:

Is this comment in the patch description (and any associated actual functionality in the patch) still relevant, now we're planning to/have moved local types out of the retained types list?

Additionally, when tracking function-local types in the DICompileUnit (via retainedTypes fields), it seems reasonable to track the clones as well.

Should this patch be reviewed for other potential issues? Because https://reviews.llvm.org/D158730 can't be merged without this one.

[dexonsmith]

LGTM.

llvm/include/llvm/IR/DebugInfo.h

public:
 
   /// Clear all lists.
   void reset();
 
 private:
   void processCompileUnit(DICompileUnit *CU);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
+  void processLocalVariable(DILocalVariable *DV);
   bool addCompileUnit(DICompileUnit *CU);
   bool addGlobalVariable(DIGlobalVariableExpression *DIG);
   bool addScope(DIScope *Scope);
   bool addSubprogram(DISubprogram *SP);
   bool addType(DIType *DT);
 
 public:
   using compile_unit_iterator =

llvm/lib/IR/DebugInfo.cpp

void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   processType(SP->getType());
   for (auto *Element : SP->getTemplateParams()) {
     if (auto *TType = dyn_cast<DITemplateTypeParameter>(Element)) {
       processType(TType->getType());
     } else if (auto *TVal = dyn_cast<DITemplateValueParameter>(Element)) {
       processType(TVal->getType());
     }
   }
+
+  for (auto *N : SP->getRetainedNodes()) {
+    if (auto *Var = dyn_cast<DILocalVariable>(N)) {
+      processLocalVariable(Var);
+    }
+  }
 }
 
 void DebugInfoFinder::processVariable(const Module &M,
                                       const DbgVariableIntrinsic &DVI) {
   auto *N = dyn_cast<MDNode>(DVI.getVariable());
   if (!N)
     return;
 
-  auto *DV = dyn_cast<DILocalVariable>(N);
+  processLocalVariable(dyn_cast<DILocalVariable>(N));
+}
+
+void DebugInfoFinder::processLocalVariable(DILocalVariable *DV) {
   if (!DV)
     return;
 
   if (!NodesSeen.insert(DV).second)
     return;
   processScope(DV->getScope());
   processType(DV->getType());
 }

llvm/lib/Transforms/Utils/CloneFunction.cpp

for (DIScope *S : DIFinder->scopes()) {
       if (LScope && MappedToSelfSPs.count(LScope->getSubprogram()))
         mapToSelfIfNew(S);
     }
 
     for (DICompileUnit *CU : DIFinder->compile_units())
       mapToSelfIfNew(CU);
 
     for (DIType *Type : DIFinder->types())
+      // Don't skip subprogram's local types.
+      if (!isa_and_present<DILocalScope>(Type->getScope()) ||
+          SPClonedWithinModule == nullptr ||
+          dyn_cast<DILocalScope>(Type->getScope())->getSubprogram() !=
+              SPClonedWithinModule)
         mapToSelfIfNew(Type);
   } else {
     assert(!SPClonedWithinModule &&
            "Subprogram should be in DIFinder->subprogram_count()...");
   }
 
   const auto RemapFlag = ModuleLevelChanges ? RF_None : RF_NoModuleLevelChanges;
   // Duplicate the metadata that is attached to the cloned function.
   // Subprograms/CUs/types that were already mapped to themselves won't be
for (Function::iterator
            BE = NewFunc->end();
        BB != BE; ++BB)
     // Loop over all instructions, fixing each one as we find it...
     for (Instruction &II : *BB)
       RemapInstruction(&II, VMap, RemapFlag, TypeMapper, Materializer);
 
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.
   if (Changes != CloneFunctionChangeType::DifferentModule)
     return;
 
   // Update !llvm.dbg.cu with compile units added to the new module if this
   // function is being cloned in isolation.
   //
   // FIXME: This is making global / module-level changes, which doesn't seem
   // like the right encapsulation  Consider dropping the requirement to update
   // !llvm.dbg.cu (either obsoleting the node, or restricting it to
   // non-discardable compile units) instead of discovering compile units by
   // visiting the metadata attached to global values, which would allow this
   // code to be deleted. Alternatively, perhaps give responsibility for this
   // update to CloneFunctionInto's callers.
   auto *NewModule = NewFunc->getParent();
   auto *NMD = NewModule->getOrInsertNamedMetadata("llvm.dbg.cu");
   // Avoid multiple insertions of the same DICompileUnit to NMD.
   SmallPtrSet<const void *, 8> Visited;
   for (auto *Operand : NMD->operands())
     Visited.insert(Operand);
   for (auto *Unit : DIFinder->compile_units()) {
     MDNode *MappedUnit =
         MapMetadata(Unit, VMap, RF_None, TypeMapper, Materializer);
     if (Visited.insert(MappedUnit).second)
       NMD->addOperand(MappedUnit);
   }
 }
 
 /// Return a copy of the specified function and add it to that function's
 /// module.  Also, any references specified in the VMap are changed to refer to
 /// their mapped value instead of the original one.  If any of the arguments to
 /// the function are in the VMap, the arguments are deleted from the resultant
 /// function.  The VMap is updated to include mappings from all of the
 /// instructions and basicblocks in the function from their old to new values.
 ///
 Function *llvm::CloneFunction(Function *F, ValueToValueMapTy &VMap,
                               ClonedCodeInfo *CodeInfo) {
   std::vector<Type *> ArgTypes;
 
   // The user might be deleting arguments to the function by specifying them in
   // the VMap.  If so, we need to not add the arguments to the arg ty vector
   //
   for (const Argument &I : F->args())
     if (VMap.count(&I) == 0) // Haven't mapped the argument to anything yet?
       ArgTypes.push_back(I.getType());
 

llvm/unittests/Transforms/Utils/CloningTest.cpp

TEST(CloneFunction, CloneFunctionWithSubprograms) {
   CloneFunctionInto(NewFunc, OldFunc, VMap,
                     CloneFunctionChangeType::GlobalChanges, Returns, "", &CCI);
 
   // This fails if the scopes in the llvm.dbg.declare variable and location
   // aren't the same.
   EXPECT_FALSE(verifyModule(*ImplModule, &errs()));
 }
 
+TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
+  StringRef ImplAssembly = R"(
+    declare void @llvm.dbg.declare(metadata, metadata, metadata)
+
+    define void @test() !dbg !3 {
+      call void @llvm.dbg.declare(metadata i8* undef, metadata !5, metadata !DIExpression()), !dbg !7
+      ret void
+    }
+
+    declare void @cloned()
+
+    !llvm.dbg.cu = !{!0}
+    !llvm.module.flags = !{!2}
+    !0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, enums: !{!14})
+    !1 = !DIFile(filename: "test.cpp",  directory: "")
+    !2 = !{i32 1, !"Debug Info Version", i32 3}
+    !3 = distinct !DISubprogram(name: "test", scope: !1, unit: !0, retainedNodes: !9)
+    !4 = distinct !DISubprogram(name: "inlined", scope: !1, unit: !0, retainedNodes: !{!5})
+    !5 = !DILocalVariable(name: "awaitables", scope: !4, type: !23)
+    !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
+    !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
+    !8 = !DILocation(line: 10, scope: !3)
+    !9 = !{!15, !17, !18}
+    !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
+    !15 = !DILocalVariable(name: "a", scope: !3)
+    !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
+    !17 = !DIImportedEntity(tag: DW_TAG_imported_declaration, name: "imported_l", file: !1, line: 14, scope: !3, entity: !16)
+    !18 = !DILabel(scope: !3, name: "l", file: !1, line: 22)
+    !22 = !DIBasicType(name: "real", size: 32, align: 32, encoding: DW_ATE_float)
+    !23 = !DIDerivedType(name: "local_float", tag: DW_TAG_const_type, baseType: !22, scope: !3)
+    !float_type = !{!23}
+  )";
+
+  LLVMContext Context;
+  SMDiagnostic Error;
+
+  auto ImplModule = parseAssemblyString(ImplAssembly, Error, Context);
+  llvm::errs() << Error.getMessage() << Error.getLineNo() << "\n";
+  EXPECT_TRUE(ImplModule != nullptr);
+  auto *Func = ImplModule->getFunction("test");
+  EXPECT_TRUE(Func != nullptr);
+  auto *ClonedFunc = ImplModule->getFunction("cloned");
+  EXPECT_TRUE(ClonedFunc != nullptr);
+
+  EXPECT_FALSE(verifyModule(*ImplModule, &errs()));
+
+  ValueToValueMapTy VMap;
+  SmallVector<ReturnInst *, 8> Returns;
+  ClonedCodeInfo CCI;
+  CloneFunctionInto(ClonedFunc, Func, VMap,
+                    CloneFunctionChangeType::GlobalChanges, Returns, "", &CCI);
+
+  EXPECT_FALSE(verifyModule(*ImplModule, &errs()));
+
+  // Check that retained and local types are copied.
+  DISubprogram *FuncSP = Func->getSubprogram();
+  DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
+  EXPECT_NE(FuncSP, nullptr);
+  EXPECT_NE(ClonedSP, nullptr);
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 3u);
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+            ClonedSP->getRetainedNodes().size());
+  for (unsigned I = 0; I < FuncSP->getRetainedNodes().size(); ++I) {
+    auto *Node = FuncSP->getRetainedNodes()[I];
+    auto *Copy = ClonedSP->getRetainedNodes()[I];
+
+    // Check that the order of retainedNodes is preserved by
+    // checking that the corresponding node has the same name.
+    if (auto *Var = dyn_cast<DILocalVariable>(Node)) {
+      auto *VarCopy = dyn_cast<DILocalVariable>(Copy);
+      EXPECT_NE(VarCopy, nullptr);
+      EXPECT_EQ(Var->getName(), VarCopy->getName());
+    } else if (auto *Label = dyn_cast<DILabel>(Node)) {
+      auto *LabelCopy = dyn_cast<DILabel>(Copy);
+      EXPECT_NE(LabelCopy, nullptr);
+      EXPECT_EQ(Label->getName(), LabelCopy->getName());
+    } else if (auto *IE = dyn_cast<DIImportedEntity>(Node)) {
+      auto *IECopy = dyn_cast<DIImportedEntity>(Copy);
+      EXPECT_NE(IECopy, nullptr);
+      EXPECT_EQ(IE->getName(), IECopy->getName());
+    }
+
+    // Check that node was copied
+    EXPECT_NE(Node, Copy);
+  }
+
+  auto *FloatType = dyn_cast<DIType>(
+      ImplModule->getNamedMetadata("float_type")->getOperand(0));
+  EXPECT_EQ(FloatType->getName(), "local_float");
+  EXPECT_TRUE(VMap.MD().contains(FloatType));
+  EXPECT_NE(FloatType, VMap.MD()[FloatType]);
+}
+
 TEST(CloneFunction, CloneFunctionWithInlinedSubprograms) {
   StringRef ImplAssembly = R"(
     declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
     define void @test() !dbg !3 {
       call void @llvm.dbg.declare(metadata i8* undef, metadata !5, metadata !DIExpression()), !dbg !7
       ret void
     }