This is an archive of the discontinued LLVM Phabricator instance.

Teach ValueMapper to use ODR uniqued types when available
ClosedPublic

Authored by tejohnson on Jan 2 2018, 10:35 AM.

Details

Summary

This is exposed during ThinLTO compilation, when we import an alias by
creating a clone of the aliasee. Without this fix the debug type is
unnecessarily cloned and we get a duplicate, undoing the uniquing.

Fixes PR36089.

Diff Detail

Repository
rL LLVM

Event Timeline

tejohnson created this revision.Jan 2 2018, 10:35 AM
pcc added a comment.Jan 19 2018, 11:14 AM

Should we instead change the ValueMapper to call DICompositeType::buildODRType if ODR uniquing is enabled?

In D41669#982160, @pcc wrote:

Should we instead change the ValueMapper to call DICompositeType::buildODRType if ODR uniquing is enabled?

We could. Confirmed change works instead of the other approach here (same test case). Let me know if you prefer that and I will change this patch.

diff --git a/lib/Transforms/Utils/ValueMapper.cpp b/lib/Transforms/Utils/ValueMapper.cpp
index 8c9ecbc3503..fc36501b145 100644

  • a/lib/Transforms/Utils/ValueMapper.cpp

+++ b/lib/Transforms/Utils/ValueMapper.cpp
@@ -25,6 +25,7 @@
#include "llvm/IR/CallSite.h"
#include "llvm/IR/Constant.h"
#include "llvm/IR/Constants.h"
+#include "llvm/IR/DebugInfoMetadata.h"
#include "llvm/IR/DerivedTypes.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/GlobalAlias.h"
@@ -536,13 +537,30 @@ Optional<Metadata *> MDNodeMapper::tryToMapOperand(const Metadata *Op) {

return None;

}

+static Metadata *cloneOrBuildODR(const MDNode &N) {
+ auto *CT = dyn_cast_or_null<DICompositeType>(&N);
+ if (CT && CT->getIdentifier() != "") {
+ auto *NewCT = DICompositeType::buildODRType(
+ CT->getContext(),
+ *getCanonicalMDString(CT->getContext(), CT->getIdentifier()),
+ CT->getTag(), CT->getRawName(), CT->getFile(), CT->getLine(),
+ CT->getScope(), CT->getBaseType(), CT->getSizeInBits(),
+ CT->getAlignInBits(), CT->getOffsetInBits(), CT->getFlags(),
+ CT->getRawElements(), CT->getRuntimeLang(), CT->getVTableHolder(),
+ CT->getRawTemplateParams());
+ if (NewCT)
+ return NewCT;
+ }
+ return MDNode::replaceWithDistinct(N.clone());
+};
+
MDNode *MDNodeMapper::mapDistinctNode(const MDNode &N) {

assert(N.isDistinct() && "Expected a distinct node");
assert(!M.getVM().getMappedMD(&N) && "Expected an unmapped node");
  • DistinctWorklist.push_back(cast<MDNode>(
  • (M.Flags & RF_MoveDistinctMDs)
  • ? M.mapToSelf(&N)
  • : M.mapToMetadata(&N, MDNode::replaceWithDistinct(N.clone()))));

+ DistinctWorklist.push_back(
+ cast<MDNode>((M.Flags & RF_MoveDistinctMDs)
+ ? M.mapToSelf(&N)
+ : M.mapToMetadata(&N, cloneOrBuildODR(N))));

return DistinctWorklist.back();

}

Ping. It turns out this fixes https://bugs.llvm.org/show_bug.cgi?id=36089 (I tested the new version that uses the ValueMapper). @pcc: let me know if you prefer the ValueMapper approach (see my previous comments), and I will update the patch officially.

pcc added a comment.Jan 30 2018, 10:08 AM

Sorry the value mapper version mostly looks good. One question.

if (CT && CT->getIdentifier() != "") {
+ auto *NewCT = DICompositeType::buildODRType

Can this be if (enableodruniquing() && CT && ...) Return CT;?

In D41669#992020, @pcc wrote:

Sorry the value mapper version mostly looks good. One question.

if (CT && CT->getIdentifier() != "") {
+ auto *NewCT = DICompositeType::buildODRType

Can this be if (enableodruniquing() && CT && ...) Return CT;?

Not sure what you mean - we don't want to return the source's CT. (And the first thing buildODRType does is check if ODR uniquing is enabled, so no need to check that here too).

I'll go ahead and update the patch with the new version.

pcc added a comment.Jan 30 2018, 10:15 AM

But doesn't ODR type uniquing mean that there is only one CT for the whole LLVMContext?

In D41669#992036, @pcc wrote:

But doesn't ODR type uniquing mean that there is only one CT for the whole LLVMContext?

That's true, with ODR type uniquing we would have invoked buildODRType during the bitcode reading process, so it should be safe to use CT. I tested that with the test case here and in PR36089 and it works. Updating the patch in a min.

Update per comments

tejohnson retitled this revision from Use ODR debug type uniquing when enabled during function cloning to Teach ValueMapper to use ODR uniqued types when available.Jan 30 2018, 10:43 AM
tejohnson edited the summary of this revision. (Show Details)
pcc accepted this revision.Jan 30 2018, 11:13 AM

LGTM with nits

lib/Transforms/Utils/ValueMapper.cpp
541 ↗(On Diff #132004)

I think this can be dyn_cast given that it is a reference.

548 ↗(On Diff #132004)

Remove semicolon

This revision is now accepted and ready to land.Jan 30 2018, 11:13 AM

address comments

tejohnson marked 2 inline comments as done.Jan 30 2018, 11:57 AM
This revision was automatically updated to reflect the committed changes.

This is unsound, as it breaks CloneModule: it causes the original/cloned module to reference GlobalValues from the new module (composite types transitively reference global values when a type has "interesting" template parameters). See https://bugs.llvm.org/show_bug.cgi?id=48841 (especially https://bugs.llvm.org/show_bug.cgi?id=48841#c22, where I have a long explanation).

@tejohnson, can you share more context about why this was necessary? How is ThinLTO using the value mapper that this happens?

There should be another way to get this optimization that's sound, but I think it'll involve priming the ValueToValueMap that gets sent into MapValue / MapMetadata.

Herald added a project: Restricted Project. · View Herald TranscriptFeb 12 2021, 4:25 PM

This is unsound, as it breaks CloneModule: it causes the original/cloned module to reference GlobalValues from the new module (composite types transitively reference global values when a type has "interesting" template parameters). See https://bugs.llvm.org/show_bug.cgi?id=48841 (especially https://bugs.llvm.org/show_bug.cgi?id=48841#c22, where I have a long explanation).

@tejohnson, can you share more context about why this was necessary? How is ThinLTO using the value mapper that this happens?

There should be another way to get this optimization that's sound, but I think it'll involve priming the ValueToValueMap that gets sent into MapValue / MapMetadata.

I need to go back and look at this some more, unfortunately it's not immediately obvious. It looks like I was making this change for unspecified reasons related to my change to import aliases as a copy of the aliasee, and then found it fixed PR36089. Need to go back and see if the test case in that bug works now or still fails without this change.

Regarding CloneModule - from my skim of the other bug you are working on it appears that this is causing an issue when invoked while cloning a module when we split ThinLTO bitcode files - is that correct? In my case I was fixing an issue during importing, not during this cloning.

This is unsound, as it breaks CloneModule: it causes the original/cloned module to reference GlobalValues from the new module (composite types transitively reference global values when a type has "interesting" template parameters). See https://bugs.llvm.org/show_bug.cgi?id=48841 (especially https://bugs.llvm.org/show_bug.cgi?id=48841#c22, where I have a long explanation).

@tejohnson, can you share more context about why this was necessary? How is ThinLTO using the value mapper that this happens?

There should be another way to get this optimization that's sound, but I think it'll involve priming the ValueToValueMap that gets sent into MapValue / MapMetadata.

I need to go back and look at this some more, unfortunately it's not immediately obvious. It looks like I was making this change for unspecified reasons related to my change to import aliases as a copy of the aliasee, and then found it fixed PR36089. Need to go back and see if the test case in that bug works now or still fails without this change.

I'd bet the testcase in that bug will still fail without this change.

Instead of changing ValueMapper, maybe the right fix is similar to the code in CloneFunctionInto (which I'm conditionalizing in https://reviews.llvm.org/D96531, expecting to commit soon, just missed the window last week where I could watch the bots): prime the ValueToValueMap with mappings-to-self for any entry points to the metadata graph that shouldn't be cloned / duplicated.

Regarding CloneModule - from my skim of the other bug you are working on it appears that this is causing an issue when invoked while cloning a module when we split ThinLTO bitcode files - is that correct? In my case I was fixing an issue during importing, not during this cloning.

Yes, that's correct. Thanks for clarifying this is during importing; it's reasonable at a high level to reuse the ODR logic; the problem is that the value mapper code has more general usecases. (Seems like metadata ownership (especially for distinct nodes) needs to be cleaned up in some way... there are a bunch of subtleties that make it hard to reason about.)

Is the source module being discarded? If so, the importer could safely use RF_ReuseAndMutateDistinctMDs (the new name for RF_MoveDistinctMDs).

This is unsound, as it breaks CloneModule: it causes the original/cloned module to reference GlobalValues from the new module (composite types transitively reference global values when a type has "interesting" template parameters). See https://bugs.llvm.org/show_bug.cgi?id=48841 (especially https://bugs.llvm.org/show_bug.cgi?id=48841#c22, where I have a long explanation).

@tejohnson, can you share more context about why this was necessary? How is ThinLTO using the value mapper that this happens?

There should be another way to get this optimization that's sound, but I think it'll involve priming the ValueToValueMap that gets sent into MapValue / MapMetadata.

I need to go back and look at this some more, unfortunately it's not immediately obvious. It looks like I was making this change for unspecified reasons related to my change to import aliases as a copy of the aliasee, and then found it fixed PR36089. Need to go back and see if the test case in that bug works now or still fails without this change.

I'd bet the testcase in that bug will still fail without this change.

Actually - I just tried it and it passes with this change reverted. I guess something else already fixed that issue in a slightly different probably more correct way?

Instead of changing ValueMapper, maybe the right fix is similar to the code in CloneFunctionInto (which I'm conditionalizing in https://reviews.llvm.org/D96531, expecting to commit soon, just missed the window last week where I could watch the bots): prime the ValueToValueMap with mappings-to-self for any entry points to the metadata graph that shouldn't be cloned / duplicated.

Regarding CloneModule - from my skim of the other bug you are working on it appears that this is causing an issue when invoked while cloning a module when we split ThinLTO bitcode files - is that correct? In my case I was fixing an issue during importing, not during this cloning.

Yes, that's correct. Thanks for clarifying this is during importing; it's reasonable at a high level to reuse the ODR logic; the problem is that the value mapper code has more general usecases. (Seems like metadata ownership (especially for distinct nodes) needs to be cleaned up in some way... there are a bunch of subtleties that make it hard to reason about.)

Is the source module being discarded? If so, the importer could safely use RF_ReuseAndMutateDistinctMDs (the new name for RF_MoveDistinctMDs).

Yes it's being discarded. And yes it is already using RF_ReuseAndMutateDistinctMDs (set by default on the IRLinker constructor initialization of the ValueMapper). Hmm, now I'm confused. The case I modified in this patch is where this flag is not set. Aha - going through the debugger for the test case included with this patch shows that while the importer itself uses a ValueMapper with RF_ReuseAndMutateDistinctMDs enabled, the functionality that imports an alias as a copy of the aliasee (replaceAliasWithAliaseee) in fact uses CloneFunction/CloneFunctionInto, so a ValueMapper without that flag. So perhaps your fix in D96531 is sufficient and this change can be reverted?

dexonsmith added a comment.EditedFeb 15 2021, 2:25 PM

This is unsound, as it breaks CloneModule: it causes the original/cloned module to reference GlobalValues from the new module (composite types transitively reference global values when a type has "interesting" template parameters). See https://bugs.llvm.org/show_bug.cgi?id=48841 (especially https://bugs.llvm.org/show_bug.cgi?id=48841#c22, where I have a long explanation).

@tejohnson, can you share more context about why this was necessary? How is ThinLTO using the value mapper that this happens?

There should be another way to get this optimization that's sound, but I think it'll involve priming the ValueToValueMap that gets sent into MapValue / MapMetadata.

I need to go back and look at this some more, unfortunately it's not immediately obvious. It looks like I was making this change for unspecified reasons related to my change to import aliases as a copy of the aliasee, and then found it fixed PR36089. Need to go back and see if the test case in that bug works now or still fails without this change.

I'd bet the testcase in that bug will still fail without this change.

Actually - I just tried it and it passes with this change reverted. I guess something else already fixed that issue in a slightly different probably more correct way?

I'm seeing ThinLTO/X86/dicompositetype-unique-alias.ll fail locally. Maybe you didn't have 22a52dfddcefad4f275eb8ad1cc0e200074c2d8a when you tried? (I probably hadn't pushed yet...)

Instead of changing ValueMapper, maybe the right fix is similar to the code in CloneFunctionInto (which I'm conditionalizing in https://reviews.llvm.org/D96531, expecting to commit soon, just missed the window last week where I could watch the bots): prime the ValueToValueMap with mappings-to-self for any entry points to the metadata graph that shouldn't be cloned / duplicated.

Regarding CloneModule - from my skim of the other bug you are working on it appears that this is causing an issue when invoked while cloning a module when we split ThinLTO bitcode files - is that correct? In my case I was fixing an issue during importing, not during this cloning.

Yes, that's correct. Thanks for clarifying this is during importing; it's reasonable at a high level to reuse the ODR logic; the problem is that the value mapper code has more general usecases. (Seems like metadata ownership (especially for distinct nodes) needs to be cleaned up in some way... there are a bunch of subtleties that make it hard to reason about.)

Is the source module being discarded? If so, the importer could safely use RF_ReuseAndMutateDistinctMDs (the new name for RF_MoveDistinctMDs).

Yes it's being discarded. And yes it is already using RF_ReuseAndMutateDistinctMDs (set by default on the IRLinker constructor initialization of the ValueMapper). Hmm, now I'm confused. The case I modified in this patch is where this flag is not set. Aha - going through the debugger for the test case included with this patch shows that while the importer itself uses a ValueMapper with RF_ReuseAndMutateDistinctMDs enabled, the functionality that imports an alias as a copy of the aliasee (replaceAliasWithAliaseee) in fact uses CloneFunction/CloneFunctionInto, so a ValueMapper without that flag. So perhaps your fix in D96531 is sufficient and this change can be reverted?

Your logic SGTM -- replaceAliasWithAliasee DOES use CloneFunction -- but as mentioned above the test is failing for me. I'm confused... maybe my tree is out-of-date, or maybe there's something subtle going on... I'll reply again if I figure it out.

This is unsound, as it breaks CloneModule: it causes the original/cloned module to reference GlobalValues from the new module (composite types transitively reference global values when a type has "interesting" template parameters). See https://bugs.llvm.org/show_bug.cgi?id=48841 (especially https://bugs.llvm.org/show_bug.cgi?id=48841#c22, where I have a long explanation).

@tejohnson, can you share more context about why this was necessary? How is ThinLTO using the value mapper that this happens?

There should be another way to get this optimization that's sound, but I think it'll involve priming the ValueToValueMap that gets sent into MapValue / MapMetadata.

I need to go back and look at this some more, unfortunately it's not immediately obvious. It looks like I was making this change for unspecified reasons related to my change to import aliases as a copy of the aliasee, and then found it fixed PR36089. Need to go back and see if the test case in that bug works now or still fails without this change.

I'd bet the testcase in that bug will still fail without this change.

Actually - I just tried it and it passes with this change reverted. I guess something else already fixed that issue in a slightly different probably more correct way?

I'm seeing ThinLTO/X86/dicompositetype-unique-alias.ll fail locally. Maybe you didn't have 22a52dfddcefad4f275eb8ad1cc0e200074c2d8a when you tried? (I probably hadn't pushed yet...)

Sorry to be unclear. That test case, added with this patch, does still fail for me with this change reverted. It will fail if there is more than one DICompositeType - please take a look to see if this looks problematic. I don't recall exactly why I was concerned about that originally. What doesn't fail for me anymore with this change reverted is the test case in the associated PR, which was an actual crash.

Note all of this was before you pushed 22a52dfddcefad4f275eb8ad1cc0e200074c2d8a so didn't have that.

Instead of changing ValueMapper, maybe the right fix is similar to the code in CloneFunctionInto (which I'm conditionalizing in https://reviews.llvm.org/D96531, expecting to commit soon, just missed the window last week where I could watch the bots): prime the ValueToValueMap with mappings-to-self for any entry points to the metadata graph that shouldn't be cloned / duplicated.

Regarding CloneModule - from my skim of the other bug you are working on it appears that this is causing an issue when invoked while cloning a module when we split ThinLTO bitcode files - is that correct? In my case I was fixing an issue during importing, not during this cloning.

Yes, that's correct. Thanks for clarifying this is during importing; it's reasonable at a high level to reuse the ODR logic; the problem is that the value mapper code has more general usecases. (Seems like metadata ownership (especially for distinct nodes) needs to be cleaned up in some way... there are a bunch of subtleties that make it hard to reason about.)

Is the source module being discarded? If so, the importer could safely use RF_ReuseAndMutateDistinctMDs (the new name for RF_MoveDistinctMDs).

Yes it's being discarded. And yes it is already using RF_ReuseAndMutateDistinctMDs (set by default on the IRLinker constructor initialization of the ValueMapper). Hmm, now I'm confused. The case I modified in this patch is where this flag is not set. Aha - going through the debugger for the test case included with this patch shows that while the importer itself uses a ValueMapper with RF_ReuseAndMutateDistinctMDs enabled, the functionality that imports an alias as a copy of the aliasee (replaceAliasWithAliaseee) in fact uses CloneFunction/CloneFunctionInto, so a ValueMapper without that flag. So perhaps your fix in D96531 is sufficient and this change can be reverted?

Your logic SGTM -- replaceAliasWithAliasee DOES use CloneFunction -- but as mentioned above the test is failing for me. I'm confused... maybe my tree is out-of-date, or maybe there's something subtle going on... I'll reply again if I figure it out.

Yes it's being discarded. And yes it is already using RF_ReuseAndMutateDistinctMDs (set by default on the IRLinker constructor initialization of the ValueMapper). Hmm, now I'm confused. The case I modified in this patch is where this flag is not set. Aha - going through the debugger for the test case included with this patch shows that while the importer itself uses a ValueMapper with RF_ReuseAndMutateDistinctMDs enabled, the functionality that imports an alias as a copy of the aliasee (replaceAliasWithAliaseee) in fact uses CloneFunction/CloneFunctionInto, so a ValueMapper without that flag. So perhaps your fix in D96531 is sufficient and this change can be reverted?

Your logic SGTM -- replaceAliasWithAliasee DOES use CloneFunction -- but as mentioned above the test is failing for me. I'm confused... maybe my tree is out-of-date, or maybe there's something subtle going on... I'll reply again if I figure it out.

Looking at it in a debugger, I think I just need to update CloneFunctionInto to catch this case as well.

Sorry to be unclear. That test case, added with this patch, does still fail for me with this change reverted. It will fail if there is more than one DICompositeType - please take a look to see if this looks problematic. I don't recall exactly why I was concerned about that originally. What doesn't fail for me anymore with this change reverted is the test case in the associated PR, which was an actual crash.

Note all of this was before you pushed 22a52dfddcefad4f275eb8ad1cc0e200074c2d8a so didn't have that.

Aha, I get it; I was reading too fast before.

I think the test you added is a good one; better not to clone the composite type unnecessarily. I posted the "revert" at https://reviews.llvm.org/D96734, and it updates CloneFunctionInto to keep the testcase passing.

Sorry to be unclear. That test case, added with this patch, does still fail for me with this change reverted. It will fail if there is more than one DICompositeType - please take a look to see if this looks problematic. I don't recall exactly why I was concerned about that originally. What doesn't fail for me anymore with this change reverted is the test case in the associated PR, which was an actual crash.

Note all of this was before you pushed 22a52dfddcefad4f275eb8ad1cc0e200074c2d8a so didn't have that.

Aha, I get it; I was reading too fast before.

I think the test you added is a good one; better not to clone the composite type unnecessarily. I posted the "revert" at https://reviews.llvm.org/D96734, and it updates CloneFunctionInto to keep the testcase passing.

Great, thanks for fixing this!