This is an archive of the discontinued LLVM Phabricator instance.

clang-misexpect: Profile Guided Validation of Performance Annotations in LLVM
AbandonedPublic

Authored by paulkirth on Aug 15 2019, 5:06 PM.

Details

Summary

This patch contains the basic functionality for reporting potentially incorrect usage of __builtin_expect() by comparing the developer's annotation against a collected PGO profile. A more detailed proposal and discussion appears on the CFE-dev mailing list (http://lists.llvm.org/pipermail/cfe-dev/2019-July/062971.html) and a prototype of the initial frontend changes appear here in D65300

We revised the work in D65300 by moving the misexpect check into the LLVM backend, and adding support for IR and sampling based profiles, in addition to frontend instrumentation.

We add new misexpect metadata tags to those instructions directly influenced by the llvm.expect intrinsic (branch, switch, and select) when lowering the intrinsics. The misexpect metadata contains information about the expected target of the intrinsic so that we can check against the correct PGO counter when emitting diagnostics, and the compiler's values for the LikelyBranchWeight and UnlikelyBranchWeight. We use these branch weight values to determine when to emit the diagnostic to the user.

A future patch should address the comment at the top of LowerExpectIntrisic.cpp to hoist the LikelyBranchWeight and UnlikelyBranchWeight values into a shared space that can be accessed outside of the LowerExpectIntrinsic pass. Once that is done, the misexpect metadata can be updated to be smaller.

In the long term, it is possible to reconstruct portions of the misexpect metadata from the existing profile data. However, we have avoided this to keep the code simple, and because some kind of metadata tag will be required to identify which branch/switch/select instructions are influenced by the use of llvm.expect

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Cool, I look forward to checking for outdated __builtin_expects in the Linux kernel, if they exist!

llvm/lib/Transforms/Utils/MisExpect.cpp
109

What's C, L, and U?

117

can std::plus be used here?

123

are the extra parens necessary?

143

Do you need to check first that NOps > 0 first? Otherwise you can likely move the creation of NOps below this conditional.

paulkirth updated this revision to Diff 217282.Aug 26 2019, 5:29 PM
paulkirth marked 3 inline comments as done.

Address Code Review

  • Give better names to extracted constants
  • Remove extraneous call to getValue()
  • Make integers const
  • Remove extra braces
  • Move NOps into branch
  • Use std::plus
paulkirth marked 5 inline comments as done.Aug 26 2019, 5:32 PM
paulkirth added inline comments.
llvm/lib/Transforms/Utils/MisExpect.cpp
109

These are the constant integers for:
C = Index into the branch weight vector
L = likely branch weight
U = unlikely branch weight

The patch renames these to be a bit more transparent.

llvm/lib/Transforms/Utils/MisExpect.cpp
109

const auto *IndexCInt

auto should be used when the type is used in the rhs of an assignment such as in a template specifier IMO. Ditto for below.

124

s/(uint64_t)0/0ULL/ might be simpler, but I don't know if long long == uint64_t on every host Clang can be built for...

paulkirth updated this revision to Diff 217285.Aug 26 2019, 5:47 PM
paulkirth marked an inline comment as done.

Revert to use of auto when extracting metadata

paulkirth marked 2 inline comments as done.Aug 26 2019, 5:49 PM
paulkirth added inline comments.
llvm/lib/Transforms/Utils/MisExpect.cpp
124

I'm not sure if that is true, either.

I think its safe to follow the convention of existing LLVM code doing more or less the same thing, though, right?

clang/lib/Frontend/CompilerInvocation.cpp
3449

The indentation here looks funny. Can you please run git-clang-format and commit the changes on top?

3454

LLVM style is no {} for single statement bodies.

llvm/include/llvm/IR/DiagnosticInfo.h
1009

Is this constructor used anywhere in this patch?

llvm/lib/IR/DiagnosticInfo.cpp
378

Why is this constructor defined out of line, when the other is in the header? Seems like they should either both be in the header, or both be out of line, with the goal being consistency.

paulkirth updated this revision to Diff 217479.Aug 27 2019, 1:03 PM

Removes unused constructor, reformat code, and remove braces

paulkirth marked 6 inline comments as done.Aug 27 2019, 1:07 PM
paulkirth added inline comments.
llvm/include/llvm/IR/DiagnosticInfo.h
1009

No, sorry, I changed this interface and failed to remove the constructor.

llvm/lib/IR/DiagnosticInfo.cpp
378

I removed the other constructor, which was unused.

llvm/lib/Transforms/Utils/MisExpect.cpp
151

Sorry if I'm going back and forth on this, but it may make sense to check the number of operands first, before accessing any.

160

do you need the misexpect:: qualifier here? I think your using statement above should make this one unnecessary?

paulkirth updated this revision to Diff 217491.Aug 27 2019, 2:20 PM
paulkirth marked 2 inline comments as done.

Remove namespace qualifier & insert check for number of arguments

paulkirth updated this revision to Diff 217494.Aug 27 2019, 2:26 PM

Change check for minimum number of metadata operands.

Branch weights must have at least 3 elements: metadata name, and a list of at least 2 branch weights

paulkirth marked 3 inline comments as done.Aug 27 2019, 2:30 PM
paulkirth added inline comments.
llvm/lib/Transforms/Utils/MisExpect.cpp
151

No problem. I've reintroduced the check. I was under the impression that metadata could not lack a 0th element, but even in that case I should have checked for additional elements.

There should always be at least 2 branch weights, otherwise branch weights are not necessary, and we should never end up here.

Should I add a comment saying that?

llvm/lib/Transforms/Utils/MisExpect.cpp
151

Sorry, I meant "check the lower bound of the operands first, ..." as NOps might be 0 which would cause MD->getOperand(0) to fail, and RealWeights to be size -1. if (Nops < 1 || Nops > 3) return;

I think a comment would be helpful, too.

paulkirth marked an inline comment as done.Aug 27 2019, 3:34 PM
paulkirth added inline comments.
llvm/lib/Transforms/Utils/MisExpect.cpp
151

Thanks for the clarification and example. I think we're on the same page, but let me make sure, since I think I might have muddied the conversation by adding some incomplete context to my reasoning. Here is my reasoning in full, and a proposed comment. Let me know if there is something I'm missing, or that I've misunderstood.

I think that the check: if(NOps <3) return; is the correct semantics here.
Operand 0 will always exist if NOps >=3.
In the case of a switch instruction NOps > 3 needs to be supported.

Any profiling data with only less than 2 branch weights implies that the target is completely deterministic, and should not have branch weights associated with it. However, even if that were that happen, we should not emit any misexpect warnings,

Proposed comment:

// Only emit misexpect diagnostics if at least 2 branch weights are present. 
// Less than 2 branch weights means that the profiling metadata is incorrect, 
// is not branch weight metadata, or that the branch is completely deterministic. 
// In these cases we should not emit any diagnostic related to misexpect.
llvm/lib/Transforms/Utils/MisExpect.cpp
151

Got it, yep that should be good.

paulkirth updated this revision to Diff 217528.Aug 27 2019, 4:36 PM

Add comment to clarify choice of operands and reasoning about profiling metadata

paulkirth marked 3 inline comments as done.Aug 27 2019, 4:37 PM

Looks good, will likely approve after these 2 questions.

llvm/lib/Transforms/Utils/MisExpect.cpp
158–159

would an assert be more appropriate then? If cases 1/2/3 above are all highly unlikely, I'd make this an assert rather than early return.

llvm/test/Transforms/PGOProfile/misexpect-branch-correct.ll
80

does the debug info need to be in all of these unit tests? If you still have the C sources handy, -g0 is extremely handy in generating more concise IR for unit tests.

paulkirth marked 2 inline comments as done.Aug 28 2019, 6:08 PM
paulkirth added inline comments.
llvm/lib/Transforms/Utils/MisExpect.cpp
158–159

I'm not sure an assert is appropriate. There are several types of profiling metadata, such as the value profiling metadata, and the types other than branch weight will have different numbers of arguments.

I'll look into the tags that fall under MD_profto see if any have less than 3 operands, but right now I'm not sure. I do know that other places that read the MD_prof tags use early return rather than an assert though.

The two examples I based my conventions on were:

  • InstrProf.cpp:987 where VP metadata is extracted after first checking the number of operands.
  • Metadata.cpp:1345 where branch weights are extracted using a less defensive approach, and do not check the number of operands before checking the metadata tag.
llvm/test/Transforms/PGOProfile/misexpect-branch-correct.ll
80

For this one, I think you're right, the debug info isn't needed, since we are checking that a diagnostic is never emitted.

In the other cases, I wanted to verify that the diagnostic flags the correct line number. If it is OK for the clang based tests to handle that aspect, then I can try to refactor the LLVM tests to be less precise about the output they are checking, and remove the debug data from the IR

nickdesaulniers accepted this revision.Aug 28 2019, 6:38 PM

Ok, thanks for the iteration on code review.

This revision is now accepted and ready to land.Aug 28 2019, 6:38 PM
leonardchan accepted this revision.Aug 29 2019, 7:31 PM
leonardchan added inline comments.
clang-tools-extra/clang-misexpect/ClangMisExpect.cpp
64 ↗(On Diff #215501)

Nit: The default and llvm_unreachable aren't needed here since you cover all cases already.

llvm/lib/IR/MDBuilder.cpp
315

nit: auto *IntType

llvm/lib/Transforms/Utils/MisExpect.cpp
56

nit: auto *B

125

nit: const unsigned NumUnlikelyTargets

130

nit: (double)ProfileCount / CaseTotal

@leonardchan can you revise your review against the latest revision? It looks to me as though you might have reviewed one of the first diffs instead of what is the latest code. For example the files in clang-tools have not been part of this revision for some time. Also when I try to respond to the comment, it appears against the very first diff. I'm happy to address any of the nits that remain, but want to make sure that what you reviewed and given approval for is what will be submitted.

paulkirth updated this revision to Diff 218016.Aug 29 2019, 8:10 PM

Address nits

  • Add pointers to uses of auto
  • Remove redundant cast
  • Use consistent type for integers
paulkirth updated this revision to Diff 218017.Aug 29 2019, 8:12 PM

Actually address nits

paulkirth marked 3 inline comments as done.Aug 29 2019, 8:13 PM
leonardchan accepted this revision.Aug 29 2019, 8:21 PM

Sorry about that. Still LGTM.

Trying to start reviewing this.
The llvm side of the patch is self contained; clang patch should be split into a dependent review.

clang/lib/CodeGen/CodeGenAction.cpp
628

This should be BadDebugInfoLoc (in getBestLocationFromDebugLoc() too)

clang/lib/CodeGen/CodeGenFunction.cpp
28–1361 ↗(On Diff #218017)

Dubious changes, drop

clang/lib/Frontend/CompilerInvocation.cpp
3449–3452

Here you are checking that -Wmisexpect is passed to clang?
This doesn't seem idiomatic. Was this copied from somewhere?
Normally this is Diags.isIgnored(diag::warn_profile_data_misexpect, ???)

llvm/include/llvm/Transforms/Utils/MisExpect.h
2

__builtin_expect() is a clang-specific thing.
Can all instances of it in all comments be reworded to be more agnostic?

llvm/lib/Transforms/Scalar/LowerExpectIntrinsic.cpp
78–87

Why can't LLVMContext::MD_prof be reused?

82

Why do you need to move this line?

84

clang is not the only llvm frontend, this should not be so specific

llvm/lib/Transforms/Utils/MisExpect.cpp
31

This doesn't look correct.

34

<cstdint>

llvm/test/Transforms/LowerExpectIntrinsic/basic.ll
288–294

; CHECK: !1 = ??? ?

paulkirth marked an inline comment as done.Aug 30 2019, 10:35 AM

Thanks for the review.

I'm working on splitting up the patch now.

I should be able to address most of your comments, but I left some inline comments for clarification/discussion.

clang/lib/CodeGen/CodeGenAction.cpp
628

That should be in a separate patch, right?

llvm/lib/Transforms/Scalar/LowerExpectIntrinsic.cpp
78–87

It didn't seem appropriate to me, but that can be changed if it is the right thing to do.

However, I don't see code elsewhere that loops over MD_prof metadata checking its tag. I see lots of examples that check if the tag matches, and then exits if it doesn't match.

If I added "misexpect" as a new MD_proftag, then code in places like extractProfTotalWeight(...) in IR/Metadata.cpp would have to be updated. I'm not sure how pervasive that pattern is, but I did want to avoid introducing systemic changes like that.

https://github.com/llvm/llvm-project/blob/9976a5bc1db5a240378a61a68c0e182060bcb554/llvm/lib/IR/Metadata.cpp#L1336

82

I did this so that the condition is available to check when performing validation, and the metadata won't be replaced by the likely/unlikely values until after I've finished the validation. Having the condition available in when performing the check provides for better diagnostics output.

paulkirth updated this revision to Diff 218164.Aug 30 2019, 2:05 PM

Address Review items

  • minimize debug info in llvm tests
  • sanitize paths in debug info
  • use more complete checks in test for LowerExpectIntrisic
  • remove references to __builtin_expect from backend code
    • update test code for new diagnostic messages
  • rename checkClangInstrumentation to checkFrontendInstrumentation
  • revert whitespace removal in CodeGenFunction.cpp
  • use more idomatic check when adding diagnostic to the backend
paulkirth marked 7 inline comments as done and an inline comment as not done.Aug 30 2019, 2:08 PM

Trying to start reviewing this.
The llvm side of the patch is self contained; clang patch should be split into a dependent review.

Looking at this, is it necessary to split up the patch? That will loose a lot of the previous comment/context, and I'd prefer to land this as a single change. Doing so has the benefit that if my patch needs to be reverted, it can be done all together. For example, if only the llvm patch is reverted, then it will likely cause other problems and make triage needlessly painful.

paulkirth updated this revision to Diff 218168.Aug 30 2019, 2:23 PM

Remove extra include from CodeGenFunction.cpp

paulkirth updated this revision to Diff 218171.Aug 30 2019, 2:29 PM

Fix comment for misexpect option

phosek accepted this revision.Aug 30 2019, 2:56 PM

Trying to start reviewing this.
The llvm side of the patch is self contained; clang patch should be split into a dependent review.

Looking at this, is it necessary to split up the patch? That will loose a lot of the previous comment/context, and I'd prefer to land this as a single change. Doing so has the benefit that if my patch needs to be reverted, it can be done all together. For example, if only the llvm patch is reverted, then it will likely cause other problems and make triage needlessly painful.

I'd actually prefer to land this as a single change, it makes it easier to revert and look at the diff later.

Missing tests for __builtin_unpredictable().

clang/lib/CodeGen/CodeGenAction.cpp
369

Comment missing

628

Yeah, let's keep that for later.

clang/lib/Frontend/CompilerInvocation.cpp
3449

Err, this works? I'd expect ! to be missing here..

llvm/lib/Transforms/Scalar/LowerExpectIntrinsic.cpp
78–87

That's not exactly what i was asking.
I'm specifically talking about how "misexpect" and "branch_weigths" carry basically the same data,
are set basically in the same place always, with the difference that "misexpect" also has the switch case index.
This is both somewhat bloaty, and is prone to getting out of sync.

I have two questions:

  1. Can "misexpect" metadata simply not be invented, but can existing LLVMContext::MD_misexpect simply be used?
  2. Can "misexpect" be changed to not store the weights, but a reference to the LLVMContext::MD_misexpect metadata that will be emitted anyway?
llvm/test/Transforms/LowerExpectIntrinsic/basic.ll
141

Here and elsewhere:
is !prof !1 no longer present?

292–293

Should there be a FIXME, are some misexpect missing here?

paulkirth added inline comments.Aug 30 2019, 6:40 PM
llvm/lib/Transforms/Scalar/LowerExpectIntrinsic.cpp
78–87

Sorry for the misunderstanding.

I think what you are suggesting is to recreate the misexpect metadata from the MD_prof "branch_weights metadata. Is that right?

I think that may be possible, but it will add complexity. While the metadata approach is very straightforward. I think there is value in keeping this simple, and then iterating on it.

Also, LowerExpectIntrinsic.cpp has a comment near the top suggesting that the LikelyBranchWeight and UnlikelyBranchWeights be hoisted to a common space. This would significantly shrink the size of the misexpect metadata if it were accessible directly. However, that is a change that should happen separately from this patch, and might require further discussion to find the correct shared space.

Ultimately, however, I do not see a way to completely remove some kind of metadata tag, as we only need to perform misexpect checks when the intrinsic was used -- which should be a small minority of cases. If there is another way, then I am happy to hear it, but again, maybe that should be another patch.

paulkirth added inline comments.Aug 30 2019, 6:40 PM
clang/lib/Frontend/CompilerInvocation.cpp
3449

I ran check-clang and check-llvm. Are there more tests I should be running?

paulkirth marked 2 inline comments as done.Aug 30 2019, 6:56 PM
paulkirth added inline comments.
llvm/test/Transforms/LowerExpectIntrinsic/basic.ll
141

The misexpect metadata tags take those slots.

br i1 %tobool, label %if.then, label %if.end, !prof !0, !misexpect !1

Unless my understanding of how the tags work is flawed, I think this is the expected behavior

292–293

This is happenstance unfortunately.

; CHECK: !4 = !{!"branch_weights", i32 1, i32 2000, i32 1}
; CHECK: !5 = !{!"branch_weights", i32 2000, i32 1, i32 1}

In both of these cases the expected index is still 0 or 1, so the metadata tags are deduplicated.

If we want to check further, then we need to modify the test to select an index greater than 1.

paulkirth updated this revision to Diff 218197.Aug 30 2019, 7:03 PM

Update LowerExpectIntrinsic/basic.ll to generate misexpect metadata for a switch

paulkirth updated this revision to Diff 218198.Aug 30 2019, 7:10 PM

Add inline checks for misexpect tags to LowerExpectIntrinstic test

paulkirth updated this revision to Diff 218200.Aug 30 2019, 7:37 PM

Add clang and LLVM tests for __builtin_unpredictable

paulkirth updated this revision to Diff 219219.Sep 6 2019, 8:31 PM

Improve diagnostic output with profile counts

lebedev.ri accepted this revision.Sep 7 2019, 12:17 AM
lebedev.ri marked an inline comment as done.

Okay, thank you for the patience, this looks good to me now too.
Please update patch description/subject - this patch is not a standalone tool.

clang/test/Profile/misexpect-branch-cold.c
4

Is there a test where -Wmisexpect isn't present, to verify that it is off-by-default?

llvm/lib/Transforms/Scalar/LowerExpectIntrinsic.cpp
78–87

nonnull

Yes.

...

Well, alright i guess, please add a note that this was at least considered into the patch description.

llvm/test/Transforms/LowerExpectIntrinsic/basic.ll
141

I was worried that here the !prof were no longer emitted, but only !misexpect were emitted.
This seems not to be the case, so all good i think.

paulkirth edited the summary of this revision. (Show Details)Sep 7 2019, 11:35 AM
paulkirth added inline comments.Sep 7 2019, 11:38 AM
clang/test/Profile/misexpect-branch-cold.c
4

We can add one, but is that necessary? Don't the tests for diagnostics cover those already?

lebedev.ri added inline comments.Sep 8 2019, 4:59 AM
clang/test/Profile/misexpect-branch-cold.c
4

To clarify: i'm interested in the case where the PGO data is provided but -Wmisexpect is *not* specified.

paulkirth marked an inline comment as done.Sep 9 2019, 12:16 PM
paulkirth added inline comments.
clang/test/Profile/misexpect-branch-cold.c
4

Actually, this is moot. That test already exists in misexpect-branch.c: line 6.

Since that file actually has a mismatched use of __builtin_expect(), it's a better candidate anyway.

// RUN: %clang_cc1 %s -O2 -o - -disable-llvm-passes -emit-llvm -fprofile-instrument-use-path=%t.profdata -verify=foo
This revision was automatically updated to reflect the committed changes.
paulkirth reopened this revision.Sep 10 2019, 3:24 PM
This revision is now accepted and ready to land.Sep 10 2019, 3:24 PM
paulkirth updated this revision to Diff 219617.EditedSep 10 2019, 3:32 PM

Add a new profdata file to use w/ misexpect-nonconst-switch.c

ASAN exposed an issue where a function may hash the same even if the number of counters is different. This means that when a PGO profile is read in, it is possible for an out of bounds write to occur due to a mismatch in the number of statements that require profile counters, and the number of counters initialized from the profile that was read in.

This change should allow the misexpect patch to reland w/o issue.

This revision was automatically updated to reflect the committed changes.
chandlerc added inline comments.
compiler-rt/trunk/lib/profile/xxhash.h
41–42 ↗(On Diff #219645)

Sorry folks, but you can't do this.

You can't depend on ADT from compiler-rt currently.

There are at least two problems here:

First problem is that this pollutes the profile library with symbols from ADT. That really doesn't seem reasonable without *significant* and invasive changes to ADT. Otherwise building LLVM and linking it with the profile library will create an ODR violation (imagine different assert levels or different versions of LLVM buing built and the host toolchain).

Second, and much more critically, we haven't gotten to 100% relicensed on ADT, so it is critical that we not depend on it from runtime libraries.

Third, a lot of this code seems to use old license headers. Please do not add any code like this to LLVM, and instead use the new LLVM license for all new code.

For now, this patch (and any related patches) need to be reverted until these are addressed. Especially the license issues.

Reverted in r371598.

Another concern that I have is cross-compilation. LLVM's ADT is not set up to be cross-compiled like the rest of compiler-rt is.

lebedev.ri added a comment.EditedSep 11 2019, 2:25 AM

Reverted in r371598.

Another concern that I have is cross-compilation. LLVM's ADT is not set up to be cross-compiled like the rest of compiler-rt is.

Uhm, i have a better question still - why xxhash is even there? it's not used in the diff, and was not reviewed, it just magically appeared in the last update of the patch:
https://reviews.llvm.org/D66324?vs=219617&id=219645

paulkirth marked an inline comment as done.Sep 11 2019, 9:01 AM
paulkirth added inline comments.
compiler-rt/trunk/lib/profile/xxhash.h
41–42 ↗(On Diff #219645)

Sorry, this looks like a mismerge somehow. My patch should't have anything from compiler-rt. I think maybe a local change got rolled in when Petr landed my patch.

phosek added inline comments.Sep 11 2019, 9:02 AM
compiler-rt/trunk/lib/profile/xxhash.h
41–42 ↗(On Diff #219645)

Sorry about that, this was my mistake, it seems two of my stales files got included in when I applied the patch locally. I'll revert the change and reland it correctly.

Reverted in r371598.

Another concern that I have is cross-compilation. LLVM's ADT is not set up to be cross-compiled like the rest of compiler-rt is.

Uhm, i have a better question still - why xxhash is even there? it's not used in the diff, and was not reviewed, it just magically appeared in the last update of the patch:
https://reviews.llvm.org/D66324?vs=219617&id=219645

It shouldn't have. My last change only modified a test file to not trigger a memory corruption bug. I'm filing a bug for that, but am waiting on a login for the bug tracker.

What I suspect happened is that when this landed some local changes got sucked in. I'm terribly sorry, it wasn't my intention to add anything significant to the patch.

paulkirth reopened this revision.Sep 11 2019, 9:05 AM
This revision is now accepted and ready to land.Sep 11 2019, 9:05 AM
paulkirth updated this revision to Diff 219728.Sep 11 2019, 9:06 AM

Revert mismerge when landing.

This revision was automatically updated to reflect the committed changes.
jrtc27 added a subscriber: jrtc27.Sep 16 2019, 9:19 AM
jrtc27 added inline comments.
llvm/trunk/include/llvm/IR/DiagnosticInfo.h
79 ↗(On Diff #219733)

Is this not going to clash with the first caller to getNextAvailablePluginDiagnosticKind? DK_FirstPluginKind is special and shouldn't have anything after it.

paulkirth added inline comments.Sep 16 2019, 9:34 PM
llvm/trunk/include/llvm/IR/DiagnosticInfo.h
79 ↗(On Diff #219733)

hmm, I wasn't aware of that. Is this documented somewhere? Also no tests seem to break from this, so that might be another area to address.

I'll upload a patch for this that moves the flag, and adds a note about the ordering.

paulkirth marked an inline comment as done.Sep 16 2019, 9:51 PM
paulkirth added inline comments.
llvm/trunk/include/llvm/IR/DiagnosticInfo.h
79 ↗(On Diff #219733)

Please see D67648 for future reference of this defect.

wenju added a subscriber: wenju.Sep 24 2019, 12:41 AM

We see error "clang (LLVM option parsing): for the --pgo-warn-misexpect option: may only occur zero or one times!" when clang::CompilerInvocation::CreateFromArgs and clang::ExecuteCompilerInvocation are called twice.

This line: if (Diags.isIgnored(diag::warn_profile_data_misexpect, SourceLocation()))
I think it should be !Diags.isIgnored.

aykevl added a subscriber: aykevl.EditedApr 2 2020, 8:27 AM

I have the same issue as @wenju. The second time I call ExecuteCompilerInvocation it will give the error above, leading me to believe some memory isn't properly cleared.

Note: the --pgo-warn-misexpect option is not passed in the compiler invocation.

Fix is easy, can you commit it as obvious?

aykevl added a comment.Apr 5 2020, 8:40 AM

@xbolva00 I could, but to me this is not trivial (I'm not familiar with the code and I'm not sure what I would break). If it's easy for you, then please do (otherwise I can send a patch for review).
I have already worked around this issue in a different way (by running the compiler invocation in a separate process).

Also hit by this issue - a lot more people are certainly going to have it as this has started landing in Linux distros.

rnk added a subscriber: rnk.May 20 2020, 3:01 PM
rnk added inline comments.
cfe/trunk/lib/Frontend/CompilerInvocation.cpp
3457 ↗(On Diff #219733)

Clang generally tries to avoid relying on LLVM option parsing if at all possible. It is not thread-safe, among other things. This causes essentially every compile to call llvm::cl::ParseCommandLineOptions, when previously it would only happen if the user passed unstable -mllvm flags.

xbolva00 added inline comments.Aug 12 2020, 9:45 AM
cfe/trunk/lib/Frontend/CompilerInvocation.cpp
3457 ↗(On Diff #219733)

And it is buggy, if you add "!" as a fix, there are many test failures...
http://45.33.8.238/linux/18098/step_7.txt

@paulkirth @phosek @leonardchan @thakis @nickdesaulniers
It seems to me that this feature is still broken an unusable - https://bugs.llvm.org/show_bug.cgi?id=45073 -and the bug was reported more than half a year ago.
If no one is willing to step up and fix it, the i can only assume that no one is using this and or cares, and this broken incomplete feature should be reverted.

lebedev.ri added a subscriber: wmi.Oct 16 2020, 5:01 AM

+1 for revert

I apologize for the late response, I somehow missed the earlier responses. We have successfully used this feature in Fuchsia and found it useful, but I agree that the issues raised need to be addressed. Unfortunately @paulkirth is no longer working on this project. I hope that someone from our team can take a look but it might take a few weeks. If you prefer, we could revert this change and then reland an improved version in the future?

I apologize for the late response, I somehow missed the earlier responses. We have successfully used this feature in Fuchsia and found it useful, but I agree that the issues raised need to be addressed. Unfortunately @paulkirth is no longer working on this project. I hope that someone from our team can take a look but it might take a few weeks. If you prefer, we could revert this change and then reland an improved version in the future?

I would very much prefer *NOT* not revert if someone is going to step up to work on these problems soon (within next 4 weeks?).

That being said, in light of that bug, my original doubts about the underlying data type (a novel MD_misexpect,
with structure different from MD_prof) have reappeared with double strength. I really think they should share underlying type.

I apologize for the late response, I somehow missed the earlier responses. We have successfully used this feature in Fuchsia and found it useful, but I agree that the issues raised need to be addressed. Unfortunately @paulkirth is no longer working on this project. I hope that someone from our team can take a look but it might take a few weeks. If you prefer, we could revert this change and then reland an improved version in the future?

I would very much prefer *NOT* not revert if someone is going to step up to work on these problems soon (within next 4 weeks?).

That being said, in light of that bug, my original doubts about the underlying data type (a novel MD_misexpect,
with structure different from MD_prof) have reappeared with double strength. I really think they should share underlying type.

I'll be posting a revert soon.

haowei added a subscriber: haowei.Nov 13 2020, 3:16 PM

I apologize for the late response, I somehow missed the earlier responses. We have successfully used this feature in Fuchsia and found it useful, but I agree that the issues raised need to be addressed. Unfortunately @paulkirth is no longer working on this project. I hope that someone from our team can take a look but it might take a few weeks. If you prefer, we could revert this change and then reland an improved version in the future?

I would very much prefer *NOT* not revert if someone is going to step up to work on these problems soon (within next 4 weeks?).

That being said, in light of that bug, my original doubts about the underlying data type (a novel MD_misexpect,
with structure different from MD_prof) have reappeared with double strength. I really think they should share underlying type.

I'll be posting a revert soon.

Sorry for the late reply. Since @paulkirth is no longer working on this project. I can take over it and start working on it full time beginning next week. It would take me some time to get familiar with the code and work on the fix though. If you still prefer reverting this change, we can work on an improved version and reland it in the future.

I apologize for the late response, I somehow missed the earlier responses. We have successfully used this feature in Fuchsia and found it useful, but I agree that the issues raised need to be addressed. Unfortunately @paulkirth is no longer working on this project. I hope that someone from our team can take a look but it might take a few weeks. If you prefer, we could revert this change and then reland an improved version in the future?

I would very much prefer *NOT* not revert if someone is going to step up to work on these problems soon (within next 4 weeks?).

That being said, in light of that bug, my original doubts about the underlying data type (a novel MD_misexpect,
with structure different from MD_prof) have reappeared with double strength. I really think they should share underlying type.

I'll be posting a revert soon.

Sorry for the late reply. Since @paulkirth is no longer working on this project. I can take over it and start working on it full time beginning next week. It would take me some time to get familiar with the code and work on the fix though. If you still prefer reverting this change, we can work on an improved version and reland it in the future.

I think revert will result in an more easily reviewable fix, yes.

lebedev.ri reopened this revision.Nov 14 2020, 2:14 AM

Reverted in rG6861d938e5c946cc7079d9849ef7560d07aa2d80, looking forward to the rework!

This revision is now accepted and ready to land.Nov 14 2020, 2:14 AM
lebedev.ri requested changes to this revision.Nov 14 2020, 2:14 AM
This revision now requires changes to proceed.Nov 14 2020, 2:14 AM
paulkirth abandoned this revision.Mar 29 2022, 10:47 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 29 2022, 10:47 AM
Herald added a subscriber: abrachet. · View Herald Transcript