Page MenuHomePhabricator

[clang] Improve Serialization/Imporing/Dumping of APValues
Needs ReviewPublic

Authored by Tyker on Jun 21 2019, 3:07 AM.

Details

Summary

Changes:

  • initializer expressions of constexpr variable are now wraped in a ConstantExpr. this is mainly used for testing purposes. the old caching system has not yet been removed.
  • Add all the missing Serialization and Importing for APValue.
  • Improve dumping of APValue when ASTContext isn't available.
  • Cleanup leftover from last patch.
  • Add Tests for Import and serialization.

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Tyker updated this revision to Diff 208015.Jul 4 2019, 4:31 AM

fixed the comment for good, sorry for that.

Tyker marked 2 inline comments as done.Jul 4 2019, 4:34 AM
Tyker added inline comments.
clang/lib/AST/ASTImporter.cpp
8503

we can have a common header, but i don't know where to put it. having a in PCH that includes a header form ASTMerge seems weird and vice versa.

martong added inline comments.Jul 5 2019, 8:24 AM
clang/lib/AST/ASTImporter.cpp
8503

Yes, that is indeed not so good.

The point is that, we should execute the APValue tests as well when we invoke ninja check-clang-astmerge. (Some people do not execute the whole check-clang verification when they change only the ASTImporter, this way the edit-test cycle can be faster)

Can we have a symbolic link from test/PCH/APValue.cpp to test/ASTMerge/APValue.cpp? Probably that would work on *nix, but not on Windows (https://stackoverflow.com/questions/5917249/git-symlinks-in-windows).
If the symlink is really not an option then I am just fine with a real copy of the file.
If we have a link or a copy then the tests will run twice, that seems ok for me, but may disturb other devs.
Actually, this is an inconvenient problem, perhaps someone had this before, so maybe a mail to cfe-dev could help us out.

Tyker updated this revision to Diff 217600.Aug 28 2019, 4:07 AM

Sorry for the long wait.

Changes:

  • Rebased on current master
  • Duplicated test file so that it runs for both importing

Sorry for the long wait.

Changes:

  • Rebased on current master
  • Duplicated test file so that it runs for both importing

Thanks for addressing the issues. The ASTImporter related changes look good to me now.

aaron.ballman added inline comments.Sep 19 2019, 7:08 AM
clang/include/clang/AST/APValue.h
512

reserveVector per naming conventions

clang/include/clang/AST/ASTContext.h
275

Why are you getting rid of this? It seems like we would still want these cleaned up.

clang/include/clang/AST/Expr.h
957

Enumerator -> Enumerators
there -> their

clang/include/clang/AST/TextNodeDumper.h
149

Good catch -- this pointed out a bug in the class that I've fixed in r372323, so you'll need to rebase.

clang/lib/AST/APValue.cpp
176

Why is this no longer a pointer to const?

748

Can this function be marked const?

clang/lib/AST/ASTImporter.cpp
8503

Conflict markers?

8749

Please don't use auto as the type is not spelled out in the initialization.

8754

Please don't use auto as the type is not spelled out in the initialization.

(Same elsewhere, I'll stop commenting on them.)

clang/lib/AST/TextNodeDumper.cpp
695–696

You'll have to rebase this too -- these changes shouldn't be needed any longer.

clang/lib/Sema/SemaDecl.cpp
11290

Can elide braces.

clang/lib/Serialization/ASTReader.cpp
9181

auto *

9187–9188

auto *

9194

auto *

9205–9211

These names don't match the current coding conventions.

9214

You should add parens here to avoid diagnostics.

clang/lib/Serialization/ASTWriter.cpp
5481

Name doesn't match coding conventions.

5492

const auto *

5514

const auto *

5518

const auto *

5534–5535

Unrelated change?

clang/lib/Serialization/ASTWriterStmt.cpp
42 ↗(On Diff #217600)

It seems like there's a lot of unrelated formatting changes in this file that should not be part of the patch.

Tyker updated this revision to Diff 221166.Sep 21 2019, 4:33 AM
Tyker marked 24 inline comments as done.

Fixed most changes requested by @aaron.ballman
test are currently no valid anymore see comments for why.

clang/include/clang/AST/ASTContext.h
275

when i added APValueCleanups i wasn't aware that there were a generic system to handle this. but with this patch APValue a cleaned up using the generic ASTContext::addDestruction.

clang/include/clang/AST/TextNodeDumper.h
149

i took a look at the revision. there is a big difference is the quality of output between APValue::dump and APValue::printPretty. i think it is possible to come quite close to printPretty's output even without the ASTContext. this would require having a default PrintingPolicy and improving dump

in this patch i was relying on the -ast-dump output for testing. i would need to find an other testing strategy or make the improvement to APValue::dump first.

clang/lib/AST/APValue.cpp
176

when imporing or deserializing, we reserve the space for elements and then import/deserialize element directly in place. so the buffer storing them is not const. that said i saw that the normal construction cast away the const.

748

this function gives access to non-const internal data. this function is private so the impact is quite limited.

Tyker updated this revision to Diff 221169.Sep 21 2019, 5:29 AM
Tyker marked 2 inline comments as done.
Tyker added inline comments.
clang/lib/AST/APValue.cpp
176

never mind this change wasn't needed.

aaron.ballman added inline comments.Sep 23 2019, 8:24 AM
clang/include/clang/AST/APValue.h
512

This was marked as done but is still an issue.

537–540

Rather than add this private bit in the middle of the public interface, you can move this to the existing private parts.

618

We're horribly inconsistent in this class, but because the other private member functions go with it, this should probably be GetMemberPointerPathPtr(). Maybe rename the get/setLValue methods from above as well?

clang/include/clang/AST/ASTContext.h
275

I don't see any new calls to addDestruction() though. Have I missed something?

clang/include/clang/AST/TextNodeDumper.h
149

there is a big difference is the quality of output between APValue::dump and APValue::printPretty.

Yes, there is.

i think it is possible to come quite close to printPretty's output even without the ASTContext. this would require having a default PrintingPolicy and improving dump

That would be much-appreciated! When I looked at it, it seemed like it may not be plausible because Stmt does not track which ASTContext it is associated with the same way that Decl does, and changing that seemed likely to cause a huge amount of interface churn.

in this patch i was relying on the -ast-dump output for testing. i would need to find an other testing strategy or make the improvement to APValue::dump first.

The issue resolved by r372323 was that we would crash on certain kinds of AST dumps. Specifically, the default AST dumper is often used during a debugging session to dump AST node information within the debugger. It was trivial to get that to crash before r372323, but with that revision, we no longer crash but get slightly uglier output (which is acceptable because it's still human-readable output).

I'm sorry for causing extra pain for you here, but I didn't want the fix from this review to accidentally become an enshrined part of the API because it's very easy to forget about this use case when working on AST dumping functionality.

clang/lib/AST/APValue.cpp
748

That makes it harder to call this helper from a constant context. I think there should be overloads (one const, one not) to handle this.

clang/test/ASTMerge/APValue/APValue.cpp
1 ↗(On Diff #221169)

Can remove the spurious newline. Also, it seems this file was added with svn properties, was that intentional (we don't usually do that, FWIW)?

2–3 ↗(On Diff #221169)

no need for -x c++ is there? This is already a C++ compilation unit.

28 ↗(On Diff #221169)

Are you planning to address this in this patch? Also, I think it's FixedPoint and not FixePoint.

clang/test/PCH/APValue.cpp
2

Spurious newline.

3

-x c++ ?

It's really unfortunate that these test files are identical copies in different directories.

29

Same comment here.

Tyker updated this revision to Diff 221384.Sep 23 2019, 12:28 PM
Tyker marked 12 inline comments as done.
Tyker retitled this revision from [clang] Improve Serialization/Imporing of APValues to [clang] Improve Serialization/Imporing/Dumping of APValues.
Tyker edited the summary of this revision. (Show Details)

fixed some changes.
see comments for others.

clang/include/clang/AST/APValue.h
512

sorry

618

We're horribly inconsistent in this class

this class has many flaws. but is far too broadly used to fix.

clang/include/clang/AST/ASTContext.h
275

the modification to use addDestruction() was made in a previous revision (https://reviews.llvm.org/D63376).
the use is currently on master in ConstantExpr::MoveIntoResult in the RSK_APValue case of the switch.
this is just a removing an unused member.

clang/include/clang/AST/TextNodeDumper.h
149

no worries, i wrote the original bug. i added APValue::dumpPretty which has almost the same output as APValue::printPretty but doesn't need an ASTContext. and is used for TextNodeDumper.

clang/lib/AST/APValue.cpp
748

this helper is not intended to be used outside of importing and serialization. it is logically part of initialization.
normal users are intended to use ArrayRef<APValue::LValuePathEntry> APValue::getLValuePath() const

clang/test/ASTMerge/APValue/APValue.cpp
1 ↗(On Diff #221169)

it wasn't intentional, i added via git add i don't think i did anything weird. is it a problem ?

2–3 ↗(On Diff #221169)

i don't know if it is normal. but i am getting an error hen i am not using -x c++
error: invalid argument '-std=gnu++2a' not allowed with 'C'

28 ↗(On Diff #221169)

i don't intend to add them in this patch or subsequent patches. i don't know how to use the features that have these representations, i don't even know if they can be stored stored in that AST. so this is untested code.
that said theses representations aren't complex. the imporing for FixePoint, ComplexInt, ComplexFloat is a no-op and for AddrLabelDiff it is trivial. for serialization, I can put an llvm_unreachable to mark them as untested if you want ?

clang/test/PCH/APValue.cpp
3

they are almost the same. the second run line is slightly different. i had a version with both run line in the same file. but martong was worried that people running partial test would only not get all test relevant for what they were testing.

after through. ASTMerge requires both serialization and importing so it would probably be sufficient ?

aaron.ballman marked an inline comment as done.Sep 23 2019, 1:02 PM
aaron.ballman added inline comments.
clang/include/clang/AST/APValue.h
618

Agreed -- I wasn't suggesting to fix the whole class, but just the new APIs that we add to the class. It looks like the private functions most consistently use a capital letter in this class, unfortunately. Best to stick with the local convention when in conflict.

clang/include/clang/AST/ASTContext.h
275

Ahhh, thank you for the explanation, I was missing that context.

clang/include/clang/AST/PrettyPrinter.h
203 ↗(On Diff #221384)

Wether -> Whether

clang/lib/AST/APValue.cpp
534

Since you're touching the code anyway, this can be const auto *.

594

Are you sure this doesn't change behavior? See the implementation of ASTContext::getAsArrayType(). Same question applies below.

609

const auto * and same question about behavior changes.

748

Nothing about this API suggests that. The name looks like a generic getter. Perhaps a more descriptive name and some comments would help?

clang/test/ASTMerge/APValue/APValue.cpp
1 ↗(On Diff #221169)

No idea; I'm on a platform where file modes are ignored. You should probably drop the svn property.

2–3 ↗(On Diff #221169)

There's no %s on that line for the source file, which is why you get that diagnostic. I'm not certain what that RUN line does, actually -- it does an AST merge with... nothing... and then prints it out?

If that's intended, then you only need the -x c++ on that one RUN line.

28 ↗(On Diff #221169)

I don't think llvm_unreachable makes a whole lot of sense unless the code is truly unreachable because there's no way to get an AST with that representation. By code inspection, the code looks reasonable but it does make me a bit uncomfortable to adopt it without tests. I suppose the FIXME is a reasonable compromise in this case, but if you have some spare cycles to investigate ways to get those representations, it would be appreciated.

Tyker updated this revision to Diff 221568.Sep 24 2019, 11:57 AM
Tyker marked 7 inline comments as done.

fixed most comments

clang/lib/AST/APValue.cpp
594

i ran the test suite after the change it there wasn't any test failures. but the test on dumping APValue are probably not as thorough as we would like them to be.
from analysis of ASTContext::getAsArrayType() the only effect i see on the element type is de-sugaring and canonicalization which shouldn't affect correctness of the output. de-sugaring requires the ASTContext but canonicalization doesn't.

i think the best way the have higher confidence is to ask rsmith what he thinks.

clang/test/ASTMerge/APValue/APValue.cpp
28 ↗(On Diff #221169)

the reason i proposed llvm_unreachable was because it passes the tests and prevents future developer from depending on the code that depend on it assuming it works.

aaron.ballman added inline comments.Sep 25 2019, 8:52 AM
clang/include/clang/AST/APValue.h
622

ReserveVector

629

SetLValueEmptyPath

clang/lib/AST/APValue.cpp
594

Yeah, I doubt we have good test coverage for all the various behaviors here. I was wondering if the qualifiers bit was handled properly with a simple cast. @rsmith is a good person to weigh in.

clang/test/ASTMerge/APValue/APValue.cpp
28 ↗(On Diff #221169)

We typically only use llvm_unreachable for situations where we believe the code path is impossible to reach, which is why I think that's the wrong approach. We could use an assertion to test the theory, however.

Tyker updated this revision to Diff 222169.Sep 27 2019, 7:18 AM
Tyker marked 3 inline comments as done.

made renamings

rsmith added inline comments.Sep 27 2019, 11:53 AM
clang/lib/AST/Expr.cpp
319

Can you use llvm_unreachable here? (Are there cases where we use RSK_None and then later find we actually have a value to store into the ConstantExpr?)

clang/lib/Serialization/ASTReader.cpp
9218

This is problematic.

ReadExpr will read a new copy of the expression, creating a distinct object. But in the case where we reach this when deserializing (for a MaterializeTemporaryExpr), we need to refer to the existing MaterializeTemporaryExpr in the initializer of its lifetime-extending declaration. We will also need to serialize the ASTContext's MaterializedTemporaryValues collection so that the temporaries lifetime-extended in a constant initializer get properly handled.

That all sounds very messy, so I think we should reconsider the model that we use for lifetime-extended materialized temporaries. As a half-baked idea:

  • When we lifetime-extend a temporary, create a MaterializedTemporaryDecl to hold its value, and modify MaterializeTemporaryExpr to refer to the MaterializedTemporaryDecl rather than to just hold the subexpression for the temporary.
  • Change the LValueBase representation to denote the declaration rather than the expression.
  • Store the constant evaluated value for a materialized temporary on the MaterializedTemporaryDecl rather than on a side-table in the ASTContext.

With that done, we should verify that all remaining Expr*s used as LValueBases are either only transiently used during evaluation or don't have these kinds of identity problems.

Tyker marked 10 inline comments as done.Oct 6 2019, 2:34 AM

update done tasks.

clang/lib/AST/APValue.cpp
594

the original question we had is whether it is correct to replace Ctx.ASTContext::getAsArrayType(ElemTy) by cast<ArrayType>(ElemTy.getCanonicalType()) in this context and the other comment below.

clang/lib/AST/Expr.cpp
319

we can put llvm_unreachable in the switch because of if (!Value.hasValue()) above the switch but we can't remove if (!Value.hasValue()).
all cases i have seen where if (!Value.hasValue()) is taken occur after a semantic error occured.

clang/lib/Serialization/ASTReader.cpp
9218

Would it be possible to adapt serialization/deserialization so that they make sure that MaterializeTemporaryExpr are unique.
by:

  • When serializing MaterializeTemporaryExpr serialize a key obtained from the pointer to the expression as it is unique.
  • When deserializing MaterializeTemporaryExpr deserializing the key, and than have a cache for previously deserialized expression that need to be unique.

This would make easier adding new Expr that require uniqueness and seem less complicated.
What do you think ?

Tyker marked an inline comment as done.Wed, Oct 23, 4:04 PM
Tyker added inline comments.
clang/lib/Serialization/ASTReader.cpp
9218

i added a review that does the refactoring https://reviews.llvm.org/D69360.