This record is constructed by hashing the bytes of the AST block in a similiar
fashion to the SIGNATURE record. This new signature only means anything if the
AST block is fully relocatable, i.e. it does not embed absolute offsets within
the PCM file. This change ensure this does not happen by repalcing these offsets
with offsets relative to the start of the AST block.
This record is constructed by hashing the bytes of the AST block in a similiar
This should address the Windows test failure as well as some offline feedback
- Ensure that DeclOffset instances can only be read by providing an offset to the AST block
- Simplify ASTSignature test by using -fdisable-module-hash
Thanks for working on this! I have a number of suggestions inline.
VERSION_MAJOR should be bumped.
Please update the first sentence to be accurate.
I don't think you'll need this comment anymore, after you fix the above. But if you keep it, please add a period at the end of the sentence.
These names and descriptions hard hard to differentiate. Is there another way of naming these that will be more clear?
(One idea I had is to create CONTROL_BLOCK_HASH and AST_BLOCK_HASH and then SIGNATURE could just be their hash-combine, but maybe you have another idea.)
Please submit this NFC change separately.
Alternately, you could change ASTFileSignature to have three fields:
(You could construct the signature by taking the SHA1 of the two block hashes.)
Rather than duplicating this code, would it be simpler to start with a patch that changes ASTFileSignature to not require this massaging? I suggest either adding a constructor that takes Hash directly, or just changing it to have the same type as Hash (and updating the serialization to expect that).
The default abbreviation (7-VBR) isn't great for hashes. Given that we're going to have two of these records, I suggest specifying a custom abbreviation.
If you were to split out a prep commit as suggested above to change SIGNATURE to be an array of 8-bit values (instead of the current array of 32-bit values), that would be a good opportunity to add an abbreviation.
Can this be relative to the start of SOURCE_MANAGER_BLOCK_ID instead?
Is there ever a reason to use M.FileName here, or is that always redundant with what's in the control block? I wonder if we can just remove this complexity.
I suggest making WriteDecls relative to the start of the DECLTYPES_BLOCK_ID. That will make the block itself (more) self-contained, and also make the offsets smaller (and therefore cheaper to store in bitcode).
I also don't think you need to actually emit that integer, it's just an implicit contract between the reader and writer.
Same here, this should be relative to the start of DECLTYPES_BLOCK_ID.
I made the decision to make all relative offsets relative to the start of the AST block rather than their own sub-block or a neighboring block, in order to prevent adding complexity in ASTReader. My reasoning that it would be best if the reader didn't have to keep track of a bunch of offsets inside the bitcode at all times, but that it could just remember the offset of the AST block once and always use that for relative offsets. I agree that conceptually making the sub-blocks relocatable too would have been nice, but I figured it wasn't worth the extra complexity in the reader.
I think encoding the signature as a blob makes the most sense if we make it an array of 20 8-bit values, which is what the hasher gives us. However it is not quite so simple, as it would need a big change in the way direct imports are serialized. The way it is currently done embeds multiple strings and signatures within a single unabbreviated record. I can replace the unabbreviated record with a sub-block that keeps track of a string table and a record for each import containing offsets inside the string table and blobs for the signature.
The control block only tracks this information for direct imports although it can maybe be extended to do keep track of this for all dependencies. This bit of the table could then become and index inside the IMPORTS record in the control block.
Not sure what you mean by I don't actually need to emit that integer? If you mean ASTBlockRange.first then I don't emit it but rather pass it to the constructor of DeclOffset so that it performs the subtraction there. Since it is a self contained type, it forces the Reader to provide a value for it, making it harder to read the raw relative offset.
I kept the same hasher when computing both of these which mitigates the cost. I don't see the need for also emitting a hash for the control block, there are some optional records that are not in both the AST block and the control block anyway.
I also think that the AST_BLOCK_HASH and the SIGNATURE are enough information already. In most cases you can deduce if the control block was different by just checking if the signatures were different and the ASTs the same.
I think the extra complexity is small and worth it. The default abbreviation is 7-VBR, which is cheaper for small offsets. I doubt we'll bother to come back and fix this later, so might as well get it right now.
|60 ↗||(On Diff #267914)|
Can you make this create? I think new code we prefer lowerCamelCase for method names.
|66–72 ↗||(On Diff #267914)|
This code looks correct, and it's better factored out like this, but I still don't understand why we convert to 32-bit numbers. Have you looked at the code simplification from storing this as an array of 8-bit numbers?
Thanks for updating to AST_BLOCK_HASH, I think that addressed the concern I had.
However, having thought further about it, I don't think it makes sense to create a new record type. I suggest just having a convention of writing the hashes out in a particular order.
In which case, the (single) record name should be something generic like SIGNATURE or SHA1_HASH.
Note: I doubt there would be additional cost to computing the AST and control block hashes separately, but I agree there isn't a compelling use for the control block hash currently (although it would be nice to confirm properties like if the CONTROL_BLOCK_HASH were typically stable when sources changed).
Note: I don't think the content of the "hashed" control block vs. unhashed control block have been carefully considered from first principles. At some point we might do that. I expect there are currently things that change the control block that would not invalidate it for importers.
Have you considered changing ASTFileSignature to have multiple fields, one for the signature and another for the AST block hash? Working with pairs and tuples is always awkward.
I don't understand why switching to 8-bit values would affect this. You can just write each 8-bit value as a separate field in the record, just like it currently does for 32-bit values. I.e., the current code seems to work just as well:
for (auto I : M.Signature) Record.push_back(I);
That said, having a single IMPORTS record is kind of ridiculous. It would be nice to clean that up (either as a follow-up or prep commit). I think your suggestion of having an IMPORTS sub-block is reasonable. A simpler approach would be just to emit IMPORT records. In the reader you can figure out implicitly how many there are (is the next record another IMPORT? If so keep reading). If you did this, I would suggest abbreviating the IMPORT record since it'll be well-structured.
For SIGNATURE record abbreviation, I wasn't think a blob, just something simple like:
auto HashAbbrev = std::make_shared<BitCodeAbbrev>(); HashAbbrev->Add(BitCodeAbbrevOp(Signature)); for (int I = 0, E = 5; I != E; ++I) HashAbbrev->Add(BitCodeAbbrevOp(BitcodeAbbrevOp::Fixed, 32));
If you switch to 8-bit:
auto HashAbbrev = std::make_shared<BitCodeAbbrev>(); HashAbbrev->Add(BitCodeAbbrevOp(Signature)); for (int I = 0, E = 20; I != E; ++I) HashAbbrev->Add(BitCodeAbbrevOp(BitcodeAbbrevOp::Fixed, 8));
Okay. That might be a nice follow-up or prep commit. If you're not interested in that tangent though I suggest adding an accessor ModuleFile::isModule that you can use here.
Ah! I misread the code.
Address @dexonsmith feedback about making the relative offsets in the AST block relative to the nearest relevant subblock.
I agree with the fact that it might be worth maintaining a control block hash, but I do think this is not the purpose of this change. I can do it as a follow up when I have some time. I am somewhat opposed to keeping them all in the same record for now, because it makes it harder to grep for one of the signatures in the output of llvm-bcanalyzer which is currently the only available tooling for looking inside the unhashed control block.
@dexonsmith the field you asked about is used here and the MacroOffsetsBase is used in other places to compute offsets
No, look at line 3706 of the Reader in the updated version. I added a comment on the relevant line.