Page MenuHomePhabricator

[SystemZ/z/OS] Add GOFFObjectFile class and details of GOFF file format
Needs ReviewPublic

Authored by yusra.syeda on Oct 8 2020, 1:56 PM.

Details

Summary

This patch details the GOFF file format and implements the GOFFObjectfile class.
This patch uses https://reviews.llvm.org/D88741

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Add check for ESD name length field size
Update loops to comply with LLVM coding standard:
https://llvm.org/docs/CodingStandards.html#don-t-evaluate-end-every-time-through-a-loop

yusra.syeda marked 3 inline comments as done.Nov 23 2020, 11:40 AM

Remove return after llvm_unreachable statement

yusra.syeda marked 2 inline comments as done.Nov 23 2020, 11:58 AM

Remove setERSymbolType and getERSymbolType functions

yusra.syeda marked an inline comment as done.Nov 23 2020, 3:13 PM
yusra.syeda added inline comments.
llvm/include/llvm/Object/GOFF.h
224

The setERSymbolType and getERSymbolType functions have been removed from this patch as they are not required.

yusra.syeda marked an inline comment as done.Nov 23 2020, 3:14 PM
yusra.syeda marked 4 inline comments as done.Nov 24 2020, 12:50 PM
yusra.syeda added inline comments.
llvm/include/llvm/Object/GOFF.h
138

This field will be added in a future patch.

157

This will also be added in a future patch.

267

Same with this field.

488

Yes, these will be coming in a future patch.

yusra.syeda marked 4 inline comments as done.Nov 24 2020, 12:50 PM

Added unit test GOFFObjectFileTest.cpp, and added GOFF case in TestFileMagic.cpp
Addressed more review comments

yusra.syeda marked 9 inline comments as done.Dec 9 2020, 8:29 AM
jhenderson added inline comments.Dec 10 2020, 1:02 AM
llvm/include/llvm/Object/GOFFObjectFile.h
51

Why is getSymbolName still returning a std::error_code and not an Error?

llvm/lib/Object/GOFFObjectFile.cpp
48

What is the right size? What is the size that the object file actually is? Please include context in the message. Also, no trailing full stop in error messages. See https://llvm.org/docs/CodingStandards.html#error-and-warning-messages.

180

(or "beginning")

begin = verb, beginning/start = nouns

291

What has actually been specified though? Which symbol (if known)?

304

What type is it actually?

Also, clang-format.

339

You didn't answer my question: "What happens if there is no symbol section in the file?"

llvm/unittests/Object/GOFFObjectFileTest.cpp
76

clang-format this file

Return Error instead of error_code for function getSymbolName
Address other review comments

yusra.syeda marked 5 inline comments as done.Dec 14 2020, 12:49 PM
yusra.syeda added inline comments.
llvm/include/llvm/Object/GOFFObjectFile.h
51

Seems like I missed this. It's been updated.

llvm/lib/Object/GOFFObjectFile.cpp
291

These are the only options for the symbolType in the ESDSymbolType enum. If it's not one of these then the type is invalid.

304

These are the only options in the ESDExecutable enum, so if it's not one of these then the type is invalid.

339

This should be unreachable and it should be an error if there is no symbol section. I updated the llvm_unreachable statement to return an Error instead.

yusra.syeda marked 2 inline comments as done.Dec 14 2020, 12:49 PM
jhenderson added inline comments.Dec 15 2020, 12:10 AM
llvm/include/llvm/Object/GOFFObjectFile.h
51

Sorry, I should have spotted this earlier. It would be a more common style for this function's signature to return an Expected rather than take the Res as an argument:

Expected<StringRef> getSymbolName(SymbolRef Symbol) const;

This would match, e.g. getSectionName or getSectionContents as good examples.

llvm/lib/Object/GOFFObjectFile.cpp
47

Please remember to clang-format your changes.

49

getBufferSize() returns size_t not int. The correct format specifier is %z for size_t.

202

Here, you'd want Error::success(), but actually, if you switch to using Expected, you'd return *NameOrErr directly.

291

I don't think you understood what I meant - when I asked this and the similar question below regarding executable type, I meant you should include the additional context, i.e. which symbol had an invalid type (i.e. the index or possibly name) and what that invalid type was, e.g. "symbol 42 has unknown type 0x12". It's important to do this properly because the user's input object might be corrupted in some way, and the code needs to make it easier for that user to find the problem.

304

Same as above - give more useful context in this message, e.g. "executable has unknown type 0x1111".

Also, I think it's more common to omit the std:: prefix from std::errc values (LLVM has its own version of this set, which partly parallels the std one). Please take a look at removing that prefix from all these std::errc instances.

344

I think it would be slightly clearly to say no symbol section found. "unable to get" sounds like there was an actual problem retrieving the section (e.g. some part of the section data was invalid), whereas "no ... found" is clear that it's simply not there.

yusra.syeda updated this revision to Diff 311981.EditedDec 15 2020, 11:56 AM

Change return type of getSymbolType function from Error to Expected<StringRef>
Also remove StringRef parameter passed by reference to the function
Update error messages to be more descriptive

yusra.syeda marked 4 inline comments as done.

Fix typo

yusra.syeda marked 5 inline comments as done.Dec 15 2020, 12:01 PM
yusra.syeda added inline comments.
llvm/lib/Object/GOFFObjectFile.cpp
291

Thanks, done.

yusra.syeda marked an inline comment as done.Dec 15 2020, 12:02 PM
jhenderson added inline comments.Dec 16 2020, 1:38 AM
llvm/include/llvm/BinaryFormat/GOFF.h
54

Does the GOFF spec specify the size of symbol types and other things like that? If so, I'd use the explicit size for these fields. For example, if symbol type is guaranteed to be a single byte, you might adopt my inline edit. The same goes for each other enum below.

llvm/lib/Object/GOFFObjectFile.cpp
137

I think this comment was referring to the "unhandled" bits below. It's been marked as done, but I don't see any response. Could you clarify more why this isn't a hard error and instead such things are being ignored?

200–204

Actually, better yet, I think you can simplify this down to a single line as suggested in the edit.

293–294

Rather than casting EsdId, use the correct print format specifier.

309–310

Same as above.

Update error statement, clean up getSymbolName function, add size to enum

yusra.syeda marked 5 inline comments as done.Dec 18 2020, 11:53 AM
yusra.syeda added inline comments.
llvm/include/llvm/BinaryFormat/GOFF.h
54

I've added the sizes where applicable.

llvm/lib/Object/GOFFObjectFile.cpp
137

These should not be errors. The GOFF reader ignores the normally less important things.

yusra.syeda marked 2 inline comments as done.Dec 18 2020, 11:53 AM

Apply clang format suggestion

I've not reviewed the testing yet, but my immediate thought is that there needs to be a lot more, handling all the different code paths.

llvm/include/llvm/BinaryFormat/GOFF.h
110

Is the field that holds this not a fixed size type? If it is, you could use uint8_t/uin16_t etc as appropriate here to match.

llvm/lib/Object/GOFFObjectFile.cpp
293–294

Now that SymbolType is defined to be a uint8_t, you should use that explicitly, i.e. something like 0x%02" PRIX8" (though you could probably simplify and omit the "02" bit, since this is an error message, and getting a fixed width field isn't really necessary.

Fix error message formatting

yusra.syeda marked an inline comment as done.Jan 8 2021, 1:43 PM
yusra.syeda added inline comments.
llvm/include/llvm/BinaryFormat/GOFF.h
110

This field is 3 bits in size.

jhenderson added inline comments.Jan 11 2021, 12:06 AM
llvm/include/llvm/BinaryFormat/GOFF.h
110

Right, okay. I'd still consider using uint8_t, as that is the smallest type that can be used here, I believe. Same probably applies elsewhere. This will allow easier print formatting.

Added size to ESDExecutable enum
Reformatted unit tests

I've not reviewed the testing yet, but my immediate thought is that there needs to be a lot more, handling all the different code paths.

@jhenderson can you please review the testing? Currently the tests construct a GOFF object with a valid sized record (80 bytes), an invalid sized record (!80 bytes), and obtains the symbol name from the ESD record. Testing for relocations will be added in a future patch.

The majority of your code is still untested as far as I can see. There appear to be three test cases you have so far:

  1. An invalid size for a GOFF object.
  2. A valid size for a GOFF object.
  3. That getSymbolName returns the name of a single symbol in the symbol table.

What about all the rest of the functionality that is included in this patch, including, but certainly not limited to, the following?

  1. More than one symbol in the symbol table.
  2. Other properties of symbols.
  3. The various properties of records.
  4. Relocations.
  5. And so on...

For each bit of code you have written, consider whether a test would fail if that bit of code was broken in some way, or didn't exist. If no test fails, then that code needs a new test case of some form. There may also be other cases where testing is appropriate, e.g. where two separate aspects of the same system interact in some way, although those are harder to judge.

llvm/lib/Object/GOFFObjectFile.cpp
65–66

I don't see a test case involving a continuation record. You should have one followed by a non-continuation record, as otherwise this aspect is not tested.

llvm/unittests/Object/GOFFObjectFileTest.cpp
21

This and the other functions below are only used in one place, if I'm not mistaken. As such, just inline them - splitting them off makes it harder to follow what the individaul tests are doing, since you have to jump around the file.

44

The valid size can be any multiple of 80 bytes. I'd recommend a second test-case that uses a size of something other than 80 bytes, e.g. 160 bytes.

What about 0 bytes? That probably needs a specific test case, as that is a multiple of 80...

53

According to the code, it needs to be a multiple of 80 bytes, so this comment isn't quite correct (it implies 160 is not a valid size).

54

Test the edge cases e.g. 79 and/or 81.

59

Rather than Failed(), use FailedWithMessage(), so that you can check the error message output.

74

This is only used once - just inline it.

75

I suspect, given the name, that the SymbolRef type is very small already (in the same manner as StringRef), and there's no real benefit in making this a const &.

80–81

You should just be able to do EXPECT_EQ(SymbolName, "Hello"); here.

kpn added a comment.Mar 9 2021, 7:56 AM

The majority of your code is still untested as far as I can see. There appear to be three test cases you have so far:

  1. An invalid size for a GOFF object.
  2. A valid size for a GOFF object.
  3. That getSymbolName returns the name of a single symbol in the symbol table.

What about all the rest of the functionality that is included in this patch, including, but certainly not limited to, the following?

  1. More than one symbol in the symbol table.
  2. Other properties of symbols.
  3. The various properties of records.
  4. Relocations.
  5. And so on...

For each bit of code you have written, consider whether a test would fail if that bit of code was broken in some way, or didn't exist. If no test fails, then that code needs a new test case of some form. There may also be other cases where testing is appropriate, e.g. where two separate aspects of the same system interact in some way, although those are harder to judge.

Here's an idea: I found, at least when running in batch, that the Binder (linker) will link an object consisting of nothing more than a HDR card, then ESD cards followed by an END card (meaning, just symbols). It will also link an object consisting of HDR, ESD, TXT, and END cards with zero relocations. Does it work that way when not running in batch? Because if it does then it might make sense to split this ticket up into a new ticket with just support for HDR+ESD+END cards.

That would make this patch smaller, and it would reduce the amount of tests that need to be written to get some initial GOFF support into the tree. The tests that @jhenderson requested would still be needed, but you'd only need the ones that were relevant to the smaller amount of code in the new ticket. A new ticket should refer back to this ticket because this ticket shows the direction you are going, and it has a bunch of comments that should probably be left for posterity. Later tickets can build on this foundation.

The LLVM community tends to prefer smaller patches over larger ones. Typically, anyway.

It's an idea. Thoughts?

In D89071#2614095, @kpn wrote:

The majority of your code is still untested as far as I can see. There appear to be three test cases you have so far:

  1. An invalid size for a GOFF object.
  2. A valid size for a GOFF object.
  3. That getSymbolName returns the name of a single symbol in the symbol table.

What about all the rest of the functionality that is included in this patch, including, but certainly not limited to, the following?

  1. More than one symbol in the symbol table.
  2. Other properties of symbols.
  3. The various properties of records.
  4. Relocations.
  5. And so on...

For each bit of code you have written, consider whether a test would fail if that bit of code was broken in some way, or didn't exist. If no test fails, then that code needs a new test case of some form. There may also be other cases where testing is appropriate, e.g. where two separate aspects of the same system interact in some way, although those are harder to judge.

Here's an idea: I found, at least when running in batch, that the Binder (linker) will link an object consisting of nothing more than a HDR card, then ESD cards followed by an END card (meaning, just symbols). It will also link an object consisting of HDR, ESD, TXT, and END cards with zero relocations. Does it work that way when not running in batch? Because if it does then it might make sense to split this ticket up into a new ticket with just support for HDR+ESD+END cards.

That would make this patch smaller, and it would reduce the amount of tests that need to be written to get some initial GOFF support into the tree. The tests that @jhenderson requested would still be needed, but you'd only need the ones that were relevant to the smaller amount of code in the new ticket. A new ticket should refer back to this ticket because this ticket shows the direction you are going, and it has a bunch of comments that should probably be left for posterity. Later tickets can build on this foundation.

The LLVM community tends to prefer smaller patches over larger ones. Typically, anyway.

It's an idea. Thoughts?

I agree this would be a quicker way to get some initial GOFF support in. I can break down this patch to support HDR + ESD + END records, and add the relevant tests mentioned by @jhenderson. I'll add support for the remaining records incrementally.

I created a smaller patch supporting just the HDR, ESD and END records here: https://reviews.llvm.org/D98437. Please continue the review in the new patch.