Page MenuHomePhabricator

[SystemZ/z/OS] Add GOFFObjectFile class and details of GOFF file format
Needs ReviewPublic

Authored by yusra.syeda on Oct 8 2020, 1:56 PM.

Details

Summary

This patch details the GOFF file format and implements the GOFFObjectfile class.
This patch uses https://reviews.llvm.org/D88741

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@jhenderson the comments you left have been addressed. Are there any other suggestions you have?

Sorry for the delay. This isn't high on my priorities, and I don't have the knowledge to properly review the file format details.

You will need some form of testing, possibly unit testing before this is ready to be put in. Also, please make sure to run clang-format on all new code, so that it conforms to LLVM style guidelines.

llvm/include/llvm/BinaryFormat/GOFF.h
6–7

This doesn't look to me to be the current license header. Please update to match the current version.

41

If this is supposed to be a byte, shouldn't it just be constexpr uint8_t PTVPrefix = 0x03?

llvm/include/llvm/Object/GOFF.h
4

This file is also using an outdated license.

106
164
llvm/include/llvm/Object/GOFFObjectFile.h
4

Update license.

35

Delete redundant blank line.

52

Isn't all of this "GOFF specific"? The class is GOFFObjectFile...

54

Why is this returning a std::error_code not an Error?

llvm/lib/Object/GOFFObjectFile.cpp
48

As stated before, avoid using error_code if possible. That includes functions like errorCodeToError which just convert from it. Instead, prefer functions like createStringError, which allow you to give more contextual error information to the user.

83
87–88
117–118
132
137

This sounds like it should be an error?

153

These sorts of comments don't add anything, in my opinion. Just delete them (the function names describe things sufficiently).

158

Add a blank line between functions.

162

Why convert between the two when you could just use uint8_t throughout?

165
166

What limits this to being specifically uint16_t in size?

168
171
181–182

What happens if the data is truncated?

263

Do you mean specifically " " means local? What about "", " " (i.e. 0, 2 spaces) etc?

283–288

Could this assertion fire if somebody wrote garbage in their object file's symbol type field? If so, it should be an error, not an assertion. (Use Error/Expected for malformed input and assertions for coder errors within LLVM).

297–300

Same question as above - should this be an error in case of malformed input?

339

Is this really unreachable? What happens if there is no symbol section in the file?

357

Blank line between functions. Same goes below.

377
380–381

Use range-based for loop here and below.

388
397

Same comment as earlier. Why not stick to uint8_t everywhere?

398
405
417

Add blank line between functions here and below.

453
kpn added a comment.Wed, Nov 11, 8:23 AM

OK, I'll bite. I do know GOFF, having implemented it in a shipping, commercial compiler before. Give me some time to take a closer look.

llvm/lib/Object/GOFFObjectFile.cpp
166

I believe the maximum record length even with continuations is 32KB. I don't know if saving two bytes of stack is worth making people reading the code doubletake, though.

A check to make sure this 32KB limit is not exceeded is needed.

kpn added inline comments.Wed, Nov 11, 9:38 AM
llvm/lib/Object/GOFFObjectFile.cpp
50–51

Hmm, @yusra.syeda, your response is mostly correct, but not entirely.

It's true that the Unix-style filesystem (IBM calls it the "Hierarchical File System", with the first implementation of it being called the "Hierarchical File System" and the second "z/FS") has no record support because Unix doesn't support records in files. Thus the 80-byte requirement on GOFF record sizes. This is true.

But there's no requirement that a program started under USS only access the Unix-style filesystem. There's no requirement that a program started under TSO or in batch only access traditional MVS datasets. Indeed, JCL even has support for Unix paths in DD statements, and TSO probably does as well (but I don't have my book handy).

So the compiler "running in USS" does _not_ mean that we are restricted to 80-byte GOFF records. Granted, disambiguation of Unix paths and MVS dataset names is a problem, but still.

I understand why you would want to leave variable sized records for implementation later. Not implementing support for MVS datasets up front is one thing. Designing your code to make it difficult if not impossible to add later is quite different. For example, random access to variable size record datasets is painful.

Are you at least looking ahead to adding RECFM=V support later?

Address some formatting comments

yusra.syeda marked 25 inline comments as done.Thu, Nov 12, 1:23 PM
jhenderson added inline comments.Thu, Nov 12, 11:59 PM
llvm/lib/Object/GOFFObjectFile.cpp
87

(reminder - comments must end in a full stop)

342

When I said the following to SymbolRef above:

These sorts of comments don't add anything, in my opinion. Just delete them (the function names describe things sufficiently).

That wasn't referring to just the one comment. Please delete all of these sort of comments.

398

Please address both edits suggested in the previous comment, not just the second one.

405

Ditto.

yusra.syeda added inline comments.Fri, Nov 13, 9:38 AM
llvm/lib/Object/GOFFObjectFile.cpp
50–51

@kpn we don't have plans on adding support for variable length GOFF records. The XL compiler supports only 80 byte records and we don't plan to add support further than what exists in the XL compiler.

Address formatting comments

yusra.syeda marked 7 inline comments as done.Fri, Nov 13, 1:29 PM
kpn added inline comments.Fri, Nov 13, 2:35 PM
llvm/include/llvm/BinaryFormat/GOFF.h
17–18

I can't find any document with Google that describes a GOFF-specific TIS, and Google also has trouble with "GOFF64" and "GOFF-64". Those version numbers I guess refer to that missing document.

Can you instead reference the exact book from IBM's "z/OS Internet Library" that describes GOFF? Include the full name, the edition number, and the year please. Then it's easy to look up the GOFF spec.

llvm/include/llvm/BinaryFormat/GOFFAda.def
10

Please expand "ADA" here at the top. This appears to be for "Extended Attributes"?

llvm/include/llvm/Object/GOFF.h
138

Missing module properties field.

157

I don't see any support for "Text Encoding". These fields are at bytes 16-21. Maybe a comment if you don't plan on ever implementing it?

224

Is this right? Bit 41.3 in "Program Management" is the "Removable Class Flag", which matches the code above this. Bit 41.4-6 are marked reserved, and bit 41.7 is unnamed but if set means "Reserve 16 bytes at beginning of class. MRG class ED records only."

So it looks like the code is writing to the wrong part of byte 41. The code that reads from that byte also appears incorrect. Shouldn't it be (41, 4, 3)?

228

This is the "Associated data ID". It's confusing having "Ada" and both "ADA" but I don't think they're related?

267

No COMMON flag?

488

I assume RLD continuation records and relocation compression are coming later.

531

For completeness this is good. But please don't ever use it.

When the Binder abends I can't tell you how useful it is to use the Unix "dd" command to slice up GOFF until the abend goes away. That's how I've had to shoot down a number of bugs in emitting GOFF. But that technique doesn't work if the END card's record count field is used.

llvm/lib/Object/GOFFObjectFile.cpp
263

It's " " that's special. My employeer's compiler uses the same symbol name because the Binder translates it into a private symbol name that uses characters. In this way multiple private symbols can be disambiguated in a listing after a link.

kpn added a reviewer: kpn.Fri, Nov 13, 2:36 PM
yusra.syeda marked 5 inline comments as done.

Change errorCodeToError to createStringError in GOFFObjectFile constructor

yusra.syeda marked an inline comment as not done.Tue, Nov 17, 8:36 AM
yusra.syeda added inline comments.
llvm/lib/Object/GOFFObjectFile.cpp
166

Thanks, I will add that check.

339

Yes, this should be unreachable and is added as an extra safety check.

Clean up some cast statements from const uint8_t * to const char *

yusra.syeda marked 2 inline comments as done.Tue, Nov 17, 1:56 PM
jhenderson added inline comments.Wed, Nov 18, 1:12 AM
llvm/lib/Object/GOFFObjectFile.cpp
47

Be more verbose with your error messages, so that they provide more useful context. See https://llvm.org/docs/CodingStandards.html#error-and-warning-messages for full details.

MaskRay added inline comments.Wed, Nov 18, 5:45 PM
llvm/lib/Object/GOFFObjectFile.cpp
323
338

llvm_unreachable does not need a return

Add check for ESD name length field size
Update loops to comply with LLVM coding standard:
https://llvm.org/docs/CodingStandards.html#don-t-evaluate-end-every-time-through-a-loop

yusra.syeda marked 3 inline comments as done.Mon, Nov 23, 11:40 AM

Remove return after llvm_unreachable statement

yusra.syeda marked 2 inline comments as done.Mon, Nov 23, 11:58 AM

Remove setERSymbolType and getERSymbolType functions

yusra.syeda marked an inline comment as done.Mon, Nov 23, 3:13 PM
yusra.syeda added inline comments.
llvm/include/llvm/Object/GOFF.h
224

The setERSymbolType and getERSymbolType functions have been removed from this patch as they are not required.

yusra.syeda marked an inline comment as done.Mon, Nov 23, 3:14 PM
yusra.syeda marked 4 inline comments as done.Tue, Nov 24, 12:50 PM
yusra.syeda added inline comments.
llvm/include/llvm/Object/GOFF.h
138

This field will be added in a future patch.

157

This will also be added in a future patch.

267

Same with this field.

488

Yes, these will be coming in a future patch.

yusra.syeda marked 4 inline comments as done.Tue, Nov 24, 12:50 PM