This is an archive of the discontinued LLVM Phabricator instance.

[llvm-ar] Add object mode option -X for AIX
ClosedPublic

Authored by DiggerLin on Jun 15 2022, 7:56 AM.

Details

Summary
  1. Added a new option object mode -X for llvm-ar. In AIX OS , there is a object mode option -X for ar command.

please see the "-X mode" part of https://www.ibm.com/docs/ko/aix/7.1?topic=ar-command

Specifies the type of object file ar should examine. The mode must be one of the following:
32
Processes only 32-bit object files
64
Processes only 64-bit object files
32_64
Processes both 32-bit and 64-bit object files
any
Processes all of the supported object files.

The default is to process 32-bit object files (ignore 64-bit objects). The mode can also be set with the OBJECT_MODE environment variable. For example, OBJECT_MODE=64 causes ar to process any 64-bit objects and ignore 32-bit objects. The -X flag overrides the OBJECT_MODE variable.

  1. Before adding the new option -X, the default behaviors of llvm-ar like -Xany, but after the adding the new option -X, the default behaviors of llvm-ar change to -X32 ,in order to let some test cases which has 32bit and 64bit object file in the same llvm-ar command, we need to add the "export OBJECT_MODE=any" into test case to change the default behaviors of llvm-ar's object mode.

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
DiggerLin requested review of this revision.Jun 15 2022, 7:56 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 15 2022, 7:56 AM
DiggerLin edited the summary of this revision. (Show Details)Jun 15 2022, 7:57 AM

Out of curious, why do we need to add -X option to llvm-ar like AIX system ar? If llvm-ar can automaticly detect the object file format on any OBJECT_MODE and generate related archive file, seems it is better than system ar?

DiggerLin retitled this revision from [AIX] add object mode -X option for llvm-ar in AIX OS. to [AIX] add object mode -X option for llvm-ar on AIX OS..Jun 15 2022, 11:30 AM

reformat llvm-ar.rst with Sphinx

MaskRay added a comment.EditedJun 15 2022, 1:00 PM

[AIX] add object mode -X option for llvm-ar on AIX OS.

Maybe [llvm-ar] Add object mode option -X for AIX?

The convention does not add a period on the subject line.

Out of curious, why do we need to add -X option to llvm-ar like AIX system ar? If llvm-ar can automaticly detect the object file format on any OBJECT_MODE and generate related archive file, seems it is better than system ar?

There is still an interface compatibility problem without the option, many build system tools (e.g. CMake) will automatically generate the flag on AIX, so it's needed to be able to use llvm-ar as a drop in replacement for the system ar.

DiggerLin retitled this revision from [AIX] add object mode -X option for llvm-ar on AIX OS. to [llvm-ar] Add object mode option -X for AIX.Jun 15 2022, 5:56 PM
DiggerLin edited the summary of this revision. (Show Details)Jun 15 2022, 6:08 PM

The pre-merge checks are showing up failures caused by this patch. Please fix them.

I've not looked at the tests yet.

llvm/docs/CommandGuide/llvm-ar.rst
299

I don't like the word "examine" here. llvm-ar is usually about manipulating archives, as much as (if not more than) inspecting them. How about "will recognise" instead of "should examine"?

308

I don't think "supported" is really necessary, so let's keep the message simple.

llvm/tools/llvm-ar/llvm-ar.cpp
649

I don't know what you mean by "SymbolList" here and why that has anything to do with bitness.

665

Please fix this function name to conform to LLVM coding standards.

670

This would be a more normal way of calling it.

675–676
680

Please fix this function name to conform to LLVM coding standards.

681

Delete the extra blank line.

686–687

Same comment change request as above.

702

Please fix this function name to conform to LLVM coding standards.

This function looks pretty much identical to the one above. Please refactor the code to avoid duplication.

739–740

Does this check need to be inside if (Filter)? It seems like it's unnecessary and you can avoid duplicating the check in the else if by moving it out.

807–808

I don't like this AT ALL. It would seem FAR better if you check the type of the object BEFORE actually doing any work with it. That way, you don't have to undo any modifications that you might have made.

1270

Please make this function name conform to LLVM style.

1328
1330

Is "OBJECT_MODE" an environment variable from AIX ar or is it specific to llvm-ar? If the latter, why do you need it?

1332

Is this the same default value as on AIX? I'd expect "any" to make much more sense as a default (but we should do what AIX ar does).

1391–1392

What's the motivation for making this AIX OS only?

1396

This message does not conform to the LLVM coding standard for error messages. Please fix it.

1399

This message does not conform to the LLVM coding standard for error messages. Please fix it.

Additionally "in no AIX OS" -> "on non-AIX OS"

DiggerLin marked 18 inline comments as done.Jun 16 2022, 10:57 AM
DiggerLin added inline comments.
llvm/tools/llvm-ar/llvm-ar.cpp
649

thanks, change to is64BitSymbolicFile

665

thanks

670

thanks

680

thanks

739–740

good catch, thanks

807–808

if we check the object BEFORE, the file will be opened checking for object mode, and we need to open the file somewhere as NewArchiveMember . otherwise we have to open the "FileName" again.

if we open the "FileName" and keep as NewArchiveMember somewhere, the problem is that if the FileName is archive , it can not be kept in the NewArchiveMember.

1270

thanks

1332

if there is no OBJECT_MODE environment variable set and no -X option in AIX "ar" , The default is to process 32-bit object files (ignore 64-bit objects).
https://www.ibm.com/docs/ko/aix/7.1?topic=ar-command

1391–1392

I think to make -X for all the non AIX OS before, but in the AIX, The default is to process 32-bit object files (ignore 64-bit objects). but for other non AIX OS, the default is to process both 32 and 64 bits object file. it maybe caused problem for the script which llvm-ar.

if you think it is better to let -X work for non AIX OS, I can make
default for AIX is 32bit, for non AIX OS, the default is any.

DiggerLin marked 9 inline comments as done.

address comment

I'm going to leave my review there, but haven't finished the test. Please address my comments and then I'll finish reviewing the test case.

llvm/test/Object/X86/archive-symbol-table.s
1 ↗(On Diff #438015)

I don't like the need for this in every single llvm-ar test. I would prefer it if this variable is set by default in the lit harness, and then just explicitly unset in the test(s) where we want to test the AIX behaviour.

llvm/test/tools/llvm-ar/invalid-option-X.test
3

"Test that the -X option is not supported on non-AIX OS."

6

No need for a check prefix other than the default CHECK when you only have one test case in the file.

7

Nit: there's a trailing space at the end of this line. Please delete it.

llvm/test/tools/llvm-ar/option-X.test
4
6–8
15

That being said, this first test case doesn't test the new option at all. It just uses the default behaviour.

16

No need for this: you haven't created the file yet, and you deleted the directory you are working in.

17

No need for %t when you cd'ed into a directory - just use a name specific to the test case, in this case something like default-create.a (i.e. it's the default behaviour for creating a new archive). Same sort of thing goes for the other test cases too.

Your new code checks for 32 versus 64 for each of a number of different formats, not just XCOFF and ELF. You should test each of them in at least one test case.

18–20

I'd also recommend using INFO-64 i.e. a hyphen instead of an underscore.

Same comments and edits apply throughout.

22–23

Should the second of these use the -NEXT suffix?

31

Add comments explaining what each test case is doing. E.g. here it is saying that the -X option overrides the environment variable.

Additionally, the norm for setting environment variables is to use the env command at the start of the RUN line you want to modify, e.g.

# RUN: env OBJECT_MODE=64 llvm-ar ...
33

This line isn't a RUN line so won't do anything, but regardless, see above.

llvm/tools/llvm-ar/llvm-ar.cpp
662

There's a real risk that if a new object file format is added that this will result in an assertion/crash. It's probably better to do a dyn_cast for ELFObjectFileBase, like the other formats and then report an error (e.g. "unsupported file format" if it is none of the known formats.

695–696

I think this phrasing is a little cleaner.

728
738

Delete this unrelated change.

840

This should probably be printed as a warning (and follow the coding standards for warnings and errors).

Same goes with the other areas reporting this kind of message.

968

@gbreynoo, can you take a look at this area of the change in particular, please, and see if it is the best we can do given how llvm-ar is structured? I'm very rusty on the code logic, so don't feel comfortable saying whether this is the best approach or coming up with a better one.

972
1391–1392

Having thought about it, I think -X should be AIX only, as it's less likely to cause a problem with another short option appearing in GNU ar and causing a name clash.

1396

Use Twine rather than std::string here to perform the concatenation. Convert to std::string only once the whole thing is concatenated (if needed).

DiggerLin updated this revision to Diff 439797.Jun 24 2022, 9:17 AM
DiggerLin marked 17 inline comments as done.
DiggerLin added inline comments.
llvm/test/tools/llvm-ar/option-X.test
15

it test the default option is as -X32 without -X option in command and no "OBJECT_MODE" set in environment.

31

thanks

jhenderson added inline comments.Jun 27 2022, 12:34 AM
llvm/test/Object/lit.local.cfg
1–2 ↗(On Diff #439797)

I'd add a comment explaining this variable briefly.

I also think it would make sense to move this into the top-level lit config, rather than a subdirectory, because it impacts all uses of llvm-ar. llvm-ar is commonly used to create inputs for tests at runtime, and these could appear in virtually any directory within llvm/test (not to mention possibly even other subprojects, such as lld, but I'm less concerned there).

llvm/test/tools/llvm-ar/option-X.test
22–23

Ping this comment? It seems to have been ignored.

24

I said this once already - don't delete archives that don't exist! The directory was deleted and recreated at the start of the test. Use a unique name for each test case too, e.g. don't reuse archive.a in multiple test cases, instead use create-default.a, create-32.a, create-64.a etc (with names that describe the test case). This way you a) don't have to rename the test, and b) can inspect the archive for each test case after running it without having to do strange things like delete parts of the test.

This applies throughout the test.

25

Did you mean to run FileCheck on this line? The 2>&1 implies you either a copy-paste error or that you meant to check for a warning.

29
38

You've still got a double space between %s and the --check-prefixes option here. Please remove it from here and all other cases.

In general, when I make a comment, please make sure you've addressed all occurrences of the issue I raised, rather than just the one, as it will speed up the review and save us both time.

180

Please remember to include a space between your # and the check directive.

201

I think you're trying to say "Test -X option with other output formats." right? You don't need to list them out specifically (the test case is clear enough).

Is there a reason you've got ELF in the earlier test cases, rather than down here? I think it would make the earlier tests simpler to use just one file format in the earlier test cases (probably XCOFF) and then just this simple test for the other ones (including ELF).

If you go down this route, make sure you don't lose test coverage that shows that llvm-ar warns for every object it doesn't process, not just the first one. This could be achieved by checking the warnings in this test case, I believe.

llvm/tools/llvm-ar/llvm-ar.cpp
699–700

I'm not sure I follow the need for this bit of code? Has something else been rebased that requires it, and if so, could you explain why it's correct to use it the way you've done (i.e. with the setOpaquePointers(true), please.

840

llvm-ar.cpp already prints one other warning in this file (see https://github.com/llvm/llvm-project/blob/ca2933f3f88a09eb485146eef776b7345e3d87d3/llvm/tools/llvm-ar/llvm-ar.cpp#L1021). To ensure consistency, we should factor all the call sites into a function that prints the warning. Note that the other case includes the toolname, for example, which this doesn't.

972–976

We should probably move this code into a separate function, so that the message is not repeated in three different cases. That function would then call the proposed generic print warning function suggested above.

DiggerLin marked 13 inline comments as done.Jun 28 2022, 6:24 AM
DiggerLin added inline comments.
llvm/test/tools/llvm-ar/option-X.test
38

thanks , I will do it.

180

thanks

201

Is there a reason you've got ELF in the earlier test cases, rather than down here? I think it would make the earlier tests simpler to use just one file format in the earlier test cases (probably XCOFF) and then just this simple test for the other ones (including ELF).

there is no special reason to got the ELF in the earlier test cases, but it is no harm to ELF in the earlier test case. And if only letting the earlier test cases only test xcoff32.o and xcoff64.o, I think it is too simple, so I put the earlier test cases test on the xcoff32.o elf32.o xcoff64.o elf64.o

If you go down this route, make sure you don't lose test coverage that shows that llvm-ar warns for every object it doesn't process, not just the first one. This could be achieved by checking the warnings in this test case, I believe

when we checking the warning functionality of "is not valid with the current object file mode" , I think we only need to check once, we do not need to check everywhere ? is it correct?

llvm/tools/llvm-ar/llvm-ar.cpp
699–700

we add bitcode object file test in test case ,it need a LLVMContext to createBinary for bit code.
https://github.com/llvm/llvm-project/blob/ca2933f3f88a09eb485146eef776b7345e3d87d3/llvm/lib/Object/SymbolicFile.cpp#L48

DiggerLin updated this revision to Diff 440614.Jun 28 2022, 7:13 AM
DiggerLin marked 4 inline comments as done.

address James' comment.

DiggerLin added inline comments.Jun 28 2022, 1:45 PM
llvm/test/tools/llvm-ar/option-X.test
201

And if only letting the earlier test cases only test xcoff32.o and xcoff64.o, I think it is too simple to test especially when testing move operation.

jhenderson added inline comments.Jun 30 2022, 12:38 AM
llvm/test/Object/lit.local.cfg
1–7 ↗(On Diff #440614)

Delete the blank line between the comment and block of code it refers to.

I've also fixed a number of grammar errors, typos, etc, and removed all references to the command-line option from the comment, in the suggested edit: the code this comment is supposed to explain is about the environment variable only, and the command-line option is irrelevant to it (users reading this comment, don't need to know about a command-line option to understand why this code exists).

1–2 ↗(On Diff #439797)

I also think it would make sense to move this into the top-level lit config, rather than a subdirectory, because it impacts all uses of llvm-ar. llvm-ar is commonly used to create inputs for tests at runtime, and these could appear in virtually any directory within llvm/test (not to mention possibly even other subprojects, such as lld, but I'm less concerned there).

You've ignored this. Please stop ignoring parts of my comments without explaining why at least.

llvm/test/tools/llvm-ar/option-X.test
14

You only test the default behaviour for archive creation, but you should also do it for the operations you test, I think.

15

Why have you dropped the warning checks from these cases?

57–58

Nit: In the -X cases, the "any" case comes before the 32_64 case. I don't mind the order, but it should be the same in all cases, for ease of review and maintenance.

72

Rather than repeatedly delete the objects, I think you should run each of these cases in a different directory (use mkdir and cd as needed), or use the new --output option added very recently (see rGbf223e43fe68) to specify a different directory each time.

74

The more common way to test for existence of a file is ls xcoff32.o. That being said, if you keep the original objects as described above, you could use cmp xcoff32.o ../xcoff32.o (or equivalent) instead for the cases where the file exists (you'd still want not ls for the missing cases).

100

You should also test the 32-bit object using -X64 case.

110
126
127

Why are you deleting objects here?

129–133

I think you should avoid this confusing situation by using a completely fresh set of objects and an archive created specifically for this test case. You could call the objects 1.o, 2.o etc, to help show the expected order. You should be able to reuse your earlier objects by simply copying them.

134

I'm not sure I understand the need for the u option in these test cases?

140–142
149
151
153

Prefer - to _ in check prefixes.

155

This should use the %errc_ENOENT lit substitution.

160

You need 32-bit equivalents of these test cases, and also to show the behaviour with the other -X options.

192–193

Why not just 32.bc and 64.bc?

201

Okay.

llvm/tools/llvm-ar/llvm-ar.cpp
840

You should quote FileName, so that it is clear if there's a space in the name.

972–976

This bit hasn't been addressed. I'm suggesting a new function like this:

static void warnInvalidObjectForFileMode(Twine Name) {
  warn(Name + " is not valid with the current object file mode");
}

To avoid repeating that string in multiple places.

DiggerLin updated this revision to Diff 442939.Jul 7 2022, 9:04 AM
DiggerLin marked 22 inline comments as done.

address comment

llvm/test/Object/lit.local.cfg
1–7 ↗(On Diff #440614)

thanks

llvm/test/tools/llvm-ar/option-X.test
14

the code

if (object::Archive::getDefaultKindForHost() == object::Archive::K_AIXBIG) {
   BitMode = getBitMode(getenv("OBJECT_MODE"));
   if (BitMode == BitModeTy::Unknown)
     BitMode = BitModeTy::Bit32;
 }

is dependent of operations , so I think we only need to test default behavior to -X32 once at here is OK.

and I also using default behavior for the

  1. Extract a 64-bit object file with option -X32(or default object mode).
15

I do not think we need to check the warning for each test. I tested the warning in the line 243~246 (etc), I do not think we need to test warning in each case.

22–23

I am sorry that I do not get the comment.

127

we need to delete all *.o before we test extract operation. I can delete the " rm -rf *.o" since I use --output for the extract operation.

129–133

here we need a same file xcoff32.o which is a 32 bit member is archive, when we want to replace with a 64 bit xcoff32.o and without option -X32, not change anything

134

yes , we do not need u here.

160

I added a -X32 of the test case , but I do not think I need to add additional -X32_64 -Xany for the "-ma" ,since we use the same
function

static bool isValidInBitMode(Binary &Bin)

for operation,
I have test -X32-64 and -Xany in other operation case.

jhenderson added inline comments.Jul 11 2022, 1:13 AM
llvm/test/lit.local.cfg
1 ↗(On Diff #442939)

I guess I should have been more specific - top-level lit configuration goes in llvm/test/lit.cfg.py, not in a lit.local.cfg in the top level.

2–5 ↗(On Diff #442939)
7 ↗(On Diff #442939)

Delete this blank line (as requested before - the logic is still the same).

llvm/test/tools/llvm-ar/option-X.test
78

Nit: there's a double space here.

95–96

I'm not sure why you've added the -s option to the cmp commands. We want the output, so that if the test fails, we'll get told where the problem is in the test log.

118
127
154
158

I already suggested this fix once...

160

If the code changed to use a different function for the move case, then there would not be test coverage for it, hence why you should test all four cases, I think.

163

This is the canonical option for dumping the archive symbol table. If that's not what you meant to do, could you explain what you are trying to check with this line, please?

Same below.

179

Comments end with a ".".

Previously you had a test case that showed the if you tried to perform an operation on an object which didn't match the -X option, you'd get a "file not found" error. Whilst I accept that it probably follows the same code paths as other cases already tested, I think it's an important enough behaviour that you should have a specific test case for it somewhere (you don't need to test both 32 and 64 bit variations for that case though, I think).

181

Not sure why you ignored parts of my original edit...

182

Ditto below.

188
llvm/tools/llvm-ar/llvm-ar.cpp
711

Name still isn't quoted in the output. Please fix (and don't mark comments as "Done" if you haven't done them).

DiggerLin marked 12 inline comments as done.Jul 12 2022, 1:45 PM
DiggerLin added inline comments.
llvm/test/tools/llvm-ar/option-X.test
163

there are two version of xcoff.o , one is 32-bit, other one is 64-bit,

the 32-bit xcoff.o has symbol var_0x1DF
the 64-bit xcoff.o has symbol var+0x1F7.

I want to check whether the only a 32-bit xcoff.o in the archive or both 32-bit and 64-bit xcoff.o in the archive, I need to check the symbol.

179

in original, the test scenario look like

  1. RUN: llvm-ar -ma elf64.o archive.a xcoff64.o 2>&1 | \
  2. RUN: FileCheck %s --check-prefix=MOVE_ERR
  3. MOVE_ERR: llvm-ar: error: xcoff64.o: No such file or directory

it because we run "rm -rf *.o" before the test scenario in original.
so when run llvm-ar -ma elf64.o archive.a xcoff64.o ,it will return
"xcoff64.o: No such file or directory"

but current we do not run "rm -rf *.o" before the

#RUN: llvm-ar -ma elf64.o archive.a xcoff64.o

there is xcoff64.o in the directory.
it return warning: 'xcoff64.o' is not valid with the current object file mode

I think we only to test the -X option work with -ma , so we need to xcoff64.o in the directory.

and from the logic for the llvm-ar.cpp it will check object file in the command line whether it exist first if we want to move the object file. it is logic is not new in the patch, so I do not think we need to test it.

181

sorry for the miss some of comment even I go the comment one by one.

DiggerLin updated this revision to Diff 444109.Jul 12 2022, 6:38 PM
DiggerLin marked 3 inline comments as done.

address comment

jhenderson added inline comments.Jul 13 2022, 1:03 AM
llvm/test/lit.cfg.py
481–487

Delete the blank line at the end of this comment. Also, some of the line wrapping is a bit off, so I recommend reflowing it.

llvm/test/tools/llvm-ar/option-X.test
78

Not addressed.

179

Thanks for the explanation - I didn't realise that the missing file was the one being added, which is indeed not something to do with this patch.

What happens if you try to move after a file with a bitness that doesn't match the object mode? That probably needs testing. I think there are two interesting cases: a) if there is only one object with the relevant name, and b) if there are two objects with the same name, but different bitness, where only the second one matches the object mode. The second case is unnecessary, if the first case's behaviour is that the object is moved to the right place anyway, even though the referenced object doesn't have the right object mode.

Outline example - update names of files as appropriate etc:

# Case 1:
llvm-ar rc -Xany archive.a xcoff64.o elf32.o elf64.o
llvm-ar ma -X64 elf32.o archive.a xcoff64.o

# Case 2, if case 1 is "elf32.o is treated as not present". First elf.o is 32-bits, second is 64.
llvm-ar rc -Xany archive.a xcoff64.o elf.o elf.o
llvm-ar ma -X64 elf.o archive.a xcoff64.o
DiggerLin updated this revision to Diff 444775.Jul 14 2022, 1:24 PM
DiggerLin marked 3 inline comments as done.
DiggerLin added inline comments.Jul 14 2022, 1:29 PM
llvm/test/tools/llvm-ar/option-X.test
179

good catch, thanks. I add two more tests for it.

182

I change elf64.o --> elf32.o , otherwise it will return "error: insertion point not found"
that is not our test purpose for the scenario.

jhenderson added inline comments.Jul 18 2022, 12:44 AM
llvm/test/tools/llvm-ar/option-X.test
247

Typo in the file name? I assume it shouldn't be "xcoff64elf64.o"!

257–258
261

There's only one 32-bit xcoff.o. The "first" implied there was more.

273
DiggerLin updated this revision to Diff 445478.Jul 18 2022, 6:42 AM
DiggerLin marked 4 inline comments as done.
DiggerLin added inline comments.
llvm/test/tools/llvm-ar/option-X.test
247

yes. it is Typo. I change to xcoff64.o,thanks.

jhenderson accepted this revision.Jul 19 2022, 12:34 AM

Looks good from my point of view, but I'd like @gbreynoo to review the area highlighted earlier before this lands.

This revision is now accepted and ready to land.Jul 19 2022, 12:34 AM
gbreynoo added inline comments.Jul 19 2022, 4:08 AM
llvm/tools/llvm-ar/llvm-ar.cpp
982

Can use warnInvalidObjectForFileMode.

I think my comment got eaten. Sorry for the delay @jhenderson. I've had a look, particularly at the section highlighted, and I this is the best place for this check. The best I can suggest is the use of a lambda or new function to tidy the IA_AddNewMember and IA_MoveNewMember cases. Something like:

auto handleNewMembers = [&](auto Members, auto MemberI, auto Child) {
  NewArchiveMember NM = getArchiveMember(*MemberI);
  if (isValidInBitMode(NM))
    addMember(Members, NM);
  else {
    // If a new member is not a valid object for the bit mode, add the old
    // member back.
    warnInvalidObjectForFileMode(*MemberI);
    addChildMember(Members, Child,
                   /*FlattenArchive=*/Thin);
  }
};

So the calls sites are:

case IA_AddNewMember: {
  handleNewMembers(Ret, MemberI, Child);
} break;
case IA_MoveNewMember: {
  handleNewMembers(Moved, MemberI, Child);
} break;

How much clearer this is is debatable.

DiggerLin updated this revision to Diff 446236.EditedJul 20 2022, 12:42 PM

address comment with a lambda function @gbreynoo

Thanks that LGTM.

This revision was landed with ongoing or failed builds.Jul 22 2022, 6:55 AM
This revision was automatically updated to reflect the committed changes.

@DiggerLin
Could you check that llvm-ar can be built? I have a link-time error:

ld.lld: error: undefined symbol: llvm::MachO::is64Bit(llvm::MachO::Architecture)
>>> referenced by llvm-ar.cpp
>>>               tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.o:(isValidInBitMode(llvm::object::Binary&))

The symbol is defined in TextAPI component which llvm-ar does not depend on.