This is an archive of the discontinued LLVM Phabricator instance.

[llvm-libtool-darwin] Add support for -static option
ClosedPublic

Authored by sameerarora101 on Jul 1 2020, 3:09 PM.

Details

Summary

Add support for creating static libraries when the input
includes only binaries (and not libraries/archives themselves)
Depends on D82923

Diff Detail

Event Timeline

sameerarora101 created this revision.Jul 1 2020, 3:09 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 1 2020, 3:09 PM
sameerarora101 marked an inline comment as done.Jul 1 2020, 3:21 PM
sameerarora101 added inline comments.
llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
48–62

5 There are some basic error handling functions available in the Support library which might do all you need them to. Consider using them before rolling your own.

These error functions themselves are using basic error handling functions available in the Support library (e.g logAllUnhandledErrors). If I use createStringError available in the support library, I too would have to pass the error object as an argument to logAllUnhandledErrors. Wouldn't I be doing something similar to this only then (I copied these error functions from llvm-lipo) ? sorry if I missed something. Thanks

Out of time. Note to self: finish review of the rest of the source code and tests.

llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
4–5

I assume the idea is that these two YAML files will be used by many different tests in the future? If so, I'd rename them simply to input1.yaml and input2.yaml to emphasise their generality (in particular, the main bit suggests it is significant, which I suspect is not true). I'd probably also rename their symbols to symbol1 and symbol2 for the same reason.

11

t is probably sufficient rather than tv since you don't check the other fields.

15

Use -NEXT suffixes where applicable.

19

--implicit-check-not=T sounds flaky to me, as file paths might start appearing in the output. It also won't indicate that there are no other symbols, merely no other global text symbols. What is the actual output here? Also, are you interested in the archive's symbol table or the symbol table of the members, because I believe this might be printing the latter. There's a --print-armap option to do the former.

26

Nit: one more space of indentation, please.

Again, I think the --implicit-check-not sounds dangerous to me. You might be better off with something like FORMAT-EMPTY: or FORMAT-NOT: {{.}} after the last check line.

28

Add some extra spacing to make this line up:

# FORMAT:      __.SYMDEF
# FORMAT-NEXT: hello.o
# FORMAT-NEXT: main.o
33–37

Similar comments apply here to the lines above.

Also, I feel like showing that the ordering has been reversed is a little subtle. Assuming the tool completely replaces the original archive, I'd add just a single object to the new one, and show that the output only has that one object in it.

46–49

Same comments as above, re. --implicit-check-not, symbol table etc.

llvm/test/tools/llvm-libtool-darwin/invalid-arguments.test
1 ↗(On Diff #274936)

I won't review this now, as I think some of it belongs in the first review, plus I'm running low on time for upstream reviewing today!

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
39

Can I confirm that the terminology actually is "statically linked library" in Darwin land? I'd typically use simply "static archive" from my ELF background.

41

Is "shared dynamic library" unnecessarily verbose, or is that the common terminology? I'd expect either "shared library" or "dynamic library" from my background, but am not familiar with Darwin development.

44–46

Can this option be delayed to a later change? I'd tentatively do it by default in this version.

FWIW, I think LLVM tools typically set deterministic to true by default, since it makes testing easier.

48–62

I was referring to the WithColor::defaultErrorHandler function, which does the actual printing of the error message and handles the Error. There's also `createFileError which takes an Error and adds a provided file name. You will still need to convert the string to an error using createStringError, but could then pass that directly to defaultErrorHandler.

69

Nit: incorrect comment format. (Use capital letter/full stop).

70–71

More idiomatic and simpler is:

if (!ObjOrErr)
  reportError(Member.MemberName, ObjOrErr.takeError());
73

Ditto.

102–103

Rather than add this as an unsupported option, I'd just only permit the static mode for now.

sameerarora101 marked 23 inline comments as done.Jul 2 2020, 11:49 AM
sameerarora101 added inline comments.
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
19

you are right, --print-armap is what I wanted. It prints:

Archive map
_symbol1 in create-static-lib.test.tmp-input1.o
_symbol2 in create-static-lib.test.tmp-input2.o


/data/users/<complete-path>/create-static-lib.test.tmp.lib(create-static-lib.test.tmp-input1.o):
0000000000000000 T _symbol1

/data/users/<complete-path>/create-static-lib.test.tmp.lib(create-static-lib.test.tmp-input2.o):
0000000000000000 T _symbol2

I have now replaced the check with :

# CHECK-SYMBOLS:      _symbol1 in
# CHECK-SYMBOLS:      _symbol2 in
# CHECK-SYMBOLS-EMPTY:

Would this be fine?
Thanks

33–37

alrighty, replaced the test by adding a single object to it.

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
39

Yup, I took this from the man pages (https://www.unix.com/man-page/osx/1/libtool/):

Libtool can create either dynamically linked shared libraries, with -dynamic, or statically linked (archive) libraries, with -static.

41

For the dynamic option the man page says:

-dynamic
Produce a dynamically linked shared library from the input files.

But I can update our description to something else too ?

(PS: I have removed the dynamic option for this version)

44–46

ok, I'll set it to true for now as well (and pass it as a default instead of creating an option).

70–71

thanks.

102–103

ok, removed dynamic option above.

sameerarora101 marked 7 inline comments as done.

Updating test and source files as per the comments:

  • removing dynamic option
  • using createStringError and createFileError
  • removing --implicit-check-nots from the test

(No time to look at tests today, sorry)

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
63

I'd expect deterministic to be capitalized in the function signature. If that is the case, this comment should match.

77

You probably don't need this blank line.

83

Let's not use auto here, as it's not immediately clear what the type of Member is.

88–91

Same comment as above re. capitalization in the comments.

98

What is ToolName used for?

102

return EXIT_FAILURE maybe? What do other tools typically do?

sameerarora101 marked 8 inline comments as done.Jul 5 2020, 2:38 PM
sameerarora101 added inline comments.
llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
98

Nothing anymore. Previously, it was being used for the error message. I have removed it now

102

llvm-objcopy does return 1. However, llvm-lipo does exit(EXIT_FAILURE) whereas some other tools (llvm-ar, llvm-as) use exit(1). I have replaced it with exit(EXIT_FAILURE) for now. Is it ok?

sameerarora101 marked 2 inline comments as done.

Rebasing and incorporating inline comments

This comment was removed by sameerarora101.
Harbormaster completed remote builds in B62946: Diff 275578.
jhenderson added inline comments.Jul 6 2020, 1:41 AM
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
16

Watch out, here and in other cases, this only shows that there is no output AFTER input2.o. It's possible that there's input before input1.o. Also, you'll not catching name corruptions resulting in a prefix/suffix of the name, since FileCheck only does sub-string matching by default, not full line matching. For example, this would fail if the following was emitted:

input-I-really-shouldn't-be-here
input1.osuffix
prefixinput2.o

You probably want to add --implicit-check-not={{.}} to the FileCheck command line, rather than the final CHECK-NAMES-NOT.

20

It would be best to check the symbol is in the right member. You can do this by using FileCheck's -D option, combined with the %t_basename (NB: check the exact name for correctness, but there should be other examples):

# RUN: ... FileCheck %s -D FILE1=%t_basename ...

# CHECK-SYMBOLS: _symbol1 in [[FILE1]]

and so on. This defines a string variable that matches the specified input string, and can be used using the [[VAR]] syntax as shown. Take a look at the FileCheck documentation for details.

Also, you should check the Archive map string to ensure there's no symbol before the first.

23

Use CHECK-SYMBOLS-NEXT here to show there's no symbol in between the two symbols.

42–43

This has the same issue as the earlier comment re. CHECK-NOT/--implicit-check-not

In fact, if you used OVERWRITE-NAMES and OVERWRITE-SYMBOLS instead of CHECK-NAMES/CHECK-SYMBOLS above, the test will still pass.

45–46

Same as above.

55–63

Same comments here as above.

llvm/test/tools/llvm-libtool-darwin/invalid-arguments.test
7 ↗(On Diff #275578)

Does the double space match the actual error message?

9–18 ↗(On Diff #275578)

These two checks plus the ELF one below probably belong in the invalid input/output arguments test.

20 ↗(On Diff #275578)

Does this message use error: as a prefix?

24 ↗(On Diff #275578)

Did you mean to use $t.lib? (probably not)

29–34 ↗(On Diff #275578)

Only these lines are required to emit a minimal ELF object. Delete everything else. Also, we prefer to use minimal amounts of padding:

--- !ELF
FileHeader:
  Class:   ELFCLASS64
  Data:    ELFDATA2LSB
  Type:    ET_REL
  Machine: EM_X86_64
llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
35–39

I'm not really familiar with the Operation type. What does it look like in the help text?

sameerarora101 marked 18 inline comments as done.Jul 6 2020, 11:05 AM
sameerarora101 added inline comments.
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
16

Thanks, I added --implicit-check-not. Furthermore, I also added -DPREFIX=create-static-lib.test.tmp as the file names are represented by [[PREFIX]]-input1.o and [[PREFIX]]-input2.o

20

Yup, I now check that the symbol is in the right member using the following:

## Check that symbols are present:
# RUN: llvm-nm --print-armap %t.lib | \
# RUN:   FileCheck %s --check-prefix=CHECK-SYMBOLS -DPREFIX=create-static-lib.test.tmp

# CHECK-SYMBOLS:      Archive map
# CHECK-SYMBOLS-NEXT: _symbol1 in [[PREFIX]]-input1.o
# CHECK-SYMBOLS-NEXT: _symbol2 in [[PREFIX]]-input2.o
# CHECK-SYMBOLS-EMPTY:

Would this be ok?

llvm/test/tools/llvm-libtool-darwin/invalid-arguments.test
7 ↗(On Diff #275578)

Yes, the actual error msg also has the double space:

Library Type:  option: must be specified at least once!
9–18 ↗(On Diff #275578)

Ok, I have placed all 4 tests into invalid-input-output-args.test now. Please lemme know in case we needed a separate test file for the first test above ## Missing -static option:

20 ↗(On Diff #275578)

yup, I added error: in the error message too now, thanks!

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
35–39

this is what help text looks like:

OVERVIEW: llvm-libtool-darwin

USAGE: llvm-libtool-darwin [options] <input files>

OPTIONS:

Color Options:

  --color             - Use colors in output (default=autodetect)

Generic Options:

  --help              - Display available options (--help-hidden for more)
  --help-list         - Display list of available options (--help-list-hidden for more)
  --version           - Display the version of this program

llvm-libtool-darwin options:

  -o                  - Alias for -output
  --output=<filename> - Specify output filename
  Library Type: 
      --static           - Produce a statically linked library from the input files

I created an enum Operation here so that in future we can add support for dynamic operation easily. I can very well make the -static option a boolean flag as well. What do you think?

sameerarora101 marked 6 inline comments as done.

Updating tests

sameerarora101 marked an inline comment as done.Jul 6 2020, 12:41 PM
sameerarora101 added inline comments.
llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
78

@jhenderson Should I replace the type of Member back to auto. clang-tidy raises a warning with StringRef?

jhenderson added inline comments.Jul 7 2020, 12:33 AM
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
12

It's not widely used, but there is %basename_t which substitutes for just the file name part of %t. This allows the test to not make assumptions about how %t expands, and also keeps the check independent of the test name. I'd recommend trying that.

19

Mostly fine, but use %basename_t here too.

35

Checking -> Check

38

Here and below, use %basename_t too, to avoid baking in file names.

45

Ditto.

51–53

Ditto.

llvm/test/tools/llvm-libtool-darwin/invalid-arguments.test
7 ↗(On Diff #275578)

Okay, bonus marks for fixing it in another patch if you want. Or file a bug against the CommandLine code.

llvm/test/tools/llvm-libtool-darwin/invalid-input-output-args.test
27

Maybe make this more generic, e.g. "Missing library type option:"

I don't really know where to put this test. It might want to be its own test case entirely, e.g. "missing-library-type.test"

44

Use %basename_t here too.

51

Use %basename_t here too.

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
23

Adding the categories sounds like a different change to me? You might want to include it alongside a test case to show that unrelated options aren't incldued.

35–39

Your current approach seems fine. I was just making sure it didn't do anything too weird.

78

Well typically you don't want to use a const & for StringRef because it is already a light-weight construct (much the same as you wouldn't use const & to a pointer or primitive type). That is probably the cause of the problem here.

98

See above re. option category change.

sameerarora101 marked 19 inline comments as done.Jul 7 2020, 8:18 AM
sameerarora101 added inline comments.
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
12

thanks, I now have -DPREFIX=%basename_t.tmp as %basename_t substitutes for the last path component of %t but without the .tmp extension

llvm/test/tools/llvm-libtool-darwin/invalid-arguments.test
7 ↗(On Diff #275578)

😂I can fix it in another patch.

llvm/test/tools/llvm-libtool-darwin/invalid-input-output-args.test
27

Isn't it similar to ## Missing output file? (That's why I thought it would go here). But I have placed it in missing-library-type.test for now.

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
23

yup, I added OptionCategory in order to prevent the general option --safepoint-ir-verifier-print-only from showing up in the help text. Added a test case for it as well. Thanks

78

ok thanks

sameerarora101 marked 5 inline comments as done.

Use %basename_t and add test for unrelated options.

smeenai added inline comments.Jul 7 2020, 6:20 PM
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
19

Might wanna add --match-full-lines here, to ensure there's no unexpected symbol name prefix or member name suffix.

33

You have an underscore instead of a dash :)

Is the purpose to ensure that there's no other members? I assume a -EMPTY would work for that. We should also check for the "Archive : " header, to ensure there's no members before the table of contents member.

48

Might be worth noting that cctools libtool warns in this case, and we don't implement that warning yet.

llvm/test/tools/llvm-libtool-darwin/hide-unrelated-options.test
1 ↗(On Diff #276081)

This seems unrelated to this diff; perhaps it should be in the previous one?

llvm/test/tools/llvm-libtool-darwin/invalid-input-output-args.test
1

Can you add -static to all of the tests in this file, so that the only invalid part of the command line is the aspect being tested?

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
23

Yup, this should either go in the previous diff or be its own diff.

Mostly looks good, barring @smeenai's comments.

llvm/test/tools/llvm-libtool-darwin/invalid-input-output-args.test
27

Similar in concept, but different behaviour (it's checking the behaviour of the cl::Required on the Library operation option, which is a different thing to the input and output files).

llvm/test/tools/llvm-libtool-darwin/missing-library-type.test
4–6

I don't think it's necessary to add --strict-whitespace here, as the exact formatting of the message isn't really the tool's responsibility. I'd get rid of the double space, as if the message was correct, and the option, so that you don't have to modify this test when you fix the command line message.

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
78

You can get rid of the const too if you want. StringRef itself is immutable, so all this does is stop you re-assigning Member.

sameerarora101 marked 11 inline comments as done.Jul 8 2020, 7:59 AM
sameerarora101 added inline comments.
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
33

Thanks for catching the underscore.

I added a check for "Archive : " header now. However, using FORMAT-EMPTY would just check that the next line (after 2nd member) has nothing on it. What I thought we wanted to check was that there is nothing at all after the second member. For eg,

Archive : ...
... __.SYMDEF
...<prefix>-input1.o
...<prefix>-input2.o

something here

would pass with FORMAT-EMPTY just below the check for second member, right? But we want it to fail.

llvm/test/tools/llvm-libtool-darwin/hide-unrelated-options.test
1 ↗(On Diff #276081)

Yup, thanks. Moved it to the previous diff D82923

sameerarora101 marked 2 inline comments as done.Jul 8 2020, 7:59 AM

Updating tests

smeenai added inline comments.Jul 8 2020, 8:04 PM
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
33

You're right, that makes sense.

smeenai accepted this revision.Jul 8 2020, 8:06 PM

LGTM, but please wait for @jhenderson too.

This revision is now accepted and ready to land.Jul 8 2020, 8:06 PM
jhenderson accepted this revision.Jul 9 2020, 12:26 AM

LGTM, with one optional suggestion.

llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
52

It might be worth adding a 2&>1 and check the output is empty here, to flag up if a warning starts getting emitted. That way, it points to where to add testing for the warning.

sameerarora101 marked 2 inline comments as done.Jul 9 2020, 8:43 AM
sameerarora101 added inline comments.
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
52

ok, added

# RUN: llvm-libtool-darwin -static -o %t.lib %t-input1.o %t-input2.o %t-input1.o 2>&1 | \
# RUN:   FileCheck %s --allow-empty --implicit-check-not={{.}}

I hope --alow-empty is the right way to go about empty input files (found it here http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20140804/229905.html). The flag is not there in the documentation

sameerarora101 marked an inline comment as done.

Updating test files

sameerarora101 marked an inline comment as done.Jul 9 2020, 8:56 AM
sameerarora101 added inline comments.
llvm/test/tools/llvm-libtool-darwin/help-message.test
10

added --implicit-check-not="--safepoint-ir-verifier-print-only as we know the headers won't be present because of LIST-NOT checks below.

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
34

enum class
+ i would add a blank line between the lines 34 and 35

sameerarora101 marked an inline comment as done.Jul 9 2020, 8:15 PM

enum -> enum class

As per @alexshap's recommendation, verifyDarwinObject -> verifyMachOObject

jhenderson added inline comments.Jul 13 2020, 12:48 AM
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
26

If I'm not mistaken, the canonical name is "Mach-O" (we use MachO for variable and tool names etc, but the file format is technically Mach-O, so we should use that in comments etc).

52

Sounds like a documentation bug. Feel free to update the doc if you get a minute. By default, FileCheck emits an error if the input is empty, but this can be overridden with the use of --allow-empty. That in turn has got me wondering whether --allow-empty should be replaced by a new option that also implies --implicit-check-not. I've raised that on llvm-dev.

llvm/test/tools/llvm-libtool-darwin/help-message.test
10

Sounds reasonable, but I'd get rid of the -- bit of the implicit-check-not pattern, so that it doesn't matter whether options are printed in the help with one or two dashes.

llvm/tools/llvm-libtool-darwin/llvm-libtool-darwin.cpp
52

Mach-O (same as mentioned elsewhere).

73

Ditto.

sameerarora101 marked 5 inline comments as done.

MachO -> Mach-O in comments and update help-message.test

sameerarora101 marked an inline comment as done.Jul 13 2020, 7:13 AM
sameerarora101 added inline comments.
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
52

ok, I'll create a diff for the doc, thanks!

sameerarora101 marked an inline comment as done.Jul 13 2020, 7:50 AM
sameerarora101 added inline comments.
llvm/test/tools/llvm-libtool-darwin/create-static-lib.test
52

added it here : D83682

jhenderson accepted this revision.Jul 14 2020, 12:36 AM

LGTM again.

Mach-O -> Darwin for archive format in comments

This revision was automatically updated to reflect the committed changes.
llvm/docs/CommandGuide/llvm-libtool-darwin.rst