This is an archive of the discontinued LLVM Phabricator instance.

ELF: Define another entry point.
Needs ReviewPublic

Authored by ruiu on Jan 26 2016, 1:28 PM.

Details

Summary

This is a drop-in replacement for the old ELF's lld::elf::link.
It takes the same parameters and does the same thing (link object files.)

Unlike elf2::link, elf2::main is guaranteed to return. Internally,
it uses fork (on Unix) or CreateProcess (on Windows), but that should
be considered as an internal detail in most cases. We may change the
implementation in future.

Diff Detail

Event Timeline

ruiu updated this revision to Diff 46039.Jan 26 2016, 1:28 PM
ruiu retitled this revision from to ELF: Define another entry point..
ruiu updated this object.
ruiu added a subscriber: llvm-commits.
ruiu updated this revision to Diff 46040.Jan 26 2016, 1:32 PM

Fixed a bug in the Windows version.

ruiu updated this revision to Diff 46046.Jan 26 2016, 1:42 PM

Made the Windows version slightly better.

rafael accepted this revision.Jan 26 2016, 2:39 PM
rafael edited edge metadata.

Would it be too bad to use the windows version for unix too for now to avoid the #ifdef?

This is fine from the perspective of providing a safe api with a really small code cost on our side. Please get a LGTM too from an intended user.

Thanks a lot for doing this.

ELF/Driver.h
31

Maybe call this one link and the old link becomes linkOrFali?

ELF/DriverUtils.cpp
139

both close and dup can fail, not sure if you care in here.

This revision is now accepted and ready to land.Jan 26 2016, 2:39 PM
ruiu added a comment.Jan 26 2016, 4:12 PM

It is better to keep Unix version being separated from Windows, because on Unix, there is no reliable way to find the executable on the filesystem. On Linux, you can always use /proc/self/exe, but that's highly platform-dependent. So the most reliable way is to not exec anything.

ELF/Driver.h
31

Done.

ELF/DriverUtils.cpp
139

We should be able to ignore errors safely here. I close stdin here just to make the child process behave nicely if it ever tries to read input from stdin (that should never succeed). This is just a safety measure.

ruiu updated this revision to Diff 46072.Jan 26 2016, 4:13 PM
ruiu edited edge metadata.
  • Updated as per Rafael's comments.
silvas added inline comments.Jan 26 2016, 4:27 PM
ELF/DriverUtils.cpp
143

It is not safe to call link after a fork, since e.g. the malloc lock can be held after forking in a multi-threaded situation.

http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html

A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. ***Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called.*** [THR] [Option Start]  Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls.

You can find a list of async-signal safe functions here: http://man7.org/linux/man-pages/man7/signal.7.html

In other words, you pretty much have to exec in order to be robust when used from a multi-threaded program (and sometimes there are threads internal to the runtime library so any program can be multi-threaded). Your LLVM_ON_UNIX==false seems portable so I think you can just delete the LLVM_ON_UNIX==true version.

194

The documentation states that this function is guaranteed to return. So it can't call error.

ruiu updated this revision to Diff 46077.Jan 26 2016, 4:38 PM
  • Use MemoryBuffer to read from an FD instead of read(2).
ruiu updated this revision to Diff 46082.Jan 26 2016, 5:01 PM
  • Update as per Sean's comments.
ruiu added inline comments.Jan 26 2016, 5:06 PM
ELF/DriverUtils.cpp
143

That is very true. I thought that I knew the risk of not calling exec right after fork, but there were more than I expected. I removed the code for Unix and merged that with Windows code.

167

Fixed.

davide accepted this revision.Jan 26 2016, 5:20 PM
davide added a reviewer: davide.
davide added a subscriber: davide.

I like it.

ELF/DriverUtils.cpp
128

The double cast is not exactly pretty, but I'm not exactly sure there's a way to avoid it.

ruiu added inline comments.Jan 26 2016, 5:25 PM
ELF/DriverUtils.cpp
128

I took this line from Clang. We may be able to remove (intptr_t), and just do (void *).

davide added inline comments.Jan 26 2016, 7:06 PM
ELF/DriverUtils.cpp
128

Fair enough, assuming GCC or clang don't complain

I *believe* (although I have not tested this) that the original version of this patch with fork() will work for me as is.

The existing version assumes that the executable-to-call is lld which is not installed on my system. Since the use case for lld-as-a-library involves linking lld into the calling process this probably makes sense.

I tried replacing getLldExecutable() body with "return std::string(Arg0)" as a hack; passing argv[0] from my compiler driver into elf2::link and handling -flavor gnu -lld-no-exec prefix as a signal to call linkOrFail works fine. Of course this requires parent process to actively participate in this form of dispatch :( I was originally assuming that we'd be able to fork() and then everything will just work without extra setup.

ruiu added a comment.Jan 26 2016, 8:43 PM

Ah, right, if you link lld with your program, lld may not exist as an independent command file on your system. You may be able to install lld along with your command, but the whole point of this patch is to provide an easy migration path for existing users, so it wouldn't make much sense, I guess.

Maybe I should switch back to use fork on Unix, with a notice saying that you shouldn't use in a multi-threaded program. Or, if wouldn't work, we can just throw this patch out and wait until we do real stuff. I want to try a few ideas to handle errors in a better way that would probably make link() always return, so maybe we can live without this.

I just reread the related threads, let me try to summarize:

  • Some user caused errors (duplicated symbols for example) are better

handled with non-fatal handling. Since that doesn't impact libraries
and we want it anyway, lets just ignore those in here (i.e.:, use this
to look at the hard issues).

  • I would really not want us to be in a situation where we have bugs

but they are not fixed. If not failing gracefully given broken files
is to be considered a bug, we *have* to fuzz the linker and fix them.
A particularly nasty case is relocations being used in ABI invalid
cases. For example, if a R_X86_64_REX_GOTPCRELX is used in a position
that doesn't follow a REX prefix, do we have to report the error or
can we produce garbage when given garbage? Note that checking means
disassembling the entire section to see if a given byte is a rex
prefix for a instruction or just a byte of the previous instruction.

The options I see then are:

A) Fuzz the linker, make sure it can handle every combination of
broken file we can throw at it. This is a lot of work and not
something I will sign up for. I can help review and benchmark, but I
would also ask that this be delayed. We are still refactoring the
linker as features are added and I losing the simplicity we have now
would be really bad for design work.

B) Provide a way of isolating the liking "task". I particularly like
this since we should be able to provide a *very* robust solution and
go on to implement the rest of the linker. There are a few
complications:

  • A library cannot know what command line options the program it is in takes.
  • There are big restrictions to using fork and threads. For posix, I think the best we can do is provide a function that

calls fork only (not exec) and is documented to do so. It is part of
its contract that it cannot be called with threads active. In a
program that wants to use multiple threads and call it, it should
setup a zygote for calling it (and we can provide code to help it do
so).

Any ideas on how to do this on windows?

C) Similar to A, but use a handler that either call exit or throws if
exceptions are enabled. This would still add a *lot* of checks, but I
don't think the error handling would degrade the code as much. This
means that lld the program would be built without exceptions, but lld
the library would require it. Like A I would ask that this be delayed
until more of the linker is done.

Cheers,
Rafael

silvas resigned from this revision.Jul 8 2016, 11:46 PM
silvas removed a reviewer: silvas.
davide removed a reviewer: davide.Sep 9 2016, 1:47 PM
davide removed a subscriber: davide.

Not sure if this will go anywhere, and I'm cleaning up my queue. Feel free to add me again things move.

espindola edited reviewers, added: espindola; removed: rafael.Mar 14 2018, 5:00 PM
This revision now requires review to proceed.Mar 14 2018, 5:00 PM