[llvm-elfabi] Emit ELF header and string table section

[haowei]

Summary

This change serves to create the initial framework for outputting ELF files from llvm-elfabi.

Sorry this is so large but this, perhaps short of the code for outputting the string table, is about as small of a setup as I can get and still have something testable. You can see the full picture here: https://reviews.llvm.org/D55864. Had I tried to get to exactly this point more directly and not by splitting it probably wouldn't be quite as large or generically written but if possible I'd like to keep some of the generality here since I'll likely add it back soon.

[haowei]

Resolve clang-tidy warnings.

[haowei]

Rebase the change and fixes some format issues.

[MaskRay]

The purpose of this tool is still a bit unclear to me. Would you mind explaining a little? If you want to create a shared object writer, you can start from yaml2obj. Then there is natural question what generation features this tool can provide while yaml2obj cannot. If you want to write an interface shared object, sh_addr can be 0.

[phosek]

In D61767#2331624, @MaskRay wrote:

The purpose of this tool is still a bit unclear to me. Would you mind explaining a little? If you want to create a shared object writer, you can start from yaml2obj. Then there is natural question what generation features this tool can provide while yaml2obj cannot. If you want to write an interface shared object, sh_addr can be 0.

We've had this discussion multiple times in the past so I'm not sure how much of that discussion we want to replay here. The somewhat short version is that yaml2obj is a testing tool intended to be used only within LLVM whereas llvm-elfabi is intended as a production tool shipped with your toolchain. The scope of these tools is also different: llvm-elfabi can generate a machine readable ABI description (we use YAML but that's an implementation detail, we're also considering other formats) for a given binary and then generate a stub from that description; yaml2obj only handles the latter part, but at the same time it also handles much broader range of use cases that are needed for testing and the YAML format it accepts has a lot of features we never want to expose to users. While I agree that there's some overlap between these tools, their use case is sufficiently different to warrant different tools in our opinion. There might be an opportunity for deduplicating some of the implementation, but that's also the case for llvm-objcopy/llvm-strip, and it's something we would like to handle separately in the future.

[MaskRay]

In D61767#2331654, @phosek wrote:

In D61767#2331624, @MaskRay wrote:

The purpose of this tool is still a bit unclear to me. Would you mind explaining a little? If you want to create a shared object writer, you can start from yaml2obj. Then there is natural question what generation features this tool can provide while yaml2obj cannot. If you want to write an interface shared object, sh_addr can be 0.

We've had this discussion multiple times in the past so I'm not sure how much of that discussion we want to replay here. The somewhat short version is that yaml2obj is a testing tool intended to be used only within LLVM whereas llvm-elfabi is intended as a production tool shipped with your toolchain. The scope of these tools is also different: llvm-elfabi can generate a machine readable ABI description (we use YAML but that's an implementation detail, we're also considering other formats) for a given binary and then generate a stub from that description; yaml2obj only handles the latter part, but at the same time it also handles much broader range of use cases that are needed for testing and the YAML format it accepts has a lot of features we never want to expose to users. While I agree that there's some overlap between these tools, their use case is sufficiently different to warrant different tools in our opinion. There might be an opportunity for deduplicating some of the implementation, but that's also the case for llvm-objcopy/llvm-strip, and it's something we would like to handle separately in the future.

This rationale is fine. However, this effort is a resurrection of a very old patch. Many tools have documentation under docs/CommandGuide and some file-level comments in the main source file while llvm-elfabi doesn't.
I think others may have similar concerns as I did. If you want to address such concerns and make the scope of the tool clear, I hope you can put the link the last discussion in this patch or have something in the documentation.

[haowei]

merge public sections in ELFStubBuilder class

[phosek]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

99–100 ↗

(On Diff #298255)

Consider this.

122 ↗

(On Diff #298255)

Nit: would it be possible to separate function bodies with empty lines? You could also consider using clang-format which should handle that.

123 ↗

(On Diff #301152)

Nit: don't use { and } when body has only a single statement.

130 ↗

(On Diff #301152)

Nit: No space.

148 ↗

(On Diff #301152)

Nit: don't use { and } only around single arm (also in this case they're unnecessary, see the comment above).

[haowei]

address reviewer comments.

[phosek]

LGTM

[grimar]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

55 ↗

(On Diff #301418)

I think you can get rid of these usings.

Elf_Phdr and Elf_Shdr are used only once, so you can inline them:

ElfHeader.e_phentsize = sizeof(typename ELFT::Phdr);
ElfHeader.e_shentsize = sizeof(typename ELFT::Shdr);

The same for Elf_Ehdr + you can change memset to

memset(&ElfHeader, 0, sizeof(ElfHeader));

113 ↗

(On Diff #301418)

You can use TYPEDEF_ELF_TYPES(ELFT) instead of adding these usings.

124 ↗

(On Diff #301418)

Please don't use auto where type is not obvious.

129 ↗

(On Diff #301418)

std::vector<OutputSection<ELFT> *> Sections = { &ShStrTab, &StrTab };

133 ↗

(On Diff #301418)

Don't use auto.

136 ↗

(On Diff #301418)

It should be OK to add an empty string to ELF string tables. Do you really need if (!Sec->Name.empty()) condition?

142 ↗

(On Diff #301418)

addrs -> addresses

145 ↗

(On Diff #301418)

Don't use auto.

148 ↗

(On Diff #301418)

Why nobits sections are special?

Also I don't understand:

1. There are just 2 sections currently: ShStrTab and StrTab. Why do you need the code for NoBits?
2. Why can't you set Align = 1 from start for them instead of having Sec->Align ? Sec->Align : 1; logic?

169 ↗

(On Diff #301418)

You don't need these 2 lines, because you do memset(&ElfHeader, 0, sizeof(Elf_Ehdr)); in initELFHeader.

560 ↗

(On Diff #301418)

You should be able to use std::string toString(Error E):

Stream << toString(BufOrError.takeError());

Then you don't need the FileReadError variable and consumeError.

I think you should be able to do just something like:

if (!BufOrError)
  return createStringError(errc::invalid_argument,
    toString(BufOrError.takeError()) + " when trying to open `" +  FilePath + "` for writing");

561 ↗

(On Diff #301418)

You don't have a test for this I think.

570 ↗

(On Diff #301418)

It is more common naming I think.

609 ↗

(On Diff #301418)

this should be llvm_unreachable I think?

llvm/test/tools/llvm-elfabi/write-elf32be-ehdr.test

16

Instead of using {{$}} everywhere, consider using FileCheck --match-full-lines.

29

No new line.

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

54

All above are unrelated formatting changes I think.

149

You don't need curly bracers around a single line:

if (SOName.getNumOccurrences() == 1)
   TargetStub->SoName = SOName;

[haowei]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

113 ↗

(On Diff #301418)

I think this is only place in this change to use so many "using" statements. It is not very cost efficient to define a marco for this case.

148 ↗

(On Diff #301418)

Most part of this change was authored by my previous co-workers. I don't know the specific reason why NoBits are handled here, but I could guess the code for NoBits are reserved in case the stub SO generated from elfabi would need any sections that has NoBits attribute. I agree it is not useful now as the following change do not use NoBits sections as well.

For align, the sh_addralign of .shstrtab and .dynstr was set to to 0 in this change. If I recall correctly both 0 and 1 are acceptable values for sh_addralign if the section does not have alignment constraints. Since alignTo function require "Align" to be non-zero, I guess that is why Sec->Align ? Sec->Align : 1 exists. I have change the sh_addralign to 1 for the string tables to avoid it.

561 ↗

(On Diff #301418)

Emmm, I tried locally on a linux machine and it seems to be difficult to put it in test case.
I have to create a file with same filename, change ownership and the permission so elfabi cannot modify it. And run llvm-elfabi to write output to this file. Is there any good way to trigger error from FileOutputBuffer?

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

54

Yes. I just run clang-format on entire file. Can I keep them?

[grimar]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

72 ↗

(On Diff #304393)

I'd use typename ELFT::Ehdr for consistency with lines below.

113 ↗

(On Diff #301418)

You are right. I thought we have it defined globally already (like LLVM_ELF_IMPORT_TYPES_ELFT), but it turns out it is only defined in llvm-readobj\ELFDumper.cpp currently.

561 ↗

(On Diff #301418)

Perhaps I am missing something (I am not sure what the problem is).
We have a few tests in LLVM (e.g see LLD tests, like lld\test\COFF\thinlto-emit-imports.ll) that
do something very close to what you describe I think: they create a file, change permission and run tool on it:

; RUN: rm -f %t3.obj.imports
; RUN: touch %t3.obj.imports
; RUN: chmod 400 %t3.obj.imports
; RUN: not lld-link -entry:main -thinlto-index-only \
; RUN:     -thinlto-emit-imports-files %t1.obj %t2.obj %t3.obj \
; RUN:     -out:%t4.exe 2>&1 | FileCheck %s --check-prefix=ERR
; ERR: cannot open {{.*}}3.obj.imports: {{P|p}}ermission denied

Can you do something like that?

llvm/test/tools/llvm-elfabi/write-elf64be-ehdr.test

3

Use --match-full-lines too?

llvm/test/tools/llvm-elfabi/write-elf64le-ehdr.test

3

Use --match-full-lines too?

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

54

It is better to avoid unnecessarily changes in patches. These changes are unrealated and the right way
to do this would be to format the entry file first and commit as NFC, that should not need a review in this case.

[haowei]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

561 ↗

(On Diff #301418)

I added "fail-file-write.test".
I thought it is discouraged to use rm, chmod commands since they are not cross-platforms.

Do I have to add anything to prevent this test from running on Windows?

[grimar]

LGTM with a nit.

llvm/lib/InterfaceStub/ELFObjHandler.cpp

561 ↗

(On Diff #301418)

Do I have to add anything to prevent this test from running on Windows?

I believe - no. touch and rm are often used in tests. chmod is a bit more rare, but
all of them are parts of GnuWin32 and works fine under windows.

llvm/test/tools/llvm-elfabi/fail-file-write.test

7 ↗

(On Diff #304652)

The last line has an excessive full stop at the end. I am not sure you need this line though.

[jhenderson]

Nothing grievously wrong here, but lots of small things that need improving. Requesting changes mostly to clear the "Accepted" state.

llvm/lib/InterfaceStub/ELFObjHandler.cpp

52 ↗

(On Diff #304652)

It's been a while since I looked at this code, but I wonder whether this is essentially a duplication of code elsewhere in LLVM. Certainly I'd expect tools like LLD, llvm-objcopy and llvm-mc to have functionality that emits an ELF Header of some description. Do we really need to repeat it here?

Also, I wonder if it would be a little more flexible/traditional to return a locally constructed ElfHeader rather than take it as an input?

70–71 ↗

(On Diff #304652)

Dead code? You use memset to zero earlier. Same goes for the EI_ABIVERSION setting earlier.

118 ↗

(On Diff #304652)

I wonder if it would be less confusing if StrTab were to be called Dynstr? .strtab is traditionally the name of the static symbol string table.

136–149 ↗

(On Diff #304652)

If I follow this code correctly, this whole loop could be dramatically simplified, as suggested.

This appears to be setting the initial address to the end of the ELF header. It's certainly not the case that the ELF header has to always be assigned memory - if it doesn't appear in a program header in a linker script, it won't be for example, so the initial address would start from zero. I note also that the program doesn't appear to be emitting program headers at all, so the use of addresses seems a little confusing to me anyway.

150 ↗

(On Diff #304652)

153 ↗

(On Diff #304652)

I suggest using a generic term rather than the variable name to avoid rotting comments should variables get renamed.

165 ↗

(On Diff #304652)

I wonder if it would be easier for this function (with the help of its child functions) to update the offset as it goes along, rather than functions needing to precalculate where the thing should be written:

void write(uint8_t *Data) const {
  write(Data, ElfHeader);
  Data += ShStrTab.Shdr.sh_offset;
  ShStrTab.Content.write(Data);
  Data += StrTab.Shdr.sh_offset;
  StrTab.Content.write(Data);
  Data = ElfHeader.e_shoff;
  writeShdr(Data, ShStrTab);
  writeShdr(Data, StrTab);
}

template <class T> static void write(uint8_t *&Data, const T &Value) {
  *reinterpret_cast<T *>(Data) = Value;
  Data += sizeof(Value);
}

void writeShdr(uint8_t *Data, const OutputSection<ELFT> &Sec) const {
  write(Data, Sec.Shdr);
}

At that point, some of the helper methods might become completely unnecessary.

167 ↗

(On Diff #304652)

Here and below, would a loop over a list of Sections make more sense going forwards, like you already do above? I don't know if there are expected to be more sections added in the future, but if there are, such a loop would be more robust than having to remember to add everything in multiple places.

170 ↗

(On Diff #304652)

It might not be important, but it seems more natural to have the SHF_ALLOC seection headers before the non SHF_ALLOC stuff in the section header table (same goes for the sections themselves).

179 ↗

(On Diff #304652)

This line makes me very nervous that this code will fail miserably where the host endianness doesn't match the target endianness with non-single byte T types.

190 ↗

(On Diff #304652)

384 ↗

(On Diff #304652)

I'm concerned that this has been changed to llvm_unreachable. Is this function only able to be called with ELF binaries? What happens if a user passes in e.g. an archive or COFF file? createBinary can return more than just ELF types, which would lead to an assertion here. This case will want testing too. Also, it seem unrelated to this patch?

589 ↗

(On Diff #304652)

The style guide says to use lower-case strings where possible for error messages.

I think this case SHOULD be an llvm_unreachable? It's not possible for a user to hit this code as far as I can tell (alternatively, write a test for it...)

148 ↗

(On Diff #301418)

I've forgotten any memory I have of the long-term goal with this tool, so this might be irrelevant, but is it possible that a user in the future will set an alignment of 0 explicitly and therefore this will fail?

llvm/test/tools/llvm-elfabi/binary-write-sheaders.test

2

Please add a comment to the top of this test stating what this test is testing. Same goes for all tests.

This test is only testing one of the 4 ELF formats (64-bit LE). You should probably have the other three too?

8

Nit: remove extra blank line.

llvm/test/tools/llvm-elfabi/fail-file-write.test

15 ↗

(On Diff #304652)

Nit: delete extra blank line.

16 ↗

(On Diff #304652)

Please check the full error here, rather than a partial one. Additionally, I would be suspicious that the error message will be different depending on whether the host OS is Linux or Windows. Please could you check? If you don't have access to both systems, let me know, and I can test the one you can't.

llvm/test/tools/llvm-elfabi/invalid-bin-target.test

2

I'd rename this test to invalid-output-target or even output-target-invalid to match the option name.

llvm/test/tools/llvm-elfabi/missing-bin-target.test

2

It would probably be easier to merge this and the above test together, so that you have all tests for --output-target in one place.

11

This probably wants an additional bit of message. Something like "error: No binary..."

llvm/test/tools/llvm-elfabi/write-elf32be-ehdr.test

2

It's not clear to me why this test is separate to the binary-write-sheaders test? Can they be merged?

llvm/test/tools/llvm-elfabi/write-elf32le-ehdr.test

2

If it doesn't have it already, I'd recommend adding a --docnum option to llvm-elfabi, like yaml2obj, so that you can have multiple test variants within the same file (in a separate prerequisite patch). In this case, I think it would make a lot of sense for it to be in the same test as the big endian test. Same with the 64-bit variants.

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

54

As llvm-elfabi is a new tool, it would be better to do the reformatting early as @grimar suggests, rather than let things linger. That will make everybody's lives easier as you'll just be able to reformat the whole file when adding new code (and only the new bits will change).

146–147

This seems unrelated to your other changes in this patch? I certainly don't see any relevant test changes.

159

I can't remember exactly the invocation (and am too lazy to check it right now), but WithColor has a default error handler function that can be passed an LLVM error I believe, so use that instead, I recommend. I'd probably also wrap it all in a local function that does the exit(1) too to reduce duplication.

Also, LLVM style guide says errors have no full stops and no leading capitalization.

[jhenderson]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

609 ↗

(On Diff #301418)

I think the wrong place was changed?

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

165

Same comment here re. default error handler function. You might want to change the TBEWriteError and other error bits similarly, in a separate patch.

[jhenderson]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

Looking at the pre-merge checks this test is failing on Windows as-is.

[grimar]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

179 ↗

(On Diff #304652)

It is only used for writing Elf_Shdr, Elf_Ehdr etc types, what I think should work fine on any hosts.
Is your concern that somebody might try to use write for regular types?

384 ↗

(On Diff #304652)

I believe this code is unreachable because readELFFile is only called for ELFs:

// First try to read as a binary (fails fast if not binary).
if (InputFileFormat.getNumOccurrences() == 0 ||
    InputFileFormat == FileFormat::ELF) {
  Expected<std::unique_ptr<ELFStub>> StubFromELF =
      readELFFile(FileReadBuffer->getMemBufferRef());

[jhenderson]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

179 ↗

(On Diff #304652)

We're probably talking past each other, or I might be forgetting something, but the fields within Elf_Shdr, Elf_Ehdr etc will be in little/big endian order right depending on the host, right? This code here is essentially just doing a byte-wise copy, by my understanding.

384 ↗

(On Diff #304652)

InputFileFormat.getNumOccurrences() == 0 - doesn't that mean an unspecified input format will be attempted to be read as ELF, but there's no guarantee it actually is ELF...?

[grimar]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

179 ↗

(On Diff #304652)

My understanding is the following:

E.g. Elf_Shdr consists of Elf_Word`/Elf_Addr/Elf_Off fields,
which are of support::detail::packed_endian_specific_integral type:

using Elf_Shdr_Base<ELFT>::sh_size;
  
template <endianness TargetEndianness>
struct Elf_Shdr_Base<ELFType<TargetEndianness, false>> {
  LLVM_ELF_IMPORT_TYPES(TargetEndianness, false)
  Elf_Word sh_name;      // Section name (index into string table)
  Elf_Word sh_type;      // Section type (SHT_*)
  Elf_Word sh_flags;     // Section flags (SHF_*)
  Elf_Addr sh_addr;      // Address where section is to be loaded
  Elf_Off sh_offset;     // File offset of section data, in bytes
  Elf_Word sh_size;      // Size of section, in bytes
  Elf_Word sh_link;      // Section type-specific header table index link
  Elf_Word sh_info;      // Section type-specific extra information
  Elf_Word sh_addralign; // Section address alignment
  Elf_Word sh_entsize;   // Size of records contained within the section
};

#define LLVM_ELF_IMPORT_TYPES_ELFT(ELFT)                                       \
  using Elf_Addr = typename ELFT::Addr;                                        \
  using Elf_Off = typename ELFT::Off;                                          \
  using Elf_Half = typename ELFT::Half;                                        \
  using Elf_Word = typename ELFT::Word;                                        \
  using Elf_Sword = typename ELFT::Sword;                                      \
  using Elf_Xword = typename ELFT::Xword;                                      \
  using Elf_Sxword = typename ELFT::Sxword;

  using Half = packed<uint16_t>;
  using Word = packed<uint32_t>;
  using Sword = packed<int32_t>;
  using Xword = packed<uint64_t>;
  using Sxword = packed<int64_t>;
  using Addr = packed<uint>;
  using Off = packed<uint>;
 
  template <typename Ty>
  using packed = support::detail::packed_endian_specific_integral<Ty, E, 1>;

This type respects endianess on assign:

struct packed_endian_specific_integral {
....
  void operator=(value_type newValue) {
    endian::write<value_type, endian, alignment>(
      (void*)Value.buffer, newValue);
  }

/// Write a value to memory with a particular endianness.
template <typename value_type, std::size_t alignment>
inline void write(void *memory, value_type value, endianness endian) {
  value = byte_swap<value_type>(value, endian);
  memcpy(LLVM_ASSUME_ALIGNED(
             memory, (detail::PickAlignment<value_type, alignment>::value)),
         &value, sizeof(value_type));
}

template <typename value_type>
inline value_type byte_swap(value_type value, endianness endian) {
  if ((endian != native) && (endian != system_endianness()))
    sys::swapByteOrder(value);
  return value;
}

384 ↗

(On Diff #304652)

Oh. I missed that there is || InputFileFormat == FileFormat::ELF, not && InputFileFormat == FileFormat::ELF.
This place looks a bit wierd to me..

[haowei]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

148 ↗

(On Diff #301418)

The section alignment is set in this builder function so I guess the case you discussed would not happen. I have no preference in using 0 or 1 here. Using 1 seems to make code simpler but less robust.

609 ↗

(On Diff #301418)

I changed the wrong place, sorry about that.

52 ↗

(On Diff #304652)

I checked writeEhdr() in obj-copy but it is not flexible enough to be used here. I agree it would be great to have a common function to write Ehdr into memory so it can be reused by these tools. But I feel it is out of scope of this change. I can draft a change to address this problem if you feel it is necessary to do so.

118 ↗

(On Diff #304652)

StrTab is renamed to DynStr.

165 ↗

(On Diff #304652)

I prefer not to do that. Using write function to determine the offset would look very messy on D89432.

167 ↗

(On Diff #304652)

Using loop would make more sense but it seems to be infeasible in this case. The "Section.Content.write" function is responsible for writing data to the MemoryBuffer but this function is not a virtual function derived by different classes (there are SymTab and Dynamic sections builder in D89432) so it would require some data structure changes to make it happen. Or if there is anything like "instanceof" that I can check and cast the types safely at run time, that would also work.

As shown in D89432, we only need to write 4 sections in a stub elf file. So it is not very cost efficient to invent some new classes just to make the loop happen. If you have better solutions please let me know.

170 ↗

(On Diff #304652)

I changed the order in the new diff

179 ↗

(On Diff #304652)

Test does not fail when writing elfheader in be or le. If you have better solution to make it more robust, please let me know.

589 ↗

(On Diff #304652)

I changed it to llvm_unreachable.

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

I have a windows workstation but it does not have development environment set up for llvm. I will see if I can set it up tomorrow. For now, let me test it on bots first.

llvm/test/tools/llvm-elfabi/missing-bin-target.test

2

merged in latest diff.

llvm/test/tools/llvm-elfabi/write-elf32be-ehdr.test

2

merged in latest diff.

llvm/test/tools/llvm-elfabi/write-elf32le-ehdr.test

2

We intended to merge elfabi tool into elf-ifs tool once the we feel ELFObjHandler is stable. So I prefer not to add additional flag(feature) to llvm-elfabi at this point.

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

54

It's reformatted in https://github.com/llvm/llvm-project/commit/ad0da312c05d5264fac71dfe417d8a1e117b6a43

146–147

writeBinaryStub also use SoName in TargetStub variable. This change moves this logic outside of the EmitTBE branch so the assignment at L140 can be written once but used by both L147 and L159.

159

I added a fatalError(Error err) function.

[jhenderson]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

561 ↗

(On Diff #305053)

153 ↗

(On Diff #304652)

This comment still has most of the typos/grammar problems I fixed. Please review my suggested edit and fix them.

167 ↗

(On Diff #304652)

FWIW, LLVM has a dyn_cast still mechanism that allows some degree of runtime type checking, but it only works within a class heirarchy (see any class that has the classof function, and how that works).

It looks like it would be straightforward to unify the section types somehow, with a virtual write function, but without actually trying it, I'm not certain it would be.

179 ↗

(On Diff #304652)

It's fine. I forgot that the Elf_* types had endianness magic built into them.

llvm/test/tools/llvm-elfabi/missing-or-invalid-output-target.test

1 ↗

(On Diff #305053)

Add descriptive comment at top of test. This should be in place for all tests.

I'd also consider renaming it output-target-error.test, which is a little more concise.

14 ↗

(On Diff #305053)

Nit: no new line at EOF.

llvm/test/tools/llvm-elfabi/write-elf32be-stub.test

1 ↗

(On Diff #305053)

Use '##' for comments, to make them stand out from other lit and FileCheck commands. This applies to all new comments you've added in this and other tests.

4–7 ↗

(On Diff #305053)

These can all be combined into a single llvm-readobj execution, without the need for separate prefixes.

llvm/test/tools/llvm-elfabi/write-elf32le-stub.test

1 ↗

(On Diff #305053)

It might make more sense to merge the ELF32/ELF64 and LE/BE tests all into a single test. Your test would look something like:

# RUN: llvm-elfabi %s --output-target=elf32-little %t32-little
# RUN: llvm-readobj -h -S --string-dump .shstrtab --string-dump .dynstr %t32-little | FileCheck %s -DCLASS="32-bit (0x1)" <other defines here for the other bits that are slightly different>
# RUN: llvm-elfabi %s --output-target=elf32-big %t32-b ig
# RUN: llvm-readobj -h -S --string-dump .shstrtab --string-dump .dynstr %t32-big | FileCheck %s -DCLASS="32-bit (0x1)" <other defines here for the other bits that are slightly different>
<same for 64-bit variants>

[haowei]

llvm/test/tools/llvm-elfabi/write-elf32le-stub.test

1 ↗

(On Diff #305053)

I combined 4 test files into 1.

[jhenderson]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

Did you get anywhere with resolving this? The test is still failing on the pre-merge bot, as noted near the top of this page.

llvm/test/tools/llvm-elfabi/output-target-error.test

1 ↗

(On Diff #305673)

[haowei]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

I finally get my windows workstation to build llvm successfully. So the reason this test failed on Windows is that, the chmod seems to be a no-op on windows. It didn't change the permission of the %t.TestDir at all. The failure at not llvm-elfabi... did not happen so the test failed.

I don't have a good workaround for this so I disabled the test under Windows for now. Do you have any good suggestion for testing this failure under Windows?

[grimar]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

I guess you should be able to change the permission for the file, if not for directory.

Our LLD test (ELF\lto\thinlto-cant-write-index.ll) has the following lines:

; Ensure lld generates error if unable to write to index files
; RUN: rm -f %t2.o.thinlto.bc
; RUN: touch %t2.o.thinlto.bc
; RUN: chmod u-w %t2.o.thinlto.bc
; RUN: not ld.lld --plugin-opt=thinlto-index-only -shared %t1.o %t2.o -o /dev/null 2>&1 | FileCheck %s
; RUN: chmod u+w %t2.o.thinlto.bc
; CHECK: cannot open {{.*}}2.o.thinlto.bc: {{P|p}}ermission denied

This test is not disabled for windows amd I believe that the last line has {{P|p}} because the message
produced is different on windows/*nix.

There are other LLD tests that are using chmod on files and are not disabled on windows.

[haowei]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

If I only change the file permission, it won't work under linux. Apparently the file write function used in elfabi will remove the file first and then recreate it. If the directory contains the file has wx permission, the file can be removed regardless of the file's own permission bits. That is why in this test, I changed the permission of the directory, not the file.

[grimar]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

I see. Perhaps it is OK to keep it unsupported at least for now.
(may be someone else have ideas, I don't have any currently).

I am not sure how your way of doing it works well though.
Usually in tests the "system-windows" is commonly used:

UNSUPPORTED: system-windows

I also observe "windows-gnu", "windows-msvc" and just "windows" versions but,
don't see "-windows-" anywhere.

[jhenderson]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

If you can get the test to work in one case for Linux and another case for Windows, how about you split this test into fail-file-write.test (for non-Windows) and fail-file-write-windows.test (for Windows) with the appropriate setup and REQUIERS/UNSUPPORTED checks according to the test?

[haowei]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

I added a windows test. However, it does not trigger the error from L530 in ELFObjHandler.cpp . Instead, it triggers the error from L539 in the same file. Maybe FileOutputBuffer::create is implemented differently on 2 platforms.

[MaskRay]

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

128

Use static for these functions

https://llvm.org/docs/CodingStandards.html#anonymous-namespaces

The coding standard also says non-exported class declarations should be written in anonymous namespaces

[haowei]

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

128

I added static.

This is not a class declaration I guess I don't need to put it into anonymous namespace.

[jhenderson]

llvm/test/tools/llvm-elfabi/fail-file-write.test

16 ↗

(On Diff #304652)

Windows file permissions are tricky at best. The important thing for these tests is that the code in llvm-elfabi that handles errors from FileOutputBuffer::create is correct. It doesn't really matter what the error actually is here.

llvm/test/tools/llvm-elfabi/output-target-error.test

1 ↗

(On Diff #305673)

Ping this?

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

128

This is not a class declaration I guess I don't need to put it into anonymous namespace.

@MaskRay might be referring to the new classes in ElfObjHandler.cpp like OutputSection etc

[haowei]

llvm/test/tools/llvm-elfabi/output-target-error.test

1 ↗

(On Diff #305673)

Sorry I missed that. Phabricator gray out the comments made me though I already fixed it.

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

128

I put those classes into anonymous namespace. Does it look good?

[MaskRay]

llvm/lib/InterfaceStub/ELFObjHandler.cpp

194 ↗

(On Diff #306527)

The coding standard uses // end anonymous namespace (no period)
Some code uses // namespace because sometimes clang-format inserts that.
Please don't invent a new style.

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

128

Please make all new functions static if not used outside the file.

[haowei]

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

128

All new functions that are not used outside of the translation unit was added with static modifier, please let me know if you found anything I missed.

[jhenderson]

Looks good from my point of view, but please wait for @MaskRay to confirm he is happy.

llvm/include/llvm/BinaryFormat/ELF.h

enum {
   EI_DATA = 5,       // Data encoding.
   EI_VERSION = 6,    // File version.
   EI_OSABI = 7,      // OS/ABI identification.
   EI_ABIVERSION = 8, // ABI version.
   EI_PAD = 9,        // Start of padding bytes.
   EI_NIDENT = 16     // Number of bytes in e_ident.
 };
 
+constexpr uint8_t ELFMAG0 = 0x7f;
+constexpr uint8_t ELFMAG1 = 'E';
+constexpr uint8_t ELFMAG2 = 'L';
+constexpr uint8_t ELFMAG3 = 'F';
+
 struct Elf32_Ehdr {
   unsigned char e_ident[EI_NIDENT]; // ELF Identification bytes
   Elf32_Half e_type;                // Type of file (see ET_* below)
   Elf32_Half e_machine;   // Required architecture for this file (see EM_*)
   Elf32_Word e_version;   // Must be equal to 1
   Elf32_Addr e_entry;     // Address to jump to in order to start program
   Elf32_Off e_phoff;      // Program header table's file offset, in bytes
   Elf32_Off e_shoff;      // Section header table's file offset, in bytes

llvm/test/tools/llvm-elfabi/binary-write-sheaders.test

+# RUN: llvm-elfabi %s --output-target=elf64-little %t
+# RUN: llvm-readobj -h %t | FileCheck %s --check-prefix=ELFHEADER
+# RUN: llvm-readobj -S %t | FileCheck %s --check-prefix=SECTIONS
+
+--- !tapi-tbe
+TbeVersion: 1.0
+Arch: x86_64
+Symbols: {}
+...
+
+# ELFHEADER: SectionHeaderCount: 2
+# ELFHEADER: StringTableSectionIndex: 1
+
+# SECTIONS:      Section {
+# SECTIONS-NEXT:   Index: 0
+# SECTIONS-NEXT:   Name:  (0)
+# SECTIONS-NEXT:   Type: SHT_NULL
+# SECTIONS-NEXT:   Flags [
+# SECTIONS-NEXT:   ]
+# SECTIONS-NEXT:   Address: 0x0
+# SECTIONS-NEXT:   Offset: 0x0
+# SECTIONS-NEXT:   Size: 0
+# SECTIONS-NEXT:   Link: 0
+# SECTIONS-NEXT:   Info: 0
+# SECTIONS-NEXT:   AddressAlignment: 0
+# SECTIONS-NEXT:   EntrySize: 0
+# SECTIONS-NEXT: }
+# SECTIONS-NEXT: Section {
+# SECTIONS-NEXT:   Index: 1
+# SECTIONS-NEXT:   Name: .dynstr
+# SECTIONS-NEXT:   Type: SHT_STRTAB
+# SECTIONS-NEXT:   Flags [
+# SECTIONS-NEXT:     SHF_ALLOC
+# SECTIONS-NEXT:   ]
+# SECTIONS-NEXT:   Address:
+# SECTIONS-NEXT:   Offset:
+# SECTIONS-NEXT:   Size:
+# SECTIONS-NEXT:   Link: 0
+# SECTIONS-NEXT:   Info: 0
+# SECTIONS-NEXT:   AddressAlignment: 0
+# SECTIONS-NEXT:   EntrySize: 0
+# SECTIONS-NEXT: }

llvm/test/tools/llvm-elfabi/invalid-bin-target.test

+# RUN: not llvm-elfabi %s --output-target=nope %t 2>&1 | FileCheck %s
+
+--- !tapi-tbe
+SoName: somelib.so
+TbeVersion: 1.0
+Arch: x86_64
+Symbols: {}
+...
+
+# CHECK: llvm-elfabi: for the --output-target option: Cannot find option named 'nope'!

llvm/test/tools/llvm-elfabi/missing-bin-target.test

+# RUN: not llvm-elfabi %s %t 2>&1 | FileCheck %s
+
+--- !tapi-tbe
+SoName: somelib.so
+TbeVersion: 1.0
+Arch: x86_64
+Symbols: {}
+...
+
+# CHECK: No binary output target specified.

llvm/test/tools/llvm-elfabi/write-elf32be-ehdr.test

+# RUN: llvm-elfabi %s --output-target=elf32-big %t
+# RUN: llvm-readobj --file-headers %t | FileCheck %s
+
+--- !tapi-tbe
+TbeVersion: 1.0
+Arch: x86_64
+Symbols: {}
+...
+
+# CHECK:      ElfHeader {
+# CHECK-NEXT:   Ident {
+# CHECK-NEXT:     Magic: (7F 45 4C 46)
+# CHECK-NEXT:     Class: 32-bit (0x1)
+# CHECK-NEXT:     DataEncoding: BigEndian (0x2)
+# CHECK-NEXT:     FileVersion: 1{{$}}
+# CHECK-NEXT:     OS/ABI: SystemV (0x0)
+# CHECK-NEXT:     ABIVersion: 0{{$}}
+# CHECK-NEXT:     Unused: (00 00 00 00 00 00 00)
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Type: SharedObject (0x3)
+# CHECK-NEXT:   Machine: EM_X86_64 (0x3E)
+# CHECK-NEXT:   Version: 1{{$}}
+# CHECK-NEXT:   Entry: 0x0{{$}}
+# CHECK:        Flags [ (0x0)
+# CHECK-NEXT:   ]
+# CHECK-NEXT:   HeaderSize: 52{{$}}
+# CHECK-NEXT:   ProgramHeaderEntrySize: 32{{$}}
+# CHECK:        SectionHeaderEntrySize: 40{{$}}

llvm/test/tools/llvm-elfabi/write-elf32le-ehdr.test

+# RUN: llvm-elfabi %s --output-target=elf32-little %t
+# RUN: llvm-readobj --file-headers %t | FileCheck %s
+
+--- !tapi-tbe
+TbeVersion: 1.0
+Arch: x86_64
+Symbols: {}
+...
+
+# CHECK:      ElfHeader {
+# CHECK-NEXT:   Ident {
+# CHECK-NEXT:     Magic: (7F 45 4C 46)
+# CHECK-NEXT:     Class: 32-bit (0x1)
+# CHECK-NEXT:     DataEncoding: LittleEndian (0x1)
+# CHECK-NEXT:     FileVersion: 1{{$}}
+# CHECK-NEXT:     OS/ABI: SystemV (0x0)
+# CHECK-NEXT:     ABIVersion: 0{{$}}
+# CHECK-NEXT:     Unused: (00 00 00 00 00 00 00)
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Type: SharedObject (0x3)
+# CHECK-NEXT:   Machine: EM_X86_64 (0x3E)
+# CHECK-NEXT:   Version: 1{{$}}
+# CHECK-NEXT:   Entry: 0x0{{$}}
+# CHECK:        Flags [ (0x0)
+# CHECK-NEXT:   ]
+# CHECK-NEXT:   HeaderSize: 52{{$}}
+# CHECK-NEXT:   ProgramHeaderEntrySize: 32{{$}}
+# CHECK:        SectionHeaderEntrySize: 40{{$}}

llvm/test/tools/llvm-elfabi/write-elf64be-ehdr.test

+# RUN: llvm-elfabi %s --output-target=elf64-big %t
+# RUN: llvm-readobj --file-headers %t | FileCheck %s
+
+--- !tapi-tbe
+TbeVersion: 1.0
+Arch: x86_64
+Symbols: {}
+...
+
+# CHECK:      ElfHeader {
+# CHECK-NEXT:   Ident {
+# CHECK-NEXT:     Magic: (7F 45 4C 46)
+# CHECK-NEXT:     Class: 64-bit (0x2)
+# CHECK-NEXT:     DataEncoding: BigEndian (0x2)
+# CHECK-NEXT:     FileVersion: 1{{$}}
+# CHECK-NEXT:     OS/ABI: SystemV (0x0)
+# CHECK-NEXT:     ABIVersion: 0{{$}}
+# CHECK-NEXT:     Unused: (00 00 00 00 00 00 00)
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Type: SharedObject (0x3)
+# CHECK-NEXT:   Machine: EM_X86_64 (0x3E)
+# CHECK-NEXT:   Version: 1{{$}}
+# CHECK-NEXT:   Entry: 0x0{{$}}
+# CHECK:        Flags [ (0x0)
+# CHECK-NEXT:   ]
+# CHECK-NEXT:   HeaderSize: 64{{$}}
+# CHECK-NEXT:   ProgramHeaderEntrySize: 56{{$}}
+# CHECK:        SectionHeaderEntrySize: 64{{$}}

llvm/test/tools/llvm-elfabi/write-elf64le-ehdr.test

+# RUN: llvm-elfabi %s --output-target=elf64-little %t
+# RUN: llvm-readobj --file-headers %t | FileCheck %s
+
+--- !tapi-tbe
+TbeVersion: 1.0
+Arch: AArch64
+Symbols: {}
+...
+
+# CHECK:      ElfHeader {
+# CHECK-NEXT:   Ident {
+# CHECK-NEXT:     Magic: (7F 45 4C 46)
+# CHECK-NEXT:     Class: 64-bit (0x2)
+# CHECK-NEXT:     DataEncoding: LittleEndian (0x1)
+# CHECK-NEXT:     FileVersion: 1{{$}}
+# CHECK-NEXT:     OS/ABI: SystemV (0x0)
+# CHECK-NEXT:     ABIVersion: 0{{$}}
+# CHECK-NEXT:     Unused: (00 00 00 00 00 00 00)
+# CHECK-NEXT:   }
+# CHECK-NEXT:   Type: SharedObject (0x3)
+# CHECK-NEXT:   Machine: EM_AARCH64 (0xB7)
+# CHECK-NEXT:   Version: 1{{$}}
+# CHECK-NEXT:   Entry: 0x0{{$}}
+# CHECK:        Flags [ (0x0)
+# CHECK-NEXT:   ]
+# CHECK-NEXT:   HeaderSize: 64{{$}}
+# CHECK-NEXT:   ProgramHeaderEntrySize: 56{{$}}
+# CHECK:        SectionHeaderEntrySize: 64{{$}}

llvm/tools/llvm-elfabi/ELFObjHandler.h

 #include "llvm/TextAPI/ELF/ELFStub.h"
 
 namespace llvm {
 
 class MemoryBuffer;
 
 namespace elfabi {
 
+enum class ELFTarget {
+  ELF32LE,
+  ELF32BE,
+  ELF64LE,
+  ELF64BE
+};
+
 /// Attempt to read a binary ELF file from a MemoryBuffer.
 Expected<std::unique_ptr<ELFStub>> readELFFile(MemoryBufferRef Buf);
 
+/// Attempt to write a binary ELF stub.
+/// This function determines appropriate ELFType using the passed ELFTarget and
+/// then writes a binary ELF stub to a specified file path.
+///
+/// @param FilePath File path for writing the ELF binary.
+/// @param Stub Source ELFStub to generate a binary ELF stub from.
+/// @param OutputFormat Target ELFType to write binary as.
+Error writeBinaryStub(StringRef FilePath, const ELFStub &Stub,
+                      ELFTarget OutputFormat);
+
 } // end namespace elfabi
 } // end namespace llvm
 
 #endif // LLVM_TOOLS_ELFABI_ELFOBJHANDLER_H

llvm/tools/llvm-elfabi/ELFObjHandler.cpp

 //===- ELFObjHandler.cpp --------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===-----------------------------------------------------------------------===/
 
 #include "ELFObjHandler.h"
+#include "llvm/MC/StringTableBuilder.h"
 #include "llvm/Object/Binary.h"
 #include "llvm/Object/ELFObjectFile.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/FileOutputBuffer.h"
+#include "llvm/Support/MathExtras.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/TextAPI/ELF/ELFStub.h"
 
+#include <functional>
+
 using llvm::MemoryBufferRef;
 using llvm::object::ELFObjectFile;
 
 using namespace llvm;
 using namespace llvm::object;
 using namespace llvm::ELF;
 
-namespace llvm {
-namespace elfabi {
+namespace {
+
+using namespace llvm::elfabi;
 
 // Simple struct to hold relevant .dynamic entries.
 struct DynamicEntries {
   uint64_t StrTabAddr = 0;
   uint64_t StrSize = 0;
   Optional<uint64_t> SONameOffset;
   std::vector<uint64_t> NeededLibNames;
   // Symbol table:
   uint64_t DynSymAddr = 0;
   // Hash tables:
   Optional<uint64_t> ElfHash;
   Optional<uint64_t> GnuHash;
 };
 
+// Lazy assumes that T is default constructable.
+// Lazy also acts like a read-only, move-only type.
+// Lazy is inteneded to be used to form build graphs for ELF stubs. ELF files
+// need to be generated in stages because different decisions depend on each
+// other. By constructing a build graph of lazy values for these decisions we
+// let them auto-resolve the order for them selves. We run the risk of
+// introducing a cyclic dependency when constructing the graphs however. To
+// check for this we overwrite the evaluation function with a so called
+// "blackhole" that causes a failure if evaluating a particular Lazy value
+// depends on the same lazy value becuase the blackhole will be called instead.
+// This type is not currently threadsafe.
+template <class T> class Lazy {
+private:
+  mutable std::function<void(T &)> Func;
+  mutable Optional<T> Value;
+
+public:
+  Lazy(const Lazy &) = delete;
+  Lazy(Lazy &&) = default;
+  // Allow Lazy values to be default constructed to an empty state.
+  Lazy() = default;
+  explicit Lazy(std::function<void(T &)> &&F) : Func(std::move(F)) {}
+
+  Lazy &operator=(std::function<void(T &)> &&F) {
+    // Once a thunk has been assigned don't allow it to change.
+    assert(!Value && !Func);
+    Func = F;
+    return *this;
+  }
+  Lazy &operator=(T &&Val) {
+    assert(!Value && !Func);
+    Value = std::move(Val);
+    return *this;
+  }
+  Lazy &operator=(const T &Val) {
+    assert(!Value && !Func);
+    Value = Val;
+    return *this;
+  }
+
+  const T &operator*() const {
+    // Assert that a value has been assigned, lazy or otherwise.
+    assert(Value || Func);
+    if (Value)
+      return *Value;
+    std::function<void(T&)> TFunc{std::move(Func)};
+    Func = [](T&) { llvm_unreachable("cycle detected"); };
+    Value.emplace();
+    TFunc(*Value);
+    return *Value;
+  }
+  const T *operator->() const { return &**this; }
+};
+
+template <class T> Lazy<T> makeLazy(std::function<void(T &)> &&F) {
+  return Lazy<T>{std::move(F)};
+}
+
 /// This function behaves similarly to StringRef::substr(), but attempts to
 /// terminate the returned StringRef at the first null terminator. If no null
 /// terminator is found, an error is returned.
 ///
 /// @param Str Source string to create a substring from.
 /// @param Offset The start index of the desired substring.
-static Expected<StringRef> terminatedSubstr(StringRef Str, size_t Offset) {
+Expected<StringRef> terminatedSubstr(StringRef Str, size_t Offset) {
   size_t StrEnd = Str.find('\0', Offset);
   if (StrEnd == StringLiteral::npos) {
     return createError(
         "String overran bounds of string table (no null terminator)");
   }
 
   size_t StrLen = StrEnd - Offset;
   return Str.substr(Offset, StrLen);

 
 /// This function populates a DynamicEntries struct using an ELFT::DynRange.
 /// After populating the struct, the members are validated with
 /// some basic sanity checks.
 ///
 /// @param Dyn Target DynamicEntries struct to populate.
 /// @param DynTable Source dynamic table.
 template <class ELFT>
-static Error populateDynamic(DynamicEntries &Dyn,
-                             typename ELFT::DynRange DynTable) {
+Error populateDynamic(DynamicEntries &Dyn, typename ELFT::DynRange DynTable) {
   if (DynTable.empty())
     return createError("No .dynamic section found");
 
   // Search .dynamic for relevant entries.
   bool FoundDynStr = false;
   bool FoundDynStrSz = false;
   bool FoundDynSym = false;
   for (auto &Entry : DynTable) {
if (!FoundDynStrSz) {
     return createError(
         "Couldn't determine dynamic string table size (no DT_STRSZ entry)");
   }
   if (!FoundDynSym) {
     return createError(
         "Couldn't locate dynamic symbol table (no DT_SYMTAB entry)");
   }
   if (Dyn.SONameOffset.hasValue() && *Dyn.SONameOffset >= Dyn.StrSize) {
-    return createStringError(
-        object_error::parse_failed,
+    return createStringError(object_error::parse_failed,
                              "DT_SONAME string offset (0x%016" PRIx64
                              ") outside of dynamic string table",
                              *Dyn.SONameOffset);
   }
   for (uint64_t Offset : Dyn.NeededLibNames) {
     if (Offset >= Dyn.StrSize) {
-      return createStringError(
-          object_error::parse_failed,
+      return createStringError(object_error::parse_failed,
                                "DT_NEEDED string offset (0x%016" PRIx64
                                ") outside of dynamic string table",
                                Offset);
     }
   }
 
   return Error::success();
 }
 
 /// This function finds the number of dynamic symbols using a GNU hash table.
 ///
 /// @param Table The GNU hash table for .dynsym.
 template <class ELFT>
-static uint64_t getDynSymtabSize(const typename ELFT::GnuHash &Table) {
+uint64_t getDynSymtabSize(const typename ELFT::GnuHash &Table) {
   using Elf_Word = typename ELFT::Word;
   if (Table.nbuckets == 0)
     return Table.symndx + 1;
   uint64_t LastSymIdx = 0;
   uint64_t BucketVal = 0;
   // Find the index of the first symbol in the last chain.
   for (Elf_Word Val : Table.buckets()) {
     BucketVal = std::max(BucketVal, (uint64_t)Val);

 
 /// This function determines the number of dynamic symbols.
 /// Without access to section headers, the number of symbols must be determined
 /// by parsing dynamic hash tables.
 ///
 /// @param Dyn Entries with the locations of hash tables.
 /// @param ElfFile The ElfFile that the section contents reside in.
 template <class ELFT>
-static Expected<uint64_t> getNumSyms(DynamicEntries &Dyn,
-                                     const ELFFile<ELFT> &ElfFile) {
+Expected<uint64_t> getNumSyms(DynamicEntries &Dyn,
   using Elf_Hash = typename ELFT::Hash;
   using Elf_GnuHash = typename ELFT::GnuHash;
   // Search GNU hash table to try to find the upper bound of dynsym.
   if (Dyn.GnuHash.hasValue()) {
     Expected<const uint8_t *> TablePtr = ElfFile.toMappedAddr(*Dyn.GnuHash);
     if (!TablePtr)
       return TablePtr.takeError();
     const Elf_GnuHash *Table =

 }
 
 /// This function extracts symbol type from a symbol's st_info member and
 /// maps it to an ELFSymbolType enum.
 /// Currently, STT_NOTYPE, STT_OBJECT, STT_FUNC, and STT_TLS are supported.
 /// Other symbol types are mapped to ELFSymbolType::Unknown.
 ///
 /// @param Info Binary symbol st_info to extract symbol type from.
-static ELFSymbolType convertInfoToType(uint8_t Info) {
+ELFSymbolType convertInfoToType(uint8_t Info) {
   Info = Info & 0xf;
   switch (Info) {
-    case ELF::STT_NOTYPE:
-      return ELFSymbolType::NoType;
-    case ELF::STT_OBJECT:
-      return ELFSymbolType::Object;
-    case ELF::STT_FUNC:
-      return ELFSymbolType::Func;
-    case ELF::STT_TLS:
-      return ELFSymbolType::TLS;
-    default:
-      return ELFSymbolType::Unknown;
   }
 }
 
 /// This function creates an ELFSymbol and populates all members using
 /// information from a binary ELFT::Sym.
 ///
 /// @param SymName The desired name of the ELFSymbol.
 /// @param RawSym ELFT::Sym to extract symbol information from.
 template <class ELFT>
-static ELFSymbol createELFSym(StringRef SymName,
-                              const typename ELFT::Sym &RawSym) {
+ELFSymbol createELFSym(StringRef SymName, const typename ELFT::Sym &RawSym) {
   ELFSymbol TargetSym(SymName);
   uint8_t Binding = RawSym.getBinding();
   if (Binding == STB_WEAK)
     TargetSym.Weak = true;
   else
     TargetSym.Weak = false;
 
   TargetSym.Undefined = RawSym.isUndefined();

 
 /// This function populates an ELFStub with symbols using information read
 /// from an ELF binary.
 ///
 /// @param TargetStub ELFStub to add symbols to.
 /// @param DynSym Range of dynamic symbols to add to TargetStub.
 /// @param DynStr StringRef to the dynamic string table.
 template <class ELFT>
-static Error populateSymbols(ELFStub &TargetStub,
-                            const typename ELFT::SymRange DynSym,
-                            StringRef DynStr) {
+Error populateSymbols(ELFStub &TargetStub, const typename ELFT::SymRange DynSym,
   // Skips the first symbol since it's the NULL symbol.
   for (auto RawSym : DynSym.drop_front(1)) {
     // If a symbol does not have global or weak binding, ignore it.
     uint8_t Binding = RawSym.getBinding();
     if (!(Binding == STB_GLOBAL || Binding == STB_WEAK))
       continue;
     // If a symbol doesn't have default or protected visibility, ignore it.
     uint8_t Visibility = RawSym.getVisibility();
for (auto RawSym : DynSym.drop_front(1)) {
     // TODO: Populate symbol warning.
   }
   return Error::success();
 }
 
 /// Returns a new ELFStub with all members populated from an ELFObjectFile.
 /// @param ElfObj Source ELFObjectFile.
 template <class ELFT>
-static Expected<std::unique_ptr<ELFStub>>
+Expected<std::unique_ptr<ELFStub>>
 buildStub(const ELFObjectFile<ELFT> &ElfObj) {
   using Elf_Dyn_Range = typename ELFT::DynRange;
   using Elf_Phdr_Range = typename ELFT::PhdrRange;
   using Elf_Sym_Range = typename ELFT::SymRange;
   using Elf_Sym = typename ELFT::Sym;
   std::unique_ptr<ELFStub> DestStub = make_unique<ELFStub>();
   const ELFFile<ELFT> *ElfFile = ElfObj.getELFFile();
   // Fetch .dynamic table.
   Expected<Elf_Dyn_Range> DynTable = ElfFile->dynamicEntries();
   if (!DynTable) {
     return DynTable.takeError();
   }
 
   // Fetch program headers.
   Expected<Elf_Phdr_Range> PHdrs = ElfFile->program_headers();
   if (!PHdrs) {
     return PHdrs.takeError();
   }
 
   // Collect relevant .dynamic entries.
   DynamicEntries DynEnt;
   if (Error Err = populateDynamic<ELFT>(DynEnt, *DynTable))
     return std::move(Err);
 
-    // Get pointer to in-memory location of .dynstr section.
   Expected<const uint8_t *> DynStrPtr =
       ElfFile->toMappedAddr(DynEnt.StrTabAddr);
   if (!DynStrPtr)
     return appendToError(DynStrPtr.takeError(),
                          "when locating .dynstr section contents");
 
   StringRef DynStr(reinterpret_cast<const char *>(DynStrPtr.get()),
                    DynEnt.StrSize);
if (!SymCount)
     return SymCount.takeError();
   if (*SymCount > 0) {
     // Get pointer to in-memory location of .dynsym section.
     Expected<const uint8_t *> DynSymPtr =
         ElfFile->toMappedAddr(DynEnt.DynSymAddr);
     if (!DynSymPtr)
       return appendToError(DynSymPtr.takeError(),
                            "when locating .dynsym section contents");
-    Elf_Sym_Range DynSyms =
-        ArrayRef<Elf_Sym>(reinterpret_cast<const Elf_Sym *>(*DynSymPtr),
-                          *SymCount);
+    Elf_Sym_Range DynSyms = ArrayRef<Elf_Sym>(
+        reinterpret_cast<const Elf_Sym *>(*DynSymPtr), *SymCount);
     Error SymReadError = populateSymbols<ELFT>(*DestStub, DynSyms, DynStr);
     if (SymReadError)
       return appendToError(std::move(SymReadError),
                            "when reading dynamic symbols");
   }
 
   return std::move(DestStub);
 }
 
+/// This initializes an ELF file header with information specific to a binary
+/// dynamic shared object.
+/// Offsets, indexes, links, etc. for section and program headers are just
+/// zero-initialized as they will be updated elsewhere.
+///
+/// @param ElfHeader Target ELFT::Ehdr to populate.
+/// @param Machine Target architecture (e_machine from ELF specifications).
+template <class ELFT>
+void initELFHeader(typename ELFT::Ehdr &ElfHeader, uint16_t Machine) {
+  using Elf_Ehdr = typename ELFT::Ehdr;
+  using Elf_Phdr = typename ELFT::Phdr;
+  using Elf_Shdr = typename ELFT::Shdr;
+
+  memset(&ElfHeader, 0, sizeof(Elf_Ehdr));
+  // ELF identification.
+  ElfHeader.e_ident[EI_MAG0] = ELFMAG0;
+  ElfHeader.e_ident[EI_MAG1] = ELFMAG1;
+  ElfHeader.e_ident[EI_MAG2] = ELFMAG2;
+  ElfHeader.e_ident[EI_MAG3] = ELFMAG3;
+  ElfHeader.e_ident[EI_CLASS] = ELFT::Is64Bits ? ELFCLASS64 : ELFCLASS32;
+  bool IsLittleEndian = ELFT::TargetEndianness == support::little;
+  ElfHeader.e_ident[EI_DATA] = IsLittleEndian ? ELFDATA2LSB : ELFDATA2MSB;
+  ElfHeader.e_ident[EI_VERSION] = EV_CURRENT;
+  ElfHeader.e_ident[EI_OSABI] = ELFOSABI_NONE;
+  ElfHeader.e_ident[EI_ABIVERSION] = 0;
+
+  // Remainder of ELF header.
+  ElfHeader.e_type = ET_DYN;
+  ElfHeader.e_machine = Machine;
+  ElfHeader.e_version = EV_CURRENT;
+  ElfHeader.e_entry = 0;
+  ElfHeader.e_flags = 0;
+  ElfHeader.e_ehsize = sizeof(Elf_Ehdr);
+  ElfHeader.e_phentsize = sizeof(Elf_Phdr);
+  ElfHeader.e_shentsize = sizeof(Elf_Shdr);
+}
+
+template <class ELFT> struct OutputSection {
+  using Elf_Shdr = typename ELFT::Shdr;
+  std::string Name;
+  Lazy<Elf_Shdr> Shdr;
+  Lazy<uint64_t> Addr;
+  Lazy<uint64_t> Offset;
+  Lazy<uint64_t> Size;
+  Lazy<uint64_t> Align;
+  uint32_t Index;
+  bool NoBits = true;
+};
+
+template <class T, class ELFT>
+struct ContentSection : public OutputSection<ELFT> {
+  Lazy<T> Content;
+  ContentSection() { this->NoBits = false; }
+};
+
+// Lazy<_> requires a default constructable type but StringTableBuilder is not
+// default constructable. This class just wraps it for the purpose of adding a
+// default constructor.
+class ELFStringTableBuilder : public StringTableBuilder {
+public:
+  ELFStringTableBuilder() : StringTableBuilder(StringTableBuilder::ELF) {}
+};
+
+template <class ELFT> class ELFBuilder {
+public:
+  using Elf_Ehdr = typename ELFT::Ehdr;
+  using Elf_Shdr = typename ELFT::Shdr;
+  using Elf_Phdr = typename ELFT::Phdr;
+  using Elf_Sym = typename ELFT::Sym;
+  using Elf_Addr = typename ELFT::Addr;
+  using Elf_Dyn = typename ELFT::Dyn;
+
+private:
+  Lazy<Elf_Ehdr> ElfHeader;
+  ContentSection<ELFStringTableBuilder, ELFT> StrTab;
+
+  template <class T> static void Write(uint8_t *Data, const T &Value) {
+    *reinterpret_cast<T *>(Data) = Value;
+  }
+  uint64_t ShdrOffset(const OutputSection<ELFT> &Sec) const {
+    return ElfHeader->e_shoff + Sec.Index * sizeof(Elf_Shdr);
+  }
+  void WriteShdr(uint8_t *Data, const OutputSection<ELFT> &Sec) const {
+    Write(Data + ShdrOffset(Sec), *Sec.Shdr);
+  }
+
+public:
+  ELFBuilder(const ELFBuilder &) = delete;
+  ELFBuilder(ELFBuilder &&) = default;
+  explicit ELFBuilder(const ELFStub &Stub) {
+    std::vector<OutputSection<ELFT> *> Sections;
+    Sections.push_back(&StrTab);
+
+    // Each Lazy value we define here should be thought of as nodes in a build
+    // graph. The dependencies of each step are implicitly captured by useage
+    // and are not explicitlly stored either. At the end we'll request each of
+    // the different parts to be written out and no extra work will be done if
+    // we transitively depend on the same thing twice. Below you can see that
+    // LastSection->Offset and LastSection->Size are depended on here despite
+    // us not setting those lazy values until later. The means by which to
+    // evaluate them will be defined later but during evaluation when we
+    // request the ElfHeader content it will go on to request the LastSection's
+    // Offset and Size which might in turn request other things be evaluated.
+    const OutputSection<ELFT> *LastSection = Sections.back();
+    ElfHeader = [this, &Stub, LastSection](Elf_Ehdr &Ehdr) {
+      initELFHeader<ELFT>(Ehdr, Stub.Arch);
+      Ehdr.e_shstrndx = StrTab.Index;
+      Ehdr.e_shnum = LastSection->Index + 1;
+      Ehdr.e_phnum = 0;
+      Ehdr.e_phoff = 0;
+      if (LastSection->NoBits)
+        Ehdr.e_shoff = alignTo(*LastSection->Offset, sizeof(Elf_Addr));
+      else
+        Ehdr.e_shoff = alignTo(*LastSection->Offset + *LastSection->Size,
+                               sizeof(Elf_Addr));
+    };
+
+    // Manually set the indexes of these.
+    uint64_t Index = 1;
+    const OutputSection<ELFT> *Prev = nullptr;
+    // TODO: Push this back for ProgramHeaders
+    uint64_t StartOffset = sizeof(Elf_Ehdr);
+    // Now set the Index, Offset, and Addr of everything.
+    for (auto Sec : Sections) {
+      Sec->Index = Index++;
+      Sec->Offset = [=](uint64_t &Offset) {
+        uint64_t Align = *Sec->Align ? *Sec->Align : 1;
+        // Don't align the offset of a NOBITS section.
+        if (Sec->NoBits)
+          Align = 1;
+        if (Prev == nullptr)
+          Offset = alignTo(StartOffset, Align);
+        else if (Prev->NoBits)
+          Offset = alignTo(*Prev->Offset, Align);
+        else
+          Offset = alignTo(*Prev->Offset + *Prev->Size, Align);
+      };
+      Sec->Addr = [=](uint64_t &Addr) {
+        uint64_t Align = *Sec->Align ? *Sec->Align : 1;
+        if (Prev == nullptr)
+          Addr = alignTo(StartOffset, Align);
+        else
+          Addr = alignTo(*Prev->Addr + *Prev->Size, Align);
+      };
+      Prev = Sec;
+    }
+
+    // Define the string table.
+    StrTab.Name = ".dynstr";
+    StrTab.Shdr = [this](Elf_Shdr &Shdr) {
+      Shdr.sh_name = StrTab.Content->getOffset(StrTab.Name);
+      Shdr.sh_type = SHT_STRTAB;
+      Shdr.sh_flags = SHF_ALLOC;
+      Shdr.sh_size = *StrTab.Size;
+      Shdr.sh_addr = *StrTab.Addr;
+      Shdr.sh_offset = *StrTab.Offset;
+    };
+    StrTab.Size = [this](uint64_t &Size) { Size = StrTab.Content->getSize(); };
+    StrTab.Align = 0;
+    StrTab.Content = [this](ELFStringTableBuilder &Builder) {
+      Builder.add(StrTab.Name);
+      Builder.finalize();
+    };
+  }
+
+  // GetSize will effectivelly compute the whole layout.
+  size_t GetSize() const {
+    return ElfHeader->e_shoff + ElfHeader->e_shnum * sizeof(Elf_Shdr);
+  }
+
+  void Write(uint8_t *Data) const {
+    Write(Data, *ElfHeader);
+    StrTab.Content->write(Data + StrTab.Shdr->sh_offset);
+    WriteShdr(Data, StrTab);
+  }
+};
+
+/// This function opens a file for writing and then writes a binary ELF stub to
+/// the file.
+///
+/// @param FilePath File path for writing the ELF binary.
+/// @param Stub Source ELFStub to generate a binary ELF stub from.
+template <class ELFT>
+Error writeELFBinaryToFile(StringRef FilePath, const ELFStub &Stub) {
+  ELFBuilder<ELFT> Builder{Stub};
+  Expected<std::unique_ptr<FileOutputBuffer>> BufOrError =
+      FileOutputBuffer::create(FilePath, Builder.GetSize());
+  if (!BufOrError) {
+    Error FileReadError = BufOrError.takeError();
+    std::string Message;
+    raw_string_ostream Stream(Message);
+    Stream << FileReadError;
+    Stream << " when trying to open `" << FilePath << "` for writing";
+    consumeError(std::move(FileReadError));
+    return createStringError(errc::invalid_argument, Stream.str().c_str());
+  }
+
+  // Write binary to file.
+  std::unique_ptr<FileOutputBuffer> Buf = std::move(*BufOrError);
+  Builder.Write(Buf->getBufferStart());
+
+  if (Error FileWriteError = Buf->commit())
+    return FileWriteError;
+
+  return Error::success();
+}
+
+} // end namespace
+
+namespace llvm {
+namespace elfabi {
+
+// This function wraps the ELFT writeELFBinaryToFile() so writeBinaryStub()
+// can be called without having to use ELFType templates directly.
+Error writeBinaryStub(StringRef FilePath, const ELFStub &Stub,
+                      ELFTarget OutputFormat) {
+  if (OutputFormat == ELFTarget::ELF32LE) {
+    return writeELFBinaryToFile<ELF32LE>(FilePath, Stub);
+  } else if (OutputFormat == ELFTarget::ELF32BE) {
+    return writeELFBinaryToFile<ELF32BE>(FilePath, Stub);
+  } else if (OutputFormat == ELFTarget::ELF64LE) {
+    return writeELFBinaryToFile<ELF64LE>(FilePath, Stub);
+  } else if (OutputFormat == ELFTarget::ELF64BE) {
+    return writeELFBinaryToFile<ELF64BE>(FilePath, Stub);
+  }
+  return createStringError(errc::invalid_argument,
+                           "Invalid binary output target");
+}
+
 Expected<std::unique_ptr<ELFStub>> readELFFile(MemoryBufferRef Buf) {
   Expected<std::unique_ptr<Binary>> BinOrErr = createBinary(Buf);
   if (!BinOrErr) {
     return BinOrErr.takeError();
   }
 
   Binary *Bin = BinOrErr->get();
   if (auto Obj = dyn_cast<ELFObjectFile<ELF32LE>>(Bin)) {

llvm/tools/llvm-elfabi/llvm-elfabi.cpp

 cl::opt<std::string>
     EmitTBE("emit-tbe",
             cl::desc("Emit a text-based ELF stub (.tbe) from the input file"),
             cl::value_desc("path"));
 cl::opt<std::string> SOName(
     "soname",
     cl::desc("Manually set the DT_SONAME entry of any emitted files"),
     cl::value_desc("name"));
+cl::opt<ELFTarget> BinaryOutputTarget(
+    "output-target", cl::desc("Create a binary stub for the specified target"),
+    cl::values(clEnumValN(ELFTarget::ELF32LE, "elf32-little",
+                          "32-bit little-endian ELF stub"),
+               clEnumValN(ELFTarget::ELF32BE, "elf32-big",
+                          "32-bit big-endian ELF stub"),
+               clEnumValN(ELFTarget::ELF64LE, "elf64-little",
+                          "64-bit little-endian ELF stub"),
+               clEnumValN(ELFTarget::ELF64BE, "elf64-big",
+                          "64-bit big-endian ELF stub")));
+cl::opt<std::string> BinaryOutputFilePath(cl::Positional, cl::desc("output"));
 
 /// writeTBE() writes a Text-Based ELF stub to a file using the latest version
 /// of the YAML parser.
 static Error writeTBE(StringRef FilePath, ELFStub &Stub) {
   std::error_code SysErr;
 
   // Open file for writing.
   raw_fd_ostream Out(FilePath, SysErr);
EC.addError(createStringError(errc::not_supported,
                                 "No file readers succeeded reading `%s` "
                                 "(unsupported/malformed file?)",
                                 FilePath.data()),
               "ReadInputFile");
   EC.escalateToFatal();
   return EC.makeError();
 }
 
 int main(int argc, char *argv[]) {
   // Parse arguments.
   cl::ParseCommandLineOptions(argc, argv);
 
   Expected<std::unique_ptr<ELFStub>> StubOrErr = readInputFile(InputFilePath);
   if (!StubOrErr) {
     Error ReadError = StubOrErr.takeError();
     WithColor::error() << ReadError << "\n";
     exit(1);
   }
 
   std::unique_ptr<ELFStub> TargetStub = std::move(StubOrErr.get());
 
-  // Write out .tbe file.
-  if (EmitTBE.getNumOccurrences() == 1) {
-    TargetStub->TbeVersion = TBEVersionCurrent;
-    if (SOName.getNumOccurrences() == 1) {
-      TargetStub->SoName = SOName;
-    }
+  // Change SoName before emitting stubs.
+
+  // Write out .tbe file.
+  if (EmitTBE.getNumOccurrences() == 1) {
+    TargetStub->TbeVersion = TBEVersionCurrent;
     Error TBEWriteError = writeTBE(EmitTBE, *TargetStub);
     if (TBEWriteError) {
       WithColor::error() << TBEWriteError << "\n";
       exit(1);
     }
   }
+
+  // Write out binary ELF stub.
+  if (BinaryOutputFilePath.getNumOccurrences() == 1) {
+    if (BinaryOutputTarget.getNumOccurrences() == 0) {
+      WithColor::error() << "No binary output target specified.\n";
+      exit(1);
+    }
+    Error BinaryWriteError = writeBinaryStub(BinaryOutputFilePath, *TargetStub,
+                                             BinaryOutputTarget);
+    if (BinaryWriteError) {
+      WithColor::error() << BinaryWriteError << "\n";
+      exit(1);
+    }
+  }
 }