[yaml2obj][ELF] Add support for program headers

[jakehehrlich]

Summary

This change adds basic support for program headers.

I need to do some testing which requires generating program headers but I can't use ld.lld or clang to produce programs that have headers. I'd also like to test some strange things that those programs may never produce.

[silvas]

It is too fragile to manually specify file offsets. The actual layout that yaml2obj does right now should be considered opaque, such that any file offsets you could manually choose are "wrong" (even if they are incidentally correct with the current layout of sections that yaml2obj chooses).

So while this is probably an interesting incremental step, we don't want this in-tree yet as-is (if I understand correctly; it might be useful to add in a basic test to make things concrete). We definitely want to have the layout portion so that we don't choose brittle offsets. I don't think it should be too difficult to add a field on the program header yaml object that lists which section names should be covered by that program header.

[jakehehrlich]

This now sets offsets to be the minimum offset of a collection of sections but does not set file size or memory size. If no sections are specified the Offset is set to zero.

[atanasyan]

And you definitely need to add some tests cases.

[jakehehrlich]

Added a test for program headers

[silvas]

In D35276#807306, @jakehehrlich wrote:

This now sets offsets to be the minimum offset of a collection of sections but does not set file size or memory size. If no sections are specified the Offset is set to zero.

Do you need the ability to manually set file size to a specific numeric value? Since that allows creating weird/broken files, I would prefer not to have it in an initial commit. Or in other words, I would consider it a separate feature specifically for creating broken ELF files (and this means it would require a separate test too).

The overall design philosophy for ELF yaml2obj is to by default create fairly sane object files (specifically, having it be impossible by default to have wild offsets / sizes). We are willing to sacrifice a bit of generality to keep this (otherwise juggling raw sizes/offsets is way too error-prone). For a couple fields we've added the ability to specify raw numeric values either to create files broken in specific ways or to create a slightly more general subset of ELF files, but those are generally added after the fact specifically for that purpose.

So according to that philosophy, I think it is worth fleshing out this initial commit to actually set the program header file size to cover the listed sections (and set mem size in a default way based on that).

test/tools/yaml2obj/program-header.yaml

29 ↗

(On Diff #106353)

The spec requires VAddr and PAddr to be equal to the offset in the file modulo the page size / alignment. Even though we are treating offsets as opaque, they are all aligned to the respective boundaries respect alignment. So that means that the VAddr and PAddr must be addresses aligned at least to the alignment of the phdr.

You might want to do something like Sec->Align = max(Sec->Align, PHdr->Align) for all sections Sec covered by phdr Phdr to ensure that the section layout respects the PHdr's alignment.

[jakehehrlich]

This comment was removed by jakehehrlich.

[jakehehrlich]

Yeah, I think I'll probably need to add a special case that lets me take over how layout occurs here. After we work this diff out I'll put together a list of use cases that I'll need and I'll post something on llvm-dev so that we can decide what would fill those use cases best.

In D35276#808629, @silvas wrote:

In D35276#807306, @jakehehrlich wrote:

This now sets offsets to be the minimum offset of a collection of sections but does not set file size or memory size. If no sections are specified the Offset is set to zero.

Do you need the ability to manually set file size to a specific numeric value? Since that allows creating weird/broken files, I would prefer not to have it in an initial commit. Or in other words, I would consider it a separate feature specifically for creating broken ELF files (and this means it would require a separate test too).

The overall design philosophy for ELF yaml2obj is to by default create fairly sane object files (specifically, having it be impossible by default to have wild offsets / sizes). We are willing to sacrifice a bit of generality to keep this (otherwise juggling raw sizes/offsets is way too error-prone). For a couple fields we've added the ability to specify raw numeric values either to create files broken in specific ways or to create a slightly more general subset of ELF files, but those are generally added after the fact specifically for that purpose.

So according to that philosophy, I think it is worth fleshing out this initial commit to actually set the program header file size to cover the listed sections (and set mem size in a default way based on that).

There's some tricky cases that need to be accounted for then.

1. If there is a PT_LOAD segment covered by 3 sections such that the middle section is a NOBITS section then that middle section should take up space in the file image. This means that section layout changes depends on the presence of a PT_LOAD segment. This is a pretty non-trivial change to make to layout. Do you want me to go ahead and add this change to this diff?
2. I think accounting PT_PHDR could be a special case that we can handle in a later diff.
3. There's some strange behavior in dealing with TLS segments and sections. Namely there's a case which contradicts case 1) above when a NOBITS section is at the end of a TLS segment but not at the end of a larger PT_LOAD segment. In this case, sometimes the NOBITS section is not given space in the file and dynamic loaders have handle it differently. I'd like to be able to test for this case. I think it's fair for something like this to be an extension.

Cases 2 and 3 are cases where the sensible thing to do isn't what will be done by just specifying which sections should be covered. I think there might be a few more cases like these lurking in the corners. Should these cases be handled by extensions as they come up? Or should we be trying to account for these in this change?
Case 1 is a little complicated to add but the correct behavior is specifiable by just listing sections that should cover the segment.

In the mean time I've got a diff and a couple tests prepared that I think handle the common normal cases sensibly. But this change fails on issues 1-3 above.

[jakehehrlich]

1. Added a test for no-bits sections
2. Modified test from previous diff
3. Memory size and file size are now computed from the sections

This does not properly handle the case where a NOBITS section is in the middle of two PROGBITS sections. There is also no way to create certain valid program headers but these are corner cases.

[silvas]

I think you should be able to deal with most of those issues by sorting the list of sections with an appropriate sorting predicate that coalesces sections that need to be adjacent, which should be fairly unintrusive.

I don't know if you really need to do that though. Again, we're mostly just trying to avoid accidentally creating really blatantly wrong files like wild offsets. Things that are semantically wrong at a higher level are not critical; for example, the contents of the .text section don't need to be real instructions. One way to look at this perhaps is to think that if you are modifying a yaml file, you aren't going to need to have a feedback loop where you manually inspect the output object file to then e.g. write in a magical offset back into the yaml file -- that's brittle, error prone, and annoying.

You want the yaml file to contains stuff that matters for testing what you care about and avoid anything that doesn't matter as much as possible.

Sometimes having yaml2obj do some work automatically for you to prevent higher-level semantic invalid stuff (like we do for relocations, which would otherwise need to be manually encoded in the section content) is a decent tradeoff, sometimes just detecting something is wrong and giving an error is a good tradeoff, sometimes letting something semantically wrong at a higher level silently go through is all that is practical. Combined with the fact that sometimes we want to create invalid stuff because yaml2obj is a testing tool, means that there's no a priori right answer.

Overall, there is a tradeoff between generality of yaml2obj (e.g. how broken the files it produces can be) and how error-prone it is to use. Conversely, how convenient it is to use is a tradeoff vs how many different testing scenarios it can be used for. So my main advice is to think about the use cases you want to use it for and strike a balance; it's meant to make life easier for writing tests. If it isn't doing that then you don't need to use it. Some use cases might be outliers for which you are really just better off checking in a binary file for the test.

Also, for the record you should be able to do a lot of this by passing a linker script to ld.lld e.g. using the PHDRS command and appropriate literal data commands. See e.g. lld/test/ELF/linkerscript/phdrs.s and lld/test/ELF/linkerscript/data-commands.s.
I'm not sure if that is enough (especially w.r.t. creating broken files) for what you want to do, but the basics of creating an output file with certain phdrs that cover certain sections with some dummy content can be done like that.

test/tools/yaml2obj/program-header.yaml

34 ↗

(On Diff #106564)

You might want to consider defaulting the alignment to be the max of the alignment of all included sections for now.

I imagine that in most tests you don't actually care about the alignment, so being able to omit it and get some sane default would be good.

Also, I think it will require a bit of extra code to ensure that manual alignment is respected (e.g. if the PT_LOAD has an alignment exceeding that of the sections it covers, it will need to be rounded down to the alignment) . So removing Align from the yaml input format altogether might be the right choice in the first patch.

tools/yaml2obj/yaml2elf.cpp

354 ↗

(On Diff #106564)

Capitalize.

[jakehehrlich]

1. Fixed capitalization typo
2. Removed the align field
3. The alignment of a segment is now the maximum alignment of all sections that have the same offset. If there are no sections, the alignment is 1.

If the alignment was set to be the maximum of all sections in the segment then an invalid case could occur where a section in the middle has a larger alignment than the first section and the segment would start at the offset of the first section while having an alignment that didn't agree with that offset.

[jakehehrlich]

In D35276#809102, @silvas wrote:

I think you should be able to deal with most of those issues by sorting the list of sections with an appropriate sorting predicate that coalesces sections that need to be adjacent, which should be fairly unintrusive.

I don't know if you really need to do that though. Again, we're mostly just trying to avoid accidentally creating really blatantly wrong files like wild offsets. Things that are semantically wrong at a higher level are not critical; for example, the contents of the .text section don't need to be real instructions. One way to look at this perhaps is to think that if you are modifying a yaml file, you aren't going to need to have a feedback loop where you manually inspect the output object file to then e.g. write in a magical offset back into the yaml file -- that's brittle, error prone, and annoying.

I'll try and keep that in mind as I propose extensions to this for my testing. This two step process was what I initially had in mind but I couldn't really think of another way of going about it. Being able to upload binaries as test cases alleviates a lot of my concerns.

Overall, there is a tradeoff between generality of yaml2obj (e.g. how broken the files it produces can be) and how error-prone it is to use. Conversely, how convenient it is to use is a tradeoff vs how many different testing scenarios it can be used for. So my main advice is to think about the use cases you want to use it for and strike a balance; it's meant to make life easier for writing tests. If it isn't doing that then you don't need to use it. Some use cases might be outliers for which you are really just better off checking in a binary file for the test.

I didn't realize it was acceptable to upload binaries! I'll probably be using that trick in the near future to test these trickier cases. I think this change will suffice to handle the less tricky cases. This all makes more sense now. I thought yaml2obj was the only way I could test some of this stuff so I really wanted full control over the output. Now that I know there's another way I agree strongly that yaml2obj should make producing sensible binaries easy.

Also, for the record you should be able to do a lot of this by passing a linker script to ld.lld e.g. using the PHDRS command and appropriate literal data commands. See e.g. lld/test/ELF/linkerscript/phdrs.s and lld/test/ELF/linkerscript/data-commands.s.
I'm not sure if that is enough (especially w.r.t. creating broken files) for what you want to do, but the basics of creating an output file with certain phdrs that cover certain sections with some dummy content can be done like that.

I think that should be good actually. And if it isn't then for the few cases where that doesn't work I can resort to some hex editing or some such other trickery.

[silvas]

In D35276#809811, @jakehehrlich wrote:

If the alignment was set to be the maximum of all sections in the segment then an invalid case could occur where a section in the middle has a larger alignment than the first section and the segment would start at the offset of the first section while having an alignment that didn't agree with that offset.

Technically the way you've handled this here is incorrect I think. For example, suppose you have two sections, one is strings (or whatever) at an alignment of 1 and another is some other data with alignment of 4. A PT_LOAD covering both must have an alignment of 4 so that the data is loaded in a way consistent with its alignment. IIUC, depending on the order of the sections, I believe that the current patch will give the PT_LOAD an alignment of 4.

You should be able to handle this by setting the alignment of the PT_LOAD to the max of all sections is must contain. Then rounding down p_offset to that alignment (and adjusting the size too).

In D35276#809847, @jakehehrlich wrote:

I didn't realize it was acceptable to upload binaries! I'll probably be using that trick in the near future to test these trickier cases. I think this change will suffice to handle the less tricky cases. This all makes more sense now. I thought yaml2obj was the only way I could test some of this stuff so I really wanted full control over the output. Now that I know there's another way I agree strongly that yaml2obj should make producing sensible binaries easy.

We try to avoid checking in binaries, but sometimes it is the most sensible way. The binaries should usually be fairly small (e.g. produced by a reduced test case, perhaps with some hex editing of the binary to corrupt some field).

Also, for the record you should be able to do a lot of this by passing a linker script to ld.lld e.g. using the PHDRS command and appropriate literal data commands. See e.g. lld/test/ELF/linkerscript/phdrs.s and lld/test/ELF/linkerscript/data-commands.s.
I'm not sure if that is enough (especially w.r.t. creating broken files) for what you want to do, but the basics of creating an output file with certain phdrs that cover certain sections with some dummy content can be done like that.

I think that should be good actually. And if it isn't then for the few cases where that doesn't work I can resort to some hex editing or some such other trickery.

One logistical detail is that using ld.lld in the tests would require the tool living in LLD's repo. This isn't impossible, but somewhat annoying if the only reason to do it is for testing.
Overall for LLD/ELF we mostly rely on inputs generated by ordinary tools unless we're testing something specifically strange or that can't be produced by ordinary tools.
So using ld.lld / llvm-mc to create your objcopy inputs is probably the thing most consistent with that. Ultimately, almost all user inputs to objcopy are going to be produced by ordinary tools so it makes sense that most test cases (except error/corrupt cases) should be generatable with such tools.

So you can look at lld/test/ELF/ and compare:

• binaries: .o files
• YAML tests: .test files (or files containing !ELF to be precise)
• tool-produced inputs: .s / .ll

The vast majority of test inputs are tool-produced.

[jakehehrlich]

Technically the way you've handled this here is incorrect I think. For example, suppose you have two sections, one is strings (or whatever) at an alignment of 1 and another is some other data with alignment of 4. A PT_LOAD covering both must have an alignment of 4 so that the data is loaded in a way consistent with its alignment. IIUC, depending on the order of the sections, I believe that the current patch will give the PT_LOAD an alignment of 4.

I'm not sure I follow. The current patch, as I intended it and as I understand it, does the following:

1. Sections are laid out appropriately according to their alignments and other constraints (this is not changed in this patch, this is just existing working code doing it's job)
2. Segments are expanded to fit the needed sections. Most notably p_offset becomes the sh_offset of the first section.
3. Segment alignment is set to the alignment of the first section (which is also the same section that p_offset got it's value from). Because the first segment's alignment and offset are valid so too should the alignment and offset of the segment be valid.

Are you concerned about alignment in memory after loading? Since the virtual address is chosen by the user I don't think that matters. If there is ever a case where the loader chooses the virtual address based on alignment then I'd agree, we should tweak p_offset to account for maximum alignment. Otherwise I think having the first section's offset agree with the segment offset is more sensible because the user will be picking the virtual address and there's nothing we can do about that.

[silvas]

In D35276#812007, @jakehehrlich wrote:

Are you concerned about alignment in memory after loading? Since the virtual address is chosen by the user I don't think that matters. If there is ever a case where the loader chooses the virtual address based on alignment then I'd agree, we should tweak p_offset to account for maximum alignment. Otherwise I think having the first section's offset agree with the segment offset is more sensible because the user will be picking the virtual address and there's nothing we can do about that.

Yes that is my concern. If you have a section that requires 2M alignment, then with your current patch a PT_LOAD covering it can end up with 4k alignment and thus the section might be loaded at an address that isn't 2M aligned. That's a bit of a technicality though (probably only matters for position independent code where the loader is choosing where to load it). Overall this LGTM.

In practice, section alignment is much smaller (~ size of a data type) compared to segment alignment (~ size of a page). A case where a section has a massive alignment is theoretically possible though.

llvm/trunk/include/llvm/ObjectYAML/ELFYAML.h

 // and are appropriately converted to the necessary endianness when
 // reading/generating binary object files.
 // The naming of these types is intended to be ELF_PREFIX, where PREFIX is
 // the common prefix of the respective constants. E.g. ELF_EM corresponds
 // to the `e_machine` constants, like `EM_X86_64`.
 // In the future, these would probably be better suited by C++11 enum
 // class's with appropriate fixed underlying type.
 LLVM_YAML_STRONG_TYPEDEF(uint16_t, ELF_ET)
+LLVM_YAML_STRONG_TYPEDEF(uint32_t, ELF_PT)
 LLVM_YAML_STRONG_TYPEDEF(uint32_t, ELF_EM)
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, ELF_ELFCLASS)
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, ELF_ELFDATA)
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, ELF_ELFOSABI)
 // Just use 64, since it can hold 32-bit values too.
 LLVM_YAML_STRONG_TYPEDEF(uint64_t, ELF_EF)
+LLVM_YAML_STRONG_TYPEDEF(uint32_t, ELF_PF)
 LLVM_YAML_STRONG_TYPEDEF(uint32_t, ELF_SHT)
 LLVM_YAML_STRONG_TYPEDEF(uint32_t, ELF_REL)
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, ELF_RSS)
 // Just use 64, since it can hold 32-bit values too.
 LLVM_YAML_STRONG_TYPEDEF(uint64_t, ELF_SHF)
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, ELF_STT)
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, ELF_STV)
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, ELF_STO)
struct FileHeader {
   ELF_ELFDATA Data;
   ELF_ELFOSABI OSABI;
   ELF_ET Type;
   ELF_EM Machine;
   ELF_EF Flags;
   llvm::yaml::Hex64 Entry;
 };
 
+struct SectionName {
+  StringRef Section;
+};
+
+struct ProgramHeader {
+  ELF_PT Type;
+  ELF_PF Flags;
+  llvm::yaml::Hex64 VAddr;
+  llvm::yaml::Hex64 PAddr;
+  std::vector<SectionName> Sections;
+};
+
 struct Symbol {
   StringRef Name;
   ELF_STT Type;
   StringRef Section;
   llvm::yaml::Hex64 Value;
   llvm::yaml::Hex64 Size;
   uint8_t Other;
 };
struct MipsABIFlags : Section {
 
   static bool classof(const Section *S) {
     return S->Kind == SectionKind::MipsABIFlags;
   }
 };
 
 struct Object {
   FileHeader Header;
+  std::vector<ProgramHeader> ProgramHeaders;
   std::vector<std::unique_ptr<Section>> Sections;
   // Although in reality the symbols reside in a section, it is a lot
   // cleaner and nicer if we read them from the YAML as a separate
   // top-level key, which automatically ensures that invariants like there
   // being a single SHT_SYMTAB section are upheld.
   LocalGlobalWeakSymbols Symbols;
 };
 
 } // end namespace ELFYAML
 } // end namespace llvm
 
+LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::ELFYAML::ProgramHeader)
 LLVM_YAML_IS_SEQUENCE_VECTOR(std::unique_ptr<llvm::ELFYAML::Section>)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::ELFYAML::Symbol)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::ELFYAML::Relocation)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::ELFYAML::SectionOrType)
+LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::ELFYAML::SectionName)
 
 namespace llvm {
 namespace yaml {
 
 template <>
 struct ScalarEnumerationTraits<ELFYAML::ELF_ET> {
   static void enumeration(IO &IO, ELFYAML::ELF_ET &Value);
 };
 
+template <> struct ScalarEnumerationTraits<ELFYAML::ELF_PT> {
+  static void enumeration(IO &IO, ELFYAML::ELF_PT &Value);
+};
+
 template <>
 struct ScalarEnumerationTraits<ELFYAML::ELF_EM> {
   static void enumeration(IO &IO, ELFYAML::ELF_EM &Value);
 };
 
 template <>
 struct ScalarEnumerationTraits<ELFYAML::ELF_ELFCLASS> {
   static void enumeration(IO &IO, ELFYAML::ELF_ELFCLASS &Value);
struct ScalarEnumerationTraits<ELFYAML::ELF_ELFOSABI> {
   static void enumeration(IO &IO, ELFYAML::ELF_ELFOSABI &Value);
 };
 
 template <>
 struct ScalarBitSetTraits<ELFYAML::ELF_EF> {
   static void bitset(IO &IO, ELFYAML::ELF_EF &Value);
 };
 
+template <> struct ScalarBitSetTraits<ELFYAML::ELF_PF> {
+  static void bitset(IO &IO, ELFYAML::ELF_PF &Value);
+};
+
 template <>
 struct ScalarEnumerationTraits<ELFYAML::ELF_SHT> {
   static void enumeration(IO &IO, ELFYAML::ELF_SHT &Value);
 };
 
 template <>
 struct ScalarBitSetTraits<ELFYAML::ELF_SHF> {
   static void bitset(IO &IO, ELFYAML::ELF_SHF &Value);
struct ScalarBitSetTraits<ELFYAML::MIPS_AFL_FLAGS1> {
   static void bitset(IO &IO, ELFYAML::MIPS_AFL_FLAGS1 &Value);
 };
 
 template <>
 struct MappingTraits<ELFYAML::FileHeader> {
   static void mapping(IO &IO, ELFYAML::FileHeader &FileHdr);
 };
 
+template <> struct MappingTraits<ELFYAML::ProgramHeader> {
+  static void mapping(IO &IO, ELFYAML::ProgramHeader &FileHdr);
+};
+
 template <>
 struct MappingTraits<ELFYAML::Symbol> {
   static void mapping(IO &IO, ELFYAML::Symbol &Symbol);
 };
 
 template <>
 struct MappingTraits<ELFYAML::LocalGlobalWeakSymbols> {
   static void mapping(IO &IO, ELFYAML::LocalGlobalWeakSymbols &Symbols);

 struct MappingTraits<ELFYAML::Object> {
   static void mapping(IO &IO, ELFYAML::Object &Object);
 };
 
 template <> struct MappingTraits<ELFYAML::SectionOrType> {
   static void mapping(IO &IO, ELFYAML::SectionOrType &sectionOrType);
 };
 
+template <> struct MappingTraits<ELFYAML::SectionName> {
+  static void mapping(IO &IO, ELFYAML::SectionName &sectionName);
+};
+
 } // end namespace yaml
 } // end namespace llvm
 
 #endif // LLVM_OBJECTYAML_ELFYAML_H

llvm/trunk/lib/ObjectYAML/ELFYAML.cpp

#define ECase(X) IO.enumCase(Value, #X, ELF::X)
   ECase(ET_REL);
   ECase(ET_EXEC);
   ECase(ET_DYN);
   ECase(ET_CORE);
 #undef ECase
   IO.enumFallback<Hex16>(Value);
 }
 
+void ScalarEnumerationTraits<ELFYAML::ELF_PT>::enumeration(
+    IO &IO, ELFYAML::ELF_PT &Value) {
+#define ECase(X) IO.enumCase(Value, #X, ELF::X)
+  ECase(PT_NULL);
+  ECase(PT_LOAD);
+  ECase(PT_DYNAMIC);
+  ECase(PT_INTERP);
+  ECase(PT_NOTE);
+  ECase(PT_SHLIB);
+  ECase(PT_PHDR);
+  ECase(PT_TLS);
+#undef ECase
+  IO.enumFallback<Hex32>(Value);
+}
+
 void ScalarEnumerationTraits<ELFYAML::ELF_EM>::enumeration(
     IO &IO, ELFYAML::ELF_EM &Value) {
 #define ECase(X) IO.enumCase(Value, #X, ELF::X)
   ECase(EM_NONE);
   ECase(EM_M32);
   ECase(EM_SPARC);
   ECase(EM_386);
   ECase(EM_68K);
case ELF::EM_MIPS:
     break;
   default:
     // Nothing to do.
     break;
   }
 #undef ECase
 }
 
+void ScalarBitSetTraits<ELFYAML::ELF_PF>::bitset(IO &IO,
+                                                 ELFYAML::ELF_PF &Value) {
+#define BCase(X) IO.bitSetCase(Value, #X, ELF::X)
+  BCase(PF_X);
+  BCase(PF_W);
+  BCase(PF_R);
+}
+
 void ScalarBitSetTraits<ELFYAML::ELF_SHF>::bitset(IO &IO,
                                                   ELFYAML::ELF_SHF &Value) {
   const auto *Object = static_cast<ELFYAML::Object *>(IO.getContext());
 #define BCase(X) IO.bitSetCase(Value, #X, ELF::X)
   BCase(SHF_WRITE);
   BCase(SHF_ALLOC);
   BCase(SHF_EXCLUDE);
   BCase(SHF_EXECINSTR);
void MappingTraits<ELFYAML::FileHeader>::mapping(IO &IO,
   IO.mapRequired("Data", FileHdr.Data);
   IO.mapOptional("OSABI", FileHdr.OSABI, ELFYAML::ELF_ELFOSABI(0));
   IO.mapRequired("Type", FileHdr.Type);
   IO.mapRequired("Machine", FileHdr.Machine);
   IO.mapOptional("Flags", FileHdr.Flags, ELFYAML::ELF_EF(0));
   IO.mapOptional("Entry", FileHdr.Entry, Hex64(0));
 }
 
+void MappingTraits<ELFYAML::ProgramHeader>::mapping(
+    IO &IO, ELFYAML::ProgramHeader &Phdr) {
+  IO.mapRequired("Type", Phdr.Type);
+  IO.mapOptional("Flags", Phdr.Flags, ELFYAML::ELF_PF(0));
+  IO.mapOptional("Sections", Phdr.Sections);
+  IO.mapOptional("VAddr", Phdr.VAddr, Hex64(0));
+  IO.mapOptional("PAddr", Phdr.PAddr, Hex64(0));
+}
+
 namespace {
 
 struct NormalizedOther {
   NormalizedOther(IO &)
       : Visibility(ELFYAML::ELF_STV(0)), Other(ELFYAML::ELF_STO(0)) {}
   NormalizedOther(IO &, uint8_t Original)
       : Visibility(Original & 0x3), Other(Original & ~0x3) {}
 
static void groupSectionMapping(IO &IO, ELFYAML::Group &group) {
   IO.mapRequired("Members", group.Members);
 }
 
 void MappingTraits<ELFYAML::SectionOrType>::mapping(
     IO &IO, ELFYAML::SectionOrType &sectionOrType) {
   IO.mapRequired("SectionOrType", sectionOrType.sectionNameOrType);
 }
 
+void MappingTraits<ELFYAML::SectionName>::mapping(
+    IO &IO, ELFYAML::SectionName &sectionName) {
+  IO.mapRequired("Section", sectionName.Section);
+}
+
 static void sectionMapping(IO &IO, ELFYAML::MipsABIFlags &Section) {
   commonSectionMapping(IO, Section);
   IO.mapOptional("Version", Section.Version, Hex16(0));
   IO.mapRequired("ISA", Section.ISALevel);
   IO.mapOptional("ISARevision", Section.ISARevision, Hex8(0));
   IO.mapOptional("ISAExtension", Section.ISAExtension,
                  ELFYAML::MIPS_AFL_EXT(Mips::AFL_EXT_NONE));
   IO.mapOptional("ASEs", Section.ASEs, ELFYAML::MIPS_AFL_ASE(0));
void MappingTraits<ELFYAML::Relocation>::mapping(IO &IO,
   IO.mapOptional("Addend", Rel.Addend, (int64_t)0);
 }
 
 void MappingTraits<ELFYAML::Object>::mapping(IO &IO, ELFYAML::Object &Object) {
   assert(!IO.getContext() && "The IO context is initialized already");
   IO.setContext(&Object);
   IO.mapTag("!ELF", true);
   IO.mapRequired("FileHeader", Object.Header);
+  IO.mapOptional("ProgramHeaders", Object.ProgramHeaders);
   IO.mapOptional("Sections", Object.Sections);
   IO.mapOptional("Symbols", Object.Symbols);
   IO.setContext(nullptr);
 }
 
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, MIPS_AFL_REG)
 LLVM_YAML_STRONG_TYPEDEF(uint8_t, MIPS_ABI_FP)
 LLVM_YAML_STRONG_TYPEDEF(uint32_t, MIPS_AFL_EXT)
 LLVM_YAML_STRONG_TYPEDEF(uint32_t, MIPS_AFL_ASE)
 LLVM_YAML_STRONG_TYPEDEF(uint32_t, MIPS_AFL_FLAGS1)
 
 } // end namespace yaml
 
 } // end namespace llvm

llvm/trunk/test/tools/yaml2obj/program-header-nobits.yaml

+# RUN: yaml2obj %s -o %t
+# RUN: llvm-readobj -program-headers %t | FileCheck %s
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+  Machine:         EM_X86_64
+Sections:
+  - Name:            .data
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC ]
+    Content:         "00000000"
+  - Name:            .after
+    Type:            SHT_NOBITS
+    Flags:           [ SHF_ALLOC ]
+    Size:            64
+ProgramHeaders:
+  - Type: PT_LOAD
+    Flags: [ PF_R ]
+    Sections:
+      - Section: .data
+      - Section: .after
+
+#CHECK:     ProgramHeaders [
+#CHECK-NEXT:  ProgramHeader {
+#CHECK-NEXT:    Type: PT_LOAD
+#CHECK-NEXT:    Offset:
+#CHECK-NEXT:    VirtualAddress:
+#CHECK-NEXT:    PhysicalAddress:
+#CHECK-NEXT:    FileSize: 4
+#CHECK-NEXT:    MemSize: 68
+#CHECK-NEXT:    Flags [
+#CHECK-NEXT:      PF_R
+#CHECK-NEXT:    ]
+#CHECK-NEXT:    Alignment:
+#CHECK-NEXT:  }
+#CHECK-NEXT:]

llvm/trunk/test/tools/yaml2obj/program-header.yaml

+# RUN: yaml2obj %s -o %t
+# RUN: llvm-readobj -program-headers %t | FileCheck %s
+
+!ELF
+FileHeader:
+  Class:           ELFCLASS64
+  Data:            ELFDATA2LSB
+  Type:            ET_EXEC
+  Machine:         EM_X86_64
+Sections:
+  - Name:            .text
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    AddressAlign:    0x0000000000001000
+    Content:         "00000000"
+  - Name:            .init
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_EXECINSTR ]
+    Content:         "00000000"
+    AddressAlign:    0x0000000000000010
+  - Name:            .data
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC ]
+    Content:         "00000000"
+    AddressAlign:    0x0000000000001000
+ProgramHeaders:
+  - Type: PT_LOAD
+    Flags: [ PF_X, PF_R ]
+    VAddr: 0xAAAA1000
+    PAddr: 0xFFFF1000
+    Sections:
+      - Section: .text
+      - Section: .init
+  - Type: PT_LOAD
+    Flags: [ PF_R ]
+    VAddr: 0xAAAA2000
+    PAddr: 0xFFFF2000
+    Sections:
+      - Section: .data
+
+#CHECK:     ProgramHeaders [
+#CHECK-NEXT:   ProgramHeader {
+#CHECK-NEXT:    Type: PT_LOAD
+#CHECK-NEXT:    Offset: 0x1000
+#CHECK-NEXT:    VirtualAddress: 0xAAAA1000
+#CHECK-NEXT:    PhysicalAddress: 0xFFFF1000
+#CHECK-NEXT:    FileSize: 20
+#CHECK-NEXT:    MemSize: 20
+#CHECK-NEXT:    Flags [
+#CHECK-NEXT:      PF_R
+#CHECK-NEXT:      PF_X
+#CHECK-NEXT:    ]
+#CHECK-NEXT:    Alignment: 4096
+#CHECK-NEXT:  }
+#CHECK-NEXT:  ProgramHeader {
+#CHECK-NEXT:    Type: PT_LOAD
+#CHECK-NEXT:    Offset: 0x2000
+#CHECK-NEXT:    VirtualAddress: 0xAAAA2000
+#CHECK-NEXT:    PhysicalAddress: 0xFFFF2000
+#CHECK-NEXT:    FileSize: 4
+#CHECK-NEXT:    MemSize: 4
+#CHECK-NEXT:    Flags [
+#CHECK-NEXT:      PF_R
+#CHECK-NEXT:    ]
+#CHECK-NEXT:    Alignment: 4096
+#CHECK-NEXT:  }
+#CHECK-NEXT:]

llvm/trunk/tools/yaml2obj/yaml2elf.cpp

 
 namespace {
 /// \brief "Single point of truth" for the ELF file construction.
 /// TODO: This class still has a ways to go before it is truly a "single
 /// point of truth".
 template <class ELFT>
 class ELFState {
   typedef typename object::ELFFile<ELFT>::Elf_Ehdr Elf_Ehdr;
+  typedef typename object::ELFFile<ELFT>::Elf_Phdr Elf_Phdr;
   typedef typename object::ELFFile<ELFT>::Elf_Shdr Elf_Shdr;
   typedef typename object::ELFFile<ELFT>::Elf_Sym Elf_Sym;
   typedef typename object::ELFFile<ELFT>::Elf_Rel Elf_Rel;
   typedef typename object::ELFFile<ELFT>::Elf_Rela Elf_Rela;
 
   /// \brief The future ".strtab" section.
   StringTableBuilder DotStrtab{StringTableBuilder::ELF};
 
   /// \brief The future ".shstrtab" section.
   StringTableBuilder DotShStrtab{StringTableBuilder::ELF};
 
   NameToIdxMap SN2I;
   NameToIdxMap SymN2I;
   const ELFYAML::Object &Doc;
 
   bool buildSectionIndex();
   bool buildSymbolIndex(std::size_t &StartIndex,
                         const std::vector<ELFYAML::Symbol> &Symbols);
   void initELFHeader(Elf_Ehdr &Header);
+  void initProgramHeaders(std::vector<Elf_Phdr> &PHeaders);
   bool initSectionHeaders(std::vector<Elf_Shdr> &SHeaders,
                           ContiguousBlobAccumulator &CBA);
   void initSymtabSectionHeader(Elf_Shdr &SHeader,
                                ContiguousBlobAccumulator &CBA);
   void initStrtabSectionHeader(Elf_Shdr &SHeader, StringRef Name,
                                StringTableBuilder &STB,
                                ContiguousBlobAccumulator &CBA);
+  void setProgramHeaderLayout(std::vector<Elf_Phdr> &PHeaders,
+                              std::vector<Elf_Shdr> &SHeaders);
   void addSymbols(const std::vector<ELFYAML::Symbol> &Symbols,
                   std::vector<Elf_Sym> &Syms, unsigned SymbolBinding);
   void writeSectionContent(Elf_Shdr &SHeader,
                            const ELFYAML::RawContentSection &Section,
                            ContiguousBlobAccumulator &CBA);
   bool writeSectionContent(Elf_Shdr &SHeader,
                            const ELFYAML::RelocationSection &Section,
                            ContiguousBlobAccumulator &CBA);
void ELFState<ELFT>::initELFHeader(Elf_Ehdr &Header) {
   Header.e_ident[EI_DATA] = IsLittleEndian ? ELFDATA2LSB : ELFDATA2MSB;
   Header.e_ident[EI_VERSION] = EV_CURRENT;
   Header.e_ident[EI_OSABI] = Doc.Header.OSABI;
   Header.e_ident[EI_ABIVERSION] = 0;
   Header.e_type = Doc.Header.Type;
   Header.e_machine = Doc.Header.Machine;
   Header.e_version = EV_CURRENT;
   Header.e_entry = Doc.Header.Entry;
+  Header.e_phoff = sizeof(Header);
   Header.e_flags = Doc.Header.Flags;
   Header.e_ehsize = sizeof(Elf_Ehdr);
+  Header.e_phentsize = sizeof(Elf_Phdr);
+  Header.e_phnum = Doc.ProgramHeaders.size();
   Header.e_shentsize = sizeof(Elf_Shdr);
-  // Immediately following the ELF header.
-  Header.e_shoff = sizeof(Header);
+  // Immediately following the ELF header and program headers.
+  Header.e_shoff =
+      sizeof(Header) + sizeof(Elf_Phdr) * Doc.ProgramHeaders.size();
   Header.e_shnum = getSectionCount();
   Header.e_shstrndx = getDotShStrTabSecNo();
 }
 
 template <class ELFT>
+void ELFState<ELFT>::initProgramHeaders(std::vector<Elf_Phdr> &PHeaders) {
+  for (const auto &YamlPhdr : Doc.ProgramHeaders) {
+    Elf_Phdr Phdr;
+    Phdr.p_type = YamlPhdr.Type;
+    Phdr.p_flags = YamlPhdr.Flags;
+    Phdr.p_vaddr = YamlPhdr.VAddr;
+    Phdr.p_paddr = YamlPhdr.PAddr;
+    PHeaders.push_back(Phdr);
+  }
+}
+
+template <class ELFT>
 bool ELFState<ELFT>::initSectionHeaders(std::vector<Elf_Shdr> &SHeaders,
                                         ContiguousBlobAccumulator &CBA) {
   // Ensure SHN_UNDEF entry is present. An all-zero section header is a
   // valid SHN_UNDEF entry since SHT_NULL == 0.
   Elf_Shdr SHeader;
   zero(SHeader);
   SHeaders.push_back(SHeader);
 
void ELFState<ELFT>::initStrtabSectionHeader(Elf_Shdr &SHeader, StringRef Name,
   SHeader.sh_name = DotShStrtab.getOffset(Name);
   SHeader.sh_type = ELF::SHT_STRTAB;
   STB.write(CBA.getOSAndAlignedOffset(SHeader.sh_offset, SHeader.sh_addralign));
   SHeader.sh_size = STB.getSize();
   SHeader.sh_addralign = 1;
 }
 
 template <class ELFT>
+void ELFState<ELFT>::setProgramHeaderLayout(std::vector<Elf_Phdr> &PHeaders,
+                                            std::vector<Elf_Shdr> &SHeaders) {
+  uint32_t PhdrIdx = 0;
+  for (auto &YamlPhdr : Doc.ProgramHeaders) {
+    auto &PHeader = PHeaders[PhdrIdx++];
+
+    if (YamlPhdr.Sections.size())
+      PHeader.p_offset = UINT32_MAX;
+    else
+      PHeader.p_offset = 0;
+
+    // Find the minimum offset for the program header.
+    for (auto SecName : YamlPhdr.Sections) {
+      uint32_t Index = 0;
+      SN2I.lookup(SecName.Section, Index);
+      const auto &SHeader = SHeaders[Index];
+      PHeader.p_offset = std::min(PHeader.p_offset, SHeader.sh_offset);
+    }
+
+    // Find the maximum offset of the end of a section in order to set p_filesz.
+    PHeader.p_filesz = 0;
+    for (auto SecName : YamlPhdr.Sections) {
+      uint32_t Index = 0;
+      SN2I.lookup(SecName.Section, Index);
+      const auto &SHeader = SHeaders[Index];
+      uint64_t EndOfSection;
+      if (SHeader.sh_type == llvm::ELF::SHT_NOBITS)
+        EndOfSection = SHeader.sh_offset;
+      else
+        EndOfSection = SHeader.sh_offset + SHeader.sh_size;
+      uint64_t EndOfSegment = PHeader.p_offset + PHeader.p_filesz;
+      EndOfSegment = std::max(EndOfSegment, EndOfSection);
+      PHeader.p_filesz = EndOfSegment - PHeader.p_offset;
+    }
+
+    // Find the memory size by adding the size of sections at the end of the
+    // segment. These should be empty (size of zero) and NOBITS sections.
+    PHeader.p_memsz = PHeader.p_filesz;
+    for (auto SecName : YamlPhdr.Sections) {
+      uint32_t Index = 0;
+      SN2I.lookup(SecName.Section, Index);
+      const auto &SHeader = SHeaders[Index];
+      if (SHeader.sh_offset == PHeader.p_offset + PHeader.p_filesz)
+        PHeader.p_memsz += SHeader.sh_size;
+    }
+
+    // Set the alignment of the segment to be the same as the maximum alignment
+    // of the the sections with the same offset so that by default the segment
+    // has a valid and sensible alignment.
+    PHeader.p_align = 1;
+    for (auto SecName : YamlPhdr.Sections) {
+      uint32_t Index = 0;
+      SN2I.lookup(SecName.Section, Index);
+      const auto &SHeader = SHeaders[Index];
+      if (SHeader.sh_offset == PHeader.p_offset)
+        PHeader.p_align = std::max(PHeader.p_align, SHeader.sh_addralign);
+    }
+  }
+}
+
+template <class ELFT>
 void ELFState<ELFT>::addSymbols(const std::vector<ELFYAML::Symbol> &Symbols,
                                 std::vector<Elf_Sym> &Syms,
                                 unsigned SymbolBinding) {
   for (const auto &Sym : Symbols) {
     Elf_Sym Symbol;
     zero(Symbol);
     if (!Sym.Name.empty())
       Symbol.st_name = DotStrtab.getOffset(Sym.Name);
if (!State.buildSymbolIndex(StartSymIndex, Doc.Symbols.Local) ||
       !State.buildSymbolIndex(StartSymIndex, Doc.Symbols.Global) ||
       !State.buildSymbolIndex(StartSymIndex, Doc.Symbols.Weak))
     return 1;
 
   Elf_Ehdr Header;
   State.initELFHeader(Header);
 
   // TODO: Flesh out section header support.
-  // TODO: Program headers.
+
+  std::vector<Elf_Phdr> PHeaders;
+  State.initProgramHeaders(PHeaders);
 
   // XXX: This offset is tightly coupled with the order that we write
   // things to `OS`.
-  const size_t SectionContentBeginOffset =
-      Header.e_ehsize + Header.e_shentsize * Header.e_shnum;
+  const size_t SectionContentBeginOffset = Header.e_ehsize +
+                                           Header.e_phentsize * Header.e_phnum +
+                                           Header.e_shentsize * Header.e_shnum;
   ContiguousBlobAccumulator CBA(SectionContentBeginOffset);
 
   // Doc might not contain .symtab, .strtab and .shstrtab sections,
   // but we will emit them, so make sure to add them to ShStrTabSHeader.
   State.DotShStrtab.add(".symtab");
   State.DotShStrtab.add(".strtab");
   State.DotShStrtab.add(".shstrtab");
 
int ELFState<ELFT>::writeELF(raw_ostream &OS, const ELFYAML::Object &Doc) {
   SHeaders.push_back(DotStrTabSHeader);
 
   // .shstrtab string table header.
   Elf_Shdr ShStrTabSHeader;
   State.initStrtabSectionHeader(ShStrTabSHeader, ".shstrtab", State.DotShStrtab,
                                 CBA);
   SHeaders.push_back(ShStrTabSHeader);
 
+  // Now we can decide segment offsets
+  State.setProgramHeaderLayout(PHeaders, SHeaders);
+
   OS.write((const char *)&Header, sizeof(Header));
+  writeArrayData(OS, makeArrayRef(PHeaders));
   writeArrayData(OS, makeArrayRef(SHeaders));
   CBA.writeBlobToStream(OS);
   return 0;
 }
 
 static bool is64Bit(const ELFYAML::Object &Doc) {
   return Doc.Header.Class == ELFYAML::ELF_ELFCLASS(ELF::ELFCLASS64);
 }