[llvm-objcopy] Introduce 'ihex-flat' output format.

[simon_tatham]

Summary

Currently, if you use llvm-objcopy to translate an ELF image into
'ihex' format, the input ELF file's entry point address will be
converted into x86-16 segment:offset style if it's less than 1MB, and
the same happens to addresses of data records. The separate record
types for a flat 32-bit address space are not used unless an address
is too big for the segment:offset style.

This is awkward for users consuming the file, who may find they need
to understand both formats of start address and both formats of data
address record. And it doesn't have any relevance to any of LLVM's
target architectures, since we support x86-32 but not x86-16. So users
wanting to write the simplest possible ihex consumer would prefer the
producer to emit the 32-bit record types unconditionally.

I haven't changed the existing format in this commit, on the
assumption that exactly matching the behavior of GNU objdump is a
useful property. Instead, I've added a new output-only format name
"ihex-flat" alongside the existing "ihex", and made the format changes
conditional on that.

[simon_tatham]

Supplementary discussion:

This is a conservative patch which changes no existing behavior. But I'd be happy to go further if people want to, by making ihex-flat the default, and relegating the old behavior to a different name, or perhaps even removing it entirely.

Rationale: this segment:offset representation of addresses was only ever important to 16-bit x86 as far as I know, and in all other situations, its sole effect is to complicate the file format unnecessarily for everyone else. And since LLVM doesn't even support x86-16 as a target architecture, 'everyone else' is quite likely to be all users!

Also, even if someone is targeting x86-16, I can't see how a hex file written in this way would be useful. Surely you would need the ability to control the precise CS:IP representation of the entry point address, so you could set it to match the expectations of the code that will be executing there? And that will not in all cases match the fixed policy here of making the segment address a multiple of 0x1000 and putting all 16 low bits of the linear address into the offset.

But the current code was put in on purpose, and as far as I can tell from the comments in D60270, the purpose was to match GNU objcopy, so for the moment I'm presuming that's useful in its own right.

[jhenderson]

I'll look at this in the coming days, but I'd appreciate @evgeny777's comments, since they were the ones who originally implemented this, so might have some specific rationale beyond matching GNU. One thought I did have though is that although LLVM as a whole may not target x86-16, there's no particularly reason why its binary manipulation tools like llvm-objcopy shouldn't work with them, if there's a use-case.

[simon_tatham]

Absolutely, I agree. Off the top of my head, the most obvious continuing use case for x86-16 is first-stage bootloaders that the PC BIOS runs in real mode. I've no idea what tools people typically use for those these days, but I wouldn't have a hard time at all believing that it might turn out to be a hodgepodge of bits and pieces from all over the place.

And I suppose that in that use case, the problem I mentioned with the entry point representation is moot anyway, because you don't get a choice about the entry point of an MBR boot sector – it's fixed at 0000:7C00 (or maybe 07C0:0000, I shamefully forget which). So you'd never need to retrieve it from the ihex file to pass on to something else.

[jhenderson]

llvm-objcopy CommandGuide docs (llvm/docs/CommandGuide) will need updating.

@MaskRay, any thoughts on this? I think it's a reasonable change, but am unsure whether it should be the "default" ihex output, or under the new format name. If the former, I'd be tempted to keep the old format around for the clients who need it, under a different name. The change should then definitely be mentioned in the release notes too.

llvm/lib/ObjCopy/ELF/ELFObject.h

275

I suggest MayUseSegmentOffset since, if I understand it correctly, you may need to use the other format even if this is true.

llvm/tools/llvm-objcopy/ObjcopyOptions.cpp

644

To keep the code simpler, I think we should omit this ihex-flat as an input format: as far as I'm aware, there's no need to support it. Users can just use ihex for the input option. In my opinion, the symmetry isn't important: note that for binary format the input and output formats are essentially unrelated, so the "symmetry" there is, if anything, confusing, but a necessary evil of being compatible with GNU.

[simon_tatham]

Sorry to have been so long getting back to this! It was relegated to my back burner for a while, but I've now found time to address the review comments so far.

[jhenderson]

Sorry for not coming to this yet: this is low priority for me, and I have a pile of other stuff I need to finish in the next couple of weeks before taking a 6 week vacation, so I doubt I'll get around to this patch until some time late January or February. Hopefully somebody else might be able to take over reviewing.

llvm/docs/CommandGuide/llvm-objcopy.rst

 -----------------
 
 The following values are currently supported by :program:`llvm-objcopy` for the
 :option:`--input-target`, :option:`--output-target`, and :option:`--target`
 options. For GNU :program:`objcopy` compatibility, the values are all bfdnames.
 
 - `binary`
 - `ihex`
+- `ihex-flat`
 - `elf32-i386`
 - `elf32-x86-64`
 - `elf64-x86-64`
 - `elf32-iamcu`
 - `elf32-littlearm`
 - `elf64-aarch64`
 - `elf64-littleaarch64`
 - `elf32-littleriscv`

 - `elf64-tradbigmips`
 - `elf64-tradlittlemips`
 - `elf32-sparc`
 - `elf32-sparcel`
 
 Additionally, all targets except `binary` and `ihex` can have `-freebsd` as a
 suffix.
 
+`ihex-flat` is permitted as an output format only. It's very similar
+to `ihex`, except that it represents all addresses as being in a flat
+32-bit address space, and never uses the x86-16 style segment:offset
+representation. This makes the output hex files slightly easier to
+consume, because fewer different record types are involved.
+
 BINARY INPUT AND OUTPUT
 -----------------------
 
 If `binary` is used as the value for :option:`--input-target`, the input file
 will be embedded as a data section in an ELF relocatable object, with symbols
 ``_binary_<file_name>_start``, ``_binary_<file_name>_end``, and
 ``_binary_<file_name>_size`` representing the start, end and size of the data,
 where ``<file_name>`` is the path of the input file as specified on the command

llvm/include/llvm/ObjCopy/CommonConfig.h

 namespace llvm {
 namespace objcopy {
 
 enum class FileFormat {
   Unspecified,
   ELF,
   Binary,
   IHex,
+  IHexFlat,
 };
 
 // This type keeps track of the machine info for various architectures. This
 // lets us map architecture names to ELF types and the e_machine value of the
 // ELF file.
 struct MachineInfo {
   MachineInfo(uint16_t EM, uint8_t ABI, bool Is64, bool IsLittle)
       : EMachine(EM), OSABI(ABI), Is64Bit(Is64), IsLittleEndian(IsLittle) {}

llvm/lib/ObjCopy/ELF/ELFObjcopy.cpp

 
 static std::unique_ptr<Writer> createWriter(const CommonConfig &Config,
                                             Object &Obj, raw_ostream &Out,
                                             ElfType OutputElfType) {
   switch (Config.OutputFormat) {
   case FileFormat::Binary:
     return std::make_unique<BinaryWriter>(Obj, Out);
   case FileFormat::IHex:
-    return std::make_unique<IHexWriter>(Obj, Out);
+    return std::make_unique<IHexWriter>(Obj, Out, true);
+  case FileFormat::IHexFlat:
+    return std::make_unique<IHexWriter>(Obj, Out, false);
   default:
     return createELFWriter(Config, Obj, Out, OutputElfType);
   }
 }
 
 template <class... Ts>
 static Error makeStringError(std::error_code EC, const Twine &Msg,
                              Ts &&...Args) {

llvm/lib/ObjCopy/ELF/ELFObject.h

 // but doesn't actually write records. It is used for output buffer size
 // calculation in IHexWriter::finalize.
 class IHexSectionWriterBase : public BinarySectionWriter {
   // 20-bit segment address
   uint32_t SegmentAddr = 0;
   // Extended linear address
   uint32_t BaseAddr = 0;
 
+  // Whether we're emitting segment:offset format at all
+  bool MayUseSegmentOffset;
+
   // Write segment address corresponding to 'Addr'
   uint64_t writeSegmentAddr(uint64_t Addr);
   // Write extended linear (base) address corresponding to 'Addr'
   uint64_t writeBaseAddr(uint64_t Addr);
 
 protected:
   // Offset in the output buffer
   uint64_t Offset = 0;
 
   void writeSection(const SectionBase *Sec, ArrayRef<uint8_t> Data);
   virtual void writeData(uint8_t Type, uint16_t Addr, ArrayRef<uint8_t> Data);
 
 public:
-  explicit IHexSectionWriterBase(WritableMemoryBuffer &Buf)
-      : BinarySectionWriter(Buf) {}
+  IHexSectionWriterBase(WritableMemoryBuffer &Buf, bool MayUseSegmentOffset)
+      : BinarySectionWriter(Buf), MayUseSegmentOffset(MayUseSegmentOffset) {}
 
   uint64_t getBufferOffset() const { return Offset; }
   Error visit(const Section &Sec) final;
   Error visit(const OwnedDataSection &Sec) final;
   Error visit(const StringTableSection &Sec) override;
   Error visit(const DynamicRelocationSection &Sec) final;
   using BinarySectionWriter::visit;
 };
 
 // Real IHEX section writer
 class IHexSectionWriter : public IHexSectionWriterBase {
 public:
-  IHexSectionWriter(WritableMemoryBuffer &Buf) : IHexSectionWriterBase(Buf) {}
+  IHexSectionWriter(WritableMemoryBuffer &Buf, bool MayUseSegmentOffset)
+      : IHexSectionWriterBase(Buf, MayUseSegmentOffset) {}
 
   void writeData(uint8_t Type, uint16_t Addr, ArrayRef<uint8_t> Data) override;
   Error visit(const StringTableSection &Sec) override;
 };
 
 class Writer {
 protected:
   Object &Obj;

 class IHexWriter : public Writer {
   struct SectionCompare {
     bool operator()(const SectionBase *Lhs, const SectionBase *Rhs) const;
   };
 
   std::set<const SectionBase *, SectionCompare> Sections;
   size_t TotalSize = 0;
 
+  bool MayUseSegmentOffset;
+
   Error checkSection(const SectionBase &Sec);
   uint64_t writeEntryPointRecord(uint8_t *Buf);
   uint64_t writeEndOfFileRecord(uint8_t *Buf);
 
 public:
   ~IHexWriter() {}
   Error finalize() override;
   Error write() override;
-  IHexWriter(Object &Obj, raw_ostream &Out) : Writer(Obj, Out) {}
+  IHexWriter(Object &Obj, raw_ostream &Out, bool MayUseSegmentOffset)
+      : Writer(Obj, Out), MayUseSegmentOffset(MayUseSegmentOffset) {}
 };
 
 class SectionBase {
 public:
   std::string Name;
   Segment *ParentSegment = nullptr;
   uint64_t HeaderOffset = 0;
   uint32_t Index = 0;

llvm/lib/ObjCopy/ELF/ELFObject.cpp

 void IHexSectionWriterBase::writeSection(const SectionBase *Sec,
                                          ArrayRef<uint8_t> Data) {
   assert(Data.size() == Sec->Size);
   const uint32_t ChunkSize = 16;
   uint32_t Addr = sectionPhysicalAddr(Sec) & 0xFFFFFFFFU;
   while (!Data.empty()) {
     uint64_t DataSize = std::min<uint64_t>(Data.size(), ChunkSize);
     if (Addr > SegmentAddr + BaseAddr + 0xFFFFU) {
-      if (Addr > 0xFFFFFU) {
+      if (Addr > 0xFFFFFU || !MayUseSegmentOffset) {
         // Write extended address record, zeroing segment address
         // if needed.
         if (SegmentAddr != 0)
           SegmentAddr = writeSegmentAddr(0U);
         BaseAddr = writeBaseAddr(Addr);
       } else {
         // We can still remain 16-bit
         SegmentAddr = writeSegmentAddr(Addr);

 
 uint64_t IHexWriter::writeEntryPointRecord(uint8_t *Buf) {
   IHexLineData HexData;
   uint8_t Data[4] = {};
   // We don't write entry point record if entry is zero.
   if (Obj.Entry == 0)
     return 0;
 
-  if (Obj.Entry <= 0xFFFFFU) {
+  if (MayUseSegmentOffset && Obj.Entry <= 0xFFFFFU) {
+    // Write x86-16 style segment:offset, with the division between the two
+    // chosen arbitrarily so that the low 16 bits all go in the offset, e.g.
+    // 0xABCDE -> 0xA000:0xBCDE.
     Data[0] = ((Obj.Entry & 0xF0000U) >> 12) & 0xFF;
     support::endian::write(&Data[2], static_cast<uint16_t>(Obj.Entry),
                            support::big);
     HexData = IHexRecord::getLine(IHexRecord::StartAddr80x86, 0, Data);
   } else {
+    // Write a 32-bit start address record assuming a flat address space,
+    // either because the address doesn't fit in 20 bits, or because we're in
+    // IHexFlat mode where the user doesn't want any segment:offset
+    // representations of anything anyway.
     support::endian::write(Data, static_cast<uint32_t>(Obj.Entry),
                            support::big);
     HexData = IHexRecord::getLine(IHexRecord::StartAddr, 0, Data);
   }
   memcpy(Buf, HexData.data(), HexData.size());
   return HexData.size();
 }
 
 uint64_t IHexWriter::writeEndOfFileRecord(uint8_t *Buf) {
   IHexLineData HexData = IHexRecord::getLine(IHexRecord::EndOfFile, 0, {});
   memcpy(Buf, HexData.data(), HexData.size());
   return HexData.size();
 }
 
 Error IHexWriter::write() {
-  IHexSectionWriter Writer(*Buf);
+  IHexSectionWriter Writer(*Buf, MayUseSegmentOffset);
   // Write sections.
   for (const SectionBase *Sec : Sections)
     if (Error Err = Sec->accept(Writer))
       return Err;
 
   uint64_t Offset = Writer.getBufferOffset();
   // Write entry point address.
   Offset += writeEntryPointRecord(
for (const SectionBase &Sec : Obj.sections())
     }
 
   std::unique_ptr<WritableMemoryBuffer> EmptyBuffer =
       WritableMemoryBuffer::getNewMemBuffer(0);
   if (!EmptyBuffer)
     return createStringError(errc::not_enough_memory,
                              "failed to allocate memory buffer of 0 bytes");
 
-  IHexSectionWriterBase LengthCalc(*EmptyBuffer);
+  IHexSectionWriterBase LengthCalc(*EmptyBuffer, MayUseSegmentOffset);
   for (const SectionBase *Sec : Sections)
     if (Error Err = Sec->accept(LengthCalc))
       return Err;
 
   // We need space to write section records + StartAddress record
   // (if start adress is not zero) + EndOfFile record.
   TotalSize = LengthCalc.getBufferOffset() +
               (Obj.Entry ? IHexRecord::getLineLength(4) : 0) +

llvm/test/tools/llvm-objcopy/ELF/ihex-writer.test

 # RUN: yaml2obj %p/Inputs/ihex-elf-sections.yaml -o %t
-# RUN: llvm-objcopy -O ihex %t - | FileCheck %s
+# RUN: llvm-objcopy -O ihex %t - | FileCheck %s --check-prefixes=CHECK,80X86
+# RUN: llvm-objcopy -O ihex-flat %t - | FileCheck %s --check-prefixes=CHECK,FLAT
 
 # Check ihex output, when we have segments in ELF file
 # In such case only sections in PT_LOAD segments will
 # be exported and their physical addresses will be used
 # RUN: yaml2obj %p/Inputs/ihex-elf-segments.yaml -o %t-segs
 # RUN: llvm-objcopy -O ihex %t-segs - | FileCheck %s --check-prefix=SEGMENTS
 
 # Check that non-load segments are ignored:
 # RUN: yaml2obj %p/Inputs/ihex-elf-pt-null.yaml -o %t2-segs
 # RUN: llvm-objcopy -O ihex %t2-segs - | FileCheck %s --check-prefix=PT_NULL
 
 # Check that sign-extended 32-bit section addresses are processed
 # correctly
 # RUN: yaml2obj %p/Inputs/ihex-elf-sections2.yaml -o %t-sec2
 # RUN: llvm-objcopy -O ihex --only-section=.text1 %t-sec2 - | FileCheck %s --check-prefix=SIGN_EXTENDED
 
 # Check that section address range overlapping 32 bit range
 # triggers an error
 # RUN: not llvm-objcopy -O ihex --only-section=.text2 %t-sec2 %t-sec2-2.hex 2>&1 | FileCheck %s --check-prefix=BAD-ADDR
 # RUN: not llvm-objcopy -O ihex --only-section=.text3 %t-sec2 %t-sec2-3.hex 2>&1 | FileCheck %s --check-prefix=BAD-ADDR2
 
 # Check that zero length section is not written
 # RUN: llvm-objcopy -O ihex --only-section=.text %t-sec2 - | FileCheck %s --check-prefix=ZERO_SIZE_SEC
 
-# Check 80x86 start address record. It is created for start
-# addresses less than 0x100000
-# RUN: llvm-objcopy -O ihex --set-start=0xFFFF %t - | FileCheck %s --check-prefix=START1
-
-# Check i386 start address record (05). It is created for
-# start addresses which doesn't fit 20 bits
-# RUN: llvm-objcopy -O ihex --set-start=0x100000 %t - | FileCheck %s --check-prefix=START2
+# Check start address records in ihex mode. 80x86 style segment:offset
+# is used for start addresses less than 0x100000, and otherwise the
+# i386 flat style is used.
+# RUN: llvm-objcopy -O ihex --set-start=0xABCDE %t - | FileCheck %s --check-prefix=START-SEG-OFF
+# RUN: llvm-objcopy -O ihex --set-start=0x100000 %t - | FileCheck %s --check-prefix=START-HIGH
+
+# Check start address records in ihex-flat mode, which should use i386
+# style unconditionally.
+# RUN: llvm-objcopy -O ihex-flat --set-start=0xABCDE %t - | FileCheck %s --check-prefix=START-FLAT
+# RUN: llvm-objcopy -O ihex-flat --set-start=0x100000 %t - | FileCheck %s --check-prefix=START-HIGH
 
 # We allow sign extended 32 bit start addresses as well.
-# RUN: llvm-objcopy -O ihex --set-start=0xFFFFFFFF80001000 %t - | FileCheck %s --check-prefix=START3
+# RUN: llvm-objcopy -O ihex --set-start=0xFFFFFFFF80001000 %t - | FileCheck %s --check-prefix=START-SIGN-EXT
 
 # Start address which exceeds 32 bit range triggers an error
 # RUN: not llvm-objcopy -O ihex --set-start=0xF00000000 %t %t6.hex 2>&1 | FileCheck %s --check-prefix=BAD-START
 
 # CHECK:      :10000000000102030405060708090A0B0C0D0E0F78
 # CHECK-NEXT: :05001000101112131491
 # CHECK-NEXT: :08FFF800303132333435363765
-# CHECK-NEXT: :020000021000EC
+# 80X86-NEXT: :020000021000EC
+#  FLAT-NEXT: :020000040001F9
 # CHECK-NEXT: :030000003839404C
 # CHECK-NEXT: :0401000040414243F5
-# CHECK-NEXT: :020000020000FC
+# 80X86-NEXT: :020000020000FC
 # CHECK-NEXT: :020000040010EA
 # CHECK-NEXT: :08FFF800505152535455565765
 # CHECK-NEXT: :020000040011E9
 # CHECK-NEXT: :03000000585960EC
 # CHECK-NEXT: :00000001FF
 
 # SEGMENTS:       :020000040010EA
 # SEGMENTS-NEXT:  :10000000000102030405060708090A0B0C0D0E0F78

 # BAD-ADDR: error: {{.*}}: Section '.text2' address range [0x{{.*}}, 0x{{.*}}] is not 32 bit
 # BAD-ADDR2: error: {{.*}}: Section '.text3' address range [0x{{.*}}, 0x{{.*}}] is not 32 bit
 
 # There shouldn't be 'ExtendedAddr' nor 'Data' records
 # ZERO_SIZE_SEC-NOT:  :02000004
 # ZERO_SIZE_SEC-NOT:  :00FFFF00
 # ZERO_SIZE_SEC:      :00000001FF
 
-# START1: :040000030000FFFFFB
-# START2: :0400000500100000E7
-# START3: :040000058000100067
+# START-SEG-OFF:  :04000003A000BCDEBF
+# START-FLAT:     :04000005000ABCDE53
+# START-HIGH:     :0400000500100000E7
+# START-SIGN-EXT: :040000058000100067
 # BAD-START: error: {{.*}}: Entry point address 0x{{.*}} overflows 32 bits

llvm/tools/llvm-objcopy/ObjcopyOptions.cpp

objcopy::parseObjcopyOptions(ArrayRef<const char *> RawArgsArr,
   }
 
   // FIXME:  Currently, we ignore the target for non-binary/ihex formats
   // explicitly specified by -I option (e.g. -Ielf32-x86-64) and guess the
   // format by llvm::object::createBinary regardless of the option value.
   Config.InputFormat = StringSwitch<FileFormat>(InputFormat)
                            .Case("binary", FileFormat::Binary)
                            .Case("ihex", FileFormat::IHex)
                            .Default(FileFormat::Unspecified);
 
   if (InputArgs.hasArg(OBJCOPY_new_symbol_visibility)) {
     const uint8_t Invalid = 0xff;
     StringRef VisibilityStr =
         InputArgs.getLastArgValue(OBJCOPY_new_symbol_visibility);
 
     ELFConfig.NewSymbolVisibility = StringSwitch<uint8_t>(VisibilityStr)
                                         .Case("default", ELF::STV_DEFAULT)
if (!Version.empty()) {
                                  Minor.str().c_str());
       COFFConfig.MinorSubsystemVersion = Number;
     }
   }
 
   Config.OutputFormat = StringSwitch<FileFormat>(OutputFormat)
                             .Case("binary", FileFormat::Binary)
                             .Case("ihex", FileFormat::IHex)
+                            .Case("ihex-flat", FileFormat::IHexFlat)
                             .Default(FileFormat::Unspecified);
   if (Config.OutputFormat == FileFormat::Unspecified) {
     if (OutputFormat.empty()) {
       Config.OutputFormat = Config.InputFormat;
     } else {
       Expected<TargetInfo> Target =
           getOutputTargetInfoByTargetName(OutputFormat);
       if (!Target)

llvm/tools/llvm-objcopy/llvm-objcopy.cpp

static Error executeObjcopyOnRawBinary(ConfigManager &ConfigMgr,
   const CommonConfig &Config = ConfigMgr.getCommonConfig();
   switch (Config.OutputFormat) {
   case FileFormat::ELF:
   // FIXME: Currently, we call elf::executeObjcopyOnRawBinary even if the
   // output format is binary/ihex or it's not given. This behavior differs from
   // GNU objcopy. See https://bugs.llvm.org/show_bug.cgi?id=42171 for details.
   case FileFormat::Binary:
   case FileFormat::IHex:
+  case FileFormat::IHexFlat:
   case FileFormat::Unspecified:
     Expected<const ELFConfig &> ELFConfig = ConfigMgr.getELFConfig();
     if (!ELFConfig)
       return ELFConfig.takeError();
 
     return elf::executeObjcopyOnRawBinary(Config, *ELFConfig, In, Out);
   }