[ELF] - Linkerscript: implemented BYTE/SHORT/LONG/QUAD commands.

[grimar]

Summary

Previously our scripts handles these commands incorrectly. For example:

SECTIONS  {
  .foo : {
 *(.foo.1)
 BYTE(0x11)
...

We accepted the script above treating BYTE as input section description. These commands are used in the wild though, eg:
https://searchcode.com/codesearch/view/42878166/

The BYTE, SHORT, LONG, and QUAD commands store one, two, four, and eight bytes (respectively). After storing the bytes, the location counter is incremented by the number of bytes
stored.

Patch implements them.

[ruiu]

ELF/LinkerScript.cpp

414

This

424

this and

436

this needs comments.

716

Don't we have a function to find an output section by name?

719–722

This code seems to write the bytes in the host endianess instead of the target endianness.

1510–1515

I don't think we need these enums. Can't this be

int Size = StringSwitch(Tok)
  .Case("BYTE", 1)
  .Case("SHORT", 2)
  ...

?

ELF/LinkerScript.h

47–52

It's time to add comments for each command.

AssignmentKind, // . = expr or <sym> = expr
OutputSectionKind,
InputSectionKind,
AssertKind, // ASSERT(expr)
BytesDataKind, // BYTE(expr), SHORT(expr), LONG(expr) or QUAD(expr)

ELF/OutputSections.cpp

1008

It needs a comment.

// Linker scripts may have BYTE()-family commands with which you
// can write arbitrary bytes to the output. Process them if any.

[grimar]

ELF/LinkerScript.cpp

414

Done.

424

Done.

436

Done.

716

I do not think we have, I created new one, it can be reused in getFiller() in following patch.

719–722

Ah, right. Fixed.

1510–1515

Done.

ELF/LinkerScript.h

47–52

Done. (btw comments looks a bit displaced because are clang-formatted)

ELF/OutputSections.cpp

1008

Done.

[• rafael]

ELF/LinkerScript.h

146

This could just store a StringRef or ArrayRef, no?

We can compute the correct (target endian) representation upfront and simplify the remaining code.

[grimar]

ELF/LinkerScript.h

146

Not sure about StringRef/ArrayRef, that could be std::vector<uint_8t>, computed in theory, but I think I can't do it.

Problem is that readBytesDataCommand() that reads command know nothing about target endianess because is not ELFT templated.

[grimar]

And I mean it is too early to know it as script can go before any other files, so I need to keep the value anyways, and in that case have a vector or something else in addition does not make much sence ?

[• rafael]

ELF/LinkerScript.cpp

713

You can probably use getSectionIndex.

[grimar]

• Addressed review comments.

ELF/LinkerScript.cpp

79

Done.

[• rafael]

LGTM with nits, but wait so see if Rui is OK with it too.

test/ELF/linkerscript/data-commands.s

3

Add a big endian test.

22

Don't check the address.

[ruiu]

ELF/LinkerScript.cpp

78

Add a blank line before this.

Maybe just

// Handle BYTE(), SHORT(), LONG(), or QUAD().

79

Use int instead of unsigned so to remove the cast to unsigned below.

83

This looks odd. You can return early.

if (I == INT_MAX)
  return;

92

I'd factor this switch out as an independent functions so that we can do something like this here.

writeInt<ELFT>(Buf + DataCmd->Offset, DataCmd->Data, DataCmd->Size);

where writeInt is defined as

writeInt(uint8_t *Buf, uint64_t Data, uint64_t Size);

[grimar]

• Addressed review comments.

ELF/LinkerScript.cpp

78

Done.

79

Done.

83

Done.

test/ELF/linkerscript/data-commands.s

3

Done.

22

Done.

[ruiu]

LGTM with a nit.

ELF/LinkerScript.cpp

744–745

This comment doesn't make sense to be written here. Probably I'd just remove.

[grimar]

ELF/LinkerScript.cpp

744–745

Yeah, initially I wrote it because we have comment for filler saying it works with big-endian,
but for this place there is no disagree between ld, gold and spec, so it was probably excessive, I removed it from commit.

ELF/LinkerScript.h

 // Config and ScriptConfig.
 void readLinkerScript(MemoryBufferRef MB);
 
 void readVersionScript(MemoryBufferRef MB);
 
 // This enum is used to implement linker script SECTIONS command.
 // https://sourceware.org/binutils/docs/ld/SECTIONS.html#SECTIONS
 enum SectionsCommandKind {
-  AssignmentKind,
+  AssignmentKind, // . = expr or <sym> = expr
   OutputSectionKind,
   InputSectionKind,
-  AssertKind
+  AssertKind,   // ASSERT(expr)
+  BytesDataKind // BYTE(expr), SHORT(expr), LONG(expr) or QUAD(expr)
 };
 
 struct BaseCommand {
   BaseCommand(int K) : Kind(K) {}
   virtual ~BaseCommand() {}
   int Kind;
 };
 
 struct SymbolAssignment : BaseCommand {

 };
 
 struct AssertCommand : BaseCommand {
   AssertCommand(Expr E) : BaseCommand(AssertKind), Expression(E) {}
   static bool classof(const BaseCommand *C);
   Expr Expression;
 };
 
+struct BytesDataCommand : BaseCommand {
+  BytesDataCommand(uint64_t Data, unsigned Size)
+      : BaseCommand(BytesDataKind), Data(Data), Size(Size) {}
+  static bool classof(const BaseCommand *C);
+  uint64_t Data;
+  unsigned Offset;
+  unsigned Size;
+};
+
 struct PhdrsCommand {
   StringRef Name;
   unsigned Type;
   bool HasFilehdr;
   bool HasPhdrs;
   unsigned Flags;
   Expr LMAExpr;
 };
public:
   void processCommands(OutputSectionFactory<ELFT> &Factory);
   void createSections(OutputSectionFactory<ELFT> &Factory);
   void adjustSectionsBeforeSorting();
 
   std::vector<PhdrEntry<ELFT>> createPhdrs();
   bool ignoreInterpSection();
 
   ArrayRef<uint8_t> getFiller(StringRef Name);
+  void writeDataBytes(StringRef Name, uint8_t *Buf);
   Expr getLma(StringRef Name);
   bool shouldKeep(InputSectionBase<ELFT> *S);
   void assignOffsets(OutputSectionCommand *Cmd);
   void assignAddresses();
   bool hasPhdrsCommands();
   uint64_t getOutputSectionAddress(StringRef Name) override;
   uint64_t getOutputSectionSize(StringRef Name) override;
   uint64_t getOutputSectionAlign(StringRef Name) override;

ELF/LinkerScript.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/StringSaver.h"
 
 using namespace llvm;
 using namespace llvm::ELF;
 using namespace llvm::object;
+using namespace llvm::support::endian;
 using namespace lld;
 using namespace lld::elf;
 
 LinkerScriptBase *elf::ScriptBase;
 ScriptConfiguration *elf::ScriptConfig;
 
 template <class ELFT> static void addRegular(SymbolAssignment *Cmd) {
   Symbol *Sym = Symtab<ELFT>::X->addRegular(Cmd->Name, STB_GLOBAL, STV_DEFAULT);

 }
 // If a symbol was in PROVIDE(), we need to define it only when
 // it is an undefined symbol.
 template <class ELFT> static bool shouldDefine(SymbolAssignment *Cmd) {
   if (Cmd->Name == ".")
     return false;
   if (!Cmd->Provide)
     return true;
   SymbolBody *B = Symtab<ELFT>::X->find(Cmd->Name);
   return B && B->isUndefined();
 }
 
 bool SymbolAssignment::classof(const BaseCommand *C) {
   return C->Kind == AssignmentKind;
 }
 
 bool OutputSectionCommand::classof(const BaseCommand *C) {
   return C->Kind == OutputSectionKind;
 }
 
 bool InputSectionDescription::classof(const BaseCommand *C) {
   return C->Kind == InputSectionKind;
 }
 
 bool AssertCommand::classof(const BaseCommand *C) {
   return C->Kind == AssertKind;
 }
 
+bool BytesDataCommand::classof(const BaseCommand *C) {
+  return C->Kind == BytesDataKind;
+}
+
 template <class ELFT> static bool isDiscarded(InputSectionBase<ELFT> *S) {
   return !S || !S->Live;
 }
 
 template <class ELFT> LinkerScript<ELFT>::LinkerScript() {}
 template <class ELFT> LinkerScript<ELFT>::~LinkerScript() {}
 
 template <class ELFT>
void LinkerScript<ELFT>::switchTo(OutputSectionBase<ELFT> *Sec) {
   flush();
   CurOutSec = Sec;
 
   Dot = alignTo(Dot, CurOutSec->getAlignment());
   CurOutSec->setVA(isTbss(CurOutSec) ? Dot + ThreadBssOffset : Dot);
 }
 
 template <class ELFT> void LinkerScript<ELFT>::process(BaseCommand &Base) {
+  // This handles the assignments to symbol or to a location counter (.)
   if (auto *AssignCmd = dyn_cast<SymbolAssignment>(&Base)) {
     if (AssignCmd->Name == ".") {
       // Update to location counter means update to section size.
       Dot = AssignCmd->Expression(Dot);
       CurOutSec->setSize(Dot - CurOutSec->getVA());
       return;
     }
     assignSectionSymbol<ELFT>(AssignCmd, CurOutSec, Dot - CurOutSec->getVA());
     return;
   }
+  // It handles the bytes data command, which places some byte value into output
+  // section and moves the location counter in according to value data size.
+  if (auto *DataCmd = dyn_cast<BytesDataCommand>(&Base)) {
+    DataCmd->Offset = Dot - CurOutSec->getVA();
+    Dot += DataCmd->Size;
+    CurOutSec->setSize(Dot - CurOutSec->getVA());
+    return;
+  }
+
+  // It handles single input section description command,
+  // calculates and assigns the offsets for each section and also
+  // updates the output section size.
   auto &ICmd = cast<InputSectionDescription>(Base);
   for (InputSectionData *ID : ICmd.Sections) {
     auto *IB = static_cast<InputSectionBase<ELFT> *>(ID);
     switchTo(IB->OutSec);
     if (auto *I = dyn_cast<InputSection<ELFT>>(IB))
       output(I);
     else if (AlreadyOutputOS.insert(CurOutSec).second)
       Dot += CurOutSec->getSize();
   }

 ArrayRef<uint8_t> LinkerScript<ELFT>::getFiller(StringRef Name) {
   for (const std::unique_ptr<BaseCommand> &Base : Opt.Commands)
     if (auto *Cmd = dyn_cast<OutputSectionCommand>(Base.get()))
       if (Cmd->Name == Name)
         return Cmd->Filler;
   return {};
 }
 
+template <class ELFT>
+static OutputSectionCommand *findOutputSection(StringRef Name) {
+  for (const std::unique_ptr<BaseCommand> &Base : ScriptConfig->Commands)
+    if (auto *Cmd = dyn_cast<OutputSectionCommand>(Base.get()))
+      if (Cmd->Name == Name)
+        return Cmd;
+  return nullptr;
+}
+
+template <class ELFT>
+void LinkerScript<ELFT>::writeDataBytes(StringRef Name, uint8_t *Buf) {
+  const endianness E = ELFT::TargetEndianness;
+
+  if (OutputSectionCommand *Cmd = findOutputSection<ELFT>(Name)) {
+    for (const std::unique_ptr<BaseCommand> &Base2 : Cmd->Commands) {
+      if (auto *DataCmd = dyn_cast<BytesDataCommand>(Base2.get())) {
+        // If the object file format of the output file has an explicit
+        // endianness, the value will be stored in that endianness.
+        switch (DataCmd->Size) {
+        case 1:
+          Buf[DataCmd->Offset] = (uint8_t)DataCmd->Data;
+          break;
+        case 2:
+          write16<E>(&Buf[DataCmd->Offset], (uint16_t)DataCmd->Data);
+          break;
+        case 4:
+          write32<E>(&Buf[DataCmd->Offset], (uint32_t)DataCmd->Data);
+          break;
+        case 8:
+          write64<E>(&Buf[DataCmd->Offset], DataCmd->Data);
+          break;
+        }
+      }
+    }
+  }
+}
+
 template <class ELFT> Expr LinkerScript<ELFT>::getLma(StringRef Name) {
   for (const std::unique_ptr<BaseCommand> &Base : Opt.Commands)
     if (auto *Cmd = dyn_cast<OutputSectionCommand>(Base.get()))
       if (Cmd->LmaExpr && Cmd->Name == Name)
         return Cmd->LmaExpr;
   return {};
 }
 
private:
   void readOutputFormat();
   void readPhdrs();
   void readSearchDir();
   void readSections();
   void readVersion();
   void readVersionScriptCommand();
 
   SymbolAssignment *readAssignment(StringRef Name);
+  BytesDataCommand *readBytesDataCommand(StringRef Tok);
   std::vector<uint8_t> readFill();
   OutputSectionCommand *readOutputSectionDescription(StringRef OutSec);
   std::vector<uint8_t> readOutputSectionFiller(StringRef Tok);
   std::vector<StringRef> readOutputSectionPhdrs();
   InputSectionDescription *readInputSectionDescription(StringRef Tok);
   Regex readFilePatterns();
   std::vector<SectionPattern> readInputSectionsList();
   InputSectionDescription *readInputSectionRules(StringRef FilePattern);
ScriptParser::readOutputSectionDescription(StringRef OutSec) {
   if (skip("ONLY_IF_RW"))
     Cmd->Constraint = ConstraintKind::ReadWrite;
   expect("{");
 
   while (!Error && !skip("}")) {
     StringRef Tok = next();
     if (SymbolAssignment *Assignment = readProvideOrAssignment(Tok, false))
       Cmd->Commands.emplace_back(Assignment);
+    else if (BytesDataCommand *Data = readBytesDataCommand(Tok))
+      Cmd->Commands.emplace_back(Data);
     else if (Tok == "FILL")
       Cmd->Filler = readFill();
     else if (Tok == "SORT")
       readSort();
     else if (peek() == "(")
       Cmd->Commands.emplace_back(readInputSectionDescription(Tok));
     else
       setError("unknown command " + Tok);
if (Tok.endswith_lower("K")) {
     Tok = Tok.drop_back();
   }
   if (Tok.getAsInteger(10, Result))
     return false;
   Result *= Suffix;
   return true;
 }
 
+BytesDataCommand *ScriptParser::readBytesDataCommand(StringRef Tok) {
+  unsigned Size = StringSwitch<unsigned>(Tok)
+                      .Case("BYTE", 1)
+                      .Case("SHORT", 2)
+                      .Case("LONG", 4)
+                      .Case("QUAD", 8)
+                      .Default(-1);
+  if (Size == (unsigned)-1)
+    return nullptr;
+
+  expect("(");
+  uint64_t Val = 0;
+  StringRef S = next();
+  if (!readInteger(S, Val))
+    setError("unexpected value: " + S);
+  expect(")");
+  return new BytesDataCommand(Val, Size);
+}
+
 Expr ScriptParser::readPrimary() {
   if (peek() == "(")
     return readParenExpr();
 
   StringRef Tok = next();
 
   if (Tok == "~") {
     Expr E = readPrimary();

ELF/OutputSections.cpp

if (!Filler.empty())
     fill(Buf, this->getSize(), Filler);
   if (Config->Threads) {
     parallel_for_each(Sections.begin(), Sections.end(),
                       [=](InputSection<ELFT> *C) { C->writeTo(Buf); });
   } else {
     for (InputSection<ELFT> *C : Sections)
       C->writeTo(Buf);
   }
+  // Linker scripts may have BYTE()-family commands with which you
+  // can write arbitrary bytes to the output. Process them if any.
+  Script<ELFT>::X->writeDataBytes(this->Name, Buf);
 }
 
 template <class ELFT>
 EhOutputSection<ELFT>::EhOutputSection()
     : OutputSectionBase<ELFT>(".eh_frame", SHT_PROGBITS, SHF_ALLOC) {}
 
 // Search for an existing CIE record or create a new one.
 // CIE records from input object files are uniquified by their contents

test/ELF/linkerscript/data-commands.s

+# REQUIRES: x86
+# RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux %s -o %t.o
+
+# RUN: echo "SECTIONS                \
+# RUN:  {                            \
+# RUN:    .foo : {                   \
+# RUN:      *(.foo.1)                \
+# RUN:      BYTE(0x11)               \
+# RUN:      *(.foo.2)                \
+# RUN:      SHORT(0x1122)            \
+# RUN:      *(.foo.3)                \
+# RUN:      LONG(0x11223344)         \
+# RUN:      *(.foo.4)                \
+# RUN:      QUAD(0x1122334455667788) \
+# RUN:    }                          \
+# RUN:  }" > %t.script
+# RUN: ld.lld -o %t --script %t.script %t.o
+# RUN: llvm-objdump -s %t | FileCheck %s
+
+# CHECK:      Contents of section .foo:
+# CHECK-NEXT:  00e8 ff11ff22 11ff4433 2211ff88 77665544
+# CHECK-NEXT:  00f8 332211
+
+.section .foo.1, "a"
+ .byte 0xFF
+
+.section .foo.2, "a"
+ .byte 0xFF
+
+.section .foo.3, "a"
+ .byte 0xFF
+
+.section .foo.4, "a"
+ .byte 0xFF