[ELF] - Linkerscript: implemented BYTE/SHORT/LONG/QUAD commands.

[grimar]

Summary

Previously our scripts handles these commands incorrectly. For example:

SECTIONS  {
  .foo : {
 *(.foo.1)
 BYTE(0x11)
...

We accepted the script above treating BYTE as input section description. These commands are used in the wild though, eg:
https://searchcode.com/codesearch/view/42878166/

The BYTE, SHORT, LONG, and QUAD commands store one, two, four, and eight bytes (respectively). After storing the bytes, the location counter is incremented by the number of bytes
stored.

Patch implements them.

[ruiu]

ELF/LinkerScript.cpp

407 ↗

(On Diff #72178)

This

417 ↗

(On Diff #72178)

this and

424 ↗

(On Diff #72178)

this needs comments.

657 ↗

(On Diff #72178)

Don't we have a function to find an output section by name?

660–663 ↗

(On Diff #72178)

This code seems to write the bytes in the host endianess instead of the target endianness.

1431–1436 ↗

(On Diff #72178)

I don't think we need these enums. Can't this be

int Size = StringSwitch(Tok)
  .Case("BYTE", 1)
  .Case("SHORT", 2)
  ...

?

ELF/LinkerScript.h

47–51 ↗

(On Diff #72178)

It's time to add comments for each command.

AssignmentKind, // . = expr or <sym> = expr
OutputSectionKind,
InputSectionKind,
AssertKind, // ASSERT(expr)
BytesDataKind, // BYTE(expr), SHORT(expr), LONG(expr) or QUAD(expr)

ELF/OutputSections.cpp

1008 ↗

(On Diff #72178)

It needs a comment.

// Linker scripts may have BYTE()-family commands with which you
// can write arbitrary bytes to the output. Process them if any.

[grimar]

ELF/LinkerScript.cpp

407 ↗

(On Diff #72178)

Done.

417 ↗

(On Diff #72178)

Done.

424 ↗

(On Diff #72178)

Done.

657 ↗

(On Diff #72178)

I do not think we have, I created new one, it can be reused in getFiller() in following patch.

660–663 ↗

(On Diff #72178)

Ah, right. Fixed.

1431–1436 ↗

(On Diff #72178)

Done.

ELF/LinkerScript.h

47–51 ↗

(On Diff #72178)

Done. (btw comments looks a bit displaced because are clang-formatted)

ELF/OutputSections.cpp

1008 ↗

(On Diff #72178)

Done.

[• rafael]

ELF/LinkerScript.h

146 ↗

(On Diff #72248)

This could just store a StringRef or ArrayRef, no?

We can compute the correct (target endian) representation upfront and simplify the remaining code.

[grimar]

ELF/LinkerScript.h

146 ↗

(On Diff #72248)

Not sure about StringRef/ArrayRef, that could be std::vector<uint_8t>, computed in theory, but I think I can't do it.

Problem is that readBytesDataCommand() that reads command know nothing about target endianess because is not ELFT templated.

[grimar]

And I mean it is too early to know it as script can go before any other files, so I need to keep the value anyways, and in that case have a vector or something else in addition does not make much sence ?

[• rafael]

ELF/LinkerScript.cpp

713 ↗

(On Diff #72248)

You can probably use getSectionIndex.

[grimar]

• Addressed review comments.

ELF/LinkerScript.cpp

713 ↗

(On Diff #72277)

Done.

[• rafael]

LGTM with nits, but wait so see if Rui is OK with it too.

test/ELF/linkerscript/data-commands.s

2 ↗

(On Diff #72277)

Add a big endian test.

21 ↗

(On Diff #72277)

Don't check the address.

[ruiu]

ELF/LinkerScript.cpp

424 ↗

(On Diff #72277)

Add a blank line before this.

Maybe just

// Handle BYTE(), SHORT(), LONG(), or QUAD().

717 ↗

(On Diff #72277)

This looks odd. You can return early.

if (I == INT_MAX)
  return;

726 ↗

(On Diff #72277)

I'd factor this switch out as an independent functions so that we can do something like this here.

writeInt<ELFT>(Buf + DataCmd->Offset, DataCmd->Data, DataCmd->Size);

where writeInt is defined as

writeInt(uint8_t *Buf, uint64_t Data, uint64_t Size);

1519 ↗

(On Diff #72277)

Use int instead of unsigned so to remove the cast to unsigned below.

[grimar]

• Addressed review comments.

ELF/LinkerScript.cpp

424 ↗

(On Diff #72277)

Done.

717 ↗

(On Diff #72277)

Done.

1519 ↗

(On Diff #72277)

Done.

test/ELF/linkerscript/data-commands.s

2 ↗

(On Diff #72277)

Done.

21 ↗

(On Diff #72277)

Done.

[ruiu]

LGTM with a nit.

ELF/LinkerScript.cpp

742–743 ↗

(On Diff #72454)

This comment doesn't make sense to be written here. Probably I'd just remove.

[grimar]

ELF/LinkerScript.cpp

742–743 ↗

(On Diff #72454)

Yeah, initially I wrote it because we have comment for filler saying it works with big-endian,
but for this place there is no disagree between ld, gold and spec, so it was probably excessive, I removed it from commit.

lld/trunk/ELF/LinkerScript.h

 // Config and ScriptConfig.
 void readLinkerScript(MemoryBufferRef MB);
 
 void readVersionScript(MemoryBufferRef MB);
 
 // This enum is used to implement linker script SECTIONS command.
 // https://sourceware.org/binutils/docs/ld/SECTIONS.html#SECTIONS
 enum SectionsCommandKind {
-  AssignmentKind,
+  AssignmentKind, // . = expr or <sym> = expr
   OutputSectionKind,
   InputSectionKind,
-  AssertKind
+  AssertKind,   // ASSERT(expr)
+  BytesDataKind // BYTE(expr), SHORT(expr), LONG(expr) or QUAD(expr)
 };
 
 struct BaseCommand {
   BaseCommand(int K) : Kind(K) {}
   virtual ~BaseCommand() {}
   int Kind;
 };
 

 };
 
 struct AssertCommand : BaseCommand {
   AssertCommand(Expr E) : BaseCommand(AssertKind), Expression(E) {}
   static bool classof(const BaseCommand *C);
   Expr Expression;
 };
 
+struct BytesDataCommand : BaseCommand {
+  BytesDataCommand(uint64_t Data, unsigned Size)
+      : BaseCommand(BytesDataKind), Data(Data), Size(Size) {}
+  static bool classof(const BaseCommand *C);
+  uint64_t Data;
+  unsigned Offset;
+  unsigned Size;
+};
+
 struct PhdrsCommand {
   StringRef Name;
   unsigned Type;
   bool HasFilehdr;
   bool HasPhdrs;
   unsigned Flags;
   Expr LMAExpr;
 };
public:
   void processCommands(OutputSectionFactory<ELFT> &Factory);
   void createSections(OutputSectionFactory<ELFT> &Factory);
   void adjustSectionsBeforeSorting();
 
   std::vector<PhdrEntry<ELFT>> createPhdrs();
   bool ignoreInterpSection();
 
   ArrayRef<uint8_t> getFiller(StringRef Name);
+  void writeDataBytes(StringRef Name, uint8_t *Buf);
   Expr getLma(StringRef Name);
   bool shouldKeep(InputSectionBase<ELFT> *S);
   void assignOffsets(OutputSectionCommand *Cmd);
   void assignAddresses();
   bool hasPhdrsCommands();
   uint64_t getOutputSectionAddress(StringRef Name) override;
   uint64_t getOutputSectionSize(StringRef Name) override;
   uint64_t getOutputSectionAlign(StringRef Name) override;

lld/trunk/ELF/LinkerScript.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/StringSaver.h"
 
 using namespace llvm;
 using namespace llvm::ELF;
 using namespace llvm::object;
+using namespace llvm::support::endian;
 using namespace lld;
 using namespace lld::elf;
 
 LinkerScriptBase *elf::ScriptBase;
 ScriptConfiguration *elf::ScriptConfig;
 
 template <class ELFT> static void addRegular(SymbolAssignment *Cmd) {
   Symbol *Sym = Symtab<ELFT>::X->addRegular(Cmd->Name, STB_GLOBAL, STV_DEFAULT);

 bool InputSectionDescription::classof(const BaseCommand *C) {
   return C->Kind == InputSectionKind;
 }
 
 bool AssertCommand::classof(const BaseCommand *C) {
   return C->Kind == AssertKind;
 }
 
+bool BytesDataCommand::classof(const BaseCommand *C) {
+  return C->Kind == BytesDataKind;
+}
+
 template <class ELFT> static bool isDiscarded(InputSectionBase<ELFT> *S) {
   return !S || !S->Live;
 }
 
 template <class ELFT> LinkerScript<ELFT>::LinkerScript() {}
 template <class ELFT> LinkerScript<ELFT>::~LinkerScript() {}
 
 template <class ELFT>
void LinkerScript<ELFT>::switchTo(OutputSectionBase<ELFT> *Sec) {
   flush();
   CurOutSec = Sec;
 
   Dot = alignTo(Dot, CurOutSec->getAlignment());
   CurOutSec->setVA(isTbss(CurOutSec) ? Dot + ThreadBssOffset : Dot);
 }
 
 template <class ELFT> void LinkerScript<ELFT>::process(BaseCommand &Base) {
+  // This handles the assignments to symbol or to a location counter (.)
   if (auto *AssignCmd = dyn_cast<SymbolAssignment>(&Base)) {
     if (AssignCmd->Name == ".") {
       // Update to location counter means update to section size.
       Dot = AssignCmd->Expression(Dot);
       CurOutSec->setSize(Dot - CurOutSec->getVA());
       return;
     }
     assignSectionSymbol<ELFT>(AssignCmd, CurOutSec, Dot - CurOutSec->getVA());
     return;
   }
+
+  // Handle BYTE(), SHORT(), LONG(), or QUAD().
+  if (auto *DataCmd = dyn_cast<BytesDataCommand>(&Base)) {
+    DataCmd->Offset = Dot - CurOutSec->getVA();
+    Dot += DataCmd->Size;
+    CurOutSec->setSize(Dot - CurOutSec->getVA());
+    return;
+  }
+
+  // It handles single input section description command,
+  // calculates and assigns the offsets for each section and also
+  // updates the output section size.
   auto &ICmd = cast<InputSectionDescription>(Base);
   for (InputSectionData *ID : ICmd.Sections) {
     auto *IB = static_cast<InputSectionBase<ELFT> *>(ID);
     switchTo(IB->OutSec);
     if (auto *I = dyn_cast<InputSection<ELFT>>(IB))
       output(I);
     else
       flush();

 ArrayRef<uint8_t> LinkerScript<ELFT>::getFiller(StringRef Name) {
   for (const std::unique_ptr<BaseCommand> &Base : Opt.Commands)
     if (auto *Cmd = dyn_cast<OutputSectionCommand>(Base.get()))
       if (Cmd->Name == Name)
         return Cmd->Filler;
   return {};
 }
 
+template <class ELFT>
+static void writeInt(uint8_t *Buf, uint64_t Data, uint64_t Size) {
+  const endianness E = ELFT::TargetEndianness;
+
+  switch (Size) {
+  case 1:
+    *Buf = (uint8_t)Data;
+    break;
+  case 2:
+    write16<E>(Buf, Data);
+    break;
+  case 4:
+    write32<E>(Buf, Data);
+    break;
+  case 8:
+    write64<E>(Buf, Data);
+    break;
+  default:
+    llvm_unreachable("unsupported Size argument");
+  }
+}
+
+template <class ELFT>
+void LinkerScript<ELFT>::writeDataBytes(StringRef Name, uint8_t *Buf) {
+  int I = getSectionIndex(Name);
+  if (I == INT_MAX)
+    return;
+
+  OutputSectionCommand *Cmd =
+      dyn_cast<OutputSectionCommand>(Opt.Commands[I].get());
+  for (const std::unique_ptr<BaseCommand> &Base2 : Cmd->Commands)
+    if (auto *DataCmd = dyn_cast<BytesDataCommand>(Base2.get()))
+      writeInt<ELFT>(&Buf[DataCmd->Offset], DataCmd->Data, DataCmd->Size);
+}
+
 template <class ELFT> Expr LinkerScript<ELFT>::getLma(StringRef Name) {
   for (const std::unique_ptr<BaseCommand> &Base : Opt.Commands)
     if (auto *Cmd = dyn_cast<OutputSectionCommand>(Base.get()))
       if (Cmd->LmaExpr && Cmd->Name == Name)
         return Cmd->LmaExpr;
   return {};
 }
 
private:
   void readOutputFormat();
   void readPhdrs();
   void readSearchDir();
   void readSections();
   void readVersion();
   void readVersionScriptCommand();
 
   SymbolAssignment *readAssignment(StringRef Name);
+  BytesDataCommand *readBytesDataCommand(StringRef Tok);
   std::vector<uint8_t> readFill();
   OutputSectionCommand *readOutputSectionDescription(StringRef OutSec);
   std::vector<uint8_t> readOutputSectionFiller(StringRef Tok);
   std::vector<StringRef> readOutputSectionPhdrs();
   InputSectionDescription *readInputSectionDescription(StringRef Tok);
   Regex readFilePatterns();
   std::vector<SectionPattern> readInputSectionsList();
   InputSectionDescription *readInputSectionRules(StringRef FilePattern);
ScriptParser::readOutputSectionDescription(StringRef OutSec) {
   if (skip("ONLY_IF_RW"))
     Cmd->Constraint = ConstraintKind::ReadWrite;
   expect("{");
 
   while (!Error && !skip("}")) {
     StringRef Tok = next();
     if (SymbolAssignment *Assignment = readProvideOrAssignment(Tok, false))
       Cmd->Commands.emplace_back(Assignment);
+    else if (BytesDataCommand *Data = readBytesDataCommand(Tok))
+      Cmd->Commands.emplace_back(Data);
     else if (Tok == "FILL")
       Cmd->Filler = readFill();
     else if (Tok == "SORT")
       readSort();
     else if (peek() == "(")
       Cmd->Commands.emplace_back(readInputSectionDescription(Tok));
     else
       setError("unknown command " + Tok);
if (Tok.endswith_lower("K")) {
     Tok = Tok.drop_back();
   }
   if (Tok.getAsInteger(10, Result))
     return false;
   Result *= Suffix;
   return true;
 }
 
+BytesDataCommand *ScriptParser::readBytesDataCommand(StringRef Tok) {
+  int Size = StringSwitch<unsigned>(Tok)
+                 .Case("BYTE", 1)
+                 .Case("SHORT", 2)
+                 .Case("LONG", 4)
+                 .Case("QUAD", 8)
+                 .Default(-1);
+  if (Size == -1)
+    return nullptr;
+
+  expect("(");
+  uint64_t Val = 0;
+  StringRef S = next();
+  if (!readInteger(S, Val))
+    setError("unexpected value: " + S);
+  expect(")");
+  return new BytesDataCommand(Val, Size);
+}
+
 Expr ScriptParser::readPrimary() {
   if (peek() == "(")
     return readParenExpr();
 
   StringRef Tok = next();
 
   if (Tok == "~") {
     Expr E = readPrimary();

lld/trunk/ELF/OutputSections.cpp

if (!Filler.empty())
     fill(Buf, this->getSize(), Filler);
   if (Config->Threads) {
     parallel_for_each(Sections.begin(), Sections.end(),
                       [=](InputSection<ELFT> *C) { C->writeTo(Buf); });
   } else {
     for (InputSection<ELFT> *C : Sections)
       C->writeTo(Buf);
   }
+  // Linker scripts may have BYTE()-family commands with which you
+  // can write arbitrary bytes to the output. Process them if any.
+  Script<ELFT>::X->writeDataBytes(this->Name, Buf);
 }
 
 template <class ELFT>
 EhOutputSection<ELFT>::EhOutputSection()
     : OutputSectionBase<ELFT>(".eh_frame", SHT_PROGBITS, SHF_ALLOC) {}
 
 // Search for an existing CIE record or create a new one.
 // CIE records from input object files are uniquified by their contents

lld/trunk/test/ELF/linkerscript/data-commands.s

+# REQUIRES: x86,mips
+# RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux %s -o %t.o
+# RUN: echo "SECTIONS                \
+# RUN:  {                            \
+# RUN:    .foo : {                   \
+# RUN:      *(.foo.1)                \
+# RUN:      BYTE(0x11)               \
+# RUN:      *(.foo.2)                \
+# RUN:      SHORT(0x1122)            \
+# RUN:      *(.foo.3)                \
+# RUN:      LONG(0x11223344)         \
+# RUN:      *(.foo.4)                \
+# RUN:      QUAD(0x1122334455667788) \
+# RUN:    }                          \
+# RUN:  }" > %t.script
+# RUN: ld.lld -o %t --script %t.script %t.o
+# RUN: llvm-objdump -s %t | FileCheck %s
+
+# CHECK:      Contents of section .foo:
+# CHECK-NEXT:   ff11ff22 11ff4433 2211ff88 77665544
+# CHECK-NEXT:   332211
+
+# RUN: llvm-mc -filetype=obj -triple=mips64-unknown-linux %s -o %tmips64be
+# RUN: ld.lld --script %t.script %tmips64be -o %t2
+# RUN: llvm-objdump -s %t2 | FileCheck %s --check-prefix=BE
+# BE:      Contents of section .foo:
+# BE-NEXT:   ff11ff11 22ff1122 3344ff11 22334455
+# BE-NEXT:   667788
+
+.section .foo.1, "a"
+ .byte 0xFF
+
+.section .foo.2, "a"
+ .byte 0xFF
+
+.section .foo.3, "a"
+ .byte 0xFF
+
+.section .foo.4, "a"
+ .byte 0xFF