[LLD] Add support for --unique option

[davidb]

Summary

Places orphan sections into a unique output section. This prevents the merging of orphan sections of the same name.
Matches behaviour of GNU ld --unique. --unique=pattern is not implemented.

Motivated user case shown in the test has 2 local symbols as they would appear if C++ source has been compiled with -ffunction-sections. The merging of these sections in the case of a partial link (-r) may limit the effectiveness of -gc-sections of a subsequent link.

[davidb]

Fix lint checks

[davidb]

rebase

[MaskRay]

Do you have a plan to implement --unique=pattern (overrides output section descriptions)? It is different from --unique (only applies to orphan sections).

The merging of these sections in the case of a partial link (-r) inhibits deadstripping.

Dead stripping is a Mach-O term. Non --unique does not inhibit --gc-sections (without -r).
Without --unique, some orphan sections may be unnecessarily kept (as a whole their "liveness" can be larger than the composing parts).

Note that -r --gc-sections is not implemented in lld.

lld/ELF/LinkerScript.cpp

695

Use else if

lld/ELF/Options.td

378

No period.

Add an item to lld/docs/ld.lld.1 (manpage)

lld/test/ELF/Inputs/unique-orphans.s

1 ↗

(On Diff #247933)

Delete the file.

lld/test/ELF/unique-orphans.s

3

Delete -unknown-linux

4

Keep one REQUIRES

7

The instructions are verbose and unnecessary.

.section .foo,"a",unique,1
.byte 1

.section .foo,"a",unique,2
.byte 2

.section .foo,"a",unique,3
.byte 3

Check both ld.lld %t.o -o %t and ld.lld -r %t.o -o %t.ro

8

Test an output section description suppresses --unique.

echo 'SECTIONS { .foo : { *(.foo) }}' > %t.script`

See test/ELF/linkerscript/input-archive.s for a recent example.

26

Check sections with llvm-readelf -S, instead of objdump (may not even be installed)

30

Not necessary. There are plenty of tests for the non --unique case.

[davidb]

In D75536#1903727, @MaskRay wrote:

Do you have a plan to implement --unique=pattern (overrides output section descriptions)? It is different from --unique (only applies to orphan sections).

I don't have a plan to implement --unique=pattern, as the default is good enough for my needs. However I would consider getting back and doing it when I next have some free cycles.

The merging of these sections in the case of a partial link (-r) inhibits deadstripping.

Dead stripping is a Mach-O term. Non --unique does not inhibit --gc-sections (without -r).
Without --unique, some orphan sections may be unnecessarily kept (as a whole their "liveness" can be larger than the composing parts).

Note that -r --gc-sections is not implemented in lld.

Sorry, force of habit. PlayStation toolchains also use the term dead-stripping for ELF binaries. I did indeed mean gc-sections.

The current workflow involves multiple link stages, involving a partial link and then an executable link. It is the second link that performs -gc-sections. We would like to keep all functions in their own individual sections after the partial link.

I could update the test to demonstrate this if it is at all useful.

[MaskRay]

In D75536#1903914, @davidb wrote:

In D75536#1903727, @MaskRay wrote:

Do you have a plan to implement --unique=pattern (overrides output section descriptions)? It is different from --unique (only applies to orphan sections).

I don't have a plan to implement --unique=pattern, as the default is good enough for my needs. However I would consider getting back and doing it when I next have some free cycles.

The merging of these sections in the case of a partial link (-r) inhibits deadstripping.

Dead stripping is a Mach-O term. Non --unique does not inhibit --gc-sections (without -r).
Without --unique, some orphan sections may be unnecessarily kept (as a whole their "liveness" can be larger than the composing parts).

Note that -r --gc-sections is not implemented in lld.

Sorry, force of habit. PlayStation toolchains also use the term dead-stripping for ELF binaries. I did indeed mean gc-sections.

The current workflow involves multiple link stages, involving a partial link and then an executable link. It is the second link that performs -gc-sections. We would like to keep all functions in their own individual sections after the partial link.

I could update the test to demonstrate this if it is at all useful.

A test is not needed. Just make the test minimal (for example, as I suggested).

Please update SUMMARY by mentioning the --gc-sections scenario and delete the mention of deadstripping.

[davidb]

• review comments

[davidb]

In D75536#1904145, @MaskRay wrote:

In D75536#1903914, @davidb wrote:

In D75536#1903727, @MaskRay wrote:

Do you have a plan to implement --unique=pattern (overrides output section descriptions)? It is different from --unique (only applies to orphan sections).

I don't have a plan to implement --unique=pattern, as the default is good enough for my needs. However I would consider getting back and doing it when I next have some free cycles.

The merging of these sections in the case of a partial link (-r) inhibits deadstripping.

Dead stripping is a Mach-O term. Non --unique does not inhibit --gc-sections (without -r).
Without --unique, some orphan sections may be unnecessarily kept (as a whole their "liveness" can be larger than the composing parts).

Note that -r --gc-sections is not implemented in lld.

Sorry, force of habit. PlayStation toolchains also use the term dead-stripping for ELF binaries. I did indeed mean gc-sections.

The current workflow involves multiple link stages, involving a partial link and then an executable link. It is the second link that performs -gc-sections. We would like to keep all functions in their own individual sections after the partial link.

I could update the test to demonstrate this if it is at all useful.

A test is not needed. Just make the test minimal (for example, as I suggested).

Please update SUMMARY by mentioning the --gc-sections scenario and delete the mention of deadstripping.

Thank you for the review comments, especially wrt the tests. I have updated the summary, just need to add a comment to the manpage, which I'll try to get round to in the morning.

[jhenderson]

lld/ELF/Options.td

377

Nit: too many blank lines here.

lld/test/ELF/unique-orphans.s

14

instance -> instances (here and below)

27

Nit: no newline at EOF.

[grimar]

lld/test/ELF/unique-orphans.s

14

Consider using ## for comments. It is a modern common way.

18

Seems you can have a single chack block:

# CHECK-3:   .foo
# CHECK-NOT: .foo

And remove --check-prefix from everywhere?

20

.text.foo -> .foo? (and above)

[MaskRay]

lld/test/ELF/unique-orphans.s

26

Add a comment, along the lines of

## Test that --unique does not affect sections specified in output section descriptions.

26

Not done. Use llvm-readelf -S

[davidb]

lld/test/ELF/unique-orphans.s

26

I added the llvm-prefix so it should pick the right objdump. There are 2x more uses of llvm-objdump than readelf in the lld test suite, 4x more in LLVM as a whole. What is the motivation for readelf?

[davidb]

• review comments

[davidb]

• fix formatting from clang-format

[davidb]

• review comments

[davidb]

• replace objdump with readelf

[davidb]

• rebase

[MaskRay]

lld/ELF/Config.h

197

Delete = false

lld/ELF/Driver.cpp

970

lexicographical order

lld/test/ELF/unique-orphans.s

15

The comment ## Test with -r is redundant and should be deleted.

[davidb]

• review comments

[davidb]

lld/ELF/Config.h

197

Why do you not want me to be explicit here? Many other options above are

lld/ELF/Driver.cpp

970

was result of rebase

[davidb]

lld/test/ELF/unique-orphans.s

15

shame.

[MaskRay]

Places orphan sections into a unique output section. This prevents the merging of orphan sections of the same name. Matches behaviour of gnu LD option --unique[=SECTION], with the exception I haven't added the wildcard SECTION support.

gnu LD -> GNU ld

Matches behaviour of GNU ld --unique. --unique=pattern is not implemented.

lld/test/ELF/unique-orphans.s

25

Nit: --check-prefix=SCRIPT

UNIQUE_ is not necessary in the context.

[grimar]

lld/ELF/Config.h

197

There is a difference. Look at sysvHash for example whitch is explicitly set to false.
sysvHash is not always set to something in the code. Hence we have to initialize it.

You always assing unique: config->unique = args.hasArg(OPT_unique);. So no need to initialize it explicitly.

[davidb]

• review comments

[davidb]

• rebase

[davidb]

• formatting

[MaskRay]

LGTM. Please wait for @grimar's opinions.

[grimar]

LGTM

lld/ELF/Config.h

struct Configuration {
   bool sysvHash = false;
   bool target1Rel;
   bool trace;
   bool thinLTOEmitImportsFiles;
   bool thinLTOIndexOnly;
   bool timeTraceEnabled;
   bool tocOptimize;
   bool undefinedVersion;
+  bool unique;
   bool useAndroidRelrTags = false;
   bool warnBackrefs;
   bool warnCommon;
   bool warnIfuncTextrel;
   bool warnMissingEntry;
   bool warnSymbolOrdering;
   bool writeAddends;
   bool zCombreloc;

lld/ELF/Driver.cpp

static void readConfigs(opt::InputArgList &args) {
   config->sectionStartMap = getSectionStartMap(args);
   config->shared = args.hasArg(OPT_shared);
   config->singleRoRx = args.hasArg(OPT_no_rosegment);
   config->soName = args.getLastArgValue(OPT_soname);
   config->sortSection = getSortSection(args);
   config->splitStackAdjustSize = args::getInteger(args, OPT_split_stack_adjust_size, 16384);
   config->strip = getStrip(args);
   config->sysroot = args.getLastArgValue(OPT_sysroot);
   config->target1Rel = args.hasFlag(OPT_target1_rel, OPT_target1_abs, false);
   config->target2 = getTarget2(args);
   config->thinLTOCacheDir = args.getLastArgValue(OPT_thinlto_cache_dir);
   config->thinLTOCachePolicy = CHECK(
       parseCachePruningPolicy(args.getLastArgValue(OPT_thinlto_cache_policy)),
       "--thinlto-cache-policy: invalid cache policy");
   config->thinLTOEmitImportsFiles = args.hasArg(OPT_thinlto_emit_imports_files);
   config->thinLTOIndexOnly = args.hasArg(OPT_thinlto_index_only) ||
                              args.hasArg(OPT_thinlto_index_only_eq);
   config->thinLTOIndexOnlyArg = args.getLastArgValue(OPT_thinlto_index_only_eq);
   config->thinLTOJobs = args::getInteger(args, OPT_thinlto_jobs, -1u);
   config->thinLTOObjectSuffixReplace =
       getOldNewOptions(args, OPT_thinlto_object_suffix_replace_eq);
   config->thinLTOPrefixReplace =
       getOldNewOptions(args, OPT_thinlto_prefix_replace_eq);
   config->timeTraceEnabled = args.hasArg(OPT_time_trace);
   config->timeTraceGranularity =
       args::getInteger(args, OPT_time_trace_granularity, 500);
   config->trace = args.hasArg(OPT_trace);
   config->undefined = args::getStrings(args, OPT_undefined);
   config->undefinedVersion =
       args.hasFlag(OPT_undefined_version, OPT_no_undefined_version, true);
+  config->unique = args.hasArg(OPT_unique);
   config->useAndroidRelrTags = args.hasFlag(
       OPT_use_android_relr_tags, OPT_no_use_android_relr_tags, false);
   config->unresolvedSymbols = getUnresolvedSymbolPolicy(args);
   config->warnBackrefs =
       args.hasFlag(OPT_warn_backrefs, OPT_no_warn_backrefs, false);
   config->warnCommon = args.hasFlag(OPT_warn_common, OPT_no_warn_common, false);
   config->warnIfuncTextrel =
       args.hasFlag(OPT_warn_ifunc_textrel, OPT_no_warn_ifunc_textrel, false);

lld/ELF/LinkerScript.cpp

void LinkerScript::addOrphanSections() {
   std::vector<OutputSection *> v;
 
   std::function<void(InputSectionBase *)> add;
   add = [&](InputSectionBase *s) {
     if (s->isLive() && !s->parent) {
       orphanSections.push_back(s);
 
       StringRef name = getOutputSectionName(s);
-      if (OutputSection *sec = findByName(sectionCommands, name)) {
+      if (config->unique) {
+        v.push_back(createSection(s, name));
+      } else if (OutputSection *sec = findByName(sectionCommands, name)) {
         sec->recordSection(s);
       } else {
         if (OutputSection *os = addInputSec(map, s, name))
           v.push_back(os);
         assert(isa<MergeInputSection>(s) ||
                 s->getOutputSection()->sectionIndex == UINT32_MAX);
       }
     }
 
-    if (config->relocatable)
-      for (InputSectionBase *depSec : s->dependentSections)
-        if (depSec->flags & SHF_LINK_ORDER)
-          add(depSec);
   };
 
   // For futher --emit-reloc handling code we need target output section
   // to be created before we create relocation output section, so we want
   // to create target sections first. We do not want priority handling
   // for synthetic sections because them are special.
   for (InputSectionBase *isec : inputSections) {
     // In -r links, SHF_LINK_ORDER sections are added while adding their parent

lld/ELF/Options.td

 defm trace_symbol: Eq<"trace-symbol", "Trace references to symbols">;
 
 defm undefined: Eq<"undefined", "Force undefined symbol during linking">,
   MetaVarName<"<symbol>">;
 
 defm undefined_glob: Eq<"undefined-glob", "Force undefined symbol during linking">,
   MetaVarName<"<pattern>">;
 
+def unique: F<"unique">, HelpText<"Creates a separate output section for every orphan input section">;
+
 defm unresolved_symbols:
   Eq<"unresolved-symbols", "Determine how to handle unresolved symbols">;
 
 defm undefined_version: B<"undefined-version",
   "Allow unused version in version script (default)",
   "Report version scripts that refer undefined symbols">;
 
 defm rsp_quoting: Eq<"rsp-quoting", "Quoting style for response files">,

lld/docs/ld.lld.1

 matches zero or more characters,
 .Cm ?
 matches any single character, and
 .Cm [...]
 matches the characters within brackets.
 All symbols that match
 a given pattern are handled as if they were given as arguments of
 .Fl -undefined .
+.It Fl -unique
+Creates a separate output section for every orphan input section.
 .It Fl -unresolved-symbols Ns = Ns Ar value
 Determine how to handle unresolved symbols.
 .It Fl -use-android-relr-tags
 Use SHT_ANDROID_RELR / DT_ANDROID_RELR* tags instead of SHT_RELR / DT_RELR*.
 .It Fl v
 Display the version number and proceed with linking if object files are
 specified.
 .It Fl V , Fl -version

lld/test/ELF/unique-orphans.s

+# REQUIRES: x86
+# RUN: llvm-mc -filetype=obj -triple=x86_64 %s -o %t.o
+
+.section .foo,"a",@progbits,unique,1
+.byte 1
+
+.section .foo,"a",@progbits,unique,2
+.byte 2
+
+.section .foo,"a",@progbits,unique,3
+.byte 3
+
+## We should have 3 instances of orphan section foo.
+## Test with -r
+# RUN: ld.lld %t.o -o %t.elf --unique 
+# RUN: llvm-readelf -S %t.elf | FileCheck %s
+
+# CHECK-COUNT-3: .foo
+# CHECK-NOT: .foo
+
+## Test that --unique does not affect sections specified in output section descriptions.
+# RUN: echo 'SECTIONS { .foo : { *(.foo) }}' > %t.script
+# RUN: ld.lld %t.o -o %t2.elf -T %t.script --unique 
+# RUN: llvm-readelf -S %t2.elf | FileCheck --check-prefix SCRIPT %s
+# SCRIPT: .foo
+# SCRIPT-NOT: .foo