In fact, can we have an option controlling this ? Does it have anything to do with -malign-data gcc switch ?

In D42748#1646442, @arichardson wrote:

FreeBSD libc should be fixed instead.

In D64930#1607709, @MaskRay wrote:

I'm not sure about breaking the (p_vaddr % p_align == 0) invariant. ld.bfd, gold, and (for the most part) lld currently provide this invariant for TLS segments in output files.

I think p_vaddr%p_align = p_offset%p_align is the invariant we must satisfy. p_vaddr%p_align=0 is not. However, p_vaddr%p_align!=0 will expose bugs (runtime address not congruent to p_vaddr modulo p_align) in existing ld.so implementations so I added a workaround to D64906.

Below are discussions about bugs in various ld.so implementations. They should not be affected by these patches but ideally they should be fixed.

Supporting n-mod-m alignment isn't free for dynamic TLS -- allocators typically have [posix_]memalign and free APIs, but I'm not aware of APIs that provide n-mod-m alignment.

When allocating TCB+static TLS blocks, ld.so should properly pad the allocated space. See my comment added to InputSection.cpp in D64906:

// Variant 1. TP will be followed by an optional gap (which is the size of 2
// pointers on ARM/AArch64, 0 on other targets), followed by alignment
// padding, then the static TLS blocks. The alignment padding is added so that
// (TP + gap + padding) is congruent to p_vaddr modulo p_align.
//
// Variant 2. Static TLS blocks, followed by alignment padding are placed
// before TP. The alignment padding is added so that (TP - padding -
// p_memsz) is congruent to p_vaddr modulo p_align.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=24606

This is a great test case! CC @kib who reviewed https://reviews.freebsd.org/D13378 @luporl for powerpc64. CC @emaste who is already subscribed...

FreeBSD rtld-elf (FreeBSD 13.0-CURRENT #89 r350361M) doesn't make runtime address of PT_TLS congruent to p_vaddr modulo p_align on i386/amd64/arm64/powerpc64:

# amd64 case
tp          = 0x8002278d0
&tlsvar     = 0x8002212c8  # this should be 1-mod-4
tlsvar[1]   = 7            # the value is correct

In D42748#1515212, @MaskRay wrote:

Regarding this patch: I don't think _DYNAMIC would be useful for -static-pie since we would need to relocate it first so _DYNAMIC could still be undefined and used to check for dynamic linker presence.

You need _DYNAMIC for -static-pie. You need _DYNAMIC to find DT_REL* tags, then perform relocations.

I had a quick look at glibc's static-pie code and it seems like glibc loads the first GOT entry for most architectures to get the address of the dynamic section.

It is one thing I don't like (I don't know any other program that needs this) about glibc (this is wrriten in x86 psABI, but probably not in other psABIs): the first entry of .got.plt or .got holds the link-time address of _DYNAMIC. At runtime a pcrel load of _GLOBAL_OFFSET_TABLE_ gets the link-time address, subtracts it from the runtime address of _DYNAMIC, then you get the load base. glibc could parse the program header to get p_vaddr of PT_DYNAMIC instead.

[Perhaps, pcrel load gets runtime address, not important.]

In D42748#1513202, @MaskRay wrote:

And if you decide to support static pie, you also need _DYNAMIC.

Are you sure about this ? For static PIE, as I understand, we miss some kind of relocator in csu.

On the linker side, static pie is -static -pie --no-dynamic-linker. The -pie causes lld to create dynamic sections.

void
_start(char **ap, void (*cleanup)(void))
{
	int argc;
	char **argv;
	char **env;

	argc = *(long *)(void *)ap;
	argv = ap + 1;
	env = ap + 2 + argc;
	handle_argv(argc, argv, env);

	if (&_DYNAMIC != NULL) {
		atexit(cleanup);
	} else {
		process_irelocs();
		_init_tls();
	}

If FreeBSD used to differentiate dynamically/statically linked programs with &_DYNAMIC != NULL, I think it is probably time to revisit.
We have two ways to have dynamic sections in a statically linked program, --export-dynamic (https://reviews.llvm.org/rL295240) or -pie (static pie). If the distinction of dynamic/static is whether the program is interpreted by PT_INTERP, isn't testing this property directly more straightforward?

No, it is not. PT_INTERP requires the program (csu) to find out the aux vector, parse it to find the binary base, and then parse ELF header and program headers. All this while the binary itself is not relocated. This is significant blow of the crt, and added complexity, which also means that every binary (not only static) now carry a code which we cannot fix because it is statically linked even into dynamic binaries.

In D42748#1511335, @MaskRay wrote:

From the description

This also causes a .dynamic section, the _DYNAMIC symtbol and a PT_DYNAMIC header to be added to the output file. This causes problems for example when trying to run such a binary on FreeBSD MIPS.

What I know is just that the presence of _DYNAMIC caused a problem but I don't have more information why it caused the problem. Without more information I can only conjecture. My intuition says it is more likely a problem if the dynamic is absent in some scenarios, I don't understand how the presence (though probably unexpected by you) caused a problem.

In D42748#1510087, @MaskRay wrote:

@arichardson
If the MIPS problem was similar to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236165 ,
moving away from &_DYNAMIC will be a more reliable approach.
To check if an executable is dynamically linked, inspecting PT_INTERP is a better choice.

Checking if a weak undefined symbol has zero address is unreliale.
Some compilers may produce a GOT-generating relocation, some may produce an absolute relocation.
After linking, you may see the relocation resolved to static 0, or see a dynamic relocation (if at runtime there is some module providing the dynamic symbol, the weak reference will resolve to non-zero)

Quoting http://www.sco.com/developers/gabi/latest/ch4.symtab.html

The behavior of weak symbols in areas not specified by this document is implementation defined. Weak symbols are intended primarily for use in system software. Applications using weak symbols are unreliable since changes in the runtime environment might cause the execution to fail.

Regarding this patch. Actually, -Bstatic (synonym of -static in ld.bfd and lld) just means: "don't look for libfoo.so when a -lfoo is seen, before next -Bdynamic". I think it is weird to use it to decide whether we should emit .dynamic . (In the compiler drivers (gcc/clang/etc), -static mean static linking, but that is different from -Bstatic/-static in ld.bfd/lld.)

This change neither improves similarity with ld.bfd nor makes behaviors reasonable that suits lld (the internals of lld are very different from ld.bfd, some behaviors of ld.bfd may not suit lld). The logic to emit .dynamic .dynsym .dynstr etc in the 3 linkers:

lld: has_dso || --shared || --pie || --export-dynamic
gold: has_dso || --shared || --pie
bfd: (--shared || --pie) || ((not -r) && info->nointerp && (info->export_dynamic || info->dynamic)) && some (almost always true) conditions

@ed I want to knore more about your motivation to add --export-dynamic to the condition in D29982. Why do you need .dynamic in a position dependent executable for CloudABI, which has no shared object dependency on the linker command line?

I do not like this. The change makes binaries linked by the default toolchain on FreeBSD, non-standard compliant. Several hundred bytes is not too high cost to pay for not having issues with all tools still not adapted to GNU hash (and never be).