Yeah, I think the only reason the isl code can get away with it is that it's related to the usage of Autoconf in that directory. llvm/cmake/config.guess doesn't have that exemption because it's being invoked from a cmake module, not an Autoconf script.

I'm pretty sure we're using this old config.guess on purpose to avoid GPL3 code.

The code changes look logical and correct to me.

Forcing retest again.

Update altivec changes after fd739804e0591468762eb87488a497a3f7d4afb0.

Add missing OpenMP TLS test for powerpcle.

• Adjust the GNU LD settings for 32 bit powerpc to more closely match reality.
• Add linker emulation test for powerpcle, as well as adding testing for the FreeBSD emulations.

Fully fix the unit test.

Re-uploading patch for part 5 now that I have the dependency tree fixed.

Redoing patch.

Add q66 to reviewers list for the targeting bits relevant to Void ppcle.

• Fix LLVM object handling unit test.

• Fix bug in clang/test/Driver/linux-header-search.cpp -- The powerpc64le test was being done with -m32 accidentally.
• Update llvm/unittests/ADT/TripleTest.cpp for powerpcle.
• Update gcc driver bits to reflect use of powerpcle in void-ppc.

Fix merge base.

• Address review comment from MaskRay.
• Incorporate changes from the Void powerpcle patchset.

Trying again..

Splitting up into multiple commits as per MaskRay's suggestion.

In D92445#2428563, @sfertile wrote:

On FreeBSD, the main use of this will be on the new powerpc64le arch, where we need to build a 32-bit LE bootloader for use with pseries. (it is easier to retarget LLVM than make a cross-endian bootloader, as it would involve rewriting filesystem code etc.)

Excuse my ignorance, but what are there technical limitations preventing writing n 64-bit LE boot loader and avoid having a 32-bit LE target all-together?

Regarding things like Altivec and VSX:
Altivec should be just fine to run in 32-bit LE.
I am undecided as to whether VSX should be banned or not. However that goes, it should be identical in powerpc64 -m32 and powerpc64le -m32.

In D92445#2427537, @Bdragon28 wrote:

In D92445#2427438, @nemanjai wrote:

This seems problematic to me for a few reasons:

1. There is no 32-bit toolchains or libraries for little endian Linux systems
2. There is no support in the ELFv2 ABI for 32-bit object mode and there may be a number of places we assume that little endian systems use ELFv2 ABI
3. There are a number of places that make the assumption that little endian systems must be 64-bit so there's no checking for 32-bit mode (which will likely result in 64-bit specific code gen)

Since this seems to be for a very specific use case, can we perhaps severely restrict the support for 32-bit little endian mode (perhaps BSD only, no Altivec/VSX, etc.)?

Technically, 32 bit PowerPC systems are neither ELFv1 nor ELFv2, they are Power Architecture 32-bit ABI (which has various extensions) that does not have the complexities that plagued ELFv1 and is a relatively "normal" ABI.

I have not seen any places that this is problematic that I am not already addressing with this patch.

In D92445#2427439, @MaskRay wrote:

This patch should be split. I suggest that you create 4 patches.

• llvm: triple change
• llvm: llvm/Object/ELFObjectFile.h llvm-readobj llvm-objdump
• clang
• lld

In D92445#2427438, @nemanjai wrote:

This seems problematic to me for a few reasons:

1. There is no 32-bit toolchains or libraries for little endian Linux systems
2. There is no support in the ELFv2 ABI for 32-bit object mode and there may be a number of places we assume that little endian systems use ELFv2 ABI
3. There are a number of places that make the assumption that little endian systems must be 64-bit so there's no checking for 32-bit mode (which will likely result in 64-bit specific code gen)

Since this seems to be for a very specific use case, can we perhaps severely restrict the support for 32-bit little endian mode (perhaps BSD only, no Altivec/VSX, etc.)?

On FreeBSD, the main use of this will be on the new powerpc64le arch, where we need to build a 32-bit LE bootloader for use with pseries. (it is easier to retarget LLVM than make a cross-endian bootloader, as it would involve rewriting filesystem code etc.)

OK, patch updated with the typo fix.

Fixing typo.

In D79916#2279918, @jrtc27 wrote:

In D79916#2279901, @Bdragon28 wrote:

In D79916#2279884, @jrtc27 wrote:

But also you really should not get warnings for unused static functions in included headers, only ones defined in the C source file itself. We'd have countless warnings in the kernel across all architectures otherwise.

I agree. But that's what it is doing when using always_inline in combination with -Wunused-function.

There is currently no real usage of always_inline in system headers though, so maybe I'm just the first to complain about it?

We use them in CheriBSD and have no such issues that I've ever noticed. When was the last time you checked (and what compiler)?

In D79916#2279884, @jrtc27 wrote:

But also you really should not get warnings for unused static functions in included headers, only ones defined in the C source file itself. We'd have countless warnings in the kernel across all architectures otherwise.

In D79916#2279866, @jrtc27 wrote:

In D79916#2279863, @Bdragon28 wrote:

In D79916#2279816, @jrtc27 wrote:

In D79916#2279812, @Bdragon28 wrote:

In D79916#2279045, @jrtc27 wrote:

This has significantly regressed FreeBSD's performance with the new version of Clang. It seems Clang does not inline functions at -O1, unlike GCC, and since FreeBSD currently compiles its kernel with -O whenever debug symbols are enabled[1] (which, of course, is almost always true), this results in all its static inline helper functions not being inlined at all, a pattern that is common in the kernel, used for things like get_curthread and the atomics implementations.

[1] This is a dubious decision made in r140400 in 2005 to provide "truer debugger stack traces" (well, before then there was ping-ponging between -O and -O2 based on concerns around correctness vs performance, but amd64 is an exception that has always used -O2 since r127180 it seems). Given that GCC will inline at -O, at least these days, the motivation seems to no longer exist, and compiling a kernel at anything other than -O2 (or maybe -O3) seems like a silly thing to do, but nevertheless it's what is currently done.

Cc: @dim @trasz

This is actually SUCH a bad idea that a kernel built with -O will *not work at all* on 32 bit powerpc platforms (presumably due to allocating stack frames in the middle of assembly fragments in the memory management that are supposed to be inlined at all times.) I had to hack kern.pre.mk to rquest -O2 at all times just to get a functioning kernel.

Well, -O0, -O1, -O2 and -O should all produce working kernels, and any cases where they don't are compiler bugs (or kernel bugs if they rely on UB) that should be fixed, not worked around by tweaking the compiler flags in a fragile way until you get the behaviour relied on. Correctness and performance are very different issues here.

As an example:

static __inline void
mtsrin(vm_offset_t va, register_t value)
{

        __asm __volatile ("mtsrin %0,%1; isync" :: "r"(value), "r"(va));
}

This code is used in the mmu when bootstrapping the cpu like so:

for (i = 0; i < 16; i++)
        mtsrin(i << ADDR_SR_SHFT, kernel_pmap->pm_sr[i]);
powerpc_sync();

sdr = (u_int)moea_pteg_table | (moea_pteg_mask >> 10);
__asm __volatile("mtsdr1 %0" :: "r"(sdr));
isync();

tlbia();

During the loop there, we are in the middle of programming the MMU segment registers in real mode, and is supposed to be doing all work out of registers. (and powerpc_sync() and isync() should be expanded to their single assembly instruction, not a function call. The whole point of calling those is that we are in an inconsistent hardware state and need to sync up before continuing execution)

If there isn't a way to force inlining, we will have to change to using preprocessor macros in cpufunc.h.

There is, it's called __attribute__((always_inline)) and supported by both GCC and Clang. But at -O0 you'll still have register allocation to deal with, so really that code is just fundamentally broken and should not be written in C. There is no way for you to guarantee stack spills are not used, it's way out of scope for C.

In D79916#2279863, @Bdragon28 wrote:

In D79916#2279816, @jrtc27 wrote:

In D79916#2279812, @Bdragon28 wrote:

In D79916#2279045, @jrtc27 wrote:

This has significantly regressed FreeBSD's performance with the new version of Clang. It seems Clang does not inline functions at -O1, unlike GCC, and since FreeBSD currently compiles its kernel with -O whenever debug symbols are enabled[1] (which, of course, is almost always true), this results in all its static inline helper functions not being inlined at all, a pattern that is common in the kernel, used for things like get_curthread and the atomics implementations.

[1] This is a dubious decision made in r140400 in 2005 to provide "truer debugger stack traces" (well, before then there was ping-ponging between -O and -O2 based on concerns around correctness vs performance, but amd64 is an exception that has always used -O2 since r127180 it seems). Given that GCC will inline at -O, at least these days, the motivation seems to no longer exist, and compiling a kernel at anything other than -O2 (or maybe -O3) seems like a silly thing to do, but nevertheless it's what is currently done.

Cc: @dim @trasz

This is actually SUCH a bad idea that a kernel built with -O will *not work at all* on 32 bit powerpc platforms (presumably due to allocating stack frames in the middle of assembly fragments in the memory management that are supposed to be inlined at all times.) I had to hack kern.pre.mk to rquest -O2 at all times just to get a functioning kernel.

Well, -O0, -O1, -O2 and -O should all produce working kernels, and any cases where they don't are compiler bugs (or kernel bugs if they rely on UB) that should be fixed, not worked around by tweaking the compiler flags in a fragile way until you get the behaviour relied on. Correctness and performance are very different issues here.

As an example:

static __inline void
mtsrin(vm_offset_t va, register_t value)
{

        __asm __volatile ("mtsrin %0,%1; isync" :: "r"(value), "r"(va));
}

This code is used in the mmu when bootstrapping the cpu like so:

for (i = 0; i < 16; i++)
        mtsrin(i << ADDR_SR_SHFT, kernel_pmap->pm_sr[i]);
powerpc_sync();

sdr = (u_int)moea_pteg_table | (moea_pteg_mask >> 10);
__asm __volatile("mtsdr1 %0" :: "r"(sdr));
isync();

tlbia();

During the loop there, we are in the middle of programming the MMU segment registers in real mode, and is supposed to be doing all work out of registers. (and powerpc_sync() and isync() should be expanded to their single assembly instruction, not a function call. The whole point of calling those is that we are in an inconsistent hardware state and need to sync up before continuing execution)

If there isn't a way to force inlining, we will have to change to using preprocessor macros in cpufunc.h.

In D79916#2279816, @jrtc27 wrote:

In D79916#2279812, @Bdragon28 wrote:

In D79916#2279045, @jrtc27 wrote:

This has significantly regressed FreeBSD's performance with the new version of Clang. It seems Clang does not inline functions at -O1, unlike GCC, and since FreeBSD currently compiles its kernel with -O whenever debug symbols are enabled[1] (which, of course, is almost always true), this results in all its static inline helper functions not being inlined at all, a pattern that is common in the kernel, used for things like get_curthread and the atomics implementations.

[1] This is a dubious decision made in r140400 in 2005 to provide "truer debugger stack traces" (well, before then there was ping-ponging between -O and -O2 based on concerns around correctness vs performance, but amd64 is an exception that has always used -O2 since r127180 it seems). Given that GCC will inline at -O, at least these days, the motivation seems to no longer exist, and compiling a kernel at anything other than -O2 (or maybe -O3) seems like a silly thing to do, but nevertheless it's what is currently done.

Cc: @dim @trasz

This is actually SUCH a bad idea that a kernel built with -O will *not work at all* on 32 bit powerpc platforms (presumably due to allocating stack frames in the middle of assembly fragments in the memory management that are supposed to be inlined at all times.) I had to hack kern.pre.mk to rquest -O2 at all times just to get a functioning kernel.

Well, -O0, -O1, -O2 and -O should all produce working kernels, and any cases where they don't are compiler bugs (or kernel bugs if they rely on UB) that should be fixed, not worked around by tweaking the compiler flags in a fragile way until you get the behaviour relied on. Correctness and performance are very different issues here.

In D79916#2279045, @jrtc27 wrote:

This has significantly regressed FreeBSD's performance with the new version of Clang. It seems Clang does not inline functions at -O1, unlike GCC, and since FreeBSD currently compiles its kernel with -O whenever debug symbols are enabled[1] (which, of course, is almost always true), this results in all its static inline helper functions not being inlined at all, a pattern that is common in the kernel, used for things like get_curthread and the atomics implementations.

[1] This is a dubious decision made in r140400 in 2005 to provide "truer debugger stack traces" (well, before then there was ping-ponging between -O and -O2 based on concerns around correctness vs performance, but amd64 is an exception that has always used -O2 since r127180 it seems). Given that GCC will inline at -O, at least these days, the motivation seems to no longer exist, and compiling a kernel at anything other than -O2 (or maybe -O3) seems like a silly thing to do, but nevertheless it's what is currently done.

Cc: @dim @trasz

That's fair. Will just use a patch on the FreeBSD side and revisit after 11.0.0 is released. Thanks.

Any chance of a backport to 11?

Use correct target for FreeBSD driver test.

Add some tests for the new target.

Here's the LLD_REPRODUCE using the cross toolchain (since I had it already and I hit the same thing on the backport, this one is probably the best one to look at)

OK, I reinterpreted 9569a1472ee7fee37f7f991d34634c5d8d1f3559 (as a prereq to reduce the churn) and this patch in the context of llvm10 (mainly re-uppercasing function names so that stuff applies and converting the MCSection::NonUniqueID bit back into ~0) so I can run it in-tree for a FreeBSD buildworld, and I hit the exact same failure, so it looks like there's an actual problem here.

I'm pretty sure I'm just tripping over c++ weirdness related to trying to use an external llvm HEAD compiler as a toolchain for the same platform. I will retry from my cross build machine.

Right, which is why it was so surprising to see it again in HEAD. I'm still trying to get a reproduce.

This fixes my problem encountered trying to build kyua and atf for lld10-native FreeBSD ppc32.

Still working on testing this one.

From a FreeBSD standpoint, I am very happy with this.

AWESOME, this fixed the clang crash I was seeing with lld10 as well! I think it's possible this might be the last crashing bug blocking ppc32 lld migration.

This patch will be needed on FreeBSD to fix some crashes for an lld10-linked FreeBSD powerpc32 world, such as a crash running /usr/bin/objdump -D.

Looks correct to me, works great, and fixes the last lld-related problem I am aware of currently on FreeBSD powerpc32.

I am now able to load pf.ko on FreeBSD powerpc32 without it crashing.

This is necessary to run large programs like clang on FreeBSD powerpc32 when built with lld, because of REL24 branches in the crt startup code (done on purpose for object reusability reasons -- the csu objects need to work on both non-pic and pic, so they just use a plain bl and rely on the link editor to do a thunk if necessary.)

Working great!

This version looks very promising so far. Will do a full freebsd buildworld test / boot on G4. I expect this version to have fixed the issue, but will verify.

Excellent! This looks like it fixed the FreeBSD putchar problem for me. I will test this with a full powerpc32 buildworld and report back.

While I ran into further problems on ppc32, I think it's just because there are further problems on ppc32 (unrelated to this one). At the very least this gets FreeBSD ppc32/lld10 as far as the rescue shell, which is great progress!

For anyone who is curious about this, copy relocations are especially needed on powerpc32, where they are used to implement global data (such as sharing a pointer between a .so and the main program) in situations where the main program's copy lives in .bss.

I am definitely in favor of this change, as the defaulting to PIC has been causing headaches in the FreeBSD kernel.

On the FreeBSD kernel side, I put some time into working out the missing parts on ppc32 and slightly extending the kernel linker to support secure-plt PIC kernel modules. Working on that over at https://reviews.freebsd.org/D22608.

This doesn't seem quite sufficient for FreeBSD kernel modules. It's emitting R_PPC_REL24 instead of R_PPC_ADDR16_HA/R_PPC_ADDR16_LO pairs for stuff like memset and memcpy, which are unusable because we're running modules in KVA and the kernel text in the 32 bit DMAP. Still need to figure out why these are being emitted this way when using freestanding.

It turns out I need the proper fix after all in my local tree, because lately I have been working on getting a working Petitboot loader binary, and that means I'm technically cross compiling code for ppc64le Linux. So yeah, it would be very nice to get this in.