[clang] add implicit include for Linux/gnu compatibility

[Origami404]

Summary

On GNU/Linux, GCC will automatically include stdc-predefs.h, while clang does
not. That is OK for glibc system because glibc includes this file manually. But
with other libc (e.g. musl), which does not manually include this, clang will
fail but GCC can get it compiled.

In 2017, D34158 try to introduce a new flag called fsystem-include-if-exists
to fix this, but it was reverted. Nearly, D106577 points out that macro
__STDC_ISO_10646__ was indeed needed, which is defined in stdc-predefs.h.
After a discussion, the community reaches a consensus, to make this macro
available with a D34158-like method.

In this patch, we port the patch in D34158 into the current clang codebase. The
change is almost the same with D34158, but we change some tests to suit current
codes:

1. c-index-test now does not accept a -ffreestanding flag, and it should not.

We choose to disable this pre-include action in Objective-C because it will
cause some c-index-test's tests on Objective-C to fail.

2. Some unit tests about parsing and re-parsing will be affected by this change.

To keep such tests passed, we add -ffreestanding flags to them.

3. The new tool, Clang-Scan-Deps, will analyze all header files to find

dependencies. After we add an implicit include, its result will have an extra
stdc-predefs.h. Currently, we choose to add -ffreestanding flags to all the
affected tests for it. But maybe making it ignore the default header will be a
better choice.

Signed-off-by: Tao Liang <[e-mail]>
Co-authored-by: YingChi Long <[e-mail]>

Link: https://reviews.llvm.org/D34158
Link: https://gcc.gnu.org/gcc-4.8/porting_to.html
Link: https://reviews.llvm.org/D106577

[inclyc]

That is OK for glibc system because glibc includes this file manually. But with other libc (e.g. musl), which does not manually include this, clang will fail but GCC can get it compiled.

FWIW currently we cannot build libedit using clang on x86_64-unknown-linux-musl and to fix this issue, they have a workaround here https://patchwork.ozlabs.org/project/buildroot/patch/[e-mail]/ which manually defined __STDC_ISO_10646__

Reproducer:

#include <stdio.h>
// #include <stdc-predef.h>

int main() {
        printf("%d", __STDC_ISO_10646__);
}

GCC accepted this code but clang complained use of undeclared identifier '__STDC_ISO_10646__'

test.c:5:15: error: use of undeclared identifier '__STDC_ISO_10646__'
        printf("%d", __STDC_ISO_10646__);
                     ^
1 error generated.

Note that glibc includes stdc-predefs.h at the source-code level, so this problem cannot be reproduced on the glibc-based systems. (The header stdio.h actually #includes stdc-predefs.h in glibc!)

However if we do not #include stdio.h, different behaviors can be seen on glibc platforms.

int main() {
  return __STDC_ISO_10646__;
}

See https://godbolt.org/z/Gd9Kbn766.

[thesamesam]

Thanks for the CC. Yep, I had the same adventure with libedit and was very confused for a moment until we dug into it and realised: https://bugs.gentoo.org/870001.

[Origami404]

I'm sorry that my patch can not pass the CI test, but I can not reproduce the CI
failure loaclly, including clang-format related problems. On my machine,
ninja check-clang shows:

[0/1] Running the Clang regression tests
llvm-lit: /home/origami/llvm/llvm-project/llvm/utils/lit/lit/llvm/config.py:456: note: using clang: /home/origami/llvm/llvm-project/build-release/bin/clang

Testing Time: 183.65s
  Skipped          :    35
  Unsupported      :  1681
  Passed           : 29839
  Expectedly Failed:    26

And git clang-format HEAD~1 show:

clang-format did not modify any files

I am on Fedora 36, with clang-format 14.0.5 and python 3.10.7. I guess the test
failure maybe lead to incorrect platform setting, so I change the new regression
test in this patch to only run on Linux now by adding -target triple.

I am not familiar with the LLVM pybindings, but I will be more than appreciative
if someone can tell me how to fix the CI failure.

[Origami404]

After digging more deeply into tests, I found that if we decide to make clang include stdc-predef.h or other things, the behavior of many libtooling-related tools (e.g. clang-tidy, clangd, and python binding) will be affected.

For example, clang-tidy/checkers/cppcoreguidelines/macro-usage.cpp tests improper macro usage. If we #include stdc-predef.h implicitly, it will give warnings to macros that are inside this header. Another example is clang-tidy/checkers/portability/restrict-system-includes-disallow.cpp, it will give errors for including stdc-predef.h.

In my opinion, we may need to spend more time discussing how to make __STD_C_* macros available. The way that D34158 take will affect too many tools using libtooling, which may not be a problem back in 2017.

By the way, I have made a patch to pass tests about clang-tidy and python binding by adding more -ffreestanding flags. But I think it's better to re-think the impact it will make instead of just making tests pass. But I still can not figure out why the git clang-format failed on the CI.

[Origami404]

ping :)

[mgorny]

I'm afraid this is above my skills. Maybe @MaskRay?

[aaronpuchert]

This include-if-exists mechanism seems brittle to me. Can we not make it dependent on the triple, i.e. include the file if we're using a libc implementation that's known to provide (and require) this file?

[Origami404]

In D137043#3935129, @aaronpuchert wrote:

This include-if-exists mechanism seems brittle to me.

Do you mean the way that we used to test a file and include it (inserting #if __has_include) is brittle or compilation flags like --include-if-exists themselves are?

Can we not make it dependent on the triple, i.e. include the file if we're using a libc implementation that's known to provide (and require) this file?

I am afarid that without depending on triples, we are not able to distinguish what environment we are in or libc we use. Can you explain more about this?

[aaronpuchert]

In D137043#3935526, @Origami404 wrote:

In D137043#3935129, @aaronpuchert wrote:

This include-if-exists mechanism seems brittle to me.

Do you mean the way that we used to test a file and include it (inserting #if __has_include) is brittle or compilation flags like --include-if-exists themselves are?

The mechanism itself. For example, we might include the file even if it doesn't belong to the C library implementation that we want to use, just because it's there.

Can we not make it dependent on the triple, i.e. include the file if we're using a libc implementation that's known to provide (and require) this file?

I am afarid that without depending on triples, we are not able to distinguish what environment we are in or libc we use. Can you explain more about this?

I was suggesting to have it depend on the triple, not be independent of it.

[Origami404]

Change implements according reviews.

[aaron.ballman]

(Removed @mibintc because she retired, added clang-vendors because this can impact other downstreams.)

This is outside my wheelhouse, so I don't have many technical comments to add. I'm hoping @erichkeane and @jyknight can weigh in given how active they were on the earlier thread.

clang/docs/ReleaseNotes.rst

401–402

clang/lib/Driver/ToolChains/Linux.cpp

640

clang/test/Driver/stdc-predef.c

5

12

19

26–27

34–35

What about Fortran?

[Origami404]

Improved comments and release notes

[Origami404]

clang/test/Driver/stdc-predef.c

34–35

Only files that using clang to compile will be affected. I have updated my expression here.

[erichkeane]

I unfortunately am equally not informed enough to help here. The code itself looks fine to me, but I'm unable to make a value judgement on the change itself.

[MaskRay]

In GCC, a *-linux* triple (this applies to musl as well) gets c_target_objs="${c_target_objs} glibc-c.o" from gcc/config.gcc.
"stdc-predef.h" is defined in glibc-c.cc. There is no musl-c.cc.
Even if musl does provide stdc-predef.h and GCC configured on *-linux-musl does include stdc-predef.h, this may be an unexpected use.
Have you tried bring up this issue on https://www.openwall.com/lists/musl/ ?

[cor3ntin]

We should take a decision on this before the move off Phab. Generally the changes look sensible to me so I'm not sure there are blockers left

clang/docs/ReleaseNotes.rst

- Clang now diagnoses use of invalid or reserved module names in a module
   suppressed for use of reserved names in a system header.
 - ``-Winteger-overflow`` will diagnose overflow in more cases. This fixes
   `Issue 58944 <https://github.com/llvm/llvm-project/issues/58944>`_.
 - Clang has an internal limit of 2GB of preprocessed source code per
   compilation, including source reachable through imported AST files such as
   PCH or modules. When Clang hits this limit, it now produces notes mentioning
   which header and source files are consuming large amounts of this space.
   ``#pragma clang __debug sloc_usage`` can also be used to request this report.
+- Clang will implicitly add ``#include "stdc-predef.h"`` on musl-based systems
+  as GCC does.
 
 Non-comprehensive list of changes in this release
 -------------------------------------------------
 - It's now possible to set the crash diagnostics directory through
   the environment variable ``CLANG_CRASH_DIAGNOSTICS_DIR``.
   The ``-fcrash-diagnostics-dir`` flag takes precedence.
 - When using header modules, inclusion of a private header and violations of
   the `use-declaration rules

clang/lib/Driver/ToolChains/Linux.cpp

void Linux::AddClangSystemIncludeArgs(const ArgList &DriverArgs,
   // system GCCs, but is often used with cross-compiling GCCs, and harmless to
   // add even when Clang is acting as-if it were a system compiler.
   addExternCSystemInclude(DriverArgs, CC1Args, concat(SysRoot, "/include"));
 
   addExternCSystemInclude(DriverArgs, CC1Args, concat(SysRoot, "/usr/include"));
 
   if (!DriverArgs.hasArg(options::OPT_nobuiltininc) && getTriple().isMusl())
     addSystemInclude(DriverArgs, CC1Args, ResourceDirInclude);
+
+  // Add an implicit include for musl-based non-free-standing systems.
+  if (!DriverArgs.hasArg(options::OPT_ffreestanding) && getTriple().isMusl()) {
+    CC1Args.push_back("-include");
+    CC1Args.push_back("stdc-predef.h");
+  }
 }
 
 void Linux::addLibStdCxxIncludePaths(const llvm::opt::ArgList &DriverArgs,
                                      llvm::opt::ArgStringList &CC1Args) const {
   // We need a detected GCC installation on Linux to provide libstdc++'s
   // headers in odd Linuxish places.
   if (!GCCInstallation.isValid())
     return;

clang/test/Driver/Inputs/stdc-predef/usr/include/stdc-predef.h

+#ifndef _STDC_PREDEF_H
+#define _STDC_PREDEF_H  1
+#define DUMMY_STDC_PREDEF 1
+#endif

clang/test/Driver/stdc-predef.c

+// Test that clang preincludes stdc-predef.h, if we are using libc that does not
+// pre-include it, e.g. Musl
+
+// Musl-based systems need this addition include.
+//
+// RUN: %clang %s -### -c 2>&1 \
+// RUN: -target x86_64-unknown-linux-musl \
+// RUN: --sysroot=%S/Inputs/stdc-predef \
+// RUN: | FileCheck -check-prefix CHECK-CPP-FLAG %s
+
+// GNU-based systems do not need this addition include.
+//
+// RUN: %clang %s -### -c -ffreestanding 2>&1 \
+// RUN: -target x86_64-unknown-linux-gnu \
+// RUN: --sysroot=%S/Inputs/stdc-predef \
+// RUN: | FileCheck --implicit-check-not "stdc-predef.h" %s
+
+// Freestanding compilations do not need this addition include.
+//
+// RUN: %clang %s -### -c 2>&1 \
+// RUN: -target x86_64-unknown-linux-musl -ffreestanding \
+// RUN: --sysroot=%S/Inputs/stdc-predef \
+// RUN: | FileCheck --implicit-check-not "stdc-predef.h" %s
+
+// Because this behavior is implemented by adding preprocessor flags in the 
+// driver, disabling preprocessing means the include flag should not appear.
+//
+// RUN: %clang -x cpp-output %s -### -c 2>&1 \
+// RUN: -target x86_64-unknown-linux-musl \
+// RUN: --sysroot=%S/Inputs/stdc-predef \
+// RUN: | FileCheck --implicit-check-not "stdc-predef.h" %s
+
+// This behavior should appear in all files clang accepted that use the C 
+// preprocessor, including C, C++ and Objective-C files, as long as the system 
+// uses musl.
+//
+// RUN: %clang -x objective-c %s -### -c 2>&1 \
+// RUN: -target x86_64-unknown-linux-musl \
+// RUN: --sysroot=%S/Inputs/stdc-predef \
+// RUN: | FileCheck -check-prefix CHECK-CPP-FLAG %s
+
+// If a musl-based system does not have this header, give an error at line 1.
+//
+// RUN: %clang %s -c -Xclang -verify=expected 2>&1 \
+// RUN: -target x86_64-unknown-linux-musl \
+// RUN: --sysroot=%S/Inputs/basic_linux_tree 
+// expected-error@1 {{'stdc-predef.h' file not found}}
+
+// Check if the file is really included by macro.
+//
+// RUN: %clang %s -c -Xclang -verify=ok -DCHECK_DUMMY=1 \
+// RUN: -target x86_64-unknown-linux-musl \
+// RUN: --sysroot=%S/Inputs/stdc-predef 
+// ok-no-diagnostics
+
+// CHECK-CPP-FLAG: "-include" "stdc-predef.h"
+int i;
+#if CHECK_DUMMY
+#if !DUMMY_STDC_PREDEF 
+  #error "Expected macro symbol DUMMY_STDC_PREDEF is not defined."
+#endif
+#endif

clang/test/Driver/stdc-predef.i

+// The automatic preinclude of stdc-predef.h should not occur if
+// the source filename indicates a preprocessed file.
+//
+// RUN: %clang %s -### -c 2>&1 \
+// RUN: --sysroot=%S/Inputs/stdc-predef \
+// RUN: -target x86_64-unknown-linux-musl \
+// RUN: | FileCheck --implicit-check-not "stdc-predef.h" %s
+
+int i;