libclang: Pass Clang install directory to driver via argv[0].

[pcc]

Summary

Various driver features, such as the sysroot path detection for Android
targets, rely on being able to find the Clang install directory (look
for callers of getDriver().getInstalledDir()). However, the install
directory isn't currently being plumbed through to the driver, which is
conventionally done via the argv[0] passed to the Driver constructor.

It looks like D14695 attempted to fix this by adding another API that
allows specifying the argv[0]. However, rather than requiring every
user of libclang to switch to this API for correct behavior, let's have
the other existing APIs work by default, by using the existing logic in
libclang for finding the install directory.

[aaron.ballman]

LGTM with a nit and a question, but can you add a release note when you land to let users know about the changes?

clang/tools/libclang/CIndex.cpp

4017–4019

Just for visual separation.

4022

I suspect this doesn't matter *too* much, but... on Windows, wouldn't this be clang.exe instead? (This isn't new to your patch, so mostly wondering if there's a separate issue we should track.)

[pcc]

clang/tools/libclang/CIndex.cpp

4022

I don't think it matters; the Driver class strips off the bin/clang part before storing it:
https://github.com/llvm/llvm-project/blob/4d18d97b594ccaa3cbd79beb4afef45e4156dc8d/clang/lib/Driver/Driver.cpp#L211

Aside from that, the only user I could find that's reachable from this function is a rather obscure codegen feature:
https://github.com/llvm/llvm-project/blob/fe27495be2040007c7b20844a9371b06156ab405/clang/lib/Frontend/CompilerInvocation.cpp#L4464
and this API doesn't do codegen.

[pcc]

clang/tools/libclang/CIndex.cpp

4022

And of course I link to a line of code that stores the filename somewhere, contradicting what I wrote... that Name field, though, seems to be only used for crash reporting and a help message:

https://github.com/llvm/llvm-project/blob/4d18d97b594ccaa3cbd79beb4afef45e4156dc8d/clang/lib/Driver/Driver.cpp#L1516
https://github.com/llvm/llvm-project/blob/4d18d97b594ccaa3cbd79beb4afef45e4156dc8d/clang/lib/Driver/Driver.cpp#L1801
https://github.com/llvm/llvm-project/blob/4d18d97b594ccaa3cbd79beb4afef45e4156dc8d/clang/lib/Driver/Driver.cpp#L1931

[Hahnfeld]

FYI this commit breaks clang/test/Index/index-file.cu for me:

warning: CUDA version 11.8 is only partially supported [-Wunknown-cuda-version]
warning: CUDA version 11.8 is only partially supported [-Wunknown-cuda-version]
/home/jhahnfel/LLVM/src/clang/test/Index/index-file.cu:8:20: error: CHECK-HOST-NOT: excluded string found in input
// CHECK-HOST-NOT: macro definition=__CUDA_ARCH__
                   ^
<stdin>:6218:48: note: found here
// CHECK: __clang_cuda_runtime_wrapper.h:70:9: macro definition=__CUDA_ARCH__ Extent=[70:9 - 70:27]
                                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It could be related to the warning about my locally installed CUDA version, but __CUDA_ARCH__ should really not be defined when compiling for the host. If you have any idea what's going on here, please let me know - I'm a bit puzzled right now...

[aaron.ballman]

In D146497#4215650, @Hahnfeld wrote:

FYI this commit breaks clang/test/Index/index-file.cu for me:

warning: CUDA version 11.8 is only partially supported [-Wunknown-cuda-version]
warning: CUDA version 11.8 is only partially supported [-Wunknown-cuda-version]
/home/jhahnfel/LLVM/src/clang/test/Index/index-file.cu:8:20: error: CHECK-HOST-NOT: excluded string found in input
// CHECK-HOST-NOT: macro definition=__CUDA_ARCH__
                   ^
<stdin>:6218:48: note: found here
// CHECK: __clang_cuda_runtime_wrapper.h:70:9: macro definition=__CUDA_ARCH__ Extent=[70:9 - 70:27]
                                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It could be related to the warning about my locally installed CUDA version, but __CUDA_ARCH__ should really not be defined when compiling for the host. If you have any idea what's going on here, please let me know - I'm a bit puzzled right now...

This is confirmed on at least this build bot: https://lab.llvm.org/buildbot/#/builders/91/builds/15271

I'll investigate locally and revert if it's not something I can fix forward.

[aaron.ballman]

In D146497#4216197, @aaron.ballman wrote:

In D146497#4215650, @Hahnfeld wrote:

FYI this commit breaks clang/test/Index/index-file.cu for me:

warning: CUDA version 11.8 is only partially supported [-Wunknown-cuda-version]
warning: CUDA version 11.8 is only partially supported [-Wunknown-cuda-version]
/home/jhahnfel/LLVM/src/clang/test/Index/index-file.cu:8:20: error: CHECK-HOST-NOT: excluded string found in input
// CHECK-HOST-NOT: macro definition=__CUDA_ARCH__
                   ^
<stdin>:6218:48: note: found here
// CHECK: __clang_cuda_runtime_wrapper.h:70:9: macro definition=__CUDA_ARCH__ Extent=[70:9 - 70:27]
                                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It could be related to the warning about my locally installed CUDA version, but __CUDA_ARCH__ should really not be defined when compiling for the host. If you have any idea what's going on here, please let me know - I'm a bit puzzled right now...

This is confirmed on at least this build bot: https://lab.llvm.org/buildbot/#/builders/91/builds/15271

I'll investigate locally and revert if it's not something I can fix forward.

The issue doesn't reproduce for me locally on my Windows machine, so it may be something specific to the VE setup. It looks like the macro is being printed on the host side when it's expected to not be present, which doesn't seem like an expected outcome from this change, so I reverted in 18d56880a89ad7d58f8543d148facebd079cef19 to hopefully bring the bot back to green (if the bot stays red for the same reason, I'll reapply the changes).

[Hahnfeld]

In D146497#4216226, @aaron.ballman wrote:

The issue doesn't reproduce for me locally on my Windows machine, so it may be something specific to the VE setup.

FWIW I'm not doing anything specific for VE, just

set(CMAKE_C_COMPILER "clang" CACHE STRING "")
set(CMAKE_CXX_COMPILER "clang++" CACHE STRING "")
set(CMAKE_BUILD_TYPE "RelWithDebInfo" CACHE STRING "")
set(LLVM_APPEND_VC_REV OFF CACHE STRING "")
set(LLVM_ENABLE_ASSERTIONS ON CACHE BOOL "")
set(LLVM_ENABLE_PROJECTS "clang" CACHE STRING "")

set(LLVM_ENABLE_LLD ON CACHE BOOL "")
set(LLVM_LINK_LLVM_DYLIB ON CACHE BOOL "")

in the file I load as the initial-cache (cmake -C).

[aaron.ballman]

In D146497#4216278, @Hahnfeld wrote:

In D146497#4216226, @aaron.ballman wrote:

The issue doesn't reproduce for me locally on my Windows machine, so it may be something specific to the VE setup.

FWIW I'm not doing anything specific for VE, just

set(CMAKE_C_COMPILER "clang" CACHE STRING "")
set(CMAKE_CXX_COMPILER "clang++" CACHE STRING "")
set(CMAKE_BUILD_TYPE "RelWithDebInfo" CACHE STRING "")
set(LLVM_APPEND_VC_REV OFF CACHE STRING "")
set(LLVM_ENABLE_ASSERTIONS ON CACHE BOOL "")
set(LLVM_ENABLE_PROJECTS "clang" CACHE STRING "")

set(LLVM_ENABLE_LLD ON CACHE BOOL "")
set(LLVM_LINK_LLVM_DYLIB ON CACHE BOOL "")

in the file I load as the initial-cache (cmake -C).

Interesting, and I was testing with llvm-lit.py. The revert did get the bot back to green in https://lab.llvm.org/buildbot/#/builders/91/builds/15275

clang/docs/ReleaseNotes.rst

 - Deprecated two functions ``clang_CXIndex_setGlobalOptions`` and
   ``clang_CXIndex_setInvocationEmissionPathOption`` in favor of the new
   function ``clang_createIndexWithOptions`` in order to improve thread safety.
 
 - Added check in ``clang_getFieldDeclBitWidth`` for whether a bit-field
   has an evaluable bit width. Fixes undefined behavior when called on a
   bit-field whose width depends on a template paramter.
 
+- ``clang_parseTranslationUnit`` and ``clang_parseTranslationUnit2`` have been
+  changed to automatically locate the Clang installation directory relative to
+  the location of the libclang binary and use it for system headers installed
+  alongside the Clang installation. It is no longer necessary to manually
+  locate such system headers or use the ``clang_parseTranslationUnit2FullArgv``
+  function for this purpose if libclang has been installed in the default
+  location.
+ 
 Static Analyzer
 ---------------
 - Fix incorrect alignment attribute on the this parameter of certain
   non-complete destructors when using the Microsoft ABI.
   (`#60465 <https://github.com/llvm/llvm-project/issues/60465>`_)
 
 .. _release-notes-sanitizers:
 

clang/include/clang-c/Index.h

 CINDEX_LINKAGE enum CXErrorCode clang_parseTranslationUnit2(
     CXIndex CIdx, const char *source_filename,
     const char *const *command_line_args, int num_command_line_args,
     struct CXUnsavedFile *unsaved_files, unsigned num_unsaved_files,
     unsigned options, CXTranslationUnit *out_TU);
 
 /**
  * Same as clang_parseTranslationUnit2 but requires a full command line
- * for \c command_line_args including argv[0]. This is useful if the standard
- * library paths are relative to the binary.
+ * for \c command_line_args including argv[0].
+ *
+ * This is useful if the driver uses paths relative to the binary and either
+ * you are targeting libclang versions older than Clang 17, or libclang is
+ * installed to a non-standard location. Clang 17 and newer will automatically
+ * use the correct argv[0] if libclang is installed in the lib directory
+ * parallel to the bin directory where the clang binary is installed.
  */
 CINDEX_LINKAGE enum CXErrorCode clang_parseTranslationUnit2FullArgv(
     CXIndex CIdx, const char *source_filename,
     const char *const *command_line_args, int num_command_line_args,
     struct CXUnsavedFile *unsaved_files, unsigned num_unsaved_files,
     unsigned options, CXTranslationUnit *out_TU);
 
 /**

clang/test/Index/record-completion-invocation.c

 // RUN: export LSAN_OPTIONS=detect_leaks=0
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -code-completion-at=%s:10:1 "-remap-file=%s,%S/Inputs/record-parsing-invocation-remap.c" %s
 // RUN: cat %t/libclang-* | FileCheck %s
 
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: env LIBCLANG_DISABLE_CRASH_RECOVERY=1 CINDEXTEST_INVOCATION_EMISSION_PATH=%t not --crash c-index-test -code-completion-at=%s:10:1 "-remap-file=%s,%S/Inputs/record-parsing-invocation-remap.c" %s
 // RUN: cat %t/libclang-* | FileCheck %s
 
-// CHECK: {"toolchain":"{{.*}}","libclang.operation":"complete","libclang.opts":1,"args":["clang","-fno-spell-checking","{{.*}}record-completion-invocation.c","-Xclang","-detailed-preprocessing-record","-fallow-editor-placeholders"],"invocation-args":["-code-completion-at={{.*}}record-completion-invocation.c:10:1"],"unsaved_file_hashes":[{"name":"{{.*}}record-completion-invocation.c","md5":"aee23773de90e665992b48209351d70e"}]}
+// CHECK: {"toolchain":"{{.*}}","libclang.operation":"complete","libclang.opts":1,"args":["{{.*}}bin{{.*}}clang","-fno-spell-checking","{{.*}}record-completion-invocation.c","-Xclang","-detailed-preprocessing-record","-fallow-editor-placeholders"],"invocation-args":["-code-completion-at={{.*}}record-completion-invocation.c:10:1"],"unsaved_file_hashes":[{"name":"{{.*}}record-completion-invocation.c","md5":"aee23773de90e665992b48209351d70e"}]}

clang/test/Index/record-parsing-invocation.c

 // RUN: mkdir %t
 // RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -test-load-source all "-remap-file=%s,%S/Inputs/record-parsing-invocation-remap.c" %s
 // RUN: cat %t/libclang-* | FileCheck --check-prefix=CHECK-UNSAVED %s
 
 #ifndef AVOID_CRASH
 #  pragma clang __debug parser_crash
 #endif
 
-// CHECK: {"toolchain":"{{.*}}","libclang.operation":"parse","libclang.opts":1,"args":["clang","-fno-spell-checking","{{.*}}record-parsing-invocation.c","-Xclang","-detailed-preprocessing-record","-fallow-editor-placeholders"]}
-// CHECK-UNSAVED: {"toolchain":"{{.*}}","libclang.operation":"parse","libclang.opts":1,"args":["clang","-fno-spell-checking","{{.*}}record-parsing-invocation.c","-Xclang","-detailed-preprocessing-record","-fallow-editor-placeholders"],"unsaved_file_hashes":[{"name":"{{.*}}record-parsing-invocation.c","md5":"aee23773de90e665992b48209351d70e"}]}
+// CHECK: {"toolchain":"{{.*}}","libclang.operation":"parse","libclang.opts":1,"args":["{{.*}}bin{{.*}}clang","-fno-spell-checking","{{.*}}record-parsing-invocation.c","-Xclang","-detailed-preprocessing-record","-fallow-editor-placeholders"]}
+// CHECK-UNSAVED: {"toolchain":"{{.*}}","libclang.operation":"parse","libclang.opts":1,"args":["{{.*}}bin{{.*}}clang","-fno-spell-checking","{{.*}}record-parsing-invocation.c","-Xclang","-detailed-preprocessing-record","-fallow-editor-placeholders"],"unsaved_file_hashes":[{"name":"{{.*}}record-parsing-invocation.c","md5":"aee23773de90e665992b48209351d70e"}]}

clang/tools/libclang/CIndex.cpp

 }
 
 enum CXErrorCode clang_parseTranslationUnit2(
     CXIndex CIdx, const char *source_filename,
     const char *const *command_line_args, int num_command_line_args,
     struct CXUnsavedFile *unsaved_files, unsigned num_unsaved_files,
     unsigned options, CXTranslationUnit *out_TU) {
   noteBottomOfStack();
+
+  if (!CIdx)
+    return CXError_InvalidArguments;
+
+  SmallString<64> ClangPath(
+      static_cast<CIndexer *>(CIdx)->getClangToolchainPath());
+  llvm::sys::path::append(ClangPath, "bin");
+  llvm::sys::path::append(ClangPath, "clang");
+
   SmallVector<const char *, 4> Args;
-  Args.push_back("clang");
+  Args.push_back(ClangPath.c_str());
   Args.append(command_line_args, command_line_args + num_command_line_args);
   return clang_parseTranslationUnit2FullArgv(
       CIdx, source_filename, Args.data(), Args.size(), unsaved_files,
       num_unsaved_files, options, out_TU);
 }
 
 enum CXErrorCode clang_parseTranslationUnit2FullArgv(
     CXIndex CIdx, const char *source_filename,