DebugInfo: Add/support new DW_LANG codes for recent C and C++ versions

[dblaikie]

Summary

This may be a breaking change for consumers if they're trying to detect
if code is C or C++, since it'll start using new codes that they may not
be ready to recognize, in which case they may fall back to non-C
handling.

[probinson]

Hmmm I might be inclined to emit 17 and 20 only under not-strict-DWARF for v5, although it makes the logic more complicated. The codes have been allocated but AFAICT the website doesn't have the new codes listed (I looked at http://wiki.dwarfstd.org/index.php/DWARF_Language_Support which doesn't even have all the v5 codes yet). @aprantl do you have an opinion on this? I tend to lean to the pedantic side on this kind of thing, but I'm persuadable.

[dblaikie]

In D138597#3954269, @probinson wrote:

Hmmm I might be inclined to emit 17 and 20 only under not-strict-DWARF for v5, although it makes the logic more complicated. The codes have been allocated but AFAICT the website doesn't have the new codes listed (I looked at http://wiki.dwarfstd.org/index.php/DWARF_Language_Support which doesn't even have all the v5 codes yet).

I think they're up here: https://dwarfstd.org/Languages.php (linked from the dwarfstd.org front page "DWARF V5 Language Codes and Requests")

@aprantl do you have an opinion on this? I tend to lean to the pedantic side on this kind of thing, but I'm persuadable.

Yeah, I've certainly got mixed feelings - maybe we don't pick up these after-release language codes, and instead produce the new language encoding (which separates language from version) as an extension, alongside the old/in-the-dwarfv5-spec-document codes? So that a DWARFv5 consumer that contains only the DWARFv5-spec-document functionality and not the new codes, and a newer consumer can read the new parts.

[probinson]

In D138597#3957356, @dblaikie wrote:

In D138597#3954269, @probinson wrote:

The codes have been allocated but AFAICT the website doesn't have the new codes listed (I looked at http://wiki.dwarfstd.org/index.php/DWARF_Language_Support which doesn't even have all the v5 codes yet).

I think they're up here: https://dwarfstd.org/Languages.php (linked from the dwarfstd.org front page "DWARF V5 Language Codes and Requests")

Oh, thanks. I had it in my head they were on the wiki, but being on the main site is better.

@aprantl do you have an opinion on this? I tend to lean to the pedantic side on this kind of thing, but I'm persuadable.

Yeah, I've certainly got mixed feelings - maybe we don't pick up these after-release language codes, and instead produce the new language encoding (which separates language from version) as an extension, alongside the old/in-the-dwarfv5-spec-document codes? So that a DWARFv5 consumer that contains only the DWARFv5-spec-document functionality and not the new codes, and a newer consumer can read the new parts.

Except the new LNAME codes aren't 1-1 with the old codes and aren't published anywhere. Well, except in the issue resolution, I guess, but that's not definitive (subject to change until DWARF 6 is published). I'm not sure which way to go here.

[dblaikie]

@aprantl do you have an opinion on this? I tend to lean to the pedantic side on this kind of thing, but I'm persuadable.

Yeah, I've certainly got mixed feelings - maybe we don't pick up these after-release language codes, and instead produce the new language encoding (which separates language from version) as an extension, alongside the old/in-the-dwarfv5-spec-document codes? So that a DWARFv5 consumer that contains only the DWARFv5-spec-document functionality and not the new codes, and a newer consumer can read the new parts.

Except the new LNAME codes aren't 1-1 with the old codes

I'm not sure that'd be a problem? They're in different attributes, etc, such that they wouldn't be read by an old consumer without explicit support for them.

and aren't published anywhere.

They're published on the website too: https://dwarfstd.org/LanguagesV6.php

[probinson]

They're published on the website too: https://dwarfstd.org/LanguagesV6.php

(sigh) sorry, distracted by other things today.

Yeah, using the v6 codes with the v6 attributes for the new ones WFM.

[aprantl]

@aprantl do you have an opinion on this? I tend to lean to the pedantic side on this kind of thing, but I'm persuadable.

As long as LLDB can deal with it I'm fine either way.
Emitting the separated DWARF 6 attribute as an extension sounds fine to me.

llvm/lib/IR/DIBuilder.cpp

159 ↗

(On Diff #477570)

Should we define a DW_LANG_HI_DWARF in the .def file?

[dblaikie]

Add llvm testing (but I'm going to rework this a lot, so this is just a snapshot for history)

[dblaikie]

Don't know that I've got it in me right now to do the more invasive change of updating the LLVM DebugInfo IR metadata to carry the DWARFv6 style language+version stuff (perhaps especially if that's going to go alongside the DWARFv5 and earlier pure language encoding)

I could do the minimum and fix the C version case to use the DWARFv5-shipped C11 encoding. And perhaps add the post-v5 backported language encodings to LLVM so at least llvm-dwarfdump can detect/render/handle them correctly for name printing, etc.

Or I could maybe do something in between - where the post-v5 backported language codes could be used in the IR and then LLVM could detect those and remap them to v5-shipped DW_AT_language + v6-not-ready-yet (using an extension form? Using the form that's not shipped in v6 yet?) attributes?

Seems like it's probably not worth emitting the v6 stuff until it's baked. And not worth emitting the post-v5 backported attributes due to risk of breakage? So maybe just do the minimum?

[probinson]

One not-yet-asked question, does gcc produce the C++17/20 codes? If so, does fstrict-dwarf affect that? The major consumers of this would be lldb (which we control) and gdb, so following gcc's lead here would seem appropriate. I can accept that listing the codes on the website "counts" as in-spec for fstrict-dwarf purposes.

Deferring the v6 language attributes seems totally fair.

[aprantl]

Agreeing with @probinson

[dblaikie]

Rebase on top of committed LLVM changes

[dblaikie]

Don't use the since-v5-published codes

[dblaikie]

In D138597#3966638, @probinson wrote:

One not-yet-asked question, does gcc produce the C++17/20 codes? If so, does fstrict-dwarf affect that?

Looks like GCC uses C++17 for both 17 and 20: https://gcc.gnu.org/git/?p=gcc.git;a=commit;f=gcc/testsuite/g%2B%2B.dg/debug/dwarf2/lang-cpp17.C;h=a90606ab979762ce4630777c2fce5c921d0a2b96 (apparently it used to use just C_plus_plus for 20: https://gcc.gnu.org/git/?p=gcc.git;a=commit;f=gcc/testsuite/g%2B%2B.dg/debug/dwarf2/lang-cpp17.C;h=a90606ab979762ce4630777c2fce5c921d0a2b96 )

The major consumers of this would be lldb (which we control) and gdb, so following gcc's lead here would seem appropriate. I can accept that listing the codes on the website "counts" as in-spec for fstrict-dwarf purposes.

Yeah, I'm leaning towards just not implementing the v5 bonus codes - they're good for new languages, but for existing languages the risk of regression seems not worth the benefit.

I left the test cases in - easy to tweak them when we decide how to render this - and left some minor refactoring in that I could pull out/commit separately.

I still haven't heard from the internal user that seemed pretty adamant that expressing these newer versions was important... I can't really picture (relative to all the other debug info incompletenessess) why it would be especially important.

llvm/lib/IR/DIBuilder.cpp

159 ↗

(On Diff #477570)

Maybe? I'm not sure - if it has to explicitly write the constant or another enum value in there, I think it's as liable to get missed when updating as this code, perhaps? Open to ideas.

[aprantl]

This minimal patch LGTM.

llvm/lib/IR/DIBuilder.cpp

159 ↗

(On Diff #477570)

I guess this isn't important enough to spend too much time designing a solution for it. Let's leave it as is.

[probinson]

LGTM. I agree with the commentary in the test.

[probinson]

Oh, right, PS4 defaults to C99. It's okay with me if you make those two unsupported for PS4.

[dblaikie]

(just a side note I discovered while looking into this: seems someone else had this idea before & came to similar conclusions: https://reviews.llvm.org/D104118#2840490 )

clang/lib/CodeGen/CGDebugInfo.cpp

void CGDebugInfo::CreateCompileUnit() {
   // accepting it as an argument? This just causes the main file name to
   // mismatch with source locations and create extra lexical scopes or
   // mismatched debug info (a CU with a DW_AT_file of "-", because that's what
   // the driver passed, but functions/other things have DW_AT_file of "<stdin>"
   // because that's what the SourceManager says)
 
   // Get absolute path name.
   SourceManager &SM = CGM.getContext().getSourceManager();
-  std::string MainFileName = CGM.getCodeGenOpts().MainFileName;
+  auto &CGO = CGM.getCodeGenOpts();
+  std::string MainFileName = CGO.MainFileName;
   if (MainFileName.empty())
     MainFileName = "<stdin>";
 
   // The main file name provided via the "-main-file-name" option contains just
   // the file name itself with no path information. This file name may have had
   // a relative path, so we look into the actual file entry for the main
   // file to determine the real absolute path for the file.
   std::string MainFileDir;
if (Optional<FileEntryRef> MainFile =
     CSKind = computeChecksum(SM.getMainFileID(), Checksum);
   }
 
   llvm::dwarf::SourceLanguage LangTag;
   const LangOptions &LO = CGM.getLangOpts();
   if (LO.CPlusPlus) {
     if (LO.ObjC)
       LangTag = llvm::dwarf::DW_LANG_ObjC_plus_plus;
-    else if (LO.CPlusPlus14 && (!CGM.getCodeGenOpts().DebugStrictDwarf ||
-                                CGM.getCodeGenOpts().DwarfVersion >= 5))
+    else if (CGO.DebugStrictDwarf && CGO.DwarfVersion < 5)
+      LangTag = llvm::dwarf::DW_LANG_C_plus_plus;
+    else if (LO.CPlusPlus14)
       LangTag = llvm::dwarf::DW_LANG_C_plus_plus_14;
-    else if (LO.CPlusPlus11 && (!CGM.getCodeGenOpts().DebugStrictDwarf ||
-                                CGM.getCodeGenOpts().DwarfVersion >= 5))
+    else if (LO.CPlusPlus11)
       LangTag = llvm::dwarf::DW_LANG_C_plus_plus_11;
     else
       LangTag = llvm::dwarf::DW_LANG_C_plus_plus;
   } else if (LO.ObjC) {
     LangTag = llvm::dwarf::DW_LANG_ObjC;
   } else if (LO.OpenCL && (!CGM.getCodeGenOpts().DebugStrictDwarf ||
                            CGM.getCodeGenOpts().DwarfVersion >= 5)) {
     LangTag = llvm::dwarf::DW_LANG_OpenCL;
   } else if (LO.RenderScript) {
     LangTag = llvm::dwarf::DW_LANG_GOOGLE_RenderScript;
+  } else if (LO.C11) {
+    LangTag = llvm::dwarf::DW_LANG_C11;
   } else if (LO.C99) {
     LangTag = llvm::dwarf::DW_LANG_C99;
   } else {
     LangTag = llvm::dwarf::DW_LANG_C89;
   }
 
   std::string Producer = getClangFullVersion();
 

clang/test/CodeGen/debug-info-preprocessed-file.i

 # 1 "/foo/bar/preprocessed-input.c"
 # 1 "<built-in>" 1
 # 1 "<built-in>" 3
 # 318 "<built-in>" 3
 # 1 "<command line>" 1
 # 1 "<built-in>" 2
 # 1 "preprocessed-input.c" 2
 
 // RUN: %clang -g -c -S -emit-llvm -o - %s | FileCheck %s
-// CHECK: !DICompileUnit(language: DW_LANG_C99, file: ![[FILE:[0-9]+]] 
+// CHECK: !DICompileUnit(language: DW_LANG_C{{.*}}, file: ![[FILE:[0-9]+]]
 // CHECK: ![[FILE]] = !DIFile(filename: "/foo/bar/preprocessed-input.c"

clang/test/CodeGen/debug-info-programming-language.c

+// RUN: %clang_cc1 -dwarf-version=3 -emit-llvm -triple %itanium_abi_triple %s -o - \
+// RUN:   -x c -std=c11 -O0 -disable-llvm-passes -debug-info-kind=limited \
+// RUN:   | FileCheck --check-prefix=CHECK-C11 %s
+// RUN: %clang_cc1 -dwarf-version=3 -emit-llvm -triple %itanium_abi_triple %s -o - \
+// RUN:   -x c -std=c17 -O0 -disable-llvm-passes -debug-info-kind=limited \
+// RUN:   | FileCheck --check-prefix=CHECK-C17 %s
+
+// CHECK-C11: !DICompileUnit(language: DW_LANG_C11
+// Update this check once support for DW_LANG_C17 is broadly supported/known in
+// consumers. Maybe we'll skip this and go to the DWARFv6 language+version
+// encoding that avoids the risk of regression when describing a language
+// version newer than what the consumer is aware of.
+// CHECK-C17: !DICompileUnit(language: DW_LANG_C11
+
+void f1(void) { }

clang/test/CodeGenCXX/debug-info-programming-language.cpp

 // RUN: %clang_cc1 -dwarf-version=5 -emit-llvm -triple %itanium_abi_triple %s -o - \
 // RUN:   -x c++ -std=c++14 -O0 -disable-llvm-passes -debug-info-kind=limited \
 // RUN:   | FileCheck --check-prefix=CHECK-CPP14 %s
 // RUN: %clang_cc1 -dwarf-version=3 -emit-llvm -triple %itanium_abi_triple %s -o - \
 // RUN:   -x c++ -std=c++14 -O0 -disable-llvm-passes -debug-info-kind=limited \
 // RUN:   | FileCheck --check-prefix=CHECK-CPP14 %s
+// RUN: %clang_cc1 -dwarf-version=3 -emit-llvm -triple %itanium_abi_triple %s -o - \
+// RUN:   -x c++ -std=c++17 -O0 -disable-llvm-passes -debug-info-kind=limited \
+// RUN:   | FileCheck --check-prefix=CHECK-CPP17 %s
+// RUN: %clang_cc1 -dwarf-version=3 -emit-llvm -triple %itanium_abi_triple %s -o - \
+// RUN:   -x c++ -std=c++20 -O0 -disable-llvm-passes -debug-info-kind=limited \
+// RUN:   | FileCheck --check-prefix=CHECK-CPP20 %s
 // RUN: %clang_cc1 -dwarf-version=3 -gstrict-dwarf -emit-llvm -triple %itanium_abi_triple %s -o - \
 // RUN:   -x c++ -std=c++14 -O0 -disable-llvm-passes -debug-info-kind=limited | FileCheck %s
 // RUN: %clang_cc1 -dwarf-version=5 -gstrict-dwarf -emit-llvm -triple %itanium_abi_triple %s -o - \
 // RUN:   -x c++ -std=c++14 -O0 -disable-llvm-passes -debug-info-kind=limited \
 // RUN:   | FileCheck --check-prefix=CHECK-CPP14 %s
 
 int main() {
   return 0;
 }
 
+// Update these tests once support for DW_LANG_C_plus_plus_17/20 is added - it's
+// a complicated tradeoff. The language codes are already published/blessed by
+// the DWARF committee, but haven't been released in a published standard yet,
+// so consumers might not be ready for these codes & could regress functionality
+// (because they wouldn't be able to identify that the language was C++). The
+// DWARFv6 language encoding, separating language from language version, would
+// remove this problem/not require new codes for new language versions and make
+// it possible to identify the base language irrespective of the version.
 // CHECK-CPP14: distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14,
+// CHECK-CPP17: distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14,
+// CHECK-CPP20: distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14,
 // CHECK: distinct !DICompileUnit(language: DW_LANG_C_plus_plus,

clang/test/PCH/debug-info-pch-container-path.c

 // REQUIRES: asserts
 
 // Modules:
 // RUN: rm -rf %t
 // RUN: mkdir %t
 // RUN: cd %t
 
 // RUN: %clang_cc1 -fmodule-format=obj -emit-pch                \
 // RUN:     -triple %itanium_abi_triple                         \
 // RUN:     -fdebug-prefix-map=%t=BUILD                         \
 // RUN:     -fdebug-prefix-map=%S=SOURCE                        \
 // RUN:     -o %t/prefix.ll %S/debug-info-limited-struct.h      \
 // RUN:   -mllvm -debug-only=pchcontainer &>%t-container.ll
 // RUN: cat %t-container.ll | FileCheck %s
 
 // CHECK: distinct !DICompileUnit(
-// CHECK-SAME:                    language: DW_LANG_C99,
+// CHECK-SAME:                    language: DW_LANG_C{{[^,]*}},
 // CHECK-SAME:                    file: ![[FILE:[0-9]+]],
 // CHECK: ![[FILE]] = !DIFile(
 // CHECK-SAME:                filename: "SOURCE/debug-info-limited-struct.h",
 // CHECK-SAME:                directory: "BUILD"