gcc extension # NNN directive lines

[urnathan]

Summary

Some time back I extended GCC's '# NNN' line marker semantics. Specifically popping to a blank filename will restore the filename to that of the popped-to include.

E.g. restore to line 5 of including file (escaped BOL #'s to avoid git eliding them):

\# 5 "" 2

GCC already had no documentation for this directive extension, I did not add any.

This was useful in developing modules tests, but turned out to also be useful with machine-generated source code. Specifically, a generated include file that itself includes fragments from elsewhere. The ability to pop to the generated include file -- with its full path prefix -- is useful for diagnostic & debug purposes. For instance something like:

Machine generated -- DO NOT EDIT
Type Var = {
\# 7 "encoded.dsl" 1 push to snippet-container
{snippet, of, code}
\# 6 " 2 // Restore to machined-generated source
,
};

// user-code
...
\#include "dsl.h"
...

That pop to "" will restore the filename to '..includepath../dsl.h', which is better than restoring to plain "dsl.h".

[bruno]

Seems like the build failures are unrelated, LGTM with the suggested clang-format changes.

[urnathan]

thanks, I wondered about that format and decided against it because it wasn't a physical line I changed. Now I know :)

[aaron.ballman]

GCC already had no documentation for this directive extension, I did not add any.

Was there a reason why GCC didn't document this? Is there a reason Clang shouldn't document it?

clang/lib/Basic/SourceManager.cpp

227

Might as well add the curly braces here thanks to the comment.

clang/lib/Lex/PPDirectives.cpp

1459

1460

Alternatively: Literal.GetStringLength()

clang/test/Preprocessor/line-directive.c

8–11

Can you add test coverage that demonstrates what happens when you pop to a file but you've not entered anything but the main file? e.g., #line "" 2 as the only line other than the RUN line?

123

[urnathan]

fine, have some documentation :)

[aaron.ballman]

LGTM aside from some minor nits in the fantastic new documentation. Thank you, Nathan!

clang/docs/LanguageExtensions.rst

1 ↗

(On Diff #385794)

Spurious newline

4045 ↗

(On Diff #385794)

4056 ↗

(On Diff #385794)

4070 ↗

(On Diff #385794)

[aaronpuchert]

clang/docs/LanguageExtensions.rst

4058 ↗

(On Diff #385814)

Produces

/home/buildbot/llvm-build-dir/clang-sphinx-docs/llvm/build/tools/clang/docs/LanguageExtensions.rst:4059:Inline literal start-string without end-string.

I guess the problem is that the characters are swapped.

[urnathan]

clang/docs/LanguageExtensions.rst

4058 ↗

(On Diff #385814)

I've pushed the obvious fix, but how did I not see this in my local build? I thought I was building docs -- is there some target I need to name?

[aaronpuchert]

clang/docs/LanguageExtensions.rst

4058 ↗

(On Diff #385814)

I think docs aren't built by default, you might need -DLLVM_ENABLE_SPHINX:BOOL=ON -DLLVM_BUILD_DOCS:BOOL=ON on the CMake command line. Then the target should be docs-clang-html.

Of course you'll need to have Sphinx installed.

clang/lib/Basic/SourceManager.cpp

 /// change the presumed \#include stack.  If it is 1, this is a file entry, if
 /// it is 2 then this is a file exit. FileKind specifies whether this is a
 /// system header or extern C system header.
 void LineTableInfo::AddLineNote(FileID FID, unsigned Offset, unsigned LineNo,
                                 int FilenameID, unsigned EntryExit,
                                 SrcMgr::CharacteristicKind FileKind) {
   std::vector<LineEntry> &Entries = LineEntries[FID];
 
-  // An unspecified FilenameID means use the last filename if available, or the
-  // main source file otherwise.
-  if (FilenameID == -1 && !Entries.empty())
-    FilenameID = Entries.back().FilenameID;
-
   assert((Entries.empty() || Entries.back().FileOffset < Offset) &&
          "Adding line entries out of order!");
 
   unsigned IncludeOffset = 0;
-  if (EntryExit == 0) {  // No #include stack change.
-    IncludeOffset = Entries.empty() ? 0 : Entries.back().IncludeOffset;
-  } else if (EntryExit == 1) {
+  if (EntryExit == 1) {
+    // Push #include
     IncludeOffset = Offset-1;
-  } else if (EntryExit == 2) {
-    assert(!Entries.empty() && Entries.back().IncludeOffset &&
+  } else {
+    const auto *PrevEntry = Entries.empty() ? nullptr : &Entries.back();
+    if (EntryExit == 2) {
+      // Pop #include
+      assert(PrevEntry && PrevEntry->IncludeOffset &&
        "PPDirectives should have caught case when popping empty include stack");
-
-    // Get the include loc of the last entries' include loc as our include loc.
-    IncludeOffset = 0;
-    if (const LineEntry *PrevEntry =
-          FindNearestLineEntry(FID, Entries.back().IncludeOffset))
+      PrevEntry = FindNearestLineEntry(FID, PrevEntry->IncludeOffset);
+    }
+    if (PrevEntry) {
       IncludeOffset = PrevEntry->IncludeOffset;
+      if (FilenameID == -1)
+        // An unspecified FilenameID means use the previous (or containing)
+        // filename if available, or the main source file otherwise.
+        FilenameID = PrevEntry->FilenameID;
+    }
   }
 
   Entries.push_back(LineEntry::get(Offset, LineNo, FilenameID, FileKind,
                                    IncludeOffset));
 }
 
 /// FindNearestLineEntry - Find the line entry nearest to FID that is before
 /// it.  If there is no line entry before Offset in FID, return null.

clang/lib/Lex/PPDirectives.cpp

if (Literal.hadError) {
       DiscardUntilEndOfDirective();
       return;
     }
     if (Literal.Pascal) {
       Diag(StrTok, diag::err_pp_linemarker_invalid_filename);
       DiscardUntilEndOfDirective();
       return;
     }
-    FilenameID = SourceMgr.getLineTableFilenameID(Literal.GetString());
 
     // If a filename was present, read any flags that are present.
     if (ReadLineMarkerFlags(IsFileEntry, IsFileExit, FileKind, *this))
       return;
+
+    // Exiting to an empty string means pop to including file
+    if (!IsFileExit || Literal.GetString().size())
+      FilenameID = SourceMgr.getLineTableFilenameID(Literal.GetString());
   }
 
   // Create a line note with this information.
   SourceMgr.AddLineNote(DigitTok.getLocation(), LineNo, FilenameID, IsFileEntry,
                         IsFileExit, FileKind);
 
   // If the preprocessor has callbacks installed, notify them of the #line
   // change.  This is used so that the line marker comes out in -E mode for

clang/test/Preprocessor/line-directive.c

 // RUN: %clang_cc1 -std=c99 -fsyntax-only -verify -pedantic %s
 // RUN: not %clang_cc1 -E %s 2>&1 | grep 'blonk.c:92:2: error: ABC'
 // RUN: not %clang_cc1 -E %s 2>&1 | grep 'blonk.c:93:2: error: DEF'
+// RUN: not %clang_cc1 -E %s 2>&1 | grep 'line-directive.c:11:2: error: MAIN7'
+// RUN: not %clang_cc1 -E %s 2>&1 | grep 'enter-1:118:2: error: ENTER1'
+// RUN: not %clang_cc1 -E %s 2>&1 | grep 'main:121:2: error: MAIN2'
+
+// a push/pop before any other line control
+# 10 "enter-0" 1
+# 11 "" 2 // pop to main file
+#error MAIN7
+// expected-error@-1{{MAIN7}}
 
 #line 'a'            // expected-error {{#line directive requires a positive integer argument}}
 #line 0              // expected-warning {{#line directive with zero argument is a GNU extension}}
 #line 00             // expected-warning {{#line directive with zero argument is a GNU extension}}
 #line 2147483648     // expected-warning {{C requires #line number to be less than 2147483648, allowed as extension}}
 #line 42             // ok
 #line 42 'a'         // expected-error {{invalid filename for #line directive}}
 #line 42 "foo/bar/baz.h"  // ok

 extern char array2_gnuline[\
 _\
 _LINE__ == 52 ? 1: -1];  /* line marker is location of first _ */
 
 // rdar://11550996
 #line 0 "line-directive.c" // expected-warning {{#line directive with zero argument is a GNU extension}}
 undefined t; // expected-error {{unknown type name 'undefined'}}
 
-
+# 115 "main"
+# 116 "enter-1" 1
+# 117 "enter-2" 1
+# 118 "" 2 // pop to enter-1
+#error ENTER1
+// expected-error@-1{{ENTER1}}
+# 121 "" 2 // pop to "main"
+#error MAIN2
+// expected-error@-1{{MAIN}}