gcc extension # NNN directive lines

[urnathan]

Summary

Some time back I extended GCC's '# NNN' line marker semantics. Specifically popping to a blank filename will restore the filename to that of the popped-to include.

E.g. restore to line 5 of including file (escaped BOL #'s to avoid git eliding them):

\# 5 "" 2

GCC already had no documentation for this directive extension, I did not add any.

This was useful in developing modules tests, but turned out to also be useful with machine-generated source code. Specifically, a generated include file that itself includes fragments from elsewhere. The ability to pop to the generated include file -- with its full path prefix -- is useful for diagnostic & debug purposes. For instance something like:

Machine generated -- DO NOT EDIT
Type Var = {
\# 7 "encoded.dsl" 1 push to snippet-container
{snippet, of, code}
\# 6 " 2 // Restore to machined-generated source
,
};

// user-code
...
\#include "dsl.h"
...

That pop to "" will restore the filename to '..includepath../dsl.h', which is better than restoring to plain "dsl.h".

[bruno]

Seems like the build failures are unrelated, LGTM with the suggested clang-format changes.

[urnathan]

thanks, I wondered about that format and decided against it because it wasn't a physical line I changed. Now I know :)

[aaron.ballman]

GCC already had no documentation for this directive extension, I did not add any.

Was there a reason why GCC didn't document this? Is there a reason Clang shouldn't document it?

clang/lib/Basic/SourceManager.cpp

228

Might as well add the curly braces here thanks to the comment.

clang/lib/Lex/PPDirectives.cpp

1459

1460

Alternatively: Literal.GetStringLength()

clang/test/Preprocessor/line-directive.c

8–11

Can you add test coverage that demonstrates what happens when you pop to a file but you've not entered anything but the main file? e.g., #line "" 2 as the only line other than the RUN line?

126

[urnathan]

fine, have some documentation :)

[aaron.ballman]

LGTM aside from some minor nits in the fantastic new documentation. Thank you, Nathan!

clang/docs/LanguageExtensions.rst

1

Spurious newline

4045

4056

4070

[aaronpuchert]

clang/docs/LanguageExtensions.rst

4059

Produces

/home/buildbot/llvm-build-dir/clang-sphinx-docs/llvm/build/tools/clang/docs/LanguageExtensions.rst:4059:Inline literal start-string without end-string.

I guess the problem is that the characters are swapped.

[urnathan]

clang/docs/LanguageExtensions.rst

4059

I've pushed the obvious fix, but how did I not see this in my local build? I thought I was building docs -- is there some target I need to name?

[aaronpuchert]

clang/docs/LanguageExtensions.rst

4059

I think docs aren't built by default, you might need -DLLVM_ENABLE_SPHINX:BOOL=ON -DLLVM_BUILD_DOCS:BOOL=ON on the CMake command line. Then the target should be docs-clang-html.

Of course you'll need to have Sphinx installed.

clang/docs/LanguageExtensions.rst

+
 =========================
 Clang Language Extensions
 =========================
 
 .. contents::
    :local:
    :depth: 1
 
.. code-block:: c
    #define FINAL_MACRO // warning: FINAL_MACRO is marked final and should not be redefined
    #undef FINAL_MACRO  // warning: FINAL_MACRO is marked final and should not be undefined
 
 This is useful for enforcing system-provided macros that should not be altered
 in user headers or code. This is controlled by ``-Wpedantic-macros``. Final 
 macros will always warn on redefinition, including situations with identical
 bodies and in system headers.
 
+Line Control
+============
+
+Clang supports a extension for source line control, which takes the
+form of a preprocessor directive starting with an unsigned integral
+constant. In addition to the standard ``#line`` directive, this form
+allows control of an include stack and header file type, which is used
+in issuing diagnostics. These lines are emitted in preprocessed
+output.
+
+.. code-block:: c
+  # <line:number> <filename:string> <header-type:numbers>
+
+The filename is optional, and if unspecified indicates no change in
+source filename. The header-type is an optional sequence of magic
+numbers as follows.
+
+* ``1`:` Push the current source file name onto the include stack and
+  enter a new file.
+
+* ``2``: Pop the include stack and return to the specified file. If
+  the filename is ``""``, the name popped from the include stack is
+  used. Otherwise there is no requirement that the specified filename
+  matches the current source when originally pushed.
+
+* ``3``: Enter a system-header region. System headers often contain
+  implementation-specific source that would normally emit a diagnostic.
+
+* ``4``: Enter an implicit `extern "C"' region. This is not required on
+  modern systems where system headers are C++-aware.
+
+At most a single ``1`` or ``2`` can be present, and values must be in
+ascending order.
+
+Examples are:
+
+.. code-block:: c
+
+   # 57 // Advance (or return) to line 57 of the current source file    
+   # 57 "frob" // Set to line 57 of "frob"
+   # 1 "foo.h" 1 // Enter "foo.h" at line 1
+   # 59 "main.c" 2 // Leave current include and return to "main.c"
+   # 1 "/usr/include/stdio.h" 1 3 // Enter a system header
+   # 60 "" 2 // return to "main.c"
+   # 1 "/usr/ancient/header.h" 1 4 // Enter an implicit extern "C" header
+
 Extended Integer Types
 ======================
 
 Clang supports a set of extended integer types under the syntax ``_ExtInt(N)``
 where ``N`` is an integer that specifies the number of bits that are used to represent
 the type, including the sign bit. The keyword ``_ExtInt`` is a type specifier, thus
 it can be used in any place a type can, including as a non-type-template-parameter,
 as the type of a bitfield, and as the underlying type of an enumeration.

clang/lib/Basic/SourceManager.cpp

 /// change the presumed \#include stack.  If it is 1, this is a file entry, if
 /// it is 2 then this is a file exit. FileKind specifies whether this is a
 /// system header or extern C system header.
 void LineTableInfo::AddLineNote(FileID FID, unsigned Offset, unsigned LineNo,
                                 int FilenameID, unsigned EntryExit,
                                 SrcMgr::CharacteristicKind FileKind) {
   std::vector<LineEntry> &Entries = LineEntries[FID];
 
-  // An unspecified FilenameID means use the last filename if available, or the
-  // main source file otherwise.
-  if (FilenameID == -1 && !Entries.empty())
-    FilenameID = Entries.back().FilenameID;
-
   assert((Entries.empty() || Entries.back().FileOffset < Offset) &&
          "Adding line entries out of order!");
 
   unsigned IncludeOffset = 0;
-  if (EntryExit == 0) {  // No #include stack change.
-    IncludeOffset = Entries.empty() ? 0 : Entries.back().IncludeOffset;
-  } else if (EntryExit == 1) {
+  if (EntryExit == 1) {
+    // Push #include
     IncludeOffset = Offset-1;
-  } else if (EntryExit == 2) {
-    assert(!Entries.empty() && Entries.back().IncludeOffset &&
-       "PPDirectives should have caught case when popping empty include stack");
-
-    // Get the include loc of the last entries' include loc as our include loc.
-    IncludeOffset = 0;
-    if (const LineEntry *PrevEntry =
-          FindNearestLineEntry(FID, Entries.back().IncludeOffset))
+  } else {
+    const auto *PrevEntry = Entries.empty() ? nullptr : &Entries.back();
+    if (EntryExit == 2) {
+      // Pop #include
+      assert(PrevEntry && PrevEntry->IncludeOffset &&
+             "PPDirectives should have caught case when popping empty include "
+             "stack");
+      PrevEntry = FindNearestLineEntry(FID, PrevEntry->IncludeOffset);
+    }
+    if (PrevEntry) {
       IncludeOffset = PrevEntry->IncludeOffset;
+      if (FilenameID == -1) {
+        // An unspecified FilenameID means use the previous (or containing)
+        // filename if available, or the main source file otherwise.
+        FilenameID = PrevEntry->FilenameID;
+      }
+    }
   }
 
   Entries.push_back(LineEntry::get(Offset, LineNo, FilenameID, FileKind,
                                    IncludeOffset));
 }
 
 /// FindNearestLineEntry - Find the line entry nearest to FID that is before
 /// it.  If there is no line entry before Offset in FID, return null.

clang/lib/Lex/PPDirectives.cpp

if (Literal.hadError) {
       DiscardUntilEndOfDirective();
       return;
     }
     if (Literal.Pascal) {
       Diag(StrTok, diag::err_pp_linemarker_invalid_filename);
       DiscardUntilEndOfDirective();
       return;
     }
-    FilenameID = SourceMgr.getLineTableFilenameID(Literal.GetString());
 
     // If a filename was present, read any flags that are present.
     if (ReadLineMarkerFlags(IsFileEntry, IsFileExit, FileKind, *this))
       return;
+
+    // Exiting to an empty string means pop to the including file
+    if (!(IsFileExit && Literal.GetString().empty()))
+      FilenameID = SourceMgr.getLineTableFilenameID(Literal.GetString());
   }
 
   // Create a line note with this information.
   SourceMgr.AddLineNote(DigitTok.getLocation(), LineNo, FilenameID, IsFileEntry,
                         IsFileExit, FileKind);
 
   // If the preprocessor has callbacks installed, notify them of the #line
   // change.  This is used so that the line marker comes out in -E mode for

clang/test/Preprocessor/line-directive.c

 // RUN: %clang_cc1 -std=c99 -fsyntax-only -verify -pedantic %s
 // RUN: not %clang_cc1 -E %s 2>&1 | grep 'blonk.c:92:2: error: ABC'
 // RUN: not %clang_cc1 -E %s 2>&1 | grep 'blonk.c:93:2: error: DEF'
+// RUN: not %clang_cc1 -E %s 2>&1 | grep 'line-directive.c:11:2: error: MAIN7'
+// RUN: not %clang_cc1 -E %s 2>&1 | grep 'enter-1:118:2: error: ENTER1'
+// RUN: not %clang_cc1 -E %s 2>&1 | grep 'main:121:2: error: MAIN2'
+
+// expected-error@+1{{cannot pop empty include stack}}
+# 20 "" 2
+
+// a push/pop before any other line control
+# 10 "enter-0" 1
+# 11 "" 2 // pop to main file
+#error MAIN7
+// expected-error@-1{{MAIN7}}
 
 #line 'a'            // expected-error {{#line directive requires a positive integer argument}}
 #line 0              // expected-warning {{#line directive with zero argument is a GNU extension}}
 #line 00             // expected-warning {{#line directive with zero argument is a GNU extension}}
 #line 2147483648     // expected-warning {{C requires #line number to be less than 2147483648, allowed as extension}}
 #line 42             // ok
 #line 42 'a'         // expected-error {{invalid filename for #line directive}}
 #line 42 "foo/bar/baz.h"  // ok

 extern char array2_gnuline[\
 _\
 _LINE__ == 52 ? 1: -1];  /* line marker is location of first _ */
 
 // rdar://11550996
 #line 0 "line-directive.c" // expected-warning {{#line directive with zero argument is a GNU extension}}
 undefined t; // expected-error {{unknown type name 'undefined'}}
 
-
+# 115 "main"
+# 116 "enter-1" 1
+# 117 "enter-2" 1
+# 118 "" 2 // pop to enter-1
+#error ENTER1
+// expected-error@-1{{ENTER1}}
+# 121 "" 2 // pop to "main"
+#error MAIN2
+// expected-error@-1{{MAIN2}}