[Preprocessor] Correct internal token parsing of newline characters in CRLF

[erichkeane]

Summary

Discovered due to a goofy git setup, the test system-headerline-directive.c (and a few others)
failed because the token-consumption will consume only the '\r' in CRLF, making the preprocessor's
printed value give the wrong line number when returning from an include. For example:

(line 1):#include <noline.h>\r\n

The "file exit" code causes the printer to try to print the 'returned to the main file' line. It looks up
what the current line number is. However, since the current 'token' is the '\n' (since only the \r was
consumed), it will give the line number as '1", not '2'. This results in a few failed tests, but more importantly,
results in error messages being incorrect when compiling a previously preprocessed file.

[erichkeane]

test/Frontend/system-header-line-directive-ms-lineendings.c

1 ↗

(On Diff #112436)

Note: this whole file is dos line endings (preserved by the above git attributes, and subversion eol-style when i go to commit it). It doesn't show in phab however for some reaosn.

[rnk]

lib/Lex/Lexer.cpp

3076–3077 ↗

(On Diff #112436)

Should we only do this in the \r case? If I understand correctly, we're basically saying, if this is a CR, and the next byte is an LF, advance one more and do the pre-processor stuff.

[erichkeane]

lib/Lex/Lexer.cpp

3076–3077 ↗

(On Diff #112436)

That is exactly what we're doing.

I debated that personally, and am a bit on the fence. It seems a number of places like to treat a '\r\n' and a '\n\r' as the same thing, though it seems a little foolish to me. If you fall toward that opinion, I'll definitely change it, just say the word :)

[erichkeane]

Switched to simply \r\n instead of both cases. This fixes the issue, is likely faster (important, since this is a performance critical part of code), and a smaller-hammer.

[rnk]

lib/Lex/Lexer.cpp

3076–3077 ↗

(On Diff #112436)

The bug probably doesn't happen in the \n\r case, because don't we count '\n's to compute our line numbers?

Anyway, yeah, I think we should make this specific to '\r'. In that case, we peek one ahead, and if we see a simple '\n' byte, we advance one more so that our line numbers stay correct.

[erichkeane]

Decided there were a few additional advantages to just handling \r\n, so Added them.

[rnk]

lgtm

cfe/trunk/lib/Lex/Lexer.cpp

case 26:  // DOS & CP/M EOF: "^Z".
     }
 
     // If Microsoft extensions are disabled, this is just random garbage.
     Kind = tok::unknown;
     break;
       
   case '\n':
   case '\r':
+    if (CurPtr[0] != Char && (CurPtr[0] == '\n' || CurPtr[0] == '\r'))
+      Char = getAndAdvanceChar(CurPtr, Result);
     // If we are inside a preprocessor directive and we see the end of line,
     // we know we are done with the directive, so return an EOD token.
     if (ParsingPreprocessorDirective) {
       // Done parsing the "line".
       ParsingPreprocessorDirective = false;
 
       // Restore comment saving mode, in case it was disabled for directive.
       if (PP)

cfe/trunk/test/Frontend/.gitattributes

+# Below test validates crlf line endings, so it should stay crlf.
+system-header-line-directive-ms-lineendings.c text eol=crlf

cfe/trunk/test/Frontend/system-header-line-directive-ms-lineendings.c

+// RUN: %clang_cc1 %s -E -o - -I %S/Inputs -isystem %S/Inputs/SystemHeaderPrefix | FileCheck %s
+#include <noline.h>
+#include <line-directive-in-system.h>
+
+#include "line-directive.h"
+
+// This tests that the line numbers for the current file are correctly outputted
+// for the include-file-completed test case.  
+
+// CHECK: # 1 "{{.*}}system-header-line-directive-ms-lineendings.c" 2
+// CHECK: # 1 "{{.*}}noline.h" 1 3
+// CHECK: foo();
+// CHECK: # 3 "{{.*}}system-header-line-directive-ms-lineendings.c" 2
+// CHECK: # 1 "{{.*}}line-directive-in-system.h" 1 3
+//      The "3" below indicates that "foo.h" is considered a system header.
+// CHECK: # 1 "foo.h" 3
+// CHECK: foo();
+// CHECK: # 4 "{{.*}}system-header-line-directive-ms-lineendings.c" 2
+// CHECK: # 1 "{{.*}}line-directive.h" 1
+// CHECK: # 10 "foo.h"{{$}}
+// CHECK: # 6 "{{.*}}system-header-line-directive-ms-lineendings.c" 2