Fix clang Lexer Windows line-ending bug

[caoz]

Summary

Add Windows line-ending support when parsing Perforce conflict marker.

This fixes 2 tests for Bug 35187 (https://bugs.llvm.org/show_bug.cgi?id=35187).

[zturner]

test/lit.cfg.py

52–57

Since the user has sed already, why wouldn't they have the actual tool dos2unix and unix2dos?

[caoz]

test/lit.cfg.py

52–57

Thanks Zachary. dos2unix and unix2dos aren't installed by default on Linux/Mac. However, if the user can be assumed to have them, I can remove these substitutions.

[belleyb]

test/lit.cfg.py

52–57

@zturner According to https://llvm.org/docs/GettingStarted.html#software, we shouldn't be relying on dos2unix nor unix2dos to be present on a build machine.

Note that the $'string' syntax (ANSI-C quotes) is a bash extension. According to the page mentioned above, the LLVM builds should only be relying on having a Bourne shell (sh). But, are there still any *nix system out there where /bin/sh isn't a link for \bin\bash ? I.e. Is relying on bash+sed fine here ?

A fully portable solution would be to write python scripts for dos2unix.py and unix2dos.py. That way, one would not be relying on build tools that LLVM isn't already using.

Any advice on how to proceed ?

lib/Lex/Lexer.cpp

unsigned Lexer::isNextPPTokenLParen() {
   if (Tok.is(tok::eof))
     return 2;
   return Tok.is(tok::l_paren);
 }
 
 /// \brief Find the end of a version control conflict marker.
 static const char *FindConflictEnd(const char *CurPtr, const char *BufferEnd,
                                    ConflictMarkerKind CMK) {
-  const char *Terminator = CMK == CMK_Perforce ? "<<<<\n" : ">>>>>>>";
-  size_t TermLen = CMK == CMK_Perforce ? 5 : 7;
+  const char *Terminator = CMK == CMK_Perforce ? "<<<<" : ">>>>>>>";
+  size_t TermLen = CMK == CMK_Perforce ? 4 : 7;
   auto RestOfBuffer = StringRef(CurPtr, BufferEnd - CurPtr).substr(TermLen);
   size_t Pos = RestOfBuffer.find(Terminator);
   while (Pos != StringRef::npos) {
     // Must occur at start of line.
     if (Pos == 0 ||
-        (RestOfBuffer[Pos - 1] != '\r' && RestOfBuffer[Pos - 1] != '\n')) {
+        (RestOfBuffer[Pos - 1] != '\r' && RestOfBuffer[Pos - 1] != '\n') ||
+        (CMK == CMK_Perforce && Pos + 4 < RestOfBuffer.size() &&
+            RestOfBuffer[Pos + 4] != '\r' && RestOfBuffer[Pos + 4] != '\n')) {
       RestOfBuffer = RestOfBuffer.substr(Pos+TermLen);
       Pos = RestOfBuffer.find(Terminator);
       continue;
     }
     return RestOfBuffer.data()+Pos;
   }
   return nullptr;
 }

test/Lexer/conflict-marker.c

-// RUN: %clang_cc1 %s -verify -fsyntax-only
+// RUN: unix2dos %s | %clang_cc1 -verify -fsyntax-only
+// RUN: dos2unix %s | %clang_cc1 -verify -fsyntax-only
 
 // Test that we recover gracefully from conflict markers left in input files.
 // PR5238
 
 // diff3 style  expected-error@+1 {{version control conflict marker in file}}
 <<<<<<< .mine
 int x = 4;
 |||||||

test/lit.cfg.py

 llvm_config.use_clang()
 
 # Propagate path to symbolizer for ASan/MSan.
 llvm_config.with_system_environment(
     ['ASAN_SYMBOLIZER_PATH', 'MSAN_SYMBOLIZER_PATH'])
 
 config.substitutions.append(('%PATH%', config.environment['PATH']))
 
+if platform.system() in ['Windows']:
+    config.substitutions.append(('dos2unix', 'sed -b "s/\r$//"'))
+    config.substitutions.append(('unix2dos', 'sed -b "s/\r*$/\r/"'))
+else:
+    config.substitutions.append(('dos2unix', "sed $'s/\r$//'"))
+    config.substitutions.append(('unix2dos', "sed $'s/\r*$/\r/'"))
 
 # For each occurrence of a clang tool name, replace it with the full path to
 # the build directory holding that tool.  We explicitly specify the directories
 # to search to ensure that we get the tools just built and not some random
 # tools that might happen to be in the user's PATH.
 tool_dirs = [config.clang_tools_dir, config.llvm_tools_dir]
 
 tools = [