[clang-scan-deps] Skip UTF-8 BOM in source minimizer

[aganea]

Summary

As per title.

[arphaman]

LGTM

[dexonsmith]

lib/Lex/DependencyDirectivesSourceMinimizer.cpp

822 ↗

(On Diff #216311)

Is skipping this the right thing, or should it also be copied to the output?

[aganea]

lib/Lex/DependencyDirectivesSourceMinimizer.cpp

822 ↗

(On Diff #216311)

The code in Lexer::InitLexer() assumes the files are always encoded as UTF-8, it simply skips over the BOM like we do here.

cfe/trunk/lib/Lex/DependencyDirectivesSourceMinimizer.cpp

bool Minimizer::lexPPLine(const char *&First, const char *const End) {
 
   if (Kind == pp_pragma_import)
     return lexPragma(First, End);
 
   // Everything else.
   return lexDefault(Kind, Id.Name, First, End);
 }
 
+static void skipUTF8ByteOrderMark(const char *&First, const char *const End) {
+  if ((End - First) >= 3 && First[0] == '\xef' && First[1] == '\xbb' &&
+      First[2] == '\xbf')
+    First += 3;
+}
+
 bool Minimizer::minimizeImpl(const char *First, const char *const End) {
+  skipUTF8ByteOrderMark(First, End);
   while (First != End)
     if (lexPPLine(First, End))
       return true;
   return false;
 }
 
 bool Minimizer::minimize() {
   bool Error = minimizeImpl(Input.begin(), Input.end());

cfe/trunk/test/Lexer/minimize_source_to_dependency_directives_utf8bom.c

+// Test UTF8 BOM at start of file
+// RUN: printf '\xef\xbb\xbf' > %t.c
+// RUN: echo '#ifdef TEST\n' >> %t.c
+// RUN: echo '#include <string>' >> %t.c
+// RUN: echo '#endif' >> %t.c
+// RUN: %clang_cc1 -DTEST -print-dependency-directives-minimized-source %t.c 2>&1 | FileCheck %s
+
+// CHECK:      #ifdef TEST
+// CHECK-NEXT: #include <string>
+// CHECK-NEXT: #endif