Show correct column nr. when multi-byte utf8 chars are used.

[erikjv]

Summary

Previously, the column number in a diagnostic would be the byte position
in the line. This results in incorrect column numbers when a multi-byte
UTF-8 character would be present in the input. This change corrects for
those multi-byte characters and for zero-length diacritic marks.

This fixes PR21144.

[efriedma]

Correctly counting columns is a bit more complicated that that... for example, consider what happens if you replace ideëen with idez̈en. See https://stackoverflow.com/questions/3634627/how-to-know-the-preferred-display-width-in-columns-of-unicode-characters .

[efriedma]

I didn't really search for it before, but it looks like LLVM already has a routine for computing column widths? See llvm::sys::unicode::columnWidthUTF8.

There are some tools which parse clang diagnostic output; we might need a flag to control this. Not sure who would know about that?

lib/Basic/SourceManager.cpp

1501 ↗

(On Diff #117660)

Instead of adding a parameter to getColumnNumber, it would probably make sense to just make this caller correct the column number afterwards.

[erikjv]

I moved all code to the TextDiagnostics, so all other interfaces still get byte offsets.

[efriedma]

Still worried about the effect on tools which parse clang diagnostics... please send a message to cfe-dev. Hopefully we'll get responses there.

lib/Frontend/TextDiagnostic.cpp

 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/Locale.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Unicode.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 
 using namespace clang;
 
 static const enum raw_ostream::Colors noteColor =
   raw_ostream::BLACK;
 static const enum raw_ostream::Colors remarkColor =
void TextDiagnostic::emitDiagnosticLoc(FullSourceLoc Loc, PresumedLoc PLoc,
   case DiagnosticOptions::Clang: OS << ':'  << LineNo; break;
   case DiagnosticOptions::MSVC:  OS << '('  << LineNo; break;
   case DiagnosticOptions::Vi:    OS << " +" << LineNo; break;
   }
 
   if (DiagOpts->ShowColumn)
     // Compute the column number.
     if (unsigned ColNo = PLoc.getColumn()) {
+      // Correct the column number for multi-byte UTF-8 code-points.
+      bool Invalid = false;
+      StringRef BufData = Loc.getBufferData(&Invalid);
+      if (!Invalid) {
+        const char *BufStart = BufData.data();
+        const char *BufEnd = BufStart + BufData.size();
+
+        // Decompose the location into a FID/Offset pair.
+        std::pair<FileID, unsigned> LocInfo = Loc.getDecomposedLoc();
+        FileID FID = LocInfo.first;
+        const SourceManager &SM = Loc.getManager();
+        const char *LineStart =
+            BufStart +
+            SM.getDecomposedLoc(SM.translateLineCol(FID, LineNo, 1)).second;
+        if (LineStart + ColNo < BufEnd) {
+          StringRef SourceLine(LineStart, ColNo);
+          int CorrectedColNo = llvm::sys::unicode::columnWidthUTF8(SourceLine);
+          if (CorrectedColNo != -1)
+            ColNo = unsigned(CorrectedColNo);
+        }
+      }
+
       if (DiagOpts->getFormat() == DiagnosticOptions::MSVC) {
         OS << ',';
         // Visual Studio 2010 or earlier expects column number to be off by one
         if (LangOpts.MSCompatibilityVersion &&
             !LangOpts.isCompatibleWithMSVC(LangOptions::MSVC2012))
           ColNo--;
       } else
         OS << ':';

test/Misc/diag-utf8.cpp

+// RUN: not %clang_cc1 -fsyntax-only %s 2>&1 | FileCheck %s
+
+struct Foo { int member; };
+
+void f(Foo foo)
+{
+    "ideeen" << foo; // CHECK: {{.*[/\\]}}diag-utf8.cpp:7:14: error: invalid operands to binary expression ('const char *' and 'Foo')
+    "ideëen" << foo; // CHECK: {{.*[/\\]}}diag-utf8.cpp:8:14: error: invalid operands to binary expression ('const char *' and 'Foo')
+    "idez̈en" << foo; // CHECK: {{.*[/\\]}}diag-utf8.cpp:9:14: error: invalid operands to binary expression ('const char *' and 'Foo')
+}