[llvm-cxxfilt] Split and demangle stdin input on certain non-alphanumerics.

[mattd]

Summary

This patch attempts to replicate GNU c++-filt behavior when splitting stdin input for demangling.

Previously, cxx-filt would split input only on spaces. Each delimited item is then demangled.
From what I have tested, GNU c++filt also splits input on any character that does not make
up the mangled name (notably commas, but also a large set of non-alphanumeric characters).

This patch splits stdin input on any character that does not belong to the Itanium mangling
format (since Itanium is currently the only supported format in llvm-cxxfilt).

This is an update to PR39990

[compnerd]

Previously we would have emitted Foo [[:space:]] . right?

llvm/tools/llvm-cxxfilt/llvm-cxxfilt.cpp

125 ↗

(On Diff #187457)

Shouldn't this be const auto &

[mattd]

In D58416#1403227, @compnerd wrote:

Previously we would have emitted Foo [[:space:]] . right?

Thanks for the feedback. Previously we emitted a literal space character, ' ', between the mangled terms that were split on [[:space:]]. This means that tabs would get replaced with single spaces when reassembling the demangled string. With this patch, we now delimit on a variety of non-space delimiters, and capture the delimiter. This patch correctly reassembles the string using the captured delimiter... not just a single space.

[mattd]

llvm/tools/llvm-cxxfilt/llvm-cxxfilt.cpp

125 ↗

(On Diff #187457)

Yes it should be const. I have corrected my local copy.

[jhenderson]

llvm/test/tools/llvm-cxxfilt/delimiters.test

30 ↗

(On Diff #187457)

It may be worth adding one more demanglable string after this one here, to show that multiple delimiters can be handled and it doesn't completely stop demangling.

llvm/tools/llvm-cxxfilt/llvm-cxxfilt.cpp

89 ↗

(On Diff #187457)

Nit: this and similar lines below are valid uses of auto.

114 ↗

(On Diff #187457)

I take it c++filt does not split on those characters?

[mattd]

llvm/tools/llvm-cxxfilt/llvm-cxxfilt.cpp

114 ↗

(On Diff #187457)

It does but in specific cases depending on if the character is a prefix or suffix to the word. For instance

echo 'Test: ._Z1F _Z1F. Done' | c++filt
Test: .F _Z1F. Done

A similar result is produced when using '$' I feel that the safest thing to do is to just let those implementation specific characters be part of the legal set of allowed characters. In fact, we already handle '.' in the suffix position, via the DotSuffix handler in include/llvm/Demangle/ItaniumDemangle.h

[mattd]

• Added a more complicated test.
• Made a few variables 'auto' instead of their iterator type.

[jhenderson]

LGTM.

[thakis]

The test fails on http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/16086

llvm/trunk/test/tools/llvm-cxxfilt/delimiters.test

+RUN: echo ",,_Z3Foo!" \
+RUN:      "_Z3Foo\""  \
+RUN:      "_Z3Foo\""  \
+RUN:      "_Z3Foo#"   \
+RUN:      "_Z3Foo%"   \
+RUN:      "_Z3Foo&"   \
+RUN:      "_Z3Foo'"   \
+RUN:      "_Z3Foo("   \
+RUN:      "_Z3Foo)"   \
+RUN:      "_Z3Foo*"   \
+RUN:      "_Z3Foo+"   \
+RUN:      "_Z3Foo,"   \
+RUN:      "_Z3Foo-"   \
+RUN:      "_Z3Foo/"   \
+RUN:      "_Z3Foo:"   \
+RUN:      "_Z3Foo;"   \
+RUN:      "_Z3Foo<"   \
+RUN:      "_Z3Foo="   \
+RUN:      "_Z3Foo>"   \
+RUN:      "_Z3Foo?"   \
+RUN:      "_Z3Foo@"   \
+RUN:      "_Z3Foo["   \
+RUN:      "_Z3Foo\\"  \
+RUN:      "_Z3Foo]"   \
+RUN:      "_Z3Foo^"   \
+RUN:      "_Z3Foo\`"  \
+RUN:      "_Z3Foo{"   \
+RUN:      "_Z3Foo|"   \
+RUN:      "_Z3Foo}"   \
+RUN:      "_Z3Foo~,," \
+RUN:      "_Z3Foo,,_Z3Bar::_Z3Baz  _Z3Foo,_Z3Bar:_Z3Baz" \
+RUN:      '_Z3Foo$ ._Z3Foo' | llvm-cxxfilt | FileCheck %s
+
+CHECK: ,,Foo!
+CHECK: Foo"
+CHECK: Foo#
+CHECK: Foo%
+CHECK: Foo&
+CHECK: Foo'
+CHECK: Foo(
+CHECK: Foo)
+CHECK: Foo*
+CHECK: Foo+
+CHECK: Foo,
+CHECK: Foo-
+CHECK: Foo/
+CHECK: Foo:
+CHECK: Foo;
+CHECK: Foo<
+CHECK: Foo=
+CHECK: Foo>
+CHECK: Foo?
+CHECK: Foo@
+CHECK: Foo[
+CHECK: Foo\
+CHECK: Foo]
+CHECK: Foo^
+CHECK: Foo`
+CHECK: Foo{
+CHECK: Foo|
+CHECK: Foo}
+CHECK: Foo~,,
+CHECK: Foo,,Bar::Baz  Foo,Bar:Baz
+CHECK: _Z3Foo$ ._Z3Foo

llvm/trunk/tools/llvm-cxxfilt/llvm-cxxfilt.cpp

if (!Undecorated &&
     Undecorated = itaniumDemangle(Decorated + 6, nullptr, nullptr, &Status);
   }
 
   std::string Result(Undecorated ? Undecorated : Mangled);
   free(Undecorated);
   return Result;
 }
 
+// Split 'Source' on any character that fails to pass 'IsLegalChar'.  The
+// returned vector consists of pairs where 'first' is the delimited word, and
+// 'second' are the delimiters following that word.
+static void SplitStringDelims(
+    StringRef Source,
+    SmallVectorImpl<std::pair<StringRef, StringRef>> &OutFragments,
+    function_ref<bool(char)> IsLegalChar) {
+  // The beginning of the input string.
+  const auto Head = Source.begin();
+
+  // Obtain any leading delimiters.
+  auto Start = std::find_if(Head, Source.end(), IsLegalChar);
+  if (Start != Head)
+    OutFragments.push_back({"", Source.slice(0, Start - Head)});
+
+  // Capture each word and the delimiters following that word.
+  while (Start != Source.end()) {
+    Start = std::find_if(Start, Source.end(), IsLegalChar);
+    auto End = std::find_if_not(Start, Source.end(), IsLegalChar);
+    auto DEnd = std::find_if(End, Source.end(), IsLegalChar);
+    OutFragments.push_back({Source.slice(Start - Head, End - Head),
+                            Source.slice(End - Head, DEnd - Head)});
+    Start = DEnd;
+  }
+}
+
+// This returns true if 'C' is a character that can show up in an
+// Itanium-mangled string.
+static bool IsLegalItaniumChar(char C) {
+  // Itanium CXX ABI [External Names]p5.1.1:
+  // '$' and '.' in mangled names are reserved for private implementations.
+  return isalnum(C) || C == '.' || C == '$' || C == '_';
+}
+
 // If 'Split' is true, then 'Mangled' is broken into individual words and each
 // word is demangled.  Otherwise, the entire string is treated as a single
 // mangled item.  The result is output to 'OS'.
 static void demangleLine(llvm::raw_ostream &OS, StringRef Mangled, bool Split) {
   std::string Result;
   if (Split) {
-    SmallVector<StringRef, 16> Words;
-    SplitString(Mangled, Words);
-    for (auto Word : Words)
-      Result += demangle(OS, Word) + ' ';
-    // Remove the trailing space character.
-    if (Result.back() == ' ')
-      Result.pop_back();
+    SmallVector<std::pair<StringRef, StringRef>, 16> Words;
+    SplitStringDelims(Mangled, Words, IsLegalItaniumChar);
+    for (const auto &Word : Words)
+      Result += demangle(OS, Word.first) + Word.second.str();
   } else
     Result = demangle(OS, Mangled);
   OS << Result << '\n';
   OS.flush();
 }
 
 int main(int argc, char **argv) {
   InitLLVM X(argc, argv);