[YAML] Add support for non-printable characters

[thegameg]

Summary

LLVM IR function names which disable mangling start with '\01' (https://www.llvm.org/docs/LangRef.html#identifiers).

When an identifier like "\01@abc@" gets dumped to MIR, it is quoted, but only with single quotes.

http://www.yaml.org/spec/1.2/spec.html#id2770814:

"The allowed character range explicitly excludes the C0 control block allowed), the surrogate block #xD800-#xDFFF, #xFFFE, and #xFFFF."

http://www.yaml.org/spec/1.2/spec.html#id2776092:

"All non-printable characters must be escaped. [...] Note that escape sequences are only interpreted in double-quoted scalars."

This patch adds support for printing non-printable characters between double quotes if needed.

[thegameg]

Also related to PR31743.

[JDevlieghere]

LGTM!

docs/YamlIO.rst

469 ↗

(On Diff #127122)

s/Simple/Single/

[thegameg]

Fix typo in docs.

[MatzeB]

LGTM too with some suggestions/nitpicks:

include/llvm/Support/YAMLTraits.h

503–565 ↗

(On Diff #127134)

Can't you combine the two functions to a single one so we only need to scan the string once? (Just keep the checks at the begin and end of needsSingleQuote() around and then perform the scan from needsDoubleQuotes() while additionally checking the things from ScalarSafeChars. Instead of returning false/true you would compute the max required quoting)

lib/Support/YAMLTraits.cpp

644–647 ↗

(On Diff #127134)

While I guess it's not strictly necessary, how about something along the lines of:

// Note: Special casing the non-printable ASCII characters.
if (S[j] == QuoteChar || (MustQuote == QuotingType::Double && S[j] <= 0x1F)) {
  output(...)
  if (S[j] <= 0x1f) {
    output("\\0x" + hex(S[j]));
  } else {
    output(Quote);
  }
  i = j+1;
}

so we get a nice \0x01 output instead of a quoted unprintable character?

[MatzeB]

llvm/trunk/include/llvm/Support/YAMLTraits.h

552–555

This part is sketchy. As we are most probably dealing with UTF-8 without having fully parsed code point here. (Not that the code/comments give any indication that we indeed have UTF-8). While for ASCII (= top bit is clear) UTF-8 and ASCII are the same, for other characters (top bit set) it is not.

I think the best/easiest thing to do here is to always trigger double quoting on (C & 0x80) != 0

[thegameg]

llvm/trunk/include/llvm/Support/YAMLTraits.h

552–555

Ah, I quickly skipped that part...

Should be fixed in r321068.

[vladimir.plyashkun]

Hello @thegameg !
Is is correct that printable UTF-8 characters are also escaped by these changes?
For example, Clang-Tidy output now uses double-quotes for UTF-8:

Message:         "parameter '\xffffffffffffffd0\xffffffffffffffbf\xffffffffffffffd0\xffffffffffffffb0\xffffffffffffffd1\xffffffffffffff80\xffffffffffffffd0\xffffffffffffffb0\xffffffffffffffd0\xffffffffffffffbc\xffffffffffffffd0\xffffffffffffffb5\xffffffffffffffd1\xffffffffffffff82\xffffffffffffffd1\xffffffffffffff80' is unused"

but previously it was simply:

Message:         'parameter ''параметр'' is unused'

I'm using snakeyaml parser which is cannot recognize these sequences. By this, i've got incorrect clang-tidy output.
Also i saw that not all the UTF-8 characters are printed in double quotes.
For example in the same output,

ReplacementText: ' /*параметр*/'

printed in the single quotes.

[thegameg]

In D41290#961801, @vladimir.plyashkun wrote:

Hello @thegameg !
Is is correct that printable UTF-8 characters are also escaped by these changes?

Sorry for breaking that, hope it's fixed in r321283.

llvm/trunk/docs/YamlIO.rst

struct ScalarTraits<MyCustomType> {
         out << value;  // do custom formatting here
       }
       static StringRef input(StringRef scalar, void*, MyCustomType &value) {
         // do custom parsing here.  Return the empty string on success,
         // or an error message on failure.
         return StringRef();
       }
       // Determine if this scalar needs quotes.
-      static bool mustQuote(StringRef) { return true; }
+      static QuotingType mustQuote(StringRef) { return QuotingType::Single; }
     };
 
 Block Scalars
 -------------
 
 YAML block scalars are string literals that are represented in YAML using the
 literal block notation, just like the example shown below:
 

llvm/trunk/include/llvm/CodeGen/MIRYamlMapping.h

template <> struct ScalarTraits<StringValue> {
   static StringRef input(StringRef Scalar, void *Ctx, StringValue &S) {
     S.Value = Scalar.str();
     if (const auto *Node =
             reinterpret_cast<yaml::Input *>(Ctx)->getCurrentNode())
       S.SourceRange = Node->getSourceRange();
     return "";
   }
 
-  static bool mustQuote(StringRef Scalar) { return needsQuotes(Scalar); }
+  static QuotingType mustQuote(StringRef S) { return needsQuotes(S); }
 };
 
 struct FlowStringValue : StringValue {
   FlowStringValue() = default;
   FlowStringValue(std::string Value) : StringValue(std::move(Value)) {}
 };
 
 template <> struct ScalarTraits<FlowStringValue> {
   static void output(const FlowStringValue &S, void *, raw_ostream &OS) {
     return ScalarTraits<StringValue>::output(S, nullptr, OS);
   }
 
   static StringRef input(StringRef Scalar, void *Ctx, FlowStringValue &S) {
     return ScalarTraits<StringValue>::input(Scalar, Ctx, S);
   }
 
-  static bool mustQuote(StringRef Scalar) { return needsQuotes(Scalar); }
+  static QuotingType mustQuote(StringRef S) { return needsQuotes(S); }
 };
 
 struct BlockStringValue {
   StringValue Value;
 
   bool operator==(const BlockStringValue &Other) const {
     return Value == Other.Value;
   }
template <> struct ScalarTraits<UnsignedValue> {
 
   static StringRef input(StringRef Scalar, void *Ctx, UnsignedValue &Value) {
     if (const auto *Node =
             reinterpret_cast<yaml::Input *>(Ctx)->getCurrentNode())
       Value.SourceRange = Node->getSourceRange();
     return ScalarTraits<unsigned>::input(Scalar, Ctx, Value.Value);
   }
 
-  static bool mustQuote(StringRef Scalar) {
+  static QuotingType mustQuote(StringRef Scalar) {
     return ScalarTraits<unsigned>::mustQuote(Scalar);
   }
 };
 
 template <> struct ScalarEnumerationTraits<MachineJumpTableInfo::JTEntryKind> {
   static void enumeration(yaml::IO &IO,
                           MachineJumpTableInfo::JTEntryKind &EntryKind) {
     IO.enumCase(EntryKind, "block-address",

llvm/trunk/include/llvm/ObjectYAML/CodeViewYAMLTypeHashing.h

 ArrayRef<uint8_t> toDebugH(const DebugHSection &DebugH,
                            BumpPtrAllocator &Alloc);
 
 } // end namespace CodeViewYAML
 
 } // end namespace llvm
 
 LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::DebugHSection)
-LLVM_YAML_DECLARE_SCALAR_TRAITS(CodeViewYAML::GlobalHash, false)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(CodeViewYAML::GlobalHash, QuotingType::None)
 LLVM_YAML_IS_SEQUENCE_VECTOR(CodeViewYAML::GlobalHash)
 
 #endif // LLVM_OBJECTYAML_CODEVIEWYAMLTYPES_H

llvm/trunk/include/llvm/ObjectYAML/CodeViewYAMLTypes.h

 
 std::vector<LeafRecord> fromDebugT(ArrayRef<uint8_t> DebugT);
 ArrayRef<uint8_t> toDebugT(ArrayRef<LeafRecord>, BumpPtrAllocator &Alloc);
 
 } // end namespace CodeViewYAML
 
 } // end namespace llvm
 
-LLVM_YAML_DECLARE_SCALAR_TRAITS(codeview::GUID, true)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(codeview::GUID, QuotingType::Single)
 
 LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::LeafRecord)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::MemberRecord)
 
 LLVM_YAML_IS_SEQUENCE_VECTOR(CodeViewYAML::LeafRecord)
 LLVM_YAML_IS_SEQUENCE_VECTOR(CodeViewYAML::MemberRecord)
 
 #endif // LLVM_OBJECTYAML_CODEVIEWYAMLTYPES_H

llvm/trunk/include/llvm/ObjectYAML/MachOYAML.h

 };
 
 // This trait is used for 16-byte chars in Mach structures used for strings
 using char_16 = char[16];
 
 template <> struct ScalarTraits<char_16> {
   static void output(const char_16 &Val, void *, raw_ostream &Out);
   static StringRef input(StringRef Scalar, void *, char_16 &Val);
-  static bool mustQuote(StringRef S);
+  static QuotingType mustQuote(StringRef S);
 };
 
 // This trait is used for UUIDs. It reads and writes them matching otool's
 // formatting style.
 using uuid_t = raw_ostream::uuid_t;
 
 template <> struct ScalarTraits<uuid_t> {
   static void output(const uuid_t &Val, void *, raw_ostream &Out);
   static StringRef input(StringRef Scalar, void *, uuid_t &Val);
-  static bool mustQuote(StringRef S);
+  static QuotingType mustQuote(StringRef S);
 };
 
 // Load Command struct mapping traits
 
 #define LOAD_COMMAND_STRUCT(LCStruct)                                          \
   template <> struct MappingTraits<MachO::LCStruct> {                          \
     static void mapping(IO &IO, MachO::LCStruct &LoadCommand);                 \
   };

llvm/trunk/include/llvm/ObjectYAML/YAML.h

if (LHS.Data.empty() && RHS.Data.empty())
     return true;
 
   return LHS.DataIsHexString == RHS.DataIsHexString && LHS.Data == RHS.Data;
 }
 
 template <> struct ScalarTraits<BinaryRef> {
   static void output(const BinaryRef &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, BinaryRef &);
-  static bool mustQuote(StringRef S) { return needsQuotes(S); }
+  static QuotingType mustQuote(StringRef S) { return needsQuotes(S); }
 };
 
 } // end namespace yaml
 
 } // end namespace llvm
 
 #endif // LLVM_OBJECTYAML_YAML_H

llvm/trunk/include/llvm/Support/YAMLTraits.h

 //===- llvm/Support/YAMLTraits.h --------------------------------*- C++ -*-===//
 //
 //                             The LLVM Linker
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_SUPPORT_YAMLTRAITS_H
 #define LLVM_SUPPORT_YAMLTRAITS_H
 
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/AlignOf.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Support/SourceMgr.h"

 ///        }
 ///      };
 template<typename T>
 struct ScalarBitSetTraits {
   // Must provide:
   // static void bitset(IO &io, T &value);
 };
 
+/// Describe which type of quotes should be used when quoting is necessary.
+/// Some non-printable characters need to be double-quoted, while some others
+/// are fine with simple-quoting, and some don't need any quoting.
+enum class QuotingType { None, Single, Double };
+
 /// This class should be specialized by type that requires custom conversion
 /// to/from a yaml scalar.  For example:
 ///
 ///    template<>
 ///    struct ScalarTraits<MyType> {
 ///      static void output(const MyType &val, void*, llvm::raw_ostream &out) {
 ///        // stream out custom formatting
 ///        out << llvm::format("%x", val);
 ///      }
 ///      static StringRef input(StringRef scalar, void*, MyType &value) {
 ///        // parse scalar and set `value`
 ///        // return empty string on success, or error string
 ///        return StringRef();
 ///      }
-///      static bool mustQuote(StringRef) { return true; }
+///      static QuotingType mustQuote(StringRef) { return QuotingType::Single; }
 ///    };
 template<typename T>
 struct ScalarTraits {
   // Must provide:
   //
   // Function to write the value as a string:
   //static void output(const T &value, void *ctxt, llvm::raw_ostream &out);
   //
   // Function to convert a string to a value.  Returns the empty
   // StringRef on success or an error string if string is malformed:
   //static StringRef input(StringRef scalar, void *ctxt, T &value);
   //
   // Function to determine if the value should be quoted.
-  //static bool mustQuote(StringRef);
+  //static QuotingType mustQuote(StringRef);
 };
 
 /// This class should be specialized by type that requires custom conversion
 /// to/from a YAML literal block scalar. For example:
 ///
 ///    template <>
 ///    struct BlockScalarTraits<MyType> {
 ///      static void output(const MyType &Value, void*, llvm::raw_ostream &Out)

 };
 
 // Test if ScalarTraits<T> is defined on type T.
 template <class T>
 struct has_ScalarTraits
 {
   using Signature_input = StringRef (*)(StringRef, void*, T&);
   using Signature_output = void (*)(const T&, void*, raw_ostream&);
-  using Signature_mustQuote = bool (*)(StringRef);
+  using Signature_mustQuote = QuotingType (*)(StringRef);
 
   template <typename U>
   static char test(SameType<Signature_input, &U::input> *,
                    SameType<Signature_output, &U::output> *,
                    SameType<Signature_mustQuote, &U::mustQuote> *);
 
   template <typename U>
   static double test(...);
return S.equals("null") || S.equals("Null") || S.equals("NULL") ||
          S.equals("~");
 }
 
 inline bool isBool(StringRef S) {
   return S.equals("true") || S.equals("True") || S.equals("TRUE") ||
          S.equals("false") || S.equals("False") || S.equals("FALSE");
 }
 
-inline bool needsQuotes(StringRef S) {
+// 5.1. Character Set
+// The allowed character range explicitly excludes the C0 control block #x0-#x1F
+// (except for TAB #x9, LF #xA, and CR #xD which are allowed), DEL #x7F, the C1
+// control block #x80-#x9F (except for NEL #x85 which is allowed), the surrogate
+// block #xD800-#xDFFF, #xFFFE, and #xFFFF.
+inline QuotingType needsQuotes(StringRef S) {
   if (S.empty())
-    return true;
+    return QuotingType::Single;
   if (isspace(S.front()) || isspace(S.back()))
-    return true;
+    return QuotingType::Single;
   if (S.front() == ',')
-    return true;
-
-  static const char ScalarSafeChars[] =
-      "abcdefghijklmnopqrstuvwxyz"
-      "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-/^., \t";
-  if (S.find_first_not_of(ScalarSafeChars) != StringRef::npos)
-    return true;
-
+    return QuotingType::Single;
   if (isNull(S))
-    return true;
+    return QuotingType::Single;
   if (isBool(S))
-    return true;
+    return QuotingType::Single;
   if (isNumeric(S))
-    return true;
+    return QuotingType::Single;
 
-  return false;
+  QuotingType MaxQuotingNeeded = QuotingType::None;
+  for (unsigned char C : S) {
+    // Alphanum is safe.
+    if (isAlnum(C))
+      continue;
+
+    switch (C) {
+    // Safe scalar characters.
+    case '_':
+    case '-':
+    case '/':
+    case '^':
+    case '.':
+    case ',':
+    case ' ':
+    // TAB (0x9), LF (0xA), CR (0xD) and NEL (0x85) are allowed.
+    case 0x9:
+    case 0xA:
+    case 0xD:
+    case 0x85:
+      continue;
+    // DEL (0x7F) are excluded from the allowed character range.
+    case 0x7F:
+      return QuotingType::Double;
+    default: {
+      // C0 control block (0x0 - 0x1F) is excluded from the allowed character
+      // range.
+      if (C <= 0x1F)
+        return QuotingType::Double;
+      // C1 control block (0x80 - 0x9F) is excluded from the allowed character
+      // range.
+      if (C >= 0x80 && C <= 0x9F)
+        return QuotingType::Double;
+
+      // The character is not safe, at least simple quoting needed.
+      MaxQuotingNeeded = QuotingType::Single;
+    }
+    }
+  }
+
+  return MaxQuotingNeeded;
 }
 
 template <typename T, typename Context>
 struct missingTraits
     : public std::integral_constant<bool,
                                     !has_ScalarEnumerationTraits<T>::value &&
                                         !has_ScalarBitSetTraits<T>::value &&
                                         !has_ScalarTraits<T>::value &&
public:
   virtual bool matchEnumScalar(const char*, bool) = 0;
   virtual bool matchEnumFallback() = 0;
   virtual void endEnumScalar() = 0;
 
   virtual bool beginBitSetScalar(bool &) = 0;
   virtual bool bitSetMatch(const char*, bool) = 0;
   virtual void endBitSetScalar() = 0;
 
-  virtual void scalarString(StringRef &, bool) = 0;
+  virtual void scalarString(StringRef &, QuotingType) = 0;
   virtual void blockScalarString(StringRef &) = 0;
 
   virtual void setError(const Twine &) = 0;
 
   template <typename T>
   void enumCase(T &Val, const char* Str, const T ConstVal) {
     if ( matchEnumScalar(Str, outputting() && Val == ConstVal) ) {
       Val = ConstVal;
else {
     io.endSequence();
   }
 }
 
 template<>
 struct ScalarTraits<bool> {
   static void output(const bool &, void* , raw_ostream &);
   static StringRef input(StringRef, void *, bool &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<StringRef> {
   static void output(const StringRef &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, StringRef &);
-  static bool mustQuote(StringRef S) { return needsQuotes(S); }
+  static QuotingType mustQuote(StringRef S) { return needsQuotes(S); }
 };
 
 template<>
 struct ScalarTraits<std::string> {
   static void output(const std::string &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, std::string &);
-  static bool mustQuote(StringRef S) { return needsQuotes(S); }
+  static QuotingType mustQuote(StringRef S) { return needsQuotes(S); }
 };
 
 template<>
 struct ScalarTraits<uint8_t> {
   static void output(const uint8_t &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, uint8_t &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<uint16_t> {
   static void output(const uint16_t &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, uint16_t &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<uint32_t> {
   static void output(const uint32_t &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, uint32_t &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<uint64_t> {
   static void output(const uint64_t &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, uint64_t &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<int8_t> {
   static void output(const int8_t &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, int8_t &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<int16_t> {
   static void output(const int16_t &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, int16_t &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<int32_t> {
   static void output(const int32_t &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, int32_t &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<int64_t> {
   static void output(const int64_t &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, int64_t &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<float> {
   static void output(const float &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, float &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<double> {
   static void output(const double &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, double &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 // For endian types, we just use the existing ScalarTraits for the underlying
 // type.  This way endian aware types are supported whenever a ScalarTraits
 // is defined for the underlying type.
 template <typename value_type, support::endianness endian, size_t alignment>
 struct ScalarTraits<support::detail::packed_endian_specific_integral<
     value_type, endian, alignment>> {
   using endian_type =
       support::detail::packed_endian_specific_integral<value_type, endian,
                                                        alignment>;
 
   static void output(const endian_type &E, void *Ctx, raw_ostream &Stream) {
     ScalarTraits<value_type>::output(static_cast<value_type>(E), Ctx, Stream);
   }
 
   static StringRef input(StringRef Str, void *Ctx, endian_type &E) {
     value_type V;
     auto R = ScalarTraits<value_type>::input(Str, Ctx, V);
     E = static_cast<endian_type>(V);
     return R;
   }
 
-  static bool mustQuote(StringRef Str) {
+  static QuotingType mustQuote(StringRef Str) {
     return ScalarTraits<value_type>::mustQuote(Str);
   }
 };
 
 // Utility for use within MappingTraits<>::mapping() method
 // to [de]normalize an object for use with YAML conversion.
 template <typename TNorm, typename TFinal>
 struct MappingNormalization {
private:
   void endFlowSequence() override;
   void beginEnumScalar() override;
   bool matchEnumScalar(const char*, bool) override;
   bool matchEnumFallback() override;
   void endEnumScalar() override;
   bool beginBitSetScalar(bool &) override;
   bool bitSetMatch(const char *, bool ) override;
   void endBitSetScalar() override;
-  void scalarString(StringRef &, bool) override;
+  void scalarString(StringRef &, QuotingType) override;
   void blockScalarString(StringRef &) override;
   void setError(const Twine &message) override;
   bool canElideEmptySequence() override;
 
   class HNode {
     virtual void anchor();
 
   public:
public:
   void endFlowSequence() override;
   void beginEnumScalar() override;
   bool matchEnumScalar(const char*, bool) override;
   bool matchEnumFallback() override;
   void endEnumScalar() override;
   bool beginBitSetScalar(bool &) override;
   bool bitSetMatch(const char *, bool ) override;
   void endBitSetScalar() override;
-  void scalarString(StringRef &, bool) override;
+  void scalarString(StringRef &, QuotingType) override;
   void blockScalarString(StringRef &) override;
   void setError(const Twine &message) override;
   bool canElideEmptySequence() override;
 
   // These are only used by operator<<. They could be private
   // if that templated operator could be made a friend.
   void beginDocuments();
   bool preflightDocument(unsigned);

 LLVM_YAML_STRONG_TYPEDEF(uint16_t, Hex16)
 LLVM_YAML_STRONG_TYPEDEF(uint32_t, Hex32)
 LLVM_YAML_STRONG_TYPEDEF(uint64_t, Hex64)
 
 template<>
 struct ScalarTraits<Hex8> {
   static void output(const Hex8 &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, Hex8 &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<Hex16> {
   static void output(const Hex16 &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, Hex16 &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<Hex32> {
   static void output(const Hex32 &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, Hex32 &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 template<>
 struct ScalarTraits<Hex64> {
   static void output(const Hex64 &, void *, raw_ostream &);
   static StringRef input(StringRef, void *, Hex64 &);
-  static bool mustQuote(StringRef) { return false; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::None; }
 };
 
 // Define non-member operator>> so that Input can stream in a document list.
 template <typename T>
 inline
 typename std::enable_if<has_DocumentListTraits<T>::value, Input &>::type
 operator>>(Input &yin, T &docList) {
   int i = 0;
#define LLVM_YAML_DECLARE_BITSET_TRAITS(Type)                                  \
   }
 
 #define LLVM_YAML_DECLARE_SCALAR_TRAITS(Type, MustQuote)                       \
   namespace llvm {                                                             \
   namespace yaml {                                                             \
   template <> struct ScalarTraits<Type> {                                      \
     static void output(const Type &Value, void *ctx, raw_ostream &Out);        \
     static StringRef input(StringRef Scalar, void *ctxt, Type &Value);         \
-    static bool mustQuote(StringRef) { return MustQuote; }                     \
+    static QuotingType mustQuote(StringRef) { return MustQuote; }         \
   };                                                                           \
   }                                                                            \
   }
 
 /// Utility for declaring that a std::vector of a particular type
 /// should be considered a YAML document list.
 #define LLVM_YAML_IS_DOCUMENT_LIST_VECTOR(_type)                               \
   namespace llvm {                                                             \

llvm/trunk/lib/ObjectYAML/CodeViewYAMLDebugSections.cpp

 LLVM_YAML_IS_SEQUENCE_VECTOR(SourceLineBlock)
 LLVM_YAML_IS_SEQUENCE_VECTOR(SourceLineInfo)
 LLVM_YAML_IS_SEQUENCE_VECTOR(InlineeSite)
 LLVM_YAML_IS_SEQUENCE_VECTOR(InlineeInfo)
 LLVM_YAML_IS_SEQUENCE_VECTOR(CrossModuleExport)
 LLVM_YAML_IS_SEQUENCE_VECTOR(YAMLCrossModuleImport)
 LLVM_YAML_IS_SEQUENCE_VECTOR(YAMLFrameData)
 
-LLVM_YAML_DECLARE_SCALAR_TRAITS(HexFormattedString, false)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(HexFormattedString, QuotingType::None)
 LLVM_YAML_DECLARE_ENUM_TRAITS(DebugSubsectionKind)
 LLVM_YAML_DECLARE_ENUM_TRAITS(FileChecksumKind)
 LLVM_YAML_DECLARE_BITSET_TRAITS(LineFlags)
 
 LLVM_YAML_DECLARE_MAPPING_TRAITS(CrossModuleExport)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(YAMLFrameData)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(YAMLCrossModuleImport)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(CrossModuleImportItem)

llvm/trunk/lib/ObjectYAML/CodeViewYAMLSymbols.cpp

 using namespace llvm::codeview;
 using namespace llvm::CodeViewYAML;
 using namespace llvm::CodeViewYAML::detail;
 using namespace llvm::yaml;
 
 LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR(TypeIndex)
 
 // We only need to declare these, the definitions are in CodeViewYAMLTypes.cpp
-LLVM_YAML_DECLARE_SCALAR_TRAITS(APSInt, false)
-LLVM_YAML_DECLARE_SCALAR_TRAITS(TypeIndex, false)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(APSInt, QuotingType::None)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(TypeIndex, QuotingType::None)
 
 LLVM_YAML_DECLARE_ENUM_TRAITS(SymbolKind)
 LLVM_YAML_DECLARE_ENUM_TRAITS(FrameCookieKind)
 
 LLVM_YAML_DECLARE_BITSET_TRAITS(CompileSym2Flags)
 LLVM_YAML_DECLARE_BITSET_TRAITS(CompileSym3Flags)
 LLVM_YAML_DECLARE_BITSET_TRAITS(ExportFlags)
 LLVM_YAML_DECLARE_BITSET_TRAITS(PublicSymFlags)
 LLVM_YAML_DECLARE_BITSET_TRAITS(LocalSymFlags)
 LLVM_YAML_DECLARE_BITSET_TRAITS(ProcSymFlags)
 LLVM_YAML_DECLARE_BITSET_TRAITS(FrameProcedureOptions)
 LLVM_YAML_DECLARE_ENUM_TRAITS(CPUType)
 LLVM_YAML_DECLARE_ENUM_TRAITS(RegisterId)
 LLVM_YAML_DECLARE_ENUM_TRAITS(TrampolineType)
 LLVM_YAML_DECLARE_ENUM_TRAITS(ThunkOrdinal)
 
 LLVM_YAML_STRONG_TYPEDEF(StringRef, TypeName)
 
-LLVM_YAML_DECLARE_SCALAR_TRAITS(TypeName, true)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(TypeName, QuotingType::Single)
 
 StringRef ScalarTraits<TypeName>::input(StringRef S, void *V, TypeName &T) {
   return ScalarTraits<StringRef>::input(S, V, T.value);
 }
 
 void ScalarTraits<TypeName>::output(const TypeName &T, void *V,
                                     raw_ostream &R) {
   ScalarTraits<StringRef>::output(T.value, V, R);

llvm/trunk/lib/ObjectYAML/CodeViewYAMLTypes.cpp

 using namespace llvm::CodeViewYAML;
 using namespace llvm::CodeViewYAML::detail;
 using namespace llvm::yaml;
 
 LLVM_YAML_IS_SEQUENCE_VECTOR(OneMethodRecord)
 LLVM_YAML_IS_SEQUENCE_VECTOR(VFTableSlotKind)
 LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR(TypeIndex)
 
-LLVM_YAML_DECLARE_SCALAR_TRAITS(TypeIndex, false)
-LLVM_YAML_DECLARE_SCALAR_TRAITS(APSInt, false)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(TypeIndex, QuotingType::None)
+LLVM_YAML_DECLARE_SCALAR_TRAITS(APSInt, QuotingType::None)
 
 LLVM_YAML_DECLARE_ENUM_TRAITS(TypeLeafKind)
 LLVM_YAML_DECLARE_ENUM_TRAITS(PointerToMemberRepresentation)
 LLVM_YAML_DECLARE_ENUM_TRAITS(VFTableSlotKind)
 LLVM_YAML_DECLARE_ENUM_TRAITS(CallingConvention)
 LLVM_YAML_DECLARE_ENUM_TRAITS(PointerKind)
 LLVM_YAML_DECLARE_ENUM_TRAITS(PointerMode)
 LLVM_YAML_DECLARE_ENUM_TRAITS(HfaKind)

llvm/trunk/lib/ObjectYAML/MachOYAML.cpp

StringRef ScalarTraits<char_16>::input(StringRef Scalar, void *, char_16 &Val) {
 
   if (Scalar.size() < 16) {
     memset((void *)&Val[Scalar.size()], 0, 16 - Scalar.size());
   }
 
   return StringRef();
 }
 
-bool ScalarTraits<char_16>::mustQuote(StringRef S) { return needsQuotes(S); }
+QuotingType ScalarTraits<char_16>::mustQuote(StringRef S) {
+  return needsQuotes(S);
+}
 
 void ScalarTraits<uuid_t>::output(const uuid_t &Val, void *, raw_ostream &Out) {
   Out.write_uuid(Val);
 }
 
 StringRef ScalarTraits<uuid_t>::input(StringRef Scalar, void *, uuid_t &Val) {
   size_t OutIdx = 0;
   for (size_t Idx = 0; Idx < Scalar.size(); ++Idx) {
     if (Scalar[Idx] == '-' || OutIdx >= 16)
       continue;
     unsigned long long TempInt;
     if (getAsUnsignedInteger(Scalar.slice(Idx, Idx + 2), 16, TempInt))
       return "invalid number";
     if (TempInt > 0xFF)
       return "out of range number";
     Val[OutIdx] = static_cast<uint8_t>(TempInt);
     ++Idx; // increment idx an extra time because we're consuming 2 chars
     ++OutIdx;
   }
   return StringRef();
 }
 
-bool ScalarTraits<uuid_t>::mustQuote(StringRef S) { return needsQuotes(S); }
+QuotingType ScalarTraits<uuid_t>::mustQuote(StringRef S) {
+  return needsQuotes(S);
+}
 
 void MappingTraits<MachOYAML::FileHeader>::mapping(
     IO &IO, MachOYAML::FileHeader &FileHdr) {
   IO.mapRequired("magic", FileHdr.magic);
   IO.mapRequired("cputype", FileHdr.cputype);
   IO.mapRequired("cpusubtype", FileHdr.cpusubtype);
   IO.mapRequired("filetype", FileHdr.filetype);
   IO.mapRequired("ncmds", FileHdr.ncmds);

llvm/trunk/lib/Support/Statistic.cpp

void llvm::PrintStatisticsJSON(raw_ostream &OS) {
 
   Stats.sort();
 
   // Print all of the statistics.
   OS << "{\n";
   const char *delim = "";
   for (const Statistic *Stat : Stats.Stats) {
     OS << delim;
-    assert(!yaml::needsQuotes(Stat->getDebugType()) &&
+    assert(yaml::needsQuotes(Stat->getDebugType()) == yaml::QuotingType::None &&
            "Statistic group/type name is simple.");
-    assert(!yaml::needsQuotes(Stat->getName()) && "Statistic name is simple");
+    assert(yaml::needsQuotes(Stat->getName()) == yaml::QuotingType::None &&
+           "Statistic name is simple");
     OS << "\t\"" << Stat->getDebugType() << '.' << Stat->getName() << "\": "
        << Stat->getValue();
     delim = ",\n";
   }
   // Print timers.
   TimerGroup::printAllJSONValues(OS, delim);
 
   OS << "\n}\n";

llvm/trunk/lib/Support/Timer.cpp

void TimerGroup::printAll(raw_ostream &OS) {
   sys::SmartScopedLock<true> L(*TimerLock);
 
   for (TimerGroup *TG = TimerGroupList; TG; TG = TG->Next)
     TG->print(OS);
 }
 
 void TimerGroup::printJSONValue(raw_ostream &OS, const PrintRecord &R,
                                 const char *suffix, double Value) {
-  assert(!yaml::needsQuotes(Name) && "TimerGroup name needs no quotes");
-  assert(!yaml::needsQuotes(R.Name) && "Timer name needs no quotes");
+  assert(yaml::needsQuotes(Name) == yaml::QuotingType::None &&
+         "TimerGroup name needs no quotes");
+  assert(yaml::needsQuotes(R.Name) == yaml::QuotingType::None &&
+         "Timer name needs no quotes");
   OS << "\t\"time." << Name << '.' << R.Name << suffix << "\": " << Value;
 }
 
 const char *TimerGroup::printJSONValues(raw_ostream &OS, const char *delim) {
   prepareToPrintList();
   for (const PrintRecord &R : TimersToPrint) {
     OS << delim;
     delim = ",\n";

llvm/trunk/lib/Support/YAMLTraits.cpp

 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/LineIterator.h"
 #include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Unicode.h"
 #include "llvm/Support/YAMLParser.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <cstdlib>
 #include <cstring>
 #include <string>
for (unsigned i = 0; i < SQ->Entries.size(); ++i) {
       if (!BitValuesUsed[i]) {
         setError(SQ->Entries[i].get(), "unknown bit value");
         return;
       }
     }
   }
 }
 
-void Input::scalarString(StringRef &S, bool) {
+void Input::scalarString(StringRef &S, QuotingType) {
   if (ScalarHNode *SN = dyn_cast<ScalarHNode>(CurrentNode)) {
     S = SN->value();
   } else {
     setError(CurrentNode, "unexpected scalar");
   }
 }
 
-void Input::blockScalarString(StringRef &S) { scalarString(S, false); }
+void Input::blockScalarString(StringRef &S) { scalarString(S, QuotingType::None); }
 
 void Input::setError(HNode *hnode, const Twine &message) {
   assert(hnode && "HNode must not be NULL");
   this->setError(hnode->_node, message);
 }
 
 void Input::setError(Node *node, const Twine &message) {
   Strm->printError(node, message);
bool Output::bitSetMatch(const char *Str, bool Matches) {
   }
   return false;
 }
 
 void Output::endBitSetScalar() {
   this->outputUpToEndOfLine(" ]");
 }
 
-void Output::scalarString(StringRef &S, bool MustQuote) {
+void Output::scalarString(StringRef &S, QuotingType MustQuote) {
   this->newLineCheck();
   if (S.empty()) {
     // Print '' for the empty string because leaving the field empty is not
     // allowed.
     this->outputUpToEndOfLine("''");
     return;
   }
-  if (!MustQuote) {
+  if (MustQuote == QuotingType::None) {
     // Only quote if we must.
     this->outputUpToEndOfLine(S);
     return;
   }
+
   unsigned i = 0;
   unsigned j = 0;
   unsigned End = S.size();
-  output("'"); // Starting single quote.
   const char *Base = S.data();
+
+  const char *const Quote = MustQuote == QuotingType::Single ? "'" : "\"";
+  const char QuoteChar = MustQuote == QuotingType::Single ? '\'' : '"';
+
+  output(Quote); // Starting quote.
+
+  // When using single-quoted strings, any single quote ' must be doubled to be
+  // escaped.
+  // When using double-quoted strings, print \x + hex for non-printable ASCII
+  // characters, and escape double quotes.
   while (j < End) {
-    // Escape a single quote by doubling it.
-    if (S[j] == '\'') {
-      output(StringRef(&Base[i], j - i + 1));
-      output("'");
+    if (S[j] == QuoteChar) {                  // Escape quotes.
+      output(StringRef(&Base[i], j - i));     // "flush".
+      if (MustQuote == QuotingType::Double) { // Print it as \"
+        output(StringLiteral("\\"));
+        output(StringRef(Quote, 1));
+      } else {                       // Single
+        output(StringLiteral("''")); // Print it as ''
+      }
+      i = j + 1;
+    } else if (MustQuote == QuotingType::Double &&
+               !sys::unicode::isPrintable(S[j])) {
+      output(StringRef(&Base[i], j - i)); // "flush"
+      output(StringLiteral("\\x"));
+
+      // Output the byte 0x0F as \x0f.
+      auto FormattedHex = format_hex_no_prefix(S[j], 2);
+      Out << FormattedHex;
+      Column += 4; // one for the '\', one for the 'x', and two for the hex
+
       i = j + 1;
     }
     ++j;
   }
   output(StringRef(&Base[i], j - i));
-  this->outputUpToEndOfLine("'"); // Ending single quote.
+  this->outputUpToEndOfLine(Quote); // Ending quote.
 }
 
 void Output::blockScalarString(StringRef &S) {
   if (!StateStack.empty())
     newLineCheck();
   output(" |");
   outputNewLine();
 

llvm/trunk/test/CodeGen/MIR/X86/escape-function-name.ll

+; RUN: llc -mtriple=x86_64-unknown-unknown -stop-after branch-folder -o - %s 2>&1 | FileCheck %s
+
+define void @"\01?f@@YAXXZ"() {
+; CHECK: name: "\x01?f@@YAXXZ"
+  ret void
+}

llvm/trunk/tools/dsymutil/DebugMap.h

 template <> struct MappingTraits<dsymutil::DebugMapObject> {
   struct YamlDMO;
   static void mapping(IO &io, dsymutil::DebugMapObject &DMO);
 };
 
 template <> struct ScalarTraits<Triple> {
   static void output(const Triple &val, void *, raw_ostream &out);
   static StringRef input(StringRef scalar, void *, Triple &value);
-  static bool mustQuote(StringRef) { return true; }
+  static QuotingType mustQuote(StringRef) { return QuotingType::Single; }
 };
 
 template <>
 struct SequenceTraits<std::vector<std::unique_ptr<dsymutil::DebugMapObject>>> {
   static size_t
   size(IO &io, std::vector<std::unique_ptr<dsymutil::DebugMapObject>> &seq);
   static dsymutil::DebugMapObject &
   element(IO &, std::vector<std::unique_ptr<dsymutil::DebugMapObject>> &seq,

llvm/trunk/unittests/Support/YAMLIOTest.cpp

static StringRef input(StringRef scalar, void* ctxt, MyCustomType &value) {
           return "malformed width";
         }
         return StringRef();
       }
       else {
           return "malformed by";
       }
     }
-    static bool mustQuote(StringRef) { return true; }
+    static QuotingType mustQuote(StringRef) { return QuotingType::Single; }
   };
 }
 }
 
 
 //
 // Test writing then reading back custom values
 //
struct ScalarTraits<MyNumber> {
     static StringRef input(StringRef scalar, void *, MyNumber &value) {
       long long n;
       if ( getAsSignedInteger(scalar, 0, n) )
         return "invalid number";
       value = n;
       return StringRef();
     }
 
-    static bool mustQuote(StringRef) { return false; }
+    static QuotingType mustQuote(StringRef) { return QuotingType::None; }
   };
 
   template <> struct ScalarTraits<MyString> {
     using Impl = ScalarTraits<StringRef>;
     static void output(const MyString &V, void *Ctx, raw_ostream &OS) {
       Impl::output(V, Ctx, OS);
     }
     static StringRef input(StringRef S, void *Ctx, MyString &V) {
       return Impl::input(S, Ctx, V.value);
     }
-    static bool mustQuote(StringRef S) { return Impl::mustQuote(S); }
+    static QuotingType mustQuote(StringRef S) {
+      return Impl::mustQuote(S);
+    }
   };
 }
 }
 
 struct NameAndNumbers {
   llvm::StringRef               name;
   std::vector<MyString>         strings;
   std::vector<MyNumber>         single;
struct ScalarTraits<FlowSeq> {
   static void output(const FlowSeq &value, void*, llvm::raw_ostream &out) {
     out << value.str;
   }
   static StringRef input(StringRef scalar, void*, FlowSeq &value) {
     value.str = scalar;
     return "";
   }
 
-  static bool mustQuote(StringRef S) { return false; }
+  static QuotingType mustQuote(StringRef S) { return QuotingType::None; }
 };
 }
 }
 
 LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR(FlowSeq)
 
 TEST(YAMLIO, TestWrapFlow) {
   std::string out;

 
 TEST(YAMLIO, InvalidInput) {
   // polluting 1 value in the sequence
   Input yin("---\n- foo:  3\n  bar:  5\n1\n- foo:  3\n  bar:  5\n...\n");
   std::vector<FooBar> Data;
   yin >> Data;
   EXPECT_TRUE((bool)yin.error());
 }
+
+TEST(YAMLIO, TestEscapedSingleQuote) {
+  std::string Id = "@abc@";
+
+  std::string out;
+  llvm::raw_string_ostream ostr(out);
+  Output xout(ostr, nullptr, 0);
+
+  llvm::yaml::EmptyContext Ctx;
+  yamlize(xout, Id, true, Ctx);
+
+  ostr.flush();
+  EXPECT_EQ("'@abc@'", out);
+}
+
+TEST(YAMLIO, TestEscapedNoQuote) {
+  std::string Id = "abc/";
+
+  std::string out;
+  llvm::raw_string_ostream ostr(out);
+  Output xout(ostr, nullptr, 0);
+
+  llvm::yaml::EmptyContext Ctx;
+  yamlize(xout, Id, true, Ctx);
+
+  ostr.flush();
+  EXPECT_EQ("abc/", out);
+}
+
+TEST(YAMLIO, TestEscapedDoubleQuoteNonPrintable) {
+  std::string Id = "\01@abc@";
+
+  std::string out;
+  llvm::raw_string_ostream ostr(out);
+  Output xout(ostr, nullptr, 0);
+
+  llvm::yaml::EmptyContext Ctx;
+  yamlize(xout, Id, true, Ctx);
+
+  ostr.flush();
+  EXPECT_EQ("\"\\x01@abc@\"", out);
+}
+
+TEST(YAMLIO, TestEscapedDoubleQuoteInsideSingleQuote) {
+  std::string Id = "abc\"fdf";
+
+  std::string out;
+  llvm::raw_string_ostream ostr(out);
+  Output xout(ostr, nullptr, 0);
+
+  llvm::yaml::EmptyContext Ctx;
+  yamlize(xout, Id, true, Ctx);
+
+  ostr.flush();
+  EXPECT_EQ("'abc\"fdf'", out);
+}
+
+TEST(YAMLIO, TestEscapedDoubleQuoteInsideDoubleQuote) {
+  std::string Id = "\01bc\"fdf";
+
+  std::string out;
+  llvm::raw_string_ostream ostr(out);
+  Output xout(ostr, nullptr, 0);
+
+  llvm::yaml::EmptyContext Ctx;
+  yamlize(xout, Id, true, Ctx);
+
+  ostr.flush();
+  EXPECT_EQ("\"\\x01bc\\\"fdf\"", out);
+}
+
+TEST(YAMLIO, TestEscapedSingleQuoteInsideSingleQuote) {
+  std::string Id = "abc'fdf";
+
+  std::string out;
+  llvm::raw_string_ostream ostr(out);
+  Output xout(ostr, nullptr, 0);
+
+  llvm::yaml::EmptyContext Ctx;
+  yamlize(xout, Id, true, Ctx);
+
+  ostr.flush();
+  EXPECT_EQ("'abc''fdf'", out);
+}