[support] copying JSON parser/writer from lldb

[aizatsky]

Summary

This CL contains a copy of JSON parser/writer that can be found
in the lldb subproject with only necessary changes:

• files now follow LLVM naming and formatting guidelines
• raw_ostream used instead of the special formatted stream
• very basic parse/print roundtrip test for JSON.

As soon as this CL lands I will prepare the one that removes JSON
parser from lldb/ and uses parent's.

[labath]

I am cc'ing a bunch of people to make sure we don't do this behind anyone's back. I don't know whether you discussed this with anyone on lldb side (it's certainly new to me), but I think it deserves a wider audience, as this is the first case (I can recall) of moving code from lldb back to llvm.

That said, I think it's a great idea. However, you should sync up with @zturner, as he has imminent plans to do a StringExtractor refactor (D24013) within lldb and you're making a copy of it here.

Some other high-level questions (not directly related to this change, but I'd like to know where you're going with this):

• despite the renaming, the code still has a distinct lldb feel to it. I presume this will be followed by a bunch of CL's to bring it more in line with the rest of llvm (?)
• you are copying StringExtractor to llvm as well. Will you switch lldb to use that copy as well, or is the plan to bypass it completely and use llvm parsing primitives for the job?
• were you able to get lldb test suite running? If not, I can help you get that working.
• mostly out of curiosity: what is the motivation for this? I guess you have a need for a JSON parser in llvm ?

cheers.

[zturner]

Having a JSON parser in LLVM seems like a reasonable idea, but I'm afraid the JSON parser that currently exists in LLDB won't do. As Pavel mentioned, the code has a definite LLDB feel to it, but more importantly it depends on StringExtractor which I think probably doesn't belong in LLVM. If you want to go down this route, I would start by deleting the StringExtractor changes from this CL. 90% of the stuff StringExtractor is used for, like hex string to number, string to integer, etc are covered by existing LLVM classes or methods.

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

[aizatsky]

In D24369#538213, @zturner wrote:

Having a JSON parser in LLVM seems like a reasonable idea, but I'm afraid the JSON parser that currently exists in LLDB won't do. As Pavel mentioned, the code has a definite LLDB feel to it, but more importantly it depends on StringExtractor which I think probably doesn't belong in LLVM. If you want to go down this route, I would start by deleting the StringExtractor changes from this CL. 90% of the stuff StringExtractor is used for, like hex string to number, string to integer, etc are covered by existing LLVM classes or methods.

My rationale was that reusing parser from a child project is better than building anything new from scratch.

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

I can't actually find json parser within clang-tidy. Only references that I see lead eventually to YAML parsing:

tool/ClangTidyMain.cpp
102:JSON array of objects:
138:Specifies a configuration in YAML/JSON format:

ClangTidyOptions.cpp
36:// Map std::pair<int, int> to a JSON array of size 2.

ClangTidyOptions.h
245:/// \brief Parses LineFilter from JSON and stores it to the \p Options.
249:/// \brief Parses configuration from JSON and returns \c ClangTidyOptions or an

[zturner]

In D24369#538422, @aizatsky wrote:

In D24369#538213, @zturner wrote:

Having a JSON parser in LLVM seems like a reasonable idea, but I'm afraid the JSON parser that currently exists in LLDB won't do. As Pavel mentioned, the code has a definite LLDB feel to it, but more importantly it depends on StringExtractor which I think probably doesn't belong in LLVM. If you want to go down this route, I would start by deleting the StringExtractor changes from this CL. 90% of the stuff StringExtractor is used for, like hex string to number, string to integer, etc are covered by existing LLVM classes or methods.

My rationale was that reusing parser from a child project is better than building anything new from scratch.

Agree that we always want to reuse code wherever possible, but LLVM has stricter requirements on code style and class design. The JSON stuff itself can probably be a good starting point since clang-tidy doesn't have one (see my comment later), but I don't think StringExtractor belongs in LLVM. Looking at the code, JSON really only uses 1 or 2 methods from `StringExtractor anyway, so I don't think it's a big thing to overcome.

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

I can't actually find json parser within clang-tidy. Only references that I see lead eventually to YAML parsing:

I was thinking of lib/Tooling/JSONCompilationDatabase, but looking at it, it seems to be pretty specialized and doesn't support arbitrary JSON.

[clayborg]

I like the idea of moving this down into LLVM. We should fully LLVM-ize the code on the first pass though and take the LLDB style out of it.

[aizatsky]

• getting rid of StringExtractor
• renaming methods to lowerCase.

[aizatsky]

In D24369#538472, @zturner wrote:

In D24369#538422, @aizatsky wrote:

In D24369#538213, @zturner wrote:

Having a JSON parser in LLVM seems like a reasonable idea, but I'm afraid the JSON parser that currently exists in LLDB won't do. As Pavel mentioned, the code has a definite LLDB feel to it, but more importantly it depends on StringExtractor which I think probably doesn't belong in LLVM. If you want to go down this route, I would start by deleting the StringExtractor changes from this CL. 90% of the stuff StringExtractor is used for, like hex string to number, string to integer, etc are covered by existing LLVM classes or methods.

My rationale was that reusing parser from a child project is better than building anything new from scratch.

Agree that we always want to reuse code wherever possible, but LLVM has stricter requirements on code style and class design. The JSON stuff itself can probably be a good starting point since clang-tidy doesn't have one (see my comment later), but I don't think StringExtractor belongs in LLVM. Looking at the code, JSON really only uses 1 or 2 methods from `StringExtractor anyway, so I don't think it's a big thing to overcome.

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

I can't actually find json parser within clang-tidy. Only references that I see lead eventually to YAML parsing:

I was thinking of lib/Tooling/JSONCompilationDatabase, but looking at it, it seems to be pretty specialized and doesn't support arbitrary JSON.

I got rid of StringExtractor and also renamed all methods to follow the guidelines.
Let me know which other style inconsistencies you see there.

[aizatsky]

remove .clang-tidy

[zturner]

include/llvm/Support/JSON.h

27

I scanned the entire patch and I'm not sure I understand the need for the covariant typedefs. The only type that ever seems to be used is JSONValue, so it seems to me like the other types are unnecessary.

Also, I'm not sure shared_ptr is necessary. Wouldn't unique_ptr work just fine? Then you could vend references to values instead of pointers.

47

I would make this a StringRef. Currently if someone calls this with a StringRef, it will make a copy. If you change the constructor to take a StringRef, then it won't take a copy regardless of whether it's called with a StringRef or a std::string.

56

I would return a StringRef here. Otherwise you're guaranteed to make a copy, if you return a StringRef it will only make a copy when you assign it to a std::string.

63

This could be a file static in the .cpp file, and also it could take a StringRef instead of a const std::string &.

202

Couple things here.

1. Key could be a StringRef
2. This method should probably be const.
3. You could make the underlying value a unique_ptr instead of a shared_ptr and return a reference. You could also have a tryGetValue in cases where you aren't sure if the value exists. Not sure what would be better.

207

DenseMap seems like a better choice here.

229–231

I don't know if the typedefs here help or hurt readability. I would just make them size_t, and remove the Iterator typedef as it doesn't seem to be used.

238

How about providing an overloaded operator[] and providing begin() and end() functions so that it can be used in a range based for syntax?

282–283

Instead of having a std::string here, which will copy the input JSON, I would make this a StringRef. Then you can delete the Index field as well, because the internal StringRef itself could always point to the beginning of the next char by using a Buffer = Buffer.drop_front(N) like pattern. This won't work if you ever have to do back-referencing across functions, but I don't think that is needed. Also if Buffer is a StringRef, a lot of the methods like peekChar() and skipSpaces become trivial one-liners that you wouldn't even need to provide a separate function for.

lib/Support/JSON.cpp

19

Input parameter should be a StringRef.

23

Should probably do an Output.reserve(S.size());

24–26

for (char CH : S)

53

Can you use C++ style casting operators here and in other similar functions?

115–116

for (const auto &I : Elements) {

187

This function should be const.

283

Don't like the idea of reimplementing string -> number parsing, but I'm not sure if LLVM has something to solve this for us.

[chandlerc]

In D24369#538472, @zturner wrote:

In D24369#538422, @aizatsky wrote:

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

There is already a JSON parser in LLVM.

I can't actually find json parser within clang-tidy. Only references that I see lead eventually to YAML parsing:

I was thinking of lib/Tooling/JSONCompilationDatabase, but looking at it, it seems to be pretty specialized and doesn't support arbitrary JSON.

It uses the YAML parser because JSON is a strict subset of YAML: https://en.wikipedia.org/wiki/JSON#YAML

Please don't add a JSON parser to LLVM. I would much rather teach LLDB to use the YAML parser and improve that parser if and where useful.

[aizatsky]

cleanup

[aizatsky]

include/llvm/Support/JSON.h

207

DenseMap doesn't seem to be defined for std::string keys.

238

done for operator[].

begin() and end() will be tricky, because underlying storage has std::unique_ptr<>

282–283

std::string => StringRef - done.

As for removing Index - I'm not sure. It is currently used in error reporting and is the only feedback we give when we encounter an error.

lib/Support/JSON.cpp

53

Removed all 3 them and added get<T> method.

include/llvm/Support/JSON.h

+//===---------------------JSON.h --------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_JSON_H
+#define LLVM_SUPPORT_JSON_H
+
+#include <inttypes.h>
+#include <map>
+#include <memory>
+#include <stdint.h>
+#include <string>
+#include <vector>
+
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace llvm {
+
+class JSONValue {
+public:
+  typedef std::shared_ptr<JSONValue> SP;
+  enum class Kind { String, Number, True, False, Null, Object, Array };
+
+  JSONValue(Kind K) : TheKind(K) {}
+  virtual ~JSONValue() = default;
+
+  virtual void write(raw_ostream &S) const = 0;
+
+  Kind getKind() const { return TheKind; }
+
+private:
+  const Kind TheKind;
+};
+
+class JSONString : public JSONValue {
+public:
+  typedef std::shared_ptr<JSONString> SP;
+
+  JSONString();
+  JSONString(const char *S);
+  JSONString(const std::string &S);
+
+  JSONString(const JSONString &S) = delete;
+  JSONString &operator=(const JSONString &S) = delete;
+
+  ~JSONString() override = default;
+
+  void write(raw_ostream &S) const override;
+
+  std::string getData() const { return Data; }
+
+  static bool classof(const JSONValue *V) {
+    return V->getKind() == JSONValue::Kind::String;
+  }
+
+private:
+  static std::string jsonStringQuoteMetachars(const std::string &);
+
+  std::string Data;
+};
+
+class JSONNumber : public JSONValue {
+public:
+  typedef std::shared_ptr<JSONNumber> SP;
+
+  // We cretae a constructor for all integer and floating point type with using
+  // templates and
+  // SFINAE to avoid having ambiguous overloads because of the implicit type
+  // promotion. If we
+  // would have constructors only with int64_t, uint64_t and double types then
+  // constructing a
+  // JSONNumber from an int32_t (or any other similar type) would fail to
+  // compile.
+
+  template <typename T, typename std::enable_if<
+                            std::is_integral<T>::value &&
+                            std::is_unsigned<T>::value>::type * = nullptr>
+  explicit JSONNumber(T U)
+      : JSONValue(JSONValue::Kind::Number), TheDataType(DataType::Unsigned) {
+    Data.Unsigned = U;
+  }
+
+  template <typename T,
+            typename std::enable_if<std::is_integral<T>::value &&
+                                    std::is_signed<T>::value>::type * = nullptr>
+  explicit JSONNumber(T S)
+      : JSONValue(JSONValue::Kind::Number), TheDataType(DataType::Signed) {
+    Data.Signed = S;
+  }
+
+  template <typename T, typename std::enable_if<
+                            std::is_floating_point<T>::value>::type * = nullptr>
+  explicit JSONNumber(T D)
+      : JSONValue(JSONValue::Kind::Number), TheDataType(DataType::Double) {
+    Data.Double = D;
+  }
+
+  ~JSONNumber() override = default;
+
+  JSONNumber(const JSONNumber &S) = delete;
+  JSONNumber &operator=(const JSONNumber &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  uint64_t getAsUnsigned() const;
+
+  int64_t getAsSigned() const;
+
+  double getAsDouble() const;
+
+  static bool classof(const JSONValue *V) {
+    return V->getKind() == JSONValue::Kind::Number;
+  }
+
+private:
+  enum class DataType : uint8_t { Unsigned, Signed, Double } TheDataType;
+
+  union {
+    uint64_t Unsigned;
+    int64_t Signed;
+    double Double;
+  } Data;
+};
+
+class JSONTrue : public JSONValue {
+public:
+  JSONTrue();
+
+  JSONTrue(const JSONTrue &S) = delete;
+  JSONTrue &operator=(const JSONTrue &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  typedef std::shared_ptr<JSONTrue> SP;
+
+  static bool classof(const JSONValue *V) {
+    return V->getKind() == JSONValue::Kind::True;
+  }
+
+  ~JSONTrue() override = default;
+};
+
+class JSONFalse : public JSONValue {
+public:
+  JSONFalse();
+
+  JSONFalse(const JSONFalse &S) = delete;
+  JSONFalse &operator=(const JSONFalse &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  typedef std::shared_ptr<JSONFalse> SP;
+
+  static bool classof(const JSONValue *V) {
+    return V->getKind() == JSONValue::Kind::False;
+  }
+
+  ~JSONFalse() override = default;
+};
+
+class JSONNull : public JSONValue {
+public:
+  JSONNull();
+
+  JSONNull(const JSONNull &S) = delete;
+  JSONNull &operator=(const JSONNull &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  typedef std::shared_ptr<JSONNull> SP;
+
+  static bool classof(const JSONValue *V) {
+    return V->getKind() == JSONValue::Kind::Null;
+  }
+
+  ~JSONNull() override = default;
+};
+
+class JSONObject : public JSONValue {
+public:
+  JSONObject();
+
+  JSONObject(const JSONObject &S) = delete;
+  JSONObject &operator=(const JSONObject &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  typedef std::shared_ptr<JSONObject> SP;
+
+  static bool classof(const JSONValue *V) {
+    return V->getKind() == JSONValue::Kind::Object;
+  }
+
+  bool setObject(const std::string &Key, JSONValue::SP Value);
+
+  JSONValue::SP getObject(const std::string &Key);
+
+  ~JSONObject() override = default;
+
+private:
+  typedef std::map<std::string, JSONValue::SP> Map;
+  typedef Map::iterator Iterator;
+  Map Elements;
+};
+
+class JSONArray : public JSONValue {
+public:
+  JSONArray();
+
+  JSONArray(const JSONArray &S) = delete;
+  JSONArray &operator=(const JSONArray &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  typedef std::shared_ptr<JSONArray> SP;
+
+  static bool classof(const JSONValue *V) {
+    return V->getKind() == JSONValue::Kind::Array;
+  }
+
+private:
+  typedef std::vector<JSONValue::SP> Vector;
+  typedef Vector::iterator Iterator;
+  typedef Vector::size_type Index;
+  typedef Vector::size_type Size;
+
+public:
+  bool setObject(Index I, JSONValue::SP Value);
+
+  bool appendObject(JSONValue::SP Value);
+
+  JSONValue::SP getObject(Index I);
+
+  Size getNumElements();
+
+  ~JSONArray() override = default;
+
+  Vector Elements;
+};
+
+class JSONParser {
+public:
+  enum Token {
+    Invalid,
+    Error,
+    ObjectStart,
+    ObjectEnd,
+    ArrayStart,
+    ArrayEnd,
+    Comma,
+    Colon,
+    String,
+    Integer,
+    Float,
+    True,
+    False,
+    Null,
+    EndOfFile
+  };
+
+  JSONParser(const char *Cstr);
+  JSONValue::SP parseJSONValue();
+
+protected:
+  int getEscapedChar(bool &WasEscaped);
+  Token getToken(std::string &Value);
+  JSONValue::SP parseJSONObject();
+  JSONValue::SP parseJSONArray();
+
+  void skipSpaces();
+  uint8_t decodeHexU8();
+  char getChar();
+  char peekChar() const;
+  size_t getBytesLeft() const { return Buffer.size() - Index; }
+
+  std::string Buffer;
+  size_t Index;
+};
+} // namespace llvm
+
+#endif // LLVM_SUPPORT_JSON_H

lib/Support/CMakeLists.txt

add_llvm_library(LLVMSupport
   FoldingSet.cpp
   FormattedStream.cpp
   GraphWriter.cpp
   Hashing.cpp
   IntEqClasses.cpp
   IntervalMap.cpp
   IntrusiveRefCntPtr.cpp
   JamCRC.cpp
+  JSON.cpp
   LEB128.cpp
   LineIterator.cpp
   Locale.cpp
   LockFileManager.cpp
   ManagedStatic.cpp
   MathExtras.cpp
   MemoryBuffer.cpp
   MemoryObject.cpp

lib/Support/JSON.cpp

+//===--------------------- JSON.cpp -----------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/JSON.h"
+
+#include <limits.h>
+#include <sstream>
+
+#include "llvm/Support/ErrorHandling.h"
+
+using namespace llvm;
+
+std::string JSONString::jsonStringQuoteMetachars(const std::string &S) {
+  if (S.find('"') == std::string::npos)
+    return S;
+
+  std::string Output;
+  const size_t Size = S.size();
+  const char *Chars = S.c_str();
+  for (size_t I = 0; I < Size; I++) {
+    unsigned char Ch = *(Chars + I);
+    if (Ch == '"') {
+      Output.push_back('\\');
+    }
+    Output.push_back(Ch);
+  }
+  return Output;
+}
+
+JSONString::JSONString() : JSONValue(JSONValue::Kind::String), Data() {}
+
+JSONString::JSONString(const char *S)
+    : JSONValue(JSONValue::Kind::String), Data(S ? S : "") {}
+
+JSONString::JSONString(const std::string &S)
+    : JSONValue(JSONValue::Kind::String), Data(S) {}
+
+void JSONString::write(raw_ostream &OS) const {
+  OS << "\"" << jsonStringQuoteMetachars(Data) << "\"";
+}
+
+uint64_t JSONNumber::getAsUnsigned() const {
+  switch (TheDataType) {
+  case DataType::Unsigned:
+    return Data.Unsigned;
+  case DataType::Signed:
+    return (uint64_t)Data.Signed;
+  case DataType::Double:
+    return (uint64_t)Data.Double;
+  }
+  llvm_unreachable("Unhandled data type");
+}
+
+int64_t JSONNumber::getAsSigned() const {
+  switch (TheDataType) {
+  case DataType::Unsigned:
+    return (int64_t)Data.Unsigned;
+  case DataType::Signed:
+    return Data.Signed;
+  case DataType::Double:
+    return (int64_t)Data.Double;
+  }
+  llvm_unreachable("Unhandled data type");
+}
+
+double JSONNumber::getAsDouble() const {
+  switch (TheDataType) {
+  case DataType::Unsigned:
+    return (double)Data.Unsigned;
+  case DataType::Signed:
+    return (double)Data.Signed;
+  case DataType::Double:
+    return Data.Double;
+  }
+  llvm_unreachable("Unhandled data type");
+}
+
+void JSONNumber::write(raw_ostream &OS) const {
+  switch (TheDataType) {
+  case DataType::Unsigned:
+    OS << Data.Unsigned;
+    break;
+  case DataType::Signed:
+    OS << Data.Signed;
+    break;
+  case DataType::Double:
+    OS << Data.Double;
+    break;
+  }
+}
+
+JSONTrue::JSONTrue() : JSONValue(JSONValue::Kind::True) {}
+
+void JSONTrue::write(raw_ostream &OS) const { OS << "true"; }
+
+JSONFalse::JSONFalse() : JSONValue(JSONValue::Kind::False) {}
+
+void JSONFalse::write(raw_ostream &OS) const { OS << "false"; }
+
+JSONNull::JSONNull() : JSONValue(JSONValue::Kind::Null) {}
+
+void JSONNull::write(raw_ostream &OS) const { OS << "null"; }
+
+JSONObject::JSONObject() : JSONValue(JSONValue::Kind::Object) {}
+
+void JSONObject::write(raw_ostream &OS) const {
+  bool First = true;
+  OS << '{';
+  auto Iter = Elements.begin(), End = Elements.end();
+  for (; Iter != End; Iter++) {
+    if (First)
+      First = false;
+    else
+      OS << ',';
+    JSONString Key(Iter->first);
+    JSONValue::SP Value(Iter->second);
+    Key.write(OS);
+    OS << ':';
+    Value->write(OS);
+  }
+  OS << '}';
+}
+
+bool JSONObject::setObject(const std::string &Key, JSONValue::SP Value) {
+  if (Key.empty() || nullptr == Value.get())
+    return false;
+  Elements[Key] = Value;
+  return true;
+}
+
+JSONValue::SP JSONObject::getObject(const std::string &Key) {
+  auto Iter = Elements.find(Key), End = Elements.end();
+  if (Iter == End)
+    return JSONValue::SP();
+  return Iter->second;
+}
+
+JSONArray::JSONArray() : JSONValue(JSONValue::Kind::Array) {}
+
+void JSONArray::write(raw_ostream &OS) const {
+  bool First = true;
+  OS << '[';
+  auto Iter = Elements.begin(), End = Elements.end();
+  for (; Iter != End; Iter++) {
+    if (First)
+      First = false;
+    else
+      OS << ',';
+    (*Iter)->write(OS);
+  }
+  OS << ']';
+}
+
+bool JSONArray::setObject(Index I, JSONValue::SP Value) {
+  if (Value.get() == nullptr)
+    return false;
+  if (I < Elements.size()) {
+    Elements[I] = Value;
+    return true;
+  }
+  if (I == Elements.size()) {
+    Elements.push_back(Value);
+    return true;
+  }
+  return false;
+}
+
+bool JSONArray::appendObject(JSONValue::SP Value) {
+  if (Value.get() == nullptr)
+    return false;
+  Elements.push_back(Value);
+  return true;
+}
+
+JSONValue::SP JSONArray::getObject(Index I) {
+  if (I < Elements.size())
+    return Elements[I];
+  return JSONValue::SP();
+}
+
+JSONArray::Size JSONArray::getNumElements() { return Elements.size(); }
+
+JSONParser::JSONParser(const char *CStr) : Index(0) {
+  if (CStr)
+    Buffer.assign(CStr);
+}
+
+JSONParser::Token JSONParser::getToken(std::string &Value) {
+  std::stringstream Error;
+
+  Value.clear();
+  skipSpaces();
+  const uint64_t StartIndex = Index;
+  const char Ch = getChar();
+  switch (Ch) {
+  case '{':
+    return Token::ObjectStart;
+  case '}':
+    return Token::ObjectEnd;
+  case '[':
+    return Token::ArrayStart;
+  case ']':
+    return Token::ArrayEnd;
+  case ',':
+    return Token::Comma;
+  case ':':
+    return Token::Colon;
+  case '\0':
+    return Token::EndOfFile;
+  case 't':
+    if (getChar() == 'r')
+      if (getChar() == 'u')
+        if (getChar() == 'e')
+          return Token::True;
+    break;
+
+  case 'f':
+    if (getChar() == 'a')
+      if (getChar() == 'l')
+        if (getChar() == 's')
+          if (getChar() == 'e')
+            return Token::False;
+    break;
+
+  case 'n':
+    if (getChar() == 'u')
+      if (getChar() == 'l')
+        if (getChar() == 'l')
+          return Token::Null;
+    break;
+
+  case '"': {
+    while (1) {
+      bool WasEscaped = false;
+      int EscapedCh = getEscapedChar(WasEscaped);
+      if (EscapedCh == -1) {
+        Error << "error: an error occurred getting a character from offset "
+              << StartIndex;
+        Value = Error.str();
+        return Token::Error;
+
+      } else {
+        const bool IsEndQuote = EscapedCh == '"';
+        const bool IsNull = EscapedCh == 0;
+        if (WasEscaped || (!IsEndQuote && !IsNull)) {
+          if (CHAR_MIN <= EscapedCh && EscapedCh <= CHAR_MAX) {
+            Value.append(1, (char)EscapedCh);
+          } else {
+            Error << "error: wide character support is needed for unicode "
+                     "character 0x"
+                  << std::hex << EscapedCh << std::dec << " at offset "
+                  << StartIndex;
+            Value = Error.str();
+            return Token::Error;
+          }
+        } else if (IsEndQuote) {
+          return Token::String;
+        } else if (IsNull) {
+          Value = "error: missing end quote for string";
+          return Token::Error;
+        }
+      }
+    }
+  } break;
+
+  case '-':
+  case '0':
+  case '1':
+  case '2':
+  case '3':
+  case '4':
+  case '5':
+  case '6':
+  case '7':
+  case '8':
+  case '9': {
+    uint64_t ExpIndex = 0;
+    bool Done = false;
+    bool GotDecimalPoint = false;
+    bool GotIntDigits = (Ch >= '0') && (Ch <= '9');
+    bool GotFracDigits = false;
+    bool GotExpDigits = false;
+    while (!Done) {
+      const char NextCh = peekChar();
+      switch (NextCh) {
+      case '0':
+      case '1':
+      case '2':
+      case '3':
+      case '4':
+      case '5':
+      case '6':
+      case '7':
+      case '8':
+      case '9':
+        if (ExpIndex != 0) {
+          GotExpDigits = true;
+        } else if (GotDecimalPoint) {
+          GotFracDigits = true;
+        } else {
+          GotIntDigits = true;
+        }
+        ++Index; // Skip this character
+        break;
+
+      case '.':
+        if (GotDecimalPoint) {
+          Error << "error: extra decimal point found at offset " << StartIndex;
+          Value = Error.str();
+          return Token::Error;
+        } else {
+          GotDecimalPoint = true;
+          ++Index; // Skip this character
+        }
+        break;
+
+      case 'e':
+      case 'E':
+        if (ExpIndex != 0) {
+          Error << "error: extra exponent character found at offset "
+                << StartIndex;
+          Value = Error.str();
+          return Token::Error;
+        } else {
+          ExpIndex = Index;
+          ++Index; // Skip this character
+        }
+        break;
+
+      case '+':
+      case '-':
+        // The '+' and '-' can only come after an exponent character...
+        if (ExpIndex == Index - 1) {
+          ++Index; // Skip the exponent sign character
+        } else {
+          Error << "error: unexpected " << NextCh << " character at offset "
+                << StartIndex;
+          Value = Error.str();
+          return Token::Error;
+        }
+        break;
+
+      default:
+        Done = true;
+        break;
+      }
+    }
+
+    if (Index > StartIndex) {
+      Value = Buffer.substr(StartIndex, Index - StartIndex);
+      if (GotDecimalPoint) {
+        if (ExpIndex != 0) {
+          // We have an exponent, make sure we got exponent digits
+          if (GotExpDigits) {
+            return Token::Float;
+          } else {
+            Error << "error: got exponent character but no exponent digits at "
+                     "offset in float value \""
+                  << Value << "\"";
+            Value = Error.str();
+            return Token::Error;
+          }
+        } else {
+          // No exponent, but we need at least one decimal after the decimal
+          // point
+          if (GotFracDigits) {
+            return Token::Float;
+          } else {
+            Error << "error: no digits after decimal point \"" << Value << "\"";
+            Value = Error.str();
+            return Token::Error;
+          }
+        }
+      } else {
+        // No decimal point
+        if (GotIntDigits) {
+          // We need at least some integer digits to make an integer
+          return Token::Integer;
+        } else {
+          Error << "error: no digits negate sign \"" << Value << "%s\"";
+          Value = Error.str();
+          return Token::Error;
+        }
+      }
+    } else {
+      Error << "error: invalid number found at offset " << StartIndex;
+      Value = Error.str();
+      return Token::Error;
+    }
+  } break;
+  default:
+    break;
+  }
+  Error << "error: failed to parse token at offset " << StartIndex
+        << " (around character '" << Ch << "')";
+  Value = Error.str();
+  return Token::Error;
+}
+
+int JSONParser::getEscapedChar(bool &WasEscaped) {
+  WasEscaped = false;
+  const char Ch = getChar();
+  if (Ch == '\\') {
+    WasEscaped = true;
+    const char Ch2 = getChar();
+    switch (Ch2) {
+    case '"':
+    case '\\':
+    case '/':
+    default:
+      break;
+
+    case 'b':
+      return '\b';
+    case 'f':
+      return '\f';
+    case 'n':
+      return '\n';
+    case 'r':
+      return '\r';
+    case 't':
+      return '\t';
+    case 'u': {
+      const int HiByte = decodeHexU8();
+      const int LoByte = decodeHexU8();
+      if (HiByte >= 0 && LoByte >= 0)
+        return HiByte << 8 | LoByte;
+      return -1;
+    } break;
+    }
+    return Ch2;
+  }
+  return Ch;
+}
+
+JSONValue::SP JSONParser::parseJSONObject() {
+  // The "JSONParser::Token::ObjectStart" token should have already been
+  // consumed
+  // by the time this function is called
+  std::unique_ptr<JSONObject> DictUp(new JSONObject());
+
+  std::string Value;
+  std::string Key;
+  while (1) {
+    JSONParser::Token Token = getToken(Value);
+
+    if (Token == JSONParser::Token::String) {
+      Key.swap(Value);
+      Token = getToken(Value);
+      if (Token == JSONParser::Token::Colon) {
+        JSONValue::SP ValueSP = parseJSONValue();
+        if (ValueSP)
+          DictUp->setObject(Key, ValueSP);
+        else
+          break;
+      }
+    } else if (Token == JSONParser::Token::ObjectEnd) {
+      return JSONValue::SP(DictUp.release());
+    } else if (Token == JSONParser::Token::Comma) {
+      continue;
+    } else {
+      break;
+    }
+  }
+  return JSONValue::SP();
+}
+
+JSONValue::SP JSONParser::parseJSONArray() {
+  // The "JSONParser::Token::ObjectStart" token should have already been
+  // consumed
+  // by the time this function is called
+  std::unique_ptr<JSONArray> ArrayUp(new JSONArray());
+
+  std::string Value;
+  while (1) {
+    skipSpaces();
+
+    char Peek = peekChar();
+    if (Peek == ']' || Peek == ',') {
+      JSONParser::Token Token = getToken(Value);
+      if (Token == JSONParser::Token::Comma) {
+        continue;
+      } else if (Token == JSONParser::Token::ArrayEnd) {
+        return JSONValue::SP(ArrayUp.release());
+      }
+    }
+
+    JSONValue::SP ValueSP = parseJSONValue();
+    if (ValueSP)
+      ArrayUp->appendObject(ValueSP);
+    else
+      break;
+  }
+  return JSONValue::SP();
+}
+
+JSONValue::SP JSONParser::parseJSONValue() {
+  std::string Value;
+  const JSONParser::Token Token = getToken(Value);
+  switch (Token) {
+  case JSONParser::Token::ObjectStart:
+    return parseJSONObject();
+
+  case JSONParser::Token::ArrayStart:
+    return parseJSONArray();
+
+  case JSONParser::Token::Integer: {
+    char *End = nullptr;
+    if (Value.front() == '-') {
+      int64_t SVal = ::strtoll(Value.c_str(), &End, 10);
+      bool Success = *End == '\0'; // all characters were used.
+      if (Success)
+        return JSONValue::SP(new JSONNumber(SVal));
+    } else {
+      uint64_t UVal = ::strtoul(Value.c_str(), &End, 10);
+      bool Success = *End == '\0'; // all characters were used.
+      if (Success)
+        return JSONValue::SP(new JSONNumber(UVal));
+    }
+  } break;
+
+  case JSONParser::Token::Float: {
+    char *End = nullptr;
+    double Val = ::strtod(Value.c_str(), &End);
+    bool Success = *End == '\0'; // all characters were used.
+    if (Success)
+      return JSONValue::SP(new JSONNumber(Val));
+  } break;
+
+  case JSONParser::Token::String:
+    return JSONValue::SP(new JSONString(Value));
+
+  case JSONParser::Token::True:
+    return JSONValue::SP(new JSONTrue());
+
+  case JSONParser::Token::False:
+    return JSONValue::SP(new JSONFalse());
+
+  case JSONParser::Token::Null:
+    return JSONValue::SP(new JSONNull());
+
+  default:
+    break;
+  }
+  return JSONValue::SP();
+}
+
+void JSONParser::skipSpaces() {
+  const size_t S = Buffer.size();
+  while (Index < S && isspace(Buffer[Index]))
+    ++Index;
+}
+
+char JSONParser::getChar() {
+  if (getBytesLeft() < 1)
+    return 0;
+  return Buffer[Index++];
+}
+
+char JSONParser::peekChar() const {
+  if (getBytesLeft() == 0)
+    return 0;
+
+  return Buffer[Index];
+}
+
+static inline int hexDigitToSInt(char Ch) {
+  if (Ch >= 'a' && Ch <= 'f')
+    return 10 + Ch - 'a';
+  if (Ch >= 'A' && Ch <= 'F')
+    return 10 + Ch - 'A';
+  if (Ch >= '0' && Ch <= '9')
+    return Ch - '0';
+  return -1;
+}
+
+uint8_t JSONParser::decodeHexU8() {
+  skipSpaces();
+  if (getBytesLeft() < 2) {
+    return -1;
+  }
+  const int HiNibble = hexDigitToSInt(Buffer[Index]);
+  const int LoNibble = hexDigitToSInt(Buffer[Index + 1]);
+  if (HiNibble == -1 || LoNibble == -1) {
+    return -1;
+  }
+  Index += 2;
+  return (uint8_t)((HiNibble << 4) + LoNibble);
+}

unittests/Support/CMakeLists.txt

add_llvm_unittest(SupportTests
   ConvertUTFTest.cpp
   DataExtractorTest.cpp
   DwarfTest.cpp
   EndianStreamTest.cpp
   EndianTest.cpp
   ErrorTest.cpp
   ErrorOrTest.cpp
   FileOutputBufferTest.cpp
+  JSONTest.cpp
   LEB128Test.cpp
   LineIteratorTest.cpp
   LockFileManagerTest.cpp
   MD5Test.cpp
   ManagedStatic.cpp
   MathExtrasTest.cpp
   MemoryBufferTest.cpp
   MemoryTest.cpp

unittests/Support/JSONTest.cpp

+//===- llvm/unittest/Support/JSONTest.cpp - JSON.cpp tests ---===//
+//
+//           The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "gtest/gtest.h"
+
+#include "llvm/Support/JSON.h"
+
+using namespace llvm;
+
+static ::testing::AssertionResult MatchRoundtrip(std::string Text) {
+  JSONParser Parser(Text.c_str());
+  auto Obj = Parser.parseJSONValue();
+
+  if (!Obj) {
+    return Text == "null"
+               ? ::testing::AssertionSuccess()
+               : ::testing::AssertionFailure() << "can't parse input: " << Text;
+  }
+
+  std::string S;
+  raw_string_ostream Out(S);
+  Obj->write(Out);
+
+  std::string Actual = Out.str();
+  if (Actual != Text) {
+    return ::testing::AssertionFailure() << "expected: " << Text
+                                         << " actual: " << Actual;
+  }
+  return ::testing::AssertionSuccess();
+}
+
+TEST(JSON, Roundtrip) {
+  EXPECT_TRUE(MatchRoundtrip("0"));
+  EXPECT_TRUE(MatchRoundtrip("3.145150e+00"));
+  EXPECT_TRUE(MatchRoundtrip("{}"));
+  EXPECT_TRUE(MatchRoundtrip("{\"a\":1,\"b\":2}"));
+  EXPECT_TRUE(MatchRoundtrip("[]"));
+  EXPECT_TRUE(MatchRoundtrip("[0]"));
+  EXPECT_TRUE(MatchRoundtrip("[1,\"two\",3]"));
+}