[support] copying JSON parser/writer from lldb

[aizatsky]

Summary

This CL contains a copy of JSON parser/writer that can be found
in the lldb subproject with only necessary changes:

• files now follow LLVM naming and formatting guidelines
• raw_ostream used instead of the special formatted stream
• very basic parse/print roundtrip test for JSON.

As soon as this CL lands I will prepare the one that removes JSON
parser from lldb/ and uses parent's.

[labath]

I am cc'ing a bunch of people to make sure we don't do this behind anyone's back. I don't know whether you discussed this with anyone on lldb side (it's certainly new to me), but I think it deserves a wider audience, as this is the first case (I can recall) of moving code from lldb back to llvm.

That said, I think it's a great idea. However, you should sync up with @zturner, as he has imminent plans to do a StringExtractor refactor (D24013) within lldb and you're making a copy of it here.

Some other high-level questions (not directly related to this change, but I'd like to know where you're going with this):

• despite the renaming, the code still has a distinct lldb feel to it. I presume this will be followed by a bunch of CL's to bring it more in line with the rest of llvm (?)
• you are copying StringExtractor to llvm as well. Will you switch lldb to use that copy as well, or is the plan to bypass it completely and use llvm parsing primitives for the job?
• were you able to get lldb test suite running? If not, I can help you get that working.
• mostly out of curiosity: what is the motivation for this? I guess you have a need for a JSON parser in llvm ?

cheers.

[zturner]

Having a JSON parser in LLVM seems like a reasonable idea, but I'm afraid the JSON parser that currently exists in LLDB won't do. As Pavel mentioned, the code has a definite LLDB feel to it, but more importantly it depends on StringExtractor which I think probably doesn't belong in LLVM. If you want to go down this route, I would start by deleting the StringExtractor changes from this CL. 90% of the stuff StringExtractor is used for, like hex string to number, string to integer, etc are covered by existing LLVM classes or methods.

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

[aizatsky]

In D24369#538213, @zturner wrote:

Having a JSON parser in LLVM seems like a reasonable idea, but I'm afraid the JSON parser that currently exists in LLDB won't do. As Pavel mentioned, the code has a definite LLDB feel to it, but more importantly it depends on StringExtractor which I think probably doesn't belong in LLVM. If you want to go down this route, I would start by deleting the StringExtractor changes from this CL. 90% of the stuff StringExtractor is used for, like hex string to number, string to integer, etc are covered by existing LLVM classes or methods.

My rationale was that reusing parser from a child project is better than building anything new from scratch.

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

I can't actually find json parser within clang-tidy. Only references that I see lead eventually to YAML parsing:

tool/ClangTidyMain.cpp
102:JSON array of objects:
138:Specifies a configuration in YAML/JSON format:

ClangTidyOptions.cpp
36:// Map std::pair<int, int> to a JSON array of size 2.

ClangTidyOptions.h
245:/// \brief Parses LineFilter from JSON and stores it to the \p Options.
249:/// \brief Parses configuration from JSON and returns \c ClangTidyOptions or an

[zturner]

In D24369#538422, @aizatsky wrote:

In D24369#538213, @zturner wrote:

Having a JSON parser in LLVM seems like a reasonable idea, but I'm afraid the JSON parser that currently exists in LLDB won't do. As Pavel mentioned, the code has a definite LLDB feel to it, but more importantly it depends on StringExtractor which I think probably doesn't belong in LLVM. If you want to go down this route, I would start by deleting the StringExtractor changes from this CL. 90% of the stuff StringExtractor is used for, like hex string to number, string to integer, etc are covered by existing LLVM classes or methods.

My rationale was that reusing parser from a child project is better than building anything new from scratch.

Agree that we always want to reuse code wherever possible, but LLVM has stricter requirements on code style and class design. The JSON stuff itself can probably be a good starting point since clang-tidy doesn't have one (see my comment later), but I don't think StringExtractor belongs in LLVM. Looking at the code, JSON really only uses 1 or 2 methods from `StringExtractor anyway, so I don't think it's a big thing to overcome.

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

I can't actually find json parser within clang-tidy. Only references that I see lead eventually to YAML parsing:

I was thinking of lib/Tooling/JSONCompilationDatabase, but looking at it, it seems to be pretty specialized and doesn't support arbitrary JSON.

[clayborg]

I like the idea of moving this down into LLVM. We should fully LLVM-ize the code on the first pass though and take the LLDB style out of it.

[aizatsky]

• getting rid of StringExtractor
• renaming methods to lowerCase.

[aizatsky]

In D24369#538472, @zturner wrote:

In D24369#538422, @aizatsky wrote:

In D24369#538213, @zturner wrote:

Having a JSON parser in LLVM seems like a reasonable idea, but I'm afraid the JSON parser that currently exists in LLDB won't do. As Pavel mentioned, the code has a definite LLDB feel to it, but more importantly it depends on StringExtractor which I think probably doesn't belong in LLVM. If you want to go down this route, I would start by deleting the StringExtractor changes from this CL. 90% of the stuff StringExtractor is used for, like hex string to number, string to integer, etc are covered by existing LLVM classes or methods.

My rationale was that reusing parser from a child project is better than building anything new from scratch.

Agree that we always want to reuse code wherever possible, but LLVM has stricter requirements on code style and class design. The JSON stuff itself can probably be a good starting point since clang-tidy doesn't have one (see my comment later), but I don't think StringExtractor belongs in LLVM. Looking at the code, JSON really only uses 1 or 2 methods from `StringExtractor anyway, so I don't think it's a big thing to overcome.

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

I can't actually find json parser within clang-tidy. Only references that I see lead eventually to YAML parsing:

I was thinking of lib/Tooling/JSONCompilationDatabase, but looking at it, it seems to be pretty specialized and doesn't support arbitrary JSON.

I got rid of StringExtractor and also renamed all methods to follow the guidelines.
Let me know which other style inconsistencies you see there.

[aizatsky]

remove .clang-tidy

[zturner]

include/llvm/Support/JSON.h

28

I scanned the entire patch and I'm not sure I understand the need for the covariant typedefs. The only type that ever seems to be used is JSONValue, so it seems to me like the other types are unnecessary.

Also, I'm not sure shared_ptr is necessary. Wouldn't unique_ptr work just fine? Then you could vend references to values instead of pointers.

48

I would make this a StringRef. Currently if someone calls this with a StringRef, it will make a copy. If you change the constructor to take a StringRef, then it won't take a copy regardless of whether it's called with a StringRef or a std::string.

57

I would return a StringRef here. Otherwise you're guaranteed to make a copy, if you return a StringRef it will only make a copy when you assign it to a std::string.

64

This could be a file static in the .cpp file, and also it could take a StringRef instead of a const std::string &.

203

Couple things here.

1. Key could be a StringRef
2. This method should probably be const.
3. You could make the underlying value a unique_ptr instead of a shared_ptr and return a reference. You could also have a tryGetValue in cases where you aren't sure if the value exists. Not sure what would be better.

208

DenseMap seems like a better choice here.

230–232

I don't know if the typedefs here help or hurt readability. I would just make them size_t, and remove the Iterator typedef as it doesn't seem to be used.

239

How about providing an overloaded operator[] and providing begin() and end() functions so that it can be used in a range based for syntax?

283–284

Instead of having a std::string here, which will copy the input JSON, I would make this a StringRef. Then you can delete the Index field as well, because the internal StringRef itself could always point to the beginning of the next char by using a Buffer = Buffer.drop_front(N) like pattern. This won't work if you ever have to do back-referencing across functions, but I don't think that is needed. Also if Buffer is a StringRef, a lot of the methods like peekChar() and skipSpaces become trivial one-liners that you wouldn't even need to provide a separate function for.

lib/Support/JSON.cpp

20

Input parameter should be a StringRef.

24

Should probably do an Output.reserve(S.size());

25–27

for (char CH : S)

54

Can you use C++ style casting operators here and in other similar functions?

116–117

for (const auto &I : Elements) {

188

This function should be const.

284

Don't like the idea of reimplementing string -> number parsing, but I'm not sure if LLVM has something to solve this for us.

[chandlerc]

In D24369#538472, @zturner wrote:

In D24369#538422, @aizatsky wrote:

I'm a bit surprised there's not already a json parser in LLVM. Maybe there's a reason for it. I think this CL needs a few more reviewers on the LLVM side, so I'm adding a few people.

There is already a JSON parser in LLVM.

I can't actually find json parser within clang-tidy. Only references that I see lead eventually to YAML parsing:

I was thinking of lib/Tooling/JSONCompilationDatabase, but looking at it, it seems to be pretty specialized and doesn't support arbitrary JSON.

It uses the YAML parser because JSON is a strict subset of YAML: https://en.wikipedia.org/wiki/JSON#YAML

Please don't add a JSON parser to LLVM. I would much rather teach LLDB to use the YAML parser and improve that parser if and where useful.

[aizatsky]

cleanup

[aizatsky]

include/llvm/Support/JSON.h

208

DenseMap doesn't seem to be defined for std::string keys.

239

done for operator[].

begin() and end() will be tricky, because underlying storage has std::unique_ptr<>

283–284

std::string => StringRef - done.

As for removing Index - I'm not sure. It is currently used in error reporting and is the only feedback we give when we encounter an error.

lib/Support/JSON.cpp

54

Removed all 3 them and added get<T> method.

include/llvm/Support/JSON.h

+//===---------------------JSON.h --------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_JSON_H
+#define LLVM_SUPPORT_JSON_H
+
+#include <inttypes.h>
+#include <map>
+#include <memory>
+#include <stdint.h>
+#include <string>
+#include <vector>
+
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace llvm {
+
+class JSONValue {
+public:
+  enum class Kind { String, Number, True, False, Null, Object, Array };
+
+  JSONValue(Kind K) : TheKind(K) {}
+  virtual ~JSONValue() = default;
+
+  virtual void write(raw_ostream &S) const = 0;
+
+  Kind kind() const { return TheKind; }
+
+private:
+  const Kind TheKind;
+};
+
+class JSONString : public JSONValue {
+public:
+  JSONString(StringRef S);
+
+  JSONString(const JSONString &S) = delete;
+  JSONString &operator=(const JSONString &S) = delete;
+
+  ~JSONString() override = default;
+
+  void write(raw_ostream &S) const override;
+
+  StringRef getData() const { return Data; }
+
+  static bool classof(const JSONValue *V) {
+    return V->kind() == JSONValue::Kind::String;
+  }
+
+private:
+  std::string Data;
+};
+
+class JSONNumber : public JSONValue {
+public:
+  // We create a constructor for all integer and floating point type with using
+  // templates and
+  // SFINAE to avoid having ambiguous overloads because of the implicit type
+  // promotion. If we
+  // would have constructors only with int64_t, uint64_t and double types then
+  // constructing a
+  // JSONNumber from an int32_t (or any other similar type) would fail to
+  // compile.
+
+  template <typename T, typename std::enable_if<
+                            std::is_integral<T>::value &&
+                            std::is_unsigned<T>::value>::type * = nullptr>
+  explicit JSONNumber(T U)
+      : JSONValue(JSONValue::Kind::Number), TheDataType(DataType::Unsigned) {
+    Data.Unsigned = U;
+  }
+
+  template <typename T,
+            typename std::enable_if<std::is_integral<T>::value &&
+                                    std::is_signed<T>::value>::type * = nullptr>
+  explicit JSONNumber(T S)
+      : JSONValue(JSONValue::Kind::Number), TheDataType(DataType::Signed) {
+    Data.Signed = S;
+  }
+
+  template <typename T, typename std::enable_if<
+                            std::is_floating_point<T>::value>::type * = nullptr>
+  explicit JSONNumber(T D)
+      : JSONValue(JSONValue::Kind::Number), TheDataType(DataType::Double) {
+    Data.Double = D;
+  }
+
+  ~JSONNumber() override = default;
+
+  JSONNumber(const JSONNumber &S) = delete;
+  JSONNumber &operator=(const JSONNumber &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  template <typename T> T get() const {
+    switch (TheDataType) {
+    case DataType::Unsigned:
+      return static_cast<T>(Data.Unsigned);
+    case DataType::Signed:
+      return static_cast<T>(Data.Signed);
+    case DataType::Double:
+      return static_cast<T>(Data.Double);
+    }
+    llvm_unreachable("Unhandled data type");
+  }
+
+  static bool classof(const JSONValue *V) {
+    return V->kind() == JSONValue::Kind::Number;
+  }
+
+private:
+  enum class DataType : uint8_t { Unsigned, Signed, Double } TheDataType;
+
+  union {
+    uint64_t Unsigned;
+    int64_t Signed;
+    double Double;
+  } Data;
+};
+
+class JSONTrue : public JSONValue {
+public:
+  JSONTrue();
+
+  JSONTrue(const JSONTrue &S) = delete;
+  JSONTrue &operator=(const JSONTrue &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  static bool classof(const JSONValue *V) {
+    return V->kind() == JSONValue::Kind::True;
+  }
+
+  ~JSONTrue() override = default;
+};
+
+class JSONFalse : public JSONValue {
+public:
+  JSONFalse();
+
+  JSONFalse(const JSONFalse &S) = delete;
+  JSONFalse &operator=(const JSONFalse &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  static bool classof(const JSONValue *V) {
+    return V->kind() == JSONValue::Kind::False;
+  }
+
+  ~JSONFalse() override = default;
+};
+
+class JSONNull : public JSONValue {
+public:
+  JSONNull();
+
+  JSONNull(const JSONNull &S) = delete;
+  JSONNull &operator=(const JSONNull &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  static bool classof(const JSONValue *V) {
+    return V->kind() == JSONValue::Kind::Null;
+  }
+
+  ~JSONNull() override = default;
+};
+
+class JSONObject : public JSONValue {
+public:
+  JSONObject();
+  ~JSONObject() override = default;
+
+  JSONObject(const JSONObject &S) = delete;
+  JSONObject &operator=(const JSONObject &S) = delete;
+
+  void write(raw_ostream &S) const override;
+
+  static bool classof(const JSONValue *V) {
+    return V->kind() == JSONValue::Kind::Object;
+  }
+
+  bool set(StringRef Key, std::unique_ptr<JSONValue> Value);
+  bool get(StringRef Key, JSONValue const **ValuePtr) const;
+
+private:
+  std::map<std::string, std::unique_ptr<JSONValue>> Elements;
+};
+
+class JSONArray : public JSONValue {
+public:
+  JSONArray();
+
+  JSONArray(const JSONArray &S) = delete;
+  JSONArray &operator=(const JSONArray &S) = delete;
+
+  ~JSONArray() override = default;
+
+  void write(raw_ostream &S) const override;
+
+  static bool classof(const JSONValue *V) {
+    return V->kind() == JSONValue::Kind::Array;
+  }
+
+public:
+  bool set(size_t I, std::unique_ptr<JSONValue> Value);
+  bool push_back(std::unique_ptr<JSONValue> Value);
+
+  size_t size() const { return Elements.size(); }
+  const JSONValue &operator[](size_t I) const;
+
+private:
+  std::vector<std::unique_ptr<JSONValue>> Elements;
+};
+
+class JSONParser {
+public:
+  enum Token {
+    Invalid,
+    Error,
+    ObjectStart,
+    ObjectEnd,
+    ArrayStart,
+    ArrayEnd,
+    Comma,
+    Colon,
+    String,
+    Integer,
+    Float,
+    True,
+    False,
+    Null,
+    EndOfFile
+  };
+
+  explicit JSONParser(StringRef S) : Buffer(S), Index(0){};
+  std::unique_ptr<JSONValue> parseJSONValue();
+
+protected:
+  int getEscapedChar(bool &WasEscaped);
+  Token getToken(std::string &Value);
+  std::unique_ptr<JSONValue> parseJSONObject();
+  std::unique_ptr<JSONValue> parseJSONArray();
+
+  void skipSpaces();
+  uint8_t decodeHexU8();
+  char getChar();
+  char peekChar() const;
+  size_t getBytesLeft() const { return Buffer.size() - Index; }
+
+  StringRef Buffer;
+  size_t Index;
+};
+} // namespace llvm
+
+#endif // LLVM_SUPPORT_JSON_H

lib/Support/CMakeLists.txt

add_llvm_library(LLVMSupport
   FoldingSet.cpp
   FormattedStream.cpp
   GraphWriter.cpp
   Hashing.cpp
   IntEqClasses.cpp
   IntervalMap.cpp
   IntrusiveRefCntPtr.cpp
   JamCRC.cpp
+  JSON.cpp
   LEB128.cpp
   LineIterator.cpp
   Locale.cpp
   LockFileManager.cpp
   ManagedStatic.cpp
   MathExtras.cpp
   MemoryBuffer.cpp
   MemoryObject.cpp

lib/Support/JSON.cpp

+//===--------------------- JSON.cpp -----------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/JSON.h"
+
+#include <limits.h>
+#include <sstream>
+
+#include "llvm/Support/ErrorHandling.h"
+
+using namespace llvm;
+
+static std::string quoteJsonString(StringRef S) {
+  if (S.find('"') == std::string::npos)
+    return S;
+
+  std::string Output;
+  Output.reserve(S.size());
+  for (char Ch : S.bytes()) {
+    if (Ch == '"')
+      Output.push_back('\\');
+    Output.push_back(Ch);
+  }
+  return Output;
+}
+
+JSONString::JSONString(StringRef S)
+    : JSONValue(JSONValue::Kind::String), Data(S) {}
+
+void JSONString::write(raw_ostream &OS) const {
+  OS << "\"" << quoteJsonString(Data) << "\"";
+}
+
+void JSONNumber::write(raw_ostream &OS) const {
+  switch (TheDataType) {
+  case DataType::Unsigned:
+    OS << Data.Unsigned;
+    break;
+  case DataType::Signed:
+    OS << Data.Signed;
+    break;
+  case DataType::Double:
+    OS << Data.Double;
+    break;
+  }
+}
+
+JSONTrue::JSONTrue() : JSONValue(JSONValue::Kind::True) {}
+
+void JSONTrue::write(raw_ostream &OS) const { OS << "true"; }
+
+JSONFalse::JSONFalse() : JSONValue(JSONValue::Kind::False) {}
+
+void JSONFalse::write(raw_ostream &OS) const { OS << "false"; }
+
+JSONNull::JSONNull() : JSONValue(JSONValue::Kind::Null) {}
+
+void JSONNull::write(raw_ostream &OS) const { OS << "null"; }
+
+JSONObject::JSONObject() : JSONValue(JSONValue::Kind::Object) {}
+
+void JSONObject::write(raw_ostream &OS) const {
+  bool First = true;
+  OS << '{';
+  for (const auto &KV : Elements) {
+    if (First)
+      First = false;
+    else
+      OS << ',';
+    OS << "\"" << quoteJsonString(KV.first) << "\"";
+    OS << ':';
+    KV.second->write(OS);
+  }
+  OS << '}';
+}
+
+bool JSONObject::set(StringRef Key, std::unique_ptr<JSONValue> Value) {
+  if (Key.empty() || nullptr == Value.get())
+    return false;
+  Elements[Key] = std::move(Value);
+  return true;
+}
+
+bool JSONObject::get(StringRef Key, JSONValue const **ValuePtr) const {
+  auto Iter = Elements.find(Key), End = Elements.end();
+  if (Iter == End)
+    return false;
+  *ValuePtr = Iter->second.get();
+  return true;
+}
+
+JSONArray::JSONArray() : JSONValue(JSONValue::Kind::Array) {}
+
+void JSONArray::write(raw_ostream &OS) const {
+  bool First = true;
+  OS << '[';
+  for (const auto &Element : Elements) {
+    if (First)
+      First = false;
+    else
+      OS << ',';
+    Element->write(OS);
+  }
+  OS << ']';
+}
+
+bool JSONArray::set(size_t I, std::unique_ptr<JSONValue> Value) {
+  if (Value.get() == nullptr)
+    return false;
+  if (I < Elements.size()) {
+    Elements[I] = std::move(Value);
+    return true;
+  }
+  if (I == Elements.size()) {
+    Elements.push_back(std::move(Value));
+    return true;
+  }
+  return false;
+}
+
+bool JSONArray::push_back(std::unique_ptr<JSONValue> Value) {
+  if (Value.get() == nullptr)
+    return false;
+  Elements.push_back(std::move(Value));
+  return true;
+}
+
+const JSONValue &JSONArray::operator[](size_t I) const { return *Elements[I]; }
+
+JSONParser::Token JSONParser::getToken(std::string &Value) {
+  std::stringstream Error;
+
+  Value.clear();
+  skipSpaces();
+  const uint64_t StartIndex = Index;
+  const char Ch = getChar();
+  switch (Ch) {
+  case '{':
+    return Token::ObjectStart;
+  case '}':
+    return Token::ObjectEnd;
+  case '[':
+    return Token::ArrayStart;
+  case ']':
+    return Token::ArrayEnd;
+  case ',':
+    return Token::Comma;
+  case ':':
+    return Token::Colon;
+  case '\0':
+    return Token::EndOfFile;
+  case 't':
+    if (getChar() == 'r')
+      if (getChar() == 'u')
+        if (getChar() == 'e')
+          return Token::True;
+    break;
+
+  case 'f':
+    if (getChar() == 'a')
+      if (getChar() == 'l')
+        if (getChar() == 's')
+          if (getChar() == 'e')
+            return Token::False;
+    break;
+
+  case 'n':
+    if (getChar() == 'u')
+      if (getChar() == 'l')
+        if (getChar() == 'l')
+          return Token::Null;
+    break;
+
+  case '"': {
+    while (1) {
+      bool WasEscaped = false;
+      int EscapedCh = getEscapedChar(WasEscaped);
+      if (EscapedCh == -1) {
+        Error << "error: an error occurred getting a character from offset "
+              << StartIndex;
+        Value = Error.str();
+        return Token::Error;
+
+      } else {
+        const bool IsEndQuote = EscapedCh == '"';
+        const bool IsNull = EscapedCh == 0;
+        if (WasEscaped || (!IsEndQuote && !IsNull)) {
+          if (CHAR_MIN <= EscapedCh && EscapedCh <= CHAR_MAX) {
+            Value.append(1, (char)EscapedCh);
+          } else {
+            Error << "error: wide character support is needed for unicode "
+                     "character 0x"
+                  << std::hex << EscapedCh << std::dec << " at offset "
+                  << StartIndex;
+            Value = Error.str();
+            return Token::Error;
+          }
+        } else if (IsEndQuote) {
+          return Token::String;
+        } else if (IsNull) {
+          Value = "error: missing end quote for string";
+          return Token::Error;
+        }
+      }
+    }
+  } break;
+
+  case '-':
+  case '0':
+  case '1':
+  case '2':
+  case '3':
+  case '4':
+  case '5':
+  case '6':
+  case '7':
+  case '8':
+  case '9': {
+    uint64_t ExpIndex = 0;
+    bool Done = false;
+    bool GotDecimalPoint = false;
+    bool GotIntDigits = (Ch >= '0') && (Ch <= '9');
+    bool GotFracDigits = false;
+    bool GotExpDigits = false;
+    while (!Done) {
+      const char NextCh = peekChar();
+      switch (NextCh) {
+      case '0':
+      case '1':
+      case '2':
+      case '3':
+      case '4':
+      case '5':
+      case '6':
+      case '7':
+      case '8':
+      case '9':
+        if (ExpIndex != 0) {
+          GotExpDigits = true;
+        } else if (GotDecimalPoint) {
+          GotFracDigits = true;
+        } else {
+          GotIntDigits = true;
+        }
+        ++Index; // Skip this character
+        break;
+
+      case '.':
+        if (GotDecimalPoint) {
+          Error << "error: extra decimal point found at offset " << StartIndex;
+          Value = Error.str();
+          return Token::Error;
+        } else {
+          GotDecimalPoint = true;
+          ++Index; // Skip this character
+        }
+        break;
+
+      case 'e':
+      case 'E':
+        if (ExpIndex != 0) {
+          Error << "error: extra exponent character found at offset "
+                << StartIndex;
+          Value = Error.str();
+          return Token::Error;
+        } else {
+          ExpIndex = Index;
+          ++Index; // Skip this character
+        }
+        break;
+
+      case '+':
+      case '-':
+        // The '+' and '-' can only come after an exponent character...
+        if (ExpIndex == Index - 1) {
+          ++Index; // Skip the exponent sign character
+        } else {
+          Error << "error: unexpected " << NextCh << " character at offset "
+                << StartIndex;
+          Value = Error.str();
+          return Token::Error;
+        }
+        break;
+
+      default:
+        Done = true;
+        break;
+      }
+    }
+
+    if (Index > StartIndex) {
+      Value = Buffer.substr(StartIndex, Index - StartIndex);
+      if (GotDecimalPoint) {
+        if (ExpIndex != 0) {
+          // We have an exponent, make sure we got exponent digits
+          if (GotExpDigits) {
+            return Token::Float;
+          } else {
+            Error << "error: got exponent character but no exponent digits at "
+                     "offset in float value \""
+                  << Value << "\"";
+            Value = Error.str();
+            return Token::Error;
+          }
+        } else {
+          // No exponent, but we need at least one decimal after the decimal
+          // point
+          if (GotFracDigits) {
+            return Token::Float;
+          } else {
+            Error << "error: no digits after decimal point \"" << Value << "\"";
+            Value = Error.str();
+            return Token::Error;
+          }
+        }
+      } else {
+        // No decimal point
+        if (GotIntDigits) {
+          // We need at least some integer digits to make an integer
+          return Token::Integer;
+        } else {
+          Error << "error: no digits negate sign \"" << Value << "%s\"";
+          Value = Error.str();
+          return Token::Error;
+        }
+      }
+    } else {
+      Error << "error: invalid number found at offset " << StartIndex;
+      Value = Error.str();
+      return Token::Error;
+    }
+  } break;
+  default:
+    break;
+  }
+  Error << "error: failed to parse token at offset " << StartIndex
+        << " (around character '" << Ch << "')";
+  Value = Error.str();
+  return Token::Error;
+}
+
+int JSONParser::getEscapedChar(bool &WasEscaped) {
+  WasEscaped = false;
+  const char Ch = getChar();
+  if (Ch == '\\') {
+    WasEscaped = true;
+    const char Ch2 = getChar();
+    switch (Ch2) {
+    case '"':
+    case '\\':
+    case '/':
+    default:
+      break;
+
+    case 'b':
+      return '\b';
+    case 'f':
+      return '\f';
+    case 'n':
+      return '\n';
+    case 'r':
+      return '\r';
+    case 't':
+      return '\t';
+    case 'u': {
+      const int HiByte = decodeHexU8();
+      const int LoByte = decodeHexU8();
+      if (HiByte >= 0 && LoByte >= 0)
+        return HiByte << 8 | LoByte;
+      return -1;
+    } break;
+    }
+    return Ch2;
+  }
+  return Ch;
+}
+
+std::unique_ptr<JSONValue> JSONParser::parseJSONObject() {
+  // The "JSONParser::Token::ObjectStart" token should have already been
+  // consumed
+  // by the time this function is called
+  std::unique_ptr<JSONObject> O(new JSONObject());
+
+  std::string Value;
+  std::string Key;
+  while (1) {
+    JSONParser::Token Token = getToken(Value);
+
+    if (Token == JSONParser::Token::String) {
+      Key.swap(Value);
+      Token = getToken(Value);
+      if (Token == JSONParser::Token::Colon) {
+        auto Value = parseJSONValue();
+        if (Value)
+          O->set(Key, std::move(Value));
+        else
+          break;
+      }
+    } else if (Token == JSONParser::Token::ObjectEnd) {
+      return std::move(O);
+    } else if (Token == JSONParser::Token::Comma) {
+      continue;
+    } else {
+      break;
+    }
+  }
+  return nullptr;
+}
+
+std::unique_ptr<JSONValue> JSONParser::parseJSONArray() {
+  // The "JSONParser::Token::ObjectStart" token should have already been
+  // consumed
+  // by the time this function is called
+  std::unique_ptr<JSONArray> Array(new JSONArray());
+
+  std::string Value;
+  while (1) {
+    skipSpaces();
+
+    char Peek = peekChar();
+    if (Peek == ']' || Peek == ',') {
+      JSONParser::Token Token = getToken(Value);
+      if (Token == JSONParser::Token::Comma) {
+        continue;
+      } else if (Token == JSONParser::Token::ArrayEnd) {
+        return std::move(Array);
+      }
+    }
+
+    auto Element = parseJSONValue();
+    if (Element)
+      Array->push_back(std::move(Element));
+    else
+      break;
+  }
+  return nullptr;
+}
+
+std::unique_ptr<JSONValue> JSONParser::parseJSONValue() {
+  std::string Value;
+  const JSONParser::Token Token = getToken(Value);
+  switch (Token) {
+  case JSONParser::Token::ObjectStart:
+    return parseJSONObject();
+
+  case JSONParser::Token::ArrayStart:
+    return parseJSONArray();
+
+  case JSONParser::Token::Integer: {
+    char *End = nullptr;
+    if (Value.front() == '-') {
+      int64_t SVal = ::strtoll(Value.c_str(), &End, 10);
+      bool Success = *End == '\0'; // all characters were used.
+      if (Success)
+        return std::unique_ptr<JSONValue>(new JSONNumber(SVal));
+    } else {
+      uint64_t UVal = ::strtoul(Value.c_str(), &End, 10);
+      bool Success = *End == '\0'; // all characters were used.
+      if (Success)
+        return std::unique_ptr<JSONValue>(new JSONNumber(UVal));
+    }
+  } break;
+
+  case JSONParser::Token::Float: {
+    char *End = nullptr;
+    double Val = ::strtod(Value.c_str(), &End);
+    bool Success = *End == '\0'; // all characters were used.
+    if (Success)
+      return std::unique_ptr<JSONValue>(new JSONNumber(Val));
+  } break;
+
+  case JSONParser::Token::String:
+    return std::unique_ptr<JSONValue>(new JSONString(Value));
+
+  case JSONParser::Token::True:
+    return std::unique_ptr<JSONValue>(new JSONTrue());
+
+  case JSONParser::Token::False:
+    return std::unique_ptr<JSONValue>(new JSONFalse());
+
+  case JSONParser::Token::Null:
+    return std::unique_ptr<JSONValue>(new JSONNull());
+
+  default:
+    break;
+  }
+  return nullptr;
+}
+
+void JSONParser::skipSpaces() {
+  const size_t S = Buffer.size();
+  while (Index < S && isspace(Buffer[Index]))
+    ++Index;
+}
+
+char JSONParser::getChar() {
+  if (getBytesLeft() < 1)
+    return 0;
+  return Buffer[Index++];
+}
+
+char JSONParser::peekChar() const {
+  if (getBytesLeft() == 0)
+    return 0;
+
+  return Buffer[Index];
+}
+
+static inline int hexDigitToSInt(char Ch) {
+  if (Ch >= 'a' && Ch <= 'f')
+    return 10 + Ch - 'a';
+  if (Ch >= 'A' && Ch <= 'F')
+    return 10 + Ch - 'A';
+  if (Ch >= '0' && Ch <= '9')
+    return Ch - '0';
+  return -1;
+}
+
+uint8_t JSONParser::decodeHexU8() {
+  skipSpaces();
+  if (getBytesLeft() < 2) {
+    return -1;
+  }
+  const int HiNibble = hexDigitToSInt(Buffer[Index]);
+  const int LoNibble = hexDigitToSInt(Buffer[Index + 1]);
+  if (HiNibble == -1 || LoNibble == -1) {
+    return -1;
+  }
+  Index += 2;
+  return (uint8_t)((HiNibble << 4) + LoNibble);
+}

unittests/Support/CMakeLists.txt

add_llvm_unittest(SupportTests
   ConvertUTFTest.cpp
   DataExtractorTest.cpp
   DwarfTest.cpp
   EndianStreamTest.cpp
   EndianTest.cpp
   ErrorTest.cpp
   ErrorOrTest.cpp
   FileOutputBufferTest.cpp
+  JSONTest.cpp
   LEB128Test.cpp
   LineIteratorTest.cpp
   LockFileManagerTest.cpp
   MD5Test.cpp
   ManagedStatic.cpp
   MathExtrasTest.cpp
   MemoryBufferTest.cpp
   MemoryTest.cpp

unittests/Support/JSONTest.cpp

+//===- llvm/unittest/Support/JSONTest.cpp - JSON.cpp tests ---===//
+//
+//           The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "gtest/gtest.h"
+
+#include "llvm/Support/JSON.h"
+
+using namespace llvm;
+
+static ::testing::AssertionResult MatchRoundtrip(std::string Text) {
+  JSONParser Parser(Text.c_str());
+  auto Obj = Parser.parseJSONValue();
+
+  if (!Obj) {
+    return Text == "null"
+               ? ::testing::AssertionSuccess()
+               : ::testing::AssertionFailure() << "can't parse input: " << Text;
+  }
+
+  std::string S;
+  raw_string_ostream Out(S);
+  Obj->write(Out);
+
+  std::string Actual = Out.str();
+  if (Actual != Text) {
+    return ::testing::AssertionFailure() << "expected: " << Text
+                                         << " actual: " << Actual;
+  }
+  return ::testing::AssertionSuccess();
+}
+
+TEST(JSON, Roundtrip) {
+  EXPECT_TRUE(MatchRoundtrip("0"));
+  EXPECT_TRUE(MatchRoundtrip("3.145150e+00"));
+  EXPECT_TRUE(MatchRoundtrip("{}"));
+  EXPECT_TRUE(MatchRoundtrip("{\"a\":1,\"b\":2}"));
+  EXPECT_TRUE(MatchRoundtrip("[]"));
+  EXPECT_TRUE(MatchRoundtrip("[0]"));
+  EXPECT_TRUE(MatchRoundtrip("[1,\"two\",3]"));
+}