[support] Provide overload to PrintNumber that use C++ types

[paulkirth]

Summary

This attempts to address an issue with overload resolution for PrintNumber
with size_t parameters on Darwin, brought up in
https://reviews.llvm.org/D146492.

On Aarch64 Darwin, uint64_t has a different typedef than size_t
(e.g., unsigned long long vs. unsigned long), whereas on Linux and
Windows they are the same.

This commit also reverts the static_cast's added in
064e2497e2ebe9ac30ac96923a26a52484300fdf, since they are no longer
needed.

[paulkirth]

llvm/include/llvm/Support/JSON.h

351–352

I didn't love that I had to change these, but the non-narrowing issue forced me to add unsigned long long. I looked into just pushing all of the unsigned integers into the uint64_t version, but ran into some strange behavior and test failures that I couldn't explain.

For example an MCA test failed, but the JSON appeared equivalent. The difference was that despite being equivalent "numbers", the values were not integers that the getInteger() API for json::Objects could resolve (i.e. it relies on them being int64_t, and some were now uint64_t).

Some of that likely is tied to the JSON rules for numeric types, but someone with more knowledge here should probably weigh in. Also template metaprogramming isn't something I do often, so there may be better ways to express these constraints.

[paulkirth]

llvm/include/llvm/Support/ScopedPrinter.h

214

Another potential approach would be to rely on conversion, and just pic a select few overloads for integer types. I usually try to avoid things like that, but 1) this is a lot of duplicated code, and 2) for many of these types the conversion is likely happening when writing to the stream.

[jhenderson]

I don't have a strong opinion on relying on conversions versus the duplicate code, so I'm happy for you to do what you prefer.

llvm/include/llvm/Support/JSON.h

351–352

I don't think the changes you've made will work in a cross-platform manner for the same sort of reasoning as why you need the unsigned long overloads of printNumber: an unsigned long is a 64-bit value on some platforms, but uint64_t might be defined to unsigned long long. In such cases, unsigned long will be neither uin64_t, nor unsigned long long, and so the enable_ifs won't work.

A 64-bit integral will have the following traits: std::is_integral, and sizeof(T) == sizeof(uint64_t) (you could hard code the latter to 8). So the current behaviour of comparing the type against uint64_t isn't really correct. I think you want to change the unsigned case to:

template <
    typename T,
    typename = std::enable_if_t<std::is_integral<T>::value &&
                                std::is_unsigned<T>::value &&
                                !std::is_same<T, bool>::value &&
                                sizeof(T) == sizeof(uint64_t)>>

The signed one should look like the following:

template <
    typename T,
    typename = std::enable_if_t<std::is_integral<T>::value &&
                                std::is_signed<T>::value &&
                                sizeof(T) == sizeof(int64_t)>>

Honestly, I'm not sure why the second one is listed as multiple enable_if_t args: I would have that that would be unnecessary, but if it is necessary (my own template metaprogramming sometimes needs a little bit of trial and error!), you could split the above into three separate ones.

Note that the comparison against bool is moved because is_signed explicitly excludes bool, but is_unsigned includes it.

You could, if you wanted, defined your own type trait that folds together the is_integral and sizeof terms, to avoid the duplication, but I'm not sure it would add enough to be worth it.

Is there any form of testing that shows that the different fundamental types create the correct underlying type?

llvm/include/llvm/Support/ScopedPrinter.h

214

Looking at the Value class above, it only has explicit overloads for uint64_t and int64_t (effectively). Other integer types happen to work because these two overloads have ended up being instantiated and so the types can convert. I might be wrong, but I suspect if you tried to use that class with e.g. an int value, but without any usage with an int64_t, you'd get a missing overload error, or something stranger (like it converting to a pointer or something)! I'm a little unsure which one char will use, and I suspect it's platform dependent.

In your case, I think you could get away with long long and unsigned long long, but in your testing, I'd test with every fundamental integer type to make sure it works without a problem.

[paulkirth]

llvm/include/llvm/Support/JSON.h

351–352

I don't think the changes you've made will work in a cross-platform manner for the same sort of reasoning as why you need the unsigned long overloads of printNumber: an unsigned long is a 64-bit value on some platforms, but uint64_t might be defined to unsigned long long. In such cases, unsigned long will be neither uin64_t, nor unsigned long long, and so the enable_ifs won't work.

A 64-bit integral will have the following traits: std::is_integral, and sizeof(T) == sizeof(uint64_t) (you could hard code the latter to 8). So the current behaviour of comparing the type against uint64_t isn't really correct. I think you want to change the unsigned case to:

template <
    typename T,
    typename = std::enable_if_t<std::is_integral<T>::value &&
                                std::is_unsigned<T>::value &&
                                !std::is_same<T, bool>::value &&
                                sizeof(T) == sizeof(uint64_t)>>

The signed one should look like the following:

template <
    typename T,
    typename = std::enable_if_t<std::is_integral<T>::value &&
                                std::is_signed<T>::value &&
                                sizeof(T) == sizeof(int64_t)>>

Huh, I forgot you could use sizeof in template expressions. That certainly cleans things up. I think for the signed case we will want: sizeof(T) <= sizeof(int64_t), since this code assumes everything smaller than uint64 is signed. It also seems that some dependent code makes similar assumptions.

Honestly, I'm not sure why the second one is listed as multiple enable_if_t args: I would have that that would be unnecessary, but if it is necessary (my own template metaprogramming sometimes needs a little bit of trial and error!), you could split the above into three separate ones.

I just went with the existing convention. I think your suggestion should work, but I haven't tried it yet.

Note that the comparison against bool is moved because is_signed explicitly excludes bool, but is_unsigned includes it.

You could, if you wanted, defined your own type trait that folds together the is_integral and sizeof terms, to avoid the duplication, but I'm not sure it would add enough to be worth it.

Is there any form of testing that shows that the different fundamental types create the correct underlying type?

The existing tests in JSONTest.cpp all seem to use int64_t/uint64_t. That makes sense to an extent, since they more or less forced anything that isn't uint64_t into int64_t. But there isn't any testing of the underlying C++ types.

llvm/include/llvm/Support/ScopedPrinter.h

214

Yeah, I the more I think about this, the more I feel like this will be simpler to implement. let me give this a shot and see if that's actually the case. And you're right, char is _probably_ going to need some kind of special handling.

[paulkirth]

Fix enable_if and update getInteger to allow more unsigned types

I may want to revisit how we split signed/unsigned.
I also still need to investigate implicit conversion.

[paulkirth]

after looking at this for a bit, I think this is probably the way to go. Using a template in ScopedPrinter.h has some problems, because there are types like ulittle32_t that won't satisfy is_integral. Using conversion also has its issues, like making some calls ambiguous.

[jhenderson]

llvm/include/llvm/Support/JSON.h

343–344

You may need to update these comments.

344–346

It's probably worth calling out that you've changed the behaviour somewhat. I fully expect the behaviour change to be harmless, but it's worth pointing out in the commit message.

351–352

I think there's a lot to be said for having testing for the fundamental types, because ultimately those are what are actually used by the compiler.

427

Are there existing unit tests for this code? I'm slightly concerned that we might accidentally break the intended behaviour without them.

llvm/include/llvm/Support/ScopedPrinter.h

215

I think C++-style casts are generally preferred.

[junhee-yoo]

llvm/include/llvm/Support/JSON.h

355

I guess this sizeof(T) <= sizeof(int64_t) and sizeof(T) <= sizeof(uint64_t) are not affect because all types are equal or smaller than 64bits types.

In my humble opinion, how about keep the intention of original code:

• for unsigned 64bits types - uint64_t:

template <typename T,
          typename = std::enable_if_t<!std::is_same_v<T, bool>&&
                                      std::is_integral_v<T> &&
                                      std::is_unsigned_v<T> &&
                                      sizeof(T) == sizeof(uint64_t)>>

• for rests - int64_t:

template <typename T,
          typename = std::enable_if_t<std::is_integral_v<T>>,
          typename = std::enable_if_t<!std::is_same_v<T, bool>>,
          typename = std::enable_if_t<std::is_integral_v<T> &&
                                      !(std::is_unsigned_v<T> &&
                                        sizeof(T) == sizeof(uint64_t))>>

If it looks too verbose, I recommend to use:

template<typenameT>
constexpr bool is_uint_64_bits_v = 
  std::is_integral_v<T> && std::is_unsigned_v<T> && sizeof(T) == sizeof(uint64_t);

[paulkirth]

Rebase and address comments.

• Add test for C++ types in JSONTest.cpp
• Restore original behavior in converting all integers representable as int64_t
• Simplify template expression.
• git clang-format

[paulkirth]

Sorry of the delay on this, I was a bit under the weather and had some other work take up more time than I planned.

I think this version makes reasonable tradeoffs between the old semantics and making use of the underlying C++ types. If there is something else that needs to be addressed, please let me know.

llvm/include/llvm/Support/JSON.h

77

I'm not 100% on using a namespace here (or naming it internal), but I didn't want to introduce this into llvm::json by accident.

If there is a preference, or an llvm coding preference I've missed, please let me know. https://llvm.org/docs/CodingStandards.html#anonymous-namespaces didn't seem to be relevant from my reading, but I'm happy to adopt something else here that is more in line w/ the rest of the codebase.

344–346

Since I've restored the behavior, I think its best to leave them as is.

427

Yes, these are tested in https://github.com/llvm/llvm-project/blob/0f9842c21c76a1c36866589770f92cc63619d36a/llvm/unittests/Support/JSONTest.cpp#L352.

There are other relevant tests in the file too, for JSON Numbers, the asUINT64() API, and conversions between them.

[jhenderson]

llvm/include/llvm/Support/JSON.h

77

I'm not sure it matters if this ends up in the llvm::json namespace, although perhaps it should just be in the llvm namespace, since it's not JSON specific.

I don't know of any specific rules here. Generally, I'd avoid anonymous namespaces (and static functions/variables) in header files, because that would result in a copy of it in every object file that uses the header, but in this particular case, I suspect it wouldn't since this is a constexpr value, so using an anonymous namespace (or possibly static - not sure if that can be used in this context), should be okay, I think?

FWIW, I'm not convinced about using an "internal" namespace.

343–344

Can we drop "long" from the comment, please?

llvm/include/llvm/Support/ScopedPrinter.h

214

I think you need a specific char overload too? char is a separate fundamental type to signed char and unsigned char.

228

Nit: missing blank line between methods.

llvm/unittests/Support/JSONTest.cpp

459

Nits. (I'm pretty sure "C++" is the "correct" way of writing it, but I could be wrong).

462

You need a signed char test case too.

[paulkirth]

Address comments

• update code comments
• add overload for char
• add missing test case for signed char
• remove internal namespace

[jhenderson]

Looks good to me.

llvm/include/llvm/Support/JSON.h

 //   - When creating Values from strings, callers must ensure they are UTF-8.
 //        with asserts on, invalid UTF-8 will crash the program
 //        with asserts off, we'll substitute the replacement character (U+FFFD)
 //     Callers can use json::isUTF8() and json::fixUTF8() for validation.
 //
 //   - When retrieving strings from Values (e.g. asString()), the result will
 //     always be valid UTF-8.
 
+template <typename T>
+constexpr bool is_uint_64_bit_v =
+    std::is_integral_v<T> && std::is_unsigned_v<T> &&
+    sizeof(T) == sizeof(uint64_t);
+
 /// Returns true if \p S is valid UTF-8, which is required for use as JSON.
 /// If it returns false, \p Offset is set to a byte offset near the first error.
 bool isUTF8(llvm::StringRef S, size_t *ErrOffset = nullptr);
 /// Replaces invalid UTF-8 sequences in \p S with the replacement character
 /// (U+FFFD). The returned string is valid UTF-8.
 /// This is much slower than isUTF8, so test that first.
 std::string fixUTF8(llvm::StringRef S);
 
public:
   // Boolean (disallow implicit conversions).
   // (The last template parameter is a dummy to keep templates distinct.)
   template <typename T,
             typename = std::enable_if_t<std::is_same<T, bool>::value>,
             bool = false>
   Value(T B) : Type(T_Boolean) {
     create<bool>(B);
   }
 
-  // Unsigned 64-bit long integers.
-  template <typename T,
-            typename = std::enable_if_t<std::is_same<T, uint64_t>::value>,
-            bool = false, bool = false>
+  // Unsigned 64-bit integers.
+  template <typename T, typename = std::enable_if_t<is_uint_64_bit_v<T>>>
   Value(T V) : Type(T_UINT64) {
     create<uint64_t>(uint64_t{V});
   }
 
   // Integers (except boolean and uint64_t).
   // Must be non-narrowing convertible to int64_t.
-  template <typename T, typename = std::enable_if_t<std::is_integral<T>::value>,
-            typename = std::enable_if_t<!std::is_same<T, bool>::value>,
-            typename = std::enable_if_t<!std::is_same<T, uint64_t>::value>>
+  template <typename T, typename = std::enable_if_t<std::is_integral_v<T>>,
+            typename = std::enable_if_t<!std::is_same_v<T, bool>>,
+            typename = std::enable_if_t<!is_uint_64_bit_v<T>>>
   Value(T I) : Type(T_Integer) {
     create<int64_t>(int64_t{I});
   }
   // Floating point. Must be non-narrowing convertible to double.
   template <typename T,
             typename = std::enable_if_t<std::is_floating_point<T>::value>,
             double * = nullptr>
   Value(T D) : Type(T_Double) {
     create<double>(double{D});
if (LLVM_LIKELY(Type == T_Double))
       return as<double>();
     if (LLVM_LIKELY(Type == T_Integer))
       return as<int64_t>();
     if (LLVM_LIKELY(Type == T_UINT64))
       return as<uint64_t>();
     return std::nullopt;
   }
   // Succeeds if the Value is a Number, and exactly representable as int64_t.
   std::optional<int64_t> getAsInteger() const {
     if (LLVM_LIKELY(Type == T_Integer))
       return as<int64_t>();
+    if (LLVM_LIKELY(Type == T_UINT64)) {
+      uint64_t U = as<uint64_t>();
+      if (LLVM_LIKELY(U <= uint64_t(std::numeric_limits<int64_t>::max()))) {
+        return U;
+      }
+    }
     if (LLVM_LIKELY(Type == T_Double)) {
       double D = as<double>();
       if (LLVM_LIKELY(std::modf(D, &D) == 0.0 &&
                       D >= double(std::numeric_limits<int64_t>::min()) &&
                       D <= double(std::numeric_limits<int64_t>::max())))
         return D;
     }
     return std::nullopt;

llvm/include/llvm/Support/ScopedPrinter.h

while (Curr > 0) {
       if (Curr & 1)
         SetFlags.emplace_back(Flag);
       Curr >>= 1;
       Flag <<= 1;
     }
     printFlagsImpl(Label, hex(Value), SetFlags);
   }
 
-  virtual void printNumber(StringRef Label, uint64_t Value) {
+  virtual void printNumber(StringRef Label, char Value) {
+    startLine() << Label << ": " << static_cast<int>(Value) << "\n";
+  }
+
+  virtual void printNumber(StringRef Label, signed char Value) {
+    startLine() << Label << ": " << static_cast<int>(Value) << "\n";
+  }
+
+  virtual void printNumber(StringRef Label, unsigned char Value) {
+    startLine() << Label << ": " << static_cast<unsigned>(Value) << "\n";
+  }
+
+  virtual void printNumber(StringRef Label, short Value) {
     startLine() << Label << ": " << Value << "\n";
   }
 
-  virtual void printNumber(StringRef Label, uint32_t Value) {
+  virtual void printNumber(StringRef Label, unsigned short Value) {
     startLine() << Label << ": " << Value << "\n";
   }
 
-  virtual void printNumber(StringRef Label, uint16_t Value) {
+  virtual void printNumber(StringRef Label, int Value) {
     startLine() << Label << ": " << Value << "\n";
   }
 
-  virtual void printNumber(StringRef Label, uint8_t Value) {
-    startLine() << Label << ": " << unsigned(Value) << "\n";
+  virtual void printNumber(StringRef Label, unsigned int Value) {
+    startLine() << Label << ": " << Value << "\n";
   }
 
-  virtual void printNumber(StringRef Label, int64_t Value) {
+  virtual void printNumber(StringRef Label, long Value) {
     startLine() << Label << ": " << Value << "\n";
   }
 
-  virtual void printNumber(StringRef Label, int32_t Value) {
+  virtual void printNumber(StringRef Label, unsigned long Value) {
     startLine() << Label << ": " << Value << "\n";
   }
 
-  virtual void printNumber(StringRef Label, int16_t Value) {
+  virtual void printNumber(StringRef Label, long long Value) {
     startLine() << Label << ": " << Value << "\n";
   }
 
-  virtual void printNumber(StringRef Label, int8_t Value) {
-    startLine() << Label << ": " << int(Value) << "\n";
+  virtual void printNumber(StringRef Label, unsigned long long Value) {
+    startLine() << Label << ": " << Value << "\n";
   }
 
   virtual void printNumber(StringRef Label, const APSInt &Value) {
     startLine() << Label << ": " << Value << "\n";
   }
 
   virtual void printNumber(StringRef Label, float Value) {
     startLine() << Label << ": " << format("%5.1f", Value) << "\n";
public:
   JSONScopedPrinter(raw_ostream &OS, bool PrettyPrint = false,
                     std::unique_ptr<DelimitedScope> &&OuterScope =
                         std::unique_ptr<DelimitedScope>{});
 
   static bool classof(const ScopedPrinter *SP) {
     return SP->getKind() == ScopedPrinter::ScopedPrinterKind::JSON;
   }
 
-  void printNumber(StringRef Label, uint64_t Value) override {
+  void printNumber(StringRef Label, char Value) override {
+    JOS.attribute(Label, Value);
+  }
+
+  void printNumber(StringRef Label, signed char Value) override {
+    JOS.attribute(Label, Value);
+  }
+
+  void printNumber(StringRef Label, unsigned char Value) override {
+    JOS.attribute(Label, Value);
+  }
+
+  void printNumber(StringRef Label, short Value) override {
     JOS.attribute(Label, Value);
   }
 
-  void printNumber(StringRef Label, uint32_t Value) override {
+  void printNumber(StringRef Label, unsigned short Value) override {
     JOS.attribute(Label, Value);
   }
 
-  void printNumber(StringRef Label, uint16_t Value) override {
+  void printNumber(StringRef Label, int Value) override {
     JOS.attribute(Label, Value);
   }
 
-  void printNumber(StringRef Label, uint8_t Value) override {
+  void printNumber(StringRef Label, unsigned int Value) override {
     JOS.attribute(Label, Value);
   }
 
-  void printNumber(StringRef Label, int64_t Value) override {
+  void printNumber(StringRef Label, long Value) override {
     JOS.attribute(Label, Value);
   }
 
-  void printNumber(StringRef Label, int32_t Value) override {
+  void printNumber(StringRef Label, unsigned long Value) override {
     JOS.attribute(Label, Value);
   }
 
-  void printNumber(StringRef Label, int16_t Value) override {
+  void printNumber(StringRef Label, long long Value) override {
     JOS.attribute(Label, Value);
   }
 
-  void printNumber(StringRef Label, int8_t Value) override {
+  void printNumber(StringRef Label, unsigned long long Value) override {
     JOS.attribute(Label, Value);
   }
 
   void printNumber(StringRef Label, float Value) override {
     JOS.attribute(Label, Value);
   }
 
   void printNumber(StringRef Label, double Value) override {

llvm/tools/llvm-readobj/ELFDumper.cpp

void LLVMELFDumper<ELFT>::printHashHistogramStats(size_t NBucket,
                                                   size_t MaxChain,
                                                   size_t TotalSyms,
                                                   ArrayRef<size_t> Count,
                                                   bool IsGnu) const {
   StringRef HistName = IsGnu ? "GnuHashHistogram" : "HashHistogram";
   StringRef BucketName = IsGnu ? "Bucket" : "Chain";
   StringRef ListName = IsGnu ? "Buckets" : "Chains";
   DictScope Outer(W, HistName);
-  W.printNumber("TotalBuckets", static_cast<uint64_t>(NBucket));
+  W.printNumber("TotalBuckets", NBucket);
   ListScope Buckets(W, ListName);
   size_t CumulativeNonZero = 0;
   for (size_t I = 0; I < MaxChain; ++I) {
     CumulativeNonZero += Count[I] * I;
     DictScope Bucket(W, BucketName);
-    W.printNumber("Length", static_cast<uint64_t>(I));
-    W.printNumber("Count", static_cast<uint64_t>(Count[I]));
+    W.printNumber("Length", I);
+    W.printNumber("Count", Count[I]);
     W.printNumber("Percentage", (float)(Count[I] * 100.0) / NBucket);
     W.printNumber("Coverage", (float)(CumulativeNonZero * 100.0) / TotalSyms);
   }
 }
 
 // Returns true if rel/rela section exists, and populates SymbolIndices.
 // Otherwise returns false.
 template <class ELFT>

llvm/unittests/Support/JSONTest.cpp

// Test with a large number that is malformed.
     const char *Str = "184467440737095516150.12.12";
     llvm::Expected<Value> Doc = parse(Str);
 
     EXPECT_EQ("[1:27, byte=27]: Invalid JSON value (number?)",
               llvm::toString(Doc.takeError()));
   }
 }
 
+template <typename T> void checkCppIntegers() {
+  Value Val = T{10};
+  T Var = 10;
+  EXPECT_EQ(Val, Var);
+
+  Val = T{std::numeric_limits<T>::max()};
+  Var = std::numeric_limits<T>::max();
+  EXPECT_EQ(Val, Var);
+
+  Val = T{std::numeric_limits<T>::min()};
+  Var = std::numeric_limits<T>::min();
+  EXPECT_EQ(Val, Var);
+}
+
+// Test that underlying C++ integer types behave as expected.
+TEST(JSONTest, CppIntegers) {
+  checkCppIntegers<char>();
+  checkCppIntegers<signed char>();
+  checkCppIntegers<unsigned char>();
+
+  checkCppIntegers<short>();
+  checkCppIntegers<unsigned short>();
+
+  checkCppIntegers<int>();
+  checkCppIntegers<unsigned int>();
+
+  checkCppIntegers<long>();
+  checkCppIntegers<unsigned long>();
+
+  checkCppIntegers<long long>();
+  checkCppIntegers<unsigned long long>();
+}
+
 // Sample struct with typical JSON-mapping rules.
 struct CustomStruct {
   CustomStruct() : B(false) {}
   CustomStruct(std::string S, std::optional<int> I, bool B)
       : S(S), I(I), B(B) {}
   std::string S;
   std::optional<int> I;
   bool B;