[compiler-rt] Add ConsumeProbability and ConsumeFloatingPoint methods to FDP.

[Dor1s]

Summary

Also slightly cleaned up the comments and changed the header's extension
back to .h as per comments on https://reviews.llvm.org/D65812.

New methods added:

• ConsumeProbability returns [0.0, 1.0] by consuming an unsigned integer value from the input data and dividing that value by the integer's max value.
• ConsumeFloatingPointInRange returns a floating point value in the given range. Relies on ConsumeProbability method. This method does not have the limitation of std::uniform_real_distribution that requires the given range to be <= the floating point type's max. If the range is too large, this implementation will additionally call ConsumeBool to decide whether the result will be in the first or the second half of the range.
• ConsumeFloatingPoint returns a floating point value in the range [std::numeric_limits<T>::lowest(), std::numeric_limits<T>::min()].

Tested on Linux, Mac, Windows.

[Dor1s]

Add parens to the condition for better readability

[Dor1s]

@morehouse Ready for review, PTAL once you get a chance :)

[morehouse]

include/fuzzer/FuzzedDataProvider.hpp

224 ↗

(On Diff #213999)

Nit: min <= max, so this is equivalent to min >= 0 || max <= 0.

234 ↗

(On Diff #213999)

This seems like it might favor the middle part of the range over others. Probably not a big issue for fuzzing, but might be worth thinking about.

[Dor1s]

include/fuzzer/FuzzedDataProvider.hpp

224 ↗

(On Diff #213999)

Oh, this is smart!

234 ↗

(On Diff #213999)

Great point! Some other methods here explicitly say in the comments that the results distribution is not uniform, so yeah it might be not a big deal, but your comments made me think more and now I have a slightly different implementation in mind. Will upload it tomorrow. Thanks, Matt!

[Dor1s]

Re-write ConsumeFloatingPointInRange, change extension back to .h

[Dor1s]

Re-written ConsumeFloatingPointInRange in another fashion. Will describe the new logic in the description.

[Dor1s]

@morehouse the CL got a little messy because of the renaming, but I'd appreciate if you could take a look at the new version of ConsumeFloatingPointInRange, it's ~20 lines. Thanks!

[morehouse]

include/fuzzer/FuzzedDataProvider.h

229 ↗

(On Diff #214162)

Nice!

compiler-rt/trunk/include/CMakeLists.txt

set(SANITIZER_HEADERS
     sanitizer/lsan_interface.h
     sanitizer/msan_interface.h
     sanitizer/netbsd_syscall_hooks.h
     sanitizer/scudo_interface.h
     sanitizer/tsan_interface.h
     sanitizer/tsan_interface_atomic.h
     )
   set(FUZZER_HEADERS
-    fuzzer/FuzzedDataProvider.hpp
+    fuzzer/FuzzedDataProvider.h
     )
 endif(COMPILER_RT_BUILD_SANITIZERS)
 
 if (COMPILER_RT_BUILD_XRAY)
   set(XRAY_HEADERS
     xray/xray_interface.h
     xray/xray_log_interface.h
     xray/xray_records.h

compiler-rt/trunk/include/fuzzer/FuzzedDataProvider.h

+//===- FuzzedDataProvider.h - Utility header for fuzz targets ---*- C++ -* ===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// A single header library providing an utility class to break up an array of
+// bytes. Whenever run on the same input, provides the same output, as long as
+// its methods are called in the same order, with the same arguments.
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_FUZZER_FUZZED_DATA_PROVIDER_H_
+#define LLVM_FUZZER_FUZZED_DATA_PROVIDER_H_
+
+#include <limits.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#include <algorithm>
+#include <cstring>
+#include <initializer_list>
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <vector>
+
+// In addition to the comments below, the API is also briefly documented at
+// https://github.com/google/fuzzing/blob/master/docs/split-inputs.md#fuzzed-data-provider
+class FuzzedDataProvider {
+ public:
+  // |data| is an array of length |size| that the FuzzedDataProvider wraps to
+  // provide more granular access. |data| must outlive the FuzzedDataProvider.
+  FuzzedDataProvider(const uint8_t *data, size_t size)
+      : data_ptr_(data), remaining_bytes_(size) {}
+  ~FuzzedDataProvider() = default;
+
+  // Returns a std::vector containing |num_bytes| of input data. If fewer than
+  // |num_bytes| of data remain, returns a shorter std::vector containing all
+  // of the data that's left. Can be used with any byte sized type, such as
+  // char, unsigned char, uint8_t, etc.
+  template <typename T> std::vector<T> ConsumeBytes(size_t num_bytes) {
+    num_bytes = std::min(num_bytes, remaining_bytes_);
+    return ConsumeBytes<T>(num_bytes, num_bytes);
+  }
+
+  // Similar to |ConsumeBytes|, but also appends the terminator value at the end
+  // of the resulting vector. Useful, when a mutable null-terminated C-string is
+  // needed, for example. But that is a rare case. Better avoid it, if possible,
+  // and prefer using |ConsumeBytes| or |ConsumeBytesAsString| methods.
+  template <typename T>
+  std::vector<T> ConsumeBytesWithTerminator(size_t num_bytes,
+                                            T terminator = 0) {
+    num_bytes = std::min(num_bytes, remaining_bytes_);
+    std::vector<T> result = ConsumeBytes<T>(num_bytes + 1, num_bytes);
+    result.back() = terminator;
+    return result;
+  }
+
+  // Returns a std::string containing |num_bytes| of input data. Using this and
+  // |.c_str()| on the resulting string is the best way to get an immutable
+  // null-terminated C string. If fewer than |num_bytes| of data remain, returns
+  // a shorter std::string containing all of the data that's left.
+  std::string ConsumeBytesAsString(size_t num_bytes) {
+    static_assert(sizeof(std::string::value_type) == sizeof(uint8_t),
+                  "ConsumeBytesAsString cannot convert the data to a string.");
+
+    num_bytes = std::min(num_bytes, remaining_bytes_);
+    std::string result(
+        reinterpret_cast<const std::string::value_type *>(data_ptr_),
+        num_bytes);
+    Advance(num_bytes);
+    return result;
+  }
+
+  // Returns a number in the range [min, max] by consuming bytes from the
+  // input data. The value might not be uniformly distributed in the given
+  // range. If there's no input data left, always returns |min|. |min| must
+  // be less than or equal to |max|.
+  template <typename T> T ConsumeIntegralInRange(T min, T max) {
+    static_assert(std::is_integral<T>::value, "An integral type is required.");
+    static_assert(sizeof(T) <= sizeof(uint64_t), "Unsupported integral type.");
+
+    if (min > max)
+      abort();
+
+    // Use the biggest type possible to hold the range and the result.
+    uint64_t range = static_cast<uint64_t>(max) - min;
+    uint64_t result = 0;
+    size_t offset = 0;
+
+    while (offset < sizeof(T) * CHAR_BIT && (range >> offset) > 0 &&
+           remaining_bytes_ != 0) {
+      // Pull bytes off the end of the seed data. Experimentally, this seems to
+      // allow the fuzzer to more easily explore the input space. This makes
+      // sense, since it works by modifying inputs that caused new code to run,
+      // and this data is often used to encode length of data read by
+      // |ConsumeBytes|. Separating out read lengths makes it easier modify the
+      // contents of the data that is actually read.
+      --remaining_bytes_;
+      result = (result << CHAR_BIT) | data_ptr_[remaining_bytes_];
+      offset += CHAR_BIT;
+    }
+
+    // Avoid division by 0, in case |range + 1| results in overflow.
+    if (range != std::numeric_limits<decltype(range)>::max())
+      result = result % (range + 1);
+
+    return static_cast<T>(min + result);
+  }
+
+  // Returns a std::string of length from 0 to |max_length|. When it runs out of
+  // input data, returns what remains of the input. Designed to be more stable
+  // with respect to a fuzzer inserting characters than just picking a random
+  // length and then consuming that many bytes with |ConsumeBytes|.
+  std::string ConsumeRandomLengthString(size_t max_length) {
+    // Reads bytes from the start of |data_ptr_|. Maps "\\" to "\", and maps "\"
+    // followed by anything else to the end of the string. As a result of this
+    // logic, a fuzzer can insert characters into the string, and the string
+    // will be lengthened to include those new characters, resulting in a more
+    // stable fuzzer than picking the length of a string independently from
+    // picking its contents.
+    std::string result;
+
+    // Reserve the anticipated capaticity to prevent several reallocations.
+    result.reserve(std::min(max_length, remaining_bytes_));
+    for (size_t i = 0; i < max_length && remaining_bytes_ != 0; ++i) {
+      char next = ConvertUnsignedToSigned<char>(data_ptr_[0]);
+      Advance(1);
+      if (next == '\\' && remaining_bytes_ != 0) {
+        next = ConvertUnsignedToSigned<char>(data_ptr_[0]);
+        Advance(1);
+        if (next != '\\')
+          break;
+      }
+      result += next;
+    }
+
+    result.shrink_to_fit();
+    return result;
+  }
+
+  // Returns a std::vector containing all remaining bytes of the input data.
+  template <typename T> std::vector<T> ConsumeRemainingBytes() {
+    return ConsumeBytes<T>(remaining_bytes_);
+  }
+
+  // Returns a std::string containing all remaining bytes of the input data.
+  // Prefer using |ConsumeRemainingBytes| unless you actually need a std::string
+  // object.
+  std::string ConsumeRemainingBytesAsString() {
+    return ConsumeBytesAsString(remaining_bytes_);
+  }
+
+  // Returns a number in the range [Type's min, Type's max]. The value might
+  // not be uniformly distributed in the given range. If there's no input data
+  // left, always returns |min|.
+  template <typename T> T ConsumeIntegral() {
+    return ConsumeIntegralInRange(std::numeric_limits<T>::min(),
+                                  std::numeric_limits<T>::max());
+  }
+
+  // Reads one byte and returns a bool, or false when no data remains.
+  bool ConsumeBool() { return 1 & ConsumeIntegral<uint8_t>(); }
+
+  // Returns a copy of the value selected from the given fixed-size |array|.
+  template <typename T, size_t size>
+  T PickValueInArray(const T (&array)[size]) {
+    static_assert(size > 0, "The array must be non empty.");
+    return array[ConsumeIntegralInRange<size_t>(0, size - 1)];
+  }
+
+  template <typename T>
+  T PickValueInArray(std::initializer_list<const T> list) {
+    // TODO(Dor1s): switch to static_assert once C++14 is allowed.
+    if (!list.size())
+      abort();
+
+    return *(list.begin() + ConsumeIntegralInRange<size_t>(0, list.size() - 1));
+  }
+
+  // Returns an enum value. The enum must start at 0 and be contiguous. It must
+  // also contain |kMaxValue| aliased to its largest (inclusive) value. Such as:
+  // enum class Foo { SomeValue, OtherValue, kMaxValue = OtherValue };
+  template <typename T> T ConsumeEnum() {
+    static_assert(std::is_enum<T>::value, "|T| must be an enum type.");
+    return static_cast<T>(ConsumeIntegralInRange<uint32_t>(
+        0, static_cast<uint32_t>(T::kMaxValue)));
+  }
+
+  // Returns a floating point number in the range [0.0, 1.0]. If there's no
+  // input data left, always returns 0.
+  template <typename T> T ConsumeProbability() {
+    static_assert(std::is_floating_point<T>::value,
+                  "A floating point type is required.");
+
+    // Use different integral types for different floating point types in order
+    // to provide better density of the resulting values.
+    using IntegralType =
+        typename std::conditional<sizeof(T) <= sizeof(uint32_t), uint32_t,
+                                  uint64_t>::type;
+
+    T result = static_cast<T>(ConsumeIntegral<IntegralType>());
+    result /= static_cast<T>(std::numeric_limits<IntegralType>::max());
+    return result;
+  }
+
+  // Returns a floating point value in the range [Type's lowest, Type's max] by
+  // consuming bytes from the input data. If there's no input data left, always
+  // returns approximately 0.
+  template <typename T> T ConsumeFloatingPoint() {
+    return ConsumeFloatingPointInRange<T>(std::numeric_limits<T>::lowest(),
+                                          std::numeric_limits<T>::max());
+  }
+
+  // Returns a floating point value in the given range by consuming bytes from
+  // the input data. If there's no input data left, returns |min|. Note that
+  // |min| must be less than or equal to |max|.
+  template <typename T> T ConsumeFloatingPointInRange(T min, T max) {
+    if (min > max)
+      abort();
+
+    T range = .0;
+    T result = min;
+    constexpr T zero(.0);
+    if (max > zero && min < zero && max > min + std::numeric_limits<T>::max()) {
+      // The diff |max - min| would overflow the given floating point type. Use
+      // the half of the diff as the range and consume a bool to decide whether
+      // the result is in the first of the second part of the diff.
+      range = (max / 2.0) - (min / 2.0);
+      if (ConsumeBool()) {
+        result += range;
+      }
+    } else {
+      range = max - min;
+    }
+
+    return result + range * ConsumeProbability<T>();
+  }
+
+  // Reports the remaining bytes available for fuzzed input.
+  size_t remaining_bytes() { return remaining_bytes_; }
+
+ private:
+  FuzzedDataProvider(const FuzzedDataProvider &) = delete;
+  FuzzedDataProvider &operator=(const FuzzedDataProvider &) = delete;
+
+  void Advance(size_t num_bytes) {
+    if (num_bytes > remaining_bytes_)
+      abort();
+
+    data_ptr_ += num_bytes;
+    remaining_bytes_ -= num_bytes;
+  }
+
+  template <typename T>
+  std::vector<T> ConsumeBytes(size_t size, size_t num_bytes_to_consume) {
+    static_assert(sizeof(T) == sizeof(uint8_t), "Incompatible data type.");
+
+    // The point of using the size-based constructor below is to increase the
+    // odds of having a vector object with capacity being equal to the length.
+    // That part is always implementation specific, but at least both libc++ and
+    // libstdc++ allocate the requested number of bytes in that constructor,
+    // which seems to be a natural choice for other implementations as well.
+    // To increase the odds even more, we also call |shrink_to_fit| below.
+    std::vector<T> result(size);
+    std::memcpy(result.data(), data_ptr_, num_bytes_to_consume);
+    Advance(num_bytes_to_consume);
+
+    // Even though |shrink_to_fit| is also implementation specific, we expect it
+    // to provide an additional assurance in case vector's constructor allocated
+    // a buffer which is larger than the actual amount of data we put inside it.
+    result.shrink_to_fit();
+    return result;
+  }
+
+  template <typename TS, typename TU> TS ConvertUnsignedToSigned(TU value) {
+    static_assert(sizeof(TS) == sizeof(TU), "Incompatible data types.");
+    static_assert(!std::numeric_limits<TU>::is_signed,
+                  "Source type must be unsigned.");
+
+    // TODO(Dor1s): change to `if constexpr` once C++17 becomes mainstream.
+    if (std::numeric_limits<TS>::is_modulo)
+      return static_cast<TS>(value);
+
+    // Avoid using implementation-defined unsigned to signer conversions.
+    // To learn more, see https://stackoverflow.com/questions/13150449.
+    if (value <= std::numeric_limits<TS>::max())
+      return static_cast<TS>(value);
+    else {
+      constexpr auto TS_min = std::numeric_limits<TS>::min();
+      return TS_min + static_cast<char>(value - TS_min);
+    }
+  }
+
+  const uint8_t *data_ptr_;
+  size_t remaining_bytes_;
+};
+
+#endif // LLVM_FUZZER_FUZZED_DATA_PROVIDER_H_

compiler-rt/trunk/include/fuzzer/FuzzedDataProvider.hpp

-//===- FuzzedDataProvider.hpp - Utility header for fuzz targets -*- C++ -* ===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-// A single header library providing an utility class to break up an array of
-// bytes. Whenever run on the same input, provides the same output, as long as
-// its methods are called in the same order, with the same arguments.
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_FUZZER_FUZZED_DATA_PROVIDER_H_
-#define LLVM_FUZZER_FUZZED_DATA_PROVIDER_H_
-
-#include <limits.h>
-#include <stddef.h>
-#include <stdint.h>
-
-#include <algorithm>
-#include <cstring>
-#include <initializer_list>
-#include <string>
-#include <type_traits>
-#include <utility>
-#include <vector>
-
-class FuzzedDataProvider {
- public:
-  // |data| is an array of length |size| that the FuzzedDataProvider wraps to
-  // provide more granular access. |data| must outlive the FuzzedDataProvider.
-  FuzzedDataProvider(const uint8_t *data, size_t size)
-      : data_ptr_(data), remaining_bytes_(size) {}
-  ~FuzzedDataProvider() = default;
-
-  // Returns a std::vector containing |num_bytes| of input data. If fewer than
-  // |num_bytes| of data remain, returns a shorter std::vector containing all
-  // of the data that's left. Can be used with any byte sized type, such as
-  // char, unsigned char, uint8_t, etc.
-  template <typename T> std::vector<T> ConsumeBytes(size_t num_bytes) {
-    num_bytes = std::min(num_bytes, remaining_bytes_);
-    return ConsumeBytes<T>(num_bytes, num_bytes);
-  }
-
-  // Similar to |ConsumeBytes|, but also appends the terminator value at the end
-  // of the resulting vector. Useful, when a mutable null-terminated C-string is
-  // needed, for example. But that is a rare case. Better avoid it, if possible,
-  // and prefer using |ConsumeBytes| or |ConsumeBytesAsString| methods.
-  template <typename T>
-  std::vector<T> ConsumeBytesWithTerminator(size_t num_bytes,
-                                            T terminator = 0) {
-    num_bytes = std::min(num_bytes, remaining_bytes_);
-    std::vector<T> result = ConsumeBytes<T>(num_bytes + 1, num_bytes);
-    result.back() = terminator;
-    return result;
-  }
-
-  // Returns a std::string containing |num_bytes| of input data. Using this and
-  // |.c_str()| on the resulting string is the best way to get an immutable
-  // null-terminated C string. If fewer than |num_bytes| of data remain, returns
-  // a shorter std::string containing all of the data that's left.
-  std::string ConsumeBytesAsString(size_t num_bytes) {
-    static_assert(sizeof(std::string::value_type) == sizeof(uint8_t),
-                  "ConsumeBytesAsString cannot convert the data to a string.");
-
-    num_bytes = std::min(num_bytes, remaining_bytes_);
-    std::string result(
-        reinterpret_cast<const std::string::value_type *>(data_ptr_),
-        num_bytes);
-    Advance(num_bytes);
-    return result;
-  }
-
-  // Returns a number in the range [min, max] by consuming bytes from the
-  // input data. The value might not be uniformly distributed in the given
-  // range. If there's no input data left, always returns |min|. |min| must
-  // be less than or equal to |max|.
-  template <typename T> T ConsumeIntegralInRange(T min, T max) {
-    static_assert(std::is_integral<T>::value, "An integral type is required.");
-    static_assert(sizeof(T) <= sizeof(uint64_t), "Unsupported integral type.");
-
-    if (min > max)
-      abort();
-
-    // Use the biggest type possible to hold the range and the result.
-    uint64_t range = static_cast<uint64_t>(max) - min;
-    uint64_t result = 0;
-    size_t offset = 0;
-
-    while (offset < sizeof(T) * CHAR_BIT && (range >> offset) > 0 &&
-           remaining_bytes_ != 0) {
-      // Pull bytes off the end of the seed data. Experimentally, this seems to
-      // allow the fuzzer to more easily explore the input space. This makes
-      // sense, since it works by modifying inputs that caused new code to run,
-      // and this data is often used to encode length of data read by
-      // |ConsumeBytes|. Separating out read lengths makes it easier modify the
-      // contents of the data that is actually read.
-      --remaining_bytes_;
-      result = (result << CHAR_BIT) | data_ptr_[remaining_bytes_];
-      offset += CHAR_BIT;
-    }
-
-    // Avoid division by 0, in case |range + 1| results in overflow.
-    if (range != std::numeric_limits<decltype(range)>::max())
-      result = result % (range + 1);
-
-    return static_cast<T>(min + result);
-  }
-
-  // Returns a std::string of length from 0 to |max_length|. When it runs out of
-  // input data, returns what remains of the input. Designed to be more stable
-  // with respect to a fuzzer inserting characters than just picking a random
-  // length and then consuming that many bytes with |ConsumeBytes|.
-  std::string ConsumeRandomLengthString(size_t max_length) {
-    // Reads bytes from the start of |data_ptr_|. Maps "\\" to "\", and maps "\"
-    // followed by anything else to the end of the string. As a result of this
-    // logic, a fuzzer can insert characters into the string, and the string
-    // will be lengthened to include those new characters, resulting in a more
-    // stable fuzzer than picking the length of a string independently from
-    // picking its contents.
-    std::string result;
-
-    // Reserve the anticipated capaticity to prevent several reallocations.
-    result.reserve(std::min(max_length, remaining_bytes_));
-    for (size_t i = 0; i < max_length && remaining_bytes_ != 0; ++i) {
-      char next = ConvertUnsignedToSigned<char>(data_ptr_[0]);
-      Advance(1);
-      if (next == '\\' && remaining_bytes_ != 0) {
-        next = ConvertUnsignedToSigned<char>(data_ptr_[0]);
-        Advance(1);
-        if (next != '\\')
-          break;
-      }
-      result += next;
-    }
-
-    result.shrink_to_fit();
-    return result;
-  }
-
-  // Returns a std::vector containing all remaining bytes of the input data.
-  template <typename T> std::vector<T> ConsumeRemainingBytes() {
-    return ConsumeBytes<T>(remaining_bytes_);
-  }
-
-  // Prefer using |ConsumeRemainingBytes| unless you actually need a std::string
-  // object.
-  // Returns a std::vector containing all remaining bytes of the input data.
-  std::string ConsumeRemainingBytesAsString() {
-    return ConsumeBytesAsString(remaining_bytes_);
-  }
-
-  // Returns a number in the range [Type's min, Type's max]. The value might
-  // not be uniformly distributed in the given range. If there's no input data
-  // left, always returns |min|.
-  template <typename T> T ConsumeIntegral() {
-    return ConsumeIntegralInRange(std::numeric_limits<T>::min(),
-                                  std::numeric_limits<T>::max());
-  }
-
-  // Reads one byte and returns a bool, or false when no data remains.
-  bool ConsumeBool() { return 1 & ConsumeIntegral<uint8_t>(); }
-
-  // Returns a copy of a value selected from a fixed-size |array|.
-  template <typename T, size_t size>
-  T PickValueInArray(const T (&array)[size]) {
-    static_assert(size > 0, "The array must be non empty.");
-    return array[ConsumeIntegralInRange<size_t>(0, size - 1)];
-  }
-
-  template <typename T>
-  T PickValueInArray(std::initializer_list<const T> list) {
-    // static_assert(list.size() > 0, "The array must be non empty.");
-    return *(list.begin() + ConsumeIntegralInRange<size_t>(0, list.size() - 1));
-  }
-
-  // Return an enum value. The enum must start at 0 and be contiguous. It must
-  // also contain |kMaxValue| aliased to its largest (inclusive) value. Such as:
-  // enum class Foo { SomeValue, OtherValue, kMaxValue = OtherValue };
-  template <typename T> T ConsumeEnum() {
-    static_assert(std::is_enum<T>::value, "|T| must be an enum type.");
-    return static_cast<T>(ConsumeIntegralInRange<uint32_t>(
-        0, static_cast<uint32_t>(T::kMaxValue)));
-  }
-
-  // Reports the remaining bytes available for fuzzed input.
-  size_t remaining_bytes() { return remaining_bytes_; }
-
- private:
-  FuzzedDataProvider(const FuzzedDataProvider &) = delete;
-  FuzzedDataProvider &operator=(const FuzzedDataProvider &) = delete;
-
-  void Advance(size_t num_bytes) {
-    if (num_bytes > remaining_bytes_)
-      abort();
-
-    data_ptr_ += num_bytes;
-    remaining_bytes_ -= num_bytes;
-  }
-
-  template <typename T>
-  std::vector<T> ConsumeBytes(size_t size, size_t num_bytes_to_consume) {
-    static_assert(sizeof(T) == sizeof(uint8_t), "Incompatible data type.");
-
-    // The point of using the size-based constructor below is to increase the
-    // odds of having a vector object with capacity being equal to the length.
-    // That part is always implementation specific, but at least both libc++ and
-    // libstdc++ allocate the requested number of bytes in that constructor,
-    // which seems to be a natural choice for other implementations as well.
-    // To increase the odds even more, we also call |shrink_to_fit| below.
-    std::vector<T> result(size);
-    std::memcpy(result.data(), data_ptr_, num_bytes_to_consume);
-    Advance(num_bytes_to_consume);
-
-    // Even though |shrink_to_fit| is also implementation specific, we expect it
-    // to provide an additional assurance in case vector's constructor allocated
-    // a buffer which is larger than the actual amount of data we put inside it.
-    result.shrink_to_fit();
-    return result;
-  }
-
-  template <typename TS, typename TU> TS ConvertUnsignedToSigned(TU value) {
-    static_assert(sizeof(TS) == sizeof(TU), "Incompatible data types.");
-    static_assert(!std::numeric_limits<TU>::is_signed,
-                  "Source type must be unsigned.");
-
-    // TODO(Dor1s): change to `if constexpr` once C++17 becomes mainstream.
-    if (std::numeric_limits<TS>::is_modulo)
-      return static_cast<TS>(value);
-
-    // Avoid using implementation-defined unsigned to signer conversions.
-    // To learn more, see https://stackoverflow.com/questions/13150449.
-    if (value <= std::numeric_limits<TS>::max())
-      return static_cast<TS>(value);
-    else {
-      constexpr auto TS_min = std::numeric_limits<TS>::min();
-      return TS_min + static_cast<char>(value - TS_min);
-    }
-  }
-
-  const uint8_t *data_ptr_;
-  size_t remaining_bytes_;
-};
-
-#endif // LLVM_FUZZER_FUZZED_DATA_PROVIDER_H_

compiler-rt/trunk/lib/fuzzer/tests/CMakeLists.txt

generate_compiler_rt_tests(FuzzerTestObjects
     LINK_FLAGS ${LIBFUZZER_UNITTEST_LINK_FLAGS} ${LIBFUZZER_TEST_RUNTIME_LINK_FLAGS})
   set_target_properties(FuzzerUnitTests PROPERTIES
     RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
 
   set(FuzzedDataProviderTestObjects)
   generate_compiler_rt_tests(FuzzedDataProviderTestObjects
     FuzzedDataProviderUnitTests "FuzzerUtils-${arch}-Test" ${arch}
     SOURCES FuzzedDataProviderUnittest.cpp ${COMPILER_RT_GTEST_SOURCE}
-    DEPS gtest ${LIBFUZZER_TEST_RUNTIME_DEPS}
+    DEPS gtest ${LIBFUZZER_TEST_RUNTIME_DEPS} ${COMPILER_RT_SOURCE_DIR}/include/fuzzer/FuzzedDataProvider.h
     CFLAGS ${LIBFUZZER_UNITTEST_CFLAGS} ${LIBFUZZER_TEST_RUNTIME_CFLAGS}
     LINK_FLAGS ${LIBFUZZER_UNITTEST_LINK_FLAGS} ${LIBFUZZER_TEST_RUNTIME_LINK_FLAGS})
   set_target_properties(FuzzedDataProviderUnitTests PROPERTIES
     RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
 endif()

compiler-rt/trunk/lib/fuzzer/tests/FuzzedDataProviderUnittest.cpp

 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 #include "gtest/gtest.h"
 #include <cstdint>
 #include <cstdlib>
 
-#include <fuzzer/FuzzedDataProvider.hpp>
+#include <fuzzer/FuzzedDataProvider.h>
 
 // The test is intentionally extensive, as behavior of |FuzzedDataProvider| must
 // not be broken, given than many fuzz targets depend on it. Changing the
 // behavior might invalidate existing corpora and make the fuzz targets using
 // |FuzzedDataProvider| to lose code coverage accumulated over time.
 
 /* A random 1KB buffer generated by:
 $ python -c "import os; print ',\n'.join([', '.join(['0x%02X' % ord(i) for i \
TEST(FuzzedDataProvider, remaining_bytes) {
   EXPECT_EQ(size_t(1024 - 1 - 8), DataProv.remaining_bytes());
 
   // Exhaust the buffer.
   EXPECT_EQ(std::vector<uint8_t>(Data + 8, Data + sizeof(Data) - 1),
             DataProv.ConsumeRemainingBytes<uint8_t>());
   EXPECT_EQ(size_t(0), DataProv.remaining_bytes());
 }
 
+TEST(FuzzedDataProvider, ConsumeProbability) {
+  FuzzedDataProvider DataProv(Data, sizeof(Data));
+  ASSERT_FLOAT_EQ(float(0.28969181), DataProv.ConsumeProbability<float>());
+  ASSERT_DOUBLE_EQ(double(0.086814121166605432),
+                   DataProv.ConsumeProbability<double>());
+  ASSERT_FLOAT_EQ(float(0.30104411), DataProv.ConsumeProbability<float>());
+  ASSERT_DOUBLE_EQ(double(0.96218831486039413),
+                   DataProv.ConsumeProbability<double>());
+  ASSERT_FLOAT_EQ(float(0.67005056), DataProv.ConsumeProbability<float>());
+  ASSERT_DOUBLE_EQ(double(0.69210584173832279),
+                   DataProv.ConsumeProbability<double>());
+
+  // Exhaust the buffer.
+  EXPECT_EQ(std::vector<uint8_t>(Data, Data + sizeof(Data) - 36),
+            DataProv.ConsumeRemainingBytes<uint8_t>());
+  ASSERT_FLOAT_EQ(float(0.0), DataProv.ConsumeProbability<float>());
+}
+
+TEST(FuzzedDataProvider, ConsumeFloatingPoint) {
+  FuzzedDataProvider DataProv(Data, sizeof(Data));
+  ASSERT_FLOAT_EQ(float(-2.8546307e+38),
+                  DataProv.ConsumeFloatingPoint<float>());
+  ASSERT_DOUBLE_EQ(double(8.0940194040236032e+307),
+                   DataProv.ConsumeFloatingPoint<double>());
+  ASSERT_FLOAT_EQ(float(271.49084),
+                  DataProv.ConsumeFloatingPointInRange<float>(123.0, 777.0));
+  ASSERT_DOUBLE_EQ(double(30.859126145478349),
+                   DataProv.ConsumeFloatingPointInRange<double>(13.37, 31.337));
+  ASSERT_FLOAT_EQ(
+      float(-903.47729),
+      DataProv.ConsumeFloatingPointInRange<float>(-999.9999, -777.77));
+  ASSERT_DOUBLE_EQ(
+      double(24.561393182922771),
+      DataProv.ConsumeFloatingPointInRange<double>(-13.37, 31.337));
+  ASSERT_FLOAT_EQ(float(1.0),
+                  DataProv.ConsumeFloatingPointInRange<float>(1.0, 1.0));
+  ASSERT_DOUBLE_EQ(double(-1.0),
+                   DataProv.ConsumeFloatingPointInRange<double>(-1.0, -1.0));
+
+  // Exhaust the buffer.
+  EXPECT_EQ((std::vector<uint8_t>(Data, Data + sizeof(Data) - 50)).size(),
+            DataProv.ConsumeRemainingBytes<uint8_t>().size());
+  ASSERT_FLOAT_EQ(float(0.0), DataProv.ConsumeProbability<float>());
+  ASSERT_NEAR(std::numeric_limits<double>::lowest(),
+              DataProv.ConsumeFloatingPoint<double>(), 1e-10);
+  ASSERT_FLOAT_EQ(float(123.0),
+                  DataProv.ConsumeFloatingPointInRange<float>(123.0, 777.0));
+  ASSERT_DOUBLE_EQ(double(-13.37), DataProv.ConsumeFloatingPointInRange<double>(
+                                       -13.37, 31.337));
+}
+
 int main(int argc, char **argv) {
   testing::InitGoogleTest(&argc, argv);
   return RUN_ALL_TESTS();
 }

compiler-rt/trunk/lib/fuzzer/utils/FuzzedDataProvider.h

 //===- FuzzedDataProvider.h - Utility header for fuzz targets ---*- C++ -* ===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
-// This a temporary copy of compiler-rt/include/fuzzer/FuzzedDataProvider.hpp.
+// This a temporary copy of compiler-rt/include/fuzzer/FuzzedDataProvider.h.
 // TODO([e-mail]): delete this copy.
 // A single header library providing an utility class to break up an array of
 // bytes. Whenever run on the same input, provides the same output, as long as
 // its methods are called in the same order, with the same arguments.
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_FUZZER_FUZZED_DATA_PROVIDER_H_
 #define LLVM_FUZZER_FUZZED_DATA_PROVIDER_H_

compiler-rt/trunk/lib/sanitizer_common/scripts/check_lint.sh

 if [ "${COMPILER_RT}" = "" ]; then
   COMPILER_RT=projects/compiler-rt
 fi
 LIT_TESTS=${COMPILER_RT}/test
 # Headers
 SANITIZER_INCLUDES=${COMPILER_RT}/include/sanitizer
 FUZZER_INCLUDES=${COMPILER_RT}/include/fuzzer
 run_lint ${SANITIZER_INCLUDES_LINT_FILTER} ${SANITIZER_INCLUDES}/*.h \
-                                           ${FUZZER_INCLUDES}/*.hpp &
+                                           ${FUZZER_INCLUDES}/*.h &
 
 # Sanitizer_common
 COMMON_RTL=${COMPILER_RT}/lib/sanitizer_common
 run_lint ${COMMON_RTL_INC_LINT_FILTER} ${COMMON_RTL}/*.cpp \
                                        ${COMMON_RTL}/*.h \
                                        ${COMMON_RTL}/tests/*.cpp &
 
 # Interception