[mlir][IR] Refactor the internal implementation of Value

[rriddle]

Summary

The current implementation of Value involves a pointer int pair with several different kinds of owners, i.e. BlockArgumentImpl*, Operation *, TrailingOpResult*. This design arose from the desire to save memory overhead for operations that have a very small number of results (generally 0-2). There are, unfortunately, many problematic aspects of the current implementation that make Values difficult to work with or just inefficient.

Operation result types are stored as a separate array on the Operation. This is very inefficient for many reasons: we use TupleType for multiple results, which can lead to huge amounts of memory usage if multi-result operations change types frequently(they do). It also means that simple methods like Value::getType/Value::setType now require complex logic to get to the desired type.

Value only has one pointer bit free, severely limiting the ability to use it in things like PointerUnion/PointerIntPair. Given that we store the kind of a Value along with the "owner" pointer, we only leave one bit free for users of Value. This creates situations where we end up nesting PointerUnions to be able to use Value in one.

As noted above, most of the methods in Value need to branch on at least 3 different cases which is both inefficient, possibly error prone, and verbose. The current storage of results also creates problems for utilities like ValueRange/TypeRange, which want to efficiently store base pointers to ranges (of which Operation* isn't really useful as one).

This revision greatly simplifies the implementation of Value by the introduction of a new ValueImpl class. This class contains all of the state shared between all of the various derived value classes; i.e. the use list, the type, and the kind. This shared implementation class provides several large benefits:

• Most of the methods on value are now branchless, and often one-liners.

• The "kind" of the value is now stored in ValueImpl instead of Value

This frees up all of Value's pointer bits, allowing for users to take full advantage of PointerUnion/PointerIntPair/etc. It also allows for storing more operation results as "inline", 6 now instead of 2, freeing up 1 word per new inline result.

• Operation result types are now stored in the result, instead of a side array

This drops the size of zero-result operations by 1 word. It also removes the memory crushing use of TupleType for operations results (which could lead up to hundreds of megabytes of "dead" TupleTypes in the context). This also allowed restructured ValueRange, making it simpler and one word smaller.

This revision does come with two conceptual downsides:

• Operation::getResultTypes no longer returns an ArrayRef<Type>

This conceptually makes some usages slower, as the iterator increment is slightly more complex.

• OpResult::getOwner is slightly more expensive, as it now requires a little bit of arithmetic

From profiling, neither of the conceptual downsides have resulted in any perceivable hit to performance. Given the advantages of the new design, most compiles are slightly faster.

[jpienaar]

Nice, thanks for the super fast improvement!

mlir/include/mlir/IR/Value.h

43–44

So, when I was looking at this I was wondering how large this impact is :)

Esp as I think this would enable simpler Type array accessing if all of these were equally spaced apart, but the performance impact/overhead may more than offset that

363

s/proper//

mlir/lib/IR/OperationSupport.cpp

482

Useful clang-tidy warning :)

mlir/lib/IR/Value.cpp

118

const_cast? (in addition)

118

This is a little bit magical for me, could you add a comment? (bonus point for ascii diagram ;-))

[rriddle]

address comments

mlir/include/mlir/IR/Value.h

43–44

When I originally made the change there was a decent memory reduction for operations, given that a very high distribution of operations have a very small number of results. To reconfirm, I just added a small instrumentation to a TF model to see what the distribution is there (I didn't do it for upstream because I know for a fact a majority of those are single result) and came up with:

Operation with 0 results : 1137
Operation with 1 results : 33587
Operation with 2 results : 156
Operation with 3 results : 36
Operation with 4 results : 352
Operation with 5 results : 121
Operation with 6 results : 320
Operation with 7 results : 217
Operation with 16 results : 1
Operation with 84 results : 5
Operation with 171 results : 2
Operation with 309 results : 1
Operation with 310 results : 3
Operation with 394 results : 2
Operation with 395 results : 3
Operation with 396 results : 3

Every result up to 6 costs 1 less word, which in this example ended up being something like ~300kb. On bigger models, it gets much more pronounced. The logic could be simpler by just keeping everything the same size, but the size savings + negligible execution time change makes it compelling for me at least.

[jpienaar]

Thanks!

[lattner]

This is a really nice design!

[nathanchance]

$ cmake -G Ninja -DCMAKE_CXX_COMPILER=$(command -v clang++) -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;debuginfo-tests;mlir" ../llvm
...

$ ninja projects/debuginfo-tests/CMakeFiles/check-gdb-mlir-support.dir/llvm-prettyprinters/gdb/mlir-support.cpp.o
...
/home/nathan/src/llvm-project/debuginfo-tests/llvm-prettyprinters/gdb/mlir-support.cpp:13:33: error: no member named 'Kind' in 'mlir::Value'
                   mlir::Value::Kind::TrailingOpResult});
                   ~~~~~~~~~~~~~^
1 error generated.

D98613 is needed to avoid other errors in this file.

[thopre]

In D97804#2625161, @nathanchance wrote:

$ cmake -G Ninja -DCMAKE_CXX_COMPILER=$(command -v clang++) -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;debuginfo-tests;mlir" ../llvm
...

$ ninja projects/debuginfo-tests/CMakeFiles/check-gdb-mlir-support.dir/llvm-prettyprinters/gdb/mlir-support.cpp.o
...
/home/nathan/src/llvm-project/debuginfo-tests/llvm-prettyprinters/gdb/mlir-support.cpp:13:33: error: no member named 'Kind' in 'mlir::Value'
                   mlir::Value::Kind::TrailingOpResult});
                   ~~~~~~~~~~~~~^
1 error generated.

D98613 is needed to avoid other errors in this file.

I'm still getting this error with top of main. Shouldn't Value be changed to a mlir::OutOfLineOpResult ? There's no more TrailingOpResult in mlir::ValueImpl.

[aboualiaa]

This comment was removed by aboualiaa.

[apoos-maximus]

In D97804#2625161, @nathanchance wrote:

$ cmake -G Ninja -DCMAKE_CXX_COMPILER=$(command -v clang++) -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;debuginfo-tests;mlir" ../llvm
...

$ ninja projects/debuginfo-tests/CMakeFiles/check-gdb-mlir-support.dir/llvm-prettyprinters/gdb/mlir-support.cpp.o
...
/home/nathan/src/llvm-project/debuginfo-tests/llvm-prettyprinters/gdb/mlir-support.cpp:13:33: error: no member named 'Kind' in 'mlir::Value'
                   mlir::Value::Kind::TrailingOpResult});
                   ~~~~~~~~~~~~~^
1 error generated.

D98613 is needed to avoid other errors in this file.

I encountered this error too while going through a 'all module build'

mlir/include/mlir/Conversion/StandardToLLVM/ConvertStandardToLLVM.h

public:
   /// is populated with argument mapping.
   Type convertFunctionSignature(FunctionType funcTy, bool isVariadic,
                                 SignatureConversion &result);
 
   /// Convert a non-empty list of types to be returned from a function into a
   /// supported LLVM IR type.  In particular, if more than one value is
   /// returned, create an LLVM IR structure type with elements that correspond
   /// to each of the MLIR types converted with `convertType`.
-  Type packFunctionResults(ArrayRef<Type> types);
+  Type packFunctionResults(TypeRange types);
 
   /// Convert a type in the context of the default or bare pointer calling
   /// convention. Calling convention sensitive types, such as MemRefType and
   /// UnrankedMemRefType, are converted following the specific rules for the
   /// calling convention. Calling convention independent types are converted
   /// following the default LLVM type conversions.
   Type convertCallingConventionType(Type type);
 

mlir/include/mlir/IR/Operation.h

public:
   operand_type_iterator operand_type_end() { return operand_end(); }
   operand_type_range getOperandTypes() { return getOperands().getTypes(); }
 
   //===--------------------------------------------------------------------===//
   // Results
   //===--------------------------------------------------------------------===//
 
   /// Return the number of results held by this operation.
-  unsigned getNumResults();
+  unsigned getNumResults() { return numResults; }
 
   /// Get the 'idx'th result of this operation.
-  OpResult getResult(unsigned idx) { return OpResult(this, idx); }
+  OpResult getResult(unsigned idx) { return OpResult(getOpResultImpl(idx)); }
 
   /// Support result iteration.
   using result_range = ResultRange;
   using result_iterator = result_range::iterator;
 
   result_iterator result_begin() { return getResults().begin(); }
   result_iterator result_end() { return getResults().end(); }
-  result_range getResults() { return result_range(this); }
+  result_range getResults() {
+    return numResults == 0 ? result_range(nullptr, 0)
+                           : result_range(getInlineOpResult(0), numResults);
+  }
 
   result_range getOpResults() { return getResults(); }
   OpResult getOpResult(unsigned idx) { return getResult(idx); }
 
   /// Support result type iteration.
   using result_type_iterator = result_range::type_iterator;
   using result_type_range = result_range::type_range;
   result_type_iterator result_type_begin() { return getResultTypes().begin(); }
   result_type_iterator result_type_end() { return getResultTypes().end(); }
-  result_type_range getResultTypes();
+  result_type_range getResultTypes() { return getResults().getTypes(); }
 
   //===--------------------------------------------------------------------===//
   // Attributes
   //===--------------------------------------------------------------------===//
 
   // Operations may optionally carry a list of attributes that associate
   // constants to names.  Attributes may be dynamically added and removed over
   // the lifetime of an operation.
private:
   /// Update the order index of this operation of this operation if necessary,
   /// potentially recomputing the order of the parent block.
   void updateOrderIfNecessary();
 
   /// Returns true if this operation has a valid order.
   bool hasValidOrder() { return orderIndex != kInvalidOrderIdx; }
 
 private:
-  Operation(Location location, OperationName name, TypeRange resultTypes,
+  Operation(Location location, OperationName name, unsigned numResults,
             unsigned numSuccessors, unsigned numRegions,
             DictionaryAttr attributes, bool hasOperandStorage);
 
   // Operations are deleted through the destroy() member because they are
   // allocated with malloc.
   ~Operation();
 
   /// Returns the additional size necessary for allocating the given objects
   /// before an Operation in-memory.
-  static size_t prefixAllocSize(unsigned numTrailingResults,
+  static size_t prefixAllocSize(unsigned numOutOfLineResults,
                                 unsigned numInlineResults) {
-    return sizeof(detail::TrailingOpResult) * numTrailingResults +
-           sizeof(detail::InLineOpResult) * numInlineResults;
+    return sizeof(detail::OutOfLineOpResult) * numOutOfLineResults +
+           sizeof(detail::InlineOpResult) * numInlineResults;
   }
   /// Returns the additional size allocated before this Operation in-memory.
   size_t prefixAllocSize() {
     unsigned numResults = getNumResults();
-    unsigned numTrailingResults = OpResult::getNumTrailing(numResults);
+    unsigned numOutOfLineResults = OpResult::getNumTrailing(numResults);
     unsigned numInlineResults = OpResult::getNumInline(numResults);
-    return prefixAllocSize(numTrailingResults, numInlineResults);
+    return prefixAllocSize(numOutOfLineResults, numInlineResults);
   }
 
   /// Returns the operand storage object.
   detail::OperandStorage &getOperandStorage() {
     assert(hasOperandStorage && "expected operation to have operand storage");
     return *getTrailingObjects<detail::OperandStorage>();
   }
 
-  /// Returns a pointer to the use list for the given trailing result.
-  detail::TrailingOpResult *getTrailingResult(unsigned resultNumber) {
-    // Trailing results are stored in reverse order after(before in memory) the
-    // inline results.
-    return reinterpret_cast<detail::TrailingOpResult *>(
-               getInlineResult(OpResult::getMaxInlineResults() - 1)) -
+  /// Returns a pointer to the use list for the given out-of-line result.
+  detail::OutOfLineOpResult *getOutOfLineOpResult(unsigned resultNumber) {
+    // Out-of-line results are stored in reverse order after (before in memory)
+    // the inline results.
+    return reinterpret_cast<detail::OutOfLineOpResult *>(getInlineOpResult(
+               detail::OpResultImpl::getMaxInlineResults() - 1)) -
            ++resultNumber;
   }
 
   /// Returns a pointer to the use list for the given inline result.
-  detail::InLineOpResult *getInlineResult(unsigned resultNumber) {
+  detail::InlineOpResult *getInlineOpResult(unsigned resultNumber) {
     // Inline results are stored in reverse order before the operation in
     // memory.
-    return reinterpret_cast<detail::InLineOpResult *>(this) - ++resultNumber;
+    return reinterpret_cast<detail::InlineOpResult *>(this) - ++resultNumber;
+  }
+
+  /// Returns a pointer to the use list for the given result, which may be
+  /// either inline or out-of-line.
+  detail::OpResultImpl *getOpResultImpl(unsigned resultNumber) {
+    unsigned maxInlineResults = detail::OpResultImpl::getMaxInlineResults();
+    if (resultNumber < maxInlineResults)
+      return getInlineOpResult(resultNumber);
+    return getOutOfLineOpResult(resultNumber - maxInlineResults);
   }
 
   /// Provide a 'getParent' method for ilist_node_with_parent methods.
   /// We mark it as a const function because ilist_node_with_parent specifically
   /// requires a 'getParent() const' method. Once ilist_node removes this
   /// constraint, we should drop the const to fit the rest of the MLIR const
   /// model.
   Block *getParent() const { return block; }
 
   /// The operation block that contains this operation.
   Block *block = nullptr;
 
   /// This holds information about the source location the operation was defined
   /// or derived from.
   Location location;
 
   /// Relative order of this operation in its parent block. Used for
   /// O(1) local dominance checks between operations.
   mutable unsigned orderIndex = 0;
 
+  const unsigned numResults;
   const unsigned numSuccs;
-  const unsigned numRegions : 30;
+  const unsigned numRegions : 31;
 
   /// This bit signals whether this operation has an operand storage or not. The
   /// operand storage may be elided for operations that are known to never have
   /// operands.
   bool hasOperandStorage : 1;
 
-  /// This holds the result types of the operation. There are three different
-  /// states recorded here:
-  /// - 0 results : The type below is null.
-  /// - 1 result  : The single result type is held here.
-  /// - N results : The type here is a tuple holding the result types.
-  /// Note: We steal a bit for 'hasSingleResult' from somewhere else so that we
-  /// can use 'resultType` in an ArrayRef<Type>.
-  bool hasSingleResult : 1;
-  Type resultType;
-
   /// This holds the name of the operation.
   OperationName name;
 
   /// This holds general named attributes for the operation.
   DictionaryAttr attrs;
 
   // allow ilist_traits access to 'block' field.
   friend struct llvm::ilist_traits<Operation>;

mlir/include/mlir/IR/OperationSupport.h

private:
   };
 
   /// This stuff is used by the TrailingObjects template.
   friend llvm::TrailingObjects<OperandStorage, OpOperand>;
 };
 } // end namespace detail
 
 //===----------------------------------------------------------------------===//
-// ResultStorage
-//===----------------------------------------------------------------------===//
-
-namespace detail {
-/// This class provides the implementation for an in-line operation result. This
-/// is an operation result whose number can be stored inline inside of the bits
-/// of an Operation*.
-struct alignas(8) InLineOpResult : public IRObjectWithUseList<OpOperand> {};
-/// This class provides the implementation for an out-of-line operation result.
-/// This is an operation result whose number cannot be stored inline inside of
-/// the bits of an Operation*.
-struct alignas(8) TrailingOpResult : public IRObjectWithUseList<OpOperand> {
-  TrailingOpResult(uint64_t trailingResultNumber)
-      : trailingResultNumber(trailingResultNumber) {}
-
-  /// Returns the parent operation of this trailing result.
-  Operation *getOwner();
-
-  /// Return the proper result number of this op result.
-  unsigned getResultNumber() {
-    return trailingResultNumber + OpResult::getMaxInlineResults();
-  }
-
-  /// The trailing result number, or the offset from the beginning of the
-  /// trailing array.
-  uint64_t trailingResultNumber;
-};
-} // end namespace detail
-
-//===----------------------------------------------------------------------===//
 // OpPrintingFlags
 //===----------------------------------------------------------------------===//
 
 /// Set of flags used to control the behavior of the various IR print methods
 /// (e.g. Operation::Print).
 class OpPrintingFlags {
 public:
   OpPrintingFlags();
private:
   SmallVector<std::pair<unsigned, NamedAttribute>, 1> operandSegments;
 };
 
 //===----------------------------------------------------------------------===//
 // ResultRange
 
 /// This class implements the result iterators for the Operation class.
 class ResultRange final
-    : public llvm::indexed_accessor_range<ResultRange, Operation *, OpResult,
-                                          OpResult, OpResult> {
+    : public llvm::detail::indexed_accessor_range_base<
+          ResultRange, detail::OpResultImpl *, OpResult, OpResult, OpResult> {
 public:
-  using indexed_accessor_range<ResultRange, Operation *, OpResult, OpResult,
-                               OpResult>::indexed_accessor_range;
-  ResultRange(Operation *op);
+  using RangeBaseT::RangeBaseT;
 
   /// Returns the types of the values within this range.
-  using type_iterator = ArrayRef<Type>::iterator;
-  using type_range = ArrayRef<Type>;
-  type_range getTypes() const;
+  using type_iterator = ValueTypeIterator<iterator>;
+  using type_range = ValueTypeRange<ResultRange>;
+  type_range getTypes() const { return {begin(), end()}; }
   auto getType() const { return getTypes(); }
 
 private:
-  /// See `llvm::indexed_accessor_range` for details.
-  static OpResult dereference(Operation *op, ptrdiff_t index);
+  /// See `llvm::detail::indexed_accessor_range_base` for details.
+  static detail::OpResultImpl *offset_base(detail::OpResultImpl *object,
+                                           ptrdiff_t index) {
+    return object->getNextResultAtOffset(index);
+  }
+  /// See `llvm::detail::indexed_accessor_range_base` for details.
+  static OpResult dereference_iterator(detail::OpResultImpl *object,
+                                       ptrdiff_t index) {
+    return offset_base(object, index);
+  }
 
-  /// Allow access to `dereference_iterator`.
-  friend llvm::indexed_accessor_range<ResultRange, Operation *, OpResult,
-                                      OpResult, OpResult>;
+  /// Allow access to `offset_base` and `dereference_iterator`.
+  friend RangeBaseT;
 };
 
 //===----------------------------------------------------------------------===//
 // ValueRange
 
-namespace detail {
-/// The type representing the owner of a ValueRange. This is either a list of
-/// values, operands, or an Operation+start index for results.
-struct ValueRangeOwner {
-  ValueRangeOwner(const Value *owner) : ptr(owner), startIndex(0) {}
-  ValueRangeOwner(OpOperand *owner) : ptr(owner), startIndex(0) {}
-  ValueRangeOwner(Operation *owner, unsigned startIndex)
-      : ptr(owner), startIndex(startIndex) {}
-  bool operator==(const ValueRangeOwner &rhs) const { return ptr == rhs.ptr; }
-
-  /// The owner pointer of the range. The owner has represents three distinct
-  /// states:
-  /// const Value *: The owner is the base to a contiguous array of Value.
-  /// OpOperand *  : The owner is the base to a contiguous array of operands.
-  /// void*        : This owner is an Operation*. It is marked as void* here
-  ///                because the definition of Operation is not visible here.
-  PointerUnion<const Value *, OpOperand *, void *> ptr;
-
-  /// Ths start index into the range. This is only used for Operation* owners.
-  unsigned startIndex;
-};
-} // end namespace detail
-
 /// This class provides an abstraction over the different types of ranges over
 /// Values. In many cases, this prevents the need to explicitly materialize a
 /// SmallVector/std::vector. This class should be used in places that are not
 /// suitable for a more derived type (e.g. ArrayRef) or a template range
 /// parameter.
 class ValueRange final
     : public llvm::detail::indexed_accessor_range_base<
-          ValueRange, detail::ValueRangeOwner, Value, Value, Value> {
+          ValueRange,
+          PointerUnion<const Value *, OpOperand *, detail::OpResultImpl *>,
+          Value, Value, Value> {
 public:
+  /// The type representing the owner of a ValueRange. This is either a list of
+  /// values, operands, or results.
+  using OwnerT =
+      PointerUnion<const Value *, OpOperand *, detail::OpResultImpl *>;
+
   using RangeBaseT::RangeBaseT;
 
   template <typename Arg,
             typename = typename std::enable_if_t<
                 std::is_constructible<ArrayRef<Value>, Arg>::value &&
                 !std::is_convertible<Arg, Value>::value>>
   ValueRange(Arg &&arg) : ValueRange(ArrayRef<Value>(std::forward<Arg>(arg))) {}
   ValueRange(const Value &value) : ValueRange(&value, /*count=*/1) {}
public:
 
   /// Returns the types of the values within this range.
   using type_iterator = ValueTypeIterator<iterator>;
   using type_range = ValueTypeRange<ValueRange>;
   type_range getTypes() const { return {begin(), end()}; }
   auto getType() const { return getTypes(); }
 
 private:
-  using OwnerT = detail::ValueRangeOwner;
-
   /// See `llvm::detail::indexed_accessor_range_base` for details.
   static OwnerT offset_base(const OwnerT &owner, ptrdiff_t index);
   /// See `llvm::detail::indexed_accessor_range_base` for details.
   static Value dereference_iterator(const OwnerT &owner, ptrdiff_t index);
 
   /// Allow access to `offset_base` and `dereference_iterator`.
   friend RangeBaseT;
 };

mlir/include/mlir/IR/TypeRange.h

 //===----------------------------------------------------------------------===//
 // TypeRange
 
 /// This class provides an abstraction over the various different ranges of
 /// value types. In many cases, this prevents the need to explicitly materialize
 /// a SmallVector/std::vector. This class should be used in places that are not
 /// suitable for a more derived type (e.g. ArrayRef) or a template range
 /// parameter.
-class TypeRange
-    : public llvm::detail::indexed_accessor_range_base<
+class TypeRange : public llvm::detail::indexed_accessor_range_base<
                       TypeRange,
-          llvm::PointerUnion<const Value *, const Type *, OpOperand *>, Type,
-          Type, Type> {
+                      llvm::PointerUnion<const Value *, const Type *,
+                                         OpOperand *, detail::OpResultImpl *>,
+                      Type, Type, Type> {
 public:
   using RangeBaseT::RangeBaseT;
   TypeRange(ArrayRef<Type> types = llvm::None);
   explicit TypeRange(OperandRange values);
   explicit TypeRange(ResultRange values);
   explicit TypeRange(ValueRange values);
   explicit TypeRange(ArrayRef<Value> values);
   explicit TypeRange(ArrayRef<BlockArgument> values)
public:
   TypeRange(std::initializer_list<Type> types)
       : TypeRange(ArrayRef<Type>(types)) {}
 
 private:
   /// The owner of the range is either:
   /// * A pointer to the first element of an array of values.
   /// * A pointer to the first element of an array of types.
   /// * A pointer to the first element of an array of operands.
-  using OwnerT = llvm::PointerUnion<const Value *, const Type *, OpOperand *>;
+  /// * A pointer to the first element of an array of results.
+  using OwnerT = llvm::PointerUnion<const Value *, const Type *, OpOperand *,
+                                    detail::OpResultImpl *>;
 
   /// See `llvm::detail::indexed_accessor_range_base` for details.
   static OwnerT offset_base(OwnerT object, ptrdiff_t index);
   /// See `llvm::detail::indexed_accessor_range_base` for details.
   static Type dereference_iterator(OwnerT object, ptrdiff_t index);
 
   /// Allow access to `offset_base` and `dereference_iterator`.
   friend RangeBaseT;
class ValueTypeRange final
     : public llvm::iterator_range<
           ValueTypeIterator<typename ValueRangeT::iterator>> {
 public:
   using llvm::iterator_range<
       ValueTypeIterator<typename ValueRangeT::iterator>>::iterator_range;
   template <typename Container>
   ValueTypeRange(Container &&c) : ValueTypeRange(c.begin(), c.end()) {}
 
+  /// Return the type at the given index.
+  Type operator[](size_t index) const {
+    assert(index < size() && "invalid index into type range");
+    return *(this->begin() + index);
+  }
+
+  /// Return the size of this range.
+  size_t size() const { return llvm::size(*this); }
+
+  /// Return first type in the range.
+  Type front() { return (*this)[0]; }
+
   /// Compare this range with another.
   template <typename OtherT>
   bool operator==(const OtherT &other) const {
     return llvm::size(*this) == llvm::size(other) &&
            std::equal(this->begin(), this->end(), other.begin());
   }
   template <typename OtherT>
   bool operator!=(const OtherT &other) const {

mlir/include/mlir/IR/TypeUtilities.h

 /// they are both scalars (not shaped), or they are both shaped types and at
 /// least one is unranked or they have compatible dimensions. Dimensions are
 /// compatible if at least one is dynamic or both are equal. The element type
 /// does not matter.
 LogicalResult verifyCompatibleShape(Type type1, Type type2);
 
 /// Returns success if the given two arrays have the same number of elements and
 /// each pair wise entries have compatible shape.
-LogicalResult verifyCompatibleShapes(ArrayRef<Type> types1,
-                                     ArrayRef<Type> types2);
+LogicalResult verifyCompatibleShapes(TypeRange types1, TypeRange types2);
 
 //===----------------------------------------------------------------------===//
 // Utility Iterators
 //===----------------------------------------------------------------------===//
 
 // An iterator for the element types of an op's operands of shaped types.
 class OperandElementTypeIterator final
     : public llvm::mapped_iterator<Operation::operand_iterator,

mlir/include/mlir/IR/Value.h

 namespace mlir {
 class AsmState;
 class BlockArgument;
 class Operation;
 class OpResult;
 class Region;
 class Value;
 
+//===----------------------------------------------------------------------===//
+// Value
+//===----------------------------------------------------------------------===//
+
 namespace detail {
-/// The internal implementation of a BlockArgument.
-class BlockArgumentImpl;
-} // end namespace detail
+
+/// The base class for all derived Value classes. It contains all of the
+/// components that are shared across Value classes.
+class alignas(8) ValueImpl : public IRObjectWithUseList<OpOperand> {
+public:
+  /// The enumeration represents the various different kinds of values the
+  /// internal representation may take. We use all of the bits from Type that we
+  /// can to store indices inline.
+  enum class Kind {
+    /// The first N kinds are all inline operation results. An inline operation
+    /// result means that the kind represents the result number. This removes
+    /// the need to store an additional index value. The derived class here is
+    /// an `OpResultImpl`.
+    InlineOpResult = 0,
+
+    /// The next kind represents a 'out-of-line' operation result. This is for
+    /// results with numbers larger than we can represent inline. The derived
+    /// class here is an `OpResultImpl`.
+    OutOfLineOpResult = 6,
+
+    /// The last kind represents a block argument. The derived class here is an
+    /// `BlockArgumentImpl`.
+    BlockArgument = 7
+  };
+
+  /// Return the type of this value.
+  Type getType() const { return typeAndKind.getPointer(); }
+
+  /// Set the type of this value.
+  void setType(Type type) { return typeAndKind.setPointer(type); }
+
+  /// Return the kind of this value.
+  Kind getKind() const { return typeAndKind.getInt(); }
+
+protected:
+  ValueImpl(Type type, Kind kind) : typeAndKind(type, kind) {}
+
+  /// The type of this result and the kind.
+  llvm::PointerIntPair<Type, 3, Kind> typeAndKind;
+};
+} // namespace detail
 
 /// This class represents an instance of an SSA value in the MLIR system,
 /// representing a computable value that has a type and a set of users. An SSA
 /// value is either a BlockArgument or the result of an operation. Note: This
 /// class has value-type semantics and is just a simple wrapper around a
 /// ValueImpl that is either owner by a block(in the case of a BlockArgument) or
 /// an Operation(in the case of an OpResult).
 class Value {
 public:
-  /// The enumeration represents the various different kinds of values the
-  /// internal representation may take. We steal 2 bits to support a total of 4
-  /// possible values.
-  enum class Kind {
-    /// The first N kinds are all inline operation results. An inline operation
-    /// result means that the kind represents the result number, and the owner
-    /// pointer is the owning `Operation*`. Note: These are packed first to make
-    /// result number lookups more efficient.
-    OpResult0 = 0,
-    OpResult1 = 1,
-
-    /// The next kind represents a 'trailing' operation result. This is for
-    /// results with numbers larger than we can represent inline. The owner here
-    /// is an `TrailingOpResult*` that points to a trailing storage on the
-    /// parent operation.
-    TrailingOpResult = 2,
-
-    /// The last kind represents a block argument. The owner here is a
-    /// `BlockArgumentImpl*`.
-    BlockArgument = 3
-  };
-
-  /// This value represents the 'owner' of the value and its kind. See the
-  /// 'Kind' enumeration above for a more detailed description of each kind of
-  /// owner.
-  struct ImplTypeTraits : public llvm::PointerLikeTypeTraits<void *> {
-    // We know that all pointers within the ImplType are aligned by 8-bytes,
-    // meaning that we can steal up to 3 bits for the different values.
-    static constexpr int NumLowBitsAvailable = 3;
-  };
-  using ImplType = llvm::PointerIntPair<void *, 2, Kind, ImplTypeTraits>;
-
-public:
-  constexpr Value(std::nullptr_t) : ownerAndKind() {}
-  Value(ImplType ownerAndKind = {}) : ownerAndKind(ownerAndKind) {}
+  Value(detail::ValueImpl *impl = nullptr) : impl(impl) {}
   Value(const Value &) = default;
   Value &operator=(const Value &) = default;
 
   template <typename U> bool isa() const {
     assert(*this && "isa<> used on a null type.");
     return U::classof(*this);
   }
 
   template <typename First, typename Second, typename... Rest>
   bool isa() const {
     return isa<First>() || isa<Second, Rest...>();
   }
-
   template <typename U> U dyn_cast() const {
-    return isa<U>() ? U(ownerAndKind) : U(nullptr);
+    return isa<U>() ? U(impl) : U(nullptr);
   }
   template <typename U> U dyn_cast_or_null() const {
-    return (*this && isa<U>()) ? U(ownerAndKind) : U(nullptr);
+    return (*this && isa<U>()) ? U(impl) : U(nullptr);
   }
   template <typename U> U cast() const {
     assert(isa<U>());
-    return U(ownerAndKind);
+    return U(impl);
   }
 
-  explicit operator bool() const { return ownerAndKind.getPointer(); }
-  bool operator==(const Value &other) const {
-    return ownerAndKind == other.ownerAndKind;
-  }
+  explicit operator bool() const { return impl; }
+  bool operator==(const Value &other) const { return impl == other.impl; }
   bool operator!=(const Value &other) const { return !(*this == other); }
 
   /// Return the type of this value.
-  Type getType() const;
+  Type getType() const { return impl->getType(); }
 
   /// Utility to get the associated MLIRContext that this value is defined in.
   MLIRContext *getContext() const { return getType().getContext(); }
 
   /// Mutate the type of this Value to be of the specified type.
   ///
   /// Note that this is an extremely dangerous operation which can create
   /// completely invalid IR very easily.  It is strongly recommended that you
   /// recreate IR objects with the right types instead of mutating them in
   /// place.
-  void setType(Type newType);
+  void setType(Type newType) { impl->setType(newType); }
 
   /// If this value is the result of an operation, return the operation that
   /// defines it.
   Operation *getDefiningOp() const;
 
   /// If this value is the result of an operation of type OpTy, return the
   /// operation that defines it.
   template <typename OpTy>
public:
   /// Return the Block in which this Value is defined.
   Block *getParentBlock();
 
   //===--------------------------------------------------------------------===//
   // UseLists
   //===--------------------------------------------------------------------===//
 
   /// Provide the use list that is attached to this value.
-  IRObjectWithUseList<OpOperand> *getUseList() const;
+  IRObjectWithUseList<OpOperand> *getUseList() const { return impl; }
 
   /// Drop all uses of this object from their respective owners.
-  void dropAllUses() const;
+  void dropAllUses() const { return getUseList()->dropAllUses(); }
 
   /// Replace all uses of 'this' value with the new value, updating anything in
   /// the IR that uses 'this' to use the other value instead.  When this returns
   /// there are zero uses of 'this'.
   void replaceAllUsesWith(Value newValue) const;
 
   /// Replace all uses of 'this' value with 'newValue', updating anything in the
   /// IR that uses 'this' to use the other value instead except if the user is
public:
 
   //===--------------------------------------------------------------------===//
   // Uses
 
   /// This class implements an iterator over the uses of a value.
   using use_iterator = ValueUseIterator<OpOperand>;
   using use_range = iterator_range<use_iterator>;
 
-  use_iterator use_begin() const;
+  use_iterator use_begin() const { return getUseList()->use_begin(); }
   use_iterator use_end() const { return use_iterator(); }
 
   /// Returns a range of all uses, which is useful for iterating over all uses.
   use_range getUses() const { return {use_begin(), use_end()}; }
 
   /// Returns true if this value has exactly one use.
-  bool hasOneUse() const;
+  bool hasOneUse() const { return getUseList()->hasOneUse(); }
 
   /// Returns true if this value has no uses.
-  bool use_empty() const;
+  bool use_empty() const { return getUseList()->use_empty(); }
 
   //===--------------------------------------------------------------------===//
   // Users
 
   using user_iterator = ValueUserIterator<use_iterator, OpOperand>;
   using user_range = iterator_range<user_iterator>;
 
   user_iterator user_begin() const { return use_begin(); }
   user_iterator user_end() const { return use_end(); }
   user_range getUsers() const { return {user_begin(), user_end()}; }
 
   //===--------------------------------------------------------------------===//
   // Utilities
 
-  /// Returns the kind of this value.
-  Kind getKind() const { return ownerAndKind.getInt(); }
-
   void print(raw_ostream &os);
   void print(raw_ostream &os, AsmState &state);
   void dump();
 
   /// Print this value as if it were an operand.
   void printAsOperand(raw_ostream &os, AsmState &state);
 
   /// Methods for supporting PointerLikeTypeTraits.
-  void *getAsOpaquePointer() const { return ownerAndKind.getOpaqueValue(); }
+  void *getAsOpaquePointer() const { return impl; }
   static Value getFromOpaquePointer(const void *pointer) {
-    Value value;
-    value.ownerAndKind.setFromOpaqueValue(const_cast<void *>(pointer));
-    return value;
+    return reinterpret_cast<detail::ValueImpl *>(const_cast<void *>(pointer));
   }
+  detail::ValueImpl *getImpl() const { return impl; }
 
   friend ::llvm::hash_code hash_value(Value arg);
 
 protected:
-  /// Returns true if the given operation result can be packed inline.
-  static bool canPackResultInline(unsigned resultNo) {
-    return resultNo < static_cast<unsigned>(Kind::TrailingOpResult);
-  }
-
-  /// Construct a value.
-  Value(detail::BlockArgumentImpl *impl);
-  Value(Operation *op, unsigned resultNo);
-
-  /// This value represents the 'owner' of the value and its kind. See the
-  /// 'Kind' enumeration above for a more detailed description of each kind of
-  /// owner.
-  ImplType ownerAndKind;
+  /// A pointer to the internal implementation of the value.
+  detail::ValueImpl *impl;
 };
 
 inline raw_ostream &operator<<(raw_ostream &os, Value value) {
   value.print(os);
   return os;
 }
 
 //===----------------------------------------------------------------------===//
 // BlockArgument
 //===----------------------------------------------------------------------===//
 
 namespace detail {
 /// The internal implementation of a BlockArgument.
-class BlockArgumentImpl : public IRObjectWithUseList<OpOperand> {
-  BlockArgumentImpl(Type type, Block *owner, int64_t index)
-      : type(type), owner(owner), index(index) {}
+class BlockArgumentImpl : public ValueImpl {
+public:
+  static bool classof(const ValueImpl *value) {
+    return value->getKind() == ValueImpl::Kind::BlockArgument;
+  }
 
-  /// The type of this argument.
-  Type type;
+private:
+  BlockArgumentImpl(Type type, Block *owner, int64_t index)
+      : ValueImpl(type, Kind::BlockArgument), owner(owner), index(index) {}
 
   /// The owner of this argument.
   Block *owner;
 
   /// The position in the argument list.
   int64_t index;
 
   /// Allow access to owner and constructor.
   friend BlockArgument;
 };
 } // end namespace detail
 
-/// Block arguments are values.
+/// This class represents an argument of a Block.
 class BlockArgument : public Value {
 public:
   using Value::Value;
 
   static bool classof(Value value) {
-    return value.getKind() == Kind::BlockArgument;
+    return llvm::isa<detail::BlockArgumentImpl>(value.getImpl());
   }
 
   /// Returns the block that owns this argument.
   Block *getOwner() const { return getImpl()->owner; }
 
-  /// Return the type of this value.
-  Type getType() const { return getImpl()->type; }
-
-  /// Set the type of this value.
-  void setType(Type newType) { getImpl()->type = newType; }
-
   /// Returns the number of this argument.
   unsigned getArgNumber() const { return getImpl()->index; }
 
 private:
   /// Allocate a new argument with the given type and owner.
   static BlockArgument create(Type type, Block *owner, int64_t index) {
     return new detail::BlockArgumentImpl(type, owner, index);
   }
 
   /// Destroy and deallocate this argument.
   void destroy() { delete getImpl(); }
 
   /// Get a raw pointer to the internal implementation.
   detail::BlockArgumentImpl *getImpl() const {
-    return reinterpret_cast<detail::BlockArgumentImpl *>(
-        ownerAndKind.getPointer());
+    return reinterpret_cast<detail::BlockArgumentImpl *>(impl);
   }
 
   /// Cache the position in the block argument list.
   void setArgNumber(int64_t index) { getImpl()->index = index; }
 
   /// Allow access to `create`, `destroy` and `setArgNumber`.
   friend Block;
 
   /// Allow access to 'getImpl'.
   friend Value;
 };
 
 //===----------------------------------------------------------------------===//
 // OpResult
 //===----------------------------------------------------------------------===//
 
+namespace detail {
+/// This class provides the implementation for an operation result.
+class alignas(8) OpResultImpl : public ValueImpl {
+public:
+  using ValueImpl::ValueImpl;
+
+  static bool classof(const ValueImpl *value) {
+    return value->getKind() != ValueImpl::Kind::BlockArgument;
+  }
+
+  /// Returns the parent operation of this result.
+  Operation *getOwner() const;
+
+  /// Returns the result number of this op result.
+  unsigned getResultNumber() const;
+
+  /// Returns the next operation result at `offset` after this result. This
+  /// method is useful when indexing the result storage of an operation, given
+  /// that there is more than one kind of operation result (with the different
+  /// kinds having different sizes) and that operations are stored in reverse
+  /// order.
+  OpResultImpl *getNextResultAtOffset(intptr_t offset);
+
+  /// Returns the maximum number of results that can be stored inline.
+  static unsigned getMaxInlineResults() {
+    return static_cast<unsigned>(Kind::OutOfLineOpResult);
+  }
+};
+
+/// This class provides the implementation for an operation result whose index
+/// can be represented "inline" in the underlying ValueImpl.
+struct InlineOpResult : public OpResultImpl {
+public:
+  InlineOpResult(Type type, unsigned resultNo)
+      : OpResultImpl(type, static_cast<ValueImpl::Kind>(resultNo)) {
+    assert(resultNo < getMaxInlineResults());
+  }
+
+  /// Return the result number of this op result.
+  unsigned getResultNumber() const { return static_cast<unsigned>(getKind()); }
+
+  static bool classof(const OpResultImpl *value) {
+    return value->getKind() != ValueImpl::Kind::OutOfLineOpResult;
+  }
+};
+
+/// This class provides the implementation for an operation result whose index
+/// cannot be represented "inline", and thus requires an additional index field.
+class OutOfLineOpResult : public OpResultImpl {
+public:
+  OutOfLineOpResult(Type type, uint64_t outOfLineIndex)
+      : OpResultImpl(type, Kind::OutOfLineOpResult),
+        outOfLineIndex(outOfLineIndex) {}
+
+  static bool classof(const OpResultImpl *value) {
+    return value->getKind() == ValueImpl::Kind::OutOfLineOpResult;
+  }
+
+  /// Return the result number of this op result.
+  unsigned getResultNumber() const {
+    return outOfLineIndex + getMaxInlineResults();
+  }
+
+  /// The trailing result number, or the offset from the beginning of the
+  /// `OutOfLineOpResult` array.
+  uint64_t outOfLineIndex;
+};
+
+/// Return the result number of this op result.
+inline unsigned OpResultImpl::getResultNumber() const {
+  if (const auto *outOfLineResult = dyn_cast<OutOfLineOpResult>(this))
+    return outOfLineResult->getResultNumber();
+  return cast<InlineOpResult>(this)->getResultNumber();
+}
+
+} // end namespace detail
+
 /// This is a value defined by a result of an operation.
 class OpResult : public Value {
 public:
   using Value::Value;
 
   static bool classof(Value value) {
-    return value.getKind() != Kind::BlockArgument;
+    return llvm::isa<detail::OpResultImpl>(value.getImpl());
   }
 
   /// Returns the operation that owns this result.
-  Operation *getOwner() const;
+  Operation *getOwner() const { return getImpl()->getOwner(); }
 
   /// Returns the number of this result.
-  unsigned getResultNumber() const;
+  unsigned getResultNumber() const { return getImpl()->getResultNumber(); }
 
-  /// Returns the maximum number of results that can be stored inline.
-  static unsigned getMaxInlineResults() {
-    return static_cast<unsigned>(Kind::TrailingOpResult);
+private:
+  /// Get a raw pointer to the internal implementation.
+  detail::OpResultImpl *getImpl() const {
+    return reinterpret_cast<detail::OpResultImpl *>(impl);
   }
 
-private:
   /// Given a number of operation results, returns the number that need to be
   /// stored inline.
   static unsigned getNumInline(unsigned numResults);
 
   /// Given a number of operation results, returns the number that need to be
   /// stored as trailing.
   static unsigned getNumTrailing(unsigned numResults);
 
   /// Allow access to constructor.
   friend Operation;
 };
 
 /// Make Value hashable.
 inline ::llvm::hash_code hash_value(Value arg) {
-  return ::llvm::hash_value(arg.ownerAndKind.getOpaqueValue());
+  return ::llvm::hash_value(arg.getImpl());
 }
 
 } // namespace mlir
 
 namespace llvm {
 
 template <> struct DenseMapInfo<mlir::Value> {
   static mlir::Value getEmptyKey() {
-    auto pointer = llvm::DenseMapInfo<void *>::getEmptyKey();
+    void *pointer = llvm::DenseMapInfo<void *>::getEmptyKey();
     return mlir::Value::getFromOpaquePointer(pointer);
   }
   static mlir::Value getTombstoneKey() {
-    auto pointer = llvm::DenseMapInfo<void *>::getTombstoneKey();
+    void *pointer = llvm::DenseMapInfo<void *>::getTombstoneKey();
     return mlir::Value::getFromOpaquePointer(pointer);
   }
   static unsigned getHashValue(mlir::Value val) {
     return mlir::hash_value(val);
   }
   static bool isEqual(mlir::Value lhs, mlir::Value rhs) { return lhs == rhs; }
 };
-
-/// Allow stealing the low bits of a value.
-template <> struct PointerLikeTypeTraits<mlir::Value> {
-public:
-  static inline void *getAsVoidPointer(mlir::Value I) {
-    return const_cast<void *>(I.getAsOpaquePointer());
-  }
-  static inline mlir::Value getFromVoidPointer(void *P) {
-    return mlir::Value::getFromOpaquePointer(P);
-  }
-  enum {
-    NumLowBitsAvailable =
-        PointerLikeTypeTraits<mlir::Value::ImplType>::NumLowBitsAvailable
-  };
-};
-
-template <> struct DenseMapInfo<mlir::BlockArgument> {
+template <>
+struct DenseMapInfo<mlir::BlockArgument> : public DenseMapInfo<mlir::Value> {
   static mlir::BlockArgument getEmptyKey() {
-    auto pointer = llvm::DenseMapInfo<void *>::getEmptyKey();
-    return mlir::BlockArgument(
-        mlir::Value::ImplType::getFromOpaqueValue(pointer));
+    void *pointer = llvm::DenseMapInfo<void *>::getEmptyKey();
+    return reinterpret_cast<mlir::detail::BlockArgumentImpl *>(pointer);
   }
   static mlir::BlockArgument getTombstoneKey() {
-    auto pointer = llvm::DenseMapInfo<void *>::getTombstoneKey();
-    return mlir::BlockArgument(
-        mlir::Value::ImplType::getFromOpaqueValue(pointer));
-  }
-  static unsigned getHashValue(mlir::BlockArgument val) {
-    return mlir::hash_value(val);
-  }
-  static bool isEqual(mlir::BlockArgument LHS, mlir::BlockArgument RHS) {
-    return LHS == RHS;
+    void *pointer = llvm::DenseMapInfo<void *>::getTombstoneKey();
+    return reinterpret_cast<mlir::detail::BlockArgumentImpl *>(pointer);
   }
 };
 
 /// Allow stealing the low bits of a value.
-template <> struct PointerLikeTypeTraits<mlir::BlockArgument> {
+template <> struct PointerLikeTypeTraits<mlir::Value> {
 public:
-  static inline void *getAsVoidPointer(mlir::Value I) {
-    return const_cast<void *>(I.getAsOpaquePointer());
+  static inline void *getAsVoidPointer(mlir::Value value) {
+    return const_cast<void *>(value.getAsOpaquePointer());
   }
-  static inline mlir::BlockArgument getFromVoidPointer(void *P) {
-    return mlir::Value::getFromOpaquePointer(P).cast<mlir::BlockArgument>();
+  static inline mlir::Value getFromVoidPointer(void *pointer) {
+    return mlir::Value::getFromOpaquePointer(pointer);
   }
   enum {
     NumLowBitsAvailable =
-        PointerLikeTypeTraits<mlir::Value>::NumLowBitsAvailable
-  };
+        PointerLikeTypeTraits<mlir::detail::ValueImpl *>::NumLowBitsAvailable
+  };
+template <>
+struct PointerLikeTypeTraits<mlir::BlockArgument>
+    : public PointerLikeTypeTraits<mlir::Value> {
+public:
+  static inline mlir::BlockArgument getFromVoidPointer(void *pointer) {
+    return reinterpret_cast<mlir::detail::BlockArgumentImpl *>(pointer);
+  }
 };
+
 } // end namespace llvm
 
 #endif

mlir/include/mlir/Interfaces/InferTypeOpInterface.td

StaticInterfaceMethod<
                     "::mlir::RegionRange":$regions,
                     "::llvm::SmallVectorImpl<::mlir::Type>&":$inferredReturnTypes)
     >,
     StaticInterfaceMethod<
       /*desc=*/"Returns whether two array of types are compatible result types"
                " for an op.",
       /*retTy=*/"bool",
       /*methodName=*/"isCompatibleReturnTypes",
-      /*args=*/(ins "::llvm::ArrayRef<::mlir::Type>":$lhs,
-                    "::llvm::ArrayRef<::mlir::Type>":$rhs),
+      /*args=*/(ins "::mlir::TypeRange":$lhs, "::mlir::TypeRange":$rhs),
       /*methodBody=*/[{
         return ConcreteOp::isCompatibleReturnTypes(lhs, rhs);
       }],
       /*defaultImplementation=*/[{
         /// Returns whether two arrays are equal as strongest check for
         /// compatibility by default.
         return lhs == rhs;
       }]

mlir/include/mlir/Transforms/DialectConversion.h

public:
 
   /// This hook simplifies defining 1-1 type conversions. This function returns
   /// the type to convert to on success, and a null type on failure.
   Type convertType(Type t);
 
   /// Convert the given set of types, filling 'results' as necessary. This
   /// returns failure if the conversion of any of the types fails, success
   /// otherwise.
-  LogicalResult convertTypes(ArrayRef<Type> types,
-                             SmallVectorImpl<Type> &results);
+  LogicalResult convertTypes(TypeRange types, SmallVectorImpl<Type> &results);
 
   /// Return true if the given type is legal for this type converter, i.e. the
   /// type converts to itself.
   bool isLegal(Type type);
   /// Return true if all of the given types are legal for this type converter.
   template <typename RangeT>
   std::enable_if_t<!std::is_convertible<RangeT, Type>::value &&
                        !std::is_convertible<RangeT, Operation *>::value,

mlir/lib/Conversion/StandardToLLVM/StandardToLLVM.cpp

void mlir::populateStdToLLVMConversionPatterns(
   populateStdToLLVMNonMemoryConversionPatterns(converter, patterns);
   populateStdToLLVMMemoryConversionPatterns(converter, patterns);
 }
 
 /// Convert a non-empty list of types to be returned from a function into a
 /// supported LLVM IR type.  In particular, if more than one value is returned,
 /// create an LLVM IR structure type with elements that correspond to each of
 /// the MLIR types converted with `convertType`.
-Type LLVMTypeConverter::packFunctionResults(ArrayRef<Type> types) {
+Type LLVMTypeConverter::packFunctionResults(TypeRange types) {
   assert(!types.empty() && "expected non-empty list of type");
 
   if (types.size() == 1)
     return convertCallingConventionType(types.front());
 
   SmallVector<Type, 8> resultTypes;
   resultTypes.reserve(types.size());
   for (auto t : types) {

mlir/lib/Dialect/Async/IR/Async.cpp

if (!op.operands().empty()) {
     llvm::interleaveComma(op.operands(), p, [&, n = 0](Value operand) mutable {
       p << operand << " as " << op.body().front().getArgument(n++) << ": "
         << operand.getType();
     });
     p << ")";
   }
 
   // -> (!async.value<!return.type>, ...)
-  p.printOptionalArrowTypeList(op.getResultTypes().drop_front(1));
+  p.printOptionalArrowTypeList(llvm::drop_begin(op.getResultTypes()));
   p.printOptionalAttrDictWithKeyword(op->getAttrs(),
                                      {kOperandSegmentSizesAttr});
   p.printRegion(op.body(), /*printEntryBlockArgs=*/false);
 }
 
 static ParseResult parseExecuteOp(OpAsmParser &parser, OperationState &result) {
   MLIRContext *ctx = result.getContext();
 

mlir/lib/IR/Operation.cpp

 }
 
 /// Overload of create that takes an existing DictionaryAttr to avoid
 /// unnecessarily uniquing a list of attributes.
 Operation *Operation::create(Location location, OperationName name,
                              TypeRange resultTypes, ValueRange operands,
                              DictionaryAttr attributes, BlockRange successors,
                              unsigned numRegions) {
+  assert(llvm::all_of(resultTypes, [](Type t) { return t; }) &&
+         "unexpected null result type");
+
   // We only need to allocate additional memory for a subset of results.
   unsigned numTrailingResults = OpResult::getNumTrailing(resultTypes.size());
   unsigned numInlineResults = OpResult::getNumInline(resultTypes.size());
   unsigned numSuccessors = successors.size();
   unsigned numOperands = operands.size();
+  unsigned numResults = resultTypes.size();
 
   // If the operation is known to have no operands, don't allocate an operand
   // storage.
   bool needsOperandStorage = true;
   if (operands.empty()) {
     if (const AbstractOperation *abstractOp = name.getAbstractOperation())
       needsOperandStorage = !abstractOp->hasTrait<OpTrait::ZeroOperands>();
   }
 
   // Compute the byte size for the operation and the operand storage. This takes
   // into account the size of the operation, its trailing objects, and its
   // prefixed objects.
   size_t byteSize =
       totalSizeToAlloc<BlockOperand, Region, detail::OperandStorage>(
           numSuccessors, numRegions, needsOperandStorage ? 1 : 0) +
       detail::OperandStorage::additionalAllocSize(numOperands);
   size_t prefixByteSize = llvm::alignTo(
       Operation::prefixAllocSize(numTrailingResults, numInlineResults),
       alignof(Operation));
   char *mallocMem = reinterpret_cast<char *>(malloc(byteSize + prefixByteSize));
   void *rawMem = mallocMem + prefixByteSize;
 
   // Create the new Operation.
   Operation *op =
-      ::new (rawMem) Operation(location, name, resultTypes, numSuccessors,
+      ::new (rawMem) Operation(location, name, numResults, numSuccessors,
                                numRegions, attributes, needsOperandStorage);
 
   assert((numSuccessors == 0 || op->mightHaveTrait<OpTrait::IsTerminator>()) &&
          "unexpected successors in a non-terminator operation");
 
   // Initialize the results.
-  for (unsigned i = 0; i < numInlineResults; ++i)
-    new (op->getInlineResult(i)) detail::InLineOpResult();
-  for (unsigned i = 0; i < numTrailingResults; ++i)
-    new (op->getTrailingResult(i)) detail::TrailingOpResult(i);
+  auto resultTypeIt = resultTypes.begin();
+  for (unsigned i = 0; i < numInlineResults; ++i, ++resultTypeIt)
+    new (op->getInlineOpResult(i)) detail::InlineOpResult(*resultTypeIt, i);
+  for (unsigned i = 0; i < numTrailingResults; ++i, ++resultTypeIt) {
+    new (op->getOutOfLineOpResult(i))
+        detail::OutOfLineOpResult(*resultTypeIt, i);
+  }
 
   // Initialize the regions.
   for (unsigned i = 0; i != numRegions; ++i)
     new (&op->getRegion(i)) Region(op);
 
   // Initialize the operands.
   if (needsOperandStorage)
     new (&op->getOperandStorage()) detail::OperandStorage(op, operands);
 
   // Initialize the successors.
   auto blockOperands = op->getBlockOperands();
   for (unsigned i = 0; i != numSuccessors; ++i)
     new (&blockOperands[i]) BlockOperand(op, successors[i]);
 
   return op;
 }
 
-Operation::Operation(Location location, OperationName name,
-                     TypeRange resultTypes, unsigned numSuccessors,
-                     unsigned numRegions, DictionaryAttr attributes,
-                     bool hasOperandStorage)
-    : location(location), numSuccs(numSuccessors), numRegions(numRegions),
-      hasOperandStorage(hasOperandStorage), hasSingleResult(false), name(name),
+Operation::Operation(Location location, OperationName name, unsigned numResults,
+                     unsigned numSuccessors, unsigned numRegions,
+                     DictionaryAttr attributes, bool hasOperandStorage)
+    : location(location), numResults(numResults), numSuccs(numSuccessors),
+      numRegions(numRegions), hasOperandStorage(hasOperandStorage), name(name),
       attrs(attributes) {
   assert(attributes && "unexpected null attribute dictionary");
-  assert(llvm::all_of(resultTypes, [](Type t) { return t; }) &&
-         "unexpected null result type");
-  if (!resultTypes.empty()) {
-    // If there is a single result it is stored in-place, otherwise use a tuple.
-    hasSingleResult = resultTypes.size() == 1;
-    if (hasSingleResult)
-      resultType = resultTypes.front();
-    else
-      resultType = TupleType::get(location->getContext(), resultTypes);
-  }
 }
 
 // Operations are deleted through the destroy() member because they are
 // allocated via malloc.
 Operation::~Operation() {
   assert(block == nullptr && "operation destroyed but still in a block");
 
   // Explicitly run the destructors for the operands.

 void Operation::dropAllDefinedValueUses() {
   dropAllUses();
 
   for (auto &region : getRegions())
     for (auto &block : region)
       block.dropAllDefinedValueUses();
 }
 
-/// Return the number of results held by this operation.
-unsigned Operation::getNumResults() {
-  if (!resultType)
-    return 0;
-  return hasSingleResult ? 1 : resultType.cast<TupleType>().size();
-}
-
-auto Operation::getResultTypes() -> result_type_range {
-  if (!resultType)
-    return llvm::None;
-  if (hasSingleResult)
-    return resultType;
-  return resultType.cast<TupleType>().getTypes();
-}
-
 void Operation::setSuccessor(Block *block, unsigned index) {
   assert(index < getNumSuccessors());
   getBlockOperands()[index].set(block);
 }
 
 /// Attempt to fold this operation using the Op's registered foldHook.
 LogicalResult Operation::fold(ArrayRef<Attribute> operands,
                               SmallVectorImpl<OpFoldResult> &results) {

 
 LogicalResult OpTrait::impl::verifySameOperandsAndResultType(Operation *op) {
   if (failed(verifyAtLeastNOperands(op, 1)) ||
       failed(verifyAtLeastNResults(op, 1)))
     return failure();
 
   auto type = op->getResult(0).getType();
   auto elementType = getElementTypeOrSelf(type);
-  for (auto resultType : op->getResultTypes().drop_front(1)) {
+  for (auto resultType : llvm::drop_begin(op->getResultTypes())) {
     if (getElementTypeOrSelf(resultType) != elementType ||
         failed(verifyCompatibleShape(resultType, type)))
       return op->emitOpError()
              << "requires the same type for all operands and results";
   }
   for (auto opType : op->getOperandTypes()) {
     if (getElementTypeOrSelf(opType) != elementType ||
         failed(verifyCompatibleShape(opType, type)))

mlir/lib/IR/OperationSupport.cpp

MutableArrayRef<OpOperand> detail::OperandStorage::resize(Operation *owner,
 
   // Update the storage representation to use the new dynamic storage.
   representation = reinterpret_cast<intptr_t>(newStorage);
   representation |= DynamicStorageBit;
   return newOperands;
 }
 
 //===----------------------------------------------------------------------===//
-// ResultStorage
-//===----------------------------------------------------------------------===//
-
-/// Returns the parent operation of this trailing result.
-Operation *detail::TrailingOpResult::getOwner() {
-  // We need to do some arithmetic to get the operation pointer. Trailing
-  // results are stored in reverse order before the inline results of the
-  // operation, so move the trailing owner up to the start of the array.
-  TrailingOpResult *trailingIt = this + (trailingResultNumber + 1);
-
-  // Move the owner past the inline op results to get to the operation.
-  auto *inlineResultIt = reinterpret_cast<InLineOpResult *>(trailingIt) +
-                         OpResult::getMaxInlineResults();
-  return reinterpret_cast<Operation *>(inlineResultIt);
-}
-
-//===----------------------------------------------------------------------===//
 // Operation Value-Iterators
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//
 // OperandRange
 
 OperandRange::OperandRange(Operation *op)
     : OperandRange(op->getOpOperands().data(), op->getNumOperands()) {}
for (OperandSegment &segment : operandSegments) {
     SmallVector<int32_t, 8> segments(attr.getValues<int32_t>());
     segments[segment.first] += diff;
     segment.second.second = DenseIntElementsAttr::get(attr.getType(), segments);
     owner->setAttr(segment.second.first, segment.second.second);
   }
 }
 
 //===----------------------------------------------------------------------===//
-// ResultRange
-
-ResultRange::ResultRange(Operation *op)
-    : ResultRange(op, /*startIndex=*/0, op->getNumResults()) {}
-
-ArrayRef<Type> ResultRange::getTypes() const {
-  return getBase()->getResultTypes().slice(getStartIndex(), size());
-}
-
-/// See `llvm::indexed_accessor_range` for details.
-OpResult ResultRange::dereference(Operation *op, ptrdiff_t index) {
-  return op->getResult(index);
-}
-
-//===----------------------------------------------------------------------===//
 // ValueRange
 
 ValueRange::ValueRange(ArrayRef<Value> values)
     : ValueRange(values.data(), values.size()) {}
 ValueRange::ValueRange(OperandRange values)
     : ValueRange(values.begin().getBase(), values.size()) {}
 ValueRange::ValueRange(ResultRange values)
-    : ValueRange(
-          {values.getBase(), static_cast<unsigned>(values.getStartIndex())},
-          values.size()) {}
+    : ValueRange(values.getBase(), values.size()) {}
 
 /// See `llvm::detail::indexed_accessor_range_base` for details.
 ValueRange::OwnerT ValueRange::offset_base(const OwnerT &owner,
                                            ptrdiff_t index) {
-  if (auto *value = owner.ptr.dyn_cast<const Value *>())
+  if (const auto *value = owner.dyn_cast<const Value *>())
     return {value + index};
-  if (auto *operand = owner.ptr.dyn_cast<OpOperand *>())
+  if (auto *operand = owner.dyn_cast<OpOperand *>())
     return {operand + index};
-  Operation *operation = reinterpret_cast<Operation *>(owner.ptr.get<void *>());
-  return {operation, owner.startIndex + static_cast<unsigned>(index)};
+  return owner.get<detail::OpResultImpl *>()->getNextResultAtOffset(index);
 }
 /// See `llvm::detail::indexed_accessor_range_base` for details.
 Value ValueRange::dereference_iterator(const OwnerT &owner, ptrdiff_t index) {
-  if (auto *value = owner.ptr.dyn_cast<const Value *>())
+  if (const auto *value = owner.dyn_cast<const Value *>())
     return value[index];
-  if (auto *operand = owner.ptr.dyn_cast<OpOperand *>())
+  if (auto *operand = owner.dyn_cast<OpOperand *>())
     return operand[index].get();
-  Operation *operation = reinterpret_cast<Operation *>(owner.ptr.get<void *>());
-  return operation->getResult(owner.startIndex + index);
+  return owner.get<detail::OpResultImpl *>()->getNextResultAtOffset(index);
 }
 
 //===----------------------------------------------------------------------===//
 // Operation Equivalency
 //===----------------------------------------------------------------------===//
 
 llvm::hash_code OperationEquivalence::computeHash(Operation *op, Flags flags) {
   // Hash operations based upon their:
   //   - Operation Name
   //   - Attributes
-  llvm::hash_code hash =
-      llvm::hash_combine(op->getName(), op->getAttrDictionary());
-
   //   - Result Types
-  ArrayRef<Type> resultTypes = op->getResultTypes();
-  switch (resultTypes.size()) {
-  case 0:
-    // We don't need to add anything to the hash.
-    break;
-  case 1:
-    // Add in the result type.
-    hash = llvm::hash_combine(hash, resultTypes.front());
-    break;
-  default:
-    // Use the type buffer as the hash, as we can guarantee it is the same for
-    // any given range of result types. This takes advantage of the fact the
-    // result types >1 are stored in a TupleType and uniqued.
-    hash = llvm::hash_combine(hash, resultTypes.data());
-    break;
-  }
+  llvm::hash_code hash = llvm::hash_combine(
+      op->getName(), op->getAttrDictionary(), op->getResultTypes());
 
   //   - Operands
   bool ignoreOperands = flags & Flags::IgnoreOperands;
   if (!ignoreOperands) {
     // TODO: Allow commutative operations to have different ordering.
     hash = llvm::hash_combine(
         hash, llvm::hash_combine_range(op->operand_begin(), op->operand_end()));
   }
if (lhs->getName() != rhs->getName())
     return false;
   // Check operand counts.
   if (lhs->getNumOperands() != rhs->getNumOperands())
     return false;
   // Compare attributes.
   if (lhs->getAttrDictionary() != rhs->getAttrDictionary())
     return false;
   // Compare result types.
-  ArrayRef<Type> lhsResultTypes = lhs->getResultTypes();
-  ArrayRef<Type> rhsResultTypes = rhs->getResultTypes();
-  if (lhsResultTypes.size() != rhsResultTypes.size())
-    return false;
-  switch (lhsResultTypes.size()) {
-  case 0:
-    break;
-  case 1:
-    // Compare the single result type.
-    if (lhsResultTypes.front() != rhsResultTypes.front())
-      return false;
-    break;
-  default:
-    // Use the type buffer for the comparison, as we can guarantee it is the
-    // same for any given range of result types. This takes advantage of the
-    // fact the result types >1 are stored in a TupleType and uniqued.
-    if (lhsResultTypes.data() != rhsResultTypes.data())
-      return false;
-    break;
-  }
+  if (lhs->getResultTypes() != rhs->getResultTypes())
   // Compare operands.
   bool ignoreOperands = flags & Flags::IgnoreOperands;
   if (ignoreOperands)
     return true;
   // TODO: Allow commutative operations to have different ordering.
   return std::equal(lhs->operand_begin(), lhs->operand_end(),
                     rhs->operand_begin());
 }

mlir/lib/IR/TypeRange.cpp

 //===- TypeRange.cpp ------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/IR/TypeRange.h"
 #include "mlir/IR/Operation.h"
+
 using namespace mlir;
 
 //===----------------------------------------------------------------------===//
 // TypeRange
 
 TypeRange::TypeRange(ArrayRef<Type> types)
     : TypeRange(types.data(), types.size()) {
   assert(llvm::all_of(types, [](Type t) { return t; }) &&
          "attempting to construct a TypeRange with null types");
 }
 TypeRange::TypeRange(OperandRange values)
     : TypeRange(values.begin().getBase(), values.size()) {}
 TypeRange::TypeRange(ResultRange values)
-    : TypeRange(values.getBase()->getResultTypes().slice(values.getStartIndex(),
-                                                         values.size())) {}
+    : TypeRange(values.getBase(), values.size()) {}
 TypeRange::TypeRange(ArrayRef<Value> values)
     : TypeRange(values.data(), values.size()) {}
 TypeRange::TypeRange(ValueRange values) : TypeRange(OwnerT(), values.size()) {
-  detail::ValueRangeOwner owner = values.begin().getBase();
-  if (auto *op = reinterpret_cast<Operation *>(owner.ptr.dyn_cast<void *>()))
-    this->base = op->getResultTypes().drop_front(owner.startIndex).data();
-  else if (auto *operand = owner.ptr.dyn_cast<OpOperand *>())
+  if (count == 0)
+    return;
+  ValueRange::OwnerT owner = values.begin().getBase();
+  if (auto *result = owner.dyn_cast<detail::OpResultImpl *>())
+    this->base = result;
+  else if (auto *operand = owner.dyn_cast<OpOperand *>())
     this->base = operand;
   else
-    this->base = owner.ptr.get<const Value *>();
+    this->base = owner.get<const Value *>();
 }
 
 /// See `llvm::detail::indexed_accessor_range_base` for details.
 TypeRange::OwnerT TypeRange::offset_base(OwnerT object, ptrdiff_t index) {
   if (const auto *value = object.dyn_cast<const Value *>())
     return {value + index};
   if (auto *operand = object.dyn_cast<OpOperand *>())
     return {operand + index};
+  if (auto *result = object.dyn_cast<detail::OpResultImpl *>())
+    return {result->getNextResultAtOffset(index)};
   return {object.dyn_cast<const Type *>() + index};
 }
+
 /// See `llvm::detail::indexed_accessor_range_base` for details.
 Type TypeRange::dereference_iterator(OwnerT object, ptrdiff_t index) {
   if (const auto *value = object.dyn_cast<const Value *>())
     return (value + index)->getType();
   if (auto *operand = object.dyn_cast<OpOperand *>())
     return (operand + index)->get().getType();
+  if (auto *result = object.dyn_cast<detail::OpResultImpl *>())
+    return result->getNextResultAtOffset(index)->getType();
   return object.dyn_cast<const Type *>()[index];
 }

mlir/lib/IR/TypeUtilities.cpp

LogicalResult mlir::verifyCompatibleShape(Type type1, Type type2) {
   if (!sType1.hasRank() || !sType2.hasRank())
     return success();
 
   return verifyCompatibleShape(sType1.getShape(), sType2.getShape());
 }
 
 /// Returns success if the given two arrays have the same number of elements and
 /// each pair wise entries have compatible shape.
-LogicalResult mlir::verifyCompatibleShapes(ArrayRef<Type> types1,
-                                           ArrayRef<Type> types2) {
+LogicalResult mlir::verifyCompatibleShapes(TypeRange types1, TypeRange types2) {
   if (types1.size() != types2.size())
     return failure();
-  for (auto it : zip_first(types1, types2))
+  for (auto it : llvm::zip_first(types1, types2))
     if (failed(verifyCompatibleShape(std::get<0>(it), std::get<1>(it))))
       return failure();
   return success();
 }
 
 OperandElementTypeIterator::OperandElementTypeIterator(
     Operation::operand_iterator it)
     : llvm::mapped_iterator<Operation::operand_iterator, Type (*)(Value)>(

mlir/lib/IR/Value.cpp

 #include "mlir/IR/Block.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Operation.h"
 #include "llvm/ADT/SmallPtrSet.h"
 
 using namespace mlir;
 using namespace mlir::detail;
 
-/// Construct a value.
-Value::Value(BlockArgumentImpl *impl)
-    : ownerAndKind(impl, Kind::BlockArgument) {}
-Value::Value(Operation *op, unsigned resultNo) {
-  assert(op->getNumResults() > resultNo && "invalid result number");
-  if (LLVM_LIKELY(canPackResultInline(resultNo))) {
-    ownerAndKind = {op, static_cast<Kind>(resultNo)};
-    return;
-  }
-
-  // If we can't pack the result directly, grab the use list from the parent op.
-  unsigned trailingNo = resultNo - OpResult::getMaxInlineResults();
-  ownerAndKind = {op->getTrailingResult(trailingNo), Kind::TrailingOpResult};
-}
-
-/// Return the type of this value.
-Type Value::getType() const {
-  if (BlockArgument arg = dyn_cast<BlockArgument>())
-    return arg.getType();
-
-  // If this is an operation result, query the parent operation.
-  OpResult result = cast<OpResult>();
-  Operation *owner = result.getOwner();
-  if (owner->hasSingleResult)
-    return owner->resultType;
-  return owner->resultType.cast<TupleType>().getType(result.getResultNumber());
-}
-
-/// Mutate the type of this Value to be of the specified type.
-void Value::setType(Type newType) {
-  if (BlockArgument arg = dyn_cast<BlockArgument>())
-    return arg.setType(newType);
-  OpResult result = cast<OpResult>();
-
-  // If the owner has a single result, simply update it directly.
-  Operation *owner = result.getOwner();
-  if (owner->hasSingleResult) {
-    owner->resultType = newType;
-    return;
-  }
-  unsigned resultNo = result.getResultNumber();
-
-  // Otherwise, rebuild the tuple if the new type is different from the current.
-  auto curTypes = owner->resultType.cast<TupleType>().getTypes();
-  if (curTypes[resultNo] == newType)
-    return;
-  auto newTypes = llvm::to_vector<4>(curTypes);
-  newTypes[resultNo] = newType;
-  owner->resultType = TupleType::get(newType.getContext(), newTypes);
-}
-
 /// If this value is the result of an Operation, return the operation that
 /// defines it.
 Operation *Value::getDefiningOp() const {
   if (auto result = dyn_cast<OpResult>())
     return result.getOwner();
   return nullptr;
 }
 
if (Operation *op = getDefiningOp())
     return op->getBlock();
   return cast<BlockArgument>().getOwner();
 }
 
 //===----------------------------------------------------------------------===//
 // Value::UseLists
 //===----------------------------------------------------------------------===//
 
-/// Provide the use list that is attached to this value.
-IRObjectWithUseList<OpOperand> *Value::getUseList() const {
-  if (BlockArgument arg = dyn_cast<BlockArgument>())
-    return arg.getImpl();
-  if (getKind() != Kind::TrailingOpResult) {
-    OpResult result = cast<OpResult>();
-    return result.getOwner()->getInlineResult(result.getResultNumber());
-  }
-
-  // Otherwise this is a trailing operation result, which contains a use list.
-  return reinterpret_cast<TrailingOpResult *>(ownerAndKind.getPointer());
-}
-
-/// Drop all uses of this object from their respective owners.
-void Value::dropAllUses() const { return getUseList()->dropAllUses(); }
-
 /// Replace all uses of 'this' value with the new value, updating anything in
 /// the IR that uses 'this' to use the other value instead.  When this returns
 /// there are zero uses of 'this'.
 void Value::replaceAllUsesWith(Value newValue) const {
   return getUseList()->replaceAllUsesWith(newValue);
 }
 
 /// Replace all uses of 'this' value with the new value, updating anything in

 
 /// Returns true if the value is used outside of the given block.
 bool Value::isUsedOutsideOfBlock(Block *block) {
   return llvm::any_of(getUsers(), [block](Operation *user) {
     return user->getBlock() != block;
   });
 }
 
-//===--------------------------------------------------------------------===//
-// Uses
-
-auto Value::use_begin() const -> use_iterator {
-  return getUseList()->use_begin();
-}
-
-/// Returns true if this value has exactly one use.
-bool Value::hasOneUse() const { return getUseList()->hasOneUse(); }
-
-/// Returns true if this value has no uses.
-bool Value::use_empty() const { return getUseList()->use_empty(); }
-
 //===----------------------------------------------------------------------===//
 // OpResult
 //===----------------------------------------------------------------------===//
 
-/// Returns the operation that owns this result.
-Operation *OpResult::getOwner() const {
-  // If the result is in-place, the `owner` is the operation.
-  void *owner = ownerAndKind.getPointer();
-  if (LLVM_LIKELY(getKind() != Kind::TrailingOpResult))
-    return static_cast<Operation *>(owner);
-
-  // Otherwise, query the trailing result for the owner.
-  return static_cast<TrailingOpResult *>(owner)->getOwner();
-}
-
-/// Return the result number of this result.
-unsigned OpResult::getResultNumber() const {
-  // If the result is in-place, we can use the kind directly.
-  if (LLVM_LIKELY(getKind() != Kind::TrailingOpResult))
-    return static_cast<unsigned>(ownerAndKind.getInt());
-  // Otherwise, query the trailing result.
-  auto *result = static_cast<TrailingOpResult *>(ownerAndKind.getPointer());
-  return result->getResultNumber();
+/// Returns the parent operation of this trailing result.
+Operation *OpResultImpl::getOwner() const {
+  // We need to do some arithmetic to get the operation pointer. Results are
+  // stored in reverse order before the operation, so move the trailing owner up
+  // to the start of the array. A rough diagram of the memory layout is:
+  //
+  // | Out-of-Line results | Inline results | Operation |
+  //
+  // Given that the results are reverse order we use the result number to know
+  // how far to jump to get to the operation. So if we are currently the 0th
+  // result, the layout would be:
+  //
+  // | Inline result 0 | Operation
+  //
+  // ^-- To get the base address of the operation, we add the result count + 1.
+  if (const auto *result = dyn_cast<InlineOpResult>(this)) {
+    result += result->getResultNumber() + 1;
+    return reinterpret_cast<Operation *>(const_cast<InlineOpResult *>(result));
+  }
+
+  // Out-of-line results are stored in an array just before the inline results.
+  const OutOfLineOpResult *outOfLineIt = (const OutOfLineOpResult *)(this);
+  outOfLineIt += (outOfLineIt->outOfLineIndex + 1);
+
+  // Move the owner past the inline results to get to the operation.
+  const auto *inlineIt = reinterpret_cast<const InlineOpResult *>(outOfLineIt);
+  inlineIt += getMaxInlineResults();
+  return reinterpret_cast<Operation *>(const_cast<InlineOpResult *>(inlineIt));
+}
+
+OpResultImpl *OpResultImpl::getNextResultAtOffset(intptr_t offset) {
+  if (offset == 0)
+    return this;
+  // We need to do some arithmetic to get the next result given that results are
+  // in reverse order, and that we need to account for the different types of
+  // results. As a reminder, the rough diagram of the memory layout is:
+  //
+  // | Out-of-Line results | Inline results | Operation |
+  //
+  // So an example operation with two results would look something like:
+  //
+  // | Inline result 1 | Inline result 0 | Operation |
+  //
+
+  // Handle the case where this result is an inline result.
+  OpResultImpl *result = this;
+  if (auto *inlineResult = dyn_cast<InlineOpResult>(this)) {
+    // Check to see how many results there are after this one before the start
+    // of the out-of-line results. If the desired offset is less than the number
+    // remaining, we can directly use the offset from the current result
+    // pointer. The following diagrams highlight the two situations.
+    //
+    // | Out-of-Line results | Inline results | Operation |
+    //                                    ^- Say we are here.
+    //                           ^- If our destination is here, we can use the
+    //                              offset directly.
+    //
+    intptr_t leftBeforeTrailing =
+        getMaxInlineResults() - inlineResult->getResultNumber() - 1;
+    if (leftBeforeTrailing >= offset)
+      return inlineResult - offset;
+
+    // Otherwise, adjust the current result pointer to the end (start in memory)
+    // of the inline result array.
+    //
+    // | Out-of-Line results | Inline results | Operation |
+    //                                    ^- Say we are here.
+    //                  ^- If our destination is here, we need to first jump to
+    //                     the end (start in memory) of the inline result array.
+    //
+    result = inlineResult - leftBeforeTrailing;
+    offset -= leftBeforeTrailing;
+  }
+
+  // If we land here, the current result is an out-of-line result and we can
+  // offset directly.
+  return reinterpret_cast<OutOfLineOpResult *>(result) - offset;
 }
 
 /// Given a number of operation results, returns the number that need to be
 /// stored inline.
 unsigned OpResult::getNumInline(unsigned numResults) {
-  return std::min(numResults, getMaxInlineResults());
+  return std::min(numResults, OpResultImpl::getMaxInlineResults());
 }
 
 /// Given a number of operation results, returns the number that need to be
 /// stored as trailing.
 unsigned OpResult::getNumTrailing(unsigned numResults) {
   // If we can pack all of the results, there is no need for additional storage.
-  unsigned maxInline = getMaxInlineResults();
+  unsigned maxInline = OpResultImpl::getMaxInlineResults();
   return numResults <= maxInline ? 0 : numResults - maxInline;
 }
 
 //===----------------------------------------------------------------------===//
 // BlockOperand
 //===----------------------------------------------------------------------===//
 
 /// Provide the use list that is attached to the given block.

mlir/lib/Transforms/Utils/DialectConversion.cpp

Type TypeConverter::convertType(Type t) {
 
   // Check to ensure that only one type was produced.
   return results.size() == 1 ? results.front() : nullptr;
 }
 
 /// Convert the given set of types, filling 'results' as necessary. This
 /// returns failure if the conversion of any of the types fails, success
 /// otherwise.
-LogicalResult TypeConverter::convertTypes(ArrayRef<Type> types,
+LogicalResult TypeConverter::convertTypes(TypeRange types,
                                           SmallVectorImpl<Type> &results) {
-  for (auto type : types)
+  for (Type type : types)
     if (failed(convertType(type, results)))
       return failure();
   return success();
 }
 
 /// Return true if the given type is legal for this type converter, i.e. the
 /// type converts to itself.
 bool TypeConverter::isLegal(Type type) { return convertType(type) == type; }