[IR] Permit load/store/alloca for struct with the same scalable vectors.

[HsiangKai]

Summary

In this patch, we try to support load/store/alloca for scalable structures. We have posted a RFC for the proposal in the mailing list. https://groups.google.com/g/llvm-dev/c/6ZK2eS4-8t0/m/PG6H1NNDBAAJ

In RISC-V vector intrinsics, we have defined types containing multiple scalable vectors. Due to the flexible configuration of vector types with LMUL, the struct is a good fit to represent these types containing multiple scalable vectors. We permit users to define auto variables using these types. That is why we need to support load, store, and alloca for the scalable structure.

To support load, store and alloca for the scalable structure, we have to achieve the following two points.

• In StructLayout, we use uint64_t for the StructSize. Currently, we have TypeSize to represent the size of data. We should modify the StructLayout using the TypeSize for data size regardless to support scalable struct or not.
• In order to handle load and store scalable struct correctly, we need to modify ComputeValueVTs() to return the correct information. The returned Offsets should be represented as TypeSize. In order to collect the correct information, we need to represent the StructSize in StructLayout using TypeSize. (refer to the first point.)

We have some limitation for the scalable structure.

• Due to the TypeSize could only represent either scalable size or fixed size, we could not mix the fixed objects and scalable objects in a struct. We only support struct with all scalable objects.
• We only need load/store/alloca scalable struct. We have no need to support GEP for scalable struct. It could alleviate the effort to expand the capabilities of IR.

[david-arm]

llvm/include/llvm/IR/DataLayout.h

630

This should also return a TypeSize to be consistent with getSizeInBits.

llvm/include/llvm/Support/TypeSize.h

148 ↗

(On Diff #328971)

I'm not sure we want to use operators here - see the TypeSize class for how we did this using isKnownXY functions. There was a long history of discussion about the use of operators, which you can see on some of the earlier patches. The problem with operators is that there are cases where you simply do not know the answer. For example, there is no compile-time answer for Fixed(8) < Scalable(4) since we don't know the value of vscale.

[HsiangKai]

Add comments before getSizeInBytes() and remove compare operators in the StackOffset.

[HsiangKai]

llvm/include/llvm/IR/DataLayout.h

630

How about keep the interface as so.

In current implementation, the elements will be scalar or scalable. The uses of StructLayout::getSizeInBytes() will be compared with StructLayout::getElementOffset() usually. Keep the interface to return uint64_t should be fine.

I added a FIXME comment before the function.

[david-arm]

llvm/include/llvm/IR/DataLayout.h

630

But isn't the problem here that the result is now wrong? We're now returning the minimum value, rather than a TypeSize that tries to express the complete value. Also, it seems quite confusing that returning a size in bits gives you TypeSize and returning a size in bytes gives you uint64_t. This is inconsistent with how sizes are returned elsewhere, for example see:

include/llvm/CodeGen/ValueTypes.h

where we have

TypeSize getSizeInBits() const
TypeSize getStoreSize() const

where the latter returns bytes. If a function must return the minimum value then I think at least the function name should reflect that with a rename, i.e. getMinSizeInBytes() so that the caller understands this is not the actual size.

[frasercrmck]

I'm curious - the restriction that it must be all scalable or all fixed is because we're using TypeSize with only one dimension? It sounds like we'd ideally want to be able to represent the size of structs with something more like StackOffset with two dimensions?

[sdesmalen]

Hi @HsiangKai, this is not a supported use-case for scalable vectors and it is explicitly called out in the LangRef:

Scalable vectors cannot be global variables or members of arrays because their size is unknown at compile time. They are allowed in structs to facilitate intrinsics returning multiple values. Structs containing scalable vectors cannot be used in loads, stores, allocas, or GEPs.

There have been a few conversations about this before. I previously left a comment on D94142 and there was also some discussion on @craig.topper's RFC to add limited support for intrinsics, which can be found here: https://lists.llvm.org/pipermail/llvm-dev/2021-January/147639.html

Supporting scalable vector types as exclusive members of structs that can be used in loads/stores/allocas is a bit of a sliding scale, because the next question will be the one @frasercrmck is asking: "Why can't you mix fixed/scalable types?".

It's been still relatively recent that we've settled on a definition for TypeSize which still needs to be supported in many places in the code-base. Mixing fixed/scalable adds another dimension of complexity, because then all interfaces that take an offset also need to consider the fact that it may be comprised of a fixed-width and scalable-width component. Also, at the moment there isn't really a use-case, there are no languages that allow or support structs/aggregates with scalable members. The original advice given for "What if C/C++ allowed such a use case?" was to solve this in Clang, so as to remove the need for such support in LLVM IR.

The only reason we wanted to allow the use-case for return-values/operands of intrinsics is so that we can model operations that return multiple values which can be scalable, such as first-faulting loads where it returns a data vector (the loaded data) and a predicate mask for the lanes that faulted.

@HsiangKai was there a specific use-case that you had in mind for this?

[craig.topper]

In D98169#2619303, @sdesmalen wrote:

Hi @HsiangKai, this is not a supported use-case for scalable vectors and it is explicitly called out in the LangRef:

Scalable vectors cannot be global variables or members of arrays because their size is unknown at compile time. They are allowed in structs to facilitate intrinsics returning multiple values. Structs containing scalable vectors cannot be used in loads, stores, allocas, or GEPs.

There have been a few conversations about this before. I previously left a comment on D94142 and there was also some discussion on @craig.topper's RFC to add limited support for intrinsics, which can be found here: https://lists.llvm.org/pipermail/llvm-dev/2021-January/147639.html

Supporting scalable vector types as exclusive members of structs that can be used in loads/stores/allocas is a bit of a sliding scale, because the next question will be the one @frasercrmck is asking: "Why can't you mix fixed/scalable types?".

It's been still relatively recent that we've settled on a definition for TypeSize which still needs to be supported in many places in the code-base. Mixing fixed/scalable adds another dimension of complexity, because then all interfaces that take an offset also need to consider the fact that it may be comprised of a fixed-width and scalable-width component. Also, at the moment there isn't really a use-case, there are no languages that allow or support structs/aggregates with scalable members. The original advice given for "What if C/C++ allowed such a use case?" was to solve this in Clang, so as to remove the need for such support in LLVM IR.

The only reason we wanted to allow the use-case for return-values/operands of intrinsics is so that we can model operations that return multiple values which can be scalable, such as first-faulting loads where it returns a data vector (the loaded data) and a predicate mask for the lanes that faulted.

@HsiangKai was there a specific use-case that you had in mind for this?

We want this to support the segment load/store intrinsics defined here https://github.com/riscv/rvv-intrinsic-doc/blob/master/intrinsic_funcs/03_vector_load_store_segment_instructions_zvlsseg.md These return 2 to 8 vectors that have been loaded into consecutive registers. I believe SVE has similar instructions. I believe SVE represents these using types wider than their normal scalable vector types and relies on the type legalizer to split them up in the backend. This works for SVE because there is only one known minimum size for all scalable vector types so the type legalizer will always split down to that minimum type.

For RISC-V vectors we already use 7 different sizes of scalable vectors to represent the ability of our instructions to operate on 2, 4, or 8 registers simultaneously. And for 1/2, 1/4, and 1/8 fractional registers. The segment load/store instructions add an extra dimension where they can produce/consume 2, 3, or 4 pairs of registers or 2 quadruples, for examples. Following the SVE strategy would give us ambiguous types for the type legalizer.

To solve this we would like to use a struct for the segment load/stores to separate them in IR. Since clang needs an address for every variable and needs to be able to load/store them we need to support load/store/alloca.

[sdesmalen]

In D98169#2619874, @craig.topper wrote:

We want this to support the segment load/store intrinsics defined here https://github.com/riscv/rvv-intrinsic-doc/blob/master/intrinsic_funcs/03_vector_load_store_segment_instructions_zvlsseg.md These return 2 to 8 vectors that have been loaded into consecutive registers. I believe SVE has similar instructions. I believe SVE represents these using types wider than their normal scalable vector types and relies on the type legalizer to split them up in the backend. This works for SVE because there is only one known minimum size for all scalable vector types so the type legalizer will always split down to that minimum type.

Thanks for providing the context!

For RISC-V vectors we already use 7 different sizes of scalable vectors to represent the ability of our instructions to operate on 2, 4, or 8 registers simultaneously. And for 1/2, 1/4, and 1/8 fractional registers. The segment load/store instructions add an extra dimension where they can produce/consume 2, 3, or 4 pairs of registers or 2 quadruples, for examples. Following the SVE strategy would give us ambiguous types for the type legalizer.

How does that look in terms of IR? Is the number of registers somehow represented in the (LLVM IR) vector type? Or are the types the same, but the compiler generates different code depending on what mode is set? For SVE we know we can split the vector because <vscale x 8 x i32> is twice the size of <vscale x 4 x i32>, regardless of the value for vscale. Indeed we know SVE vectors area multiple of 128bits, and therefore that <vscale x 4 x i32> is legal. In order to make any assumptions about splitting/legalization, the compiler will need to know which types are legal, and so would expect the compiler to know the mode (2, 4 ,8) for RVV when generating the code, and therefore have similar knowledge about which types are legal and how the vectors are represented/split into registers. How does that lead to ambiguous types?

To solve this we would like to use a struct for the segment load/stores to separate them in IR. Since clang needs an address for every variable and needs to be able to load/store them we need to support load/store/alloca.

These (C/C++-level) intrinsics are probably implemented using target-specific intrinsics or perhaps a common LLVM IR intrinsic like masked.load, which should be able to take/return a struct with scalable members after D94142. If so, it should be possible to handle this in Clang by emitting extractvalue instructions and storing each member individually. That would avoid any changes to LLVM IR. Is that something you've considered?

If we do need to make this work for scalable vectors, I think it needs a message to the mailing list because it's a change to the LangRef and capabilities of scalable vectors, given previous discussions on this topic. I'd like to avoid giving the impression that we're quietly moving the goalpost on what scalable vectors can do in IR.

[HsiangKai]

This patch is a proof of concept patch. We should have a formal discussion about the idea before reviewing this patch.

Some related patches:
We model segment load/store types in D97264.
We intend to separate the uses of scalable types and scalable vector types in D98161.

[HsiangKai]

• Change the return type of getSizeInBytes() to TypeSize.
• Mark the places we need to take care.

[HsiangKai]

Use TypeSize for offsets instead of StackOffset.

The memebers are all scalable or all fixed objects. We could use TypeSize for offsets. In other places in the current implementation, it uses TypeSize as the offset type. Use TypeSize for member offsets is more consistent with the current implementation.

[HsiangKai]

Fix test failed.

[HsiangKai]

Consider scalable struct types in passes processing StructType and remove TODO comments.

[craig.topper]

llvm/include/llvm/IR/DataLayout.h

627

The is special code in DataLayout::getStructLayout that allocates the size of StructLayout to account for the needed size of MemberOffsets. That code does not appear to be updated in this patch.

I'm not sure we want to use SmallVector here. The minimum size is increasing the size of every StructLayout object. SmallVector contains the number of elements which is redundant with the NumElements in the StructLayout itself. SmallVector also contains an additional field for the capacity of the allocated memory.

Can we go back to the variable sized array and update DataLayout::getStructLayout to use TypeSize instead of uint64_t to calculate the needed space?

llvm/lib/Analysis/MemoryBuiltins.cpp

354

Was it already a bug that scalable vectors weren't checked for here?

llvm/lib/Analysis/ScalarEvolution.cpp

3447

"could not" -> "cannot"

llvm/lib/CodeGen/Analysis.cpp

88–89

Can we remove the conditional here and always get the the struct layout? Then we don't need to check for null StructLayout in the loop. This was one of the change we did previously when we allowed scalable vectors in structs for intrinsic returns.

103

Is the operand reordering required? I would hope that TypeSize multiplied by unsigned can be in either order.

llvm/lib/CodeGen/GlobalISel/CallLowering.cpp

773

Does this need to be addressed as part of this review?

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

1891

Maybe we need a new entrypoint to ComputeValueVTs that does this automatically based on the type passed to argument 2?

llvm/lib/IR/Type.cpp

187

Please address this lint warnig

188

I'm slightly concerned that containsScalableVectorType() is recursive and we're now calling isScalableType in a bunch of places. This could be bad for deeply nested structs.

On that subject, are we allowing or preventing struct of scalable vectors to be part of other structs?

543

One -> Element.

[HsiangKai]

• Add an assertion to check if it is a scalable struct type before creating GEP instructions.
• Address comments.

[HsiangKai]

Fix build fail.

[craig.topper]

llvm/lib/CodeGen/Analysis.cpp

88–89

Nevermind. We need this to support a mix of scalable and scalars for intrinsic returns.

[HsiangKai]

Use TrailingObjects for MemberOffsets.

[craig.topper]

@HsiangKai can you rebase this on trunk. Some of the TrailingObjects changes would go away.

[HsiangKai]

Rebase.

[HsiangKai]

In D98169#2622089, @sdesmalen wrote:

In D98169#2619874, @craig.topper wrote:

We want this to support the segment load/store intrinsics defined here https://github.com/riscv/rvv-intrinsic-doc/blob/master/intrinsic_funcs/03_vector_load_store_segment_instructions_zvlsseg.md These return 2 to 8 vectors that have been loaded into consecutive registers. I believe SVE has similar instructions. I believe SVE represents these using types wider than their normal scalable vector types and relies on the type legalizer to split them up in the backend. This works for SVE because there is only one known minimum size for all scalable vector types so the type legalizer will always split down to that minimum type.

Thanks for providing the context!

For RISC-V vectors we already use 7 different sizes of scalable vectors to represent the ability of our instructions to operate on 2, 4, or 8 registers simultaneously. And for 1/2, 1/4, and 1/8 fractional registers. The segment load/store instructions add an extra dimension where they can produce/consume 2, 3, or 4 pairs of registers or 2 quadruples, for examples. Following the SVE strategy would give us ambiguous types for the type legalizer.

How does that look in terms of IR? Is the number of registers somehow represented in the (LLVM IR) vector type? Or are the types the same, but the compiler generates different code depending on what mode is set? For SVE we know we can split the vector because <vscale x 8 x i32> is twice the size of <vscale x 4 x i32>, regardless of the value for vscale. Indeed we know SVE vectors area multiple of 128bits, and therefore that <vscale x 4 x i32> is legal. In order to make any assumptions about splitting/legalization, the compiler will need to know which types are legal, and so would expect the compiler to know the mode (2, 4 ,8) for RVV when generating the code, and therefore have similar knowledge about which types are legal and how the vectors are represented/split into registers. How does that lead to ambiguous types?

To solve this we would like to use a struct for the segment load/stores to separate them in IR. Since clang needs an address for every variable and needs to be able to load/store them we need to support load/store/alloca.

These (C/C++-level) intrinsics are probably implemented using target-specific intrinsics or perhaps a common LLVM IR intrinsic like masked.load, which should be able to take/return a struct with scalable members after D94142. If so, it should be possible to handle this in Clang by emitting extractvalue instructions and storing each member individually. That would avoid any changes to LLVM IR. Is that something you've considered?

We have defined types containing multiple scalable vectors and we permit users to use these types to define auto variables. That is why we need load, store and alloca capabilities for scalable structure.

If we do need to make this work for scalable vectors, I think it needs a message to the mailing list because it's a change to the LangRef and capabilities of scalable vectors, given previous discussions on this topic. I'd like to avoid giving the impression that we're quietly moving the goalpost on what scalable vectors can do in IR.

I have posted a RFC for the proposal in the mailing list.
https://groups.google.com/g/llvm-dev/c/6ZK2eS4-8t0/m/PG6H1NNDBAAJ

llvm/include/llvm/CodeGen/Analysis.h

 /// EVTs that represent all the individual underlying
 /// non-aggregate types that comprise it.
 ///
 /// If Offsets is non-null, it points to a vector to be filled in
 /// with the in-memory offsets of each of the individual values.
 ///
 void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
                      SmallVectorImpl<EVT> &ValueVTs,
-                     SmallVectorImpl<uint64_t> *Offsets = nullptr,
-                     uint64_t StartingOffset = 0);
+                     SmallVectorImpl<TypeSize> *Offsets = nullptr,
+                     TypeSize StartingOffset = TypeSize::Fixed(0));
+void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
+                     SmallVectorImpl<EVT> &ValueVTs,
+                     SmallVectorImpl<TypeSize> *Offsets = nullptr) {
+  TypeSize StartingOffset =
+      Ty->isScalableType() ? TypeSize::Scalable(0) : TypeSize::Fixed(0);
+  return ComputeValueVTs(TLI, DL, Ty, ValueVTs, Offsets, StartingOffset);
+}
 
 /// Variant of ComputeValueVTs that also produces the memory VTs.
 void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
                      SmallVectorImpl<EVT> &ValueVTs,
                      SmallVectorImpl<EVT> *MemVTs,
-                     SmallVectorImpl<uint64_t> *Offsets = nullptr,
-                     uint64_t StartingOffset = 0);
+                     SmallVectorImpl<TypeSize> *Offsets = nullptr,
+                     TypeSize StartingOffset = TypeSize::Fixed(0));
+void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
+                     SmallVectorImpl<EVT> &ValueVTs,
+                     SmallVectorImpl<EVT> *MemVTs,
+                     SmallVectorImpl<TypeSize> *Offsets = nullptr) {
+  TypeSize StartingOffset =
+      Ty->isScalableType() ? TypeSize::Scalable(0) : TypeSize::Fixed(0);
+  return ComputeValueVTs(TLI, DL, Ty, ValueVTs, MemVTs, Offsets,
+                         StartingOffset);
+}
 
 /// computeValueLLTs - Given an LLVM IR type, compute a sequence of
 /// LLTs that represent all the individual underlying
 /// non-aggregate types that comprise it.
 ///
 /// If Offsets is non-null, it points to a vector to be filled in
 /// with the in-memory offsets of each of the individual values.
 ///

llvm/include/llvm/IR/DataLayout.h

 
 inline LLVMTargetDataRef wrap(const DataLayout *P) {
   return reinterpret_cast<LLVMTargetDataRef>(const_cast<DataLayout *>(P));
 }
 
 /// Used to lazily calculate structure layout information for a target machine,
 /// based on the DataLayout structure.
 class StructLayout {
-  uint64_t StructSize;
+  TypeSize StructSize;
   Align StructAlignment;
   unsigned IsPadded : 1;
   unsigned NumElements : 31;
-  uint64_t MemberOffsets[1]; // variable sized array!
+  TypeSize MemberOffsets[1]; // variable sized array!
 
 public:
-  uint64_t getSizeInBytes() const { return StructSize; }
+  TypeSize getSizeInBytes() const { return StructSize; }
 
-  uint64_t getSizeInBits() const { return 8 * StructSize; }
+  TypeSize getSizeInBits() const { return 8 * StructSize; }
 
   Align getAlignment() const { return StructAlignment; }
 
   /// Returns whether the struct has padding or not between its fields.
   /// NB: Padding in nested element is not taken into account.
   bool hasPadding() const { return IsPadded; }
 
   /// Given a valid byte offset into the structure, returns the structure
   /// index that contains it.
   unsigned getElementContainingOffset(uint64_t Offset) const;
 
-  uint64_t getElementOffset(unsigned Idx) const {
+  TypeSize getElementOffset(unsigned Idx) const {
     assert(Idx < NumElements && "Invalid element idx!");
     return MemberOffsets[Idx];
   }
 
-  uint64_t getElementOffsetInBits(unsigned Idx) const {
+  TypeSize getElementOffsetInBits(unsigned Idx) const {
     return getElementOffset(Idx) * 8;
   }
 
 private:
   friend class DataLayout; // Only DataLayout can create this class
 
   StructLayout(StructType *ST, const DataLayout &DL);
 };
case Type::PointerTyID:
     return TypeSize::Fixed(getPointerSizeInBits(Ty->getPointerAddressSpace()));
   case Type::ArrayTyID: {
     ArrayType *ATy = cast<ArrayType>(Ty);
     return ATy->getNumElements() *
            getTypeAllocSizeInBits(ATy->getElementType());
   }
   case Type::StructTyID:
     // Get the layout annotation... which is lazily created on demand.
-    return TypeSize::Fixed(
-                        getStructLayout(cast<StructType>(Ty))->getSizeInBits());
+    return getStructLayout(cast<StructType>(Ty))->getSizeInBits();
   case Type::IntegerTyID:
     return TypeSize::Fixed(Ty->getIntegerBitWidth());
   case Type::HalfTyID:
   case Type::BFloatTyID:
     return TypeSize::Fixed(16);
   case Type::FloatTyID:
     return TypeSize::Fixed(32);
   case Type::DoubleTyID:

llvm/include/llvm/IR/Instructions.h

GetElementPtrInst::GetElementPtrInst(Type *PointeeType, Value *Ptr,
                                      Instruction *InsertBefore)
     : Instruction(getGEPReturnType(PointeeType, Ptr, IdxList), GetElementPtr,
                   OperandTraits<GetElementPtrInst>::op_end(this) - Values,
                   Values, InsertBefore),
       SourceElementType(PointeeType),
       ResultElementType(getIndexedType(PointeeType, IdxList)) {
   assert(ResultElementType ==
          cast<PointerType>(getType()->getScalarType())->getElementType());
+  assert(!(SourceElementType->isStructTy() &&
+           SourceElementType->isScalableType()) &&
+         "No support GEP for scalable struct types.");
   init(Ptr, IdxList, NameStr);
 }
 
 GetElementPtrInst::GetElementPtrInst(Type *PointeeType, Value *Ptr,
                                      ArrayRef<Value *> IdxList, unsigned Values,
                                      const Twine &NameStr,
                                      BasicBlock *InsertAtEnd)
     : Instruction(getGEPReturnType(PointeeType, Ptr, IdxList), GetElementPtr,
                   OperandTraits<GetElementPtrInst>::op_end(this) - Values,
                   Values, InsertAtEnd),
       SourceElementType(PointeeType),
       ResultElementType(getIndexedType(PointeeType, IdxList)) {
   assert(ResultElementType ==
          cast<PointerType>(getType()->getScalarType())->getElementType());
+  assert(!(SourceElementType->isStructTy() &&
+           SourceElementType->isScalableType()) &&
+         "No support GEP for scalable struct types.");
   init(Ptr, IdxList, NameStr);
 }
 
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(GetElementPtrInst, Value)
 
 //===----------------------------------------------------------------------===//
 //                               ICmpInst Class
 //===----------------------------------------------------------------------===//

llvm/include/llvm/IR/Type.h

public:
   /// Return true if this is a pointer type or a vector of pointer types.
   bool isPtrOrPtrVectorTy() const { return getScalarType()->isPointerTy(); }
 
   /// True if this is an instance of VectorType.
   inline bool isVectorTy() const {
     return getTypeID() == ScalableVectorTyID || getTypeID() == FixedVectorTyID;
   }
 
+  /// True if this is an instance of scalable types.
+  bool isScalableType() const;
+
   /// Return true if this type could be converted with a lossless BitCast to
   /// type 'Ty'. For example, i8* to i32*. BitCasts are valid for types of the
   /// same size only where no re-interpretation of the bits is done.
   /// Determine if this type could be losslessly bitcast to Ty
   bool canLosslesslyBitCastTo(Type *Ty) const;
 
   /// Return true if this type is empty, that is, it has no elements or all of
   /// its elements are empty.

llvm/lib/Analysis/ConstantFolding.cpp

Constant *SymbolicallyEvaluateGEP(const GEPOperator *GEP,
                                   const DataLayout &DL,
                                   const TargetLibraryInfo *TLI) {
   const GEPOperator *InnermostGEP = GEP;
   bool InBounds = GEP->isInBounds();
 
   Type *SrcElemTy = GEP->getSourceElementType();
   Type *ResElemTy = GEP->getResultElementType();
   Type *ResTy = GEP->getType();
-  if (!SrcElemTy->isSized() || isa<ScalableVectorType>(SrcElemTy))
+  if (!SrcElemTy->isSized() || SrcElemTy->isScalableType())
     return nullptr;
 
   if (Constant *C = CastGEPIndices(SrcElemTy, Ops, ResTy,
                                    GEP->getInRangeIndex(), DL, TLI))
     return C;
 
   Constant *Ptr = Ops[0];
   if (!Ptr->getType()->isPointerTy())

llvm/lib/Analysis/Loads.cpp

 }
 
 bool llvm::isDereferenceableAndAlignedPointer(const Value *V, Type *Ty,
                                               MaybeAlign MA,
                                               const DataLayout &DL,
                                               const Instruction *CtxI,
                                               const DominatorTree *DT,
                                               const TargetLibraryInfo *TLI) {
-  // For unsized types or scalable vectors we don't know exactly how many bytes
+  // For unsized types or scalable types we don't know exactly how many bytes
   // are dereferenced, so bail out.
-  if (!Ty->isSized() || isa<ScalableVectorType>(Ty))
+  if (!Ty->isSized() || Ty->isScalableType())
     return false;
 
   // When dereferenceability information is provided by a dereferenceable
   // attribute, we know exactly how many bytes are dereferenceable. If we can
   // determine the exact offset to the attributed variable, we can use that
   // information here.
 
   // Require ABI alignment for loads without alignment specification

llvm/lib/Analysis/MemoryBuiltins.cpp

 static Value *computeArraySize(const CallInst *CI, const DataLayout &DL,
                                const TargetLibraryInfo *TLI,
                                bool LookThroughSExt = false) {
   if (!CI)
     return nullptr;
 
   // The size of the malloc's result type must be known to determine array size.
   Type *T = getMallocAllocatedType(CI, TLI);
-  if (!T || !T->isSized())
+  if (!T || !T->isSized() || T->isScalableType())
     return nullptr;
 
   unsigned ElementSize = DL.getTypeAllocSize(T);
   if (StructType *ST = dyn_cast<StructType>(T))
     ElementSize = DL.getStructLayout(ST)->getSizeInBytes();
 
   // If malloc call's arg can be determined to be a multiple of ElementSize,
   // return the multiple.  Otherwise, return NULL.

llvm/lib/Analysis/ScalarEvolution.cpp

SCEV::NoWrapFlags OffsetWrap =
       GEP->isInBounds() ? SCEV::FlagNSW : SCEV::FlagAnyWrap;
 
   Type *CurTy = GEP->getType();
   bool FirstIter = true;
   SmallVector<const SCEV *, 4> Offsets;
   for (const SCEV *IndexExpr : IndexExprs) {
     // Compute the (potentially symbolic) offset in bytes for this index.
     if (StructType *STy = dyn_cast<StructType>(CurTy)) {
+      assert(!STy->isScalableType() &&
+             "Scalable struct cannot be used in GEP.");
       // For a struct, add the member offset.
       ConstantInt *Index = cast<SCEVConstant>(IndexExpr)->getValue();
       unsigned FieldNo = Index->getZExtValue();
       const SCEV *FieldOffset = getOffsetOfExpr(IntIdxTy, STy, FieldNo);
       Offsets.push_back(FieldOffset);
 
       // Update CurTy to the type of the field at Index.
       CurTy = STy->getTypeAtIndex(Index);

llvm/lib/CodeGen/Analysis.cpp

 /// non-aggregate types that comprise it.
 ///
 /// If Offsets is non-null, it points to a vector to be filled in
 /// with the in-memory offsets of each of the individual values.
 ///
 void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
                            Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
                            SmallVectorImpl<EVT> *MemVTs,
-                           SmallVectorImpl<uint64_t> *Offsets,
-                           uint64_t StartingOffset) {
+                           SmallVectorImpl<TypeSize> *Offsets,
+                           TypeSize StartingOffset) {
   // Given a struct type, recursively traverse the elements.
   if (StructType *STy = dyn_cast<StructType>(Ty)) {
-    // If the Offsets aren't needed, don't query the struct layout. This allows
-    // us to support structs with scalable vectors for operations that don't
-    // need offsets.
-    const StructLayout *SL = Offsets ? DL.getStructLayout(STy) : nullptr;
-    for (StructType::element_iterator EB = STy->element_begin(),
-                                      EI = EB,
+    const StructLayout *SL = DL.getStructLayout(STy);
+    for (StructType::element_iterator EB = STy->element_begin(), EI = EB,
                                       EE = STy->element_end();
-         EI != EE; ++EI) {
-      // Don't compute the element offset if we didn't get a StructLayout above.
-      uint64_t EltOffset = SL ? SL->getElementOffset(EI - EB) : 0;
+         EI != EE; ++EI)
       ComputeValueVTs(TLI, DL, *EI, ValueVTs, MemVTs, Offsets,
-                      StartingOffset + EltOffset);
-    }
+                      StartingOffset + SL->getElementOffset(EI - EB));
     return;
   }
   // Given an array type, recursively traverse the elements.
   if (ArrayType *ATy = dyn_cast<ArrayType>(Ty)) {
     Type *EltTy = ATy->getElementType();
-    uint64_t EltSize = DL.getTypeAllocSize(EltTy).getFixedValue();
+    TypeSize EltSize = DL.getTypeAllocSize(EltTy);
     for (unsigned i = 0, e = ATy->getNumElements(); i != e; ++i)
       ComputeValueVTs(TLI, DL, EltTy, ValueVTs, MemVTs, Offsets,
                       StartingOffset + i * EltSize);
     return;
   }
   // Interpret void as zero return values.
   if (Ty->isVoidTy())
     return;
   // Base case: we can get an EVT for this LLVM IR type.
   ValueVTs.push_back(TLI.getValueType(DL, Ty));
   if (MemVTs)
     MemVTs->push_back(TLI.getMemValueType(DL, Ty));
   if (Offsets)
     Offsets->push_back(StartingOffset);
 }
 
 void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
                            Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
-                           SmallVectorImpl<uint64_t> *Offsets,
-                           uint64_t StartingOffset) {
+                           SmallVectorImpl<TypeSize> *Offsets,
+                           TypeSize StartingOffset) {
   return ComputeValueVTs(TLI, DL, Ty, ValueVTs, /*MemVTs=*/nullptr, Offsets,
                          StartingOffset);
 }
 
 void llvm::computeValueLLTs(const DataLayout &DL, Type &Ty,
                             SmallVectorImpl<LLT> &ValueTys,
                             SmallVectorImpl<uint64_t> *Offsets,
                             uint64_t StartingOffset) {
   // Given a struct type, recursively traverse the elements.
   if (StructType *STy = dyn_cast<StructType>(&Ty)) {
+    assert(!STy->isScalableType() && "Unexpected scalable struct type.");
     // If the Offsets aren't needed, don't query the struct layout. This allows
     // us to support structs with scalable vectors for operations that don't
     // need offsets.
     const StructLayout *SL = Offsets ? DL.getStructLayout(STy) : nullptr;
     for (unsigned I = 0, E = STy->getNumElements(); I != E; ++I) {
       uint64_t EltOffset = SL ? SL->getElementOffset(I) : 0;
       computeValueLLTs(DL, *STy->getElementType(I), ValueTys, Offsets,
                        StartingOffset + EltOffset);

llvm/lib/CodeGen/CodeGenPrepare.cpp

static bool splitMergedValStore(StoreInst &SI, const DataLayout &DL,
                                 const TargetLowering &TLI) {
   // Handle simple but common cases only.
   Type *StoreType = SI.getValueOperand()->getType();
 
   // The code below assumes shifting a value by <number of bits>,
   // whereas scalable vectors would have to be shifted by
   // <2log(vscale) + number of bits> in order to store the
   // low/high parts. Bailing out for now.
-  if (isa<ScalableVectorType>(StoreType))
+  if (StoreType->isScalableType())
     return false;
 
   if (!DL.typeSizeEqualsStoreSize(StoreType) ||
       DL.getTypeSizeInBits(StoreType) == 0)
     return false;
 
   unsigned HalfValBitSize = DL.getTypeSizeInBits(StoreType) / 2;
   Type *SplitStoreType = Type::getIntNTy(SI.getContext(), HalfValBitSize);

llvm/lib/CodeGen/GlobalISel/CallLowering.cpp

 
 void CallLowering::splitToValueTypes(const ArgInfo &OrigArg,
                                      SmallVectorImpl<ArgInfo> &SplitArgs,
                                      const DataLayout &DL,
                                      CallingConv::ID CallConv) const {
   LLVMContext &Ctx = OrigArg.Ty->getContext();
 
   SmallVector<EVT, 4> SplitVTs;
-  SmallVector<uint64_t, 4> Offsets;
-  ComputeValueVTs(*TLI, DL, OrigArg.Ty, SplitVTs, &Offsets, 0);
+  ComputeValueVTs(*TLI, DL, OrigArg.Ty, SplitVTs);
 
   if (SplitVTs.size() == 0)
     return;
 
   if (SplitVTs.size() == 1) {
     // No splitting to do, but we want to replace the original type (e.g. [1 x
     // double] -> double).
     SplitArgs.emplace_back(OrigArg.Regs[0], SplitVTs[0].getTypeForEVT(Ctx),

 void CallLowering::insertSRetLoads(MachineIRBuilder &MIRBuilder, Type *RetTy,
                                    ArrayRef<Register> VRegs, Register DemoteReg,
                                    int FI) const {
   MachineFunction &MF = MIRBuilder.getMF();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   const DataLayout &DL = MF.getDataLayout();
 
   SmallVector<EVT, 4> SplitVTs;
-  SmallVector<uint64_t, 4> Offsets;
-  ComputeValueVTs(*TLI, DL, RetTy, SplitVTs, &Offsets, 0);
+  SmallVector<TypeSize, 4> Offsets;
+  ComputeValueVTs(*TLI, DL, RetTy, SplitVTs, &Offsets);
 
   assert(VRegs.size() == SplitVTs.size());
 
   unsigned NumValues = SplitVTs.size();
   Align BaseAlign = DL.getPrefTypeAlign(RetTy);
   Type *RetPtrTy = RetTy->getPointerTo(DL.getAllocaAddrSpace());
   LLT OffsetLLTy = getLLTForType(*DL.getIntPtrType(RetPtrTy), DL);
 
   MachinePointerInfo PtrInfo = MachinePointerInfo::getFixedStack(MF, FI);
 
   for (unsigned I = 0; I < NumValues; ++I) {
     Register Addr;
-    MIRBuilder.materializePtrAdd(Addr, DemoteReg, OffsetLLTy, Offsets[I]);
+    // FIXME: Considering scalable struct types in GlobalISel.
+    MIRBuilder.materializePtrAdd(Addr, DemoteReg, OffsetLLTy,
+                                 Offsets[I].getKnownMinSize());
     auto *MMO = MF.getMachineMemOperand(PtrInfo, MachineMemOperand::MOLoad,
                                         MRI.getType(VRegs[I]).getSizeInBytes(),
                                         commonAlignment(BaseAlign, Offsets[I]));
     MIRBuilder.buildLoad(VRegs[I], Addr, *MMO);
   }
 }
 
 void CallLowering::insertSRetStores(MachineIRBuilder &MIRBuilder, Type *RetTy,
                                     ArrayRef<Register> VRegs,
                                     Register DemoteReg) const {
   MachineFunction &MF = MIRBuilder.getMF();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   const DataLayout &DL = MF.getDataLayout();
 
   SmallVector<EVT, 4> SplitVTs;
-  SmallVector<uint64_t, 4> Offsets;
-  ComputeValueVTs(*TLI, DL, RetTy, SplitVTs, &Offsets, 0);
+  SmallVector<TypeSize, 4> Offsets;
+  ComputeValueVTs(*TLI, DL, RetTy, SplitVTs, &Offsets);
 
   assert(VRegs.size() == SplitVTs.size());
 
   unsigned NumValues = SplitVTs.size();
   Align BaseAlign = DL.getPrefTypeAlign(RetTy);
   unsigned AS = DL.getAllocaAddrSpace();
   LLT OffsetLLTy =
       getLLTForType(*DL.getIntPtrType(RetTy->getPointerTo(AS)), DL);
 
   MachinePointerInfo PtrInfo(AS);
 
   for (unsigned I = 0; I < NumValues; ++I) {
     Register Addr;
-    MIRBuilder.materializePtrAdd(Addr, DemoteReg, OffsetLLTy, Offsets[I]);
+    // FIXME: Considering scalable struct types in GlobalISel.
+    MIRBuilder.materializePtrAdd(Addr, DemoteReg, OffsetLLTy,
+                                 Offsets[I].getKnownMinSize());
     auto *MMO = MF.getMachineMemOperand(PtrInfo, MachineMemOperand::MOStore,
                                         MRI.getType(VRegs[I]).getSizeInBytes(),
                                         commonAlignment(BaseAlign, Offsets[I]));
     MIRBuilder.buildStore(VRegs[I], Addr, *MMO);
   }
 }
 
 void CallLowering::insertSRetIncomingArgument(
     const Function &F, SmallVectorImpl<ArgInfo> &SplitArgs, Register &DemoteReg,
     MachineRegisterInfo &MRI, const DataLayout &DL) const {
   unsigned AS = DL.getAllocaAddrSpace();
   DemoteReg = MRI.createGenericVirtualRegister(
       LLT::pointer(AS, DL.getPointerSizeInBits(AS)));
 
   Type *PtrTy = PointerType::get(F.getReturnType(), AS);
 
   SmallVector<EVT, 1> ValueVTs;
   ComputeValueVTs(*TLI, DL, PtrTy, ValueVTs);
 
   // NOTE: Assume that a pointer won't get split into more than one VT.
   assert(ValueVTs.size() == 1);
 
   ArgInfo DemoteArg(DemoteReg, ValueVTs[0].getTypeForEVT(PtrTy->getContext()));
   setArgFlags(DemoteArg, AttributeList::ReturnIndex, DL, F);
   DemoteArg.Flags[0].setSRet();
   SplitArgs.insert(SplitArgs.begin(), DemoteArg);

 void CallLowering::getReturnInfo(CallingConv::ID CallConv, Type *RetTy,
                                  AttributeList Attrs,
                                  SmallVectorImpl<BaseArgInfo> &Outs,
                                  const DataLayout &DL) const {
   LLVMContext &Context = RetTy->getContext();
   ISD::ArgFlagsTy Flags = ISD::ArgFlagsTy();
 
   SmallVector<EVT, 4> SplitVTs;
   ComputeValueVTs(*TLI, DL, RetTy, SplitVTs);
   addArgFlagsFromAttributes(Flags, Attrs, AttributeList::ReturnIndex);
 
   for (EVT VT : SplitVTs) {
     unsigned NumParts =
         TLI->getNumRegistersForCallingConv(Context, CallConv, VT);
     MVT RegVT = TLI->getRegisterTypeForCallingConv(Context, CallConv, VT);
     Type *PartTy = EVT(RegVT).getTypeForEVT(Context);
 

llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

for (const Instruction &I : BB) {
             FrameIndex = MF->getFrameInfo().CreateFixedObject(
                 TySize, 0, /*IsImmutable=*/false, /*isAliased=*/true);
             MF->getFrameInfo().setObjectAlignment(FrameIndex, Alignment);
           } else {
             FrameIndex = MF->getFrameInfo().CreateStackObject(TySize, Alignment,
                                                               false, AI);
           }
 
-          // Scalable vectors may need a special StackID to distinguish
+          // Scalable types may need a special StackID to distinguish
           // them from other (fixed size) stack objects.
-          if (isa<ScalableVectorType>(Ty))
+          if (Ty->isScalableType()) {
+            assert(Ty->isSized() && "It is only permitted to alloca struct "
+                                    "with the same scalable vector types.");
             MF->getFrameInfo().setStackID(FrameIndex,
                                           TFI->getStackIDForScalableVectors());
+          }
 
           StaticAllocaMap[AI] = FrameIndex;
           // Update the catch handler information.
           if (Iter != CatchObjects.end()) {
             for (int *CatchObjPtr : Iter->second)
               *CatchObjPtr = FrameIndex;
           }
         } else {
for (const PHINode &PN : BB.phis()) {
       if (PN.getType()->isEmptyTy())
         continue;
 
       DebugLoc DL = PN.getDebugLoc();
       unsigned PHIReg = ValueMap[&PN];
       assert(PHIReg && "PHI node does not have an assigned virtual register!");
 
       SmallVector<EVT, 4> ValueVTs;
       ComputeValueVTs(*TLI, MF->getDataLayout(), PN.getType(), ValueVTs);
       for (EVT VT : ValueVTs) {
         unsigned NumRegisters = TLI->getNumRegisters(Fn->getContext(), VT);
         const TargetInstrInfo *TII = MF->getSubtarget().getInstrInfo();
         for (unsigned i = 0; i != NumRegisters; ++i)
           BuildMI(MBB, DL, TII->get(TargetOpcode::PHI), PHIReg + i);
         PHIReg += NumRegisters;
       }
     }

 ///
 /// In the case that the given value has struct or array type, this function
 /// will assign registers for each member or element.
 ///
 Register FunctionLoweringInfo::CreateRegs(Type *Ty, bool isDivergent) {
   const TargetLowering *TLI = MF->getSubtarget().getTargetLowering();
 
   SmallVector<EVT, 4> ValueVTs;
   ComputeValueVTs(*TLI, MF->getDataLayout(), Ty, ValueVTs);
 
   Register FirstReg;
   for (unsigned Value = 0, e = ValueVTs.size(); Value != e; ++Value) {
     EVT ValueVT = ValueVTs[Value];
     MVT RegisterVT = TLI->getRegisterType(Ty->getContext(), ValueVT);
 
     unsigned NumRegs = TLI->getNumRegisters(Ty->getContext(), ValueVT);
     for (unsigned i = 0; i != NumRegs; ++i) {

 /// ComputePHILiveOutRegInfo - Compute LiveOutInfo for a PHI's destination
 /// register based on the LiveOutInfo of its operands.
 void FunctionLoweringInfo::ComputePHILiveOutRegInfo(const PHINode *PN) {
   Type *Ty = PN->getType();
   if (!Ty->isIntegerTy() || Ty->isVectorTy())
     return;
 
   SmallVector<EVT, 1> ValueVTs;
   ComputeValueVTs(*TLI, MF->getDataLayout(), Ty, ValueVTs);
   assert(ValueVTs.size() == 1 &&
          "PHIs with non-vector integer types should have a single VT.");
   EVT IntVT = ValueVTs[0];
 
   if (TLI->getNumRegisters(PN->getContext(), IntVT) != 1)
     return;
   IntVT = TLI->getTypeToTransformTo(PN->getContext(), IntVT);
   unsigned BitWidth = IntVT.getSizeInBits();

 }
 
 const Value *
 FunctionLoweringInfo::getValueFromVirtualReg(Register Vreg) {
   if (VirtReg2Value.empty()) {
     SmallVector<EVT, 4> ValueVTs;
     for (auto &P : ValueMap) {
       ValueVTs.clear();
       ComputeValueVTs(*TLI, Fn->getParent()->getDataLayout(),
                       P.first->getType(), ValueVTs);
       unsigned Reg = P.second;
       for (EVT VT : ValueVTs) {
         unsigned NumRegisters = TLI->getNumRegisters(Fn->getContext(), VT);
         for (unsigned i = 0, e = NumRegisters; i != e; ++i)
           VirtReg2Value[Reg++] = P.first;
       }
     }
   }
   return VirtReg2Value.lookup(Vreg);
 }

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

 }
 
 SDValue SelectionDAG::getMemBasePlusOffset(SDValue Base, TypeSize Offset,
                                            const SDLoc &DL,
                                            const SDNodeFlags Flags) {
   EVT VT = Base.getValueType();
   SDValue Index;
 
+  if (Offset.getKnownMinSize() == 0)
+    return Base;
+
   if (Offset.isScalable())
     Index = getVScale(DL, Base.getValueType(),
                       APInt(Base.getValueSizeInBits().getFixedSize(),
                             Offset.getKnownMinSize()));
   else
     Index = getConstant(Offset.getFixedSize(), DL, VT);
 
   return getMemBasePlusOffset(Base, Index, DL, Flags);

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

 RegsForValue::RegsForValue(const SmallVector<unsigned, 4> &regs, MVT regvt,
                            EVT valuevt, Optional<CallingConv::ID> CC)
     : ValueVTs(1, valuevt), RegVTs(1, regvt), Regs(regs),
       RegCount(1, regs.size()), CallConv(CC) {}
 
 RegsForValue::RegsForValue(LLVMContext &Context, const TargetLowering &TLI,
                            const DataLayout &DL, unsigned Reg, Type *Ty,
                            Optional<CallingConv::ID> CC) {
   ComputeValueVTs(TLI, DL, Ty, ValueVTs);
 
   CallConv = CC;
 
   for (EVT ValueVT : ValueVTs) {
     unsigned NumRegs =
         isABIMangled()
             ? TLI.getNumRegistersForCallingConv(Context, CC.getValue(), ValueVT)
             : TLI.getNumRegisters(Context, ValueVT);
if (const ConstantDataSequential *CDS =
       return NodeMap[V] = DAG.getBuildVector(VT, getCurSDLoc(), Ops);
     }
 
     if (C->getType()->isStructTy() || C->getType()->isArrayTy()) {
       assert((isa<ConstantAggregateZero>(C) || isa<UndefValue>(C)) &&
              "Unknown struct or array constant!");
 
       SmallVector<EVT, 4> ValueVTs;
       ComputeValueVTs(TLI, DAG.getDataLayout(), C->getType(), ValueVTs);
       unsigned NumElts = ValueVTs.size();
       if (NumElts == 0)
         return SDValue(); // empty struct
       SmallVector<SDValue, 4> Constants(NumElts);
       for (unsigned i = 0; i != NumElts; ++i) {
         EVT EltVT = ValueVTs[i];
         if (isa<UndefValue>(C))
           Constants[i] = DAG.getUNDEF(EltVT);
void SelectionDAGBuilder::visitRet(const ReturnInst &I) {
   if (!FuncInfo.CanLowerReturn) {
     unsigned DemoteReg = FuncInfo.DemoteRegister;
     const Function *F = I.getParent()->getParent();
 
     // Emit a store of the return value through the virtual register.
     // Leave Outs empty so that LowerReturn won't try to load return
     // registers the usual way.
     SmallVector<EVT, 1> PtrValueVTs;
     ComputeValueVTs(TLI, DL,
                     F->getReturnType()->getPointerTo(
                         DAG.getDataLayout().getAllocaAddrSpace()),
                     PtrValueVTs);
 
     SDValue RetPtr = DAG.getCopyFromReg(DAG.getEntryNode(), getCurSDLoc(),
                                         DemoteReg, PtrValueVTs[0]);
     SDValue RetOp = getValue(I.getOperand(0));
 
     SmallVector<EVT, 4> ValueVTs, MemVTs;
-    SmallVector<uint64_t, 4> Offsets;
+    SmallVector<TypeSize, 4> Offsets;
     ComputeValueVTs(TLI, DL, I.getOperand(0)->getType(), ValueVTs, &MemVTs,
                     &Offsets);
     unsigned NumValues = ValueVTs.size();
 
     SmallVector<SDValue, 4> Chains(NumValues);
     Align BaseAlign = DL.getPrefTypeAlign(I.getOperand(0)->getType());
     for (unsigned i = 0; i != NumValues; ++i) {
       // An aggregate return value cannot wrap around the address space, so
       // offsets to its parts don't wrap either.
-      SDValue Ptr = DAG.getObjectPtrOffset(getCurSDLoc(), RetPtr,
-                                           TypeSize::Fixed(Offsets[i]));
+      SDValue Ptr = DAG.getObjectPtrOffset(getCurSDLoc(), RetPtr, Offsets[i]);
 
       SDValue Val = RetOp.getValue(RetOp.getResNo() + i);
       if (MemVTs[i] != ValueVTs[i])
         Val = DAG.getPtrExtOrTrunc(Val, getCurSDLoc(), MemVTs[i]);
       Chains[i] = DAG.getStore(
           Chain, getCurSDLoc(), Val,
           // FIXME: better loc info would be nice.
           Ptr, MachinePointerInfo::getUnknownStack(DAG.getMachineFunction()),
-          commonAlignment(BaseAlign, Offsets[i]));
+          commonAlignment(BaseAlign, Offsets[i].getKnownMinSize()));
     }
 
     Chain = DAG.getNode(ISD::TokenFactor, getCurSDLoc(),
                         MVT::Other, Chains);
   } else if (I.getNumOperands() != 0) {
     SmallVector<EVT, 4> ValueVTs;
     ComputeValueVTs(TLI, DL, I.getOperand(0)->getType(), ValueVTs);
     unsigned NumValues = ValueVTs.size();
     if (NumValues) {
       SDValue RetOp = getValue(I.getOperand(0));
 
       const Function *F = I.getParent()->getParent();
 
       bool NeedsRegBlock = TLI.functionArgumentNeedsConsecutiveRegisters(
           I.getOperand(0)->getType(), F->getCallingConv(),
void SelectionDAGBuilder::visitLandingPad(const LandingPadInst &LP) {
   // for its exception pointer and selector value. The extraction of exception
   // pointer or selector value from token type landingpads is not currently
   // supported.
   if (LP.getType()->isTokenTy())
     return;
 
   SmallVector<EVT, 2> ValueVTs;
   SDLoc dl = getCurSDLoc();
   ComputeValueVTs(TLI, DAG.getDataLayout(), LP.getType(), ValueVTs);
   assert(ValueVTs.size() == 2 && "Only two-valued landingpads are supported");
 
   // Get the two live-in registers as SDValues. The physregs have already been
   // copied into virtual registers.
   SDValue Ops[2];
   if (FuncInfo.ExceptionPointerVirtReg) {
     Ops[0] = DAG.getZExtOrTrunc(
         DAG.getCopyFromReg(DAG.getEntryNode(), dl,

 static bool hasOnlySelectUsers(const Value *Cond) {
   return llvm::all_of(Cond->users(), [](const Value *V) {
     return isa<SelectInst>(V);
   });
 }
 
 void SelectionDAGBuilder::visitSelect(const User &I) {
   SmallVector<EVT, 4> ValueVTs;
   ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(), I.getType(),
                   ValueVTs);
   unsigned NumValues = ValueVTs.size();
   if (NumValues == 0) return;
 
   SmallVector<SDValue, 4> Values(NumValues);
   SDValue Cond     = getValue(I.getOperand(0));
   SDValue LHSVal   = getValue(I.getOperand(1));
   SDValue RHSVal   = getValue(I.getOperand(2));
void SelectionDAGBuilder::visitInsertValue(const User &I) {
   Type *ValTy = Op1->getType();
   bool IntoUndef = isa<UndefValue>(Op0);
   bool FromUndef = isa<UndefValue>(Op1);
 
   unsigned LinearIndex = ComputeLinearIndex(AggTy, Indices);
 
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   SmallVector<EVT, 4> AggValueVTs;
   ComputeValueVTs(TLI, DAG.getDataLayout(), AggTy, AggValueVTs);
   SmallVector<EVT, 4> ValValueVTs;
   ComputeValueVTs(TLI, DAG.getDataLayout(), ValTy, ValValueVTs);
 
   unsigned NumAggValues = AggValueVTs.size();
   unsigned NumValValues = ValValueVTs.size();
   SmallVector<SDValue, 4> Values(NumAggValues);
 
   // Ignore an insertvalue that produces an empty object
   if (!NumAggValues) {
     setValue(&I, DAG.getUNDEF(MVT(MVT::Other)));
void SelectionDAGBuilder::visitExtractValue(const User &I) {
   Type *AggTy = Op0->getType();
   Type *ValTy = I.getType();
   bool OutOfUndef = isa<UndefValue>(Op0);
 
   unsigned LinearIndex = ComputeLinearIndex(AggTy, Indices);
 
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   SmallVector<EVT, 4> ValValueVTs;
   ComputeValueVTs(TLI, DAG.getDataLayout(), ValTy, ValValueVTs);
 
   unsigned NumValValues = ValValueVTs.size();
 
   // Ignore a extractvalue that produces an empty object
   if (!NumValValues) {
     setValue(&I, DAG.getUNDEF(MVT(MVT::Other)));
     return;
   }
void SelectionDAGBuilder::visitLoad(const LoadInst &I) {
   Type *Ty = I.getType();
   Align Alignment = I.getAlign();
 
   AAMDNodes AAInfo;
   I.getAAMetadata(AAInfo);
   const MDNode *Ranges = I.getMetadata(LLVMContext::MD_range);
 
   SmallVector<EVT, 4> ValueVTs, MemVTs;
-  SmallVector<uint64_t, 4> Offsets;
+  SmallVector<TypeSize, 4> Offsets;
   ComputeValueVTs(TLI, DAG.getDataLayout(), Ty, ValueVTs, &MemVTs, &Offsets);
   unsigned NumValues = ValueVTs.size();
   if (NumValues == 0)
     return;
 
   bool isVolatile = I.isVolatile();
 
   SDValue Root;
   bool ConstantMemory = false;
void SelectionDAGBuilder::visitLoad(const LoadInst &I) {
 
   // An aggregate load cannot wrap around the address space, so offsets to its
   // parts don't wrap either.
   SDNodeFlags Flags;
   Flags.setNoUnsignedWrap(true);
 
   SmallVector<SDValue, 4> Values(NumValues);
   SmallVector<SDValue, 4> Chains(std::min(MaxParallelChains, NumValues));
-  EVT PtrVT = Ptr.getValueType();
 
   MachineMemOperand::Flags MMOFlags
     = TLI.getLoadMemOperandFlags(I, DAG.getDataLayout());
 
   unsigned ChainI = 0;
   for (unsigned i = 0; i != NumValues; ++i, ++ChainI) {
     // Serializing loads here may result in excessive register pressure, and
     // TokenFactor places arbitrary choke points on the scheduler. SD scheduling
     // could recover a bit by hoisting nodes upward in the chain by recognizing
     // they are side-effect free or do not alias. The optimizer should really
     // avoid this case by converting large object/array copies to llvm.memcpy
     // (MaxParallelChains should always remain as failsafe).
     if (ChainI == MaxParallelChains) {
       assert(PendingLoads.empty() && "PendingLoads must be serialized first");
       SDValue Chain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other,
                                   makeArrayRef(Chains.data(), ChainI));
       Root = Chain;
       ChainI = 0;
     }
-    SDValue A = DAG.getNode(ISD::ADD, dl,
-                            PtrVT, Ptr,
-                            DAG.getConstant(Offsets[i], dl, PtrVT),
-                            Flags);
+    SDValue A = DAG.getMemBasePlusOffset(Ptr, Offsets[i], dl, Flags);
 
-    SDValue L = DAG.getLoad(MemVTs[i], dl, Root, A,
-                            MachinePointerInfo(SV, Offsets[i]), Alignment,
-                            MMOFlags, AAInfo, Ranges);
+    SDValue L =
+        DAG.getLoad(MemVTs[i], dl, Root, A,
+                    MachinePointerInfo(SV, Offsets[i].getKnownMinSize()),
+                    Alignment, MMOFlags, AAInfo, Ranges);
     Chains[ChainI] = L.getValue(1);
 
     if (MemVTs[i] != ValueVTs[i])
       L = DAG.getZExtOrTrunc(L, dl, ValueVTs[i]);
 
     Values[i] = L;
   }
 
setValue(&I, DAG.getNode(ISD::MERGE_VALUES, dl,
                            DAG.getVTList(ValueVTs), Values));
 }
 
 void SelectionDAGBuilder::visitStoreToSwiftError(const StoreInst &I) {
   assert(DAG.getTargetLoweringInfo().supportSwiftError() &&
          "call visitStoreToSwiftError when backend supports swifterror");
 
   SmallVector<EVT, 4> ValueVTs;
-  SmallVector<uint64_t, 4> Offsets;
+  SmallVector<TypeSize, 4> Offsets;
   const Value *SrcV = I.getOperand(0);
   ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(),
                   SrcV->getType(), ValueVTs, &Offsets);
-  assert(ValueVTs.size() == 1 && Offsets[0] == 0 &&
+  assert(ValueVTs.size() == 1 && Offsets[0].getKnownMinSize() == 0 &&
          "expect a single EVT for swifterror");
 
   SDValue Src = getValue(SrcV);
   // Create a virtual register, then update the virtual register.
   Register VReg =
       SwiftError.getOrCreateVRegDefAt(&I, FuncInfo.MBB, I.getPointerOperand());
   // Chain, DL, Reg, N or Chain, DL, Reg, N, Glue
   // Chain can be getRoot or getControlRoot.
void SelectionDAGBuilder::visitLoadFromSwiftError(const LoadInst &I) {
   assert(
       (!AA ||
        !AA->pointsToConstantMemory(MemoryLocation(
            SV, LocationSize::precise(DAG.getDataLayout().getTypeStoreSize(Ty)),
            AAInfo))) &&
       "load_from_swift_error should not be constant memory");
 
   SmallVector<EVT, 4> ValueVTs;
-  SmallVector<uint64_t, 4> Offsets;
+  SmallVector<TypeSize, 4> Offsets;
   ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(), Ty,
                   ValueVTs, &Offsets);
-  assert(ValueVTs.size() == 1 && Offsets[0] == 0 &&
+  assert(ValueVTs.size() == 1 && Offsets[0].getKnownMinSize() == 0 &&
          "expect a single EVT for swifterror");
 
   // Chain, DL, Reg, VT, Glue or Chain, DL, Reg, VT
   SDValue L = DAG.getCopyFromReg(
       getRoot(), getCurSDLoc(),
       SwiftError.getOrCreateVRegUseAt(&I, FuncInfo.MBB, SV), ValueVTs[0]);
 
   setValue(&I, L);
if (TLI.supportSwiftError()) {
 
     if (const AllocaInst *Alloca = dyn_cast<AllocaInst>(PtrV)) {
       if (Alloca->isSwiftError())
         return visitStoreToSwiftError(I);
     }
   }
 
   SmallVector<EVT, 4> ValueVTs, MemVTs;
-  SmallVector<uint64_t, 4> Offsets;
+  SmallVector<TypeSize, 4> Offsets;
   ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(),
                   SrcV->getType(), ValueVTs, &MemVTs, &Offsets);
   unsigned NumValues = ValueVTs.size();
   if (NumValues == 0)
     return;
 
   // Get the lowered operands. Note that we do this after
   // checking if NumResults is zero, because with zero results
   // the operands won't have values in the map.
void SelectionDAGBuilder::visitStore(const StoreInst &I) {
   for (unsigned i = 0; i != NumValues; ++i, ++ChainI) {
     // See visitLoad comments.
     if (ChainI == MaxParallelChains) {
       SDValue Chain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other,
                                   makeArrayRef(Chains.data(), ChainI));
       Root = Chain;
       ChainI = 0;
     }
-    SDValue Add =
-        DAG.getMemBasePlusOffset(Ptr, TypeSize::Fixed(Offsets[i]), dl, Flags);
+    SDValue Add = DAG.getMemBasePlusOffset(Ptr, Offsets[i], dl, Flags);
     SDValue Val = SDValue(Src.getNode(), Src.getResNo() + i);
     if (MemVTs[i] != ValueVTs[i])
       Val = DAG.getPtrExtOrTrunc(Val, dl, MemVTs[i]);
     SDValue St =
-        DAG.getStore(Root, dl, Val, Add, MachinePointerInfo(PtrV, Offsets[i]),
+        DAG.getStore(Root, dl, Val, Add,
+                     MachinePointerInfo(PtrV, Offsets[i].getKnownMinSize()),
                      Alignment, MMOFlags, AAInfo);
     Chains[ChainI] = St;
   }
 
   SDValue StoreNode = DAG.getNode(ISD::TokenFactor, dl, MVT::Other,
                                   makeArrayRef(Chains.data(), ChainI));
   DAG.setRoot(StoreNode);
 }
if (const ConstantInt *CI = dyn_cast<ConstantInt>(Arg)) {
       Ops.push_back(DAG.getTargetConstant(*CI, SDLoc(), VT));
     } else {
       Ops.push_back(
           DAG.getTargetConstantFP(*cast<ConstantFP>(Arg), SDLoc(), VT));
     }
   }
 
   SmallVector<EVT, 4> ValueVTs;
   ComputeValueVTs(TLI, DAG.getDataLayout(), I.getType(), ValueVTs);
 
   if (HasChain)
     ValueVTs.push_back(MVT::Other);
 
   SDVTList VTs = DAG.getVTList(ValueVTs);
 
   // Create the node.
   SDValue Result;

 }
 
 void SelectionDAGBuilder::visitConstrainedFPIntrinsic(
     const ConstrainedFPIntrinsic &FPI) {
   SDLoc sdl = getCurSDLoc();
 
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   SmallVector<EVT, 4> ValueVTs;
   ComputeValueVTs(TLI, DAG.getDataLayout(), FPI.getType(), ValueVTs);
   ValueVTs.push_back(MVT::Other); // Out chain
 
   // We do not need to serialize constrained FP intrinsics against
   // each other or against (nonvolatile) loads, so they can be
   // chained like loads.
   SDValue Chain = DAG.getRoot();
   SmallVector<SDValue, 4> Opers;
   Opers.push_back(Chain);

 }
 
 void SelectionDAGBuilder::visitVectorPredicationIntrinsic(
     const VPIntrinsic &VPIntrin) {
   unsigned Opcode = getISDForVPIntrinsic(VPIntrin);
 
   SmallVector<EVT, 4> ValueVTs;
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   ComputeValueVTs(TLI, DAG.getDataLayout(), VPIntrin.getType(), ValueVTs);
   SDVTList VTs = DAG.getVTList(ValueVTs);
 
   // Request operands.
   SmallVector<SDValue, 7> OpValues;
   for (int i = 0; i < (int)VPIntrin.getNumArgOperands(); ++i)
     OpValues.push_back(getValue(VPIntrin.getArgOperand(i)));
 
   SDLoc DL = getCurSDLoc();

 void SelectionDAGBuilder::emitInlineAsmError(const CallBase &Call,
                                              const Twine &Message) {
   LLVMContext &Ctx = *DAG.getContext();
   Ctx.emitError(&Call, Message);
 
   // Make sure we leave the DAG in a valid state
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   SmallVector<EVT, 1> ValueVTs;
   ComputeValueVTs(TLI, DAG.getDataLayout(), Call.getType(), ValueVTs);
 
   if (ValueVTs.empty())
     return;
 
   SmallVector<SDValue, 1> Ops;
   for (unsigned i = 0, e = ValueVTs.size(); i != e; ++i)
     Ops.push_back(DAG.getUNDEF(ValueVTs[i]));
 
void SelectionDAGBuilder::visitPatchpoint(const CallBase &CB,
   if (HasGlue)
     Ops.push_back(*(Call->op_end()-1));
 
   SDVTList NodeTys;
   if (IsAnyRegCC && HasDef) {
     // Create the return types based on the intrinsic definition
     const TargetLowering &TLI = DAG.getTargetLoweringInfo();
     SmallVector<EVT, 3> ValueVTs;
     ComputeValueVTs(TLI, DAG.getDataLayout(), CB.getType(), ValueVTs);
     assert(ValueVTs.size() == 1 && "Expected only one return value type.");
 
     // There is always a chain and a glue type at the end
     ValueVTs.push_back(MVT::Other);
     ValueVTs.push_back(MVT::Glue);
     NodeTys = DAG.getVTList(ValueVTs);
   } else
     NodeTys = DAG.getVTList(MVT::Other, MVT::Glue);

 /// FIXME: When all targets are
 /// migrated to using LowerCall, this hook should be integrated into SDISel.
 std::pair<SDValue, SDValue>
 TargetLowering::LowerCallTo(TargetLowering::CallLoweringInfo &CLI) const {
   // Handle the incoming return values from the call.
   CLI.Ins.clear();
   Type *OrigRetTy = CLI.RetTy;
   SmallVector<EVT, 4> RetTys;
-  SmallVector<uint64_t, 4> Offsets;
+  SmallVector<TypeSize, 4> Offsets;
   auto &DL = CLI.DAG.getDataLayout();
   ComputeValueVTs(*this, DL, CLI.RetTy, RetTys, &Offsets);
 
   if (CLI.IsPostTypeLegalization) {
     // If we are lowering a libcall after legalization, split the return type.
     SmallVector<EVT, 4> OldRetTys;
-    SmallVector<uint64_t, 4> OldOffsets;
+    SmallVector<TypeSize, 4> OldOffsets;
     RetTys.swap(OldRetTys);
     Offsets.swap(OldOffsets);
 
     for (size_t i = 0, e = OldRetTys.size(); i != e; ++i) {
       EVT RetVT = OldRetTys[i];
-      uint64_t Offset = OldOffsets[i];
+      TypeSize Offset = OldOffsets[i];
       MVT RegisterVT = getRegisterType(CLI.RetTy->getContext(), RetVT);
       unsigned NumRegs = getNumRegisters(CLI.RetTy->getContext(), RetVT);
       unsigned RegisterVTByteSZ = RegisterVT.getSizeInBits() / 8;
       RetTys.append(NumRegs, RegisterVT);
       for (unsigned j = 0; j != NumRegs; ++j)
-        Offsets.push_back(Offset + j * RegisterVTByteSZ);
+        Offsets.push_back(Offset + TypeSize::Fixed(j * RegisterVTByteSZ));
     }
   }
 
   SmallVector<ISD::OutputArg, 4> Outs;
   GetReturnInfo(CLI.CallConv, CLI.RetTy, getReturnAttrs(CLI), Outs, *this, DL);
 
   bool CanLowerReturn =
       this->CanLowerReturn(CLI.CallConv, CLI.DAG.getMachineFunction(),
#endif
   if (!CanLowerReturn) {
     // The instruction result is the result of loading from the
     // hidden sret parameter.
     SmallVector<EVT, 1> PVTs;
     Type *PtrRetTy = OrigRetTy->getPointerTo(DL.getAllocaAddrSpace());
 
     ComputeValueVTs(*this, DL, PtrRetTy, PVTs);
     assert(PVTs.size() == 1 && "Pointers should fit in one register");
-    EVT PtrVT = PVTs[0];
 
     unsigned NumValues = RetTys.size();
     ReturnValues.resize(NumValues);
     SmallVector<SDValue, 4> Chains(NumValues);
 
     // An aggregate return value cannot wrap around the address space, so
     // offsets to its parts don't wrap either.
     SDNodeFlags Flags;
     Flags.setNoUnsignedWrap(true);
 
     MachineFunction &MF = CLI.DAG.getMachineFunction();
     Align HiddenSRetAlign = MF.getFrameInfo().getObjectAlign(DemoteStackIdx);
     for (unsigned i = 0; i < NumValues; ++i) {
-      SDValue Add = CLI.DAG.getNode(ISD::ADD, CLI.DL, PtrVT, DemoteStackSlot,
-                                    CLI.DAG.getConstant(Offsets[i], CLI.DL,
-                                                        PtrVT), Flags);
-      SDValue L = CLI.DAG.getLoad(
-          RetTys[i], CLI.DL, CLI.Chain, Add,
-          MachinePointerInfo::getFixedStack(CLI.DAG.getMachineFunction(),
-                                            DemoteStackIdx, Offsets[i]),
+      SDValue Add = CLI.DAG.getMemBasePlusOffset(DemoteStackSlot, Offsets[i],
+                                                 CLI.DL, Flags);
+
+      SDValue L =
+          CLI.DAG.getLoad(RetTys[i], CLI.DL, CLI.Chain, Add,
+                          MachinePointerInfo::getFixedStack(
+                              CLI.DAG.getMachineFunction(), DemoteStackIdx,
+                              Offsets[i].getKnownMinSize()),
                           HiddenSRetAlign);
       ReturnValues[i] = L;
       Chains[i] = L.getValue(1);
     }
 
     CLI.Chain = CLI.DAG.getNode(ISD::TokenFactor, CLI.DL, MVT::Other, Chains);
   } else {
     // Collect the legal value parts into potentially illegal values
     // that correspond to the original function's return values.

llvm/lib/IR/DataLayout.cpp

 #include <utility>
 
 using namespace llvm;
 
 //===----------------------------------------------------------------------===//
 // Support for StructLayout
 //===----------------------------------------------------------------------===//
 
-StructLayout::StructLayout(StructType *ST, const DataLayout &DL) {
+StructLayout::StructLayout(StructType *ST, const DataLayout &DL)
+    : StructSize(0, false) {
   assert(!ST->isOpaque() && "Cannot get layout of opaque structs");
-  StructSize = 0;
+  // We permit scalable vector struct as "sized" only when all the elements
+  // are the same scalable vector types.
+  if (ST->isScalableType())
+    StructSize = TypeSize::Scalable(0);
   IsPadded = false;
   NumElements = ST->getNumElements();
 
   // Loop over each of the elements, placing them in memory.
   for (unsigned i = 0, e = NumElements; i != e; ++i) {
     Type *Ty = ST->getElementType(i);
     const Align TyAlign = ST->isPacked() ? Align(1) : DL.getABITypeAlign(Ty);
 
     // Add padding if necessary to align the data element properly.
-    if (!isAligned(TyAlign, StructSize)) {
+    // Scalable vector struct must be the same scalable vector types. They have
+    // no alignment issues.
+    if (!StructSize.isScalable() &&
+        !isAligned(TyAlign, StructSize.getFixedSize())) {
       IsPadded = true;
-      StructSize = alignTo(StructSize, TyAlign);
+      StructSize = TypeSize::Fixed(alignTo(StructSize.getFixedSize(), TyAlign));
     }
 
     // Keep track of maximum alignment constraint.
     StructAlignment = std::max(TyAlign, StructAlignment);
 
-    MemberOffsets[i] = StructSize;
+    MemberOffsets.push_back(StructSize);
     // Consume space for this data item
-    StructSize += DL.getTypeAllocSize(Ty).getFixedValue();
+    StructSize += DL.getTypeAllocSize(Ty);
   }
 
   // Add padding to the end of the struct so that it could be put in an array
   // and all array elements would be aligned correctly.
-  if (!isAligned(StructAlignment, StructSize)) {
+  if (!StructSize.isScalable() &&
+      !isAligned(StructAlignment, StructSize.getFixedSize())) {
     IsPadded = true;
-    StructSize = alignTo(StructSize, StructAlignment);
+    StructSize =
+        TypeSize::Fixed(alignTo(StructSize.getFixedSize(), StructAlignment));
   }
 }
 
 /// getElementContainingOffset - Given a valid offset into the structure,
 /// return the structure index that contains it.
-unsigned StructLayout::getElementContainingOffset(uint64_t Offset) const {
-  const uint64_t *SI =
-    std::upper_bound(&MemberOffsets[0], &MemberOffsets[NumElements], Offset);
-  assert(SI != &MemberOffsets[0] && "Offset not in structure type!");
+unsigned StructLayout::getElementContainingOffset(uint64_t O) const {
+  TypeSize Offset(O, StructSize.isScalable());
+  const TypeSize *SI =
+      std::upper_bound(MemberOffsets.begin(), MemberOffsets.end(), Offset,
+                       [](TypeSize lhs, TypeSize rhs) -> bool {
+                         return TypeSize::isKnownLT(lhs, rhs);
+                       });
+  assert(SI != MemberOffsets.begin() && "Offset not in structure type!");
   --SI;
-  assert(*SI <= Offset && "upper_bound didn't work");
-  assert((SI == &MemberOffsets[0] || *(SI-1) <= Offset) &&
-         (SI+1 == &MemberOffsets[NumElements] || *(SI+1) > Offset) &&
-         "Upper bound didn't work!");
+  assert(TypeSize::isKnownLE(*SI, Offset) && "upper_bound didn't work");
+  assert(
+      (SI == MemberOffsets.begin() || TypeSize::isKnownLE(*(SI - 1), Offset)) &&
+      (SI + 1 == MemberOffsets.end() ||
+       TypeSize::isKnownLT(Offset, *(SI + 1))) &&
 
   // Multiple fields can have the same offset if any of them are zero sized.
   // For example, in { i32, [0 x i32], i32 }, searching for offset 4 will stop
   // at the i32 element, because it is the last element at that offset.  This is
   // the right one to return, because anything after it will have a higher
   // offset, implying that this element is non-empty.
-  return SI-&MemberOffsets[0];
+  return SI - MemberOffsets.begin();
 }
 
 //===----------------------------------------------------------------------===//
 // LayoutAlignElem, LayoutAlign support
 //===----------------------------------------------------------------------===//
 
 LayoutAlignElem LayoutAlignElem::get(AlignTypeEnum align_type, Align abi_align,
                                      Align pref_align, uint32_t bit_width) {
const StructLayout *DataLayout::getStructLayout(StructType *Ty) const {
 
   StructLayoutMap *STM = static_cast<StructLayoutMap*>(LayoutMap);
   StructLayout *&SL = (*STM)[Ty];
   if (SL) return SL;
 
   // Otherwise, create the struct layout.  Because it is variable length, we
   // malloc it, then use placement new.
   int NumElts = Ty->getNumElements();
-  StructLayout *L = (StructLayout *)
-      safe_malloc(sizeof(StructLayout)+(NumElts-1) * sizeof(uint64_t));
+  StructLayout *L = (StructLayout *)safe_malloc(
+      sizeof(StructLayout) + (NumElts - 1) * sizeof(TypeSize));
 
   // Set SL before calling StructLayout's ctor.  The ctor could cause other
   // entries to be added to TheMap, invalidating our reference.
   SL = L;
 
   new (L) StructLayout(Ty, *this);
 
   return L;

llvm/lib/IR/Type.cpp

if (auto *ATy = dyn_cast<ArrayType>(this))
     return ATy->getElementType()->isSized(Visited);
 
   if (auto *VTy = dyn_cast<VectorType>(this))
     return VTy->getElementType()->isSized(Visited);
 
   return cast<StructType>(this)->isSized(Visited);
 }
 
+bool Type::isScalableType() const {
+  if (getTypeID() == ScalableVectorTyID)
+    return true;
+
+  if (const auto *STy = dyn_cast<StructType>(this)) {
+    if (STy->containsScalableVectorType())
+      return true;
+  }
+  return false;
+}
+
 //===----------------------------------------------------------------------===//
 //                          Primitive 'Type' data
 //===----------------------------------------------------------------------===//
 
 Type *Type::getVoidTy(LLVMContext &C) { return &C.pImpl->VoidTy; }
 Type *Type::getLabelTy(LLVMContext &C) { return &C.pImpl->LabelTy; }
 Type *Type::getHalfTy(LLVMContext &C) { return &C.pImpl->HalfTy; }
 Type *Type::getBFloatTy(LLVMContext &C) { return &C.pImpl->BFloatTy; }
if (isOpaque())
     return false;
 
   if (Visited && !Visited->insert(const_cast<StructType*>(this)).second)
     return false;
 
   // Okay, our struct is sized if all of the elements are, but if one of the
   // elements is opaque, the struct isn't sized *yet*, but may become sized in
   // the future, so just bail out without caching.
+  Type *FirstTy = getNumElements() > 0 ? elements()[0] : nullptr;
+  bool IsFirstElementScalable = false;
+  if (FirstTy)
+    IsFirstElementScalable = isa<ScalableVectorType>(FirstTy);
   for (Type *Ty : elements()) {
-    // If the struct contains a scalable vector type, don't consider it sized.
-    // This prevents it from being used in loads/stores/allocas/GEPs.
+    if (IsFirstElementScalable) {
+      // We do not permit mix scalar types with scalable types within struct.
+      if (!isa<ScalableVectorType>(Ty))
+        return false;
+      // All the scalable types within struct should be the same.
+      if (FirstTy != Ty)
+        return false;
+    } else {
       if (isa<ScalableVectorType>(Ty))
         return false;
+    }
     if (!Ty->isSized(Visited))
       return false;
   }
 
   // Here we cheat a bit and cast away const-ness. The goal is to memoize when
   // we find a sized type, as types can only move from opaque to sized, not the
   // other way.
   const_cast<StructType*>(this)->setSubclassData(

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

for (const Argument &Arg : Fn.args()) {
     // We're basically throwing away everything passed into us and starting over
     // to get accurate in-memory offsets. The "PartOffset" is completely useless
     // to us as computed in Ins.
     //
     // We also need to figure out what type legalization is trying to do to get
     // the correct memory offsets.
 
     SmallVector<EVT, 16> ValueVTs;
-    SmallVector<uint64_t, 16> Offsets;
-    ComputeValueVTs(*this, DL, BaseArgTy, ValueVTs, &Offsets, ArgOffset);
+    SmallVector<TypeSize, 16> Offsets;
+    ComputeValueVTs(*this, DL, BaseArgTy, ValueVTs, &Offsets,
+                    TypeSize::Fixed(ArgOffset));
 
     for (unsigned Value = 0, NumValues = ValueVTs.size();
          Value != NumValues; ++Value) {
       uint64_t BasePartOffset = Offsets[Value];
 
       EVT ArgVT = ValueVTs[Value];
       EVT MemVT = ArgVT;
       MVT RegisterVT = getRegisterTypeForCallingConv(Ctx, CC, ArgVT);

llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp

 /// NOTE: This is a band-aid for code that expects ComputeValueVTs to return the
 /// same number of types as the Ins/Outs arrays in LowerFormalArguments,
 /// LowerCall, and LowerReturn.
 static void ComputePTXValueVTs(const TargetLowering &TLI, const DataLayout &DL,
                                Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
                                SmallVectorImpl<uint64_t> *Offsets = nullptr,
                                uint64_t StartingOffset = 0) {
   SmallVector<EVT, 16> TempVTs;
-  SmallVector<uint64_t, 16> TempOffsets;
+  SmallVector<TypeSize, 16> TempOffsets;
 
   // Special case for i128 - decompose to (i64, i64)
   if (Ty->isIntegerTy(128)) {
     ValueVTs.push_back(EVT(MVT::i64));
     ValueVTs.push_back(EVT(MVT::i64));
 
     if (Offsets) {
       Offsets->push_back(StartingOffset + 0);
if (StructType *STy = dyn_cast<StructType>(Ty)) {
     for(auto *EI : STy->elements()) {
       ComputePTXValueVTs(TLI, DL, EI, ValueVTs, Offsets,
                          StartingOffset + SL->getElementOffset(ElementNum));
       ++ElementNum;
     }
     return;
   }
 
-  ComputeValueVTs(TLI, DL, Ty, TempVTs, &TempOffsets, StartingOffset);
+  ComputeValueVTs(TLI, DL, Ty, TempVTs, &TempOffsets,
+                  TypeSize::Fixed(StartingOffset));
   for (unsigned i = 0, e = TempVTs.size(); i != e; ++i) {
     EVT VT = TempVTs[i];
     uint64_t Off = TempOffsets[i];
     // Split vectors into individual elements, except for v2f16, which
     // we will pass as a single scalar.
     if (VT.isVector()) {
       unsigned NumElts = VT.getVectorNumElements();
       EVT EltVT = VT.getVectorElementType();

llvm/lib/Target/X86/X86CallLowering.cpp

bool X86CallLowering::splitToValueTypes(const ArgInfo &OrigArg,
                                         SmallVectorImpl<ArgInfo> &SplitArgs,
                                         const DataLayout &DL,
                                         MachineRegisterInfo &MRI,
                                         SplitArgTy PerformArgSplit) const {
   const X86TargetLowering &TLI = *getTLI<X86TargetLowering>();
   LLVMContext &Context = OrigArg.Ty->getContext();
 
   SmallVector<EVT, 4> SplitVTs;
-  SmallVector<uint64_t, 4> Offsets;
-  ComputeValueVTs(TLI, DL, OrigArg.Ty, SplitVTs, &Offsets, 0);
+  SmallVector<TypeSize, 4> Offsets;
+  ComputeValueVTs(TLI, DL, OrigArg.Ty, SplitVTs, &Offsets, TypeSize::Fixed(0));
   assert(OrigArg.Regs.size() == 1 && "Can't handle multple regs yet");
 
   if (OrigArg.Ty->isVoidTy())
     return true;
 
   EVT VT = SplitVTs[0];
   unsigned NumParts = TLI.getNumRegisters(Context, VT);
 

llvm/lib/Transforms/IPO/ArgumentPromotion.cpp

bool ArgumentPromotionPass::isDenselyPacked(Type *type, const DataLayout &DL) {
 
   // For array types, check for padding within members.
   if (ArrayType *seqTy = dyn_cast<ArrayType>(type))
     return isDenselyPacked(seqTy->getElementType(), DL);
 
   if (!isa<StructType>(type))
     return true;
 
+  // Be conservative for scalable struct.
+  if (type->isScalableType())
+    return false;
+
   // Check for padding within and between elements of a struct.
   StructType *StructTy = cast<StructType>(type);
   const StructLayout *Layout = DL.getStructLayout(StructTy);
   uint64_t StartPos = 0;
   for (unsigned i = 0, E = StructTy->getNumElements(); i < E; ++i) {
     Type *ElTy = StructTy->getElementType(i);
     if (!isDenselyPacked(ElTy, DL))
       return false;

llvm/lib/Transforms/IPO/AttributorAttributes.cpp

if (Offset) {
     // Add 0 index to look through the pointer.
     assert((uint64_t)Offset < DL.getTypeAllocSize(PtrElemTy) &&
            "Offset out of bounds");
     Indices.push_back(Constant::getNullValue(IRB.getInt32Ty()));
 
     Type *Ty = PtrElemTy;
     do {
       auto *STy = dyn_cast<StructType>(Ty);
-      if (!STy)
+      if (!STy || STy->isScalableType())
         // Non-aggregate type, we cast and make byte-wise progress now.
         break;
 
       const StructLayout *SL = DL.getStructLayout(STy);
       if (int64_t(SL->getSizeInBytes()) < Offset)
         break;
 
       uint64_t Idx = SL->getElementContainingOffset(Offset);

llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp

if (NumElements == 1) {
                                                   ".unpack");
       AAMDNodes AAMD;
       LI.getAAMetadata(AAMD);
       NewLoad->setAAMetadata(AAMD);
       return IC.replaceInstUsesWith(LI, IC.Builder.CreateInsertValue(
         UndefValue::get(T), NewLoad, 0, Name));
     }
 
+    if (ST->isScalableType())
+      return nullptr;
+
     // We don't want to break loads with padding here as we'd loose
     // the knowledge that padding exists for the rest of the pipeline.
     const DataLayout &DL = IC.getDataLayout();
     auto *SL = DL.getStructLayout(ST);
     if (SL->hasPadding())
       return nullptr;
 
     const auto Align = LI.getAlign();
if (auto *ST = dyn_cast<StructType>(T)) {
     // If the struct only have one element, we unpack.
     unsigned Count = ST->getNumElements();
     if (Count == 1) {
       V = IC.Builder.CreateExtractValue(V, 0);
       combineStoreToNewValue(IC, SI, V);
       return true;
     }
 
+    if (ST->isScalableType())
+      return false;
+
     // We don't want to break loads with padding here as we'd loose
     // the knowledge that padding exists for the rest of the pipeline.
     const DataLayout &DL = IC.getDataLayout();
     auto *SL = DL.getStructLayout(ST);
     if (SL->hasPadding())
       return false;
 
     const auto Align = SI.getAlign();

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

 /// Given a pointer type and a constant offset, determine whether or not there
 /// is a sequence of GEP indices into the pointed type that will land us at the
 /// specified offset. If so, fill them into NewIndices and return the resultant
 /// element type, otherwise return null.
 Type *
 InstCombinerImpl::FindElementAtOffset(PointerType *PtrTy, int64_t Offset,
                                       SmallVectorImpl<Value *> &NewIndices) {
   Type *Ty = PtrTy->getElementType();
-  if (!Ty->isSized())
+  if (!Ty->isSized() || Ty->isScalableType())
     return nullptr;
 
   // Start with the index over the outer type.  Note that the type size
   // might be zero (even if the offset isn't zero) if the indexed type
   // is something like [0 x {int, int}]
   Type *IndexTy = DL.getIndexType(PtrTy);
   int64_t FirstIdx = 0;
   if (int64_t TySize = DL.getTypeAllocSize(Ty)) {

llvm/lib/Transforms/Scalar/SROA.cpp

if (ArrayType *ArrTy = dyn_cast<ArrayType>(Ty)) {
 
     Offset -= NumSkippedElements * ElementSize;
     Indices.push_back(IRB.getInt(NumSkippedElements));
     return getNaturalGEPRecursively(IRB, DL, Ptr, ElementTy, Offset, TargetTy,
                                     Indices, NamePrefix);
   }
 
   StructType *STy = dyn_cast<StructType>(Ty);
-  if (!STy)
+  if (!STy || STy->isScalableType())
     return nullptr;
 
   const StructLayout *SL = DL.getStructLayout(STy);
   uint64_t StructOffset = Offset.getZExtValue();
   if (StructOffset >= SL->getSizeInBytes())
     return nullptr;
   unsigned Index = SL->getElementContainingOffset(StructOffset);
   Offset -= APInt(Offset.getBitWidth(), SL->getElementOffset(Index));

 ///
 /// This removes no-op aggregate types wrapping an underlying type. It will
 /// strip as many layers of types as it can without changing either the type
 /// size or the allocated size.
 static Type *stripAggregateTypeWrapping(const DataLayout &DL, Type *Ty) {
   if (Ty->isSingleValueType())
     return Ty;
 
-  uint64_t AllocSize = DL.getTypeAllocSize(Ty).getFixedSize();
-  uint64_t TypeSize = DL.getTypeSizeInBits(Ty).getFixedSize();
+  TypeSize AllocSize = DL.getTypeAllocSize(Ty);
+  TypeSize TypeSize = DL.getTypeSizeInBits(Ty);
 
   Type *InnerTy;
   if (ArrayType *ArrTy = dyn_cast<ArrayType>(Ty)) {
     InnerTy = ArrTy->getElementType();
   } else if (StructType *STy = dyn_cast<StructType>(Ty)) {
     const StructLayout *SL = DL.getStructLayout(STy);
     unsigned Index = SL->getElementContainingOffset(0);
     InnerTy = STy->getElementType(Index);
   } else {
     return Ty;
   }
 
-  if (AllocSize > DL.getTypeAllocSize(InnerTy).getFixedSize() ||
-      TypeSize > DL.getTypeSizeInBits(InnerTy).getFixedSize())
+  if (TypeSize::isKnownGT(AllocSize, DL.getTypeAllocSize(InnerTy)) ||
+      TypeSize::isKnownGT(TypeSize, DL.getTypeSizeInBits(InnerTy)))
     return Ty;
 
   return stripAggregateTypeWrapping(DL, InnerTy);
 }
 
 /// Try to find a partition of the aggregate type passed in for a given
 /// offset and size.
 ///
if (isa<ArrayType>(Ty) || isa<VectorType>(Ty)) {
     assert(Size > ElementSize);
     uint64_t NumElements = Size / ElementSize;
     if (NumElements * ElementSize != Size)
       return nullptr;
     return ArrayType::get(ElementTy, NumElements);
   }
 
   StructType *STy = dyn_cast<StructType>(Ty);
-  if (!STy)
+  if (!STy || STy->isScalableType())
     return nullptr;
 
   const StructLayout *SL = DL.getStructLayout(STy);
   if (Offset >= SL->getSizeInBytes())
     return nullptr;
   uint64_t EndOffset = Offset + Size;
   if (EndOffset > SL->getSizeInBytes())
     return nullptr;

llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp

for (;;) {
     while (StructType *STy = dyn_cast<StructType>(ElTy)) {
       bool FoundFieldNo = false;
       // An empty struct has no fields.
       if (STy->getNumElements() == 0) break;
       // Field offsets are known. See if a constant offset falls within any of
       // the struct fields.
       if (Ops.empty())
         break;
+      // Be conservative for scalable struct.
+      if (STy->isScalableType())
+        break;
       if (const SCEVConstant *C = dyn_cast<SCEVConstant>(Ops[0]))
         if (SE.getTypeSizeInBits(C->getType()) <= 64) {
           const StructLayout &SL = *DL.getStructLayout(STy);
           uint64_t FullOffset = C->getValue()->getZExtValue();
           if (FullOffset < SL.getSizeInBytes()) {
             unsigned ElIdx = SL.getElementContainingOffset(FullOffset);
             GepIndices.push_back(
                 ConstantInt::get(Type::getInt32Ty(Ty->getContext()), ElIdx));

llvm/test/CodeGen/RISCV/rvv/load-store-scalable-struct.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv64 -mattr=+m,+d,+experimental-v -verify-machineinstrs \
+; RUN:   --riscv-no-aliases < %s | FileCheck %s
+
+target triple = "riscv64-unknown-unknown-elf"
+
+%struct.test = type { <vscale x 1 x double>, <vscale x 1 x double> }
+
+define <vscale x 1 x double> @test(%struct.test* %addr, i64 %vl) {
+; CHECK-LABEL: test:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    addi sp, sp, -16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    csrrs a2, vlenb, zero
+; CHECK-NEXT:    slli a2, a2, 1
+; CHECK-NEXT:    sub sp, sp, a2
+; CHECK-NEXT:    csrrs a2, vlenb, zero
+; CHECK-NEXT:    add a3, a0, a2
+; CHECK-NEXT:    vl1re64.v v25, (a3)
+; CHECK-NEXT:    vl1re64.v v26, (a0)
+; CHECK-NEXT:    addi a0, sp, 16
+; CHECK-NEXT:    add a0, a0, a2
+; CHECK-NEXT:    vs1r.v v25, (a0)
+; CHECK-NEXT:    addi a2, sp, 16
+; CHECK-NEXT:    vs1r.v v26, (a2)
+; CHECK-NEXT:    vl1re64.v v25, (a0)
+; CHECK-NEXT:    addi a0, sp, 16
+; CHECK-NEXT:    vl1re64.v v26, (a0)
+; CHECK-NEXT:    vsetvli a0, a1, e64,m1,ta,mu
+; CHECK-NEXT:    vfadd.vv v8, v26, v25
+; CHECK-NEXT:    csrrs a0, vlenb, zero
+; CHECK-NEXT:    slli a0, a0, 1
+; CHECK-NEXT:    add sp, sp, a0
+; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    jalr zero, 0(ra)
+entry:
+  %ret = alloca %struct.test, align 8
+  %val = load %struct.test, %struct.test* %addr
+  store %struct.test %val, %struct.test* %ret, align 8
+  %0 = load %struct.test, %struct.test* %ret, align 8
+  %1 = extractvalue %struct.test %0, 0
+  %2 = extractvalue %struct.test %0, 1
+  %3 = call <vscale x 1 x double> @llvm.riscv.vfadd.nxv1f64.nxv1f64(
+    <vscale x 1 x double> %1, <vscale x 1 x double> %2, i64 %vl)
+  ret <vscale x 1 x double> %3
+}
+
+declare <vscale x 1 x double> @llvm.riscv.vfadd.nxv1f64.nxv1f64(
+  <vscale x 1 x double>,
+  <vscale x 1 x double>,
+  i64);

llvm/test/Other/load-scalable-vector-struct.ll

+; RUN: opt -S -verify < %s 2>&1 | FileCheck %s
+
+%struct.test = type { <vscale x 1 x i32>, <vscale x 1 x i32> }
+
+define <vscale x 1 x i32> @load(%struct.test* %x) {
+; CHECK: %a = load %struct.test, %struct.test* %x, align 4
+; CHECK: %b = extractvalue %struct.test %a, 1
+; CHECK: ret <vscale x 1 x i32> %b
+  %a = load %struct.test, %struct.test* %x
+  %b = extractvalue %struct.test %a, 1
+  ret <vscale x 1 x i32> %b
+}

llvm/test/Other/store-scalable-vector-struct.ll

+; RUN: opt -S -verify < %s 2>&1 | FileCheck %s
+
+%struct.test = type { <vscale x 1 x i32>, <vscale x 1 x i32> }
+
+define void @store(%struct.test* %x, <vscale x 1 x i32> %y, <vscale x 1 x i32> %z) {
+; CHECK: %a = insertvalue %struct.test undef, <vscale x 1 x i32> %y, 0
+; CHECK: %b = insertvalue %struct.test %a, <vscale x 1 x i32> %z, 1
+; CHECK: store %struct.test %b, %struct.test* %x
+; CHECK: ret void
+  %a = insertvalue %struct.test undef, <vscale x 1 x i32> %y, 0
+  %b = insertvalue %struct.test %a, <vscale x 1 x i32> %z, 1
+  store %struct.test %b, %struct.test* %x
+  ret void
+}