[1/11][IR] Permit load/store/alloca for struct of the same scalable vector type

[eopXD]

Summary

This patch-set aims to simplify the existing RVV segment load/store
intrinsics to use a type that represents a tuple of vectors instead.

To achieve this, first we need to relax the current limitation for an
aggregate type to be a target of load/store/alloca when the aggregate
type contains homogeneous scalable vector types. Then to adjust the
prolog of an LLVM function during lowering to clang. Finally we
re-define the RVV segment load/store intrinsics to use the tuple types.

The pull request under the RVV intrinsic specification is
riscv-non-isa/rvv-intrinsic-doc#198

---

This is the 1st patch of the patch-set. This patch is originated from
D98169.

This patch allows aggregate type (StructType) that contains homogeneous
scalable vector types to be a target of load/store/alloca. The RFC of
this patch was posted in LLVM Discourse.

https://discourse.llvm.org/t/rfc-ir-permit-load-store-alloca-for-struct-of-the-same-scalable-vector-type/69527

The main changes in this patch are:

Extend StructLayout::StructSize from uint64_t to TypeSize to
accommodate an expression of scalable size.

Allow StructType:isSized to also return true for homogeneous
scalable vector types.

Let Type::isScalableTy return true when Type is StructType
and contains scalable vectors

Extra description is added in the LLVM Language Reference Manual on the
relaxation of this patch.

Authored-by: Hsiangkai Wang <[e-mail]>
Co-Authored-by: eop Chen <[e-mail]>

[nikic]

isSized() should return true for these structs. They are sized for our definition of sized.

Is there an RFC for this on discourse?

[eopXD]

In D146872#4221495, @nikic wrote:

Is there an RFC for this on discourse?

Hi @nikic,

Thanks for the swift comment, just opened up an RFC (https://discourse.llvm.org/t/rfc-ir-permit-load-store-alloca-for-struct-of-the-same-scalable-vector-type/69527).
Hope this time we can explore a way to have RVV tuple types.

Regards,

eop Chen

[kito-cheng]

I think you also need to update LangRef.rst at least this sentence : https://github.com/llvm/llvm-project/blob/main/llvm/docs/LangRef.rst#L738

:ref:`Scalable vectors <t_vector>` cannot be global variables or members of
arrays because their size is unknown at compile time. They are allowed in
structs to facilitate intrinsics returning multiple values. Structs containing
scalable vectors cannot be used in loads, stores, allocas, or GEPs.

[craig.topper]

llvm/include/llvm/IR/DataLayout.h

636

This is a weird sentence. A "caller" is a function in the code base, it doesn't have a size.

"Should only be called for structs with scalable size"

llvm/include/llvm/IR/DerivedTypes.h

291

Missing period at end of sentence

292

isContainHomogeneousScalableVectorType -> containsHomogenousScalableVectorTypes.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6966 ↗

(On Diff #508307)

Why is this needed?

llvm/lib/IR/DataLayout.cpp

51

You already checked containsScalableVectorType in the if above. You don't need to check it in the ?: too

63–69

Use "homogenous scalable vector types" here instead of "tuple types"

llvm/lib/IR/Type.cpp

628

As @nikic said. We need to return true for homogenous scalable vector structs. I put this check in when I allowed structs containing scalable vectors for intrinsic return values. See https://reviews.llvm.org/D94142

[eopXD]

Changes:

• Address comments from Craig
  • Let isSize return true for homogeneous scalable vector structs.
  • Function rename isContainHomogeneousScalableVectorType -> containsHomogenousScalableVectorTypes
• Fix return value for DataLayout::getTypeSizeInBits
• Let Type::isScalableTy return true for StructType with scalable vectors

[eopXD]

Also addressed comments from Kito:

• Added description in Language Reference Manual for the relaxation added in this patch.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6966 ↗

(On Diff #508307)

No this is not needed here. The original intent was to return base when a scalable structure was queried here.

[nikic]

Generally, please re-review the whole patch and remove special cases that are no longer needed now that isSized() returns the correct value.

llvm/include/llvm/IR/DataLayout.h

630

Why not make these methods return TypeSize instead (and drop the separate getMinSize APIs)?

675

Assertion can be restored.

llvm/lib/AsmParser/LLParser.cpp

7627

These changes should no longer be necessary now that isSized() is fixed.

llvm/lib/IR/Type.cpp

622

Debugging leftover.

[eopXD]

Change: Remove conditions of checking containsHomogeneousScalableVectorTypes since now isSized admits homogeneous scalable vector structs

[eopXD]

Changes:

• Address comments from Nikic
  • Restore assertion
  • Return TypeSize for getSizeInBytes and getSizeInBits.
• Fix check under containsHomogeneousScalableVectorTypes

[eopXD]

Prune unnecessary changes.

[eopXD]

Recover original code under Verifier.cpp.

[craig.topper]

llvm/docs/LangRef.rst

747–754

Need to work the exception into this sentence. Right now its an absolute statement and I wouldn't expect an exception if I didn't read the next paragraph.

May add "with one exception described below" to the end of the sentence?

756

Don't capitalize "Structs".

759

Don't capitalize "Structs"

10281

Maybe "Structs containing scalable vectors cannot be used in allocas unless all fields are the same scalable vector type"?

10282

Same

llvm/include/llvm/CodeGen/Analysis.h

75

Why the last field called Offset here but StartingOffset in the other two signatures?

llvm/lib/CodeGen/Analysis.cpp

96

TypeSize::get(0, StartingOffset.isScalable())

135

TypeSize::get(Offset, Ty->isScalableTy())

186

This assert contradicts the comment on the next line. Scalable vectors are fine if Offsets is null

llvm/lib/CodeGen/CodeGenPrepare.cpp

7684 ↗

(On Diff #511088)

Is this information we can get from DL.getTypeSize? Not sure when StructLayout is created and whether that would be cheaper than walking every field of a struct looking for scalable vectors.

llvm/lib/IR/DataLayout.cpp

94

The Caller is not a structure.

[eopXD]

Address comments from Craig. Comment resolution are described under the comment threads.

llvm/docs/LangRef.rst

747–754

Merged the exception into the same paragraph so users can be aware of this.

756

The existing paragraph uses "Structs" instead of "structs".

Structs containing scalable vectors cannot be used in loads, stores, allocas, or GEPs.

Un-capitalized it.

llvm/lib/CodeGen/Analysis.cpp

186

Removed assertion.

llvm/lib/IR/DataLayout.cpp

94

Caller under SROA is ArrayType.
Caller under ConsantFolding, ModuleSummaryAnalysis, TypeMetadataUtils is ConstantStruct.
Caller under Target is StructType, but by grep-ing I don't find user of the function LLVMElementAtOffset.

I say it is safe to remove the assertion here.

[craig.topper]

llvm/lib/IR/DataLayout.cpp

94

I was referring to a word usage problem. The word caller refers to the function that contains the code calls this method on an object. The caller is not the same thing as the object this method acts on.

[eopXD]

Recover assertion and adjust wording.

llvm/lib/IR/DataLayout.cpp

94

Recovered the method and adjusted the wording.

[eopXD]

Resolve test failure - LLVM.tools/llvm-as::slow-ptrtoint.ll

• Add memoize when containsHomogeneousScalableVectorTypes() == true under StructType::isSized.
• Add memoize for StructType::containsScalableVectorType.

[nikic]

llvm/docs/LangRef.rst

752

The types should probably be wrapped in double backticks.

llvm/include/llvm/IR/DerivedTypes.h

223

NotContain

292

definition of

llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

160

Use isScalableTy()?

llvm/lib/IR/DataLayout.cpp

63–69

double "be the"

llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp

527

Can this be an assertion? As this is related to GEP handling.

[eopXD]

Address comments from Nikic.

[nikic]

It looks like there is a test failure in sme-aarch64-svcount.ll now.

We should also have a verifier test that checks that these are rejected in GEPs. We should also test the InstCombine bailouts (I think you can just take your "Other" tests and move them to InstCombine and run -passes=instcombine instead of -passes=verify. Those should cover the relevant code paths.)

[craig.topper]

llvm/docs/LangRef.rst

753

"can used" -> "can be used"

llvm/lib/IR/Type.cpp

466

If this structure is opaque, elements will be empty and we would set the SCDB_NotContainScalableVector flag. That seems wrong since it could gain scalable vector types if it becomes non-opaque.

[eopXD]

Address comments from Nikic and Craig.

• Bail out when visiting extractvalue under InstructionCombining.cpp, added test case llvm/test/Transforms/InstCombine/scalable-vector-struct-extractvalue.ll
• Check for GEP with scalable vector under LLParser.cpp, added test case llvm/test/Verifier/scalable-vector-struct-gep.ll
• Removed assertion in GetElementPtrInst::GetElementPtrInst under Instructions.h, the verifier will take case of this check.
• Fix grammatical error under LangRef.rst.
• Adjust test case sme-aarch64-svcount.ll based on change under FunctionLoweringInfo.cpp

llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

160

Changing this to checking Ty->isScalableTy() results in changes the code generation of llvm/test/CodeGen/AArch64/sme-aarch64-svcount.ll. The change makes sense to me, but the code generation is worse here, @sdesmalen may you confirm if this is reasonable?

[nikic]

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

2976

This looks too aggressive -- is it enough to only guard the LoadInst code below?

[eopXD]

Apply Nikic's suggestion to bail out when aggregate type contains scalable vector type and used in a load instruction.

[eopXD]

Adjust checking sequence under LLParser.cpp to resolve test case failure for LLVM.Assembler::unsized-recursive-type.ll.

[craig.topper]

llvm/include/llvm/IR/DerivedTypes.h

222

Contain->Contains

llvm/lib/IR/Type.cpp

481

is -> are

[eopXD]

Address comments from Craig.

[eopXD]

Minor change.

[eopXD]

Ping, thank you.

[eopXD]

Ping again, thank you.

[craig.topper]

llvm/lib/Analysis/ScalarEvolution.cpp

3768

Maybe just check in getOffsetOfExpr where we implicitly cast getElementOffset to uint64_t? We just need to make sure the TypeSize from getElementOffset isn't scalable.

llvm/lib/IR/DataLayout.cpp

49

Would StructSize be default constructed to TypeSize::Fixed(0) anyway?

50–51

Since we only support homogenous scalable structs. I wonder if we could changes StructSize to scalable on the first iteration of the below loop if the first type comes back scalable? That would save us needing to walk the entire struct hierachy separately here.

Though it gets memoized so maybe not a big deal.

94

"Cannot get element at offset for structure containing scalable vector types"

llvm/lib/IR/Type.cpp

639

If the element is a struct containing scalable vectors, should we no longer be considered sized? That would mean we have a nested struct.

llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp

768

Could we check the size from the SL below is not scalable?

1296

Same question

llvm/lib/Transforms/Scalar/SROA.cpp

3979

Can we check the size of the StructLayout?

llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp

526

"GEPs are not supported on structures containing scalable vectors"

[eopXD]

Address comments from Craig.

llvm/lib/Analysis/ScalarEvolution.cpp

3768

Moved the assertion to check assert(!SL->getSizeInBits().isScalable() under ScalarEvolution::getOffsetOfExpr.

llvm/lib/IR/DataLayout.cpp

49

The ctors of TypeSize are:

constexpr TypeSize(ScalarTy Quantity, bool Scalable)
    : FixedOrScalableQuantity(Quantity, Scalable) {}

static constexpr TypeSize getFixed(ScalarTy ExactSize) {
  return TypeSize(ExactSize, false);
}
static constexpr TypeSize getScalable(ScalarTy MinimumSize) {
  return TypeSize(MinimumSize, true);
}
static constexpr TypeSize get(ScalarTy Quantity, bool Scalable) {
  return TypeSize(Quantity, Scalable);
}
static constexpr TypeSize Fixed(ScalarTy ExactSize) {
  return TypeSize(ExactSize, false);
}
static constexpr TypeSize Scalable(ScalarTy MinimumSize) {
  return TypeSize(MinimumSize, true);
}

Am I using it incorrectly here?

50–51

Though calling Type::isScalableTy still results in calling containsScalableVectorType, adjusted the code as requested anyway.

llvm/lib/IR/Type.cpp

639

Good point. Nested structs containing scalable vector will not be consider sized. Added extra check to guard this.

[craig.topper]

llvm/lib/IR/DataLayout.cpp

49

You're right. There is no default constructor. Though it's base class has one.

llvm/lib/IR/Type.cpp

638

Could we call isScalableTy instead of isa<ScalableVectorType>(Ty) above?

llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp

767

Does SL->getSizeInBits().isScalable() below make this unnecessary?

[eopXD]

llvm/lib/IR/DataLayout.cpp

49

Marking this thread as done.

llvm/lib/IR/Type.cpp

638

Yes you are correct. Corrected the code.

llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp

767

Sorry, forgot to delete this one. Deleted.

[eopXD]

Address comments from Craig.

[craig.topper]

llvm/lib/IR/Type.cpp

635

Doesn't checking isScalableTy() make this unnecessary?

[eopXD]

Address comments from Craig, also removed infinite loop bug in containsScalableVectorType.

llvm/lib/IR/Type.cpp

635

Yes you are correct. Furthermore calling isScalableTy uncovered a bug that I forgot to record visited Type that would result in infinite loop case for recursively-defined types.

[craig.topper]

LGTM

[eopXD]

@nikic Is there any more concern on landing this?

[eopXD]

Rebase to latest main.

[nikic]

llvm/docs/LangRef.rst

750

contains -> contain

752

kind -> kinds

llvm/include/llvm/IR/DerivedTypes.h

223

Contain -> Contains

294

a false -> false

llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

155–158

structure -> structs
vector -> vectors

llvm/lib/IR/Type.cpp

620

This has no relation to opaque types.

llvm/test/Other/load-scalable-vector-struct.ll

3

Move these two tests into llvm/test/Assembler and combine them.

llvm/test/Transforms/InstCombine/scalable-vector-struct-extractvalue.ll

3

You've added three bailouts in InstCombine -- does this test case cover all three of them?

Also, don't we need a test case for the check in SROA as well?

[eopXD]

Address comments from Nikic.

• Fixed grammatical errors.
• Merged llvm/test/Other/store-scalable-vector-struct.ll and llvm/test/Other/load-scalable-vector-struct.ll into llvm/test/Assembler/scalable-vector-struct.ll
• Added store situation test case for llvm/test/Transforms/InstCombine/scalable-vector-struct.ll. The two test case should cover the three checks under load/store/extractvalue.
• Added test case llvm/test/Transforms/SROA/scalable-vector-struct.ll

[nikic]

LGTM

[eopXD]

Thank you Nikic and Craig for the patient reviewals :)

[zixuan-wu]

llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

160

Changing this to checking Ty->isScalableTy() results in changes the code generation of llvm/test/CodeGen/AArch64/sme-aarch64-svcount.ll. The change makes sense to me, but the code generation is worse here, @sdesmalen may you confirm if this is reasonable?

Hi. As Ty->isScalableTy() contains target scalable type, it extends the semantic. So it's not only for scalable vector type. I think we can pass the ty as argument to distinguish them in target TFI, then getStackIDForScalableVectors would be better to be named 'getStackIDForScalableTy'

[eopXD]

llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

160

Thank you for checking this out. I will come back to revisit this when I finished implementing the other RVV intrinsic features for v1.0.

[paulwalker-arm]

llvm/test/CodeGen/RISCV/rvv/alloca-load-store-scalable-struct.ll

19–20

Apologies for any naivety here but is the code generation here correct? I'm trying to figure out the patch's objective because it looks like it has enable support within the IR but ignored code generation. My interpretation of this output is that whilst it correctly allocates the vscale relative stack space:

csrrs a2, vlenb, zero
slli a2, a2, 1
sub sp, sp, a2

it then simply falls back to fixed length offsets when accessing it:

addi a2, a0, 8
vl1re64.v v8, (a2)

Is my interpretation here correct? or have I misunderstood something?

[paulwalker-arm]

llvm/test/CodeGen/RISCV/rvv/alloca-load-store-scalable-struct.ll

19–20

I've pushed a test for SVE under https://reviews.llvm.org/rGac2a7637fe74 which shows similar behaviour. So now I'm pretty sure this is just broken. I'm working on a fix that I'll post later, probably Monday.

llvm/docs/LangRef.rst

 an optional :ref:`global attributes <glattrs>` and
 an optional list of attached :ref:`metadata <metadata>`.
 
 Variables and aliases can have a
 :ref:`Thread Local Storage Model <tls_model>`.
 
 :ref:`Scalable vectors <t_vector>` cannot be global variables or members of
 arrays because their size is unknown at compile time. They are allowed in
-structs to facilitate intrinsics returning multiple values. Structs containing
-scalable vectors cannot be used in loads, stores, allocas, or GEPs.
+structs to facilitate intrinsics returning multiple values. Generally, structs
+containing scalable vectors are not considered "sized" and cannot be used in
+loads, stores, allocas, or GEPs. The only exception to this rule is for structs
+that contains scalable vectors of the same type (e.g. ``{<vscale x 2 x i32>,
+<vscale x 2 x i32>}`` contains the same type while ``{<vscale x 2 x i32>,
+<vscale x 2 x i64>}`` doesn't). These kind of structs (we may call them
+homogeneous scalable vector structs) are considered sized and can be used in
+loads, stores, allocas, but not GEPs.
 
 Syntax::
 
       @<GlobalVarName> = [Linkage] [PreemptionSpecifier] [Visibility]
                          [DLLStorageClass] [ThreadLocal]
                          [(unnamed_addr|local_unnamed_addr)] [AddrSpace]
                          [ExternallyInitialized]
                          <global | constant> <Type> [<InitializerConstant>]
                          [, section "name"] [, partition "name"]
                          [, comdat [($name)]] [, align <Alignment>]
                          [, no_sanitize_address] [, no_sanitize_hwaddress]
                          [, sanitize_address_dyninit] [, sanitize_memtag]
                          (, !name !N)*

 alignment may not be greater than ``1 << 32``.
 
 The alignment is only optional when parsing textual IR; for in-memory IR,
 it is always present. If not specified, the target can choose to align the
 allocation on any convenient boundary compatible with the type.
 
 '``type``' may be any sized type.
 
+Structs containing scalable vectors cannot be used in allocas unless all
+fields are the same scalable vector type (e.g. ``{<vscale x 2 x i32>,
+<vscale x 2 x i32>}`` contains the same type while ``{<vscale x 2 x i32>,
+<vscale x 2 x i64>}`` doesn't).
+
 Semantics:
 """"""""""
 
 Memory is allocated; a pointer is returned. The allocated memory is
 uninitialized, and loading from uninitialized memory produces an undefined
 value. The operation itself is undefined if there is insufficient stack
 space for the allocation.'``alloca``'d memory is automatically released
 when the function returns. The '``alloca``' instruction is commonly used

llvm/include/llvm/CodeGen/Analysis.h

 /// EVTs that represent all the individual underlying
 /// non-aggregate types that comprise it.
 ///
 /// If Offsets is non-null, it points to a vector to be filled in
 /// with the in-memory offsets of each of the individual values.
 ///
 void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
                      SmallVectorImpl<EVT> &ValueVTs,
-                     SmallVectorImpl<uint64_t> *Offsets = nullptr,
+                     SmallVectorImpl<TypeSize> *Offsets,
+                     TypeSize StartingOffset);
+void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
+                     SmallVectorImpl<EVT> &ValueVTs,
+                     SmallVectorImpl<TypeSize> *Offsets = nullptr,
                      uint64_t StartingOffset = 0);
+void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
+                     SmallVectorImpl<EVT> &ValueVTs,
+                     SmallVectorImpl<uint64_t> *FixedOffsets,
+                     uint64_t StartingOffset);
 
 /// Variant of ComputeValueVTs that also produces the memory VTs.
 void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
                      SmallVectorImpl<EVT> &ValueVTs,
                      SmallVectorImpl<EVT> *MemVTs,
-                     SmallVectorImpl<uint64_t> *Offsets = nullptr,
+                     SmallVectorImpl<TypeSize> *Offsets,
+                     TypeSize StartingOffset);
+void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
+                     SmallVectorImpl<EVT> &ValueVTs,
+                     SmallVectorImpl<EVT> *MemVTs,
+                     SmallVectorImpl<TypeSize> *Offsets = nullptr,
                      uint64_t StartingOffset = 0);
+void ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL, Type *Ty,
+                     SmallVectorImpl<EVT> &ValueVTs,
+                     SmallVectorImpl<EVT> *MemVTs,
+                     SmallVectorImpl<uint64_t> *FixedOffsets,
+                     uint64_t StartingOffset);
 
 /// computeValueLLTs - Given an LLVM IR type, compute a sequence of
 /// LLTs that represent all the individual underlying
 /// non-aggregate types that comprise it.
 ///
 /// If Offsets is non-null, it points to a vector to be filled in
 /// with the in-memory offsets of each of the individual values.
 ///

llvm/include/llvm/IR/DataLayout.h

 }
 
 inline LLVMTargetDataRef wrap(const DataLayout *P) {
   return reinterpret_cast<LLVMTargetDataRef>(const_cast<DataLayout *>(P));
 }
 
 /// Used to lazily calculate structure layout information for a target machine,
 /// based on the DataLayout structure.
-class StructLayout final : public TrailingObjects<StructLayout, uint64_t> {
-  uint64_t StructSize;
+class StructLayout final : public TrailingObjects<StructLayout, TypeSize> {
+  TypeSize StructSize;
   Align StructAlignment;
   unsigned IsPadded : 1;
   unsigned NumElements : 31;
 
 public:
-  uint64_t getSizeInBytes() const { return StructSize; }
+  TypeSize getSizeInBytes() const { return StructSize; }
 
-  uint64_t getSizeInBits() const { return 8 * StructSize; }
+  TypeSize getSizeInBits() const { return 8 * StructSize; }
 
   Align getAlignment() const { return StructAlignment; }
 
   /// Returns whether the struct has padding or not between its fields.
   /// NB: Padding in nested element is not taken into account.
   bool hasPadding() const { return IsPadded; }
 
   /// Given a valid byte offset into the structure, returns the structure
   /// index that contains it.
-  unsigned getElementContainingOffset(uint64_t Offset) const;
+  unsigned getElementContainingOffset(uint64_t FixedOffset) const;
 
-  MutableArrayRef<uint64_t> getMemberOffsets() {
-    return llvm::MutableArrayRef(getTrailingObjects<uint64_t>(),
-                                     NumElements);
+  MutableArrayRef<TypeSize> getMemberOffsets() {
+    return llvm::MutableArrayRef(getTrailingObjects<TypeSize>(), NumElements);
   }
 
-  ArrayRef<uint64_t> getMemberOffsets() const {
-    return llvm::ArrayRef(getTrailingObjects<uint64_t>(), NumElements);
+  ArrayRef<TypeSize> getMemberOffsets() const {
+    return llvm::ArrayRef(getTrailingObjects<TypeSize>(), NumElements);
   }
 
-  uint64_t getElementOffset(unsigned Idx) const {
+  TypeSize getElementOffset(unsigned Idx) const {
     assert(Idx < NumElements && "Invalid element idx!");
     return getMemberOffsets()[Idx];
   }
 
-  uint64_t getElementOffsetInBits(unsigned Idx) const {
+  TypeSize getElementOffsetInBits(unsigned Idx) const {
     return getElementOffset(Idx) * 8;
   }
 
 private:
   friend class DataLayout; // Only DataLayout can create this class
 
   StructLayout(StructType *ST, const DataLayout &DL);
 
-  size_t numTrailingObjects(OverloadToken<uint64_t>) const {
+  size_t numTrailingObjects(OverloadToken<TypeSize>) const {
     return NumElements;
   }
 };
 
 // The implementation of this method is provided inline as it is particularly
 // well suited to constant folding when called on a specific Type subclass.
 inline TypeSize DataLayout::getTypeSizeInBits(Type *Ty) const {
   assert(Ty->isSized() && "Cannot getTypeInfo() on a type that is unsized!");
   switch (Ty->getTypeID()) {
   case Type::LabelTyID:
     return TypeSize::Fixed(getPointerSizeInBits(0));
   case Type::PointerTyID:
     return TypeSize::Fixed(getPointerSizeInBits(Ty->getPointerAddressSpace()));
   case Type::ArrayTyID: {
     ArrayType *ATy = cast<ArrayType>(Ty);
     return ATy->getNumElements() *
            getTypeAllocSizeInBits(ATy->getElementType());
   }
   case Type::StructTyID:
     // Get the layout annotation... which is lazily created on demand.
-    return TypeSize::Fixed(
-                        getStructLayout(cast<StructType>(Ty))->getSizeInBits());
+    return getStructLayout(cast<StructType>(Ty))->getSizeInBits();
   case Type::IntegerTyID:
     return TypeSize::Fixed(Ty->getIntegerBitWidth());
   case Type::HalfTyID:
   case Type::BFloatTyID:
     return TypeSize::Fixed(16);
   case Type::FloatTyID:
     return TypeSize::Fixed(32);
   case Type::DoubleTyID:

llvm/include/llvm/IR/DerivedTypes.h

 class StructType : public Type {
   StructType(LLVMContext &C) : Type(C, StructTyID) {}
 
   enum {
     /// This is the contents of the SubClassData field.
     SCDB_HasBody = 1,
     SCDB_Packed = 2,
     SCDB_IsLiteral = 4,
-    SCDB_IsSized = 8
+    SCDB_IsSized = 8,
+    SCDB_ContainsScalableVector = 16,
+    SCDB_NotContainScalableVector = 32
   };
 
   /// For a named struct that actually has a name, this is a pointer to the
   /// symbol table entry (maintained by LLVMContext) for the struct.
   /// This is null if the type is an literal struct or if it is a identified
   /// type that has an empty name.
   void *SymbolTableEntry = nullptr;
 
public:
   bool isOpaque() const { return (getSubclassData() & SCDB_HasBody) == 0; }
 
   /// isSized - Return true if this is a sized type.
   bool isSized(SmallPtrSetImpl<Type *> *Visited = nullptr) const;
 
   /// Returns true if this struct contains a scalable vector.
   bool containsScalableVectorType() const;
 
+  /// Returns true if this struct contains homogeneous scalable vector types.
+  /// Note that the definition of homogeneous scalable vector type is not
+  /// recursive here. That means the following structure will return a false
+  /// when calling this function.
+  /// {{<vscale x 2 x i32>, <vscale x 4 x i64>},
+  ///  {<vscale x 2 x i32>, <vscale x 4 x i64>}}
+  bool containsHomogeneousScalableVectorTypes() const;
+
   /// Return true if this is a named struct that has a non-empty name.
   bool hasName() const { return SymbolTableEntry != nullptr; }
 
   /// Return the name for this struct type if it has an identity.
   /// This may return an empty string for an unnamed struct type.  Do not call
   /// this on an literal type.
   StringRef getName() const;
 

llvm/include/llvm/IR/Type.h

public:
   /// Return true if this is a target extension type.
   bool isTargetExtTy() const { return getTypeID() == TargetExtTyID; }
 
   /// Return true if this is a target extension type with a scalable layout.
   bool isScalableTargetExtTy() const;
 
   /// Return true if this is a scalable vector type or a target extension type
   /// with a scalable layout.
-  bool isScalableTy() const {
-    return getTypeID() == ScalableVectorTyID || isScalableTargetExtTy();
-  }
+  bool isScalableTy() const;
 
   /// Return true if this is a FP type or a vector of FP.
   bool isFPOrFPVectorTy() const { return getScalarType()->isFloatingPointTy(); }
 
   /// Return true if this is 'label'.
   bool isLabelTy() const { return getTypeID() == LabelTyID; }
 
   /// Return true if this is 'metadata'.

llvm/lib/Analysis/ScalarEvolution.cpp

SCEV::NoWrapFlags OffsetWrap =
     AssumeInBoundsFlags ? SCEV::FlagNSW : SCEV::FlagAnyWrap;
 
   Type *CurTy = GEP->getType();
   bool FirstIter = true;
   SmallVector<const SCEV *, 4> Offsets;
   for (const SCEV *IndexExpr : IndexExprs) {
     // Compute the (potentially symbolic) offset in bytes for this index.
     if (StructType *STy = dyn_cast<StructType>(CurTy)) {
       // For a struct, add the member offset.
       ConstantInt *Index = cast<SCEVConstant>(IndexExpr)->getValue();
       unsigned FieldNo = Index->getZExtValue();
       const SCEV *FieldOffset = getOffsetOfExpr(IntIdxTy, STy, FieldNo);
       Offsets.push_back(FieldOffset);
 
       // Update CurTy to the type of the field at Index.
       CurTy = STy->getTypeAtIndex(Index);
     } else {

 }
 
 const SCEV *ScalarEvolution::getOffsetOfExpr(Type *IntTy,
                                              StructType *STy,
                                              unsigned FieldNo) {
   // We can bypass creating a target-independent constant expression and then
   // folding it back into a ConstantInt. This is just a compile-time
   // optimization.
-  return getConstant(
-      IntTy, getDataLayout().getStructLayout(STy)->getElementOffset(FieldNo));
+  const StructLayout *SL = getDataLayout().getStructLayout(STy);
+  assert(!SL->getSizeInBits().isScalable() &&
+         "Cannot get offset for structure containing scalable vector types");
+  return getConstant(IntTy, SL->getElementOffset(FieldNo));
 }
 
 const SCEV *ScalarEvolution::getUnknown(Value *V) {
   // Don't attempt to do anything other than create a SCEVUnknown object
   // here.  createSCEV only calls getUnknown after checking for all other
   // interesting possibilities, and any other code that calls getUnknown
   // is doing so in order to hide a value from SCEV canonicalization.
 

llvm/lib/AsmParser/LLParser.cpp

int LLParser::parseAlloc(Instruction *&Inst, PerFunctionState &PFS) {
   if (Size && !Size->getType()->isIntegerTy())
     return error(SizeLoc, "element count must have integer type");
 
   SmallPtrSet<Type *, 4> Visited;
   if (!Alignment && !Ty->isSized(&Visited))
     return error(TyLoc, "Cannot allocate unsized type");
   if (!Alignment)
     Alignment = M->getDataLayout().getPrefTypeAlign(Ty);
   AllocaInst *AI = new AllocaInst(Ty, AddrSpace, Size, *Alignment);
   AI->setUsedWithInAlloca(IsInAlloca);
   AI->setSwiftError(IsSwiftError);
   Inst = AI;
   return AteExtraComma ? InstExtraComma : InstNormal;
 }
 
 /// parseLoad
 ///   ::= 'load' 'volatile'? TypeAndValue (',' 'align' i32)?
while (EatIfPresent(lltok::comma)) {
     }
     Indices.push_back(Val);
   }
 
   SmallPtrSet<Type*, 4> Visited;
   if (!Indices.empty() && !Ty->isSized(&Visited))
     return error(Loc, "base element of getelementptr must be sized");
 
+  auto *STy = dyn_cast<StructType>(Ty);
+  if (STy && STy->containsScalableVectorType())
+    return error(Loc, "getelementptr cannot target structure that contains "
+                      "scalable vector type");
+
   if (!GetElementPtrInst::getIndexedType(Ty, Indices))
     return error(Loc, "invalid getelementptr indices");
   Inst = GetElementPtrInst::Create(Ty, Ptr, Indices);
   if (InBounds)
     cast<GetElementPtrInst>(Inst)->setIsInBounds(true);
   return AteExtraComma ? InstExtraComma : InstNormal;
 }
 

llvm/lib/CodeGen/Analysis.cpp

 /// non-aggregate types that comprise it.
 ///
 /// If Offsets is non-null, it points to a vector to be filled in
 /// with the in-memory offsets of each of the individual values.
 ///
 void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
                            Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
                            SmallVectorImpl<EVT> *MemVTs,
-                           SmallVectorImpl<uint64_t> *Offsets,
-                           uint64_t StartingOffset) {
+                           SmallVectorImpl<TypeSize> *Offsets,
+                           TypeSize StartingOffset) {
   // Given a struct type, recursively traverse the elements.
   if (StructType *STy = dyn_cast<StructType>(Ty)) {
     // If the Offsets aren't needed, don't query the struct layout. This allows
     // us to support structs with scalable vectors for operations that don't
     // need offsets.
     const StructLayout *SL = Offsets ? DL.getStructLayout(STy) : nullptr;
     for (StructType::element_iterator EB = STy->element_begin(),
                                       EI = EB,
                                       EE = STy->element_end();
          EI != EE; ++EI) {
       // Don't compute the element offset if we didn't get a StructLayout above.
-      uint64_t EltOffset = SL ? SL->getElementOffset(EI - EB) : 0;
+      TypeSize EltOffset = SL ? SL->getElementOffset(EI - EB)
+                              : TypeSize::get(0, StartingOffset.isScalable());
       ComputeValueVTs(TLI, DL, *EI, ValueVTs, MemVTs, Offsets,
                       StartingOffset + EltOffset);
     }
     return;
   }
   // Given an array type, recursively traverse the elements.
   if (ArrayType *ATy = dyn_cast<ArrayType>(Ty)) {
     Type *EltTy = ATy->getElementType();
-    uint64_t EltSize = DL.getTypeAllocSize(EltTy).getFixedValue();
+    TypeSize EltSize = DL.getTypeAllocSize(EltTy);
     for (unsigned i = 0, e = ATy->getNumElements(); i != e; ++i)
       ComputeValueVTs(TLI, DL, EltTy, ValueVTs, MemVTs, Offsets,
                       StartingOffset + i * EltSize);
     return;
   }
   // Interpret void as zero return values.
   if (Ty->isVoidTy())
     return;
   // Base case: we can get an EVT for this LLVM IR type.
   ValueVTs.push_back(TLI.getValueType(DL, Ty));
   if (MemVTs)
     MemVTs->push_back(TLI.getMemValueType(DL, Ty));
   if (Offsets)
     Offsets->push_back(StartingOffset);
 }
 
 void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
                            Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
-                           SmallVectorImpl<uint64_t> *Offsets,
-                           uint64_t StartingOffset) {
+                           SmallVectorImpl<TypeSize> *Offsets,
+                           TypeSize StartingOffset) {
   return ComputeValueVTs(TLI, DL, Ty, ValueVTs, /*MemVTs=*/nullptr, Offsets,
                          StartingOffset);
 }
 
+void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
+                           Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
+                           SmallVectorImpl<TypeSize> *Offsets,
+                           uint64_t StartingOffset) {
+  TypeSize Offset = TypeSize::get(StartingOffset, Ty->isScalableTy());
+  return ComputeValueVTs(TLI, DL, Ty, ValueVTs, Offsets, Offset);
+}
+
+void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
+                           Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
+                           SmallVectorImpl<uint64_t> *FixedOffsets,
+                           uint64_t StartingOffset) {
+  TypeSize Offset = TypeSize::get(StartingOffset, Ty->isScalableTy());
+  SmallVector<TypeSize, 4> Offsets;
+  if (FixedOffsets)
+    ComputeValueVTs(TLI, DL, Ty, ValueVTs, &Offsets, Offset);
+  else
+    ComputeValueVTs(TLI, DL, Ty, ValueVTs, nullptr, Offset);
+
+  if (FixedOffsets)
+    for (TypeSize Offset : Offsets)
+      FixedOffsets->push_back(Offset.getKnownMinValue());
+}
+
+void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
+                           Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
+                           SmallVectorImpl<EVT> *MemVTs,
+                           SmallVectorImpl<TypeSize> *Offsets,
+                           uint64_t StartingOffset) {
+  TypeSize Offset = TypeSize::get(StartingOffset, Ty->isScalableTy());
+  return ComputeValueVTs(TLI, DL, Ty, ValueVTs, MemVTs, Offsets, Offset);
+}
+
+void llvm::ComputeValueVTs(const TargetLowering &TLI, const DataLayout &DL,
+                           Type *Ty, SmallVectorImpl<EVT> &ValueVTs,
+                           SmallVectorImpl<EVT> *MemVTs,
+                           SmallVectorImpl<uint64_t> *FixedOffsets,
+                           uint64_t StartingOffset) {
+  TypeSize Offset = TypeSize::get(StartingOffset, Ty->isScalableTy());
+  SmallVector<TypeSize, 4> Offsets;
+  if (FixedOffsets)
+    ComputeValueVTs(TLI, DL, Ty, ValueVTs, MemVTs, &Offsets, Offset);
+  else
+    ComputeValueVTs(TLI, DL, Ty, ValueVTs, MemVTs, nullptr, Offset);
+
+  if (FixedOffsets)
+    for (TypeSize Offset : Offsets)
+      FixedOffsets->push_back(Offset.getKnownMinValue());
+}
+
 void llvm::computeValueLLTs(const DataLayout &DL, Type &Ty,
                             SmallVectorImpl<LLT> &ValueTys,
                             SmallVectorImpl<uint64_t> *Offsets,
                             uint64_t StartingOffset) {
   // Given a struct type, recursively traverse the elements.
   if (StructType *STy = dyn_cast<StructType>(&Ty)) {
     // If the Offsets aren't needed, don't query the struct layout. This allows
     // us to support structs with scalable vectors for operations that don't
     // need offsets.
     const StructLayout *SL = Offsets ? DL.getStructLayout(STy) : nullptr;
     for (unsigned I = 0, E = STy->getNumElements(); I != E; ++I) {
       uint64_t EltOffset = SL ? SL->getElementOffset(I) : 0;
       computeValueLLTs(DL, *STy->getElementType(I), ValueTys, Offsets,
                        StartingOffset + EltOffset);
     }

llvm/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp

for (const Instruction &I : BB) {
             FrameIndex = MF->getFrameInfo().CreateFixedObject(
                 TySize, 0, /*IsImmutable=*/false, /*isAliased=*/true);
             MF->getFrameInfo().setObjectAlignment(FrameIndex, Alignment);
           } else {
             FrameIndex = MF->getFrameInfo().CreateStackObject(TySize, Alignment,
                                                               false, AI);
           }
 
-          // Scalable vectors may need a special StackID to distinguish
-          // them from other (fixed size) stack objects.
-          if (isa<ScalableVectorType>(Ty))
+          // Scalable vectors and structure that contains scalable vector may
+          // need a special StackID to distinguish them from other (fixed size)
+          // stack objects.
+          if (Ty->isScalableTy())
             MF->getFrameInfo().setStackID(FrameIndex,
                                           TFI->getStackIDForScalableVectors());
 
           StaticAllocaMap[AI] = FrameIndex;
           // Update the catch handler information.
           if (Iter != CatchObjects.end()) {
             for (int *CatchObjPtr : Iter->second)
               *CatchObjPtr = FrameIndex;
           }
         } else {

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (!FuncInfo.CanLowerReturn) {
 
     SDValue RetPtr =
         DAG.getCopyFromReg(Chain, getCurSDLoc(), DemoteReg, PtrValueVTs[0]);
     SDValue RetOp = getValue(I.getOperand(0));
 
     SmallVector<EVT, 4> ValueVTs, MemVTs;
     SmallVector<uint64_t, 4> Offsets;
     ComputeValueVTs(TLI, DL, I.getOperand(0)->getType(), ValueVTs, &MemVTs,
-                    &Offsets);
+                    &Offsets, 0);
     unsigned NumValues = ValueVTs.size();
 
     SmallVector<SDValue, 4> Chains(NumValues);
     Align BaseAlign = DL.getPrefTypeAlign(I.getOperand(0)->getType());
     for (unsigned i = 0; i != NumValues; ++i) {
       // An aggregate return value cannot wrap around the address space, so
       // offsets to its parts don't wrap either.
       SDValue Ptr = DAG.getObjectPtrOffset(getCurSDLoc(), RetPtr,
if (TLI.supportSwiftError()) {
     }
   }
 
   SDValue Ptr = getValue(SV);
 
   Type *Ty = I.getType();
   SmallVector<EVT, 4> ValueVTs, MemVTs;
   SmallVector<uint64_t, 4> Offsets;
-  ComputeValueVTs(TLI, DAG.getDataLayout(), Ty, ValueVTs, &MemVTs, &Offsets);
+  ComputeValueVTs(TLI, DAG.getDataLayout(), Ty, ValueVTs, &MemVTs, &Offsets, 0);
   unsigned NumValues = ValueVTs.size();
   if (NumValues == 0)
     return;
 
   Align Alignment = I.getAlign();
   AAMDNodes AAInfo = I.getAAMetadata();
   const MDNode *Ranges = I.getMetadata(LLVMContext::MD_range);
   bool isVolatile = I.isVolatile();

 void SelectionDAGBuilder::visitStoreToSwiftError(const StoreInst &I) {
   assert(DAG.getTargetLoweringInfo().supportSwiftError() &&
          "call visitStoreToSwiftError when backend supports swifterror");
 
   SmallVector<EVT, 4> ValueVTs;
   SmallVector<uint64_t, 4> Offsets;
   const Value *SrcV = I.getOperand(0);
   ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(),
-                  SrcV->getType(), ValueVTs, &Offsets);
+                  SrcV->getType(), ValueVTs, &Offsets, 0);
   assert(ValueVTs.size() == 1 && Offsets[0] == 0 &&
          "expect a single EVT for swifterror");
 
   SDValue Src = getValue(SrcV);
   // Create a virtual register, then update the virtual register.
   Register VReg =
       SwiftError.getOrCreateVRegDefAt(&I, FuncInfo.MBB, I.getPointerOperand());
   // Chain, DL, Reg, N or Chain, DL, Reg, N, Glue
assert(
        !AA->pointsToConstantMemory(MemoryLocation(
            SV, LocationSize::precise(DAG.getDataLayout().getTypeStoreSize(Ty)),
            I.getAAMetadata()))) &&
       "load_from_swift_error should not be constant memory");
 
   SmallVector<EVT, 4> ValueVTs;
   SmallVector<uint64_t, 4> Offsets;
   ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(), Ty,
-                  ValueVTs, &Offsets);
+                  ValueVTs, &Offsets, 0);
   assert(ValueVTs.size() == 1 && Offsets[0] == 0 &&
          "expect a single EVT for swifterror");
 
   // Chain, DL, Reg, VT, Glue or Chain, DL, Reg, VT
   SDValue L = DAG.getCopyFromReg(
       getRoot(), getCurSDLoc(),
       SwiftError.getOrCreateVRegUseAt(&I, FuncInfo.MBB, SV), ValueVTs[0]);
 
if (const AllocaInst *Alloca = dyn_cast<AllocaInst>(PtrV)) {
       if (Alloca->isSwiftError())
         return visitStoreToSwiftError(I);
     }
   }
 
   SmallVector<EVT, 4> ValueVTs, MemVTs;
   SmallVector<uint64_t, 4> Offsets;
   ComputeValueVTs(DAG.getTargetLoweringInfo(), DAG.getDataLayout(),
-                  SrcV->getType(), ValueVTs, &MemVTs, &Offsets);
+                  SrcV->getType(), ValueVTs, &MemVTs, &Offsets, 0);
   unsigned NumValues = ValueVTs.size();
   if (NumValues == 0)
     return;
 
   // Get the lowered operands. Note that we do this after
   // checking if NumResults is zero, because with zero results
   // the operands won't have values in the map.
   SDValue Src = getValue(SrcV);

 std::pair<SDValue, SDValue>
 TargetLowering::LowerCallTo(TargetLowering::CallLoweringInfo &CLI) const {
   // Handle the incoming return values from the call.
   CLI.Ins.clear();
   Type *OrigRetTy = CLI.RetTy;
   SmallVector<EVT, 4> RetTys;
   SmallVector<uint64_t, 4> Offsets;
   auto &DL = CLI.DAG.getDataLayout();
-  ComputeValueVTs(*this, DL, CLI.RetTy, RetTys, &Offsets);
+  ComputeValueVTs(*this, DL, CLI.RetTy, RetTys, &Offsets, 0);
 
   if (CLI.IsPostTypeLegalization) {
     // If we are lowering a libcall after legalization, split the return type.
     SmallVector<EVT, 4> OldRetTys;
     SmallVector<uint64_t, 4> OldOffsets;
     RetTys.swap(OldRetTys);
     Offsets.swap(OldOffsets);
 

llvm/lib/IR/DataLayout.cpp

 #include <utility>
 
 using namespace llvm;
 
 //===----------------------------------------------------------------------===//
 // Support for StructLayout
 //===----------------------------------------------------------------------===//
 
-StructLayout::StructLayout(StructType *ST, const DataLayout &DL) {
+StructLayout::StructLayout(StructType *ST, const DataLayout &DL)
+    : StructSize(TypeSize::Fixed(0)) {
   assert(!ST->isOpaque() && "Cannot get layout of opaque structs");
-  StructSize = 0;
   IsPadded = false;
   NumElements = ST->getNumElements();
 
   // Loop over each of the elements, placing them in memory.
   for (unsigned i = 0, e = NumElements; i != e; ++i) {
     Type *Ty = ST->getElementType(i);
+    if (i == 0 && Ty->isScalableTy())
+      StructSize = TypeSize::Scalable(0);
+
     const Align TyAlign = ST->isPacked() ? Align(1) : DL.getABITypeAlign(Ty);
 
     // Add padding if necessary to align the data element properly.
-    if (!isAligned(TyAlign, StructSize)) {
+    // Currently the only structure with scalable size will be the homogeneous
+    // scalable vector types. Homogeneous scalable vector types have members of
+    // the same data type so no alignment issue will happen. The condition here
+    // assumes so and needs to be adjusted if this assumption changes (e.g. we
+    // support structures with arbitrary scalable data type, or structure that
+    // contains both fixed size and scalable size data type members).
+    if (!StructSize.isScalable() && !isAligned(TyAlign, StructSize)) {
       IsPadded = true;
-      StructSize = alignTo(StructSize, TyAlign);
+      StructSize = TypeSize::Fixed(alignTo(StructSize, TyAlign));
     }
 
     // Keep track of maximum alignment constraint.
     StructAlignment = std::max(TyAlign, StructAlignment);
 
     getMemberOffsets()[i] = StructSize;
     // Consume space for this data item
-    StructSize += DL.getTypeAllocSize(Ty).getFixedValue();
+    StructSize += DL.getTypeAllocSize(Ty);
   }
 
   // Add padding to the end of the struct so that it could be put in an array
   // and all array elements would be aligned correctly.
-  if (!isAligned(StructAlignment, StructSize)) {
+  if (!StructSize.isScalable() && !isAligned(StructAlignment, StructSize)) {
     IsPadded = true;
-    StructSize = alignTo(StructSize, StructAlignment);
+    StructSize = TypeSize::Fixed(alignTo(StructSize, StructAlignment));
   }
 }
 
 /// getElementContainingOffset - Given a valid offset into the structure,
 /// return the structure index that contains it.
-unsigned StructLayout::getElementContainingOffset(uint64_t Offset) const {
-  ArrayRef<uint64_t> MemberOffsets = getMemberOffsets();
-  auto SI = llvm::upper_bound(MemberOffsets, Offset);
+unsigned StructLayout::getElementContainingOffset(uint64_t FixedOffset) const {
+  assert(!StructSize.isScalable() &&
+         "Cannot get element at offset for structure containing scalable "
+         "vector types");
+  TypeSize Offset = TypeSize::Fixed(FixedOffset);
+  ArrayRef<TypeSize> MemberOffsets = getMemberOffsets();
+
+  const auto *SI =
+      std::upper_bound(MemberOffsets.begin(), MemberOffsets.end(), Offset,
+                       [](TypeSize LHS, TypeSize RHS) -> bool {
+                         return TypeSize::isKnownLT(LHS, RHS);
+                       });
   assert(SI != MemberOffsets.begin() && "Offset not in structure type!");
   --SI;
-  assert(*SI <= Offset && "upper_bound didn't work");
-  assert((SI == MemberOffsets.begin() || *(SI - 1) <= Offset) &&
-         (SI + 1 == MemberOffsets.end() || *(SI + 1) > Offset) &&
-         "Upper bound didn't work!");
+  assert(TypeSize::isKnownLE(*SI, Offset) && "upper_bound didn't work");
+  assert(
+      (SI == MemberOffsets.begin() || TypeSize::isKnownLE(*(SI - 1), Offset)) &&
+      (SI + 1 == MemberOffsets.end() ||
+       TypeSize::isKnownGT(*(SI + 1), Offset)) &&
 
   // Multiple fields can have the same offset if any of them are zero sized.
   // For example, in { i32, [0 x i32], i32 }, searching for offset 4 will stop
   // at the i32 element, because it is the last element at that offset.  This is
   // the right one to return, because anything after it will have a higher
   // offset, implying that this element is non-empty.
   return SI - MemberOffsets.begin();
 }
const StructLayout *DataLayout::getStructLayout(StructType *Ty) const {
 
   StructLayoutMap *STM = static_cast<StructLayoutMap*>(LayoutMap);
   StructLayout *&SL = (*STM)[Ty];
   if (SL) return SL;
 
   // Otherwise, create the struct layout.  Because it is variable length, we
   // malloc it, then use placement new.
   StructLayout *L = (StructLayout *)safe_malloc(
-      StructLayout::totalSizeToAlloc<uint64_t>(Ty->getNumElements()));
+      StructLayout::totalSizeToAlloc<TypeSize>(Ty->getNumElements()));
 
   // Set SL before calling StructLayout's ctor.  The ctor could cause other
   // entries to be added to TheMap, invalidating our reference.
   SL = L;
 
   new (L) StructLayout(Ty, *this);
 
   return L;

llvm/lib/IR/Type.cpp

 }
 
 bool Type::isOpaquePointerTy() const {
   if (auto *PTy = dyn_cast<PointerType>(this))
     return PTy->isOpaque();
   return false;
 }
 
+bool Type::isScalableTy() const {
+  if (const auto *STy = dyn_cast<StructType>(this))
+    return STy->containsScalableVectorType();
+  return getTypeID() == ScalableVectorTyID || isScalableTargetExtTy();
+}
+
 const fltSemantics &Type::getFltSemantics() const {
   switch (getTypeID()) {
   case HalfTyID: return APFloat::IEEEhalf();
   case BFloatTyID: return APFloat::BFloat();
   case FloatTyID: return APFloat::IEEEsingle();
   case DoubleTyID: return APFloat::IEEEdouble();
   case X86_FP80TyID: return APFloat::x87DoubleExtended();
   case FP128TyID: return APFloat::IEEEquad();
if (Insertion.second) {
     // The struct type was found. Just return it.
     ST = *Insertion.first;
   }
 
   return ST;
 }
 
 bool StructType::containsScalableVectorType() const {
+  if ((getSubclassData() & SCDB_ContainsScalableVector) != 0)
+    return true;
+
+  if ((getSubclassData() & SCDB_NotContainScalableVector) != 0)
+    return false;
+
   for (Type *Ty : elements()) {
-    if (isa<ScalableVectorType>(Ty))
+    if (isa<ScalableVectorType>(Ty)) {
+      const_cast<StructType *>(this)->setSubclassData(
+          getSubclassData() | SCDB_ContainsScalableVector);
       return true;
-    if (auto *STy = dyn_cast<StructType>(Ty))
-      if (STy->containsScalableVectorType())
+    }
+    if (auto *STy = dyn_cast<StructType>(Ty)) {
+      if (STy->containsScalableVectorType()) {
+        const_cast<StructType *>(this)->setSubclassData(
+            getSubclassData() | SCDB_ContainsScalableVector);
         return true;
       }
+    }
+  }
+
+  // For structures that are opaque, return false but do not set the
+  // SCDB_NotContainScalableVector flag since it may gain scalable vector type
+  // when it becomes non-opaque.
+  if (!isOpaque())
+    const_cast<StructType *>(this)->setSubclassData(
+        getSubclassData() | SCDB_NotContainScalableVector);
+  return false;
+}
 
+bool StructType::containsHomogeneousScalableVectorTypes() const {
+  Type *FirstTy = getNumElements() > 0 ? elements()[0] : nullptr;
+  if (!FirstTy || !isa<ScalableVectorType>(FirstTy))
     return false;
+  for (Type *Ty : elements())
+    if (Ty != FirstTy)
+      return false;
+  return true;
 }
 
 void StructType::setBody(ArrayRef<Type*> Elements, bool isPacked) {
   assert(isOpaque() && "Struct body already set!");
 
   setSubclassData(getSubclassData() | SCDB_HasBody);
   if (isPacked)
     setSubclassData(getSubclassData() | SCDB_Packed);
if (isOpaque())
     return false;
 
   if (Visited && !Visited->insert(const_cast<StructType*>(this)).second)
     return false;
 
   // Okay, our struct is sized if all of the elements are, but if one of the
   // elements is opaque, the struct isn't sized *yet*, but may become sized in
   // the future, so just bail out without caching.
+  // The ONLY special case of opaque types inside a Struct that is considered
+  // sized is when the elements are homogeneous of a scalable vector type.
+  if (containsHomogeneousScalableVectorTypes()) {
+    const_cast<StructType *>(this)->setSubclassData(getSubclassData() |
+                                                    SCDB_IsSized);
+    return true;
+  }
   for (Type *Ty : elements()) {
     // If the struct contains a scalable vector type, don't consider it sized.
-    // This prevents it from being used in loads/stores/allocas/GEPs.
-    if (isa<ScalableVectorType>(Ty))
+    // This prevents it from being used in loads/stores/allocas/GEPs. The ONLY
+    // special case right now is a structure of homogenous scalable vector
+    // types and is handled by the if-statement before this for-loop.
+    // If the element is a struct containing scalable vectors, it should not be
+    // considered sized even if it contains homogeneous scalable vector tyoes.
+    if (Ty->isScalableTy())
+      return false;
+    if (StructType *STy = dyn_cast<StructType>(Ty);
+        STy && containsHomogeneousScalableVectorTypes())
       return false;
     if (!Ty->isSized(Visited))
       return false;
   }
 
   // Here we cheat a bit and cast away const-ness. The goal is to memoize when
   // we find a sized type, as types can only move from opaque to sized, not the
   // other way.
   const_cast<StructType*>(this)->setSubclassData(
     getSubclassData() | SCDB_IsSized);
   return true;

llvm/lib/IR/Verifier.cpp

 
 void Verifier::visitGetElementPtrInst(GetElementPtrInst &GEP) {
   Type *TargetTy = GEP.getPointerOperandType()->getScalarType();
 
   Check(isa<PointerType>(TargetTy),
         "GEP base pointer is not a vector or a vector of pointers", &GEP);
   Check(GEP.getSourceElementType()->isSized(), "GEP into unsized type!", &GEP);
 
+  if (auto *STy = dyn_cast<StructType>(GEP.getSourceElementType()))
+    Check(!STy->containsScalableVectorType(),
+          "getelementptr cannot target structure that contains scalable vector"
+          "type",
+          &GEP);
+
   SmallVector<Value *, 16> Idxs(GEP.indices());
   Check(
       all_of(Idxs, [](Value *V) { return V->getType()->isIntOrIntVectorTy(); }),
       "GEP indexes must be integers", &GEP);
   Type *ElTy =
       GetElementPtrInst::getIndexedType(GEP.getSourceElementType(), Idxs);
   Check(ElTy, "Invalid indices for GEP pointer type!", &GEP);
 

llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp

if (auto *ST = dyn_cast<StructType>(T)) {
     if (NumElements == 1) {
       LoadInst *NewLoad = IC.combineLoadToNewType(LI, ST->getTypeAtIndex(0U),
                                                   ".unpack");
       NewLoad->setAAMetadata(LI.getAAMetadata());
       return IC.replaceInstUsesWith(LI, IC.Builder.CreateInsertValue(
         PoisonValue::get(T), NewLoad, 0, Name));
     }
 
     // We don't want to break loads with padding here as we'd loose
     // the knowledge that padding exists for the rest of the pipeline.
     const DataLayout &DL = IC.getDataLayout();
     auto *SL = DL.getStructLayout(ST);
+
+    // Don't unpack for structure with scalable vector.
+    if (SL->getSizeInBits().isScalable())
+      return nullptr;
+
     if (SL->hasPadding())
       return nullptr;
 
     const auto Align = LI.getAlign();
     auto *Addr = LI.getPointerOperand();
     auto *IdxType = Type::getInt32Ty(T->getContext());
     auto *Zero = ConstantInt::get(IdxType, 0);
 
if (auto *ST = dyn_cast<StructType>(T)) {
     // If the struct only have one element, we unpack.
     unsigned Count = ST->getNumElements();
     if (Count == 1) {
       V = IC.Builder.CreateExtractValue(V, 0);
       combineStoreToNewValue(IC, SI, V);
       return true;
     }
 
     // We don't want to break loads with padding here as we'd loose
     // the knowledge that padding exists for the rest of the pipeline.
     const DataLayout &DL = IC.getDataLayout();
     auto *SL = DL.getStructLayout(ST);
+
+    // Don't unpack for structure with scalable vector.
+    if (SL->getSizeInBits().isScalable())
+      return false;
+
     if (SL->hasPadding())
       return false;
 
     const auto Align = SI.getAlign();
 
     SmallString<16> EltName = V->getName();
     EltName += ".elt";
     auto *Addr = SI.getPointerOperand();

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

 Instruction *InstCombinerImpl::visitExtractValueInst(ExtractValueInst &EV) {
   Value *Agg = EV.getAggregateOperand();
 
   if (!EV.hasIndices())
     return replaceInstUsesWith(EV, Agg);
 
   if (Value *V = simplifyExtractValueInst(Agg, EV.getIndices(),
                                           SQ.getWithInstruction(&EV)))
     return replaceInstUsesWith(EV, V);
 
   if (InsertValueInst *IV = dyn_cast<InsertValueInst>(Agg)) {
     // We're extracting from an insertvalue instruction, compare the indices
     const unsigned *exti, *exte, *insi, *inse;
     for (exti = EV.idx_begin(), insi = IV->idx_begin(),
          exte = EV.idx_end(), inse = IV->idx_end();
          exti != exte && insi != inse;
          ++exti, ++insi) {
if (insi == inse)
       return ExtractValueInst::Create(IV->getInsertedValueOperand(),
                                       ArrayRef(exti, exte));
   }
 
   if (Instruction *R = foldExtractOfOverflowIntrinsic(EV))
     return R;
 
   if (LoadInst *L = dyn_cast<LoadInst>(Agg)) {
+    // Bail out if the aggregate contains scalable vector type
+    if (auto *STy = dyn_cast<StructType>(Agg->getType());
+        STy && STy->containsScalableVectorType())
+      return nullptr;
+
     // If the (non-volatile) load only has one use, we can rewrite this to a
     // load from a GEP. This reduces the size of the load. If a load is used
     // only by extractvalue instructions then this either must have been
     // optimized before, or it is a struct with padding, in which case we
     // don't want to do the transformation as it loses padding knowledge.
     if (L->isSimple() && L->hasOneUse()) {
       // extractvalue has integer indices, getelementptr has Value*s. Convert.
       SmallVector<Value*, 4> Indices;

llvm/lib/Transforms/Scalar/SROA.cpp

if (NumElements * ElementSize != Size)
       return nullptr;
     return ArrayType::get(ElementTy, NumElements);
   }
 
   StructType *STy = dyn_cast<StructType>(Ty);
   if (!STy)
     return nullptr;
 
   const StructLayout *SL = DL.getStructLayout(STy);
+
+  if (SL->getSizeInBits().isScalable())
+    return nullptr;
+
   if (Offset >= SL->getSizeInBytes())
     return nullptr;
   uint64_t EndOffset = Offset + Size;
   if (EndOffset > SL->getSizeInBytes())
     return nullptr;
 
   unsigned Index = SL->getElementContainingOffset(Offset);
   Offset -= SL->getElementOffset(Index);

llvm/lib/Transforms/Utils/ScalarEvolutionExpander.cpp

for (;;) {
       while (StructType *STy = dyn_cast<StructType>(ElTy)) {
         bool FoundFieldNo = false;
         // An empty struct has no fields.
         if (STy->getNumElements() == 0) break;
         // Field offsets are known. See if a constant offset falls within any of
         // the struct fields.
         if (Ops.empty())
           break;
+        assert(
+            !STy->containsScalableVectorType() &&
+            "GEPs are not supported on structures containing scalable vectors");
         if (const SCEVConstant *C = dyn_cast<SCEVConstant>(Ops[0]))
           if (SE.getTypeSizeInBits(C->getType()) <= 64) {
             const StructLayout &SL = *DL.getStructLayout(STy);
             uint64_t FullOffset = C->getValue()->getZExtValue();
             if (FullOffset < SL.getSizeInBytes()) {
               unsigned ElIdx = SL.getElementContainingOffset(FullOffset);
               GepIndices.push_back(
                   ConstantInt::get(Type::getInt32Ty(Ty->getContext()), ElIdx));

llvm/test/CodeGen/AArch64/sme-aarch64-svcount.ll

 ; CHECKO3:       // %bb.0:
 ; CHECKO3-NEXT:    sub sp, sp, #16
 ; CHECKO3-NEXT:    add x8, sp, #14
 ; CHECKO3-NEXT:    str p0, [x8]
 ; CHECKO3-NEXT:    add sp, sp, #16
 ; CHECKO3-NEXT:    ret
 ; CHECK-O0-LABEL: test_alloca_store_reload:
 ; CHECK-O0:       // %bb.0:
-; CHECK-O0-NEXT:    sub sp, sp, #16
-; CHECK-O0-NEXT:    add x8, sp, #14
-; CHECK-O0-NEXT:    str p0, [x8]
-; CHECK-O0-NEXT:    ldr p0, [x8]
-; CHECK-O0-NEXT:    add sp, sp, #16
+; CHECK-O0-NEXT:    str x29, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-O0-NEXT:    addvl sp, sp, #-1
+; CHECK-O0-NEXT:    str p0, [sp, #7, mul vl]
+; CHECK-O0-NEXT:    ldr p0, [sp, #7, mul vl]
+; CHECK-O0-NEXT:    addvl sp, sp, #1
+; CHECK-O0-NEXT:    ldr x29, [sp], #16 // 8-byte Folded Reload
 ; CHECK-O0-NEXT:    ret
 ;
 ; CHECK-O3-LABEL: test_alloca_store_reload:
 ; CHECK-O3:       // %bb.0:
-; CHECK-O3-NEXT:    sub sp, sp, #16
-; CHECK-O3-NEXT:    add x8, sp, #14
-; CHECK-O3-NEXT:    str p0, [x8]
-; CHECK-O3-NEXT:    add sp, sp, #16
+; CHECK-O3-NEXT:    str x29, [sp, #-16]! // 8-byte Folded Spill
+; CHECK-O3-NEXT:    addvl sp, sp, #-1
+; CHECK-O3-NEXT:    str p0, [sp, #7, mul vl]
+; CHECK-O3-NEXT:    addvl sp, sp, #1
+; CHECK-O3-NEXT:    ldr x29, [sp], #16 // 8-byte Folded Reload
 ; CHECK-O3-NEXT:    ret
   %ptr = alloca target("aarch64.svcount"), align 1
   store target("aarch64.svcount") %val, ptr %ptr
   %res = load target("aarch64.svcount"), ptr %ptr
   ret target("aarch64.svcount") %res
 }
 
 ;

llvm/test/CodeGen/RISCV/rvv/alloca-load-store-scalable-struct.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
+; RUN: llc -mtriple=riscv64 -mattr=+m,+d,+v -verify-machineinstrs \
+; RUN:   --riscv-no-aliases < %s | FileCheck %s
+
+target triple = "riscv64-unknown-unknown-elf"
+
+%struct.test = type { <vscale x 1 x double>, <vscale x 1 x double> }
+
+define <vscale x 1 x double> @test(%struct.test* %addr, i64 %vl) {
+; CHECK-LABEL: test:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    addi sp, sp, -16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    csrrs a2, vlenb, zero
+; CHECK-NEXT:    slli a2, a2, 1
+; CHECK-NEXT:    sub sp, sp, a2
+; CHECK-NEXT:    .cfi_escape 0x0f, 0x0d, 0x72, 0x00, 0x11, 0x10, 0x22, 0x11, 0x02, 0x92, 0xa2, 0x38, 0x00, 0x1e, 0x22 # sp + 16 + 2 * vlenb
+; CHECK-NEXT:    addi a2, a0, 8
+; CHECK-NEXT:    vl1re64.v v8, (a2)
+; CHECK-NEXT:    vl1re64.v v9, (a0)
+; CHECK-NEXT:    addi a0, sp, 24
+; CHECK-NEXT:    vs1r.v v8, (a0)
+; CHECK-NEXT:    addi a2, sp, 16
+; CHECK-NEXT:    vs1r.v v9, (a2)
+; CHECK-NEXT:    vl1re64.v v8, (a0)
+; CHECK-NEXT:    vl1re64.v v9, (a2)
+; CHECK-NEXT:    vsetvli zero, a1, e64, m1, ta, ma
+; CHECK-NEXT:    vfadd.vv v8, v9, v8
+; CHECK-NEXT:    csrrs a0, vlenb, zero
+; CHECK-NEXT:    slli a0, a0, 1
+; CHECK-NEXT:    add sp, sp, a0
+; CHECK-NEXT:    addi sp, sp, 16
+; CHECK-NEXT:    jalr zero, 0(ra)
+entry:
+  %ret = alloca %struct.test, align 8
+  %val = load %struct.test, %struct.test* %addr
+  store %struct.test %val, %struct.test* %ret, align 8
+  %0 = load %struct.test, %struct.test* %ret, align 8
+  %1 = extractvalue %struct.test %0, 0
+  %2 = extractvalue %struct.test %0, 1
+  %3 = call <vscale x 1 x double> @llvm.riscv.vfadd.nxv1f64.nxv1f64.i64(
+    <vscale x 1 x double> poison,
+    <vscale x 1 x double> %1,
+    <vscale x 1 x double> %2, i64 %vl)
+  ret <vscale x 1 x double> %3
+}
+
+declare <vscale x 1 x double> @llvm.riscv.vfadd.nxv1f64.nxv1f64.i64(
+  <vscale x 1 x double>,
+  <vscale x 1 x double>,
+  <vscale x 1 x double>,
+  i64)

llvm/test/Other/load-scalable-vector-struct.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -passes=verify -S < %s 2>&1 | FileCheck %s
+
+%struct.test = type { <vscale x 1 x i32>, <vscale x 1 x i32> }
+
+define <vscale x 1 x i32> @load(%struct.test* %x) {
+; CHECK-LABEL: define <vscale x 1 x i32> @load
+; CHECK-SAME: (ptr [[X:%.*]]) {
+; CHECK-NEXT:    [[A:%.*]] = load [[STRUCT_TEST:%.*]], ptr [[X]], align 4
+; CHECK-NEXT:    [[B:%.*]] = extractvalue [[STRUCT_TEST]] [[A]], 1
+; CHECK-NEXT:    ret <vscale x 1 x i32> [[B]]
+;
+  %a = load %struct.test, %struct.test* %x
+  %b = extractvalue %struct.test %a, 1
+  ret <vscale x 1 x i32> %b
+}

llvm/test/Other/store-scalable-vector-struct.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -passes=verify -S < %s 2>&1 | FileCheck %s
+
+%struct.test = type { <vscale x 1 x i32>, <vscale x 1 x i32> }
+
+define void @store(%struct.test* %x, <vscale x 1 x i32> %y, <vscale x 1 x i32> %z) {
+; CHECK-LABEL: define void @store
+; CHECK-SAME: (ptr [[X:%.*]], <vscale x 1 x i32> [[Y:%.*]], <vscale x 1 x i32> [[Z:%.*]]) {
+; CHECK-NEXT:    [[A:%.*]] = insertvalue [[STRUCT_TEST:%.*]] undef, <vscale x 1 x i32> [[Y]], 0
+; CHECK-NEXT:    [[B:%.*]] = insertvalue [[STRUCT_TEST]] [[A]], <vscale x 1 x i32> [[Z]], 1
+; CHECK-NEXT:    store [[STRUCT_TEST]] [[B]], ptr [[X]], align 4
+; CHECK-NEXT:    ret void
+;
+  %a = insertvalue %struct.test undef, <vscale x 1 x i32> %y, 0
+  %b = insertvalue %struct.test %a, <vscale x 1 x i32> %z, 1
+  store %struct.test %b, %struct.test* %x
+  ret void
+}

llvm/test/Transforms/InstCombine/scalable-vector-struct-extractvalue.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
+; RUN: opt -passes=instcombine -S < %s 2>&1 | FileCheck %s
+
+%struct.test = type { <vscale x 1 x i32>, <vscale x 1 x i32> }
+
+define <vscale x 1 x i32> @load(%struct.test* %x) {
+; CHECK-LABEL: define <vscale x 1 x i32> @load
+; CHECK-SAME: (ptr [[X:%.*]]) {
+; CHECK-NEXT:    [[A:%.*]] = load [[STRUCT_TEST:%.*]], ptr [[X]], align 4
+; CHECK-NEXT:    [[B:%.*]] = extractvalue [[STRUCT_TEST]] [[A]], 1
+; CHECK-NEXT:    ret <vscale x 1 x i32> [[B]]
+;
+  %a = load %struct.test, %struct.test* %x
+  %b = extractvalue %struct.test %a, 1
+  ret <vscale x 1 x i32> %b
+}

llvm/test/Verifier/scalable-vector-struct-gep.ll

+; RUN: not opt -S -passes=verify < %s 2>&1 | FileCheck %s
+
+%struct.test = type { <vscale x 1 x double>, <vscale x 1 x double> }
+
+define void @gep(ptr %a) {
+; CHECK: error: getelementptr cannot target structure that contains scalable vector type
+  %a.addr = getelementptr %struct.test, ptr %a, i32 0
+  ret void
+}