Scalable vector core instruction support + size queries

[huntergr]

Summary

Implements basic size queries to support scalable vectors in LLVM IR.

Adds simple tests for IR instructions usable with scalable vector types to ensure that they can be parsed and printed back out without dropping the scalable flag; used the size queries and getElementCount in a couple of places to get it working.

Adds a few scalable size query calls in codegen and tablegen to keep existing tests working, including the recent SVE calling convention test.

Adds checks in various backends which don't support scalable vectors to skip over them when registering legalization and DAG actions.

[huntergr]

• Unified ScalableSize representation
• Changed to uint64_t + boolean, since we no longer allow scalable vectors in aggregates
• Removed aggregate and mixed unit tests

[rengolin]

This looks ok to me, but I'd rather more people look at it before approving.

Thanks!

[greened]

I know this isn't ready for merge, but since the mailing list discussion has died down it seems like maybe we should move the discussion here. If so, it would be helpful to have comments on all the routines explaining what they do and how they differ from the existing routines, in order to aid discussion.

[huntergr]

Updated based on where scalable size queries were required when running with an SVE-capable loop vectorizer on several codebases (all of LNT, various flavours of SpecCPU, lots of HPC and embedded benchmarks).

[rovka]

Hi Graham,

I think this patch is difficult to review. It covers many different source files with only a small unit test to check the correctness. This isn't very robust against future changes and it makes it hard to know exactly what is and isn't supported.

I would find it much easier to review with an incremental strategy based on regression tests. For instance, with ToT opt, the attached testcase fails (error: '%r' defined with type '<4 x i1>' but expected '<vscale x 4 x i1>'). I would add a patch to fix that, and maybe other similar, really simple cases. We could then proceed to more complex examples, run some of the passes that come after the vectorizer on them, and progressively fix the places required to make them pass, with focused tests for each hurdle that we run into. It shouldn't be too hard to reduce such snippets from the tests you've already been running.

We would eventually end up with something like this patch, but I think that an incremental approach would have the advantage of quicker reviews with more focused discussions, more regression tests, and we would know how much was supported at any point. Does anyone else have a similar opinion?

simplest.ll415 BDownload

[huntergr]

Hi Diana,

Thanks for the comments.

In D53137#1597022, @rovka wrote:

I think this patch is difficult to review. It covers many different source files with only a small unit test to check the correctness. This isn't very robust against future changes and it makes it hard to know exactly what is and isn't supported.

Yeah, I was worried about that -- this is basically the size queries alone without anything actually using scalable vectors. It demonstrates roughly where changes will be needed, but doesn't actually change the surrounding code to use e.g. getElementCount instead of getNumElements.

I would find it much easier to review with an incremental strategy based on regression tests. For instance, with ToT opt, the attached testcase fails (error: '%r' defined with type '<4 x i1>' but expected '<vscale x 4 x i1>'). I would add a patch to fix that, and maybe other similar, really simple cases. We could then proceed to more complex examples, run some of the passes that come after the vectorizer on them, and progressively fix the places required to make them pass, with focused tests for each hurdle that we run into. It shouldn't be too hard to reduce such snippets from the tests you've already been running.

An incremental approach sounds good; assuming nobody objects, I'll remove most of the code in this patch and just leave the core mechanism behind (in enforcing mode) and add in that test case. We can fill in the other cases as we enable codegen/acle/autovec in separate patches.

[rengolin]

In D53137#1597022, @rovka wrote:

I would find it much easier to review with an incremental strategy based on regression tests. For instance, with ToT opt, the attached testcase fails (error: '%r' defined with type '<4 x i1>' but expected '<vscale x 4 x i1>'). I would add a patch to fix that, and maybe other similar, really simple cases. We could then proceed to more complex examples, run some of the passes that come after the vectorizer on them, and progressively fix the places required to make them pass, with focused tests for each hurdle that we run into. It shouldn't be too hard to reduce such snippets from the tests you've already been running.

+1 to incremental approach and more tests!

This change is mostly mechanical, but you're absolutely right we need to be aware of unwanted side effects. I wrongly assumed mechanical == NFC, but this clearly isn't. Thanks for the thorough review!

I think tests need to be strict on what it should support. Not necessarily test for *all* errors, but add test for the supported cases and some negative tests for the obvious unsupported stuff.

Each step of the way, revert negative tests when new features are added (and adding more tests, too!), we can make sure it's stable and robust.

[rovka]

In D53137#1597088, @huntergr wrote:

Hi Diana,

Thanks for the comments.

In D53137#1597022, @rovka wrote:

I think this patch is difficult to review. It covers many different source files with only a small unit test to check the correctness. This isn't very robust against future changes and it makes it hard to know exactly what is and isn't supported.

Yeah, I was worried about that -- this is basically the size queries alone without anything actually using scalable vectors. It demonstrates roughly where changes will be needed, but doesn't actually change the surrounding code to use e.g. getElementCount instead of getNumElements.

I would find it much easier to review with an incremental strategy based on regression tests. For instance, with ToT opt, the attached testcase fails (error: '%r' defined with type '<4 x i1>' but expected '<vscale x 4 x i1>'). I would add a patch to fix that, and maybe other similar, really simple cases. We could then proceed to more complex examples, run some of the passes that come after the vectorizer on them, and progressively fix the places required to make them pass, with focused tests for each hurdle that we run into. It shouldn't be too hard to reduce such snippets from the tests you've already been running.

An incremental approach sounds good; assuming nobody objects, I'll remove most of the code in this patch and just leave the core mechanism behind (in enforcing mode) and add in that test case. We can fill in the other cases as we enable codegen/acle/autovec in separate patches.

Actually my test case falls in the "replace getNumElements with getElementCount" category, rather than size queries per se. I don't think either of them is more important than the other, but you might run into the former while writing tests for the latter.

[huntergr]

• Removed most of the changes in favour of reintroducing them in separate patches later with appropriate tests.
• Added tests for core IR instructions to make sure they don't drop the scalable flag.
• Fixed up a couple places which broke the new tests.

[greened]

llvm/include/llvm/CodeGen/ValueTypes.h

299 ↗

(On Diff #212548)

Needs a comment about what this returns.

305 ↗

(On Diff #212548)

Needs a comment about what this returns.

321 ↗

(On Diff #212548)

Needs a comment about what this returns.

326 ↗

(On Diff #212548)

Needs a comment about what this returns.

336 ↗

(On Diff #212548)

Needs a comment about what this returns.

340 ↗

(On Diff #212548)

Needs a comment about what this returns.

llvm/include/llvm/IR/DataLayout.h

441 ↗

(On Diff #212548)

Needs comments about what these return.

452 ↗

(On Diff #212548)

Needs a comment about what this returns.

458 ↗

(On Diff #212548)

Needs a comment about what this returns.

470 ↗

(On Diff #212548)

Needs a comment about what this returns.

475 ↗

(On Diff #212548)

Needs a comment about what this returns.

497 ↗

(On Diff #212548)

Needs a comment about what this returns.

504 ↗

(On Diff #212548)

Needs a comment about what this returns.

517 ↗

(On Diff #212548)

Needs a comment about what this returns.

[greened]

llvm/include/llvm/IR/DataLayout.h

522 ↗

(On Diff #212548)

Needs a comment about what this returns.

llvm/include/llvm/Support/MachineValueType.h

672 ↗

(On Diff #212548)

Not sure why the other methods here don't have comments but we should probably have one here to say what this does.

720 ↗

(On Diff #212548)

Needs a comment about what this returns.

840 ↗

(On Diff #212548)

Needs a comment about what this returns.

845 ↗

(On Diff #212548)

Needs a comment about what this returns.

855 ↗

(On Diff #212548)

Needs a comment about what this returns.

860 ↗

(On Diff #212548)

Needs a comment about what this returns.

llvm/include/llvm/Support/ScalableSize.h

41 ↗

(On Diff #212548)

Needs comments about what this is and what the fields represent.

100 ↗

(On Diff #212548)

This needs an explanatory comment.

105 ↗

(On Diff #212548)

Needs a comment about what this returns.

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

4612 ↗

(On Diff #212548)

This comment should probably be moved right before the if below.

llvm/lib/IR/DataLayout.cpp

744 ↗

(On Diff #212548)

I might help to clarify that this comment only applies to scalable types, at least as far as I understand this changes here.

llvm/lib/Target/Hexagon/HexagonISelLowering.cpp

1438 ↗

(On Diff #212548)

What about other backends? Do they need similar checks (both here and below)?

[rovka]

Several more general comments:

• Should all the getXSize assert when called on a scalable type? I see that for MVT::getSizeInBits and for Type::getPrimitiveSize, but not for the others. This should also be made clear in the comments for each of them.
• Great test for the IR, thanks!
• I don't see any test for the CodeGen stuff though. Is it possible to add one? (If not, maybe add the changes to EVT etc when we can actually test them).
• Ditto for TableGen (or if that's too difficult/hairy to test, just update the commit message to explain exactly why the change belongs in this patch).

llvm/include/llvm/CodeGen/ValueTypes.h

214 ↗

(On Diff #212548)

Why does this one need to change? We're only looking at MinSize anyway, isn't getSizeInBits good enough?

llvm/include/llvm/IR/DataLayout.h

517 ↗

(On Diff #212548)

Where is this used?

llvm/include/llvm/Support/ScalableSize.h

41 ↗

(On Diff #212548)

Maybe make this a class, so people can't just get at the MinSize directly? Otherwise, it doesn't make much sense to have accessors for the fields (and especially getFixedSize, which can be completely circumvented via direct access).

48 ↗

(On Diff #212548)

Isn't this already deleted because we have a constructor just above?

51 ↗

(On Diff #212548)

I think it's more canonical to say return std::tie(MinSize, Scalable) == std::tie(RHS.MinSize, RHS.Scalable).

57 ↗

(On Diff #212548)

Should be implemented in terms of operator==.

64 ↗

(On Diff #212548)

Needs a comment saying that you can't compare scalable sizes and fixed sizes.
You can simplify the code by replacing the check for Scalable with an assert and removing the unreachable at the end.
Also, all the other comparison operators below should be implemented in terms of this.

92 ↗

(On Diff #212548)

Would be nice to also have a non-member one for symmetry (so you can write 2 * SomeSize, not just SomeSize * 2).

llvm/lib/CodeGen/ValueTypes.cpp

106 ↗

(On Diff #212548)

Should also assert that it's not called on a scalable type.

[huntergr]

Thanks for the reviews; I left a couple of inline comments, and will make the requested changes.

llvm/include/llvm/CodeGen/ValueTypes.h

214 ↗

(On Diff #212548)

getSizeInBits will assert (actually, the underlying MVT version for it will) for a scalable vector type, so can't be used.

However, I introduced the 'getMinSizeInBits' function after writing this, so it should be used here.

llvm/lib/Target/Hexagon/HexagonISelLowering.cpp

1438 ↗

(On Diff #212548)

I'll take a look, but the Hexagon backend was the only one which asserted when running make check-all.

[huntergr]

• Added comments explaining the new methods
• Added tests for the MVT/EVT/DataLayout interfaces
• Refactored ScalableSize class operators to build on '==' and '<'
• Added checks in various backends that don't support scalable vectors to prevent legalizing operations involving scalable MVTs

[sdesmalen]

Thanks @huntergr for working on this!

This patch can probably be split into two separate patches, which make them easier to review;

• One that fixes the Targets to ignore scalable types (see comment)
• Another one that adds the interface for scalable size queries.

On the interfaces itself, I personally find ScalableSize a bit of a misnomer (see comment for other suggestions) because it describes both scalable and fixed-width sizes, which is not what the name suggests. But perhaps more importantly, my concern is that with the interface as defined in this patch, there is little incentive to move the LLVM codebase to work on ScalableSize. Especially for new code that gets added, the interface should let developers make a conscious decision whether their code is fixed-width only, scalable-width only, or agnostic to this property. I would much rather see getSizeInBits() return a ScalableSize object, which in turn only has getFixedSize() and getMinScalableSize() as query methods. If the code is agnostic to whether the size of an object is fixed or scalable, the code should simply operate on the ScalableSize object itself, rather than operating on MinSize.

I realise that the entire code-base currently expects 'unsigned' or 'int', but this can easily be fixed by adding a (hopefully temporary) overloaded cast operator to the struct that produces a scalar value, like:

/// Casts to a uint64_t if this is a fixed-width size.
///
/// NOTE: This interface is obsolescent and will be removed
/// in a future version of LLVM in favour of getFixedSize().
operator uint64_t() const {
  assert(isFixed() && "Expected fixed size data type");
  return getFixedSize();
}

We can then update the codebase piece by piece, incrementally making use of ScalableSize or its interfaces. When all that is done, the codebase should compile without errors when building for LLVM_TARGETS_TO_BUILD=AArch64 after we remove the overloaded operator.

llvm/include/llvm/Support/ScalableSize.h

83 ↗

(On Diff #213589)

All the comparison operators assert that the types are both fixed-width or both scalable.
Is there value in also adding the following interfaces?

// Returns true if A is known to be at least as big as B, e.g.
// If A is scalable, and B is not, returns A.MinSize >= B.MinSize
// If A is not scalable, and B is, returns false regardless of MinSize.
bool knownGreaterOrEqual(const ScalableSize &B) const { ... }
// or alternatively: "atLeastAsBigAs"?

and

// Returns true if A is known to always be larger than B, e.g.
// If A is scalable, and B is not, returns A.MinSize > B.MinSize
// If A is not scalable, and B is, returns false regardless of MinSize.
bool knownGreater(const ScalableSize &B) const { ... }

120 ↗

(On Diff #213589)

I think this only really makes sense in the context of scalable vectors, and don't think we want to expose it as a property for both fixed/scalable vectors, e.g.:

uint64_t getMinSize() const {
  assert (isScalable() && "MinSize only makes sense in the context of a scalable vector,"
                          " use getFixedSize() instead");
  return MinSize;
}

For cases where we need to use it in a comparison (to know if it is at least a given number of bytes) we can do this in a separate comparison interface.

125 ↗

(On Diff #213589)

bool ScalableSize::isScalable() proves to me that the name ScalableSize is a misnomer. If you have a ScalableSize object, I'd expect it to always represent a scalable size.

Can we rename this to something more generic like ObjectSize or PrimitiveSize? (suggestions welcome)

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

9177 ↗

(On Diff #213589)

OutputArg::PartOffset is defined explicitly as a byte offset, so until we change that, this should be getFixedSize().

llvm/lib/IR/Instructions.cpp

3039 ↗

(On Diff #213589)

MinSize suggests that it could be larger at runtime, so SrcBits.getMinSize() == 0 would always be true. If you instead overload operator bool() which checks for Fixed size and Scalable size to be 0, you can rewrite this as (!SrcBits || !DestBits) .

llvm/lib/Target/X86/X86ISelLowering.cpp

721 ↗

(On Diff #213589)

Adding MVT::fixedwidth_[integer_|fp_]vector_valuetypes() seems like a more natural interface than skipping based on isScalableVector(). There already seem to be iterators for MVT::(integer_|fp_)_scalable_vector_valuetypes.

You can do that in a separate patch, so this patch can focus solely on scalable size queries.

[huntergr]

llvm/include/llvm/Support/ScalableSize.h

83 ↗

(On Diff #213589)

If you can think of an immediate use case, sure.

Otherwise I'd leave it to another patch which requires that information.

120 ↗

(On Diff #213589)

This is used for the alignment checks at the moment, where we just need to know the minimum for scalable and exact for fixed.

I'd rather not have code that looks like

unsigned Align;
if (VTy->isScalable())
  Align = getMinSize(VTy);
else
  Align = getFixedSize(VTy);

in several places, when it could be done with one.

We can certainly bikeshed the names, though, and come up with a more explicit one which acknowledges it can represent a known quantity from fixed or scalable vectors.

I would also need to convert a larger part of the codebase to use scalable types in order to use this right now, and I've already had pushback on the size of the changes.

125 ↗

(On Diff #213589)

Yeah, I figured it needed a rename, but plain 'Size' was likely to cause problems. ObjectSize or TypeSize might work.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

9177 ↗

(On Diff #213589)

Then we would assert when running unit tests.

I'm trying to juggle between an enormous patch that fixes everything 'properly', and something which gets us partway and keeps us running but needs fixes elsewhere later (which seemed to be what other reviewers would prefer).

I could certainly add a comment here to explain that changes are needed, though.

llvm/lib/IR/Instructions.cpp

3039 ↗

(On Diff #213589)

No. 0 * vscale is still 0 for any value of vscale.

The ScalableSize class says the Scalable flag indicates the total size is an integer multiple of the known minimum size.

I suspect a better way of doing this test would be to explicitly check for the elements of one being pointer type and not the other, instead of relying on a hack with the size.

llvm/lib/Target/X86/X86ISelLowering.cpp

721 ↗

(On Diff #213589)

Yeah, I thought of that after I'd submitted the patch. Will do.

[huntergr]

In D53137#1621351, @sdesmalen wrote:

Thanks @huntergr for working on this!

This patch can probably be split into two separate patches, which make them easier to review;

• One that fixes the Targets to ignore scalable types (see comment)
• Another one that adds the interface for scalable size queries.

Ok, will do.

On the interfaces itself, I personally find ScalableSize a bit of a misnomer (see comment for other suggestions) because it describes both scalable and fixed-width sizes, which is not what the name suggests. But perhaps more importantly, my concern is that with the interface as defined in this patch, there is little incentive to move the LLVM codebase to work on ScalableSize. Especially for new code that gets added, the interface should let developers make a conscious decision whether their code is fixed-width only, scalable-width only, or agnostic to this property. I would much rather see getSizeInBits() return a ScalableSize object, which in turn only has getFixedSize() and getMinScalableSize() as query methods. If the code is agnostic to whether the size of an object is fixed or scalable, the code should simply operate on the ScalableSize object itself, rather than operating on MinSize.

See some of the inline comments -- there are a few places where we'd just end up duplicating code if used that way. The names can certainly be improved for clarity, though, and we could state that the (scalable|fixed)-only interfaces should be used in preference to a joint one.

I realise that the entire code-base currently expects 'unsigned' or 'int', but this can easily be fixed by adding a (hopefully temporary) overloaded cast operator to the struct that produces a scalar value, like:

/// Casts to a uint64_t if this is a fixed-width size.
///
/// NOTE: This interface is obsolescent and will be removed
/// in a future version of LLVM in favour of getFixedSize().
operator uint64_t() const {
  assert(isFixed() && "Expected fixed size data type");
  return getFixedSize();
}

We can then update the codebase piece by piece, incrementally making use of ScalableSize or its interfaces. When all that is done, the codebase should compile without errors when building for LLVM_TARGETS_TO_BUILD=AArch64 after we remove the overloaded operator.

I'm fine with that approach if others approve; I was trying to minimize the overall impact on the codebase though.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

9177 ↗

(On Diff #213589)

A followup thought on this; would it be preferable to split the EVT/MVT side of this patch out into a separate one, and fix this properly there?

[sdesmalen]

In D53137#1622713, @huntergr wrote:

See some of the inline comments -- there are a few places where we'd just end up duplicating code if used that way. The names can certainly be improved for clarity, though, and we could state that the (scalable|fixed)-only interfaces should be used in preference to a joint one.

Yes, adding some words to state that in the description of the method would be good. There are indeed cases where MinSize is needed directly (like in DAGCombiner.cpp where it uses MaximumLegalStoreInBits = MinSize), but such a case is very explicit (it needs to specifically query the known part of the size), which I think is not very common. We should strive to reduce those uses as much as possible. For Alignment for example, I think we can use a separate method (see comment).

I realise that the entire code-base currently expects 'unsigned' or 'int', but this can easily be fixed by adding a (hopefully temporary) overloaded cast operator to the struct that produces a scalar value, like:

/// Casts to a uint64_t if this is a fixed-width size.
///
/// NOTE: This interface is obsolescent and will be removed
/// in a future version of LLVM in favour of getFixedSize().
operator uint64_t() const {
  assert(isFixed() && "Expected fixed size data type");
  return getFixedSize();
}

We can then update the codebase piece by piece, incrementally making use of ScalableSize or its interfaces. When all that is done, the codebase should compile without errors when building for LLVM_TARGETS_TO_BUILD=AArch64 after we remove the overloaded operator.

I'm fine with that approach if others approve; I was trying to minimize the overall impact on the codebase though.

Great! Looking forward to hear how others feel about it.

llvm/include/llvm/Support/ScalableSize.h

83 ↗

(On Diff #213589)

One use-case is in DAGCombiner.cpp, where it currently has:

VT.getMinSizeInBits() >= MaximumLegalStoreInBits

which would become:

VT.getSizeInBits().isKnownGreaterOrEqual(MaximumLegalStoreInBits)

This removes another use-case for using MinSize directly.

120 ↗

(On Diff #213589)

For Alignment, because it is so common, I think it is worth adding a separate method:

unsigned getNaturalAlignment() const { return (unsigned) getKnownSize(); };

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

9177 ↗

(On Diff #213589)

Right, I see. At some point this class should probably use StackOffset to represent such offsets. For now, I'd say this line warrants a "FIXME" comment.

9177 ↗

(On Diff #213589)

A followup thought on this; would it be preferable to split the EVT/MVT side of this patch out into a separate one, and fix this properly there?

I think so.

llvm/lib/IR/Instructions.cpp

3039 ↗

(On Diff #213589)

Perhaps I am a little pedantic with how I read this, because I expect the minimum size of every object to be 0, always :)

So perhaps instead of using the name MinSize, a name like KnownSize would be better suited. (which would also nicely match with the isKnownGreaterOrEqual() suggestion for DAGCombiner.cpp).

llvm/lib/Target/X86/X86ISelLowering.cpp

721 ↗

(On Diff #213589)

Thanks!

[huntergr]

Created D66339 to split out the code to skip scalable vector types in other backends.

[huntergr]

• Split out backend code into separate patches
• Renamed 'Min' to 'KnownMin' in method names.
• Added a few more comments.

I tried to replace the comparison of minimum sizes to 0 in Instructions.cpp (for the bitcast checks), but that statement represents quite a lot. Could be scalar pointers, could be a vector of pointers (both of which I checked for in experiments), but there are several other types which report a size of 0 and which are checked for in some unit tests, so the number of checks ended up being fairly substantial. I've left it alone for now, but if reviewers would prefer I extract a method to explicitly check for all conditions represented by the size check I can do so.

[huntergr]

Does anyone like Sander's suggestion to make ScalableSize (or whatever we end up naming it) the return value for all size queries and provide an overloaded cast operator to transparently work with existing code comparing against unsigned values? Or is it preferable to keep the current split?

[rovka]

In D53137#1650774, @huntergr wrote:

Does anyone like Sander's suggestion to make ScalableSize (or whatever we end up naming it) the return value for all size queries and provide an overloaded cast operator to transparently work with existing code comparing against unsigned values? Or is it preferable to keep the current split?

+1 from me, I definitely like Sander's suggestion. I thought we were heading towards ScalableSize-only functions anyway and that this was just a temporary path until we cleaned up the codebase from unsigned. Using a cast operator from the start seems like an even better way of getting there, since that way at least we hopefully won't get that many new uses of unsigned in the interim.

[greened]

In D53137#1652189, @rovka wrote:

+1 from me, I definitely like Sander's suggestion. I thought we were heading towards ScalableSize-only functions anyway and that this was just a temporary path until we cleaned up the codebase from unsigned. Using a cast operator from the start seems like an even better way of getting there, since that way at least we hopefully won't get that many new uses of unsigned in the interim.

+1

[huntergr]

• Changed existing interface to return ScalableSize objects and added a (mostly) transparent conversion, as per Sander's suggestion.
• Removed new interfaces for DataLayout and Type.
• Fixed cases where the transparent conversion doesn't quite work (e.g. std::max/min, where the types must be the same)

I suspect 'ScalableSize' is the wrong term now; 'TypeSize' may be better. Thoughts?

[rovka]

I suspect 'ScalableSize' is the wrong term now; 'TypeSize' may be better. Thoughts?

I agree, TypeSize sounds better. Maybe we can replace the public constructor with 2 static methods, TypeSize::Fixed(Size) and TypeSize::Scalable(Size), so we don't always have to spell out /* Scalable =*/.

llvm/include/llvm/IR/DataLayout.h

454 ↗

(On Diff #221283)

We already overload operator /, why not overload + as well so we don't have to change the body of this method?

487 ↗

(On Diff #221283)

Can we add a version of alignTo that works with ScalableSize instead?

656 ↗

(On Diff #221283)

Maybe just return VTy->getElementCount() * getTypeSizeInBits(VTy->getElementType()).getFixedSize().

[huntergr]

llvm/include/llvm/IR/DataLayout.h

454 ↗

(On Diff #221283)

Scaling a size with * or / has a clear meaning to me, since it's independent of vscale; getting a vector that's half the size or four times larger just works.

Using + (or -) on the other hand doesn't seem to be as clear; I wasn't sure if a standalone int should be automatically treated as being the same as the TypeSize, or always considered Fixed. If we try for the former I can imagine quite a few bugs arising.

I could add a roundBitsToNearestByteSize method to move the arithmetic elsewhere if that would be acceptable?

656 ↗

(On Diff #221283)

There's no support for generating a TypeSize from an ElementCount in that way; is that an interface you feel is useful?

(I'll certainly change the getKnownMinSize to getFixedSize though, since we're just referring to a scalar)

[huntergr]

• Renamed ScalableSize to TypeSize, including header name.
• added alignTo function that takes and returns a TypeSize. I wasn't sure if this should be added to MathExtras.h where the other variants live, so just kept it in TypeSize.h for now

[rovka]

This looks good to me, maybe wait a while to see if anyone else has any further comments.

llvm/include/llvm/IR/DataLayout.h

454 ↗

(On Diff #221283)

You're right, + on TypeSizes would be confusing. This looks ok as-is then, no need to fiddle with it more.

656 ↗

(On Diff #221283)

Actually, no, now that I think about it a bit more it might be clearer to spell it out this way.

llvm/include/llvm/Support/TypeSize.h

122 ↗

(On Diff #222593)

Microscopic nit: punctuation.

143 ↗

(On Diff #222593)

Ditto.

148 ↗

(On Diff #222593)

Ditto.

[sdesmalen]

Thanks @huntergr , I think this interface looks really nice. LGTM!

llvm/include/llvm/IR/DataLayout.h

456 ↗

(On Diff #222593)

nit: use TypeSize instead of auto.

622 ↗

(On Diff #222593)

Nice, I like the interface of TypeSize::Fixed() for fixed-size types.

llvm/lib/IR/Instructions.cpp

2990 ↗

(On Diff #222593)

nit: auto -> TypeSize ?

3052 ↗

(On Diff #222593)

nit: auto -> TypeSize ?

[huntergr]

Hmm, forgot to add the last round of minor fixes before committing. Sorry about that, will push them as well.

include/llvm/CodeGen/BasicTTIImpl.h

unsigned getMemoryOpCost(unsigned Opcode, Type *Src, unsigned Alignment,
                        unsigned AddressSpace, const Instruction *I = nullptr) {
     assert(!Src->isVoidTy() && "Invalid type");
     std::pair<unsigned, MVT> LT = getTLI()->getTypeLegalizationCost(DL, Src);
 
     // Assuming that all loads of legal types cost 1.
     unsigned Cost = LT.first;
 
     if (Src->isVectorTy() &&
-        Src->getPrimitiveSizeInBits() < LT.second.getSizeInBits()) {
+      Src->getScalableSizeInBits().Unscaled < LT.second.getSizeInBits()) {
       // This is a vector load that legalizes to a larger type than the vector
       // itself. Unless the corresponding extending load or truncating store is
       // legal, then this will scalarize.
       TargetLowering::LegalizeAction LA = TargetLowering::Expand;
       EVT MemVT = getTLI()->getValueType(DL, Src);
       if (Opcode == Instruction::Store)
         LA = getTLI()->getTruncStoreAction(LT.second, MemVT);
       else

include/llvm/IR/DataLayout.h

public:
   /// Layout size of the index used in GEP calculation.
   /// The function should be called with pointer or vector of pointers type.
   unsigned getIndexTypeSizeInBits(Type *Ty) const;
 
   unsigned getPointerTypeSize(Type *Ty) const {
     return getPointerTypeSizeInBits(Ty) / 8;
   }
 
+  struct ScalableSize {
+    uint64_t Unscaled;
+    uint64_t Scaled;
+
+    ScalableSize(uint64_t Unscaled, uint64_t Scaled)
+      : Unscaled(Unscaled), Scaled(Scaled) {}
+
+    ScalableSize() {}
+
+    bool operator==(const ScalableSize RHS) const {
+      return Unscaled == RHS.Unscaled && Scaled == RHS.Scaled;
+    }
+
+    bool operator!=(const ScalableSize RHS) const {
+      return Unscaled != RHS.Unscaled || Scaled != RHS.Scaled;
+    }
+
+    bool operator<(const ScalableSize& RHS) const {
+      // Fixed-length fast case
+      if (Scaled == RHS.Scaled)
+        return Unscaled < RHS.Unscaled;
+
+      // Scalable fast case
+      if (Unscaled == RHS.Unscaled)
+        return Scaled < RHS.Scaled;
+
+      // Runtime dependent, no matter the unscaled difference.
+      if (Scaled > RHS.Scaled)
+        return false;
+
+      // We know Scaled < RHS.Scaled now.
+      if (Unscaled <= RHS.Unscaled)
+        return true;
+
+      // {largeU, smallS} vs. {smallU, largeS}
+      // if the unscaled difference is less than the scaled difference,
+      // then this is true. Otherwise it's runtime dependent.
+      auto SDiff = RHS.Scaled - Scaled;
+      auto UDiff = Unscaled - RHS.Unscaled;
+      return UDiff < SDiff;
+    }
+
+    bool operator<=(const ScalableSize& RHS) const {
+      // Fixed-length fast case
+      if (Scaled == RHS.Scaled)
+        return Unscaled <= RHS.Unscaled;
+
+      // Scalable fast case
+      if (Unscaled == RHS.Unscaled)
+        return Scaled <= RHS.Scaled;
+
+      // Runtime dependent, no matter the unscaled difference.
+      if (Scaled > RHS.Scaled)
+        return false;
+
+      // We know Scaled < RHS.Scaled now.
+      if (Unscaled <= RHS.Unscaled)
+        return true;
+
+      // {largeU, smallS} vs. {smallU, largeS}
+      // if the unscaled difference is less than or equal to the scaled
+      // difference, then this is true. Otherwise it's runtime dependent.
+      auto SDiff = RHS.Scaled - Scaled;
+      auto UDiff = Unscaled - RHS.Unscaled;
+      return UDiff <= SDiff;
+    }
+
+    bool operator>(const ScalableSize& RHS) const {
+      // Fixed-length fast case
+      if (Scaled == RHS.Scaled)
+        return Unscaled > RHS.Unscaled;
+
+      // Scalable fast case
+      if (Unscaled == RHS.Unscaled)
+        return Scaled > RHS.Scaled;
+
+      // Runtime dependent, no matter the unscaled difference.
+      if (Scaled < RHS.Scaled)
+        return false;
+
+      // We know Scaled > RHS.Scaled now.
+      if (Unscaled >= RHS.Unscaled)
+        return true;
+
+      // {smallU, largeS} vs. {largeU, smallS}
+      // if the unscaled difference is less than the scaled difference,
+      // then this is true. Otherwise it's runtime dependent.
+      auto SDiff = Scaled - RHS.Scaled;
+      auto UDiff = RHS.Unscaled - Unscaled;
+      return UDiff < SDiff;
+    }
+
+    bool operator>=(const ScalableSize& RHS) const {
+      // Fixed-length fast case
+      if (Scaled == RHS.Scaled)
+        return Unscaled >= RHS.Unscaled;
+
+      // Scalable fast case
+      if (Unscaled == RHS.Unscaled)
+        return Scaled >= RHS.Scaled;
+
+      // Runtime dependent, no matter the unscaled difference.
+      if (Scaled < RHS.Scaled)
+        return false;
+
+      // We know Scaled > RHS.Scaled now.
+      if (Unscaled >= RHS.Unscaled)
+        return true;
+
+      // {smallU, largeS} vs. {largeU, smallS}
+      // if the unscaled difference is less than or equal to the scaled
+      // difference, then this is true. Otherwise it's runtime dependent.
+      auto SDiff = Scaled - RHS.Scaled;
+      auto UDiff = RHS.Unscaled - Unscaled;
+      return UDiff <= SDiff;
+    }
+
+    ScalableSize& operator+=(const ScalableSize& RHS) {
+      Unscaled += RHS.Unscaled;
+      Scaled += RHS.Scaled;
+      return *this;
+    }
+
+    ScalableSize operator*(uint64_t Count) {
+      return ScalableSize(Unscaled * Count, Scaled * Count);
+    }
+  };
+
   /// Size examples:
   ///
   /// Type        SizeInBits  StoreSizeInBits  AllocSizeInBits[*]
   /// ----        ----------  ---------------  ---------------
   ///  i1            1           8                8
   ///  i8            8           8                8
   ///  i19          19          24               32
   ///  i32          32          32               32
   ///  i100        100         104              128
   ///  i128        128         128              128
   ///  Float        32          32               32
   ///  Double       64          64               64
   ///  X86_FP80     80          80               96
   ///
   /// [*] The alloc size depends on the alignment, and thus on the target.
   ///     These values are for x86-32 linux.
 
   /// Returns the number of bits necessary to hold the specified type.
   ///
   /// For example, returns 36 for i36 and 80 for x86_fp80. The type passed must
   /// have a size (Type::isSized() must return true).
   uint64_t getTypeSizeInBits(Type *Ty) const;
 
+  ScalableSize getScalableTypeSizeInBits(Type *Ty) const;
+
   /// Returns the maximum number of bytes that may be overwritten by
   /// storing the specified type.
   ///
   /// For example, returns 5 for i36 and 10 for x86_fp80.
   uint64_t getTypeStoreSize(Type *Ty) const {
     return (getTypeSizeInBits(Ty) + 7) / 8;
   }
 
+  ScalableSize getScalableTypeStoreSize(Type *Ty) const {
+    // Is overloading bits/bytes wise?
+    auto Bits = getScalableTypeSizeInBits(Ty);
+    return ScalableSize((Bits.Unscaled+7)/8,
+                        (Bits.Scaled+7)/8);
+  }
+
   /// Returns the maximum number of bits that may be overwritten by
   /// storing the specified type; always a multiple of 8.
   ///
   /// For example, returns 40 for i36 and 80 for x86_fp80.
   uint64_t getTypeStoreSizeInBits(Type *Ty) const {
     return 8 * getTypeStoreSize(Ty);
   }
 
   /// Returns the offset in bytes between successive objects of the
   /// specified type, including alignment padding.
   ///
   /// This is the amount that alloca reserves for this type. For example,
   /// returns 12 or 16 for x86_fp80, depending on alignment.
   uint64_t getTypeAllocSize(Type *Ty) const {
     // Round up to the next alignment boundary.
     return alignTo(getTypeStoreSize(Ty), getABITypeAlignment(Ty));
   }
 
+  ScalableSize getScalableTypeAllocSize(Type *Ty) const {
+    auto Bytes = getScalableTypeStoreSize(Ty);
+    auto *VTy = dyn_cast<VectorType>(Ty);
+    if (!VTy || !VTy->isScalable())
+      Bytes.Unscaled = alignTo(Bytes.Unscaled, getABITypeAlignment(Ty));
+
+    return Bytes;
+  }
+
   /// Returns the offset in bits between successive objects of the
   /// specified type, including alignment padding; always a multiple of 8.
   ///
   /// This is the amount that alloca reserves for this type. For example,
   /// returns 96 or 128 for x86_fp80, depending on alignment.
   uint64_t getTypeAllocSizeInBits(Type *Ty) const {
     return 8 * getTypeAllocSize(Ty);
   }
 
+  ScalableSize getScalableTypeAllocSizeInBits(Type *Ty) const {
+    return getScalableTypeAllocSize(Ty) * 8;
+  }
+
   /// Returns the minimum ABI-required alignment for the specified type.
   unsigned getABITypeAlignment(Type *Ty) const;
 
   /// Returns the minimum ABI-required alignment for an integer type of
   /// the specified bitwidth.
   unsigned getABIIntegerTypeAlignment(unsigned BitWidth) const;
 
   /// Returns the preferred stack/global alignment for the specified

 
 inline LLVMTargetDataRef wrap(const DataLayout *P) {
   return reinterpret_cast<LLVMTargetDataRef>(const_cast<DataLayout *>(P));
 }
 
 /// Used to lazily calculate structure layout information for a target machine,
 /// based on the DataLayout structure.
 class StructLayout {
-  uint64_t StructSize;
+  DataLayout::ScalableSize StructSize;
   unsigned StructAlignment;
   unsigned IsPadded : 1;
   unsigned NumElements : 31;
-  uint64_t MemberOffsets[1]; // variable sized array!
+  DataLayout::ScalableSize MemberOffsets[1]; // variable sized array!
 
 public:
-  uint64_t getSizeInBytes() const { return StructSize; }
+  uint64_t getSizeInBytes() const { return StructSize.Unscaled; }
+
+  DataLayout::ScalableSize getScalableSizeInBytes() const { return StructSize; }
 
-  uint64_t getSizeInBits() const { return 8 * StructSize; }
+  uint64_t getSizeInBits() const { return 8 * StructSize.Unscaled; }
+
+  DataLayout::ScalableSize getScalableSizeInBits() const {
+    return DataLayout::ScalableSize(StructSize.Unscaled * 8,
+                                    StructSize.Scaled * 8);
+  }
 
   unsigned getAlignment() const { return StructAlignment; }
 
   /// Returns whether the struct has padding or not between its fields.
   /// NB: Padding in nested element is not taken into account.
   bool hasPadding() const { return IsPadded; }
 
   /// Given a valid byte offset into the structure, returns the structure
   /// index that contains it.
   unsigned getElementContainingOffset(uint64_t Offset) const;
 
   uint64_t getElementOffset(unsigned Idx) const {
     assert(Idx < NumElements && "Invalid element idx!");
-    return MemberOffsets[Idx];
+    return MemberOffsets[Idx].Unscaled;
   }
 
   uint64_t getElementOffsetInBits(unsigned Idx) const {
     return getElementOffset(Idx) * 8;
   }
 
 private:
   friend class DataLayout; // Only DataLayout can create this class
case Type::VectorTyID: {
     VectorType *VTy = cast<VectorType>(Ty);
     return VTy->getNumElements() * getTypeSizeInBits(VTy->getElementType());
   }
   default:
     llvm_unreachable("DataLayout::getTypeSizeInBits(): Unsupported type");
   }
 }
 
+inline DataLayout::ScalableSize
+DataLayout::getScalableTypeSizeInBits(Type *Ty) const {
+  switch(Ty->getTypeID()) {
+  default:
+    return {getTypeSizeInBits(Ty), 0ULL};
+  case Type::ArrayTyID: {
+    ArrayType *ATy = cast<ArrayType>(Ty);
+    return getScalableTypeAllocSizeInBits(ATy->getElementType()) *
+           ATy->getNumElements();
+  }
+  case Type::StructTyID:
+    return getStructLayout(cast<StructType>(Ty))->getScalableSizeInBits();
+  case Type::VectorTyID: {
+    VectorType *VTy = cast<VectorType>(Ty);
+    auto EltCnt = VTy->getElementCount();
+    uint64_t MinBits = EltCnt.Min * getTypeSizeInBits(VTy->getElementType());
+    if (EltCnt.Scalable)
+      return {0ULL, MinBits};
+    else
+      return {MinBits, 0ULL};
+  }
+  }
+}
+
 } // end namespace llvm
 
 #endif // LLVM_IR_DATALAYOUT_H

include/llvm/IR/Type.h

public:
   ///
   unsigned getPrimitiveSizeInBits() const LLVM_READONLY;
 
   /// If this is a vector type, return the getPrimitiveSizeInBits value for the
   /// element type. Otherwise return the getPrimitiveSizeInBits value for this
   /// type.
   unsigned getScalarSizeInBits() const LLVM_READONLY;
 
+  struct ScalableSize {
+    unsigned Unscaled;
+    unsigned Scaled;
+
+    ScalableSize(unsigned Unscaled, unsigned Scaled)
+      : Unscaled(Unscaled), Scaled(Scaled) {}
+
+    bool operator==(const ScalableSize& RHS) const {
+      return Unscaled == RHS.Unscaled && Scaled == RHS.Scaled;
+    }
+
+    bool operator!=(const ScalableSize& RHS) const {
+      return Unscaled != RHS.Unscaled || Scaled != RHS.Scaled;
+    }
+
+    bool operator<(const ScalableSize& RHS) const {
+      // Fixed-length fast case
+      if (Scaled == RHS.Scaled)
+        return Unscaled < RHS.Unscaled;
+
+      // Scalable fast case
+      if (Unscaled == RHS.Unscaled)
+        return Scaled < RHS.Scaled;
+
+      // Runtime dependent, no matter the unscaled difference.
+      if (Scaled > RHS.Scaled)
+        return false;
+
+      // We know Scaled < RHS.Scaled now.
+      if (Unscaled <= RHS.Unscaled)
+        return true;
+
+      // {largeU, smallS} vs. {smallU, largeS}
+      // if the unscaled difference is less than the scaled difference,
+      // then this is true. Otherwise it's runtime dependent.
+      auto SDiff = RHS.Scaled - Scaled;
+      auto UDiff = Unscaled - RHS.Unscaled;
+      return UDiff < SDiff;
+    }
+
+    bool operator<=(const ScalableSize& RHS) const {
+      // Fixed-length fast case
+      if (Scaled == RHS.Scaled)
+        return Unscaled <= RHS.Unscaled;
+
+      // Scalable fast case
+      if (Unscaled == RHS.Unscaled)
+        return Scaled <= RHS.Scaled;
+
+      // Runtime dependent, no matter the unscaled difference.
+      if (Scaled > RHS.Scaled)
+        return false;
+
+      // We know Scaled < RHS.Scaled now.
+      if (Unscaled <= RHS.Unscaled)
+        return true;
+
+      // {largeU, smallS} vs. {smallU, largeS}
+      // if the unscaled difference is less than the scaled difference,
+      // then this is true. Otherwise it's runtime dependent.
+      auto SDiff = RHS.Scaled - Scaled;
+      auto UDiff = Unscaled - RHS.Unscaled;
+      return UDiff <= SDiff;
+    }
+
+    bool operator>(const ScalableSize& RHS) const {
+      // Fixed-length fast case
+      if (Scaled == RHS.Scaled)
+        return Unscaled > RHS.Unscaled;
+
+      // Scalable fast case
+      if (Unscaled == RHS.Unscaled)
+        return Scaled > RHS.Scaled;
+
+      // Runtime dependent, no matter the unscaled difference.
+      if (Scaled < RHS.Scaled)
+        return false;
+
+      // We know Scaled > RHS.Scaled now.
+      if (Unscaled >= RHS.Unscaled)
+        return true;
+
+      // {smallU, largeS} vs. {largeU, smallS}
+      // if the unscaled difference is less than the scaled difference,
+      // then this is true. Otherwise it's runtime dependent.
+      auto SDiff = Scaled - RHS.Scaled;
+      auto UDiff = RHS.Unscaled - Unscaled;
+      return UDiff < SDiff;
+    }
+
+    bool operator>=(const ScalableSize& RHS) const {
+      // Fixed-length fast case
+      if (Scaled == RHS.Scaled)
+        return Unscaled >= RHS.Unscaled;
+
+      // Scalable fast case
+      if (Unscaled == RHS.Unscaled)
+        return Scaled >= RHS.Scaled;
+
+      // Runtime dependent, no matter the unscaled difference.
+      if (Scaled < RHS.Scaled)
+        return false;
+
+      // We know Scaled > RHS.Scaled now.
+      if (Unscaled >= RHS.Unscaled)
+        return true;
+
+      // {smallU, largeS} vs. {largeU, smallS}
+      // if the unscaled difference is less than or equal to the scaled
+      // difference, then this is true. Otherwise it's runtime dependent.
+      auto SDiff = Scaled - RHS.Scaled;
+      auto UDiff = RHS.Unscaled - Unscaled;
+      return UDiff <= SDiff;
+    }
+  };
+
+  ScalableSize getScalableSizeInBits() const LLVM_READONLY;
+
   /// Return the width of the mantissa of this type. This is only valid on
   /// floating-point types. If the FP type does not have a stable mantissa (e.g.
   /// ppc long double), this method returns -1.
   int getFPMantissaWidth() const;
 
   /// If this is a vector type, return the element type, otherwise return
   /// 'this'.
   Type *getScalarType() const {

lib/Analysis/ConstantFolding.cpp

if (CE->getOpcode() == Instruction::BitCast)
     if (Constant *LoadedC = ConstantFoldLoadThroughBitcastExpr(CE, Ty, DL))
       return LoadedC;
 
   // Instead of loading constant c string, use corresponding integer value
   // directly if string length is small enough.
   StringRef Str;
   if (getConstantStringInfo(CE, Str) && !Str.empty()) {
     size_t StrLen = Str.size();
-    unsigned NumBits = Ty->getPrimitiveSizeInBits();
+    unsigned NumBits = Ty->getScalableSizeInBits().Unscaled;
     // Replace load with immediate integer if the result is an integer or fp
     // value.
     if ((NumBits >> 3) == StrLen + 1 && (NumBits & 7) == 0 &&
         (isa<IntegerType>(Ty) || Ty->isFloatingPointTy())) {
       APInt StrVal(NumBits, 0);
       APInt SingleChar(NumBits, 0);
       if (DL.isLittleEndian()) {
         for (unsigned char C : reverse(Str.bytes())) {

lib/Analysis/InstructionSimplify.cpp

if (isa<CastInst>(LHS) && (isa<Constant>(RHS) || isa<CastInst>(RHS))) {
     Instruction *LI = cast<CastInst>(LHS);
     Value *SrcOp = LI->getOperand(0);
     Type *SrcTy = SrcOp->getType();
     Type *DstTy = LI->getType();
 
     // Turn icmp (ptrtoint x), (ptrtoint/constant) into a compare of the input
     // if the integer type is the same size as the pointer type.
     if (MaxRecurse && isa<PtrToIntInst>(LI) &&
-        Q.DL.getTypeSizeInBits(SrcTy) == DstTy->getPrimitiveSizeInBits()) {
+        Q.DL.getTypeSizeInBits(SrcTy) == DstTy->getScalableSizeInBits().Unscaled) {
       if (Constant *RHSC = dyn_cast<Constant>(RHS)) {
         // Transfer the cast to the constant.
         if (Value *V = SimplifyICmpInst(Pred, SrcOp,
                                         ConstantExpr::getIntToPtr(RHSC, SrcTy),
                                         Q, MaxRecurse-1))
           return V;
       } else if (PtrToIntInst *RI = dyn_cast<PtrToIntInst>(RHS)) {
         if (RI->getOperand(0)->getType() == SrcTy)

lib/CodeGen/CodeGenPrepare.cpp

static bool despeculateCountZeros(IntrinsicInst *CountZeros,
   // If it's cheap to speculate, there's nothing to do.
   auto IntrinsicID = CountZeros->getIntrinsicID();
   if ((IntrinsicID == Intrinsic::cttz && TLI->isCheapToSpeculateCttz()) ||
       (IntrinsicID == Intrinsic::ctlz && TLI->isCheapToSpeculateCtlz()))
     return false;
 
   // Only handle legal scalar cases. Anything else requires too much work.
   Type *Ty = CountZeros->getType();
-  unsigned SizeInBits = Ty->getPrimitiveSizeInBits();
+  unsigned SizeInBits = Ty->getScalableSizeInBits().Unscaled;
   if (Ty->isVectorTy() || SizeInBits > DL->getLargestLegalIntTypeSizeInBits())
     return false;
 
   // The intrinsic will be sunk behind a compare against zero and branch.
   BasicBlock *StartBlock = CountZeros->getParent();
   BasicBlock *CallBlock = StartBlock->splitBasicBlock(CountZeros, "cond.false");
 
   // Create another block after the count zero intrinsic. A PHI will be added

lib/IR/AutoUpgrade.cpp

NewFn = Intrinsic::getDeclaration(F->getParent(),
                                       Intrinsic::x86_xop_vfrcz_sd);
     return true;
   }
   // Upgrade any XOP PERMIL2 index operand still using a float/double vector.
   if (Name.startswith("xop.vpermil2")) { // Added in 3.9
     auto Idx = F->getFunctionType()->getParamType(2);
     if (Idx->isFPOrFPVectorTy()) {
       rename(F);
-      unsigned IdxSize = Idx->getPrimitiveSizeInBits();
+      unsigned IdxSize = Idx->getScalableSizeInBits().Unscaled;
       unsigned EltSize = Idx->getScalarSizeInBits();
       Intrinsic::ID Permil2ID;
       if (EltSize == 64 && IdxSize == 128)
         Permil2ID = Intrinsic::x86_xop_vpermil2pd;
       else if (EltSize == 32 && IdxSize == 128)
         Permil2ID = Intrinsic::x86_xop_vpermil2ps;
       else if (EltSize == 64 && IdxSize == 256)
         Permil2ID = Intrinsic::x86_xop_vpermil2pd_256;
static Value* UpgradeMaskToInt(IRBuilder<> &Builder, CallInst &CI) {
   return Builder.CreateSExt(Mask, ReturnOp, "vpmovm2");
 }
 
 // Replace intrinsic with unmasked version and a select.
 static bool upgradeAVX512MaskToSelect(StringRef Name, IRBuilder<> &Builder,
                                       CallInst &CI, Value *&Rep) {
   Name = Name.substr(12); // Remove avx512.mask.
 
-  unsigned VecWidth = CI.getType()->getPrimitiveSizeInBits();
+  unsigned VecWidth = CI.getType()->getScalableSizeInBits().Unscaled;
   unsigned EltWidth = CI.getType()->getScalarSizeInBits();
   Intrinsic::ID IID;
   if (Name.startswith("max.p")) {
     if (VecWidth == 128 && EltWidth == 32)
       IID = Intrinsic::x86_sse_max_ps;
     else if (VecWidth == 128 && EltWidth == 64)
       IID = Intrinsic::x86_sse2_max_pd;
     else if (VecWidth == 256 && EltWidth == 32)
if (IsX86 && (Name.startswith("sse2.pcmp") ||
       bool CmpEq = Name[9] == 'e';
       Rep = Builder.CreateICmp(CmpEq ? ICmpInst::ICMP_EQ : ICmpInst::ICMP_SGT,
                                CI->getArgOperand(0), CI->getArgOperand(1));
       Rep = Builder.CreateSExt(Rep, CI->getType(), "");
     } else if (IsX86 && (Name.startswith("avx512.broadcastm"))) {
       Type *ExtTy = Type::getInt32Ty(C);
       if (CI->getOperand(0)->getType()->isIntegerTy(8))
         ExtTy = Type::getInt64Ty(C);
-      unsigned NumElts = CI->getType()->getPrimitiveSizeInBits() /
+      unsigned NumElts = CI->getType()->getScalableSizeInBits().Unscaled /
                          ExtTy->getPrimitiveSizeInBits();
       Rep = Builder.CreateZExt(CI->getArgOperand(0), ExtTy);
       Rep = Builder.CreateVectorSplat(NumElts, Rep);
     } else if (IsX86 && (Name == "sse.sqrt.ss" ||
                          Name == "sse2.sqrt.sd")) {
       Value *Vec = CI->getArgOperand(0);
       Value *Elt0 = Builder.CreateExtractElement(Vec, (uint64_t)0);
       Function *Intr = Intrinsic::getDeclaration(F->getParent(),
if (IsX86 && (Name.startswith("sse2.pcmp") ||
       Rep = Builder.CreateInsertElement(CI->getArgOperand(0), EltOp,
                                         ConstantInt::get(I32Ty, 0));
     } else if (IsX86 && Name.startswith("avx512.mask.pcmp")) {
       // "avx512.mask.pcmpeq." or "avx512.mask.pcmpgt."
       bool CmpEq = Name[16] == 'e';
       Rep = upgradeMaskedCompare(Builder, *CI, CmpEq ? 0 : 6, true);
     } else if (IsX86 && Name.startswith("avx512.mask.fpclass.p")) {
       Type *OpTy = CI->getArgOperand(0)->getType();
-      unsigned VecWidth = OpTy->getPrimitiveSizeInBits();
+      unsigned VecWidth = OpTy->getScalableSizeInBits().Unscaled;
       unsigned EltWidth = OpTy->getScalarSizeInBits();
       Intrinsic::ID IID;
       if (VecWidth == 128 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_fpclass_ps_128;
       else if (VecWidth == 256 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_fpclass_ps_256;
       else if (VecWidth == 512 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_fpclass_ps_512;
       else if (VecWidth == 128 && EltWidth == 64)
         IID = Intrinsic::x86_avx512_fpclass_pd_128;
       else if (VecWidth == 256 && EltWidth == 64)
         IID = Intrinsic::x86_avx512_fpclass_pd_256;
       else if (VecWidth == 512 && EltWidth == 64)
         IID = Intrinsic::x86_avx512_fpclass_pd_512;
       else
         llvm_unreachable("Unexpected intrinsic");
 
       Rep = Builder.CreateCall(Intrinsic::getDeclaration(F->getParent(), IID),
                                { CI->getOperand(0), CI->getArgOperand(1) });
       Rep = ApplyX86MaskOn1BitsVec(Builder, Rep, CI->getArgOperand(2));
     } else if (IsX86 && Name.startswith("avx512.mask.cmp.p")) {
       Type *OpTy = CI->getArgOperand(0)->getType();
-      unsigned VecWidth = OpTy->getPrimitiveSizeInBits();
+      unsigned VecWidth = OpTy->getScalableSizeInBits().Unscaled;
       unsigned EltWidth = OpTy->getScalarSizeInBits();
       Intrinsic::ID IID;
       if (VecWidth == 128 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_cmp_ps_128;
       else if (VecWidth == 256 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_cmp_ps_256;
       else if (VecWidth == 512 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_cmp_ps_512;
if (IsX86 && (Name.startswith("sse2.pcmp") ||
                                           { 0, 1, 0, 1 });
       else
         Rep = Builder.CreateShuffleVector(Load, UndefValue::get(Load->getType()),
                                           { 0, 1, 2, 3, 0, 1, 2, 3 });
     } else if (IsX86 && (Name.startswith("avx512.mask.shuf.i") ||
                          Name.startswith("avx512.mask.shuf.f"))) {
       unsigned Imm = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
       Type *VT = CI->getType();
-      unsigned NumLanes = VT->getPrimitiveSizeInBits() / 128;
+      unsigned NumLanes = VT->getScalableSizeInBits().Unscaled / 128;
       unsigned NumElementsInLane = 128 / VT->getScalarSizeInBits();
       unsigned ControlBitsMask = NumLanes - 1;
       unsigned NumControlBits = NumLanes / 2;
       SmallVector<uint32_t, 8> ShuffleMask(0);
 
       for (unsigned l = 0; l != NumLanes; ++l) {
         unsigned LaneMask = (Imm >> (l * NumControlBits)) & ControlBitsMask;
         // We actually need the other source.

       Value *PassThru = IsMaskZ ? llvm::Constant::getNullValue(CI->getType()) :
                         IsMask3 ? CI->getArgOperand(2) :
                                   CI->getArgOperand(0);
 
       Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
     } else if (IsX86 && (Name.startswith("avx512.mask.pternlog.") ||
                          Name.startswith("avx512.maskz.pternlog."))) {
       bool ZeroMask = Name[11] == 'z';
-      unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
+      unsigned VecWidth = CI->getType()->getScalableSizeInBits().Unscaled;
       unsigned EltWidth = CI->getType()->getScalarSizeInBits();
       Intrinsic::ID IID;
       if (VecWidth == 128 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_pternlog_d_128;
       else if (VecWidth == 256 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_pternlog_d_256;
       else if (VecWidth == 512 && EltWidth == 32)
         IID = Intrinsic::x86_avx512_pternlog_d_512;

                                Args);
       Value *PassThru = ZeroMask ? ConstantAggregateZero::get(CI->getType())
                                  : CI->getArgOperand(0);
       Rep = EmitX86Select(Builder, CI->getArgOperand(4), Rep, PassThru);
     } else if (IsX86 && (Name.startswith("avx512.mask.vpmadd52") ||
                          Name.startswith("avx512.maskz.vpmadd52"))) {
       bool ZeroMask = Name[11] == 'z';
       bool High = Name[20] == 'h' || Name[21] == 'h';
-      unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
+      unsigned VecWidth = CI->getType()->getScalableSizeInBits().Unscaled;
       Intrinsic::ID IID;
       if (VecWidth == 128 && !High)
         IID = Intrinsic::x86_avx512_vpmadd52l_uq_128;
       else if (VecWidth == 256 && !High)
         IID = Intrinsic::x86_avx512_vpmadd52l_uq_256;
       else if (VecWidth == 512 && !High)
         IID = Intrinsic::x86_avx512_vpmadd52l_uq_512;
       else if (VecWidth == 128 && High)

       Value *PassThru = ZeroMask ? ConstantAggregateZero::get(CI->getType())
                                  : CI->getArgOperand(0);
       Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
     } else if (IsX86 && (Name.startswith("avx512.mask.vpermi2var.") ||
                          Name.startswith("avx512.mask.vpermt2var.") ||
                          Name.startswith("avx512.maskz.vpermt2var."))) {
       bool ZeroMask = Name[11] == 'z';
       bool IndexForm = Name[17] == 'i';
-      unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
+      unsigned VecWidth = CI->getType()->getScalableSizeInBits().Unscaled;
       unsigned EltWidth = CI->getType()->getScalarSizeInBits();
       bool IsFloat = CI->getType()->isFPOrFPVectorTy();
       Intrinsic::ID IID;
       if (VecWidth == 128 && EltWidth == 32 && IsFloat)
         IID = Intrinsic::x86_avx512_vpermi2var_ps_128;
       else if (VecWidth == 128 && EltWidth == 32 && !IsFloat)
         IID = Intrinsic::x86_avx512_vpermi2var_d_128;
       else if (VecWidth == 128 && EltWidth == 64 && IsFloat)

                                                          CI->getType());
       Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
     } else if (IsX86 && (Name.startswith("avx512.mask.vpdpbusd.") ||
                          Name.startswith("avx512.maskz.vpdpbusd.") ||
                          Name.startswith("avx512.mask.vpdpbusds.") ||
                          Name.startswith("avx512.maskz.vpdpbusds."))) {
       bool ZeroMask = Name[11] == 'z';
       bool IsSaturating = Name[ZeroMask ? 21 : 20] == 's';
-      unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
+      unsigned VecWidth = CI->getType()->getScalableSizeInBits().Unscaled;
       Intrinsic::ID IID;
       if (VecWidth == 128 && !IsSaturating)
         IID = Intrinsic::x86_avx512_vpdpbusd_128;
       else if (VecWidth == 256 && !IsSaturating)
         IID = Intrinsic::x86_avx512_vpdpbusd_256;
       else if (VecWidth == 512 && !IsSaturating)
         IID = Intrinsic::x86_avx512_vpdpbusd_512;
       else if (VecWidth == 128 && IsSaturating)

                                  : CI->getArgOperand(0);
       Rep = EmitX86Select(Builder, CI->getArgOperand(3), Rep, PassThru);
     } else if (IsX86 && (Name.startswith("avx512.mask.vpdpwssd.") ||
                          Name.startswith("avx512.maskz.vpdpwssd.") ||
                          Name.startswith("avx512.mask.vpdpwssds.") ||
                          Name.startswith("avx512.maskz.vpdpwssds."))) {
       bool ZeroMask = Name[11] == 'z';
       bool IsSaturating = Name[ZeroMask ? 21 : 20] == 's';
-      unsigned VecWidth = CI->getType()->getPrimitiveSizeInBits();
+      unsigned VecWidth = CI->getType()->getScalableSizeInBits().Unscaled;
       Intrinsic::ID IID;
       if (VecWidth == 128 && !IsSaturating)
         IID = Intrinsic::x86_avx512_vpdpwssd_128;
       else if (VecWidth == 256 && !IsSaturating)
         IID = Intrinsic::x86_avx512_vpdpwssd_256;
       else if (VecWidth == 512 && !IsSaturating)
         IID = Intrinsic::x86_avx512_vpdpwssd_512;
       else if (VecWidth == 128 && IsSaturating)

lib/IR/DataLayout.cpp

 
 //===----------------------------------------------------------------------===//
 // Support for StructLayout
 //===----------------------------------------------------------------------===//
 
 StructLayout::StructLayout(StructType *ST, const DataLayout &DL) {
   assert(!ST->isOpaque() && "Cannot get layout of opaque structs");
   StructAlignment = 0;
-  StructSize = 0;
+  StructSize = {0, 0};
   IsPadded = false;
   NumElements = ST->getNumElements();
 
   // Loop over each of the elements, placing them in memory.
   for (unsigned i = 0, e = NumElements; i != e; ++i) {
     Type *Ty = ST->getElementType(i);
+    VectorType *VTy = dyn_cast<VectorType>(Ty);
+    bool IsScalable = VTy && VTy->isScalable();
+
+    // TODO: Figure out what to do for alignment with scalable vectors. If
+    // the scaled bytes are a multiple of the required alignment we're fine,
+    // but if they aren't I don't think we have a way of generating a
+    // simple address at compile time -- we would need to align at runtime.
+    // For now, just insist that all structs containing scalable vectors are
+    // packed.
+    assert((!IsScalable || ST->isPacked()) &&
+           "Unable to determine alignment of unpacked scalable struct");
     unsigned TyAlign = ST->isPacked() ? 1 : DL.getABITypeAlignment(Ty);
 
     // Add padding if necessary to align the data element properly.
-    if ((StructSize & (TyAlign-1)) != 0) {
+    if ((StructSize.Unscaled & (TyAlign-1)) != 0) {
       IsPadded = true;
-      StructSize = alignTo(StructSize, TyAlign);
+      StructSize = {alignTo(StructSize.Unscaled, TyAlign), 0};
     }
 
     // Keep track of maximum alignment constraint.
     StructAlignment = std::max(TyAlign, StructAlignment);
 
     MemberOffsets[i] = StructSize;
-    StructSize += DL.getTypeAllocSize(Ty); // Consume space for this data item
+
+    // Consume space for this data item
+    StructSize += DL.getScalableTypeAllocSize(Ty);
   }
 
   // Empty structures have alignment of 1 byte.
   if (StructAlignment == 0) StructAlignment = 1;
 
   // Add padding to the end of the struct so that it could be put in an array
   // and all array elements would be aligned correctly.
-  if ((StructSize & (StructAlignment-1)) != 0) {
+  if ((StructSize.Unscaled & (StructAlignment-1)) != 0) {
     IsPadded = true;
-    StructSize = alignTo(StructSize, StructAlignment);
+    StructSize = {alignTo(StructSize.Unscaled, StructAlignment), 0};
   }
 }
 
 /// getElementContainingOffset - Given a valid offset into the structure,
 /// return the structure index that contains it.
 unsigned StructLayout::getElementContainingOffset(uint64_t Offset) const {
-  const uint64_t *SI =
-    std::upper_bound(&MemberOffsets[0], &MemberOffsets[NumElements], Offset);
+  const DataLayout::ScalableSize SOffset(Offset, 0ULL);
+  const DataLayout::ScalableSize *SI =
+    std::upper_bound(&MemberOffsets[0], &MemberOffsets[NumElements], SOffset);
   assert(SI != &MemberOffsets[0] && "Offset not in structure type!");
   --SI;
-  assert(*SI <= Offset && "upper_bound didn't work");
-  assert((SI == &MemberOffsets[0] || *(SI-1) <= Offset) &&
-         (SI+1 == &MemberOffsets[NumElements] || *(SI+1) > Offset) &&
+  assert(*SI <= SOffset && "upper_bound didn't work");
+  assert((SI == &MemberOffsets[0] || *(SI-1) <= SOffset) &&
+         (SI+1 == &MemberOffsets[NumElements] || *(SI+1) > SOffset) &&
          "Upper bound didn't work!");
 
   // Multiple fields can have the same offset if any of them are zero sized.
   // For example, in { i32, [0 x i32], i32 }, searching for offset 4 will stop
   // at the i32 element, because it is the last element at that offset.  This is
   // the right one to return, because anything after it will have a higher
   // offset, implying that this element is non-empty.
   return SI-&MemberOffsets[0];
const StructLayout *DataLayout::getStructLayout(StructType *Ty) const {
   StructLayoutMap *STM = static_cast<StructLayoutMap*>(LayoutMap);
   StructLayout *&SL = (*STM)[Ty];
   if (SL) return SL;
 
   // Otherwise, create the struct layout.  Because it is variable length, we
   // malloc it, then use placement new.
   int NumElts = Ty->getNumElements();
   StructLayout *L = (StructLayout *)
-      safe_malloc(sizeof(StructLayout)+(NumElts-1) * sizeof(uint64_t));
+      safe_malloc(sizeof(StructLayout) +
+                  (NumElts-1) * sizeof(DataLayout::ScalableSize));
 
   // Set SL before calling StructLayout's ctor.  The ctor could cause other
   // entries to be added to TheMap, invalidating our reference.
   SL = L;
 
   new (L) StructLayout(Ty, *this);
 
   return L;

lib/IR/Instructions.cpp

if (VectorType *SrcVecTy = dyn_cast<VectorType>(SrcTy))
     if (VectorType *DestVecTy = dyn_cast<VectorType>(DestTy))
       if (SrcVecTy->getNumElements() == DestVecTy->getNumElements()) {
         // An element by element cast.  Valid if casting the elements is valid.
         SrcTy = SrcVecTy->getElementType();
         DestTy = DestVecTy->getElementType();
       }
 
   // Get the bit sizes, we'll need these
-  unsigned SrcBits = SrcTy->getPrimitiveSizeInBits();   // 0 for ptr
-  unsigned DestBits = DestTy->getPrimitiveSizeInBits(); // 0 for ptr
+  auto SrcBits = SrcTy->getScalableSizeInBits();   // 0 for ptr
+  auto DestBits = DestTy->getScalableSizeInBits(); // 0 for ptr
 
   // Run through the possibilities ...
   if (DestTy->isIntegerTy()) {               // Casting to integral
     if (SrcTy->isIntegerTy())                // Casting from integral
         return true;
     if (SrcTy->isFloatingPointTy())   // Casting from floating pt
       return true;
     if (SrcTy->isVectorTy())          // Casting from vector
bool CastInst::isBitCastable(Type *SrcTy, Type *DestTy) {
   }
 
   if (PointerType *DestPtrTy = dyn_cast<PointerType>(DestTy)) {
     if (PointerType *SrcPtrTy = dyn_cast<PointerType>(SrcTy)) {
       return SrcPtrTy->getAddressSpace() == DestPtrTy->getAddressSpace();
     }
   }
 
-  unsigned SrcBits = SrcTy->getPrimitiveSizeInBits();   // 0 for ptr
-  unsigned DestBits = DestTy->getPrimitiveSizeInBits(); // 0 for ptr
+  auto SrcBits = SrcTy->getScalableSizeInBits();   // 0 for ptr
+  auto DestBits = DestTy->getScalableSizeInBits(); // 0 for ptr
 
   // Could still have vectors of pointers if the number of elements doesn't
   // match
-  if (SrcBits == 0 || DestBits == 0)
+  if (SrcBits.Unscaled == 0 || DestBits.Unscaled == 0)
     return false;
 
   if (SrcBits != DestBits)
     return false;
 
   if (DestTy->isX86_MMXTy() || SrcTy->isX86_MMXTy())
     return false;
 
case Instruction::BitCast: {
     // BitCast implies a no-op cast of type only. No bits change.
     // However, you can't cast pointers to anything but pointers.
     if (!SrcPtrTy != !DstPtrTy)
       return false;
 
     // For non-pointer cases, the cast is okay if the source and destination bit
     // widths are identical.
     if (!SrcPtrTy)
-      return SrcTy->getPrimitiveSizeInBits() == DstTy->getPrimitiveSizeInBits();
+      return SrcTy->getScalableSizeInBits() ==
+        DstTy->getScalableSizeInBits();
 
     // If both are pointers then the address spaces must match.
     if (SrcPtrTy->getAddressSpace() != DstPtrTy->getAddressSpace())
       return false;
 
     // A vector of pointers must have the same number of elements.
     VectorType *SrcVecTy = dyn_cast<VectorType>(SrcTy);
     VectorType *DstVecTy = dyn_cast<VectorType>(DstTy);

lib/IR/Type.cpp

unsigned Type::getPrimitiveSizeInBits() const {
   case Type::HalfTyID: return 16;
   case Type::FloatTyID: return 32;
   case Type::DoubleTyID: return 64;
   case Type::X86_FP80TyID: return 80;
   case Type::FP128TyID: return 128;
   case Type::PPC_FP128TyID: return 128;
   case Type::X86_MMXTyID: return 64;
   case Type::IntegerTyID: return cast<IntegerType>(this)->getBitWidth();
-  case Type::VectorTyID:  return cast<VectorType>(this)->getBitWidth();
+  case Type::VectorTyID: return 0; // TODO: Just an assert for scalable vectors?
   default: return 0;
   }
 }
 
 unsigned Type::getScalarSizeInBits() const {
   return getScalarType()->getPrimitiveSizeInBits();
 }
 
+Type::ScalableSize Type::getScalableSizeInBits() const {
+  auto ID = getTypeID();
+  if (ID == Type::HalfTyID || ID == Type::FloatTyID || ID == Type::DoubleTyID ||
+      ID == Type::X86_FP80TyID || ID == Type::FP128TyID ||
+      ID == Type::PPC_FP128TyID || ID == Type::X86_MMXTyID ||
+      ID == Type::IntegerTyID)
+      return { getPrimitiveSizeInBits(), 0 };
+
+  if (auto *VTy = dyn_cast<VectorType>(this)) {
+    if (VTy->isScalable())
+      return { 0, VTy->getBitWidth() };
+    else
+      return { VTy->getBitWidth(), 0 };
+  }
+
+  return {0, 0};
+}
+
 int Type::getFPMantissaWidth() const {
   if (auto *VTy = dyn_cast<VectorType>(this))
     return VTy->getElementType()->getFPMantissaWidth();
   assert(isFloatingPointTy() && "Not a floating point type!");
   if (getTypeID() == HalfTyID) return 11;
   if (getTypeID() == FloatTyID) return 24;
   if (getTypeID() == DoubleTyID) return 53;
   if (getTypeID() == X86_FP80TyID) return 64;

lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

void AMDGPUAsmPrinter::getSIProgramInfo(SIProgramInfo &ProgInfo,
 
   // Account for extra SGPRs and VGPRs reserved for debugger use.
   ProgInfo.NumSGPR += ExtraSGPRs;
 
   // Ensure there are enough SGPRs and VGPRs for wave dispatch, where wave
   // dispatch registers are function args.
   unsigned WaveDispatchNumSGPR = 0, WaveDispatchNumVGPR = 0;
   for (auto &Arg : MF.getFunction().args()) {
-    unsigned NumRegs = (Arg.getType()->getPrimitiveSizeInBits() + 31) / 32;
+    unsigned NumRegs = (Arg.getType()->getScalableSizeInBits().Unscaled + 31) / 32;
     if (Arg.hasAttribute(Attribute::InReg))
       WaveDispatchNumSGPR += NumRegs;
     else
       WaveDispatchNumVGPR += NumRegs;
   }
   ProgInfo.NumSGPR = std::max(ProgInfo.NumSGPR, WaveDispatchNumSGPR);
   ProgInfo.NumVGPR = std::max(ProgInfo.NumVGPR, WaveDispatchNumVGPR);
 

lib/Target/PowerPC/PPCTargetTransformInfo.cpp

bool IsVSXType = ST->hasVSX() &&
                    (LT.second == MVT::v2f64 || LT.second == MVT::v2i64);
   bool IsQPXType = ST->hasQPX() &&
                    (LT.second == MVT::v4f64 || LT.second == MVT::v4f32);
 
   // VSX has 32b/64b load instructions. Legalization can handle loading of
   // 32b/64b to VSR correctly and cheaply. But BaseT::getMemoryOpCost and
   // PPCTargetLowering can't compute the cost appropriately. So here we
   // explicitly check this case.
-  unsigned MemBytes = Src->getPrimitiveSizeInBits();
+  unsigned MemBytes = Src->getScalableSizeInBits().Unscaled;
   if (Opcode == Instruction::Load && ST->hasVSX() && IsAltivecType &&
       (MemBytes == 64 || (ST->hasP8Vector() && MemBytes == 32)))
     return 1;
 
   // Aligned loads and stores are easy.
   unsigned SrcBytes = LT.second.getStoreSize();
   if (!SrcBytes || !Alignment || Alignment >= SrcBytes)
     return Cost;

lib/Target/SystemZ/SystemZTargetTransformInfo.cpp

 //
 // SystemZ cost model.
 //
 //===----------------------------------------------------------------------===//
 
 int SystemZTTIImpl::getIntImmCost(const APInt &Imm, Type *Ty) {
   assert(Ty->isIntegerTy());
 
-  unsigned BitSize = Ty->getPrimitiveSizeInBits();
+  unsigned BitSize = Ty->getScalableSizeInBits().Unscaled;
   // There is no cost model for constants with a bit size of 0. Return TCC_Free
   // here, so that constant hoisting will ignore this constant.
   if (BitSize == 0)
     return TTI::TCC_Free;
   // No cost model for operations on integers larger than 64 bit implemented yet.
   if (BitSize > 64)
     return TTI::TCC_Free;
 
int SystemZTTIImpl::getIntImmCost(const APInt &Imm, Type *Ty) {
 
   return 4 * TTI::TCC_Basic;
 }
 
 int SystemZTTIImpl::getIntImmCost(unsigned Opcode, unsigned Idx,
                                   const APInt &Imm, Type *Ty) {
   assert(Ty->isIntegerTy());
 
-  unsigned BitSize = Ty->getPrimitiveSizeInBits();
+  unsigned BitSize = Ty->getScalableSizeInBits().Unscaled;
   // There is no cost model for constants with a bit size of 0. Return TCC_Free
   // here, so that constant hoisting will ignore this constant.
   if (BitSize == 0)
     return TTI::TCC_Free;
   // No cost model for operations on integers larger than 64 bit implemented yet.
   if (BitSize > 64)
     return TTI::TCC_Free;
 
int SystemZTTIImpl::getIntImmCost(unsigned Opcode, unsigned Idx,
 
   return SystemZTTIImpl::getIntImmCost(Imm, Ty);
 }
 
 int SystemZTTIImpl::getIntImmCost(Intrinsic::ID IID, unsigned Idx,
                                   const APInt &Imm, Type *Ty) {
   assert(Ty->isIntegerTy());
 
-  unsigned BitSize = Ty->getPrimitiveSizeInBits();
+  unsigned BitSize = Ty->getScalableSizeInBits().Unscaled;
   // There is no cost model for constants with a bit size of 0. Return TCC_Free
   // here, so that constant hoisting will ignore this constant.
   if (BitSize == 0)
     return TTI::TCC_Free;
   // No cost model for operations on integers larger than 64 bit implemented yet.
   if (BitSize > 64)
     return TTI::TCC_Free;
 
static unsigned getElSizeLog2Diff(Type *Ty0, Type *Ty1) {
 
   return (Log2_32(Bits0) - Log2_32(Bits1));
 }
 
 // Return the number of instructions needed to truncate SrcTy to DstTy.
 unsigned SystemZTTIImpl::
 getVectorTruncCost(Type *SrcTy, Type *DstTy) {
   assert (SrcTy->isVectorTy() && DstTy->isVectorTy());
-  assert (SrcTy->getPrimitiveSizeInBits() > DstTy->getPrimitiveSizeInBits() &&
+  assert (SrcTy->getScalableSizeInBits() > DstTy->getScalableSizeInBits() &&
           "Packing must reduce size of vector type.");
   assert (SrcTy->getVectorNumElements() == DstTy->getVectorNumElements() &&
           "Packing should not change number of elements.");
 
   // TODO: Since fp32 is expanded, the extract cost should always be 0.
 
   unsigned NumParts = getNumVectorRegs(SrcTy);
   if (NumParts <= 2)

lib/Target/X86/X86ISelLowering.cpp

for (unsigned i = 0, e = Op.getNumOperands(); i != e; ++i) {
       SrcEltBits[i] = RawBits.zextOrTrunc(SrcEltSizeInBits);
     }
     return CastBitData(UndefSrcElts, SrcEltBits);
   }
 
   // Extract constant bits from constant pool vector.
   if (auto *Cst = getTargetConstantFromNode(Op)) {
     Type *CstTy = Cst->getType();
-    if (!CstTy->isVectorTy() || (SizeInBits != CstTy->getPrimitiveSizeInBits()))
+    if (!CstTy->isVectorTy() ||
+        (CstTy->getScalableSizeInBits().Unscaled != SizeInBits))
       return false;
 
     unsigned SrcEltSizeInBits = CstTy->getScalarSizeInBits();
     unsigned NumSrcElts = CstTy->getVectorNumElements();
 
     APInt UndefSrcElts(NumSrcElts, 0);
     SmallVector<APInt, 64> SrcEltBits(NumSrcElts, APInt(SrcEltSizeInBits, 0));
     for (unsigned i = 0; i != NumSrcElts; ++i)
bool X86TargetLowering::isVectorShiftByScalarCheap(Type *Ty) const {
   unsigned Bits = Ty->getScalarSizeInBits();
 
   // 8-bit shifts are always expensive, but versions with a scalar amount aren't
   // particularly cheaper than those without.
   if (Bits == 8)
     return false;
 
   // XOP has v16i8/v8i16/v4i32/v2i64 variable vector shifts.
-  if (Subtarget.hasXOP() && Ty->getPrimitiveSizeInBits() == 128 &&
+  if (Subtarget.hasXOP() && Ty->getScalableSizeInBits().Unscaled == 128 &&
       (Bits == 8 || Bits == 16 || Bits == 32 || Bits == 64))
     return false;
 
   // AVX2 has vpsllv[dq] instructions (and other shifts) that make variable
   // shifts just as cheap as scalar ones.
   if (Subtarget.hasAVX2() && (Bits == 32 || Bits == 64))
     return false;
 

lib/Target/X86/X86ShuffleDecodeConstantPool.cpp

static bool extractConstantMask(const Constant *C, unsigned MaskEltSizeInBits,
   Type *CstTy = C->getType();
   if (!CstTy->isVectorTy())
     return false;
 
   Type *CstEltTy = CstTy->getVectorElementType();
   if (!CstEltTy->isIntegerTy())
     return false;
 
-  unsigned CstSizeInBits = CstTy->getPrimitiveSizeInBits();
+  unsigned CstSizeInBits = CstTy->getScalableSizeInBits().Unscaled;
   unsigned CstEltSizeInBits = CstTy->getScalarSizeInBits();
   unsigned NumCstElts = CstTy->getVectorNumElements();
 
   assert((CstSizeInBits % MaskEltSizeInBits) == 0 &&
          "Unaligned shuffle mask size");
 
   unsigned NumMaskElts = CstSizeInBits / MaskEltSizeInBits;
   UndefElts = APInt(NumMaskElts, 0);
for (unsigned i = 0; i != NumMaskElts; ++i) {
     RawMask[i] = EltBits.getZExtValue();
   }
 
   return true;
 }
 
 void DecodePSHUFBMask(const Constant *C, SmallVectorImpl<int> &ShuffleMask) {
   Type *MaskTy = C->getType();
-  unsigned MaskTySize = MaskTy->getPrimitiveSizeInBits();
+  unsigned MaskTySize = MaskTy->getScalableSizeInBits().Unscaled;
   (void)MaskTySize;
   assert((MaskTySize == 128 || MaskTySize == 256 || MaskTySize == 512) &&
          "Unexpected vector size.");
 
   // The shuffle mask requires a byte vector.
   APInt UndefElts;
   SmallVector<uint64_t, 64> RawMask;
   if (!extractConstantMask(C, 8, UndefElts, RawMask))
else {
       ShuffleMask.push_back(Index);
     }
   }
 }
 
 void DecodeVPERMILPMask(const Constant *C, unsigned ElSize,
                         SmallVectorImpl<int> &ShuffleMask) {
   Type *MaskTy = C->getType();
-  unsigned MaskTySize = MaskTy->getPrimitiveSizeInBits();
+  unsigned MaskTySize = MaskTy->getScalableSizeInBits().Unscaled;
   (void)MaskTySize;
   assert((MaskTySize == 128 || MaskTySize == 256 || MaskTySize == 512) &&
          "Unexpected vector size.");
   assert((ElSize == 32 || ElSize == 64) && "Unexpected vector element size.");
 
   // The shuffle mask requires elements the same size as the target.
   APInt UndefElts;
   SmallVector<uint64_t, 16> RawMask;
for (unsigned i = 0; i != NumElts; ++i) {
 
     ShuffleMask.push_back(Index);
   }
 }
 
 void DecodeVPERMIL2PMask(const Constant *C, unsigned M2Z, unsigned ElSize,
                          SmallVectorImpl<int> &ShuffleMask) {
   Type *MaskTy = C->getType();
-  unsigned MaskTySize = MaskTy->getPrimitiveSizeInBits();
+  unsigned MaskTySize = MaskTy->getScalableSizeInBits().Unscaled;
   (void)MaskTySize;
   assert((MaskTySize == 128 || MaskTySize == 256) && "Unexpected vector size.");
 
   // The shuffle mask requires elements the same size as the target.
   APInt UndefElts;
   SmallVector<uint64_t, 8> RawMask;
   if (!extractConstantMask(C, ElSize, UndefElts, RawMask))
     return;
for (unsigned i = 0; i != NumElts; ++i) {
 
     int Src = (Selector >> 2) & 0x1;
     Index += Src * NumElts;
     ShuffleMask.push_back(Index);
   }
 }
 
 void DecodeVPPERMMask(const Constant *C, SmallVectorImpl<int> &ShuffleMask) {
-  assert(C->getType()->getPrimitiveSizeInBits() == 128 &&
+  assert(C->getType()->getScalableSizeInBits().Unscaled == 128 &&
          "Unexpected vector size.");
 
   // The shuffle mask requires a byte vector.
   APInt UndefElts;
   SmallVector<uint64_t, 16> RawMask;
   if (!extractConstantMask(C, 8, UndefElts, RawMask))
     return;
 
for (unsigned i = 0; i != NumElts; ++i) {
     }
     ShuffleMask.push_back((int)Index);
   }
 }
 
 void DecodeVPERMVMask(const Constant *C, unsigned ElSize,
                       SmallVectorImpl<int> &ShuffleMask) {
   Type *MaskTy = C->getType();
-  unsigned MaskTySize = MaskTy->getPrimitiveSizeInBits();
+  unsigned MaskTySize = MaskTy->getScalableSizeInBits().Unscaled;
   (void)MaskTySize;
   assert((MaskTySize == 128 || MaskTySize == 256 || MaskTySize == 512) &&
          "Unexpected vector size.");
   assert((ElSize == 8 || ElSize == 16 || ElSize == 32 || ElSize == 64) &&
          "Unexpected vector element size.");
 
   // The shuffle mask requires elements the same size as the target.
   APInt UndefElts;
for (unsigned i = 0; i != NumElts; ++i) {
     int Index = RawMask[i] & (NumElts - 1);
     ShuffleMask.push_back(Index);
   }
 }
 
 void DecodeVPERMV3Mask(const Constant *C, unsigned ElSize,
                        SmallVectorImpl<int> &ShuffleMask) {
   Type *MaskTy = C->getType();
-  unsigned MaskTySize = MaskTy->getPrimitiveSizeInBits();
+  unsigned MaskTySize = MaskTy->getScalableSizeInBits().Unscaled;
   (void)MaskTySize;
   assert((MaskTySize == 128 || MaskTySize == 256 || MaskTySize == 512) &&
          "Unexpected vector size.");
   assert((ElSize == 8 || ElSize == 16 || ElSize == 32 || ElSize == 64) &&
          "Unexpected vector element size.");
 
   // The shuffle mask requires elements the same size as the target.
   APInt UndefElts;

lib/Transforms/InstCombine/InstCombineCalls.cpp

static Value *simplifyX86immShift(const IntrinsicInst &II,
   if (!CAZ && !CDV && !CInt)
     return nullptr;
 
   APInt Count(64, 0);
   if (CDV) {
     // SSE2/AVX2 uses all the first 64-bits of the 128-bit vector
     // operand to compute the shift amount.
     auto VT = cast<VectorType>(CDV->getType());
-    unsigned BitWidth = VT->getElementType()->getPrimitiveSizeInBits();
+    unsigned BitWidth = VT->getElementType()->getScalableSizeInBits().Unscaled;
     assert((64 % BitWidth) == 0 && "Unexpected packed shift size");
     unsigned NumSubElts = 64 / BitWidth;
 
     // Concatenate the sub-elements to create the 64-bit value.
     for (unsigned i = 0; i != NumSubElts; ++i) {
       unsigned SubEltIdx = (NumSubElts - 1) - i;
       auto SubElt = cast<ConstantInt>(CDV->getElementAsConstant(SubEltIdx));
       Count <<= BitWidth;
       Count |= SubElt->getValue().zextOrTrunc(64);
     }
   }
   else if (CInt)
     Count = CInt->getValue();
 
   auto Vec = II.getArgOperand(0);
   auto VT = cast<VectorType>(Vec->getType());
   auto SVT = VT->getElementType();
   unsigned VWidth = VT->getNumElements();
-  unsigned BitWidth = SVT->getPrimitiveSizeInBits();
+  unsigned BitWidth = SVT->getScalableSizeInBits().Unscaled;
 
   // If shift-by-zero then just return the original value.
   if (Count.isNullValue())
     return Vec;
 
   // Handle cases when Shift >= BitWidth.
   if (Count.uge(BitWidth)) {
     // If LogicalShift - just return zero.
static Value *simplifyX86pack(IntrinsicInst &II, bool IsSigned) {
   Value *Arg1 = II.getArgOperand(1);
   Type *ResTy = II.getType();
 
   // Fast all undef handling.
   if (isa<UndefValue>(Arg0) && isa<UndefValue>(Arg1))
     return UndefValue::get(ResTy);
 
   Type *ArgTy = Arg0->getType();
-  unsigned NumLanes = ResTy->getPrimitiveSizeInBits() / 128;
+  unsigned NumLanes = ResTy->getScalableSizeInBits().Unscaled / 128;
   unsigned NumDstElts = ResTy->getVectorNumElements();
   unsigned NumSrcElts = ArgTy->getVectorNumElements();
   assert(NumDstElts == (2 * NumSrcElts) && "Unexpected packing types");
 
   unsigned NumDstEltsPerLane = NumDstElts / NumLanes;
   unsigned NumSrcEltsPerLane = NumSrcElts / NumLanes;
   unsigned DstScalarSizeInBits = ResTy->getScalarSizeInBits();
   assert(ArgTy->getScalarSizeInBits() == (2 * DstScalarSizeInBits) &&
Instruction *InstCombiner::visitCallInst(CallInst &CI) {
   case Intrinsic::x86_avx512_psll_q_512:
   case Intrinsic::x86_avx512_psll_w_512: {
     if (Value *V = simplifyX86immShift(*II, Builder))
       return replaceInstUsesWith(*II, V);
 
     // SSE2/AVX2 uses only the first 64-bits of the 128-bit vector
     // operand to compute the shift amount.
     Value *Arg1 = II->getArgOperand(1);
-    assert(Arg1->getType()->getPrimitiveSizeInBits() == 128 &&
+    assert(Arg1->getType()->getScalableSizeInBits().Unscaled == 128 &&
            "Unexpected packed shift size");
     unsigned VWidth = Arg1->getType()->getVectorNumElements();
 
     if (Value *V = SimplifyDemandedVectorEltsLow(Arg1, VWidth, VWidth / 2)) {
       II->setArgOperand(1, V);
       return II;
     }
     break;
if (Value *V = simplifyX86insertps(*II, Builder))
       return replaceInstUsesWith(*II, V);
     break;
 
   case Intrinsic::x86_sse4a_extrq: {
     Value *Op0 = II->getArgOperand(0);
     Value *Op1 = II->getArgOperand(1);
     unsigned VWidth0 = Op0->getType()->getVectorNumElements();
     unsigned VWidth1 = Op1->getType()->getVectorNumElements();
-    assert(Op0->getType()->getPrimitiveSizeInBits() == 128 &&
-           Op1->getType()->getPrimitiveSizeInBits() == 128 && VWidth0 == 2 &&
+    assert(Op0->getType()->getScalableSizeInBits().Unscaled == 128 &&
+             Op1->getType()->getScalableSizeInBits().Unscaled == 128 && VWidth0 == 2 &&
            VWidth1 == 16 && "Unexpected operand sizes");
 
     // See if we're dealing with constant values.
     Constant *C1 = dyn_cast<Constant>(Op1);
     ConstantInt *CILength =
         C1 ? dyn_cast_or_null<ConstantInt>(C1->getAggregateElement((unsigned)0))
            : nullptr;
     ConstantInt *CIIndex =
case Intrinsic::x86_sse4a_extrq: {
     break;
   }
 
   case Intrinsic::x86_sse4a_extrqi: {
     // EXTRQI: Extract Length bits starting from Index. Zero pad the remaining
     // bits of the lower 64-bits. The upper 64-bits are undefined.
     Value *Op0 = II->getArgOperand(0);
     unsigned VWidth = Op0->getType()->getVectorNumElements();
-    assert(Op0->getType()->getPrimitiveSizeInBits() == 128 && VWidth == 2 &&
-           "Unexpected operand size");
+    assert(Op0->getType()->getScalableSizeInBits().Unscaled == 128 &&
+           VWidth == 2 && "Unexpected operand size");
 
     // See if we're dealing with constant values.
     ConstantInt *CILength = dyn_cast<ConstantInt>(II->getArgOperand(1));
     ConstantInt *CIIndex = dyn_cast<ConstantInt>(II->getArgOperand(2));
 
     // Attempt to simplify to a constant or shuffle vector.
     if (Value *V = simplifyX86extrq(*II, Op0, CILength, CIIndex, Builder))
       return replaceInstUsesWith(*II, V);
 
     // EXTRQI only uses the lowest 64-bits of the first 128-bit vector
     // operand.
     if (Value *V = SimplifyDemandedVectorEltsLow(Op0, VWidth, 1)) {
       II->setArgOperand(0, V);
       return II;
     }
     break;
   }
 
   case Intrinsic::x86_sse4a_insertq: {
     Value *Op0 = II->getArgOperand(0);
     Value *Op1 = II->getArgOperand(1);
     unsigned VWidth = Op0->getType()->getVectorNumElements();
-    assert(Op0->getType()->getPrimitiveSizeInBits() == 128 &&
-           Op1->getType()->getPrimitiveSizeInBits() == 128 && VWidth == 2 &&
-           Op1->getType()->getVectorNumElements() == 2 &&
+    assert(Op0->getType()->getScalableSizeInBits().Unscaled == 128 &&
+             Op1->getType()->getScalableSizeInBits().Unscaled == 128 &&
+           VWidth == 2 && Op1->getType()->getVectorNumElements() == 2 &&
            "Unexpected operand size");
 
     // See if we're dealing with constant values.
     Constant *C1 = dyn_cast<Constant>(Op1);
     ConstantInt *CI11 =
         C1 ? dyn_cast_or_null<ConstantInt>(C1->getAggregateElement((unsigned)1))
            : nullptr;
 

   case Intrinsic::x86_sse4a_insertqi: {
     // INSERTQI: Extract lowest Length bits from lower half of second source and
     // insert over first source starting at Index bit. The upper 64-bits are
     // undefined.
     Value *Op0 = II->getArgOperand(0);
     Value *Op1 = II->getArgOperand(1);
     unsigned VWidth0 = Op0->getType()->getVectorNumElements();
     unsigned VWidth1 = Op1->getType()->getVectorNumElements();
-    assert(Op0->getType()->getPrimitiveSizeInBits() == 128 &&
-           Op1->getType()->getPrimitiveSizeInBits() == 128 && VWidth0 == 2 &&
-           VWidth1 == 2 && "Unexpected operand sizes");
+    assert(Op0->getType()->getScalableSizeInBits().Unscaled == 128 &&
+             Op1->getType()->getScalableSizeInBits().Unscaled == 128 &&
+           VWidth0 == 2 && VWidth1 == 2 && "Unexpected operand sizes");
 
     // See if we're dealing with constant values.
     ConstantInt *CILength = dyn_cast<ConstantInt>(II->getArgOperand(2));
     ConstantInt *CIIndex = dyn_cast<ConstantInt>(II->getArgOperand(3));
 
     // Attempt to simplify to a constant or shuffle vector.
     if (CILength && CIIndex) {
       APInt Len = CILength->getValue().zextOrTrunc(6);
case Intrinsic::x86_avx2_pblendvb: {
 
     // Convert to a vector select if we can bypass casts and find a boolean
     // vector condition value.
     Value *BoolVec;
     Mask = peekThroughBitcast(Mask);
     if (match(Mask, m_SExt(m_Value(BoolVec))) &&
         BoolVec->getType()->isVectorTy() &&
         BoolVec->getType()->getScalarSizeInBits() == 1) {
-      assert(Mask->getType()->getPrimitiveSizeInBits() ==
-             II->getType()->getPrimitiveSizeInBits() &&
+      assert(Mask->getType()->getScalableSizeInBits() ==
+               II->getType()->getScalableSizeInBits()&&
              "Not expecting mask and operands with different sizes");
 
       unsigned NumMaskElts = Mask->getType()->getVectorNumElements();
       unsigned NumOperandElts = II->getType()->getVectorNumElements();
       if (NumMaskElts == NumOperandElts)
         return SelectInst::Create(BoolVec, Op1, Op0);
 
       // If the mask has less elements than the operands, each mask bit maps to

lib/Transforms/InstCombine/InstCombineCasts.cpp

static Instruction *foldVecTruncToExtElt(TruncInst &Trunc, InstCombiner &IC) {
   ConstantInt *ShiftVal = nullptr;
   if (!match(TruncOp, m_CombineOr(m_BitCast(m_Value(VecInput)),
                                   m_LShr(m_BitCast(m_Value(VecInput)),
                                          m_ConstantInt(ShiftVal)))) ||
       !isa<VectorType>(VecInput->getType()))
     return nullptr;
 
   VectorType *VecType = cast<VectorType>(VecInput->getType());
-  unsigned VecWidth = VecType->getPrimitiveSizeInBits();
+  unsigned VecWidth = VecType->getScalableSizeInBits().Unscaled;
   unsigned DestWidth = DestType->getPrimitiveSizeInBits();
   unsigned ShiftAmount = ShiftVal ? ShiftVal->getZExtValue() : 0;
 
   if ((VecWidth % DestWidth != 0) || (ShiftAmount % DestWidth != 0))
     return nullptr;
 
   // If the element type of the vector doesn't match the result type,
   // bitcast it to a vector type that we can extract from.

lib/Transforms/InstCombine/InstCombineSimplifyDemanded.cpp

case Instruction::Call: {
     case Intrinsic::x86_avx512_packssdw_512:
     case Intrinsic::x86_avx512_packsswb_512:
     case Intrinsic::x86_avx512_packusdw_512:
     case Intrinsic::x86_avx512_packuswb_512: {
       auto *Ty0 = II->getArgOperand(0)->getType();
       unsigned InnerVWidth = Ty0->getVectorNumElements();
       assert(VWidth == (InnerVWidth * 2) && "Unexpected input size");
 
-      unsigned NumLanes = Ty0->getPrimitiveSizeInBits() / 128;
+      unsigned NumLanes = Ty0->getScalableSizeInBits().Unscaled / 128;
       unsigned VWidthPerLane = VWidth / NumLanes;
       unsigned InnerVWidthPerLane = InnerVWidth / NumLanes;
 
       // Per lane, pack the elements of the first input and then the second.
       // e.g.
       // v8i16 PACK(v4i32 X, v4i32 Y) - (X[0..3],Y[0..3])
       // v32i8 PACK(v16i16 X, v16i16 Y) - (X[0..7],Y[0..7]),(X[8..15],Y[8..15])
       for (int OpNum = 0; OpNum != 2; ++OpNum) {

unittests/IR/VectorTypesTest.cpp

 //===--- llvm/unittest/IR/VectorTypesTest.cpp - vector types unit tests ---===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
+#include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/LLVMContext.h"
+#include "llvm/IR/Type.h"
 #include "gtest/gtest.h"
 using namespace llvm;
 
 namespace {
 TEST(VectorTypesTest, FixedLength) {
   LLVMContext Ctx;
 
   Type *Int16Ty = Type::getInt16Ty(Ctx);
TEST(VectorTypesTest, Scalable) {
   ASSERT_TRUE(ScV4Int64Ty->isScalable());
   VectorType *ScV2Int64Ty = VectorType::get(Int64Ty, EltCnt/2);
   ASSERT_TRUE(ScV2Int64Ty->isScalable());
   VectorType *ScV8Int64Ty = VectorType::get(Int64Ty, EltCnt*2);
   ASSERT_TRUE(ScV8Int64Ty->isScalable());
   VectorType *ScV4Float64Ty = VectorType::get(Float64Ty, EltCnt);
   ASSERT_TRUE(ScV4Float64Ty->isScalable());
 
-
   EXPECT_EQ(VectorType::getExtendedElementVectorType(ScV8Int16Ty), ScV8Int32Ty);
   EXPECT_EQ(VectorType::getTruncatedElementVectorType(ScV8Int32Ty),
             ScV8Int16Ty);
 
   EXPECT_EQ(VectorType::getHalfElementsVectorType(ScV4Int64Ty), ScV2Int64Ty);
   EXPECT_EQ(VectorType::getDoubleElementsVectorType(ScV4Int64Ty), ScV8Int64Ty);
 
   EXPECT_EQ(VectorType::getInteger(ScV4Float64Ty), ScV4Int64Ty);
 
   EltCnt = ScV8Int64Ty->getElementCount();
   EXPECT_EQ(EltCnt.Min, 8U);
   ASSERT_TRUE(EltCnt.Scalable);
 }
 
+TEST(VectorTypesTest, FixedLenComparisons) {
+  LLVMContext Ctx;
+
+  Type *Int32Ty = Type::getInt32Ty(Ctx);
+  Type *Int64Ty = Type::getInt64Ty(Ctx);
+
+  VectorType *V2Int32Ty = VectorType::get(Int32Ty, 2);
+  VectorType *V4Int32Ty = VectorType::get(Int32Ty, 4);
+
+  VectorType *V2Int64Ty = VectorType::get(Int64Ty, 2);
+
+  Type::ScalableSize V2I32Len = V2Int32Ty->getScalableSizeInBits();
+  EXPECT_EQ(V2I32Len.Unscaled, 64U);
+  EXPECT_EQ(V2I32Len.Scaled, 0U);
+
+  EXPECT_LT(V2Int32Ty->getScalableSizeInBits(),
+            V4Int32Ty->getScalableSizeInBits());
+  EXPECT_GT(V2Int64Ty->getScalableSizeInBits(),
+            V2Int32Ty->getScalableSizeInBits());
+  EXPECT_EQ(V4Int32Ty->getScalableSizeInBits(),
+            V2Int64Ty->getScalableSizeInBits());
+  EXPECT_NE(V2Int32Ty->getScalableSizeInBits(),
+            V2Int64Ty->getScalableSizeInBits());
+
+  DataLayout DL("");
+  SmallVector<llvm::Type *, 2> Fields({V2Int32Ty, V4Int32Ty});
+  auto *StructTy = StructType::get(Ctx, Fields, /*isPacked=*/true);
+  auto SLen = DL.getScalableTypeSizeInBits(StructTy);
+  EXPECT_EQ(SLen.Unscaled, 192ULL);
+  EXPECT_EQ(SLen.Scaled, 0ULL);
+
+  auto *ArrayTy = ArrayType::get(V4Int32Ty, 8);
+  auto ALen = DL.getScalableTypeSizeInBits(ArrayTy);
+  EXPECT_EQ(ALen.Unscaled, 1024ULL);
+  EXPECT_EQ(ALen.Scaled, 0ULL);
+}
+
+TEST(VectorTypesTest, ScalableComparisons) {
+  LLVMContext Ctx;
+
+  Type *Int32Ty = Type::getInt32Ty(Ctx);
+  Type *Int64Ty = Type::getInt64Ty(Ctx);
+
+  VectorType *ScV2Int32Ty = VectorType::get(Int32Ty, {2, true});
+  VectorType *ScV4Int32Ty = VectorType::get(Int32Ty, {4, true});
+
+  VectorType *ScV2Int64Ty = VectorType::get(Int64Ty, {2, true});
+
+  Type::ScalableSize ScV2I32Len = ScV2Int32Ty->getScalableSizeInBits();
+  EXPECT_EQ(ScV2I32Len.Unscaled, 0U);
+  EXPECT_EQ(ScV2I32Len.Scaled, 64U);
+
+  EXPECT_LT(ScV2Int32Ty->getScalableSizeInBits(),
+            ScV4Int32Ty->getScalableSizeInBits());
+  EXPECT_GT(ScV2Int64Ty->getScalableSizeInBits(),
+            ScV2Int32Ty->getScalableSizeInBits());
+  EXPECT_EQ(ScV4Int32Ty->getScalableSizeInBits(),
+            ScV2Int64Ty->getScalableSizeInBits());
+  EXPECT_NE(ScV2Int32Ty->getScalableSizeInBits(),
+            ScV2Int64Ty->getScalableSizeInBits());
+
+  DataLayout DL("");
+  SmallVector<llvm::Type *, 2> Fields({ScV2Int32Ty, ScV4Int32Ty});
+  auto *StructTy = StructType::get(Ctx, Fields, /*isPacked=*/true);
+  auto SLen = DL.getScalableTypeSizeInBits(StructTy);
+  EXPECT_EQ(SLen.Unscaled, 0ULL);
+  EXPECT_EQ(SLen.Scaled, 192ULL);
+
+  auto *ArrayTy = ArrayType::get(ScV4Int32Ty, 8);
+  auto ALen = DL.getScalableTypeSizeInBits(ArrayTy);
+  EXPECT_EQ(ALen.Unscaled, 0ULL);
+  EXPECT_EQ(ALen.Scaled, 1024ULL);
+}
+
+TEST(VectorTypesTest, MixedComparisons) {
+  LLVMContext Ctx;
+  DataLayout DL("");
+
+  Type *Int32Ty = Type::getInt32Ty(Ctx);
+  Type *Int64Ty = Type::getInt64Ty(Ctx);
+
+  VectorType *V2I32Ty = VectorType::get(Int32Ty, {2, false});
+  VectorType *V2I64Ty = VectorType::get(Int64Ty, {2, false});
+  VectorType *ScV2I32Ty = VectorType::get(Int32Ty, {2, true});
+  VectorType *ScV2I64Ty = VectorType::get(Int64Ty, {2, true});
+
+  // Compare a fixed-size-only object vs. scalable-size-only object with
+  // same base size
+  // False: {X, 0} == {0, X}
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() ==
+                 ScV2I64Ty->getScalableSizeInBits());
+  // True: {X, 0} != {0, X}
+  EXPECT_TRUE(V2I64Ty->getScalableSizeInBits() !=
+                ScV2I64Ty->getScalableSizeInBits());
+  // False: {X, 0} < {0, X}
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() <
+                 ScV2I64Ty->getScalableSizeInBits());
+  // True: {X, 0} <= {0, X}
+  EXPECT_TRUE(V2I64Ty->getScalableSizeInBits() <=
+                ScV2I64Ty->getScalableSizeInBits());
+  // False: {X, 0} > {0, X}
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() >
+                 ScV2I64Ty->getScalableSizeInBits());
+  // False: {X, 0} >= {0, X}
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() >=
+                 ScV2I64Ty->getScalableSizeInBits());
+  // True: {0, X} >= {X, 0}
+  EXPECT_TRUE(ScV2I64Ty->getScalableSizeInBits() >=
+                V2I64Ty->getScalableSizeInBits());
+
+  // Compare a fixed-size-only object vs. scalable-size-only object with
+  // the scalable object having a larger base size
+  // False: {X, 0} == {0, Y} where X < Y
+  EXPECT_FALSE(V2I32Ty->getScalableSizeInBits() ==
+                 ScV2I64Ty->getScalableSizeInBits());
+  // True: {X, 0} != {0, Y} where X < Y
+  EXPECT_TRUE(V2I32Ty->getScalableSizeInBits() !=
+                ScV2I64Ty->getScalableSizeInBits());
+  // True: {X, 0} < {0, Y} where X < Y
+  EXPECT_TRUE(V2I32Ty->getScalableSizeInBits() <
+                ScV2I64Ty->getScalableSizeInBits());
+  // True: {X, 0} <= {0, Y} where X < Y
+  EXPECT_TRUE(V2I32Ty->getScalableSizeInBits() <=
+                ScV2I64Ty->getScalableSizeInBits());
+  // False: {0, Y} < {X, 0} where X < Y
+  EXPECT_FALSE(ScV2I64Ty->getScalableSizeInBits() <=
+                 V2I32Ty->getScalableSizeInBits());
+  // False: {X, 0} > {0, Y} where X < Y
+  EXPECT_FALSE(V2I32Ty->getScalableSizeInBits() >
+                 ScV2I64Ty->getScalableSizeInBits());
+  // True: {0, Y} > {X, 0} where X < Y
+  EXPECT_TRUE(ScV2I64Ty->getScalableSizeInBits() >
+                V2I32Ty->getScalableSizeInBits());
+  // False: {X, 0} >= {0, Y} where X < Y
+  EXPECT_FALSE(V2I32Ty->getScalableSizeInBits() >=
+                 ScV2I64Ty->getScalableSizeInBits());
+  // True: {0, Y} >= {X, 0} where X < Y
+  EXPECT_TRUE(ScV2I64Ty->getScalableSizeInBits() >=
+                V2I32Ty->getScalableSizeInBits());
+
+  // Compare a fixed-size-only object vs. a scalable-size-only object with
+  // the scalable object having a smaller base size
+  // False: {X, 0} == {0, Y} where X > Y
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() ==
+                 ScV2I32Ty->getScalableSizeInBits());
+  // True: {X, 0} != {0, Y} where X > Y
+  EXPECT_TRUE(V2I64Ty->getScalableSizeInBits() !=
+                ScV2I32Ty->getScalableSizeInBits());
+  // False: {X, 0} < {0, Y} where X > Y
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() <
+                 ScV2I32Ty->getScalableSizeInBits());
+  // False: {0, Y} < {X, 0} where X > Y
+  EXPECT_FALSE(ScV2I32Ty->getScalableSizeInBits() <
+                 V2I64Ty->getScalableSizeInBits());
+  // False: {X, 0} <= {0, Y} where X > Y
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() <=
+                 ScV2I32Ty->getScalableSizeInBits());
+  // False: {0, Y} <= {X, 0} where X > Y
+  EXPECT_FALSE(ScV2I32Ty->getScalableSizeInBits() <=
+                 V2I64Ty->getScalableSizeInBits());
+  // False: {X, 0} > {0, Y} where X > Y
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() >
+                 ScV2I32Ty->getScalableSizeInBits());
+  // False: {0, Y} > {X, 0} where X > Y
+  EXPECT_FALSE(ScV2I32Ty->getScalableSizeInBits() >
+                 V2I64Ty->getScalableSizeInBits());
+  // False: {X, 0} >= {0, Y} where X > Y
+  EXPECT_FALSE(V2I64Ty->getScalableSizeInBits() >=
+                 ScV2I32Ty->getScalableSizeInBits());
+  // False: {0, Y} >= {X, 0} where X > Y
+  EXPECT_FALSE(ScV2I32Ty->getScalableSizeInBits() >=
+                 V2I64Ty->getScalableSizeInBits());
+
+  SmallVector<llvm::Type *, 2> Fields({V2I32Ty, ScV2I64Ty});
+  auto *U64S128Ty = StructType::get(Ctx, Fields, /*isPacked=*/true);
+  auto SLen = DL.getScalableTypeSizeInBits(U64S128Ty);
+  EXPECT_EQ(SLen.Unscaled, 64ULL);
+  EXPECT_EQ(SLen.Scaled, 128ULL);
+
+  Fields = {V2I64Ty, ScV2I64Ty};
+  auto U128S128Ty = StructType::get(Ctx, Fields, /*isPacked=*/true);
+
+  // False: {X, Y} == {Z, Y} where X < Z
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U64S128Ty) ==
+               DL.getScalableTypeSizeInBits(U128S128Ty));
+  // True: {X, Y} != {Z, Y} where X < Z
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U64S128Ty) !=
+              DL.getScalableTypeSizeInBits(U128S128Ty));
+  // True: {X, Y} < {Z, Y} where X < Z
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U64S128Ty) <
+              DL.getScalableTypeSizeInBits(U128S128Ty));
+  // True: {X, Y} <= {Z, Y} where X < Z
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U64S128Ty) <=
+              DL.getScalableTypeSizeInBits(U128S128Ty));
+  // False: {Z, Y} <= {X, Y} where X < Z
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U128S128Ty) <=
+               DL.getScalableTypeSizeInBits(U64S128Ty));
+  // False: {X, Y} > {Z, Y} where X < Z
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U64S128Ty) >
+               DL.getScalableTypeSizeInBits(U128S128Ty));
+  // False: {X, Y} >= {Z, Y} where X < Z
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U64S128Ty) >=
+               DL.getScalableTypeSizeInBits(U128S128Ty));
+  // True: {Z, Y} >= {X, Y} where X < Z
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U128S128Ty) >=
+              DL.getScalableTypeSizeInBits(U64S128Ty));
+
+  Fields = {V2I64Ty, ScV2I32Ty};
+  auto U128S64Ty = StructType::get(Ctx, Fields, /*isPacked=*/true);
+
+  // False: {X, Y} == {X, Z} where Y > Z
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U128S128Ty) ==
+               DL.getScalableTypeSizeInBits(U128S64Ty));
+  // True: {X, Y} != {X, Z} where Y > Z
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U128S128Ty) !=
+              DL.getScalableTypeSizeInBits(U128S64Ty));
+  // False: {X, Y} < {X, Z} where Y > Z
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U128S128Ty) <
+               DL.getScalableTypeSizeInBits(U128S64Ty));
+  // False {X, Y} <= {X, Z} where Y > Z
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U128S128Ty) <=
+               DL.getScalableTypeSizeInBits(U128S64Ty));
+  // True {X, Z} <= {X, Y} where Y > Z
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U128S64Ty) <=
+              DL.getScalableTypeSizeInBits(U128S128Ty));
+  // True: {X, Y} > {X, Z} where Y > Z
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U128S128Ty) >
+              DL.getScalableTypeSizeInBits(U128S64Ty));
+  // False: {X, Z} > {X, Y} where Y > Z
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U128S64Ty) >
+               DL.getScalableTypeSizeInBits(U128S128Ty));
+  // True: {X, Y} >= {X, Z} where Y > Z
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U128S128Ty) >=
+              DL.getScalableTypeSizeInBits(U128S64Ty));
+
+
+
+
+  // False: {W, X} == {Y, Z} where W == Z && X == Y
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U64S128Ty) ==
+               DL.getScalableTypeSizeInBits(U128S64Ty));
+  // True: {W, X} != {Y, Z} where W == Z && X == Y
+  EXPECT_TRUE(DL.getScalableTypeSizeInBits(U64S128Ty) !=
+              DL.getScalableTypeSizeInBits(U128S64Ty));
+  // False: {W, X} < {Y, Z} where W == Z && X == Y
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U64S128Ty) <
+               DL.getScalableTypeSizeInBits(U128S64Ty));
+  // False: {W, X} <= {Y, Z} where W == Z && X == Y
+  EXPECT_FALSE(DL.getScalableTypeSizeInBits(U64S128Ty) <=
+               DL.getScalableTypeSizeInBits(U128S64Ty));
+
+
+
+  auto *ArrayTy = ArrayType::get(U128S64Ty, 3);
+  auto ALen = DL.getScalableTypeSizeInBits(ArrayTy);
+  EXPECT_EQ(ALen.Unscaled, 384ULL);
+  EXPECT_EQ(ALen.Scaled, 192ULL);
+}
+
 }