[llvm] Use pointer index type for more GEP offsets (pre-codegen)

[krzysz00]

Summary

Many uses of getIntPtrType() were using that type to calculate the
neened type for GEP offset arguments. However, some time ago,
DataLayout was extended to support pointers where the size of the
pointer is not equal to the size of the values used to index it.

Much code was already migrated to, for example, use getIndexSizeInBits
instead of getPtrSizeInBits, but some rewrites still used
getIntPtrType() to get the type for GEP offsets.

This commit changes uses of getIntPtrType() to getIndexType() where
they are involved in a GEP-related calculation.

In at least one case (bounds check insertion) this resolves a compiler
crash that the new test added here would previously trigger.

This commit does not impact

• C library-related rewriting (memcpy()), which are operating under

the assumption that intptr_t == size_t. While all the mechanisms for
breaking this assumption now exist, doing so is outside the scope of
this commit.

• Code generation and below. Note that the use of getIntPtrType() in

CodeGenPrepare will be changed in a future commit.

• Usage of getIntPtrType() in any backend

Depends on D143435

[arichardson]

Thanks for this patch! In the CHERI fork I added a rather ugly workaround in getIngPtrType() for this instead of updating all callers (https://github.com/CTSRD-CHERI/llvm-project/blob/19d402e23fcaa197e1d40547da403dc17e13c7ae/llvm/lib/IR/DataLayout.cpp#L862). I will try applying the patch to CHERI LLVM to see if there are any further places that need updating.

[arichardson]

llvm/include/llvm/IR/IRBuilder.h

572

I'd suggest removing the default zero here to ensure that callers always pass the address space explicitly.

[foad]

llvm/lib/Transforms/Scalar/NaryReassociate.cpp

454

Not necessarily related to your patch, but the "OrTrunc" makes me nervous (here and in a couple of other places in this patch). My understanding of the index type in datalayout is that it is a preferred type to use GEP indexes, but shouldn't actually have any semantic effect, so it would be wrong to truncate an index to the index type unless you can prove that it won't change the value.

[krzysz00]

@arichardson I'd like to also draw your attention to https://reviews.llvm.org/D143526, its global isel equivalent.

It'd be nice to fix up SelectionDAG to also generate ADD nodes using the index width and not the data width, but that's not what I'm trying first.

llvm/lib/Transforms/Scalar/NaryReassociate.cpp

454

That's not my understanding of that field. The index field is maximal size of an offset into a pointer in address space N. For example, CHERI's capability pointers are (to my understanding) 128:128:128:64 meaning that the pointer value is 128 bits long, but only 64 bits of offset can be applied to it.

That is, in C terms, the pointer type is intptr_t and the index field is size_t. This patch fixes up a bunch of code that normalized pointer offsets to intptr_t instead of size_t, since LLVM historically assumed they were the same.

[krzysz00]

... turns out I just wanted getIndexType() this whole time

[arsenm]

llvm/include/llvm-c/Target.h

249 ↗

(On Diff #495283)

Adding new stuff to the C API should be done separately and should get unit tests

llvm/lib/IR/DataLayout.cpp

878

Can this just be inline in the header?

llvm/lib/Transforms/Scalar/NaryReassociate.cpp

355

nary reassociate not tested

llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp

694

slsr not tested

llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp

327

Vectorizer not tested

[krzysz00]

Add tests

[krzysz00]

While I was there, found another case where the constant offset extractor was called with pointer bitwidth APInt and not an index bitwidth APInt.

[arsenm]

llvm/lib/Transforms/InstCombine/InstCombineCompares.cpp

283–287

Don't think this one is tested

llvm/lib/Transforms/Scalar/ConstantHoisting.cpp

414

Don't think this one is tested

llvm/lib/Transforms/Scalar/MergeICmps.cpp

160

Don't think this one is tested

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

930

Doesn't creating a ptrtoint from a non-integral pointer break horribly?

llvm/lib/Transforms/Utils/FunctionComparator.cpp

752

Don't think this one is tested

[krzysz00]

Add test for icmp change, guard the inttoptr based GEP lowering so it doesn't reach fat pointers or non-integeral ones.

[krzysz00]

llvm/lib/Transforms/InstCombine/InstCombineCompares.cpp

283–287

You're right, added a test.

llvm/lib/Transforms/Scalar/ConstantHoisting.cpp

414

GEPOperator::accumulateConstantOffset asserts that its input bitwidth has bitwidth equal to the index size of the GEP base, so this was a straight-up bug.

llvm/lib/Transforms/Scalar/MergeICmps.cpp

160

Same thing with accumulateConstantOffset

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

930

Good catch, I've added guards to make sure non-integral address spaces don't get sent this direction. And that fat pointers don't enter this code path either.

llvm/lib/Transforms/Utils/FunctionComparator.cpp

752

Same note re accumulateConstantOffset()

[jrtc27]

llvm/lib/Transforms/Scalar/NaryReassociate.cpp

454

This is correct; see SelectionDAGBuilder::visitGetElementPtr for example where the use of the index type is explicit (and there's some commentary, too). It is legal to use an oversized type in the IR, but it will eventually be legalised. Undersized types are implicitly sign-extended.

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

794

Arguably this is now a slight misnomer

921

IR is not C, plus the assert documents that anyway?

926

Why not assert they're the same type?

930

Hm, weirdly I don't see any changes to this function in CHERI LLVM, which makes me wonder how we don't hit issues here on CHERI-RISC-V given the RISC-V backend doesn't return true for STI->useAA()...

1063

Comment probably needs updating to match

1064–1072

Why is AddrSpace before the comment but IsFatPointer after?

llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp

677–678

If you're changing the indentation below please also fix the curlies (and break) to match

[arsenm]

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

1064–1072

Don't know what FatPointer is meaning here, but isNonIntegralAddressSpace should cover it?

[jrtc27]

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

1064–1072

Non-integral is overly strong, it also implies instability of the value if converted to an integer multiple times, which is not true of CHERI, nor AMDGPU AFAIK

[krzysz00]

Address review comments

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

794

... yeah, I should fix that

1064–1072

Yeah, this is safeguarding against the possibility of having the buffer descriptor be an integral value, where you need to be very careful when adding things (to avoid carries into the metadata).

llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp

677–678

I don't think this is changing the indentation?

[jrtc27]

llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp

677–678

Phabricator seems to think you've de-indented by 2 spaces

[jrtc27]

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

1066

As discussed on discourse, this isn't really what that means, at least not for CHERI, which was one of the influences for how the current pointer index type came to be (D42123)

[arichardson]

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

930

Probably due to the DL->getIntPtrType hack that we have (which been hopefully be deleted soon after merging up to this change).

[jrtc27]

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

930

It would still generate a metadata-stripping ptrtoint+...+inttoptr sequence though, that would only stop it asserting on using the wrong type for ptrtoint

[foad]

llvm/lib/Transforms/Scalar/NaryReassociate.cpp

454

If it has a semantic effect on IR then that needs to be clearly documented in the LangRef. Literally the only mention of it in the document is: "The fourth parameter <idx> is a size of index that used for address calculation."

[krzysz00]

Update documentation on GEP since it didn't get fully updated

[krzysz00]

What's missing here that prevents a landing? I feel like I missed something in the sea of comments

[arsenm]

In D143437#4142104, @krzysz00 wrote:

What's missing here that prevents a landing? I feel like I missed something in the sea of comments

I'm being a stickler for testing every single piece here and there are a few missing (e.g. MergeICmps.cpp)

[krzysz00]

@arsenm, I can try to put together tests for some of the remaining stuff (like the ICmp merge). However, unless I'm forgetting something, the remaaining untested bits are the ones that call accumulateConstantOffset(), which, per https://github.com/llvm/llvm-project/blob/main/llvm/lib/IR/Operator.cpp#L94-L96 , will crash if you mix up pointer bitwidth and index bitwidth. I'm not sure that fixing up those crashes (when most of the other code in LLVM has been fixed for this) justifies a test?

[arichardson]

In D143437#4145600, @krzysz00 wrote:

@arsenm, I can try to put together tests for some of the remaining stuff (like the ICmp merge). However, unless I'm forgetting something, the remaaining untested bits are the ones that call accumulateConstantOffset(), which, per https://github.com/llvm/llvm-project/blob/main/llvm/lib/IR/Operator.cpp#L94-L96 , will crash if you mix up pointer bitwidth and index bitwidth. I'm not sure that fixing up those crashes (when most of the other code in LLVM has been fixed for this) justifies a test?

You could always pre-commit a not --crash test that is fixed by this commit?

[krzysz00]

I didn't know not --crash was a Thing, thanks!

I'll try and put those together tomorrow

[krzysz00]

Use precommited tests for accumulateConstantOffset() calls

[arichardson]

I would check the output of the no longer crashing tests but otherwise LGTM. Please wait for @arsenm to confirm as well before committing though.

llvm/test/Transforms/MergeICmps/X86/distinct-index-width-crash.ll

1

Maybe also check the output here to make sure it's somewhat sensible?

[krzysz00]

Update tests to include non-trivial CHECK lines

[krzysz00]

Missed a bad test, whoops

[krzysz00]

Update for planned new address spaces (p7 = descriptor + offset, p8 = descriptor)

[krzysz00]

@arsenm Any objections to landing this? I think I've added all the needed "don't crash" tests

[krzysz00]

Rebase updates

[krzysz00]

Fix buffer tests

llvm/docs/LangRef.rst

 ``addrspacecast (CST to TYPE)``
     Convert a constant pointer or constant vector of pointer, CST, to another
     TYPE in a different address space. The constraints of the operands are the
     same as those for the :ref:`addrspacecast instruction <i_addrspacecast>`.
 ``getelementptr (TY, CSTPTR, IDX0, IDX1, ...)``, ``getelementptr inbounds (TY, CSTPTR, IDX0, IDX1, ...)``
     Perform the :ref:`getelementptr operation <i_getelementptr>` on
     constants. As with the :ref:`getelementptr <i_getelementptr>`
     instruction, the index list may have one or more indexes, which are
-    required to make sense for the type of "pointer to TY".
+    required to make sense for the type of "pointer to TY". These indexes
+    may be implicitly sign-extended or truncated to match the index size
+    of CSTPTR's address space.
 ``select (COND, VAL1, VAL2)``
     Perform the :ref:`select operation <i_select>` on constants.
 ``icmp COND (VAL1, VAL2)``
     Perform the :ref:`icmp operation <i_icmp>` on constants.
 ``fcmp COND (VAL1, VAL2)``
     Perform the :ref:`fcmp operation <i_fcmp>` on constants.
 ``extractelement (VAL, IDX)``
     Perform the :ref:`extractelement operation <i_extractelement>` on

 multiple atomic stores. The type of the pointee must be an integer, pointer, or
 floating-point type whose bit width is a power of two greater than or equal to
 eight and less than or equal to a target-specific size limit.  ``align`` must be
 explicitly specified on atomic loads, and the load has undefined behavior if the
 alignment is not set to a value which is at least the size in bytes of the
 pointee. ``!nontemporal`` does not have any defined semantics for atomic loads.
 
 The optional constant ``align`` argument specifies the alignment of the
-operation (that is, the alignment of the memory address). It is the 
+operation (that is, the alignment of the memory address). It is the
 responsibility of the code emitter to ensure that the alignment information is
-correct. Overestimating the alignment results in undefined behavior. 
+correct. Overestimating the alignment results in undefined behavior.
 Underestimating the alignment may produce less efficient code. An alignment of
-1 is always safe. The maximum possible alignment is ``1 << 32``. An alignment 
+1 is always safe. The maximum possible alignment is ``1 << 32``. An alignment
 value higher than the size of the loaded type implies memory up to the
 alignment value bytes can be safely loaded without trapping in the default
 address space. Access of the high bytes can interfere with debugging tools, so
 should not be accessed if the function has the ``sanitize_thread`` or
 ``sanitize_address`` attributes.
 
 The alignment is only optional when parsing textual IR; for in-memory IR, it is
 always present. An omitted ``align`` argument means that the operation has the

 multiple atomic stores. The type of the pointee must be an integer, pointer, or
 floating-point type whose bit width is a power of two greater than or equal to
 eight and less than or equal to a target-specific size limit.  ``align`` must be
 explicitly specified on atomic stores, and the store has undefined behavior if
 the alignment is not set to a value which is at least the size in bytes of the
 pointee. ``!nontemporal`` does not have any defined semantics for atomic stores.
 
 The optional constant ``align`` argument specifies the alignment of the
-operation (that is, the alignment of the memory address). It is the 
+operation (that is, the alignment of the memory address). It is the
 responsibility of the code emitter to ensure that the alignment information is
-correct. Overestimating the alignment results in undefined behavior. 
+correct. Overestimating the alignment results in undefined behavior.
 Underestimating the alignment may produce less efficient code. An alignment of
-1 is always safe. The maximum possible alignment is ``1 << 32``. An alignment 
+1 is always safe. The maximum possible alignment is ``1 << 32``. An alignment
 value higher than the size of the loaded type implies memory up to the
 alignment value bytes can be safely loaded without trapping in the default
 address space. Access of the high bytes can interfere with debugging tools, so
 should not be accessed if the function has the ``sanitize_thread`` or
 ``sanitize_address`` attributes.
 
 The alignment is only optional when parsing textual IR; for in-memory IR, it is
 always present. An omitted ``align`` argument means that the operation has the
*  In cases where the base is a vector of pointers, the ``inbounds`` keyword
    applies to each of the computations element-wise.
 
 These rules are based on the assumption that no allocated object may cross
 the unsigned address space boundary, and no allocated object may be larger
 than half the pointer index type space.
 
 If the ``inbounds`` keyword is not present, the offsets are added to the
 base address with silently-wrapping two's complement arithmetic. If the
-offsets have a different width from the pointer, they are sign-extended
-or truncated to the width of the pointer. The result value of the
-``getelementptr`` may be outside the object pointed to by the base
+offsets have a different width from the pointer's index type, they are
+sign-extended or truncated to the width of the pointer's index type. The result
+value of the ``getelementptr`` may be outside the object pointed to by the base
 pointer. The result value may not necessarily be used to access memory
 though, even if it happens to point into allocated storage. See the
 :ref:`Pointer Aliasing Rules <pointeraliasing>` section for more
 information.
 
 If the ``inrange`` keyword is present before any index, loading from or
 storing to any pointer derived from the ``getelementptr`` has undefined
 behavior if the load or store would access memory outside of the bounds of

llvm/include/llvm/IR/DataLayout.h

Type *getLargestLegalIntType(LLVMContext &C) const {
     unsigned LargestSize = getLargestLegalIntTypeSizeInBits();
     return (LargestSize == 0) ? nullptr : Type::getIntNTy(C, LargestSize);
   }
 
   /// Returns the size of largest legal integer type size, or 0 if none
   /// are set.
   unsigned getLargestLegalIntTypeSizeInBits() const;
 
+  /// Returns the type of a GEP index in AddressSpace.
+  /// If it was not specified explicitly, it will be the integer type of the
+  /// pointer width - IntPtrType.
+  IntegerType *getIndexType(LLVMContext &C, unsigned AddressSpace) const;
+
   /// Returns the type of a GEP index.
   /// If it was not specified explicitly, it will be the integer type of the
   /// pointer width - IntPtrType.
   Type *getIndexType(Type *PtrTy) const;
 
   /// Returns the offset from the beginning of the type for the specified
   /// indices.
   ///

llvm/include/llvm/IR/IRBuilder.h

#endif
   }
 
   /// Fetch the type of an integer with size at least as big as that of a
   /// pointer in the given address space.
   IntegerType *getIntPtrTy(const DataLayout &DL, unsigned AddrSpace = 0) {
     return DL.getIntPtrType(Context, AddrSpace);
   }
 
+  /// Fetch the type of an integer that should be used to index GEP operations
+  /// within AddressSpace.
+  IntegerType *getIndexTy(const DataLayout &DL, unsigned AddrSpace) {
+    return DL.getIndexType(Context, AddrSpace);
+  }
+
   //===--------------------------------------------------------------------===//
   // Intrinsic creation methods
   //===--------------------------------------------------------------------===//
 
   /// Create and insert a memset to the specified pointer and the
   /// specified value.
   ///
   /// If the pointer isn't an i8*, it will be converted. If a TBAA tag is

llvm/lib/Analysis/MemoryBuiltins.cpp

 
 SizeOffsetEvalType ObjectSizeOffsetEvaluator::visitAllocaInst(AllocaInst &I) {
   if (!I.getAllocatedType()->isSized())
     return unknown();
 
   // must be a VLA
   assert(I.isArrayAllocation());
 
-  // If needed, adjust the alloca's operand size to match the pointer size.
-  // Subsequent math operations expect the types to match.
+  // If needed, adjust the alloca's operand size to match the pointer indexing
+  // size. Subsequent math operations expect the types to match.
   Value *ArraySize = Builder.CreateZExtOrTrunc(
-      I.getArraySize(), DL.getIntPtrType(I.getContext()));
+      I.getArraySize(),
+      DL.getIndexType(I.getContext(), DL.getAllocaAddrSpace()));
   assert(ArraySize->getType() == Zero->getType() &&
-         "Expected zero constant to have pointer type");
+         "Expected zero constant to have pointer index type");
 
   Value *Size = ConstantInt::get(ArraySize->getType(),
                                  DL.getTypeAllocSize(I.getAllocatedType()));
   Size = Builder.CreateMul(Size, ArraySize);
   return std::make_pair(Size, Zero);
 }
 
 SizeOffsetEvalType ObjectSizeOffsetEvaluator::visitCallBase(CallBase &CB) {

llvm/lib/IR/DataLayout.cpp

Type *DataLayout::getIntPtrType(Type *Ty) const {
   IntegerType *IntTy = IntegerType::get(Ty->getContext(), NumBits);
   if (VectorType *VecTy = dyn_cast<VectorType>(Ty))
     return VectorType::get(IntTy, VecTy);
   return IntTy;
 }
 
 Type *DataLayout::getSmallestLegalIntType(LLVMContext &C, unsigned Width) const {
   for (unsigned LegalIntWidth : LegalIntWidths)
     if (Width <= LegalIntWidth)
       return Type::getIntNTy(C, LegalIntWidth);
   return nullptr;
 }
 
 unsigned DataLayout::getLargestLegalIntTypeSizeInBits() const {
   auto Max = std::max_element(LegalIntWidths.begin(), LegalIntWidths.end());
   return Max != LegalIntWidths.end() ? *Max : 0;
 }
 
+IntegerType *DataLayout::getIndexType(LLVMContext &C,
+                                      unsigned AddressSpace) const {
+  return IntegerType::get(C, getIndexSizeInBits(AddressSpace));
+}
+
 Type *DataLayout::getIndexType(Type *Ty) const {
   assert(Ty->isPtrOrPtrVectorTy() &&
          "Expected a pointer or pointer vector type.");
   unsigned NumBits = getIndexTypeSizeInBits(Ty);
   IntegerType *IntTy = IntegerType::get(Ty->getContext(), NumBits);
   if (VectorType *VecTy = dyn_cast<VectorType>(Ty))
     return VectorType::get(IntTy, VecTy);
   return IntTy;

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

for (MDNode *Type : Types) {
               ->getZExtValue();
       if (COffset == Offset)
         return true;
     }
     return false;
   }
 
   if (auto GEP = dyn_cast<GEPOperator>(V)) {
-    APInt APOffset(DL.getPointerSizeInBits(0), 0);
+    APInt APOffset(DL.getIndexSizeInBits(0), 0);
     bool Result = GEP->accumulateConstantOffset(DL, APOffset);
     if (!Result)
       return false;
     COffset += APOffset.getZExtValue();
     return isKnownTypeIdMember(TypeId, DL, GEP->getPointerOperand(), COffset);
   }
 
   if (auto Op = dyn_cast<Operator>(V)) {

llvm/lib/Transforms/InstCombine/InstCombineCompares.cpp

if ((i & 8) == 0 && i >= 64 && SecondTrueElement == Overdefined &&
         FalseRangeEnd == Overdefined)
       return nullptr;
   }
 
   // Now that we've scanned the entire array, emit our new comparison(s).  We
   // order the state machines in complexity of the generated code.
   Value *Idx = GEP->getOperand(2);
 
-  // If the index is larger than the pointer size of the target, truncate the
-  // index down like the GEP would do implicitly.  We don't have to do this for
-  // an inbounds GEP because the index can't be out of range.
+  // If the index is larger than the pointer offset size of the target, truncate
+  // the index down like the GEP would do implicitly.  We don't have to do this
+  // for an inbounds GEP because the index can't be out of range.
   if (!GEP->isInBounds()) {
-    Type *IntPtrTy = DL.getIntPtrType(GEP->getType());
-    unsigned PtrSize = IntPtrTy->getIntegerBitWidth();
-    if (Idx->getType()->getPrimitiveSizeInBits().getFixedValue() > PtrSize)
-      Idx = Builder.CreateTrunc(Idx, IntPtrTy);
+    Type *PtrIdxTy = DL.getIndexType(GEP->getType());
+    unsigned OffsetSize = PtrIdxTy->getIntegerBitWidth();
+    if (Idx->getType()->getPrimitiveSizeInBits().getFixedValue() > OffsetSize)
+      Idx = Builder.CreateTrunc(Idx, PtrIdxTy);
   }
 
   // If inbounds keyword is not present, Idx * ElementSize can overflow.
   // Let's assume that ElementSize is 2 and the wanted value is at offset 0.
   // Then, there are two possible values for Idx to match offset 0:
   // 0x00..00, 0x80..00.
   // Emitting 'icmp eq Idx, 0' isn't correct in this case because the
   // comparison is false if Idx was 0x80..00.

llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp

if (C->getValue().getActiveBits() <= 64) {
       //
       BasicBlock::iterator It(New);
       while (isa<AllocaInst>(*It) || isa<DbgInfoIntrinsic>(*It))
         ++It;
 
       // Now that I is pointing to the first non-allocation-inst in the block,
       // insert our getelementptr instruction...
       //
-      Type *IdxTy = IC.getDataLayout().getIntPtrType(AI.getType());
+      Type *IdxTy = IC.getDataLayout().getIndexType(AI.getType());
       Value *NullIdx = Constant::getNullValue(IdxTy);
       Value *Idx[2] = {NullIdx, NullIdx};
       Instruction *GEP = GetElementPtrInst::CreateInBounds(
           NewTy, New, Idx, New->getName() + ".sub");
       IC.InsertNewInstBefore(GEP, *It);
 
       // Now make everything use the getelementptr instead of the original
       // allocation.
       return IC.replaceInstUsesWith(AI, GEP);
     }
   }
 
   if (isa<UndefValue>(AI.getArraySize()))
     return IC.replaceInstUsesWith(AI, Constant::getNullValue(AI.getType()));
 
-  // Ensure that the alloca array size argument has type intptr_t, so that
-  // any casting is exposed early.
-  Type *IntPtrTy = IC.getDataLayout().getIntPtrType(AI.getType());
-  if (AI.getArraySize()->getType() != IntPtrTy) {
-    Value *V = IC.Builder.CreateIntCast(AI.getArraySize(), IntPtrTy, false);
+  // Ensure that the alloca array size argument has type equal to the offset
+  // size of the alloca() pointer, which, in the tyical case, is intptr_t,
+  // so that any casting is exposed early.
+  Type *PtrIdxTy = IC.getDataLayout().getIndexType(AI.getType());
+  if (AI.getArraySize()->getType() != PtrIdxTy) {
+    Value *V = IC.Builder.CreateIntCast(AI.getArraySize(), PtrIdxTy, false);
     return IC.replaceOperand(AI, 0, V);
   }
 
   return nullptr;
 }
 
 namespace {
 // If I and V are pointers in different address space, it is not allowed to

llvm/lib/Transforms/Instrumentation/BoundsChecking.cpp

if (!ObjSizeEval.bothKnown(SizeOffset)) {
     ++ChecksUnable;
     return nullptr;
   }
 
   Value *Size   = SizeOffset.first;
   Value *Offset = SizeOffset.second;
   ConstantInt *SizeCI = dyn_cast<ConstantInt>(Size);
 
-  Type *IntTy = DL.getIntPtrType(Ptr->getType());
-  Value *NeededSizeVal = IRB.CreateTypeSize(IntTy, NeededSize);
+  Type *IndexTy = DL.getIndexType(Ptr->getType());
+  Value *NeededSizeVal = IRB.CreateTypeSize(IndexTy, NeededSize);
 
   auto SizeRange = SE.getUnsignedRange(SE.getSCEV(Size));
   auto OffsetRange = SE.getUnsignedRange(SE.getSCEV(Offset));
   auto NeededSizeRange = SE.getUnsignedRange(SE.getSCEV(NeededSizeVal));
 
   // three checks are required to ensure safety:
   // . Offset >= 0  (since the offset is given from the base ptr)
   // . Size >= Offset  (unsigned)
   // . Size - Offset >= NeededSize  (unsigned)
   //
   // optimization: if Size >= 0 (signed), skip 1st check
   // FIXME: add NSW/NUW here?  -- we dont care if the subtraction overflows
   Value *ObjSize = IRB.CreateSub(Size, Offset);
   Value *Cmp2 = SizeRange.getUnsignedMin().uge(OffsetRange.getUnsignedMax())
                     ? ConstantInt::getFalse(Ptr->getContext())
                     : IRB.CreateICmpULT(Size, Offset);
   Value *Cmp3 = SizeRange.sub(OffsetRange)
                         .getUnsignedMin()
                         .uge(NeededSizeRange.getUnsignedMax())
                     ? ConstantInt::getFalse(Ptr->getContext())
                     : IRB.CreateICmpULT(ObjSize, NeededSizeVal);
   Value *Or = IRB.CreateOr(Cmp2, Cmp3);
   if ((!SizeCI || SizeCI->getValue().slt(0)) &&
       !SizeRange.getSignedMin().isNonNegative()) {
-    Value *Cmp1 = IRB.CreateICmpSLT(Offset, ConstantInt::get(IntTy, 0));
+    Value *Cmp1 = IRB.CreateICmpSLT(Offset, ConstantInt::get(IndexTy, 0));
     Or = IRB.CreateOr(Cmp1, Or);
   }
 
   return Or;
 }
 
 /// Adds run-time bounds checks to memory accessing instructions.
 ///

llvm/lib/Transforms/Scalar/ConstantHoisting.cpp

if (ConstExpr->getType()->isVectorTy())
     return;
 
   GlobalVariable *BaseGV = dyn_cast<GlobalVariable>(ConstExpr->getOperand(0));
   if (!BaseGV)
     return;
 
   // Get offset from the base GV.
   PointerType *GVPtrTy = cast<PointerType>(BaseGV->getType());
-  IntegerType *PtrIntTy = DL->getIntPtrType(*Ctx, GVPtrTy->getAddressSpace());
-  APInt Offset(DL->getTypeSizeInBits(PtrIntTy), /*val*/0, /*isSigned*/true);
+  IntegerType *OffsetTy = DL->getIndexType(*Ctx, GVPtrTy->getAddressSpace());
+  APInt Offset(DL->getTypeSizeInBits(OffsetTy), /*val*/ 0, /*isSigned*/ true);
   auto *GEPO = cast<GEPOperator>(ConstExpr);
 
   // TODO: If we have a mix of inbounds and non-inbounds GEPs, then basing a
   // non-inbounds GEP on an inbounds GEP is potentially incorrect. Restrict to
   // inbounds GEP for now -- alternatively, we could drop inbounds from the
   // constant expression,
   if (!GEPO->isInBounds())
     return;
 
   if (!GEPO->accumulateConstantOffset(*DL, Offset))
     return;
 
   if (!Offset.isIntN(32))
     return;
 
   // A constant GEP expression that has a GlobalVariable as base pointer is
   // usually lowered to a load from constant pool. Such operation is unlikely
   // to be cheaper than compute it by <Base + Offset>, which can be lowered to
   // an ADD instruction or folded into Load/Store instruction.
   InstructionCost Cost =
-      TTI->getIntImmCostInst(Instruction::Add, 1, Offset, PtrIntTy,
+      TTI->getIntImmCostInst(Instruction::Add, 1, Offset, OffsetTy,
                              TargetTransformInfo::TCK_SizeAndLatency, Inst);
   ConstCandVecType &ExprCandVec = ConstGEPCandMap[BaseGV];
   ConstCandMapType::iterator Itr;
   bool Inserted;
   ConstPtrUnionType Cand = ConstExpr;
   std::tie(Itr, Inserted) = ConstCandMap.insert(std::make_pair(Cand, 0));
   if (Inserted) {
     ExprCandVec.push_back(ConstantCandidate(

llvm/lib/Transforms/Scalar/MergeICmps.cpp

BCEAtom visitICmpLoadOperand(Value *const Val, BaseIdentifier &BaseId) {
   const auto &DL = LoadI->getModule()->getDataLayout();
   if (!isDereferenceablePointer(Addr, LoadI->getType(), DL)) {
     LLVM_DEBUG(dbgs() << "not dereferenceable\n");
     // We need to make sure that we can do comparison in any order, so we
     // require memory to be unconditionally dereferenceable.
     return {};
   }
 
-  APInt Offset = APInt(DL.getPointerTypeSizeInBits(Addr->getType()), 0);
+  APInt Offset = APInt(DL.getIndexTypeSizeInBits(Addr->getType()), 0);
   Value *Base = Addr;
   auto *GEP = dyn_cast<GetElementPtrInst>(Addr);
   if (GEP) {
     LLVM_DEBUG(dbgs() << "GEP\n");
     if (GEP->isUsedOutsideOfBlock(LoadI->getParent())) {
       LLVM_DEBUG(dbgs() << "used outside of block\n");
       return {};
     }

llvm/lib/Transforms/Scalar/NaryReassociate.cpp

if (GTI.isSequential()) {
       }
     }
   }
   return nullptr;
 }
 
 bool NaryReassociatePass::requiresSignExtension(Value *Index,
                                                 GetElementPtrInst *GEP) {
-  unsigned PointerSizeInBits =
-      DL->getPointerSizeInBits(GEP->getType()->getPointerAddressSpace());
-  return cast<IntegerType>(Index->getType())->getBitWidth() < PointerSizeInBits;
+  unsigned IndexSizeInBits =
+      DL->getIndexSizeInBits(GEP->getType()->getPointerAddressSpace());
+  return cast<IntegerType>(Index->getType())->getBitWidth() < IndexSizeInBits;
 }
 
 GetElementPtrInst *
 NaryReassociatePass::tryReassociateGEPAtIndex(GetElementPtrInst *GEP,
                                               unsigned I, Type *IndexedType) {
   Value *IndexToSplit = GEP->getOperand(I + 1);
   if (SExtInst *SExt = dyn_cast<SExtInst>(IndexToSplit)) {
     IndexToSplit = SExt->getOperand(0);
NaryReassociatePass::tryReassociateGEPAtIndex(GetElementPtrInst *GEP,
   //
   // sizeof(S) = 100 is indivisible by sizeof(int64) = 8.
   //
   // TODO: bail out on this case for now. We could emit uglygep.
   if (IndexedSize % ElementSize != 0)
     return nullptr;
 
   // NewGEP = &Candidate[RHS * (sizeof(IndexedType) / sizeof(Candidate[0])));
-  Type *IntPtrTy = DL->getIntPtrType(GEP->getType());
-  if (RHS->getType() != IntPtrTy)
-    RHS = Builder.CreateSExtOrTrunc(RHS, IntPtrTy);
+  Type *PtrIdxTy = DL->getIndexType(GEP->getType());
+  if (RHS->getType() != PtrIdxTy)
+    RHS = Builder.CreateSExtOrTrunc(RHS, PtrIdxTy);
   if (IndexedSize != ElementSize) {
     RHS = Builder.CreateMul(
-        RHS, ConstantInt::get(IntPtrTy, IndexedSize / ElementSize));
+        RHS, ConstantInt::get(PtrIdxTy, IndexedSize / ElementSize));
   }
   GetElementPtrInst *NewGEP = cast<GetElementPtrInst>(
       Builder.CreateGEP(GEP->getResultElementType(), Candidate, RHS));
   NewGEP->setIsInBounds(GEP->isInBounds());
   NewGEP->takeName(GEP);
   return NewGEP;
 }
 

llvm/lib/Transforms/Scalar/SeparateConstOffsetFromGEP.cpp

private:
   /// (http://llvm.org/docs/LangRef.html#id181).
   ///
   /// This canonicalization is very likely already done in clang and
   /// instcombine. Therefore, the program will probably remain the same.
   ///
   /// Returns true if the module changes.
   ///
   /// Verified in @i32_add in split-gep.ll
-  bool canonicalizeArrayIndicesToPointerSize(GetElementPtrInst *GEP);
+  bool canonicalizeArrayIndicesToIndexSize(GetElementPtrInst *GEP);
 
   /// Optimize sext(a)+sext(b) to sext(a+b) when a+b can't sign overflow.
   /// SeparateConstOffsetFromGEP distributes a sext to leaves before extracting
   /// the constant offset. After extraction, it becomes desirable to reunion the
   /// distributed sexts. For example,
   ///
   ///                              &a[sext(i +nsw (j +nsw 5)]
   ///   => distribute              &a[sext(i) +nsw (sext(j) +nsw 5)]
int64_t ConstantOffsetExtractor::Find(Value *Idx, GetElementPtrInst *GEP,
                                       const DominatorTree *DT) {
   // If Idx is an index of an inbound GEP, Idx is guaranteed to be non-negative.
   return ConstantOffsetExtractor(GEP, DT)
       .find(Idx, /* SignExtended */ false, /* ZeroExtended */ false,
             GEP->isInBounds())
       .getSExtValue();
 }
 
-bool SeparateConstOffsetFromGEP::canonicalizeArrayIndicesToPointerSize(
+bool SeparateConstOffsetFromGEP::canonicalizeArrayIndicesToIndexSize(
     GetElementPtrInst *GEP) {
   bool Changed = false;
-  Type *IntPtrTy = DL->getIntPtrType(GEP->getType());
+  Type *PtrIdxTy = DL->getIndexType(GEP->getType());
   gep_type_iterator GTI = gep_type_begin(*GEP);
   for (User::op_iterator I = GEP->op_begin() + 1, E = GEP->op_end();
        I != E; ++I, ++GTI) {
     // Skip struct member indices which must be i32.
     if (GTI.isSequential()) {
-      if ((*I)->getType() != IntPtrTy) {
-        *I = CastInst::CreateIntegerCast(*I, IntPtrTy, true, "idxprom", GEP);
+      if ((*I)->getType() != PtrIdxTy) {
+        *I = CastInst::CreateIntegerCast(*I, PtrIdxTy, true, "idxprom", GEP);
         Changed = true;
       }
     }
   }
   return Changed;
 }
 
 int64_t
for (unsigned I = 1, E = GEP->getNumOperands(); I != E; ++I, ++GTI) {
     }
   }
   return AccumulativeByteOffset;
 }
 
 void SeparateConstOffsetFromGEP::lowerToSingleIndexGEPs(
     GetElementPtrInst *Variadic, int64_t AccumulativeByteOffset) {
   IRBuilder<> Builder(Variadic);
-  Type *IntPtrTy = DL->getIntPtrType(Variadic->getType());
+  Type *PtrIndexTy = DL->getIndexType(Variadic->getType());
 
   Type *I8PtrTy =
       Builder.getInt8PtrTy(Variadic->getType()->getPointerAddressSpace());
   Value *ResultPtr = Variadic->getOperand(0);
   Loop *L = LI->getLoopFor(Variadic->getParent());
   // Check if the base is not loop invariant or used more than once.
   bool isSwapCandidate =
       L && L->isLoopInvariant(ResultPtr) &&
void SeparateConstOffsetFromGEP::lowerToSingleIndexGEPs(
   for (unsigned I = 1, E = Variadic->getNumOperands(); I != E; ++I, ++GTI) {
     if (GTI.isSequential()) {
       Value *Idx = Variadic->getOperand(I);
       // Skip zero indices.
       if (ConstantInt *CI = dyn_cast<ConstantInt>(Idx))
         if (CI->isZero())
           continue;
 
-      APInt ElementSize = APInt(IntPtrTy->getIntegerBitWidth(),
+      APInt ElementSize = APInt(PtrIndexTy->getIntegerBitWidth(),
                                 DL->getTypeAllocSize(GTI.getIndexedType()));
       // Scale the index by element size.
       if (ElementSize != 1) {
         if (ElementSize.isPowerOf2()) {
           Idx = Builder.CreateShl(
-              Idx, ConstantInt::get(IntPtrTy, ElementSize.logBase2()));
+              Idx, ConstantInt::get(PtrIndexTy, ElementSize.logBase2()));
         } else {
-          Idx = Builder.CreateMul(Idx, ConstantInt::get(IntPtrTy, ElementSize));
+          Idx =
+              Builder.CreateMul(Idx, ConstantInt::get(PtrIndexTy, ElementSize));
         }
       }
       // Create an ugly GEP with a single index for each index.
       ResultPtr =
           Builder.CreateGEP(Builder.getInt8Ty(), ResultPtr, Idx, "uglygep");
       if (FirstResult == nullptr)
         FirstResult = ResultPtr;
     }
   }
 
   // Create a GEP with the constant offset index.
   if (AccumulativeByteOffset != 0) {
-    Value *Offset = ConstantInt::get(IntPtrTy, AccumulativeByteOffset);
+    Value *Offset = ConstantInt::get(PtrIndexTy, AccumulativeByteOffset);
     ResultPtr =
         Builder.CreateGEP(Builder.getInt8Ty(), ResultPtr, Offset, "uglygep");
   } else
     isSwapCandidate = false;
 
   // If we created a GEP with constant index, and the base is loop invariant,
   // then we swap the first one with it, so LICM can move constant GEP out
   // later.
   auto *FirstGEP = dyn_cast_or_null<GetElementPtrInst>(FirstResult);
   auto *SecondGEP = dyn_cast<GetElementPtrInst>(ResultPtr);
   if (isSwapCandidate && isLegalToSwapOperand(FirstGEP, SecondGEP, L))
     swapGEPOperand(FirstGEP, SecondGEP);
 
   if (ResultPtr->getType() != Variadic->getType())
     ResultPtr = Builder.CreateBitCast(ResultPtr, Variadic->getType());
 
   Variadic->replaceAllUsesWith(ResultPtr);
   Variadic->eraseFromParent();
 }
 
 void
 SeparateConstOffsetFromGEP::lowerToArithmetics(GetElementPtrInst *Variadic,
                                                int64_t AccumulativeByteOffset) {
   IRBuilder<> Builder(Variadic);
   Type *IntPtrTy = DL->getIntPtrType(Variadic->getType());
+  assert(IntPtrTy == DL->getIndexType(Variadic->getType()) &&
+         "Pointer type must match index type for arithmetic-based lowering of "
+         "split GEPs");
 
   Value *ResultPtr = Builder.CreatePtrToInt(Variadic->getOperand(0), IntPtrTy);
   gep_type_iterator GTI = gep_type_begin(*Variadic);
   // Create ADD/SHL/MUL arithmetic operations for each sequential indices. We
   // don't create arithmetics for structure indices, as they are accumulated
   // in the constant offset index.
   for (unsigned I = 1, E = Variadic->getNumOperands(); I != E; ++I, ++GTI) {
     if (GTI.isSequential()) {
       Value *Idx = Variadic->getOperand(I);
       // Skip zero indices.
bool SeparateConstOffsetFromGEP::splitGEP(GetElementPtrInst *GEP) {
   if (GEP->getType()->isVectorTy())
     return false;
 
   // The backend can already nicely handle the case where all indices are
   // constant.
   if (GEP->hasAllConstantIndices())
     return false;
 
-  bool Changed = canonicalizeArrayIndicesToPointerSize(GEP);
+  bool Changed = canonicalizeArrayIndicesToIndexSize(GEP);
 
   bool NeedsExtraction;
   int64_t AccumulativeByteOffset = accumulateByteOffset(GEP, NeedsExtraction);
 
   if (!NeedsExtraction)
     return Changed;
 
   TargetTransformInfo &TTI = GetTTI(*GEP->getFunction());
bool SeparateConstOffsetFromGEP::splitGEP(GetElementPtrInst *GEP) {
   // TODO(jingyue): do some range analysis to keep as many inbounds as
   // possible. GEPs with inbounds are more friendly to alias analysis.
   bool GEPWasInBounds = GEP->isInBounds();
   GEP->setIsInBounds(false);
 
   // Lowers a GEP to either GEPs with a single index or arithmetic operations.
   if (LowerGEP) {
     // As currently BasicAA does not analyze ptrtoint/inttoptr, do not lower to
     // arithmetic operations if the target uses alias analysis in codegen.
-    if (TTI.useAA())
+    // Additionally, pointers that aren't integral (and so can't be safely
+    // converted to integers) or those whose offset size is different from their
+    // pointer size (which means that doing integer arithmetic on them could
+    // affect that data) can't be lowered in this way.
+    unsigned AddrSpace = GEP->getPointerAddressSpace();
+    bool PointerHasExtraData = DL->getPointerSizeInBits(AddrSpace) !=
+                               DL->getIndexSizeInBits(AddrSpace);
+    if (TTI.useAA() || DL->isNonIntegralAddressSpace(AddrSpace) ||
+        PointerHasExtraData)
       lowerToSingleIndexGEPs(GEP, AccumulativeByteOffset);
     else
       lowerToArithmetics(GEP, AccumulativeByteOffset);
     return true;
   }
 
   // No need to create another GEP if the accumulative byte offset is 0.
   if (AccumulativeByteOffset == 0)
bool SeparateConstOffsetFromGEP::splitGEP(GetElementPtrInst *GEP) {
   Instruction *NewGEP = GEP->clone();
   NewGEP->insertBefore(GEP);
 
   // Per ANSI C standard, signed / unsigned = unsigned and signed % unsigned =
   // unsigned.. Therefore, we cast ElementTypeSizeOfGEP to signed because it is
   // used with unsigned integers later.
   int64_t ElementTypeSizeOfGEP = static_cast<int64_t>(
       DL->getTypeAllocSize(GEP->getResultElementType()));
-  Type *IntPtrTy = DL->getIntPtrType(GEP->getType());
+  Type *PtrIdxTy = DL->getIndexType(GEP->getType());
   if (AccumulativeByteOffset % ElementTypeSizeOfGEP == 0) {
     // Very likely. As long as %gep is naturally aligned, the byte offset we
     // extracted should be a multiple of sizeof(*%gep).
     int64_t Index = AccumulativeByteOffset / ElementTypeSizeOfGEP;
     NewGEP = GetElementPtrInst::Create(GEP->getResultElementType(), NewGEP,
-                                       ConstantInt::get(IntPtrTy, Index, true),
+                                       ConstantInt::get(PtrIdxTy, Index, true),
                                        GEP->getName(), GEP);
     NewGEP->copyMetadata(*GEP);
     // Inherit the inbounds attribute of the original GEP.
     cast<GetElementPtrInst>(NewGEP)->setIsInBounds(GEPWasInBounds);
   } else {
     // Unlikely but possible. For example,
     // #pragma pack(1)
     // struct S {
if (AccumulativeByteOffset % ElementTypeSizeOfGEP == 0) {
     //
     // Emit an uglygep in this case.
     IRBuilder<> Builder(GEP);
     Type *I8PtrTy =
         Builder.getInt8Ty()->getPointerTo(GEP->getPointerAddressSpace());
 
     NewGEP = cast<Instruction>(Builder.CreateGEP(
         Builder.getInt8Ty(), Builder.CreateBitCast(NewGEP, I8PtrTy),
-        {ConstantInt::get(IntPtrTy, AccumulativeByteOffset, true)}, "uglygep",
+        {ConstantInt::get(PtrIdxTy, AccumulativeByteOffset, true)}, "uglygep",
         GEPWasInBounds));
 
     NewGEP->copyMetadata(*GEP);
     NewGEP = cast<Instruction>(Builder.CreateBitCast(NewGEP, GEP->getType()));
   }
 
   GEP->replaceAllUsesWith(NewGEP);
   GEP->eraseFromParent();

llvm/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp

 
 void StraightLineStrengthReduce::allocateCandidatesAndFindBasisForGEP(
     const SCEV *B, ConstantInt *Idx, Value *S, uint64_t ElementSize,
     Instruction *I) {
   // I = B + sext(Idx *nsw S) * ElementSize
   //   = B + (sext(Idx) * sext(S)) * ElementSize
   //   = B + (sext(Idx) * ElementSize) * sext(S)
   // Casting to IntegerType is safe because we skipped vector GEPs.
-  IntegerType *IntPtrTy = cast<IntegerType>(DL->getIntPtrType(I->getType()));
+  IntegerType *PtrIdxTy = cast<IntegerType>(DL->getIndexType(I->getType()));
   ConstantInt *ScaledIdx = ConstantInt::get(
-      IntPtrTy, Idx->getSExtValue() * (int64_t)ElementSize, true);
+      PtrIdxTy, Idx->getSExtValue() * (int64_t)ElementSize, true);
   allocateCandidatesAndFindBasis(Candidate::GEP, B, ScaledIdx, S, I);
 }
 
 void StraightLineStrengthReduce::factorArrayIndex(Value *ArrayIdx,
                                                   const SCEV *Base,
                                                   uint64_t ElementSize,
                                                   GetElementPtrInst *GEP) {
   // At least, ArrayIdx = ArrayIdx *nsw 1.
for (unsigned I = 1, E = GEP->getNumOperands(); I != E; ++I, ++GTI) {
     IndexExprs[I - 1] = SE->getZero(OrigIndexExpr->getType());
 
     // The base of this candidate is GEP's base plus the offsets of all
     // indices except this current one.
     const SCEV *BaseExpr = SE->getGEPExpr(cast<GEPOperator>(GEP), IndexExprs);
     Value *ArrayIdx = GEP->getOperand(I);
     uint64_t ElementSize = DL->getTypeAllocSize(GTI.getIndexedType());
     if (ArrayIdx->getType()->getIntegerBitWidth() <=
-        DL->getPointerSizeInBits(GEP->getAddressSpace())) {
-      // Skip factoring if ArrayIdx is wider than the pointer size, because
-      // ArrayIdx is implicitly truncated to the pointer size.
+        DL->getIndexSizeInBits(GEP->getAddressSpace())) {
+      // Skip factoring if ArrayIdx is wider than the index size, because
+      // ArrayIdx is implicitly truncated to the index size.
       factorArrayIndex(ArrayIdx, BaseExpr, ElementSize, GEP);
     }
     // When ArrayIdx is the sext of a value, we try to factor that value as
     // well.  Handling this case is important because array indices are
-    // typically sign-extended to the pointer size.
+    // typically sign-extended to the pointer index size.
     Value *TruncatedArrayIdx = nullptr;
     if (match(ArrayIdx, m_SExt(m_Value(TruncatedArrayIdx))) &&
         TruncatedArrayIdx->getType()->getIntegerBitWidth() <=
-            DL->getPointerSizeInBits(GEP->getAddressSpace())) {
+            DL->getIndexSizeInBits(GEP->getAddressSpace())) {
       // Skip factoring if TruncatedArrayIdx is wider than the pointer size,
       // because TruncatedArrayIdx is implicitly truncated to the pointer size.
       factorArrayIndex(TruncatedArrayIdx, BaseExpr, ElementSize, GEP);
     }
 
     IndexExprs[I - 1] = OrigIndexExpr;
   }
 }
if (match(Bump, m_Neg(m_Value(NegBump)))) {
       // Y = X + 2 * INT_MAX
       //
       // Neither + and * in the resultant expression are nsw.
       Reduced = Builder.CreateAdd(Basis.Ins, Bump);
     }
     break;
   }
   case Candidate::GEP:
     {
-      Type *IntPtrTy = DL->getIntPtrType(C.Ins->getType());
-      bool InBounds = cast<GetElementPtrInst>(C.Ins)->isInBounds();
-      if (BumpWithUglyGEP) {
-        // C = (char *)Basis + Bump
-        unsigned AS = Basis.Ins->getType()->getPointerAddressSpace();
-        Type *CharTy = Type::getInt8PtrTy(Basis.Ins->getContext(), AS);
-        Reduced = Builder.CreateBitCast(Basis.Ins, CharTy);
-        Reduced =
-            Builder.CreateGEP(Builder.getInt8Ty(), Reduced, Bump, "", InBounds);
-        Reduced = Builder.CreateBitCast(Reduced, C.Ins->getType());
-      } else {
-        // C = gep Basis, Bump
-        // Canonicalize bump to pointer size.
-        Bump = Builder.CreateSExtOrTrunc(Bump, IntPtrTy);
-        Reduced = Builder.CreateGEP(
-            cast<GetElementPtrInst>(Basis.Ins)->getResultElementType(),
-            Basis.Ins, Bump, "", InBounds);
-      }
+    Type *OffsetTy = DL->getIndexType(C.Ins->getType());
+      Bump = Builder.CreateSExtOrTrunc(Bump, OffsetTy);
+          cast<GetElementPtrInst>(Basis.Ins)->getResultElementType(), Basis.Ins,
+          Bump, "", InBounds);
       break;
     }
   default:
     llvm_unreachable("C.CandidateKind is invalid");
   };
   Reduced->takeName(C.Ins);
   C.Ins->replaceAllUsesWith(Reduced);
   // Unlink C.Ins so that we can skip other candidates also corresponding to

llvm/lib/Transforms/Utils/FunctionComparator.cpp

int FunctionComparator::cmpGEPs(const GEPOperator *GEPL,
   unsigned int ASR = GEPR->getPointerAddressSpace();
 
   if (int Res = cmpNumbers(ASL, ASR))
     return Res;
 
   // When we have target data, we can reduce the GEP down to the value in bytes
   // added to the address.
   const DataLayout &DL = FnL->getParent()->getDataLayout();
-  unsigned BitWidth = DL.getPointerSizeInBits(ASL);
-  APInt OffsetL(BitWidth, 0), OffsetR(BitWidth, 0);
+  unsigned OffsetBitWidth = DL.getIndexSizeInBits(ASL);
+  APInt OffsetL(OffsetBitWidth, 0), OffsetR(OffsetBitWidth, 0);
   if (GEPL->accumulateConstantOffset(DL, OffsetL) &&
       GEPR->accumulateConstantOffset(DL, OffsetR))
     return cmpAPInts(OffsetL, OffsetR);
   if (int Res =
           cmpTypes(GEPL->getSourceElementType(), GEPR->getSourceElementType()))
     return Res;
 
   if (int Res = cmpNumbers(GEPL->getNumOperands(), GEPR->getNumOperands()))

llvm/lib/Transforms/Vectorize/LoadStoreVectorizer.cpp

bool Vectorizer::isConsecutiveAccess(Value *A, Value *B) {
   Type *PtrBTy = getLoadStoreType(B);
   if (PtrA == PtrB ||
       PtrATy->isVectorTy() != PtrBTy->isVectorTy() ||
       DL.getTypeStoreSize(PtrATy) != DL.getTypeStoreSize(PtrBTy) ||
       DL.getTypeStoreSize(PtrATy->getScalarType()) !=
           DL.getTypeStoreSize(PtrBTy->getScalarType()))
     return false;
 
-  unsigned PtrBitWidth = DL.getPointerSizeInBits(ASA);
-  APInt Size(PtrBitWidth, DL.getTypeStoreSize(PtrATy));
+  unsigned PtrOffsetWidth = DL.getIndexSizeInBits(ASA);
+  APInt Size(PtrOffsetWidth, DL.getTypeStoreSize(PtrATy));
 
   return areConsecutivePointers(PtrA, PtrB, Size);
 }
 
 bool Vectorizer::areConsecutivePointers(Value *PtrA, Value *PtrB,
                                         APInt PtrDelta, unsigned Depth) const {
-  unsigned PtrBitWidth = DL.getPointerTypeSizeInBits(PtrA->getType());
-  APInt OffsetA(PtrBitWidth, 0);
-  APInt OffsetB(PtrBitWidth, 0);
+  unsigned OffsetBitWidth = DL.getIndexTypeSizeInBits(PtrA->getType());
+  APInt OffsetA(OffsetBitWidth, 0);
+  APInt OffsetB(OffsetBitWidth, 0);
   PtrA = PtrA->stripAndAccumulateInBoundsConstantOffsets(DL, OffsetA);
   PtrB = PtrB->stripAndAccumulateInBoundsConstantOffsets(DL, OffsetB);
 
   unsigned NewPtrBitWidth = DL.getTypeStoreSizeInBits(PtrA->getType());
 
   if (NewPtrBitWidth != DL.getTypeStoreSizeInBits(PtrB->getType()))
     return false;
 

llvm/test/Instrumentation/BoundsChecking/simple.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=bounds-checking -S | FileCheck %s
-target datalayout = "e-p:64:64:64-p1:16:16:16-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
+target datalayout = "e-p:64:64:64-p1:16:16:16-p2:64:64:64:48-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 
 @.str = private constant [8 x i8] c"abcdefg\00"
 
 @.str_as1 = private addrspace(1) constant [8 x i8] c"abcdefg\00"
 
+@.str_as2 = private addrspace(2) constant [8 x i8] c"abcdefg\00"
+
 
 declare noalias ptr @malloc(i64) nounwind allocsize(0)
 declare noalias ptr @calloc(i64, i64) nounwind allocsize(0,1)
 declare noalias ptr @realloc(ptr nocapture allocptr, i64) nounwind allocsize(1)
 
 define void @f1() nounwind {
 ; CHECK-LABEL: @f1(
 ; CHECK-NEXT:    [[TMP1:%.*]] = tail call ptr @malloc(i64 32)

 ; CHECK-NEXT:    call void @llvm.trap() #[[ATTR6]]
 ; CHECK-NEXT:    unreachable
 ;
   %idx = getelementptr inbounds [8 x i8], ptr addrspace(1) @.str_as1, i64 0, i64 %x
   %1 = load i8, ptr addrspace(1) %idx, align 4
   ret void
 }
 
+define void @f5_as2(i32 %x) nounwind {;
+; CHECK-LABEL: @f5_as2(
+; CHECK-NEXT:    [[X_C:%.*]] = sext i32 [[X:%.*]] to i48
+; CHECK-NEXT:    [[TMP1:%.*]] = add i48 0, [[X_C]]
+; CHECK-NEXT:    [[IDX:%.*]] = getelementptr inbounds [8 x i8], ptr addrspace(2) @.str_as2, i32 0, i32 [[X]]
+; CHECK-NEXT:    [[TMP2:%.*]] = sub i48 8, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp ult i48 8, [[TMP1]]
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp ult i48 [[TMP2]], 1
+; CHECK-NEXT:    [[TMP5:%.*]] = or i1 [[TMP3]], [[TMP4]]
+; CHECK-NEXT:    br i1 [[TMP5]], label [[TRAP:%.*]], label [[TMP6:%.*]]
+; CHECK:       6:
+; CHECK-NEXT:    [[TMP7:%.*]] = load i8, ptr addrspace(2) [[IDX]], align 4
+; CHECK-NEXT:    ret void
+; CHECK:       trap:
+; CHECK-NEXT:    call void @llvm.trap() #[[ATTR6]]
+; CHECK-NEXT:    unreachable
+;
+  %idx = getelementptr inbounds [8 x i8], ptr addrspace(2) @.str_as2, i32 0, i32 %x
+  %1 = load i8, ptr addrspace(2) %idx, align 4
+  ret void
+}
+
 define void @f6(i64 %x) nounwind {
 ; CHECK-LABEL: @f6(
 ; CHECK-NEXT:    [[TMP1:%.*]] = alloca i128, align 8
 ; CHECK-NEXT:    [[TMP2:%.*]] = load i128, ptr [[TMP1]], align 4
 ; CHECK-NEXT:    ret void
 ;
   %1 = alloca i128
   %2 = load i128, ptr %1, align 4

llvm/test/Transforms/InstCombine/alloca-intptr-not-sizet.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=instcombine -S | FileCheck %s
 target datalayout = "p0:64:64:64-p7:128:128:128:32-A7"
 
 define void @test_array_alloca_intptr_not_sizet(i64 %size, ptr %dest) {
 ; CHECK-LABEL: @test_array_alloca_intptr_not_sizet(
-; CHECK-NEXT:    [[TMP1:%.*]] = zext i64 [[SIZE:%.*]] to i128
-; CHECK-NEXT:    [[ALLOCA:%.*]] = alloca i8, i128 [[TMP1]], align 1, addrspace(7)
+; CHECK-NEXT:    [[TMP1:%.*]] = trunc i64 [[SIZE:%.*]] to i32
+; CHECK-NEXT:    [[ALLOCA:%.*]] = alloca i8, i32 [[TMP1]], align 1, addrspace(7)
 ; CHECK-NEXT:    store ptr addrspace(7) [[ALLOCA]], ptr [[DEST:%.*]], align 16
 ; CHECK-NEXT:    ret void
 ;
   %alloca = alloca i8, i64 %size, addrspace(7)
   store ptr addrspace(7) %alloca, ptr %dest
   ret void
 }

llvm/test/Transforms/InstCombine/load-cmp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -passes=instcombine -S -data-layout="p:32:32:32-p1:16:16:16-n8:16:32:64" < %s | FileCheck %s
+; RUN: opt -passes=instcombine -S -data-layout="p:32:32:32-p1:16:16:16-p2:128:128:128:32-n8:16:32:64" < %s | FileCheck %s
 
 @G16 = internal constant [10 x i16] [i16 35, i16 82, i16 69, i16 81, i16 85,
   i16 73, i16 82, i16 69, i16 68, i16 0]
 
 @G16_as1 = internal addrspace(1) constant [10 x i16] [i16 35, i16 82, i16 69, i16 81, i16 85,
   i16 73, i16 82, i16 69, i16 68, i16 0]
 
+@G16_as2 = internal addrspace(2) constant [10 x i16] [i16 35, i16 82, i16 69, i16 81, i16 85,
+  i16 73, i16 82, i16 69, i16 68, i16 0]
+
 @GD = internal constant [6 x double]
   [double -10.0, double 1.0, double 4.0, double 2.0, double -20.0, double -40.0]
 
 %Foo = type { i32, i32, i32, i32 }
 
 @GS = internal constant %Foo { i32 1, i32 4, i32 9, i32 14 }
 
 @GStructArr = internal constant [4 x %Foo] [ %Foo { i32 1, i32 4, i32 9, i32 14 },

 ;
   %p = getelementptr [10 x i16], ptr addrspace(1) @G16_as1, i16 0, i32 %x
   %q = load i16, ptr addrspace(1) %p
   %r = icmp eq i16 %q, 0
   ret i1 %r
 
 }
 
+define i1 @test1_noinbounds_as2(i64 %x) {
+; CHECK-LABEL: @test1_noinbounds_as2(
+; CHECK-NEXT:    [[TMP1:%.*]] = and i64 [[X:%.*]], 2147483647
+; CHECK-NEXT:    [[R:%.*]] = icmp eq i64 [[TMP1]], 9
+; CHECK-NEXT:    ret i1 [[R]]
+;
+  %p = getelementptr [10 x i16], ptr addrspace(2) @G16_as2, i16 0, i64 %x
+  %q = load i16, ptr addrspace(2) %p
+  %r = icmp eq i16 %q, 0
+  ret i1 %r
+
+}
+
 define i1 @test2(i32 %X) {
 ; CHECK-LABEL: @test2(
 ; CHECK-NEXT:    [[R:%.*]] = icmp ne i32 [[X:%.*]], 4
 ; CHECK-NEXT:    ret i1 [[R]]
 ;
   %P = getelementptr inbounds [10 x i16], ptr @G16, i32 0, i32 %X
   %Q = load i16, ptr %P
   %R = icmp slt i16 %Q, 85

llvm/test/Transforms/LoadStoreVectorizer/AMDGPU/merge-vectors.ll

 ; RUN: opt -mtriple=amdgcn-amd-amdhsa -passes=load-store-vectorizer -S -o - %s | FileCheck %s
 
-target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5"
+target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-ni:7"
 
 ; CHECK-LABEL: @merge_v2i32_v2i32(
 ; CHECK: load <4 x i32>
 ; CHECK: store <4 x i32> zeroinitializer
 define amdgpu_kernel void @merge_v2i32_v2i32(ptr addrspace(1) nocapture %a, ptr addrspace(1) nocapture readonly %b) #0 {
 entry:
   %a.1 = getelementptr inbounds <2 x i32>, ptr addrspace(1) %a, i64 1
   %b.1 = getelementptr inbounds <2 x i32>, ptr addrspace(1) %b, i64 1
entry:
   %ld.c.idx.1 = load <2 x i16>, ptr addrspace(1) %b.1, align 4
 
   store <2 x i16> zeroinitializer, ptr addrspace(1) %a, align 4
   store <2 x i16> zeroinitializer, ptr addrspace(1) %a.1, align 4
 
   ret void
 }
 
+; CHECK-LABEL: @merge_fat_ptrs(
+; CHECK: load <4 x i16>
+; CHECK: store <4 x i16> zeroinitializer
+define amdgpu_kernel void @merge_fat_ptrs(ptr addrspace(7) nocapture %a, ptr addrspace(7) nocapture readonly %b) #0 {
+entry:
+  %a.1 = getelementptr inbounds <2 x i16>, ptr addrspace(7) %a, i32 1
+  %b.1 = getelementptr inbounds <2 x i16>, ptr addrspace(7) %b, i32 1
+
+  %ld.c = load <2 x i16>, ptr addrspace(7) %b, align 4
+  %ld.c.idx.1 = load <2 x i16>, ptr addrspace(7) %b.1, align 4
+
+  store <2 x i16> zeroinitializer, ptr addrspace(7) %a, align 4
+  store <2 x i16> zeroinitializer, ptr addrspace(7) %a.1, align 4
+
+  ret void
+}
+
 ; Ideally this would be merged
 ; CHECK-LABEL: @merge_load_i32_v2i16(
 ; CHECK: load i32,
 ; CHECK: load <2 x i16>
 define amdgpu_kernel void @merge_load_i32_v2i16(ptr addrspace(1) nocapture %a) #0 {
 entry:
   %a.1 = getelementptr inbounds i32, ptr addrspace(1) %a, i32 1
 
   %ld.0 = load i32, ptr addrspace(1) %a
   %ld.1 = load <2 x i16>, ptr addrspace(1) %a.1
 
   ret void
 }
 
 attributes #0 = { nounwind }
 attributes #1 = { nounwind readnone }

llvm/test/Transforms/LowerTypeTests/distinct-index-width-crash.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -passes=lowertypetests < %s | FileCheck %s
+target datalayout = "e-p:64:64:64:32"
+
+@a = constant i32 1, !type !0
+@b = constant [2 x i32] [i32 2, i32 3], !type !1
+
+!0 = !{i32 0, !"typeid1"}
+!1 = !{i32 4, !"typeid1"}
+
+declare i1 @llvm.type.test(ptr %ptr, metadata %bitset) nounwind readnone
+
+define i1 @bar() {
+; CHECK-LABEL: @bar(
+; CHECK-NEXT:    ret i1 true
+;
+  %x = call i1 @llvm.type.test(ptr getelementptr ([2 x i32], ptr @b, i32 0, i32 1), metadata !"typeid1")
+  ret i1 %x
+}

llvm/test/Transforms/MergeFunc/different-index-width-gep-crash.ll

+; RUN: opt -passes=mergefunc -disable-output < %s
+
+target datalayout = "e-p:64:64-p2:128:128:128:32"
+
+define void @foo(ptr addrspace(2) %x) {
+    %tmp = getelementptr i32, ptr addrspace(2) %x, i32 1
+    ret void
+}
+
+define void @bar(ptr addrspace(2) %x) {
+    %tmp = getelementptr i32, ptr addrspace(2) %x, i32 1
+    ret void
+}

llvm/test/Transforms/MergeICmps/X86/distinct-index-width-crash.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=mergeicmps -verify-dom-info -S | FileCheck %s
+
+target triple = "x86_64"
+
+; This is very much not an x86 ABI, in current use, but we're testing
+; that we've fixed a bug where accumulateConstantOffset() was called incorrectly.
+target datalayout = "e-p:64:64:64:32"
+
+; Define a cunstom data layout that has index width < pointer width
+; and make sure that doesn't mreak anything
+define void @fat_ptrs(ptr dereferenceable(16) %a, ptr dereferenceable(16) %b) {
+; CHECK-LABEL: @fat_ptrs(
+; CHECK-NEXT:  bb0:
+; CHECK-NEXT:    [[PTR_A1:%.*]] = getelementptr inbounds [2 x i64], ptr [[A:%.*]], i32 0, i32 1
+; CHECK-NEXT:    [[PTR_B1:%.*]] = getelementptr inbounds [2 x i64], ptr [[B:%.*]], i32 0, i32 1
+; CHECK-NEXT:    br label %"bb1+bb2"
+; CHECK:       "bb1+bb2":
+; CHECK-NEXT:    [[MEMCMP:%.*]] = call i32 @memcmp(ptr [[A]], ptr [[B]], i64 16)
+; CHECK-NEXT:    [[TMP0:%.*]] = icmp eq i32 [[MEMCMP]], 0
+; CHECK-NEXT:    br label [[BB3:%.*]]
+; CHECK:       bb3:
+; CHECK-NEXT:    ret void
+;
+bb0:
+  %ptr_a1 = getelementptr inbounds [2 x i64], ptr %a, i32 0, i32 1
+  %ptr_b1 = getelementptr inbounds [2 x i64], ptr %b, i32 0, i32 1
+  br label %bb1
+
+bb1:                                              ; preds = %bb0
+  %a0 = load i64, ptr %a
+  %b0 = load i64, ptr %b
+  %cond0 = icmp eq i64 %a0, %b0
+  br i1 %cond0, label %bb2, label %bb3
+
+bb2:                                              ; preds = %bb1
+  %a1 = load i64, ptr %ptr_a1
+  %b1 = load i64, ptr %ptr_b1
+  %cond1 = icmp eq i64 %a1, %b1
+  br label %bb3
+
+bb3:                                              ; preds = %bb2, %bb1
+  %necessary = phi i1 [ %cond1, %bb2 ], [ false, %bb1 ]
+  ret void
+}

llvm/test/Transforms/NaryReassociate/nary-gep.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=nary-reassociate -S | FileCheck %s
+
+target datalayout = "e-p:64:64-p1:32:32:32-p2:128:128:128:32-i64:64-v16:16-v32:32-n32:64"
+
+; Even though addrspace(2) has 128 bit pointers, no sign extension needed
+; becasue it has 32-bit offsets.
+define void @no_sext_fat_pointer(ptr addrspace(2) %a, i32 %i, i32 %j) {
+; CHECK-LABEL: @no_sext_fat_pointer(
+; CHECK-NEXT:    [[V2:%.*]] = getelementptr float, ptr addrspace(2) [[A:%.*]], i32 [[I:%.*]]
+; CHECK-NEXT:    call void @foo(ptr addrspace(2) [[V2]])
+; CHECK-NEXT:    [[V3:%.*]] = getelementptr float, ptr addrspace(2) [[V2]], i32 [[J:%.*]]
+; CHECK-NEXT:    call void @foo(ptr addrspace(2) [[V3]])
+; CHECK-NEXT:    ret void
+;
+  %v1 = add i32 %i, %j
+  %v2 = getelementptr float, ptr addrspace(2) %a, i32 %i
+  call void @foo(ptr addrspace(2) %v2)
+  %v3 = getelementptr float, ptr addrspace(2) %a, i32 %v1
+  call void @foo(ptr addrspace(2) %v3)
+  ret void
+}
+
+declare void @foo(ptr addrspace(2))

llvm/test/Transforms/SeparateConstOffsetFromGEP/pointer-type-not-offset-type.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=separate-const-offset-from-gep -S | FileCheck %s
 target datalayout = "p0:64:64:64-p1:128:128:128:32"
 
 define void @should_be_trunc(ptr addrspace(1) %ptr,  i64 %index, ptr %result) {
 ; CHECK-LABEL: @should_be_trunc(
-; CHECK-NEXT:    [[IDXPROM:%.*]] = sext i64 [[INDEX:%.*]] to i128
-; CHECK-NEXT:    [[GEP:%.*]] = getelementptr ptr addrspace(1), ptr addrspace(1) [[PTR:%.*]], i128 [[IDXPROM]]
+; CHECK-NEXT:    [[IDXPROM:%.*]] = trunc i64 [[INDEX:%.*]] to i32
+; CHECK-NEXT:    [[GEP:%.*]] = getelementptr ptr addrspace(1), ptr addrspace(1) [[PTR:%.*]], i32 [[IDXPROM]]
 ; CHECK-NEXT:    store ptr addrspace(1) [[GEP]], ptr [[RESULT:%.*]], align 16
 ; CHECK-NEXT:    ret void
 ;
   %gep = getelementptr ptr addrspace(1), ptr addrspace(1) %ptr, i64 %index
   store ptr addrspace(1) %gep, ptr %result
   ret void
 }

llvm/test/Transforms/StraightLineStrengthReduce/slsr-gep.ll

 ; RUN: opt < %s -passes=slsr,gvn -S | FileCheck %s
 ; RUN: opt < %s -passes='slsr,gvn' -S | FileCheck %s
 
-target datalayout = "e-i64:64-v16:16-v32:32-n16:32:64-p:64:64:64-p1:32:32:32"
+target datalayout = "e-i64:64-v16:16-v32:32-n16:32:64-p:64:64:64-p1:32:32:32-p2:128:128:128:32"
 
 ; foo(input[0]);
 ; foo(input[s]);
 ; foo(input[s * 2]);
 ;   =>
 ; p0 = &input[0];
 ; foo(*p);
 ; p1 = p0 + s;
; CHECK-LABEL: @slsr_gep_32bit_pointer(
   %p2 = getelementptr inbounds i32, ptr addrspace(1) %input, i64 %s2
   ; %s2 is wider than the pointer size of addrspace(1), so do not factor it.
 ; CHECK: %p2 = getelementptr inbounds i32, ptr addrspace(1) %input, i64 %s2
   call void @baz(ptr addrspace(1) %p2)
 
   ret void
 }
 
+define void @slsr_gep_fat_pointer(ptr addrspace(2) %input, i32 %s) {
+  ; p1 = &input[s]
+  %p1 = getelementptr inbounds i32, ptr addrspace(2) %input, i32 %s
+  call void @baz2(ptr addrspace(2) %p1)
+
+  ; p2 = &input[s * 2]
+  %s2 = mul nsw i32 %s, 2
+  %p2 = getelementptr inbounds i32, ptr addrspace(2) %input, i32 %s2
+; CHECK: %p2 = getelementptr inbounds i32, ptr addrspace(2) %p1, i32 %s
+  ; Use index bitwidth, not pointer size (i128)
+  call void @baz2(ptr addrspace(2) %p2)
+
+  ret void
+}
+
+
 declare void @foo(ptr)
 declare void @bar(ptr)
 declare void @baz(ptr addrspace(1))
+declare void @baz2(ptr addrspace(2))