[ConstFold] Constant fold load through invariant group intrinsics

[aeubanks]

Summary

Loads should be able to look through invariant group intrinsics when
we're trying to constant fold.

This fixes some major code size regressions due to RTTI when
-fstrict-vtable-pointers is turned on.

Since invariant group intrinsics can't be constants,
ConstantFoldLoadFromConstPtr had to be changed to accept Values instead
of Constants. A majority of this patch is fixing up functions to work
with a Value instead of a Constant.

SymbolicallyEvaluateGEP has a new option to strip pointers specifically
for load constant folding. We can't strip all-zero GEPs since we walk
GlobalVariable constants using GEP indices, so introduce a new pointer
stripping type. We need this to simplify GEPs that have an operand that
derive from an invariant group intrinsic. Otherwise we can never see
that the GEP pointer operand could be a constant GlobalVariable.

The new pointer stripping mode also looks through local aliases to local
aliasees, fixing devirtualization with MSVC RTTI.

[aeubanks]

update

[nikic]

This looks like a pretty serious layering violation, in that constants folding is now no longer folding constants. LLVM pretty strictly separates ConstantFold (folding instructions with all-constant operands), InstSimplify (folding instructions to a constant or existing instruction) and InstCombine (folding arbitrarily). This patch blurs the line between ConstantFold and InstSimplify.

llvm/test/Transforms/InstCombine/invariant.group-load.ll

4

Please prefer update_test_checks.py unless there is some specific reason not to use it (e.g. huge output). Also, if this already folds with just -instsimplify, please test it there instead.

[aeubanks]

In D100923#2706417, @nikic wrote:

This looks like a pretty serious layering violation, in that constants folding is now no longer folding constants. LLVM pretty strictly separates ConstantFold (folding instructions with all-constant operands), InstSimplify (folding instructions to a constant or existing instruction) and InstCombine (folding arbitrarily). This patch blurs the line between ConstantFold and InstSimplify.

That makes sense.
So instead I should add a SimplifyFold to llvm::SimplifyInstruction?

[aeubanks]

abandoning in favor of D101103 and D101100

llvm/include/llvm/Analysis/ConstantFolding.h

 class APInt;
 template <typename T> class ArrayRef;
 class CallBase;
 class Constant;
 class ConstantExpr;
 class DSOLocalEquivalent;
 class DataLayout;
 class Function;
+class GEPOperator;
 class GlobalValue;
 class Instruction;
 class TargetLibraryInfo;
 class Type;
+class Value;
 
 /// If this constant is a constant offset from a global, return the global and
 /// the constant. Because of constantexprs, this function is recursive.
 /// If the global is part of a dso_local_equivalent constant, return it through
 /// `Equiv` if it is provided.
 bool IsConstantOffsetFromGlobal(Constant *C, GlobalValue *&GV, APInt &Offset,
                                 const DataLayout &DL,
                                 DSOLocalEquivalent **DSOEquiv = nullptr);

 
 /// Attempt to constant fold a shufflevector instruction with the
 /// specified operands and mask.  See class ShuffleVectorInst for a description
 /// of the mask representation. The constant result is returned if successful;
 /// if not, null is returned.
 Constant *ConstantFoldShuffleVectorInstruction(Constant *V1, Constant *V2,
                                                ArrayRef<int> Mask);
 
-/// ConstantFoldLoadFromConstPtr - Return the value that a load from C would
+/// ConstantFoldLoadFromValue - Return the value that a load from V would
 /// produce if it is constant and determinable.  If this is not determinable,
 /// return null.
-Constant *ConstantFoldLoadFromConstPtr(Constant *C, Type *Ty, const DataLayout &DL);
+Constant *ConstantFoldLoadFromValue(Value *V, Type *Ty, const DataLayout &DL);
 
-/// ConstantFoldLoadThroughGEPConstantExpr - Given a constant and a
-/// getelementptr constantexpr, return the constant value being addressed by the
+/// ConstantFoldLoadThroughGEP - Given a constant and a
+/// getelementptr, return the constant value being addressed by the
 /// constant expression, or null if something is funny and we can't decide.
-Constant *ConstantFoldLoadThroughGEPConstantExpr(Constant *C, ConstantExpr *CE,
-                                                 Type *Ty,
-                                                 const DataLayout &DL);
+Constant *ConstantFoldLoadThroughGEP(Constant *C, GEPOperator *GEP, Type *Ty,
 
 /// ConstantFoldLoadThroughGEPIndices - Given a constant and getelementptr
 /// indices (with an *implied* zero pointer index that is not in the list),
 /// return the constant value being addressed by a virtual load, or null if
 /// something is funny and we can't decide.
 Constant *ConstantFoldLoadThroughGEPIndices(Constant *C,
                                             ArrayRef<Constant *> Indices);
 

llvm/lib/Analysis/ConstantFolding.cpp

 #include <cfenv>
 #include <cmath>
 #include <cstddef>
 #include <cstdint>
 
 using namespace llvm;
 
 namespace {
+Constant *SymbolicallyEvaluateGEP(const GEPOperator *GEP, Value *Ptr,
+                                  ArrayRef<Constant *> Idxs,
+                                  const DataLayout &DL,
+                                  const TargetLibraryInfo *TLI,
+                                  bool StripForAliasAnalysis);
 
 //===----------------------------------------------------------------------===//
 // Constant Folding internal helper functions
 //===----------------------------------------------------------------------===//
 
 static Constant *foldConstVectorToAPInt(APInt &Result, Type *DestTy,
                                         Constant *C, Type *SrcEltTy,
                                         unsigned NumSrcElts,
for (unsigned i = 1; i != BytesLoaded; ++i) {
       ResultVal <<= 8;
       ResultVal |= RawBytes[i];
     }
   }
 
   return ConstantInt::get(IntType->getContext(), ResultVal);
 }
 
-Constant *ConstantFoldLoadThroughBitcastExpr(ConstantExpr *CE, Type *DestTy,
+Constant *ConstantFoldLoadThroughBitcastExpr(BitCastOperator *BC, Type *DestTy,
                                              const DataLayout &DL) {
-  auto *SrcPtr = CE->getOperand(0);
+  auto *SrcPtr = BC->getOperand(0);
   if (!SrcPtr->getType()->isPointerTy())
     return nullptr;
 
-  return ConstantFoldLoadFromConstPtr(SrcPtr, DestTy, DL);
+  return ConstantFoldLoadFromValue(SrcPtr, DestTy, DL);
 }
 
 } // end anonymous namespace
 
-Constant *llvm::ConstantFoldLoadFromConstPtr(Constant *C, Type *Ty,
-                                             const DataLayout &DL) {
+Constant *llvm::ConstantFoldLoadFromValue(Value *V, Type *Ty,
   // First, try the easy cases:
-  if (auto *GV = dyn_cast<GlobalVariable>(C))
+  if (auto *GV = dyn_cast<GlobalVariable>(V))
     if (GV->isConstant() && GV->hasDefinitiveInitializer())
       return ConstantFoldLoadThroughBitcast(GV->getInitializer(), Ty, DL);
 
-  if (auto *GA = dyn_cast<GlobalAlias>(C))
+  if (auto *GA = dyn_cast<GlobalAlias>(V))
     if (GA->getAliasee() && !GA->isInterposable())
-      return ConstantFoldLoadFromConstPtr(GA->getAliasee(), Ty, DL);
+      return ConstantFoldLoadFromValue(GA->getAliasee(), Ty, DL);
 
-  // If the loaded value isn't a constant expr, we can't handle it.
-  auto *CE = dyn_cast<ConstantExpr>(C);
-  if (!CE)
+  if (auto *GEP = dyn_cast<GEPOperator>(V)) {
+    SmallVector<Constant *> Idxs;
+    for (unsigned I = 1, E = GEP->getNumOperands(); I != E; ++I) {
+      if (auto *Idx = dyn_cast<Constant>(GEP->getOperand(I)))
+        Idxs.push_back(Idx);
+      else
         return nullptr;
-
-  if (CE->getOpcode() == Instruction::GetElementPtr) {
-    if (auto *GV = dyn_cast<GlobalVariable>(CE->getOperand(0))) {
+    }
+    if (auto *Simplified = SymbolicallyEvaluateGEP(
+            GEP, GEP->getPointerOperand(), Idxs, DL, nullptr,
+            /*StripForAliasAnalysis*/ true)) {
+      if (auto *SimplifiedGEP = dyn_cast<GEPOperator>(Simplified)) {
+        if (auto *GV =
+                dyn_cast<GlobalVariable>(SimplifiedGEP->getPointerOperand())) {
           if (GV->isConstant() && GV->hasDefinitiveInitializer()) {
-        if (Constant *V = ConstantFoldLoadThroughGEPConstantExpr(
-                GV->getInitializer(), CE, Ty, DL))
+            if (Constant *V = ConstantFoldLoadThroughGEP(GV->getInitializer(),
+                                                         GEP, Ty, DL)) {
+              assert(V->getType() == Ty && "wrong type");
               return V;
             }
           }
         }
+      } else {
+        return ConstantFoldLoadFromValue(Simplified, Ty, DL);
+      }
+    }
+  }
 
-  if (CE->getOpcode() == Instruction::BitCast)
-    if (Constant *LoadedC = ConstantFoldLoadThroughBitcastExpr(CE, Ty, DL))
+  if (auto *BC = dyn_cast<BitCastOperator>(V)) {
+    if (Constant *LoadedC = ConstantFoldLoadThroughBitcastExpr(BC, Ty, DL)) {
+      assert(LoadedC->getType() == Ty && "wrong type");
       return LoadedC;
+    }
+  }
+
+  if (auto *Call = dyn_cast<CallBase>(V)) {
+    if (Call->getIntrinsicID() == Intrinsic::strip_invariant_group ||
+        Call->getIntrinsicID() == Intrinsic::launder_invariant_group) {
+      return ConstantFoldLoadFromValue(Call->getOperand(0), Ty, DL);
+    }
+  }
 
   // Instead of loading constant c string, use corresponding integer value
   // directly if string length is small enough.
   StringRef Str;
-  if (getConstantStringInfo(CE, Str) && !Str.empty()) {
+  if (getConstantStringInfo(V, Str) && !Str.empty()) {
     size_t StrLen = Str.size();
     unsigned NumBits = Ty->getPrimitiveSizeInBits();
     // Replace load with immediate integer if the result is an integer or fp
     // value.
     if ((NumBits >> 3) == StrLen + 1 && (NumBits & 7) == 0 &&
         (isa<IntegerType>(Ty) || Ty->isFloatingPointTy())) {
       APInt StrVal(NumBits, 0);
       APInt SingleChar(NumBits, 0);
       if (DL.isLittleEndian()) {
         for (unsigned char C : reverse(Str.bytes())) {
           SingleChar = static_cast<uint64_t>(C);
           StrVal = (StrVal << 8) | SingleChar;
         }
       } else {
         for (unsigned char C : Str.bytes()) {
           SingleChar = static_cast<uint64_t>(C);
           StrVal = (StrVal << 8) | SingleChar;
         }
         // Append NULL at the end.
         SingleChar = 0;
         StrVal = (StrVal << 8) | SingleChar;
       }
 
-      Constant *Res = ConstantInt::get(CE->getContext(), StrVal);
+      Constant *Res = ConstantInt::get(V->getContext(), StrVal);
       if (Ty->isFloatingPointTy())
         Res = ConstantExpr::getBitCast(Res, Ty);
       return Res;
     }
   }
 
   // If this load comes from anywhere in a constant global, and if the global
   // is all undef or zero, we know what it loads.
-  if (auto *GV = dyn_cast<GlobalVariable>(getUnderlyingObject(CE))) {
+  if (auto *GV = dyn_cast<GlobalVariable>(getUnderlyingObject(V))) {
     if (GV->isConstant() && GV->hasDefinitiveInitializer()) {
       if (GV->getInitializer()->isNullValue())
         return Constant::getNullValue(Ty);
       if (isa<UndefValue>(GV->getInitializer()))
         return UndefValue::get(Ty);
     }
   }
 
   // Try hard to fold loads from bitcasted strange and non-type-safe things.
-  return FoldReinterpretLoadFromConstPtr(CE, Ty, DL);
+  if (auto *C = dyn_cast<Constant>(V))
+    return FoldReinterpretLoadFromConstPtr(C, Ty, DL);
+
+  return nullptr;
 }
 
 namespace {
 
 Constant *ConstantFoldLoadInst(const LoadInst *LI, const DataLayout &DL) {
   if (LI->isVolatile()) return nullptr;
 
-  if (auto *C = dyn_cast<Constant>(LI->getOperand(0)))
-    return ConstantFoldLoadFromConstPtr(C, LI->getType(), DL);
-
-  return nullptr;
+  return ConstantFoldLoadFromValue(LI->getOperand(0), LI->getType(), DL);
 }
 
 /// One of Op0/Op1 is a constant expression.
 /// Attempt to symbolically evaluate the result of a binary operator merging
 /// these together.  If target data info is available, it is provided as DL,
 /// otherwise DL is null.
 Constant *SymbolicallyEvaluateBinop(unsigned Opc, Constant *Op0, Constant *Op1,
                                     const DataLayout &DL) {
if (IsConstantOffsetFromGlobal(Op0, GV1, Offs1, DL))
       }
   }
 
   return nullptr;
 }
 
 /// If array indices are not pointer-sized integers, explicitly cast them so
 /// that they aren't implicitly casted by the getelementptr.
-Constant *CastGEPIndices(Type *SrcElemTy, ArrayRef<Constant *> Ops,
-                         Type *ResultTy, Optional<unsigned> InRangeIndex,
-                         const DataLayout &DL, const TargetLibraryInfo *TLI) {
+Constant *CastGEPIndices(Type *SrcElemTy, Constant *Ptr,
+                         ArrayRef<Constant *> Idxs, Type *ResultTy,
+                         Optional<unsigned> InRangeIndex, const DataLayout &DL,
+                         const TargetLibraryInfo *TLI) {
   Type *IntIdxTy = DL.getIndexType(ResultTy);
   Type *IntIdxScalarTy = IntIdxTy->getScalarType();
 
   bool Any = false;
   SmallVector<Constant*, 32> NewIdxs;
-  for (unsigned i = 1, e = Ops.size(); i != e; ++i) {
-    if ((i == 1 ||
-         !isa<StructType>(GetElementPtrInst::getIndexedType(
-             SrcElemTy, Ops.slice(1, i - 1)))) &&
-        Ops[i]->getType()->getScalarType() != IntIdxScalarTy) {
+  for (unsigned i = 0, e = Idxs.size(); i != e; ++i) {
+    if ((i == 0 || !isa<StructType>(GetElementPtrInst::getIndexedType(
+                       SrcElemTy, Idxs.slice(0, i)))) &&
+        Idxs[i]->getType()->getScalarType() != IntIdxScalarTy) {
       Any = true;
-      Type *NewType = Ops[i]->getType()->isVectorTy()
-                          ? IntIdxTy
-                          : IntIdxScalarTy;
-      NewIdxs.push_back(ConstantExpr::getCast(CastInst::getCastOpcode(Ops[i],
-                                                                      true,
-                                                                      NewType,
-                                                                      true),
-                                              Ops[i], NewType));
+      Type *NewType =
+          Idxs[i]->getType()->isVectorTy() ? IntIdxTy : IntIdxScalarTy;
+      NewIdxs.push_back(ConstantExpr::getCast(
+          CastInst::getCastOpcode(Idxs[i], true, NewType, true), Idxs[i],
+          NewType));
     } else
-      NewIdxs.push_back(Ops[i]);
+      NewIdxs.push_back(Idxs[i]);
   }
 
   if (!Any)
     return nullptr;
 
   Constant *C = ConstantExpr::getGetElementPtr(
-      SrcElemTy, Ops[0], NewIdxs, /*InBounds=*/false, InRangeIndex);
+      SrcElemTy, Ptr, NewIdxs, /*InBounds=*/false, InRangeIndex);
   return ConstantFoldConstant(C, DL, TLI);
 }
 
-/// Strip the pointer casts, but preserve the address space information.
-Constant *StripPtrCastKeepAS(Constant *Ptr, Type *&ElemTy) {
+/// Strip pointer casts, but preserve address space information.
+Value *StripPtrCastKeepAS(Value *Ptr, Type *&ElemTy,
+                          bool StripForAliasAnalysis) {
   assert(Ptr->getType()->isPointerTy() && "Not a pointer type");
   auto *OldPtrTy = cast<PointerType>(Ptr->getType());
-  Ptr = cast<Constant>(Ptr->stripPointerCasts());
-  auto *NewPtrTy = cast<PointerType>(Ptr->getType());
+  Constant *NewPtr;
+  if (StripForAliasAnalysis)
+    NewPtr = dyn_cast<Constant>(Ptr->stripPointerCastsForAliasAnalysis());
+  else
+    NewPtr = dyn_cast<Constant>(Ptr->stripPointerCasts());
+  if (!NewPtr)
+    return Ptr;
+  auto *NewPtrTy = cast<PointerType>(NewPtr->getType());
 
   ElemTy = NewPtrTy->getPointerElementType();
 
   // Preserve the address space number of the pointer.
   if (NewPtrTy->getAddressSpace() != OldPtrTy->getAddressSpace()) {
     NewPtrTy = ElemTy->getPointerTo(OldPtrTy->getAddressSpace());
-    Ptr = ConstantExpr::getPointerCast(Ptr, NewPtrTy);
+    NewPtr = ConstantExpr::getPointerCast(NewPtr, NewPtrTy);
   }
-  return Ptr;
+  return NewPtr;
 }
 
 /// If we can symbolically evaluate the GEP constant expression, do so.
-Constant *SymbolicallyEvaluateGEP(const GEPOperator *GEP,
-                                  ArrayRef<Constant *> Ops,
+Constant *SymbolicallyEvaluateGEP(const GEPOperator *GEP, Value *Ptr,
+                                  ArrayRef<Constant *> Idxs,
                                   const DataLayout &DL,
-                                  const TargetLibraryInfo *TLI) {
+                                  const TargetLibraryInfo *TLI,
+                                  bool StripForAliasAnalysis) {
   const GEPOperator *InnermostGEP = GEP;
   bool InBounds = GEP->isInBounds();
 
   Type *SrcElemTy = GEP->getSourceElementType();
   Type *ResElemTy = GEP->getResultElementType();
   Type *ResTy = GEP->getType();
   if (!SrcElemTy->isSized() || isa<ScalableVectorType>(SrcElemTy))
     return nullptr;
 
-  if (Constant *C = CastGEPIndices(SrcElemTy, Ops, ResTy,
+  if (Constant *PtrC = dyn_cast<Constant>(Ptr)) {
+    if (Constant *C = CastGEPIndices(SrcElemTy, PtrC, Idxs, ResTy,
                                      GEP->getInRangeIndex(), DL, TLI))
       return C;
+  }
 
-  Constant *Ptr = Ops[0];
   if (!Ptr->getType()->isPointerTy())
     return nullptr;
 
   Type *IntIdxTy = DL.getIndexType(Ptr->getType());
 
   // If this is a constant expr gep that is effectively computing an
   // "offsetof", fold it into 'cast int Size to T*' instead of 'gep 0, 0, 12'
-  for (unsigned i = 1, e = Ops.size(); i != e; ++i)
-      if (!isa<ConstantInt>(Ops[i])) {
-
-        // If this is "gep i8* Ptr, (sub 0, V)", fold this as:
-        // "inttoptr (sub (ptrtoint Ptr), V)"
-        if (Ops.size() == 2 && ResElemTy->isIntegerTy(8)) {
-          auto *CE = dyn_cast<ConstantExpr>(Ops[1]);
-          assert((!CE || CE->getType() == IntIdxTy) &&
-                 "CastGEPIndices didn't canonicalize index types!");
-          if (CE && CE->getOpcode() == Instruction::Sub &&
-              CE->getOperand(0)->isNullValue()) {
-            Constant *Res = ConstantExpr::getPtrToInt(Ptr, CE->getType());
+  for (unsigned i = 0, e = Idxs.size(); i != e; ++i) {
+    if (!isa<ConstantInt>(Idxs[i])) {
+      if (Idxs.size() == 1 && ResElemTy->isIntegerTy(8)) {
+        auto *CE = dyn_cast<ConstantExpr>(Idxs[0]);
+          if (auto *PtrC = dyn_cast<Constant>(Ptr)) {
+            Constant *Res = ConstantExpr::getPtrToInt(PtrC, CE->getType());
             Res = ConstantExpr::getSub(Res, CE->getOperand(1));
             Res = ConstantExpr::getIntToPtr(Res, ResTy);
             return ConstantFoldConstant(Res, DL, TLI);
           }
         }
-        return nullptr;
-      }
+      }
+  }
 
   unsigned BitWidth = DL.getTypeSizeInBits(IntIdxTy);
-  APInt Offset =
-      APInt(BitWidth,
-            DL.getIndexedOffsetInType(
-                SrcElemTy,
-                makeArrayRef((Value * const *)Ops.data() + 1, Ops.size() - 1)));
-  Ptr = StripPtrCastKeepAS(Ptr, SrcElemTy);
+  APInt Offset = APInt(
+      BitWidth,
+          SrcElemTy, makeArrayRef((Value *const *)Idxs.begin(), Idxs.size())));
+  Ptr = StripPtrCastKeepAS(Ptr, SrcElemTy, StripForAliasAnalysis);
 
   // If this is a GEP of a GEP, fold it all into a single GEP.
   while (auto *GEP = dyn_cast<GEPOperator>(Ptr)) {
     InnermostGEP = GEP;
     InBounds &= GEP->isInBounds();
 
     SmallVector<Value *, 4> NestedOps(GEP->op_begin() + 1, GEP->op_end());
 
     // Do not try the incorporate the sub-GEP if some index is not a number.
     bool AllConstantInt = true;
     for (Value *NestedOp : NestedOps)
       if (!isa<ConstantInt>(NestedOp)) {
         AllConstantInt = false;
         break;
       }
     if (!AllConstantInt)
       break;
 
-    Ptr = cast<Constant>(GEP->getOperand(0));
+    Ptr = dyn_cast<Constant>(GEP->getOperand(0));
+    if (!Ptr)
+      return nullptr;
     SrcElemTy = GEP->getSourceElementType();
     Offset += APInt(BitWidth, DL.getIndexedOffsetInType(SrcElemTy, NestedOps));
-    Ptr = StripPtrCastKeepAS(Ptr, SrcElemTy);
+    Ptr = StripPtrCastKeepAS(Ptr, SrcElemTy, StripForAliasAnalysis);
   }
 
   // If the base value for this address is a literal integer value, fold the
   // getelementptr to the resulting integer value casted to the pointer type.
   APInt BasePtr(BitWidth, 0);
   if (auto *CE = dyn_cast<ConstantExpr>(Ptr)) {
     if (CE->getOpcode() == Instruction::IntToPtr) {
       if (auto *Base = dyn_cast<ConstantInt>(CE->getOperand(0)))
         BasePtr = Base->getValue().zextOrTrunc(BitWidth);
     }
   }
 
   auto *PTy = cast<PointerType>(Ptr->getType());
-  if ((Ptr->isNullValue() || BasePtr != 0) &&
+  auto *PtrC = dyn_cast<Constant>(Ptr);
+  if (!PtrC)
+    return nullptr;
+  if ((PtrC->isNullValue() || BasePtr != 0) &&
       !DL.isNonIntegralPointerType(PTy)) {
-    Constant *C = ConstantInt::get(Ptr->getContext(), Offset + BasePtr);
+    Constant *C = ConstantInt::get(PtrC->getContext(), Offset + BasePtr);
     return ConstantExpr::getIntToPtr(C, ResTy);
   }
 
   // Otherwise form a regular getelementptr. Recompute the indices so that
   // we eliminate over-indexing of the notional static type array bounds.
   // This makes it easy to determine if the getelementptr is "inbounds".
   // Also, this helps GlobalOpt do SROA on GlobalVariables.
   Type *Ty = PTy;
if (SrcElemTy == InnermostGEP->getSourceElementType() &&
         NewIdxs.size() > *LastIRIndex) {
       InRangeIndex = LastIRIndex;
       for (unsigned I = 0; I <= *LastIRIndex; ++I)
         if (NewIdxs[I] != InnermostGEP->getOperand(I + 1))
           return nullptr;
     }
 
   // Create a GEP.
-  Constant *C = ConstantExpr::getGetElementPtr(SrcElemTy, Ptr, NewIdxs,
+  Constant *C = ConstantExpr::getGetElementPtr(SrcElemTy, PtrC, NewIdxs,
                                                InBounds, InRangeIndex);
   assert(C->getType()->getPointerElementType() == Ty &&
          "Computed GetElementPtr has unexpected type!");
 
   // If we ended up indexing a member with a type that doesn't match
   // the type of what the original indices indexed, add a cast.
   if (Ty != ResElemTy)
     C = FoldBitCast(C, ResTy, DL);
Constant *ConstantFoldInstOperandsImpl(const Value *InstOrCE, unsigned Opcode,
 
   if (Instruction::isBinaryOp(Opcode))
     return ConstantFoldBinaryOpOperands(Opcode, Ops[0], Ops[1], DL);
 
   if (Instruction::isCast(Opcode))
     return ConstantFoldCastOperand(Opcode, Ops[0], DestTy, DL);
 
   if (auto *GEP = dyn_cast<GEPOperator>(InstOrCE)) {
-    if (Constant *C = SymbolicallyEvaluateGEP(GEP, Ops, DL, TLI))
+    if (Constant *C =
+            SymbolicallyEvaluateGEP(GEP, Ops[0], Ops.slice(1), DL, TLI,
+                                    /*StripForAliasAnalysis*/ false))
       return C;
 
     return ConstantExpr::getGetElementPtr(GEP->getSourceElementType(), Ops[0],
                                           Ops.slice(1), GEP->isInBounds(),
                                           GEP->getInRangeIndex());
   }
 
   if (auto *CE = dyn_cast<ConstantExpr>(InstOrCE))
for (Value *Incoming : PN->incoming_values()) {
         return nullptr;
       CommonValue = C;
     }
 
     // If we reach here, all incoming values are the same constant or undef.
     return CommonValue ? CommonValue : UndefValue::get(PN->getType());
   }
 
+  if (const auto *LI = dyn_cast<LoadInst>(I))
+    return ConstantFoldLoadInst(LI, DL);
+
   // Scan the operand list, checking to see if they are all constants, if so,
   // hand off to ConstantFoldInstOperandsImpl.
   if (!all_of(I->operands(), [](Use &U) { return isa<Constant>(U); }))
     return nullptr;
 
   SmallDenseMap<Constant *, Constant *> FoldedOps;
   SmallVector<Constant *, 8> Ops;
   for (const Use &OpU : I->operands()) {
     auto *Op = cast<Constant>(&OpU);
     // Fold the Instruction's operands.
     Op = ConstantFoldConstantImpl(Op, DL, TLI, FoldedOps);
     Ops.push_back(Op);
   }
 
   if (const auto *CI = dyn_cast<CmpInst>(I))
     return ConstantFoldCompareInstOperands(CI->getPredicate(), Ops[0], Ops[1],
                                            DL, TLI);
 
-  if (const auto *LI = dyn_cast<LoadInst>(I))
-    return ConstantFoldLoadInst(LI, DL);
-
   if (auto *IVI = dyn_cast<InsertValueInst>(I)) {
     return ConstantExpr::getInsertValue(
                                 cast<Constant>(IVI->getAggregateOperand()),
                                 cast<Constant>(IVI->getInsertedValueOperand()),
                                 IVI->getIndices());
   }
 
   if (auto *EVI = dyn_cast<ExtractValueInst>(I)) {
Constant *llvm::ConstantFoldCastOperand(unsigned Opcode, Constant *C,
   case Instruction::FPToSI:
   case Instruction::AddrSpaceCast:
       return ConstantExpr::getCast(Opcode, C, DestTy);
   case Instruction::BitCast:
     return FoldBitCast(C, DestTy, DL);
   }
 }
 
-Constant *llvm::ConstantFoldLoadThroughGEPConstantExpr(Constant *C,
-                                                       ConstantExpr *CE,
-                                                       Type *Ty,
-                                                       const DataLayout &DL) {
-  if (!CE->getOperand(1)->isNullValue())
+Constant *llvm::ConstantFoldLoadThroughGEP(Constant *C, GEPOperator *GEP,
+                                           Type *Ty, const DataLayout &DL) {
+  auto *O = dyn_cast<Constant>(GEP->getOperand(1));
+  if (!O || !O->isNullValue())
     return nullptr;  // Do not allow stepping over the value!
 
   // Loop over all of the operands, tracking down which value we are
   // addressing.
-  for (unsigned i = 2, e = CE->getNumOperands(); i != e; ++i) {
-    C = C->getAggregateElement(CE->getOperand(i));
+  for (unsigned I = 2, E = GEP->getNumOperands(); I != E; ++I) {
+    auto *Idx = dyn_cast<Constant>(GEP->getOperand(I));
+    if (!Idx)
+      return nullptr;
+    C = C->getAggregateElement(Idx);
     if (!C)
       return nullptr;
   }
   return ConstantFoldLoadThroughBitcast(C, Ty, DL);
 }
 
 Constant *
 llvm::ConstantFoldLoadThroughGEPIndices(Constant *C,
static Constant *ConstantFoldFixedVectorCall(
   Type *Ty = FVTy->getElementType();
 
   switch (IntrinsicID) {
   case Intrinsic::masked_load: {
     auto *SrcPtr = Operands[0];
     auto *Mask = Operands[2];
     auto *Passthru = Operands[3];
 
-    Constant *VecData = ConstantFoldLoadFromConstPtr(SrcPtr, FVTy, DL);
+    Constant *VecData = ConstantFoldLoadFromValue(SrcPtr, FVTy, DL);
 
     SmallVector<Constant *, 32> NewElements;
     for (unsigned I = 0, E = FVTy->getNumElements(); I != E; ++I) {
       auto *MaskElt = Mask->getAggregateElement(I);
       if (!MaskElt)
         break;
       auto *PassthruElt = Passthru->getAggregateElement(I);
       auto *VecElt = VecData ? VecData->getAggregateElement(I) : nullptr;

llvm/lib/Analysis/InstructionSimplify.cpp

if (!AllowRefinement && canCreatePoison(cast<Operator>(I)))
     return nullptr;
 
   if (CmpInst *C = dyn_cast<CmpInst>(I))
     return ConstantFoldCompareInstOperands(C->getPredicate(), ConstOps[0],
                                            ConstOps[1], Q.DL, Q.TLI);
 
   if (LoadInst *LI = dyn_cast<LoadInst>(I))
     if (!LI->isVolatile())
-      return ConstantFoldLoadFromConstPtr(ConstOps[0], LI->getType(), Q.DL);
+      return ConstantFoldLoadFromValue(ConstOps[0], LI->getType(), Q.DL);
 
   return ConstantFoldInstOperands(I, ConstOps, Q.DL, Q.TLI);
 }
 
 Value *llvm::SimplifyWithOpReplaced(Value *V, Value *Op, Value *RepOp,
                                     const SimplifyQuery &Q,
                                     bool AllowRefinement) {
   return ::SimplifyWithOpReplaced(V, Op, RepOp, Q, AllowRefinement,
static Value *SimplifyRelativeLoad(Constant *Ptr, Constant *Offset,
 
   uint64_t OffsetInt = OffsetConstInt->getSExtValue();
   if (OffsetInt % 4 != 0)
     return nullptr;
 
   Constant *C = ConstantExpr::getGetElementPtr(
       Int32Ty, ConstantExpr::getBitCast(Ptr, Int32PtrTy),
       ConstantInt::get(Int64Ty, OffsetInt / 4));
-  Constant *Loaded = ConstantFoldLoadFromConstPtr(C, Int32Ty, DL);
+  Constant *Loaded = ConstantFoldLoadFromValue(C, Int32Ty, DL);
   if (!Loaded)
     return nullptr;
 
   auto *LoadedCE = dyn_cast<ConstantExpr>(Loaded);
   if (!LoadedCE)
     return nullptr;
 
   if (LoadedCE->getOpcode() == Instruction::Trunc) {

llvm/lib/Analysis/ScalarEvolution.cpp

for (unsigned i = 0, e = I->getNumOperands(); i != e; ++i) {
     Operands[i] = C;
   }
 
   if (CmpInst *CI = dyn_cast<CmpInst>(I))
     return ConstantFoldCompareInstOperands(CI->getPredicate(), Operands[0],
                                            Operands[1], DL, TLI);
   if (LoadInst *LI = dyn_cast<LoadInst>(I)) {
     if (!LI->isVolatile())
-      return ConstantFoldLoadFromConstPtr(Operands[0], LI->getType(), DL);
+      return ConstantFoldLoadFromValue(Operands[0], LI->getType(), DL);
   }
   return ConstantFoldInstOperands(I, Operands, DL, TLI);
 }
 
 
 // If every incoming value to PN except the one for BB is a specific Constant,
 // return that, else return nullptr.
 static Constant *getOtherIncomingValue(PHINode *PN, BasicBlock *BB) {
if (Instruction *I = dyn_cast<Instruction>(SU->getValue())) {
         if (MadeImprovement) {
           Constant *C = nullptr;
           const DataLayout &DL = getDataLayout();
           if (const CmpInst *CI = dyn_cast<CmpInst>(I))
             C = ConstantFoldCompareInstOperands(CI->getPredicate(), Operands[0],
                                                 Operands[1], DL, &TLI);
           else if (const LoadInst *Load = dyn_cast<LoadInst>(I)) {
             if (!Load->isVolatile())
-              C = ConstantFoldLoadFromConstPtr(Operands[0], Load->getType(),
-                                               DL);
+              C = ConstantFoldLoadFromValue(Operands[0], Load->getType(), DL);
           } else
             C = ConstantFoldInstOperands(I, Operands, DL, &TLI);
           if (!C) return V;
           return getSCEV(C);
         }
       }
     }
 

llvm/lib/CodeGen/ExpandMemCmp.cpp

RhsSource = Builder.CreateConstGEP1_64(
         OffsetBytes);
     LhsAlign = commonAlignment(LhsAlign, OffsetBytes);
     RhsAlign = commonAlignment(RhsAlign, OffsetBytes);
   }
   LhsSource = Builder.CreateBitCast(LhsSource, LoadSizeType->getPointerTo());
   RhsSource = Builder.CreateBitCast(RhsSource, LoadSizeType->getPointerTo());
 
   // Create a constant or a load from the source.
-  Value *Lhs = nullptr;
-  if (auto *C = dyn_cast<Constant>(LhsSource))
-    Lhs = ConstantFoldLoadFromConstPtr(C, LoadSizeType, DL);
+  Value *Lhs = ConstantFoldLoadFromValue(LhsSource, LoadSizeType, DL);
   if (!Lhs)
     Lhs = Builder.CreateAlignedLoad(LoadSizeType, LhsSource, LhsAlign);
 
-  Value *Rhs = nullptr;
-  if (auto *C = dyn_cast<Constant>(RhsSource))
-    Rhs = ConstantFoldLoadFromConstPtr(C, LoadSizeType, DL);
+  Value *Rhs = ConstantFoldLoadFromValue(RhsSource, LoadSizeType, DL);
   if (!Rhs)
     Rhs = Builder.CreateAlignedLoad(LoadSizeType, RhsSource, RhsAlign);
 
   // Swap bytes if required.
   if (NeedsBSwap) {
     Function *Bswap = Intrinsic::getDeclaration(CI->getModule(),
                                                 Intrinsic::bswap, LoadSizeType);
     Lhs = Builder.CreateCall(Bswap, Lhs);

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (const Constant *LoadInput = dyn_cast<Constant>(PtrVal)) {
     Type *LoadTy =
         Type::getIntNTy(PtrVal->getContext(), LoadVT.getScalarSizeInBits());
     if (LoadVT.isVector())
       LoadTy = FixedVectorType::get(LoadTy, LoadVT.getVectorNumElements());
 
     LoadInput = ConstantExpr::getBitCast(const_cast<Constant *>(LoadInput),
                                          PointerType::getUnqual(LoadTy));
 
-    if (const Constant *LoadCst = ConstantFoldLoadFromConstPtr(
+    if (const Constant *LoadCst = ConstantFoldLoadFromValue(
             const_cast<Constant *>(LoadInput), LoadTy, *Builder.DL))
       return Builder.getValue(LoadCst);
   }
 
   // Otherwise, we have to emit the load.  If the pointer is to unfoldable but
   // still constant memory, the input chain can be the entry node.
   SDValue Root;
   bool ConstantMemory = false;

llvm/lib/Transforms/IPO/GlobalOpt.cpp

if (LoadInst *LI = dyn_cast<LoadInst>(U)) {
           Changed = true;
         }
       }
     } else if (StoreInst *SI = dyn_cast<StoreInst>(U)) {
       // Store must be unreachable or storing Init into the global.
       SI->eraseFromParent();
       Changed = true;
     } else if (ConstantExpr *CE = dyn_cast<ConstantExpr>(U)) {
-      if (CE->getOpcode() == Instruction::GetElementPtr) {
+      if (auto *GEP = dyn_cast<GEPOperator>(CE)) {
         Constant *SubInit = nullptr;
         if (Init)
-          SubInit = ConstantFoldLoadThroughGEPConstantExpr(
-              Init, CE, V->getType()->getPointerElementType(), DL);
+          SubInit = ConstantFoldLoadThroughGEP(
+              Init, GEP, V->getType()->getPointerElementType(), DL);
         Changed |= CleanupConstantGlobalUsers(CE, SubInit, DL, GetTLI);
       } else if ((CE->getOpcode() == Instruction::BitCast &&
                   CE->getType()->isPointerTy()) ||
                  CE->getOpcode() == Instruction::AddrSpaceCast) {
         // Pointer cast, delete any stores and memsets to the global.
         Changed |= CleanupConstantGlobalUsers(CE, nullptr, DL, GetTLI);
       }
 
       if (CE->use_empty()) {
         CE->destroyConstant();
         Changed = true;
       }
     } else if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(U)) {
       // Do not transform "gepinst (gep constexpr (GV))" here, because forming
       // "gepconstexpr (gep constexpr (GV))" will cause the two gep's to fold
       // and will invalidate our notion of what Init is.
       Constant *SubInit = nullptr;
       if (!isa<ConstantExpr>(GEP->getOperand(0))) {
         ConstantExpr *CE = dyn_cast_or_null<ConstantExpr>(
             ConstantFoldInstruction(GEP, DL, &GetTLI(*GEP->getFunction())));
         if (Init && CE && CE->getOpcode() == Instruction::GetElementPtr)
-          SubInit = ConstantFoldLoadThroughGEPConstantExpr(
-              Init, CE, V->getType()->getPointerElementType(), DL);
+          SubInit = ConstantFoldLoadThroughGEP(
+              Init, cast<GEPOperator>(GEP),
+              V->getType()->getPointerElementType(), DL);
 
         // If the initializer is an all-null value and we have an inbounds GEP,
         // we already know what the result of any load from that GEP is.
         // TODO: Handle splats.
         if (Init && isa<ConstantAggregateZero>(Init) && GEP->isInBounds())
           SubInit = Constant::getNullValue(GEP->getResultElementType());
       }
       Changed |= CleanupConstantGlobalUsers(GEP, SubInit, DL, GetTLI);

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

if (isInstructionTriviallyDead(I, &TLI)) {
       ++NumDeadInst;
       continue;
     }
 
     if (!DebugCounter::shouldExecute(VisitCounter))
       continue;
 
     // Instruction isn't dead, see if we can constant propagate it.
-    if (!I->use_empty() &&
-        (I->getNumOperands() == 0 || isa<Constant>(I->getOperand(0)))) {
+    if (!I->use_empty()) {
       if (Constant *C = ConstantFoldInstruction(I, DL, &TLI)) {
         LLVM_DEBUG(dbgs() << "IC: ConstFold to: " << *C << " from: " << *I
                           << '\n');
 
         // Add operands to the worklist.
         replaceInstUsesWith(*I, C);
         ++NumConstProp;
         if (isInstructionTriviallyDead(I, &TLI))
do {
     // We have now visited this block!  If we've already been here, ignore it.
     if (!Visited.insert(BB).second)
       continue;
 
     for (BasicBlock::iterator BBI = BB->begin(), E = BB->end(); BBI != E; ) {
       Instruction *Inst = &*BBI++;
 
       // ConstantProp instruction if trivially constant.
-      if (!Inst->use_empty() &&
-          (Inst->getNumOperands() == 0 || isa<Constant>(Inst->getOperand(0))))
+      if (!Inst->use_empty())
         if (Constant *C = ConstantFoldInstruction(Inst, DL, TLI)) {
           LLVM_DEBUG(dbgs() << "IC: ConstFold to: " << *C << " from: " << *Inst
                             << '\n');
           Inst->replaceAllUsesWith(C);
           ++NumConstProp;
           if (isInstructionTriviallyDead(Inst, TLI))
             Inst->eraseFromParent();
           MadeIRChange = true;

llvm/lib/Transforms/Utils/Evaluator.cpp

if (CE->getOpcode() == Instruction::GetElementPtr &&
       ConstantInt *CI = dyn_cast<ConstantInt>(*std::next(CE->op_begin()));
       if (!CI || !CI->isZero()) return false;
 
       // The remaining indices must be compile-time known integers within the
       // notional bounds of the corresponding static array types.
       if (!CE->isGEPWithNoNotionalOverIndexing())
         return false;
 
-      return ConstantFoldLoadThroughGEPConstantExpr(
-          GV->getInitializer(), CE,
+      return ConstantFoldLoadThroughGEP(
+          GV->getInitializer(), cast<GEPOperator>(CE),
           cast<GEPOperator>(CE)->getResultElementType(), DL);
     } else if (CE->getOpcode() == Instruction::BitCast &&
                isa<GlobalVariable>(CE->getOperand(0))) {
       // A constantexpr bitcast from a pointer to another pointer is a no-op,
       // and we know how to evaluate it by moving the bitcast from the pointer
       // operand to the value operand.
       // Do not allow weak/*_odr/linkonce/dllimport/dllexport linkage or
       // external globals.
if (GlobalVariable *GV = dyn_cast<GlobalVariable>(P)) {
     return nullptr;
   }
 
   if (ConstantExpr *CE = dyn_cast<ConstantExpr>(P)) {
     switch (CE->getOpcode()) {
     // Handle a constantexpr getelementptr.
     case Instruction::GetElementPtr:
       if (auto *I = getInitializer(CE->getOperand(0)))
-        return ConstantFoldLoadThroughGEPConstantExpr(I, CE, Ty, DL);
+        return ConstantFoldLoadThroughGEP(I, cast<GEPOperator>(CE), Ty, DL);
       break;
     // Handle a constantexpr bitcast.
     case Instruction::BitCast:
       // We're evaluating a load through a pointer that was bitcast to a
       // different type. See if the "from" pointer has recently been stored.
       // If it hasn't, we may still be able to find a stored pointer by
       // introspecting the type.
       Constant *Val =

llvm/lib/Transforms/Utils/SCCPSolver.cpp

if (auto *GV = dyn_cast<GlobalVariable>(Ptr)) {
         if (It != TrackedGlobals.end()) {
           mergeInValue(IV, &I, It->second, getMaxWidenStepsOpts());
           return;
         }
       }
     }
 
     // Transform load from a constant into a constant if possible.
-    if (Constant *C = ConstantFoldLoadFromConstPtr(Ptr, I.getType(), DL)) {
+    if (Constant *C = ConstantFoldLoadFromValue(Ptr, I.getType(), DL)) {
       if (isa<UndefValue>(C))
         return;
       return (void)markConstant(IV, &I, C);
     }
   }
 
   // Fall back to metadata.
   mergeInValue(&I, getValueFromMetadata(&I));

llvm/lib/Transforms/Utils/SimplifyLibCalls.cpp

static Value *optimizeMemCmpConstantSize(CallInst *CI, Value *LHS, Value *RHS,
   // memcmp(S1,S2,N/8)==0 -> (*(intN_t*)S1 != *(intN_t*)S2)==0
   // TODO: The case where both inputs are constants does not need to be limited
   // to legal integers or equality comparison. See block below this.
   if (DL.isLegalInteger(Len * 8) && isOnlyUsedInZeroEqualityComparison(CI)) {
     IntegerType *IntType = IntegerType::get(CI->getContext(), Len * 8);
     unsigned PrefAlignment = DL.getPrefTypeAlignment(IntType);
 
     // First, see if we can fold either argument to a constant.
-    Value *LHSV = nullptr;
-    if (auto *LHSC = dyn_cast<Constant>(LHS)) {
-      LHSC = ConstantExpr::getBitCast(LHSC, IntType->getPointerTo());
-      LHSV = ConstantFoldLoadFromConstPtr(LHSC, IntType, DL);
-    }
-    Value *RHSV = nullptr;
-    if (auto *RHSC = dyn_cast<Constant>(RHS)) {
-      RHSC = ConstantExpr::getBitCast(RHSC, IntType->getPointerTo());
-      RHSV = ConstantFoldLoadFromConstPtr(RHSC, IntType, DL);
-    }
+    Value *LHSV = ConstantFoldLoadFromValue(LHS, IntType, DL);
+    Value *RHSV = ConstantFoldLoadFromValue(RHS, IntType, DL);
 
     // Don't generate unaligned loads. If either source is constant data,
     // alignment doesn't matter for that source because there is no load.
     if ((LHSV || getKnownAlignment(LHS, DL, CI) >= PrefAlignment) &&
         (RHSV || getKnownAlignment(RHS, DL, CI) >= PrefAlignment)) {
       if (!LHSV) {
         Type *LHSPtrTy =
             IntType->getPointerTo(LHS->getType()->getPointerAddressSpace());

llvm/lib/Transforms/Utils/VNCoercion.cpp

if (Offset) {
     Src = ConstantExpr::getBitCast(Src,
                                    Type::getInt8PtrTy(Src->getContext(), AS));
     Constant *OffsetCst =
         ConstantInt::get(Type::getInt64Ty(Src->getContext()), (unsigned)Offset);
     Src = ConstantExpr::getGetElementPtr(Type::getInt8Ty(Src->getContext()),
                                          Src, OffsetCst);
   }
   Src = ConstantExpr::getBitCast(Src, PointerType::get(LoadTy, AS));
-  if (ConstantFoldLoadFromConstPtr(Src, LoadTy, DL))
+  if (ConstantFoldLoadFromValue(Src, LoadTy, DL))
     return Offset;
   return -1;
 }
 
 template <class T, class HelperClass>
 static T *getStoreValueForLoadHelper(T *SrcVal, unsigned Offset, Type *LoadTy,
                                      HelperClass &Helper,
                                      const DataLayout &DL) {
if (Offset) {
     Src = ConstantExpr::getBitCast(Src,
                                    Type::getInt8PtrTy(Src->getContext(), AS));
     Constant *OffsetCst =
         ConstantInt::get(Type::getInt64Ty(Src->getContext()), (unsigned)Offset);
     Src = ConstantExpr::getGetElementPtr(Type::getInt8Ty(Src->getContext()),
                                          Src, OffsetCst);
   }
   Src = ConstantExpr::getBitCast(Src, PointerType::get(LoadTy, AS));
-  return ConstantFoldLoadFromConstPtr(Src, LoadTy, DL);
+  return ConstantFoldLoadFromValue(Src, LoadTy, DL);
 }
 
 /// This function is called when we have a
 /// memdep query of a load that ends up being a clobbering mem intrinsic.
 Value *getMemInstValueForLoad(MemIntrinsic *SrcInst, unsigned Offset,
                               Type *LoadTy, Instruction *InsertPt,
                               const DataLayout &DL) {
   IRBuilder<> Builder(InsertPt);

llvm/test/Transforms/InstCombine/invariant.group-load.ll

+; RUN: opt -passes=instcombine -S < %s | FileCheck %s
+
+; CHECK: ret i64 3
+
+@A = linkonce_odr hidden constant { i64, i64 } { i64 2, i64 3 }
+
+declare i8* @llvm.strip.invariant.group.p0i8(i8* %p)
+
+define i64 @f() {
+        %p = bitcast { i64, i64 }* @A to i8*
+        %a = call i8* @llvm.strip.invariant.group.p0i8(i8* %p)
+        %b = getelementptr i8, i8* %a, i32 8
+        %c = bitcast i8* %b to i64*
+        %d = load i64, i64* %c
+        ret i64 %d
+}