[CSSPGO] Unblock optimizations with pseudo probe instrumentation part 3.

[hoy]

Summary

This patch continues unblocking optimizations that are blocked by pseudo probe instrumentation.

Not exactly like DbgIntrinsics, PseudoProbe intrinsic has other attributes (such as mayread, maywrite, mayhaveSideEffect) that can block optimizations. The issues fixed are:

• Flipped default param of getFirstNonPHIOrDbg API to skip pseudo probes
• Unblocked CSE by avoiding pseudo probe from clobbering memory SSA
• Unblocked induction variable simpliciation
• Allow empty loop deletion by treating probe intrinsic isDroppable
• Some refactoring.

[wenlei]

Thanks for the changes to reduce probe overhead. As we chatted off patch, some information and verification of this change's impact on profile quality would be helpful.

[hoy]

In D110847#3035672, @wenlei wrote:

Thanks for the changes to reduce probe overhead. As we chatted off patch, some information and verification of this change's impact on profile quality would be helpful.

The original patch contained several general changes, e.g, flipping the default values of those skipping APIs, which were observed to affect profile quality. After narrowing down the offending changes that are in SimplifyCFG, reverting them brought back the profile quality while still reduced the probe overhead in a decent amount.

[hoy]

Updating D110847: [CSSPGO] Unblock optimizations with pseudo probe instrumentation part 3.

[wenlei]

In D110847#3055368, @hoy wrote:

In D110847#3035672, @wenlei wrote:

Thanks for the changes to reduce probe overhead. As we chatted off patch, some information and verification of this change's impact on profile quality would be helpful.

The original patch contained several general changes, e.g, flipping the default values of those skipping APIs, which were observed to affect profile quality. After narrowing down the offending changes that are in SimplifyCFG, reverting them brought back the profile quality while still reduced the probe overhead in a decent amount.

What are the specific optimizations that were critical for profile quality in SimplifyCFG? On the other hand, it looks to me that some of the other unblocked optimizations like InstCombine could also have some impact on profile quality.
The change looks good, but I think it'd also be nice to have some insight and reasoning as to why certain optimizations have be blocked for profile quality, beyond experiment results which could vary from one workload to another.

[hoy]

In D110847#3058762, @wenlei wrote:

In D110847#3055368, @hoy wrote:

In D110847#3035672, @wenlei wrote:

Thanks for the changes to reduce probe overhead. As we chatted off patch, some information and verification of this change's impact on profile quality would be helpful.

The original patch contained several general changes, e.g, flipping the default values of those skipping APIs, which were observed to affect profile quality. After narrowing down the offending changes that are in SimplifyCFG, reverting them brought back the profile quality while still reduced the probe overhead in a decent amount.

What are the specific optimizations that were critical for profile quality in SimplifyCFG? On the other hand, it looks to me that some of the other unblocked optimizations like InstCombine could also have some impact on profile quality.
The change looks good, but I think it'd also be nice to have some insight and reasoning as to why certain optimizations have be blocked for profile quality, beyond experiment results which could vary from one workload to another.

Good question. In general SimplifyCFG is more disruptive to the flow graph by folding, merging or removing blocks. Such flow graph changes affected the execution frequency of blocks, and samples collected on the new blocks have a problem to correlate to original blocks. A particular case is

/// If this basic block is simple enough, and if a predecessor branches to us
/// and one of our successors, fold the block into the predecessor and use
/// logical operations to pick the right destination.
bool llvm::FoldBranchToCommonDest(BranchInst *BI, DomTreeUpdater *DTU,

Compared to SimplifyCFG, InstCombine is more like optimizing on instruction level. Moving an instruction around or folding instructions are likely having a smaller impact on the shape of flow graph and block frequencies.

[wenlei]

In D110847#3058794, @hoy wrote:

In D110847#3058762, @wenlei wrote:

In D110847#3055368, @hoy wrote:

In D110847#3035672, @wenlei wrote:

Thanks for the changes to reduce probe overhead. As we chatted off patch, some information and verification of this change's impact on profile quality would be helpful.

The original patch contained several general changes, e.g, flipping the default values of those skipping APIs, which were observed to affect profile quality. After narrowing down the offending changes that are in SimplifyCFG, reverting them brought back the profile quality while still reduced the probe overhead in a decent amount.

What are the specific optimizations that were critical for profile quality in SimplifyCFG? On the other hand, it looks to me that some of the other unblocked optimizations like InstCombine could also have some impact on profile quality.
The change looks good, but I think it'd also be nice to have some insight and reasoning as to why certain optimizations have be blocked for profile quality, beyond experiment results which could vary from one workload to another.

Good question. In general SimplifyCFG is more disruptive to the flow graph by folding, merging or removing blocks. Such flow graph changes affected the execution frequency of blocks, and samples collected on the new blocks have a problem to correlate to original blocks. A particular case is

/// If this basic block is simple enough, and if a predecessor branches to us
/// and one of our successors, fold the block into the predecessor and use
/// logical operations to pick the right destination.
bool llvm::FoldBranchToCommonDest(BranchInst *BI, DomTreeUpdater *DTU,

Compared to SimplifyCFG, InstCombine is more like optimizing on instruction level. Moving an instruction around or folding instructions are likely having a smaller impact on the shape of flow graph and block frequencies.

Ok, while some are destructive, perhaps there're still recoverable cases (e.g. SimplifyCondBranchToCondBranch). But we can deal with them later.

llvm/include/llvm/IR/BasicBlock.h

public:
   Instruction* getFirstNonPHI() {
     return const_cast<Instruction *>(
                        static_cast<const BasicBlock *>(this)->getFirstNonPHI());
   }
 
   /// Returns a pointer to the first instruction in this block that is not a
   /// PHINode or a debug intrinsic, or any pseudo operation if \c SkipPseudoOp
   /// is true.
-  const Instruction *getFirstNonPHIOrDbg(bool SkipPseudoOp = false) const;
-  Instruction *getFirstNonPHIOrDbg(bool SkipPseudoOp = false) {
+  const Instruction *getFirstNonPHIOrDbg(bool SkipPseudoOp = true) const;
+  Instruction *getFirstNonPHIOrDbg(bool SkipPseudoOp = true) {
     return const_cast<Instruction *>(
         static_cast<const BasicBlock *>(this)->getFirstNonPHIOrDbg(
             SkipPseudoOp));
   }
 
   /// Returns a pointer to the first instruction in this block that is not a
   /// PHINode, a debug intrinsic, or a lifetime intrinsic, or any pseudo
   /// operation if \c SkipPseudoOp is true.
   const Instruction *
-  getFirstNonPHIOrDbgOrLifetime(bool SkipPseudoOp = false) const;
-  Instruction *getFirstNonPHIOrDbgOrLifetime(bool SkipPseudoOp = false) {
+  getFirstNonPHIOrDbgOrLifetime(bool SkipPseudoOp = true) const;
+  Instruction *getFirstNonPHIOrDbgOrLifetime(bool SkipPseudoOp = true) {
     return const_cast<Instruction *>(
         static_cast<const BasicBlock *>(this)->getFirstNonPHIOrDbgOrLifetime(
             SkipPseudoOp));
   }
 
   /// Returns an iterator to the first instruction in this block that is
   /// suitable for inserting a non-PHI instruction.
   ///
   /// In particular, it skips all PHIs and LandingPad instructions.
   const_iterator getFirstInsertionPt() const;
   iterator getFirstInsertionPt() {
     return static_cast<const BasicBlock *>(this)
                                           ->getFirstInsertionPt().getNonConst();
   }
 
   /// Return a const iterator range over the instructions in the block, skipping
   /// any debug instructions. Skip any pseudo operations as well if \c
   /// SkipPseudoOp is true.
   iterator_range<filter_iterator<BasicBlock::const_iterator,
                                  std::function<bool(const Instruction &)>>>
-  instructionsWithoutDebug(bool SkipPseudoOp = false) const;
+  instructionsWithoutDebug(bool SkipPseudoOp = true) const;
 
   /// Return an iterator range over the instructions in the block, skipping any
   /// debug instructions. Skip and any pseudo operations as well if \c
   /// SkipPseudoOp is true.
   iterator_range<
       filter_iterator<BasicBlock::iterator, std::function<bool(Instruction &)>>>
-  instructionsWithoutDebug(bool SkipPseudoOp = false);
+  instructionsWithoutDebug(bool SkipPseudoOp = true);
 
   /// Return the size of the basic block ignoring debug instructions
   filter_iterator<BasicBlock::const_iterator,
                   std::function<bool(const Instruction &)>>::difference_type
   sizeWithoutDebug() const;
 
   /// Unlink 'this' from the containing function, but do not delete it.
   void removeFromParent();

llvm/lib/Analysis/InlineCost.cpp

CallAnalyzer::analyzeBlock(BasicBlock *BB,
   for (Instruction &I : *BB) {
     // FIXME: Currently, the number of instructions in a function regardless of
     // our ability to simplify them during inline to constants or dead code,
     // are actually used by the vector bonus heuristic. As long as that's true,
     // we have to special case debug intrinsics here to prevent differences in
     // inlining due to debug symbols. Eventually, the number of unsimplified
     // instructions shouldn't factor into the cost computation, but until then,
     // hack around it here.
-    if (isa<DbgInfoIntrinsic>(I))
-      continue;
-
-    // Skip pseudo-probes.
-    if (isa<PseudoProbeInst>(I))
+    // Similarly, skip pseudo-probes.
+    if (I.isDebugOrPseudoInst())
       continue;
 
     // Skip ephemeral values.
     if (EphValues.count(&I))
       continue;
 
     ++NumInstructions;
     if (isa<ExtractElementInst>(I) || I.getType()->isVectorTy())

llvm/lib/Analysis/MemorySSA.cpp

if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(DefInst)) {
     // (including creating MemoryAccesses for them): we just end up inventing
     // clobbers where they don't really exist at all. Please see D43269 for
     // context.
     switch (II->getIntrinsicID()) {
     case Intrinsic::invariant_start:
     case Intrinsic::invariant_end:
     case Intrinsic::assume:
     case Intrinsic::experimental_noalias_scope_decl:
+    case Intrinsic::pseudoprobe:
       return {false, AliasResult(AliasResult::NoAlias)};
     case Intrinsic::dbg_addr:
     case Intrinsic::dbg_declare:
     case Intrinsic::dbg_label:
     case Intrinsic::dbg_value:
       llvm_unreachable("debuginfo shouldn't have associated defs!");
     default:
       break;
MemoryUseOrDef *MemorySSA::createNewAccess(Instruction *I,
   // FIXME: Replace this special casing with a more accurate modelling of
   // assume's control dependency.
   if (IntrinsicInst *II = dyn_cast<IntrinsicInst>(I)) {
     switch (II->getIntrinsicID()) {
     default:
       break;
     case Intrinsic::assume:
     case Intrinsic::experimental_noalias_scope_decl:
+    case Intrinsic::pseudoprobe:
       return nullptr;
     }
   }
 
   // Using a nonstandard AA pipelines might leave us with unexpected modref
   // results for I, so add a check to not model instructions that may not read
   // from or write to memory. This is necessary for correctness.
   if (!I->mayReadFromMemory() && !I->mayWriteToMemory())

llvm/lib/CodeGen/Analysis.cpp

bool llvm::isInTailCallPosition(const CallBase &Call, const TargetMachine &TM) {
 
   // If I will have a chain, make sure no other instruction that will have a
   // chain interposes between I and the return.
   // Check for all calls including speculatable functions.
   for (BasicBlock::const_iterator BBI = std::prev(ExitBB->end(), 2);; --BBI) {
     if (&*BBI == &Call)
       break;
     // Debug info intrinsics do not get in the way of tail call optimization.
-    if (isa<DbgInfoIntrinsic>(BBI))
-      continue;
     // Pseudo probe intrinsics do not block tail call optimization either.
-    if (isa<PseudoProbeInst>(BBI))
+    if (BBI->isDebugOrPseudoInst())
       continue;
     // A lifetime end, assume or noalias.decl intrinsic should not stop tail
     // call optimization.
     if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(BBI))
       if (II->getIntrinsicID() == Intrinsic::lifetime_end ||
           II->getIntrinsicID() == Intrinsic::assume ||
           II->getIntrinsicID() == Intrinsic::experimental_noalias_scope_decl)
         continue;

llvm/lib/IR/User.cpp

MutableArrayRef<uint8_t> User::getDescriptor() {
   auto *DI = reinterpret_cast<DescriptorInfo *>(getIntrusiveOperands()) - 1;
   assert(DI->SizeInBytes != 0 && "Should not have had a descriptor otherwise!");
 
   return MutableArrayRef<uint8_t>(
       reinterpret_cast<uint8_t *>(DI) - DI->SizeInBytes, DI->SizeInBytes);
 }
 
 bool User::isDroppable() const {
-  return isa<AssumeInst>(this);
+  return isa<AssumeInst>(this) || isa<PseudoProbeInst>(this);
 }
 
 //===----------------------------------------------------------------------===//
 //                         User operator new Implementations
 //===----------------------------------------------------------------------===//
 
 void *User::allocateFixedOperandUser(size_t Size, unsigned Us,
                                      unsigned DescBytes) {

llvm/lib/Transforms/IPO/GlobalDCE.cpp

 ModulePass *llvm::createGlobalDCEPass() {
   return new GlobalDCELegacyPass();
 }
 
 /// Returns true if F is effectively empty.
 static bool isEmptyFunction(Function *F) {
   BasicBlock &Entry = F->getEntryBlock();
   for (auto &I : Entry) {
-    if (isa<DbgInfoIntrinsic>(I))
+    if (I.isDebugOrPseudoInst())
       continue;
     if (auto *RI = dyn_cast<ReturnInst>(&I))
       return !RI->getReturnValue();
     break;
   }
   return false;
 }
 

llvm/lib/Transforms/IPO/GlobalOpt.cpp

 /// the code so we simply check for 'ret'.
 static bool cxxDtorIsEmpty(const Function &Fn) {
   // FIXME: We could eliminate C++ destructors if they're readonly/readnone and
   // nounwind, but that doesn't seem worth doing.
   if (Fn.isDeclaration())
     return false;
 
   for (auto &I : Fn.getEntryBlock()) {
-    if (isa<DbgInfoIntrinsic>(I))
+    if (I.isDebugOrPseudoInst())
       continue;
     if (isa<ReturnInst>(I))
       return true;
     break;
   }
   return false;
 }
 

llvm/lib/Transforms/InstCombine/InstCombineCalls.cpp

 removeTriviallyEmptyRange(IntrinsicInst &EndI, InstCombinerImpl &IC,
                           std::function<bool(const IntrinsicInst &)> IsStart) {
   // We start from the end intrinsic and scan backwards, so that InstCombine
   // has already processed (and potentially removed) all the instructions
   // before the end intrinsic.
   BasicBlock::reverse_iterator BI(EndI), BE(EndI.getParent()->rend());
   for (; BI != BE; ++BI) {
     if (auto *I = dyn_cast<IntrinsicInst>(&*BI)) {
-      if (isa<DbgInfoIntrinsic>(I) ||
+      if (I->isDebugOrPseudoInst() ||
           I->getIntrinsicID() == EndI.getIntrinsicID())
         continue;
       if (IsStart(*I)) {
         if (haveSameOperands(EndI, *I, EndI.arg_size())) {
           IC.eraseInstFromFunction(*I);
           IC.eraseInstFromFunction(EndI);
           return true;
         }

llvm/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp

bool InstCombinerImpl::mergeStoreIntoSuccessor(StoreInst &SI) {
   if (!OtherBr || BBI == OtherBB->begin())
     return false;
 
   // If the other block ends in an unconditional branch, check for the 'if then
   // else' case. There is an instruction before the branch.
   StoreInst *OtherStore = nullptr;
   if (OtherBr->isUnconditional()) {
     --BBI;
-    // Skip over debugging info.
-    while (isa<DbgInfoIntrinsic>(BBI) ||
+    // Skip over debugging info and pseudo probes.
+    while (BBI->isDebugOrPseudoInst() ||
            (isa<BitCastInst>(BBI) && BBI->getType()->isPointerTy())) {
       if (BBI==OtherBB->begin())
         return false;
       --BBI;
     }
     // If this isn't a store, isn't a store to the same location, or is not the
     // right kind of store, bail out.
     OtherStore = dyn_cast<StoreInst>(BBI);

llvm/lib/Transforms/InstCombine/InstructionCombining.cpp

Instruction *InstCombinerImpl::visitUnconditionalBranchInst(BranchInst &BI) {
   assert(BI.isUnconditional() && "Only for unconditional branches.");
 
   // If this store is the second-to-last instruction in the basic block
   // (excluding debug info and bitcasts of pointers) and if the block ends with
   // an unconditional branch, try to move the store to the successor block.
 
   auto GetLastSinkableStore = [](BasicBlock::iterator BBI) {
     auto IsNoopInstrForStoreMerging = [](BasicBlock::iterator BBI) {
-      return isa<DbgInfoIntrinsic>(BBI) ||
+      return BBI->isDebugOrPseudoInst() ||
              (isa<BitCastInst>(BBI) && BBI->getType()->isPointerTy());
     };
 
     BasicBlock::iterator FirstInstr = BBI->getParent()->begin();
     do {
       if (BBI != FirstInstr)
         --BBI;
     } while (BBI != FirstInstr && IsNoopInstrForStoreMerging(BBI));

llvm/lib/Transforms/Scalar/EarlyCSE.cpp

for (Instruction &Inst : make_early_inc_range(BB->getInstList())) {
     }
 
     // Skip sideeffect intrinsics, for the same reason as assume intrinsics.
     if (match(&Inst, m_Intrinsic<Intrinsic::sideeffect>())) {
       LLVM_DEBUG(dbgs() << "EarlyCSE skipping sideeffect: " << Inst << '\n');
       continue;
     }
 
+    // Skip pseudoprobe intrinsics, for the same reason as assume intrinsics.
+    if (match(&Inst, m_Intrinsic<Intrinsic::pseudoprobe>())) {
+      LLVM_DEBUG(dbgs() << "EarlyCSE skipping pseudoprobe: " << Inst << '\n');
+      continue;
+    }
+
     // We can skip all invariant.start intrinsics since they only read memory,
     // and we can forward values across it. For invariant starts without
     // invariant ends, we can use the fact that the invariantness never ends to
     // start a scope in the current generaton which is true for all future
     // generations.  Also, we dont need to consume the last store since the
     // semantics of invariant.start allow us to perform   DSE of the last
     // store, if there was a store following invariant.start. Consider:
     //

llvm/lib/Transforms/Scalar/IndVarSimplify.cpp

while (I != Preheader->begin()) {
     // Otherwise, sink it to the exit block.
     Instruction *ToMove = &*I;
     bool Done = false;
 
     if (I != Preheader->begin()) {
       // Skip debug info intrinsics.
       do {
         --I;
-      } while (isa<DbgInfoIntrinsic>(I) && I != Preheader->begin());
+      } while (I->isDebugOrPseudoInst() && I != Preheader->begin());
 
-      if (isa<DbgInfoIntrinsic>(I) && I == Preheader->begin())
+      if (I->isDebugOrPseudoInst() && I == Preheader->begin())
         Done = true;
     } else {
       Done = true;
     }
 
     MadeAnyChanges = true;
     ToMove->moveBefore(*ExitBlock, InsertPt);
     if (Done) break;

llvm/lib/Transforms/Utils/CloneFunction.cpp

if (DIFinder && TheModule)
       DIFinder->processInstruction(*TheModule, I);
 
     Instruction *NewInst = I.clone();
     if (I.hasName())
       NewInst->setName(I.getName() + NameSuffix);
     NewBB->getInstList().push_back(NewInst);
     VMap[&I] = NewInst; // Add instruction map to value.
 
-    hasCalls |= (isa<CallInst>(I) && !isa<DbgInfoIntrinsic>(I));
+    hasCalls |= (isa<CallInst>(I) && !I.isDebugOrPseudoInst());
     if (const AllocaInst *AI = dyn_cast<AllocaInst>(&I)) {
       if (!AI->isStaticAlloca()) {
         hasDynamicAllocas = true;
       }
     }
   }
 
   if (CodeInfo) {
if (!isa<PHINode>(NewInst)) {
         }
       }
     }
 
     if (II->hasName())
       NewInst->setName(II->getName() + NameSuffix);
     VMap[&*II] = NewInst; // Add instruction map to value.
     NewBB->getInstList().push_back(NewInst);
-    hasCalls |= (isa<CallInst>(II) && !isa<DbgInfoIntrinsic>(II));
+    hasCalls |= (isa<CallInst>(II) && !II->isDebugOrPseudoInst());
 
     if (CodeInfo) {
       CodeInfo->OrigVMap[&*II] = NewInst;
       if (auto *CB = dyn_cast<CallBase>(&*II))
         if (CB->hasOperandBundles())
           CodeInfo->OperandBundleCallSites.push_back(NewInst);
     }
 

llvm/lib/Transforms/Utils/SimplifyCFG.cpp

if (isa<AssumeInst>(V))
       return true;
     return isSafeToSpeculativelyExecute(V) &&
            all_of(V->users(),
                   [&](const User *U) { return EphValues.count(U); });
   };
 
   // Walk the loop in reverse so that we can identify ephemeral values properly
   // (values only feeding assumes).
-  for (Instruction &I : reverse(BB->instructionsWithoutDebug())) {
+  for (Instruction &I : reverse(BB->instructionsWithoutDebug(false))) {
     // Can't fold blocks that contain noduplicate or convergent calls.
     if (CallInst *CI = dyn_cast<CallInst>(&I))
       if (CI->cannotDuplicate() || CI->isConvergent())
         return false;
 
     // Ignore ephemeral values which are deleted during codegen.
     if (IsEphemeral(&I))
       EphValues.insert(&I);
if (PN->getType()->isIntegerTy(1) &&
     return Changed;
 
   // If all PHI nodes are promotable, check to make sure that all instructions
   // in the predecessor blocks can be promoted as well. If not, we won't be able
   // to get rid of the control flow, so it's not worth promoting to select
   // instructions.
   for (BasicBlock *IfBlock : IfBlocks)
     for (BasicBlock::iterator I = IfBlock->begin(); !I->isTerminator(); ++I)
-      if (!AggressiveInsts.count(&*I) && !isa<DbgInfoIntrinsic>(I) &&
-          !isa<PseudoProbeInst>(I)) {
+      if (!AggressiveInsts.count(&*I) && !I->isDebugOrPseudoInst()) {
         // This is not an aggressive instruction that we can promote.
         // Because of this, we won't be able to get rid of the control flow, so
         // the xform is not worth it.
         return Changed;
       }
 
   // If either of the blocks has it's address taken, we can't do this fold.
   if (any_of(IfBlocks,
auto IsWorthwhile = [&](BasicBlock *BB, ArrayRef<StoreInst *> FreeStores) {
     if (!BB)
       return true;
     // Heuristic: if the block can be if-converted/phi-folded and the
     // instructions inside are all cheap (arithmetic/GEPs), it's worthwhile to
     // thread this store.
     InstructionCost Cost = 0;
     InstructionCost Budget =
         PHINodeFoldingThreshold * TargetTransformInfo::TCC_Basic;
-    for (auto &I : BB->instructionsWithoutDebug()) {
+    for (auto &I : BB->instructionsWithoutDebug(false)) {
       // Consider terminator instruction to be free.
       if (I.isTerminator())
         continue;
       // If this is one the stores that we want to speculate out of this BB,
       // then don't count it's cost, consider it to be free.
       if (auto *S = dyn_cast<StoreInst>(&I))
         if (llvm::find(FreeStores, S))
           continue;
static bool SimplifyCondBranchToCondBranch(BranchInst *PBI, BranchInst *BI,
   if (MergeCondStores && mergeConditionalStores(PBI, BI, DTU, DL, TTI))
     return true;
 
   // If this is a conditional branch in an empty block, and if any
   // predecessors are a conditional branch to one of our destinations,
   // fold the conditions into logical ops and one cond br.
 
   // Ignore dbg intrinsics.
-  if (&*BB->instructionsWithoutDebug().begin() != BI)
+  if (&*BB->instructionsWithoutDebug(false).begin() != BI)
     return false;
 
   int PBIOp, BIOp;
   if (PBI->getSuccessor(0) == BI->getSuccessor(0)) {
     PBIOp = 0;
     BIOp = 0;
   } else if (PBI->getSuccessor(0) == BI->getSuccessor(1)) {
     PBIOp = 0;
GetCaseResults(SwitchInst *SI, ConstantInt *CaseVal, BasicBlock *CaseDest,
                const DataLayout &DL, const TargetTransformInfo &TTI) {
   // The block from which we enter the common destination.
   BasicBlock *Pred = SI->getParent();
 
   // If CaseDest is empty except for some side-effect free instructions through
   // which we can constant-propagate the CaseVal, continue to its successor.
   SmallDenseMap<Value *, Constant *> ConstantPool;
   ConstantPool.insert(std::make_pair(SI->getCondition(), CaseVal));
-  for (Instruction &I :CaseDest->instructionsWithoutDebug()) {
+  for (Instruction &I : CaseDest->instructionsWithoutDebug(false)) {
     if (I.isTerminator()) {
       // If the terminator is a simple branch, continue to the next block.
       if (I.getNumSuccessors() != 1 || I.isExceptionalTerminator())
         return false;
       Pred = CaseDest;
       CaseDest = I.getSuccessor(0);
     } else if (Constant *C = ConstantFold(&I, DL, ConstantPool)) {
       // Instruction is side-effect free and constant.
if (isValueEqualityComparison(SI)) {
 
     Value *Cond = SI->getCondition();
     if (SelectInst *Select = dyn_cast<SelectInst>(Cond))
       if (SimplifySwitchOnSelect(SI, Select))
         return requestResimplify();
 
     // If the block only contains the switch, see if we can fold the block
     // away into any preds.
-    if (SI == &*BB->instructionsWithoutDebug().begin())
+    if (SI == &*BB->instructionsWithoutDebug(false).begin())
       if (FoldValueComparisonIntoPredecessors(SI, Builder))
         return requestResimplify();
   }
 
   // Try to transform the switch into an icmp and a branch.
   if (TurnSwitchRangeIntoICmp(SI, Builder))
     return requestResimplify();
 

llvm/lib/Transforms/Vectorize/VectorCombine.cpp

auto FoldInst = [this, &MadeChange](Instruction &I) {
     MadeChange |= foldSingleElementStore(I);
   };
   for (BasicBlock &BB : F) {
     // Ignore unreachable basic blocks.
     if (!DT.isReachableFromEntry(&BB))
       continue;
     // Use early increment range so that we can erase instructions in loop.
     for (Instruction &I : make_early_inc_range(BB)) {
-      if (isa<DbgInfoIntrinsic>(I))
+      if (I.isDebugOrPseudoInst())
         continue;
       FoldInst(I);
     }
   }
 
   while (!Worklist.isEmpty()) {
     Instruction *I = Worklist.removeOne();
     if (!I)

llvm/test/Transforms/SampleProfile/pseudo-probe-cse.ll

+; RUN: opt < %s -S -early-cse-memssa | FileCheck %s
+
+define i16 @f1() readonly {
+  ret i16 0
+}
+
+declare void @f2()
+
+; Check that EarlyCSE correctly handles pseudo probes that don't have
+; a MemoryAccess. 
+
+define void @f3() {
+; CHECK-LABEL: @f3(
+; CHECK-NEXT:    [[CALL1:%.*]] = call i16 @f1()
+; CHECK-NEXT:    call void @llvm.pseudoprobe
+; CHECK-NEXT:    ret void
+;
+  %call1 = call i16 @f1()
+  call void @llvm.pseudoprobe(i64 6878943695821059507, i64 9, i32 0, i64 -1)
+  %call2 = call i16 @f1()
+  ret void
+}
+
+
+; Function Attrs: inaccessiblememonly nounwind willreturn
+declare void @llvm.pseudoprobe(i64, i64, i32, i64) #0
+
+attributes #0 = { inaccessiblememonly nounwind willreturn }
 No newline at end of file

llvm/test/Transforms/SampleProfile/pseudo-probe-loop-deletion.ll

+; RUN: opt %s -passes=loop-deletion -S | FileCheck %s --check-prefixes=CHECK
+
+%class.Loc.95 = type { %class.Domain.96 }
+%class.Domain.96 = type { %class.DomainBase.97 }
+%class.DomainBase.97 = type { [3 x %struct.WrapNoInit] }
+%struct.WrapNoInit = type { %class.Loc }
+%class.Loc = type { %class.Domain.67 }
+%class.Domain.67 = type { %class.DomainBase.68 }
+%class.DomainBase.68 = type { i32 }
+
+define dso_local void @foo(%class.Loc.95* %0) {
+; CHECK-LABEL: @foo(
+; CHECK-NEXT:    br label [[foo:%.*]]
+; CHECK:       foo.exit:
+; CHECK-NEXT:    ret void
+;
+  br label %2
+
+2:                                                ; preds = %4, %1
+  %.0.i.i = phi %class.Loc.95* [ undef, %1 ], [ %5, %4 ]
+  %3 = icmp ne %class.Loc.95* %.0.i.i, %0
+  br i1 %3, label %4, label %foo.exit
+
+4:                                                ; preds = %2
+  call void @llvm.pseudoprobe(i64 6878943695821059507, i64 9, i32 0, i64 -1)
+  %5 = getelementptr inbounds %class.Loc.95, %class.Loc.95* %.0.i.i, i32 1
+  br label %2
+
+foo.exit:            ; preds = %2
+  ret void
+}
+
+declare void @llvm.pseudoprobe(i64, i64, i32, i64)  #1
+
+attributes #1 = { willreturn readnone norecurse nofree }