Lower llvm.objectsize earlier in our optimization pipeline

[george.burgess.iv]

Summary

For context, please see "cleaning up ‘br i1 false’ cases in CodeGenPrepare"

I don't know if InstCombine is a great place for this, or if we'd prefer to have some kind of LowerBestEffortPostOptimizationIntrinsics pass, or ...

Suggestions for how to better test this are appreciated.

Summarizing my appearing-soon response on said email thread, I ran this on a large project that has a clang-tailored FORTIFY implementation. With this change, we fail to lower (e.g. we return failure values for) 1.9% more calls to objectsize, but we also end up lowering quite a few more objectsize intrinsics in total. The most likely explanation for this seems to be that we're able to DCE most of those "new" failures before hitting CGP. So, I think a 1.9% degradation is likely to be an overstatement.

During that test, I added an llvm_unreachable to CGP's objectsize lowering logic. It wasn't hit.

[efriedma]

I don't really like adding arbitrary flags to existing passes; better to have a separate pass, I think.

[george.burgess.iv]

Switch to having a pass to do this, as requested.

[george.burgess.iv]

I don't really like adding arbitrary flags to existing passes; better to have a separate pass, I think.

SGTM; better names for the pass are appreciated :)

[george.burgess.iv]

Ping :)

[george.burgess.iv]

Ping :)

[george.burgess.iv]

Ping :)

[dexonsmith]

This approach seems reasonable, but it would be great to have a chance to run some numbers internally to be sure we not losing something important.

You mentioned a 1.9% degradation with the current placement. Did you happen to try any other locations in the pipeline?

[george.burgess.iv]

You mentioned a 1.9% degradation with the current placement. Did you happen to try any other locations in the pipeline?

I did not, though I'd be happy to do so if anyone has recommendations :)

Looking at this, it appears that we may be running this pass too early with LTO enabled. I'll update this patch shortly to rebase it + delay running this pass until link-time if we're going to be running some flavor of LTO, and will collect new accuracy numbers for ${large_project} soonish.

Any other numbers/vetting would be highly appreciated.

[george.burgess.iv]

Rebase, and only run this in the non-prelink full-LTO pipeline

Apparently I misread the ThinLTO code, so we were fine with that. Full-LTO would lower these before linking, though, which is suboptimal.

There doesn't appear to be a full-LTO test-case (and I can't figure out how to flip PrepareForLTO without going through e.g. cfe), so making a test for this behavior on full-LTO seems a bit icky. Happy to try to add some cruft to do so if people want.

[george.burgess.iv]

And numbers for building ${large_project}, as promised:

I built each compiler/ran the numbers twice with the patch and twice without, and all of the numbers were stable across reruns.

Across ~29,000 TUs that had at least one attempt by LLVM to lower an @llvm.objectsize intrinsic:

• With this patch, we lowered 585,692 successfully, and we failed to lower 649,086, for a total of 1,234,778 "forceful" lowerings (e.g. either we return success, or we returned a "we're giving up; here's a conservative answer" value). This is a 52.6% failure rate.
• Without this patch, we lowered 589,559 successfully, and we failed to lower 640,908, for a total of 1,230,507 "forceful" lowerings. This is a 52.1% failure rate.

It's interesting to note that the "with the patch" numbers included 30 more TUs than the "without the patch" numbers. Like said, a TU only "counts" here if it had one or more objectsize calls that LLVM had to lower, so it would seem that some number of our failed lowerings with the patch are part of functions or blocks of code that just get DCE'd later on.

(For users of C/C++/ObjC specifically, I'd like to emphasize that these numbers don't include any calls to __builtin_object_size that clang could lower without the help of LLVM.)

[george.burgess.iv]

Looks like this was done in https://reviews.llvm.org/D65280; closing

include/llvm/InitializePasses.h

 void initializeLoopStrengthReducePass(PassRegistry&);
 void initializeLoopUnrollAndJamPass(PassRegistry&);
 void initializeLoopUnrollPass(PassRegistry&);
 void initializeLoopUnswitchPass(PassRegistry&);
 void initializeLoopVectorizePass(PassRegistry&);
 void initializeLoopVersioningLICMPass(PassRegistry&);
 void initializeLoopVersioningPassPass(PassRegistry&);
 void initializeLowerAtomicLegacyPassPass(PassRegistry&);
+void initializeLowerBestEffortIntrinsicsLegacyPassPass(PassRegistry&);
 void initializeLowerEmuTLSPass(PassRegistry&);
 void initializeLowerExpectIntrinsicPass(PassRegistry&);
 void initializeLowerGuardIntrinsicLegacyPassPass(PassRegistry&);
 void initializeLowerIntrinsicsPass(PassRegistry&);
 void initializeLowerInvokeLegacyPassPass(PassRegistry&);
 void initializeLowerSwitchPass(PassRegistry&);
 void initializeLowerTypeTestsPass(PassRegistry&);
 void initializeMIRCanonicalizerPass(PassRegistry &);

include/llvm/LinkAllPasses.h

ForcePassLinking() {
       (void) llvm::createLoopStrengthReducePass();
       (void) llvm::createLoopRerollPass();
       (void) llvm::createLoopUnrollPass();
       (void) llvm::createLoopUnrollAndJamPass();
       (void) llvm::createLoopUnswitchPass();
       (void) llvm::createLoopVersioningLICMPass();
       (void) llvm::createLoopIdiomPass();
       (void) llvm::createLoopRotatePass();
+      (void) llvm::createLowerBestEffortIntrinsicsPass();
       (void) llvm::createLowerExpectIntrinsicPass();
       (void) llvm::createLowerInvokePass();
       (void) llvm::createLowerSwitchPass();
       (void) llvm::createNaryReassociatePass();
       (void) llvm::createObjCARCAAWrapperPass();
       (void) llvm::createObjCARCAPElimPass();
       (void) llvm::createObjCARCExpandPass();
       (void) llvm::createObjCARCContractPass();

include/llvm/Passes/PassBuilder.h

public:
   /// transformations. It also cannot generally be run repeatedly on a module
   /// without potentially seriously regressing either runtime performance of
   /// the code or serious code size growth.
   ///
   /// Note that \p Level cannot be `O0` here. The pipelines produced are
   /// only intended for use when attempting to optimize code. If frontends
   /// require some transformations for semantic reasons, they should explicitly
   /// build them.
-  ModulePassManager buildModuleOptimizationPipeline(OptimizationLevel Level,
-                                                    bool DebugLogging = false);
+  ModulePassManager
+  buildModuleOptimizationPipeline(OptimizationLevel Level,
+                                  bool DebugLogging = false,
+                                  bool ForceLowerIntrinsics = true);
 
   /// Build a per-module default optimization pipeline.
   ///
   /// This provides a good default optimization pipeline for per-module
   /// optimization and code generation without any link-time optimization. It
   /// typically correspond to frontend "-O[123]" options for optimization
   /// levels \c O1, \c O2 and \c O3 resp.
   ///
   /// Note that \p Level cannot be `O0` here. The pipelines produced are
   /// only intended for use when attempting to optimize code. If frontends
   /// require some transformations for semantic reasons, they should explicitly
   /// build them.
-  ModulePassManager buildPerModuleDefaultPipeline(OptimizationLevel Level,
-                                                  bool DebugLogging = false);
+  ModulePassManager
+  buildPerModuleDefaultPipeline(OptimizationLevel Level,
+                                bool DebugLogging = false,
+                                bool ForceLowerIntrinsics = true);
 
   /// Build a pre-link, ThinLTO-targeting default optimization pipeline to
   /// a pass manager.
   ///
   /// This adds the pre-link optimizations tuned to prepare a module for
   /// a ThinLTO run. It works to minimize the IR which needs to be analyzed
   /// without making irreversible decisions which could be made better during
   /// the LTO run.

include/llvm/Transforms/Scalar.h

 // in the source address space if using the destination address space is slower
 // on the target.
 //
 FunctionPass *createInferAddressSpacesPass();
 extern char &InferAddressSpacesID;
 
 //===----------------------------------------------------------------------===//
 //
+// LowerBestEffortIntrinsics - Forces all best-effort intrinsics, like
+// llvm.objectsize, to be lowered immediately.
+FunctionPass *createLowerBestEffortIntrinsicsPass();
+
+//===----------------------------------------------------------------------===//
+//
 // LowerExpectIntrinsics - Removes llvm.expect intrinsics and creates
 // "block_weights" metadata.
 FunctionPass *createLowerExpectIntrinsicPass();
 
 //===----------------------------------------------------------------------===//
 //
 // PartiallyInlineLibCalls - Tries to inline the fast path of library
 // calls such as sqrt.

include/llvm/Transforms/Scalar/LowerBestEffortIntrinsics.h

+//===- LowerBestEffortIntrinsics.h - LowerBestEffortIntrinsics --*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// The header file for the LowerBestEffortIntrinsicsPass pass.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_SCALAR_LOWERBESTEFFORTINTRINSICS_H
+#define LLVM_TRANSFORMS_SCALAR_LOWERBESTEFFORTINTRINSICS_H
+
+#include "llvm/IR/Function.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+struct LowerBestEffortIntrinsicsPass
+    : PassInfoMixin<LowerBestEffortIntrinsicsPass> {
+  /// \brief Run the pass over the function.
+  ///
+  /// This will lower all of the best-effort llvm.objectsize intrinsics in the
+  // given module.
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &);
+};
+} // namespace llvm
+
+#endif

lib/CodeGen/CodeGenPrepare.cpp

for (auto &Arg : CI->arg_operands()) {
       return optimizeMemoryInst(CI, Arg, Arg->getType(), AS);
     }
 
   IntrinsicInst *II = dyn_cast<IntrinsicInst>(CI);
   if (II) {
     switch (II->getIntrinsicID()) {
     default: break;
     case Intrinsic::objectsize: {
-      // Lower all uses of llvm.objectsize.*
       ConstantInt *RetVal =
-          lowerObjectSizeCall(II, *DL, TLInfo, /*MustSucceed=*/true);
-      // Substituting this can cause recursive simplifications, which can
-      // invalidate our iterator.  Use a WeakTrackingVH to hold onto it in case
-      // this
-      // happens.
-      Value *CurValue = &*CurInstIterator;
-      WeakTrackingVH IterHandle(CurValue);
-
-      replaceAndRecursivelySimplify(CI, RetVal, TLInfo, nullptr);
 
-      // If the iterator instruction was recursively deleted, start over at the
-      // start of the block.
-      if (IterHandle != CurValue) {
-        CurInstIterator = BB->begin();
-        SunkAddrs.clear();
-      }
+      // In a regular optimization pipeline, we have a pass to forcibly lower
+      // this intrinsic, so we have a chance to simplify code that depends on
+      // its result.
+      //
+      // Hence, if we've reached this, either:
+      // - the optimization pipeline is misconfigured,
+      // - we're just running a small set of of manually-specified passes on the
+      //   IR, or
+      // - there's a bug elsewhere.
+      //
+      // In any case, there's probably not much reason to try to simplify users
+      // of `RetVal`.
+      II->replaceAllUsesWith(RetVal);
+      II->eraseFromParent();
       return true;
     }
     case Intrinsic::aarch64_stlxr:
     case Intrinsic::aarch64_stxr: {
       ZExtInst *ExtVal = dyn_cast<ZExtInst>(CI->getArgOperand(0));
       if (!ExtVal || !ExtVal->hasOneUse() ||
           ExtVal->getParent() == CI->getParent())
         return false;

lib/Passes/PassBuilder.cpp

 #include "llvm/Transforms/Scalar/LoopPredication.h"
 #include "llvm/Transforms/Scalar/LoopRotation.h"
 #include "llvm/Transforms/Scalar/LoopSimplifyCFG.h"
 #include "llvm/Transforms/Scalar/LoopSink.h"
 #include "llvm/Transforms/Scalar/LoopStrengthReduce.h"
 #include "llvm/Transforms/Scalar/LoopUnrollAndJamPass.h"
 #include "llvm/Transforms/Scalar/LoopUnrollPass.h"
 #include "llvm/Transforms/Scalar/LowerAtomic.h"
+#include "llvm/Transforms/Scalar/LowerBestEffortIntrinsics.h"
 #include "llvm/Transforms/Scalar/LowerExpectIntrinsic.h"
 #include "llvm/Transforms/Scalar/LowerGuardIntrinsic.h"
 #include "llvm/Transforms/Scalar/MemCpyOptimizer.h"
 #include "llvm/Transforms/Scalar/MergedLoadStoreMotion.h"
 #include "llvm/Transforms/Scalar/NaryReassociate.h"
 #include "llvm/Transforms/Scalar/NewGVN.h"
 #include "llvm/Transforms/Scalar/PartiallyInlineLibCalls.h"
 #include "llvm/Transforms/Scalar/Reassociate.h"
MPM.addPass(
       createModuleToPostOrderCGSCCPassAdaptor(createDevirtSCCRepeatedPass(
           std::move(MainCGPipeline), MaxDevirtIterations)));
 
   return MPM;
 }
 
 ModulePassManager
 PassBuilder::buildModuleOptimizationPipeline(OptimizationLevel Level,
-                                             bool DebugLogging) {
+                                             bool DebugLogging,
+                                             bool ForceLowerIntrinsics) {
   ModulePassManager MPM(DebugLogging);
 
   // Optimize globals now that the module is fully simplified.
   MPM.addPass(GlobalOptPass());
   MPM.addPass(GlobalDCEPass());
 
   // Run partial inlining pass to partially inline functions that have
   // large bodies.
PassBuilder::buildModuleOptimizationPipeline(OptimizationLevel Level,
   // memory operations.
   MPM.addPass(RequireAnalysisPass<GlobalsAA, Module>());
 
   FunctionPassManager OptimizePM(DebugLogging);
   OptimizePM.addPass(Float2IntPass());
   // FIXME: We need to run some loop optimizations to re-rotate loops after
   // simplify-cfg and others undo their rotation.
 
+  if (ForceLowerIntrinsics)
+    OptimizePM.addPass(LowerBestEffortIntrinsicsPass());
+
   // Optimize the loop execution. These passes operate on entire loop nests
   // rather than on each loop in an inside-out manner, and so they are actually
   // function passes.
 
   for (auto &C : VectorizerStartEPCallbacks)
     C(OptimizePM, Level);
 
   // First rotate loops that may have been un-rotated by prior passes.
PassBuilder::buildModuleOptimizationPipeline(OptimizationLevel Level,
   MPM.addPass(GlobalDCEPass());
   MPM.addPass(ConstantMergePass());
 
   return MPM;
 }
 
 ModulePassManager
 PassBuilder::buildPerModuleDefaultPipeline(OptimizationLevel Level,
-                                           bool DebugLogging) {
+                                           bool DebugLogging,
+                                           bool ForceLowerIntrinsics) {
   assert(Level != O0 && "Must request optimizations for the default pipeline!");
 
   ModulePassManager MPM(DebugLogging);
 
   // Force any function attributes we want the rest of the pipeline to observe.
   MPM.addPass(ForceFunctionAttrsPass());
 
   // Apply module pipeline start EP callback.
ModulePassManager PassBuilder::buildThinLTODefaultPipeline(
 
   return MPM;
 }
 
 ModulePassManager
 PassBuilder::buildLTOPreLinkDefaultPipeline(OptimizationLevel Level,
                                             bool DebugLogging) {
   assert(Level != O0 && "Must request optimizations for the default pipeline!");
+  // Skip force-lowering best-effort intrinsics until we have a global view of
+  // the program.
+  //
   // FIXME: We should use a customized pre-link pipeline!
-  return buildPerModuleDefaultPipeline(Level, DebugLogging);
+  return buildPerModuleDefaultPipeline(Level, DebugLogging,
+                                       /*ForceLowerIntrinsics=*/false);
 }
 
 ModulePassManager
 PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level, bool DebugLogging,
                                      ModuleSummaryIndex *ExportSummary) {
   assert(Level != O0 && "Must request optimizations for the default pipeline!");
   ModulePassManager MPM(DebugLogging);
 

lib/Passes/PassRegistry.def

 FUNCTION_PASS("gvn-hoist", GVNHoistPass())
 FUNCTION_PASS("instcombine", InstCombinePass())
 FUNCTION_PASS("instsimplify", InstSimplifyPass())
 FUNCTION_PASS("invalidate<all>", InvalidateAllAnalysesPass())
 FUNCTION_PASS("float2int", Float2IntPass())
 FUNCTION_PASS("no-op-function", NoOpFunctionPass())
 FUNCTION_PASS("libcalls-shrinkwrap", LibCallsShrinkWrapPass())
 FUNCTION_PASS("loweratomic", LowerAtomicPass())
+FUNCTION_PASS("lower-best-effort-intrinsics", LowerBestEffortIntrinsicsPass())
 FUNCTION_PASS("lower-expect", LowerExpectIntrinsicPass())
 FUNCTION_PASS("lower-guard-intrinsic", LowerGuardIntrinsicPass())
 FUNCTION_PASS("guard-widening", GuardWideningPass())
 FUNCTION_PASS("gvn", GVN())
 FUNCTION_PASS("loop-simplify", LoopSimplifyPass())
 FUNCTION_PASS("loop-sink", LoopSinkPass())
 FUNCTION_PASS("lowerinvoke", LowerInvokePass())
 FUNCTION_PASS("mem2reg", PromotePass())

lib/Transforms/IPO/PassManagerBuilder.cpp

void PassManagerBuilder::populateModulePassManager(
   // Thus both Float2Int and LoopRotate have to preserve AliasAnalysis for
   // this to work. Fortunately, it is trivial to preserve AliasAnalysis
   // (doing nothing preserves it as it is required to be conservatively
   // correct in the face of IR changes).
   MPM.add(createGlobalsAAWrapperPass());
 
   MPM.add(createFloat2IntPass());
 
+  // If we're in the pre-link stage for LTO, we want to avoid lowering these
+  // intrinsics for now: we'll have a global view later, which may give us
+  // higher-quality answers.
+  //
+  // PrepareForThinLTO gave up above, so it's not a problem.
+  if (!PrepareForLTO)
+    MPM.add(createLowerBestEffortIntrinsicsPass());
+
   addExtensionsToPM(EP_VectorizerStart, MPM);
 
   // Re-rotate loops in all our loop nests. These may have fallout out of
   // rotated form due to GVN or other transformations, and the vectorizer relies
   // on the rotated form. Disable header duplication at -Oz.
   MPM.add(createLoopRotatePass(SizeLevel == 2 ? 0 : -1));
 
   // Distribute loops to allow partial vectorization.  I.e. isolate dependences

lib/Transforms/Scalar/CMakeLists.txt

add_llvm_library(LLVMScalarOpts
   LoopRotation.cpp
   LoopSimplifyCFG.cpp
   LoopStrengthReduce.cpp
   LoopUnrollPass.cpp
   LoopUnrollAndJamPass.cpp
   LoopUnswitch.cpp
   LoopVersioningLICM.cpp
   LowerAtomic.cpp
+  LowerBestEffortIntrinsics.cpp
   LowerExpectIntrinsic.cpp
   LowerGuardIntrinsic.cpp
   MemCpyOptimizer.cpp
   MergeICmps.cpp
   MergedLoadStoreMotion.cpp
   NaryReassociate.cpp
   NewGVN.cpp
   PartiallyInlineLibCalls.cpp

lib/Transforms/Scalar/LowerBestEffortIntrinsics.cpp

+//===- LowerBestEffortIntrinsics.cpp - Lower best-effort intrinsics -------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass forcibly lowers 'best-effort' intrinsics that generally require
+// some amount of optimization to be run in order to produce high-quality
+// results. Namely, llvm.objectsize and (coming soon) llvm.is.constant.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Scalar/LowerBestEffortIntrinsics.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/ADT/iterator_range.h"
+#include "llvm/Analysis/GlobalsModRef.h"
+#include "llvm/Analysis/MemoryBuiltins.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
+#include "llvm/IR/BasicBlock.h"
+#include "llvm/IR/CallSite.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/LLVMContext.h"
+#include "llvm/IR/MDBuilder.h"
+#include "llvm/IR/Metadata.h"
+#include "llvm/Pass.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Transforms/Scalar.h"
+
+using namespace llvm;
+
+static bool runOnFunction(Function &F, const TargetLibraryInfo *TLI) {
+  const DataLayout &DL = F.getParent()->getDataLayout();
+
+  bool Changed = false;
+  for (BasicBlock &BB : F)
+    for (auto Iter = BB.begin(); Iter != BB.end();) {
+      Instruction *I = &*Iter;
+      // We may erase `I`; don't hang on to the iterator.
+      ++Iter;
+
+      auto *II = dyn_cast<IntrinsicInst>(I);
+      if (!II || II->getIntrinsicID() != llvm::Intrinsic::objectsize)
+        continue;
+
+      ConstantInt *NewVal =
+          lowerObjectSizeCall(II, DL, TLI, /*MustSucceed=*/true);
+      II->replaceAllUsesWith(NewVal);
+      II->eraseFromParent();
+      Changed = true;
+    }
+
+  return Changed;
+}
+
+PreservedAnalyses
+LowerBestEffortIntrinsicsPass::run(Function &F, FunctionAnalysisManager &FAM) {
+  const TargetLibraryInfo *TLI = &FAM.getResult<TargetLibraryAnalysis>(F);
+  if (!::runOnFunction(F, TLI))
+    return PreservedAnalyses::all();
+
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  PA.preserve<AAManager>();
+  PA.preserve<GlobalsAA>();
+  return PA;
+}
+
+namespace {
+/// Legacy pass for lowering best-effort intrinsics out of the IR.
+struct LowerBestEffortIntrinsicsLegacyPass : FunctionPass {
+  static char ID;
+  LowerBestEffortIntrinsicsLegacyPass() : FunctionPass(ID) {
+    initializeLowerBestEffortIntrinsicsLegacyPassPass(
+        *PassRegistry::getPassRegistry());
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<TargetLibraryInfoWrapperPass>();
+    AU.setPreservesCFG();
+  }
+
+  bool runOnFunction(Function &F) override {
+    const TargetLibraryInfo *TLI =
+        &getAnalysis<TargetLibraryInfoWrapperPass>().getTLI();
+    return ::runOnFunction(F, TLI);
+  }
+};
+} // namespace
+
+char LowerBestEffortIntrinsicsLegacyPass::ID = 0;
+INITIALIZE_PASS(LowerBestEffortIntrinsicsLegacyPass,
+                "lower-best-effort-intrinsics", "Lower Best-Effort Intrinsics",
+                false, false)
+
+FunctionPass *llvm::createLowerBestEffortIntrinsicsPass() {
+  return new LowerBestEffortIntrinsicsLegacyPass();
+}

lib/Transforms/Scalar/Scalar.cpp

void llvm::initializeScalarOpts(PassRegistry &Registry) {
   initializeLoopStrengthReducePass(Registry);
   initializeLoopRerollPass(Registry);
   initializeLoopUnrollPass(Registry);
   initializeLoopUnrollAndJamPass(Registry);
   initializeLoopUnswitchPass(Registry);
   initializeLoopVersioningLICMPass(Registry);
   initializeLoopIdiomRecognizeLegacyPassPass(Registry);
   initializeLowerAtomicLegacyPassPass(Registry);
+  initializeLowerBestEffortIntrinsicsLegacyPassPass(Registry);
   initializeLowerExpectIntrinsicPass(Registry);
   initializeLowerGuardIntrinsicLegacyPassPass(Registry);
   initializeMemCpyOptLegacyPassPass(Registry);
   initializeMergeICmpsPass(Registry);
   initializeMergedLoadStoreMotionLegacyPassPass(Registry);
   initializeNaryReassociateLegacyPassPass(Registry);
   initializePartiallyInlineLibCallsLegacyPassPass(Registry);
   initializeReassociateLegacyPassPass(Registry);

test/Other/new-pm-defaults.ll

 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: GlobalDCEPass
 ; CHECK-O-NEXT: Running pass: EliminateAvailableExternallyPass
 ; CHECK-O-NEXT: Running pass: ReversePostOrderFunctionAttrsPass
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Running pass: ModuleToFunctionPassAdaptor<{{.*}}PassManager{{.*}}>
 ; CHECK-O-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-O-NEXT: Running pass: Float2IntPass
+; CHECK-O-NEXT: Running pass: LowerBestEffortIntrinsicsPass
 ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-O-NEXT: Running pass: FunctionToLoopPassAdaptor<{{.*}}LoopRotatePass
 ; CHECK-O-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Finished llvm::Function pass manager run.
 ; CHECK-O-NEXT: Running pass: LoopDistributePass
 ; CHECK-O-NEXT: Running pass: LoopVectorizePass

test/Other/new-pm-thinlto-defaults.ll

 ; CHECK-POSTLINK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-POSTLINK-O-NEXT: Running pass: GlobalDCEPass
 ; CHECK-POSTLINK-O-NEXT: Running pass: EliminateAvailableExternallyPass
 ; CHECK-POSTLINK-O-NEXT: Running pass: ReversePostOrderFunctionAttrsPass
 ; CHECK-POSTLINK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-POSTLINK-O-NEXT: Running pass: ModuleToFunctionPassAdaptor<{{.*}}PassManager{{.*}}>
 ; CHECK-POSTLINK-O-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-POSTLINK-O-NEXT: Running pass: Float2IntPass
+; CHECK-POSTLINK-O-NEXT: Running pass: LowerBestEffortIntrinsicsPass
 ; CHECK-POSTLINK-O-NEXT: Running pass: FunctionToLoopPassAdaptor<{{.*}}LoopRotatePass
 ; CHECK-POSTLINK-O-NEXT: Starting llvm::Function pass manager run
 ; CHECK-POSTLINK-O-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-POSTLINK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-POSTLINK-O-NEXT: Finished llvm::Function pass manager run
 ; CHECK-POSTLINK-O-NEXT: Running pass: LoopDistributePass
 ; CHECK-POSTLINK-O-NEXT: Running pass: LoopVectorizePass
 ; CHECK-POSTLINK-O-NEXT: Running analysis: BlockFrequencyAnalysis

test/Other/opt-O2-pipeline.ll

 ; CHECK-NEXT:     Deduce function attributes in RPO
 ; CHECK-NEXT:     Global Variable Optimizer
 ; CHECK-NEXT:       Unnamed pass: implement Pass::getPassName()
 ; CHECK-NEXT:     Dead Global Elimination
 ; CHECK-NEXT:     CallGraph Construction
 ; CHECK-NEXT:     Globals Alias Analysis
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Float to int
+; CHECK-NEXT:       Lower Best-Effort Intrinsics
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Canonicalize natural loops
 ; CHECK-NEXT:       LCSSA Verifier
 ; CHECK-NEXT:       Loop-Closed SSA Form Pass
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       Scalar Evolution Analysis

test/Other/opt-O3-pipeline.ll

 ; CHECK-NEXT:     Deduce function attributes in RPO
 ; CHECK-NEXT:     Global Variable Optimizer
 ; CHECK-NEXT:       Unnamed pass: implement Pass::getPassName()
 ; CHECK-NEXT:     Dead Global Elimination
 ; CHECK-NEXT:     CallGraph Construction
 ; CHECK-NEXT:     Globals Alias Analysis
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Float to int
+; CHECK-NEXT:       Lower Best-Effort Intrinsics
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Canonicalize natural loops
 ; CHECK-NEXT:       LCSSA Verifier
 ; CHECK-NEXT:       Loop-Closed SSA Form Pass
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       Scalar Evolution Analysis

test/Other/opt-Os-pipeline.ll

 ; CHECK-NEXT:     Deduce function attributes in RPO
 ; CHECK-NEXT:     Global Variable Optimizer
 ; CHECK-NEXT:       Unnamed pass: implement Pass::getPassName()
 ; CHECK-NEXT:     Dead Global Elimination
 ; CHECK-NEXT:     CallGraph Construction
 ; CHECK-NEXT:     Globals Alias Analysis
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Float to int
+; CHECK-NEXT:       Lower Best-Effort Intrinsics
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Canonicalize natural loops
 ; CHECK-NEXT:       LCSSA Verifier
 ; CHECK-NEXT:       Loop-Closed SSA Form Pass
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       Scalar Evolution Analysis

test/Other/opt-hot-cold-split.ll

 ; CHECK-NEXT:     Deduce function attributes in RPO
 ; CHECK-NEXT:     Global Variable Optimizer
 ; CHECK-NEXT:       Unnamed pass: implement Pass::getPassName()
 ; CHECK-NEXT:     Dead Global Elimination
 ; CHECK-NEXT:     CallGraph Construction
 ; CHECK-NEXT:     Globals Alias Analysis
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Float to int
+; CHECK-NEXT:       Lower Best-Effort Intrinsics
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Canonicalize natural loops
 ; CHECK-NEXT:       LCSSA Verifier
 ; CHECK-NEXT:       Loop-Closed SSA Form Pass
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       Scalar Evolution Analysis

test/Transforms/LowerBestEffortIntrinsics/objectsize.ll

+; RUN: opt -S -lower-best-effort-intrinsics < %s | FileCheck %s
+; RUN: opt -S -passes='function(lower-best-effort-intrinsics)' < %s | FileCheck %s
+
+; CHECK-LABEL: define i32 @unresolvable
+define i32 @unresolvable(i8* %p) {
+  ; CHECK: ret i32 -1
+  %1 = call i32 @llvm.objectsize.i32.p0i8(i8* %p, i1 false, i1 true)
+  ret i32 %1
+}
+
+; CHECK-LABEL: define i32 @trivially_resolvable
+define i32 @trivially_resolvable() {
+  ; CHECK: ret i32 1
+  %p = alloca i8
+  %1 = call i32 @llvm.objectsize.i32.p0i8(i8* %p, i1 false, i1 true)
+  ret i32 %1
+}
+
+; CHECK-LABEL: define i1 @only_objectsize
+define i1 @only_objectsize(i1 %x) {
+  ; CHECK: call i1 @llvm.expect
+  %1 = call i1 @llvm.expect.i1(i1 %x, i1 0)
+  ret i1 %1
+}
+
+declare i32 @llvm.objectsize.i32.p0i8(i8*, i1, i1) nounwind readnone
+declare i1 @llvm.expect.i1(i1, i1) nounwind readnone