[GlobalISel] Create an opt-in unified GlobalISel + SelectionDAG pipeline.

[aemerson]

Summary

In order to enable LTO with GlobalISel, we need to be able to select between
GlobalISel and SDISel via some mechanism in the bitcode. Usually these kinds
of codegen options are done with function attributes and passes will inspect
these at link-time compilation to decide what to do. However, when that decision
involves changing the pass pipeline itself this model breaks down. We can't
change the pass pipeline once it's been set up on a per-function basis.

This patch adds a unified ISel pipeline that's opt-in for targets, and enabled
for AArch64 only at the moment. It basically just schedules the GlobalISel
passes before SelectionDAG selection, much like the fallback mechanism. However,
it uses a different query to decide whether or not to run the GlobalISel passes
at all (rather than the FailedISel MF property).

Each GlobalISel pass needs to be taught to *not* run unless the TargetMachine
indicates that GlobalISel was enabled, or if there's a "use-globalisel" function
attribute present.

There are a few things that break when doing this.

• MachineDominator construction seems to crash when it's given a function which

doesn't have any blocks, which happens when it tries to run right after the
IRTranslator which bails out immediately. This is worked around by just creating
an empty block temporarily.

• llc's handy -run-pass testing option breaks unless we somehow know in the pass

that's being run that we have been specifically requested. Some module metadata
is added to communicate to the passes in question that they need to run.

[paquette]

llvm/lib/CodeGen/ResetMachineFunctionPass.cpp

78–81

delete commented out code?

llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

412–413

This comment makes it sound like this doesn't work at all?

If GISel skips a function, then SelectionDAG will just not select it?

llvm/lib/CodeGen/TargetPassConfig.cpp

1349–1352

maybe more succinct

if (!MF.getTarget().getGlobalISel() && !F.hasFnAttribute("use-globalisel"))
  return true;

[arsenm]

Why is this mechanism necessary? I think it's a bit much to change the selector for LTO per module/function as it's not really part of the program semantics, and therefore shouldn't be in the IR

[aemerson]

In D79769#2032002, @arsenm wrote:

Why is this mechanism necessary? I think it's a bit much to change the selector for LTO per module/function as it's not really part of the program semantics, and therefore shouldn't be in the IR

As I replied to James on the dev list:

I think being able to select per function would actually be a nice feature for debugging and bug reducing purposes. The real reason I tried to not go for a linker flag was because it would be an exceptional case just for GlobalISel. No other codegen option seems to rely on a linker flag (maybe I’m wrong here?). Passing it via bitcode just seemed the Right Thing To Do, even if it was a pain to implement.

If the consensus is that a one off linker flag for GlobalISel is the right way to go, then that’s ok with me.

llvm/lib/CodeGen/ResetMachineFunctionPass.cpp

78–81

Yep.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

412–413

The comment is obsolete so should be disregarded.

llvm/include/llvm/CodeGen/TargetPassConfig.h

 
 namespace llvm {
 
 class LLVMTargetMachine;
 struct MachineSchedContext;
 class PassConfigImpl;
 class ScheduleDAGInstrs;
 class CSEConfigBase;
+class MachineFunction;
 
 // The old pass manager infrastructure is hidden in a legacy namespace now.
 namespace legacy {
 
 class PassManagerBase;
 
 } // end namespace legacy
 
public:
 
   /// Return true if the optimized regalloc pipeline is enabled.
   bool getOptimizeRegAlloc() const;
 
   /// Return true if the default global register allocator is in use and
   /// has not be overriden on the command line with '-regalloc=...'
   bool usingDefaultRegAlloc() const;
 
+  enum class SelectorType { SelectionDAG, FastISel, GlobalISel };
+
   /// High level function that adds all passes necessary to go from llvm IR
   /// representation to the MI representation.
   /// Adds IR based lowering and target specific optimization passes and finally
   /// the core instruction selection passes.
   /// \returns true if an error occurred, false otherwise.
   bool addISelPasses();
 
   /// Add common target configurable passes that perform LLVM IR to IR
public:
   /// instructions, thereby constraining all generic virtual registers to
   /// register classes.
   virtual bool addGlobalInstructionSelect() { return true; }
 
   /// Add the complete, standard set of LLVM CodeGen passes.
   /// Fully developed targets will not generally override this.
   virtual void addMachinePasses();
 
+  /// Add a unified pipeline of GlobalISel and SelectionDAG isel passes.
+  /// The GISel passes are scheduled first, and then SDISel. Targets must
+  /// take care to ensure their custom passes are aware that two selectors
+  /// may be running.
+  virtual bool addUnifiedISelPipeline(SelectorType Selector);
+
   /// Create an instance of ScheduleDAGInstrs to be run within the standard
   /// MachineScheduler pass for this function and target at the current
   /// optimization level.
   ///
   /// This can also be used to plug a new MachineSchedStrategy into an instance
   /// of the standard ScheduleDAGMI:
   ///   return new ScheduleDAGMI(C, std::make_unique<MyStrategy>(C), /*RemoveKillFlags=*/false)
   ///
public:
 
   /// Check whether continuous CSE should be enabled in GISel passes.
   /// By default, it's enabled for non O0 levels.
   virtual bool isGISelCSEEnabled() const;
 
   /// Returns the CSEConfig object to use for the current optimization level.
   virtual std::unique_ptr<CSEConfigBase> getCSEConfig() const;
 
+  /// Returns true if we should skip running GlobalISel passes on the given
+  /// MachineFunction \p MF based on the current codegen options in
+  /// TargetMachine and the attributes on \p MF.
+  static bool skipGISelPassesForFn(MachineFunction &MF);
+
 protected:
   // Helper to verify the analysis is really immutable.
   void setOpt(bool &Opt, bool Val);
 
   /// Methods with trivial inline returns are convenient points in the common
   /// codegen pass pipeline where targets may insert passes. Methods with
   /// out-of-line standard implementations are major CodeGen stages called by
   /// addMachinePasses. Some targets may override major stages when inserting

llvm/include/llvm/Target/TargetMachine.h

public:
   /// Parse out the target's MachineFunctionInfo from the YAML reprsentation.
   virtual bool parseMachineFunctionInfo(const yaml::MachineFunctionInfo &,
                                         PerFunctionMIParsingState &PFS,
                                         SMDiagnostic &Error,
                                         SMRange &SourceRange) const {
     return false;
   }
 
+  /// Return true if the target wants to use a unified GlobalISel + SelectionDAG
+  /// pipeline. This is necessary for GlobalISel LTO support, as we need to be
+  /// able to choose an instruction selector based on attributes or metadata in
+  /// the bitcode.
+  virtual bool useUnifiedISelPipeline() const { return false; }
+
   /// This method returns a pointer to the specified type of
   /// TargetSubtargetInfo.  In debug builds, it verifies that the object being
   /// returned is of the correct type.
   template <typename STC> const STC &getSubtarget(const Function &F) const {
     return *static_cast<const STC*>(getSubtargetImpl(F));
   }
 
   /// Create a DataLayout.
public:
 
   /// Overrides the optimization level.
   void setOptLevel(CodeGenOpt::Level Level);
 
   void setFastISel(bool Enable) { Options.EnableFastISel = Enable; }
   bool getO0WantsFastISel() { return O0WantsFastISel; }
   void setO0WantsFastISel(bool Enable) { O0WantsFastISel = Enable; }
   void setGlobalISel(bool Enable) { Options.EnableGlobalISel = Enable; }
+  bool getGlobalISel() const { return Options.EnableGlobalISel; }
+
   void setGlobalISelAbort(GlobalISelAbortMode Mode) {
     Options.GlobalISelAbort = Mode;
   }
   void setMachineOutliner(bool Enable) {
     Options.EnableMachineOutliner = Enable;
   }
   void setSupportsDefaultOutlining(bool Enable) {
     Options.SupportsDefaultOutlining = Enable;

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

return std::any_of(BB.rbegin(), BB.rend(), [](const Instruction &I) {
     const auto *CI = dyn_cast<CallInst>(&I);
     return CI && CI->isMustTailCall();
   });
 }
 
 bool IRTranslator::runOnMachineFunction(MachineFunction &CurMF) {
   MF = &CurMF;
   const Function &F = MF->getFunction();
+  if (TargetPassConfig::skipGISelPassesForFn(CurMF)) {
+    // If we're using a unified isel pipeline with SDISel, then some analyses
+    // like MachineDominator may try to run right after the translator. This
+    // doesn't end well when the translator leaves just an empty function
+    // without any basic blocks. So here we just create an entry block so that
+    // the MF is at least well-formed, even though the MF will be discarded
+    // later by the ResetMachineFunction pass.
+    MachineBasicBlock *EntryBB = MF->CreateMachineBasicBlock();
+    MF->push_back(EntryBB);
+    return true;
+  }
   if (F.empty())
     return false;
   GISelCSEAnalysisWrapper &Wrapper =
       getAnalysis<GISelCSEAnalysisWrapperPass>().getCSEWrapper();
   // Set the CSEConfig and run the analysis.
   GISelCSEInfo *CSEInfo = nullptr;
   TPC = &getAnalysis<TargetPassConfig>();
   bool EnableCSE = EnableCSEInIRTranslator.getNumOccurrences()

llvm/lib/CodeGen/GlobalISel/InstructionSelect.cpp

void InstructionSelect::getAnalysisUsage(AnalysisUsage &AU) const {
   MachineFunctionPass::getAnalysisUsage(AU);
 }
 
 bool InstructionSelect::runOnMachineFunction(MachineFunction &MF) {
   // If the ISel pipeline failed, do not bother running that pass.
   if (MF.getProperties().hasProperty(
           MachineFunctionProperties::Property::FailedISel))
     return false;
+  if (TargetPassConfig::skipGISelPassesForFn(MF))
+    return false;
 
-  LLVM_DEBUG(dbgs() << "Selecting function: " << MF.getName() << '\n');
+  LLVM_DEBUG(dbgs() << "GlobalISel selecting function: " << MF.getName() << '\n');
   GISelKnownBits &KB = getAnalysis<GISelKnownBitsAnalysis>().get(MF);
 
   const TargetPassConfig &TPC = getAnalysis<TargetPassConfig>();
   InstructionSelector *ISel = MF.getSubtarget().getInstructionSelector();
   CodeGenCoverage CoverageInfo;
   assert(ISel && "Cannot work without InstructionSelector");
   ISel->setupMF(MF, KB, CoverageInfo);
 

llvm/lib/CodeGen/GlobalISel/Legalizer.cpp

Legalizer::legalizeMachineFunction(MachineFunction &MF, const LegalizerInfo &LI,
   return {Changed, /*FailedOn*/ nullptr};
 }
 
 bool Legalizer::runOnMachineFunction(MachineFunction &MF) {
   // If the ISel pipeline failed, do not bother running that pass.
   if (MF.getProperties().hasProperty(
           MachineFunctionProperties::Property::FailedISel))
     return false;
+  if (TargetPassConfig::skipGISelPassesForFn(MF))
+    return false;
   LLVM_DEBUG(dbgs() << "Legalize Machine IR for: " << MF.getName() << '\n');
   init(MF);
   const TargetPassConfig &TPC = getAnalysis<TargetPassConfig>();
   GISelCSEAnalysisWrapper &Wrapper =
       getAnalysis<GISelCSEAnalysisWrapperPass>().getCSEWrapper();
   MachineOptimizationRemarkEmitter MORE(MF, /*MBFI=*/nullptr);
 
   const size_t NumBlocks = MF.size();

llvm/lib/CodeGen/GlobalISel/Localizer.cpp

 //===- Localizer.cpp ---------------------- Localize some instrs -*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// This file implements the Localizer class.
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/GlobalISel/Localizer.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/TargetLowering.h"
+#include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Support/Debug.h"
 
 #define DEBUG_TYPE "localizer"
 
 using namespace llvm;
 
 char Localizer::ID = 0;

 }
 
 bool Localizer::runOnMachineFunction(MachineFunction &MF) {
   // If the ISel pipeline failed, do not bother running that pass.
   if (MF.getProperties().hasProperty(
           MachineFunctionProperties::Property::FailedISel))
     return false;
 
+  if (TargetPassConfig::skipGISelPassesForFn(MF))
+    return false;
+
   // Don't run the pass if the target asked so.
   if (DoNotRunPass(MF))
     return false;
 
   LLVM_DEBUG(dbgs() << "Localize instructions for: " << MF.getName() << '\n');
 
   init(MF);
 
   // Keep track of the instructions we localized. We'll do a second pass of
   // intra-block localization to further reduce live ranges.
   LocalizedSetVecT LocalizedInstrs;
 
   bool Changed = localizeInterBlock(MF, LocalizedInstrs);
   Changed |= localizeIntraBlock(LocalizedInstrs);
   return Changed;
 }

llvm/lib/CodeGen/GlobalISel/RegBankSelect.cpp

bool RegBankSelect::assignInstr(MachineInstr &MI) {
   return applyMapping(MI, *BestMapping, RepairPts);
 }
 
 bool RegBankSelect::runOnMachineFunction(MachineFunction &MF) {
   // If the ISel pipeline failed, do not bother running that pass.
   if (MF.getProperties().hasProperty(
           MachineFunctionProperties::Property::FailedISel))
     return false;
+  if (TargetPassConfig::skipGISelPassesForFn(MF))
+    return false;
 
   LLVM_DEBUG(dbgs() << "Assign register banks for: " << MF.getName() << '\n');
   const Function &F = MF.getFunction();
   Mode SaveOptMode = OptMode;
   if (F.hasOptNone())
     OptMode = Mode::Fast;
   init(MF);
 

llvm/lib/CodeGen/GlobalISel/Utils.cpp

 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineOptimizationRemarkEmitter.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/StackProtector.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/Constants.h"
+#include "llvm/Target/TargetMachine.h"
 
 #define DEBUG_TYPE "globalisel-utils"
 
 using namespace llvm;
 
 Register llvm::constrainRegToClass(MachineRegisterInfo &MRI,
                                    const TargetInstrInfo &TII,
                                    const RegisterBankInfo &RBI, Register Reg,

llvm/lib/CodeGen/ResetMachineFunctionPass.cpp

 
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/CodeGen/StackProtector.h"
+#include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Support/Debug.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "reset-machine-function"
 
 STATISTIC(NumFunctionsReset, "Number of functions reset");
bool runOnMachineFunction(MachineFunction &MF) override {
         MF.reset();
         if (EmitFallbackDiag) {
           const Function &F = MF.getFunction();
           DiagnosticInfoISelFallback DiagFallback(F);
           F.getContext().diagnose(DiagFallback);
         }
         return true;
       }
+      if (TargetPassConfig::skipGISelPassesForFn(MF)) {
+        /*        using MFP = MachineFunctionProperties::Property;
+        MF.getProperties().reset(MFP::Legalized);
+        MF.getProperties().reset(MFP::RegBankSelected);
+        MF.getProperties().reset(MFP::Selected);*/
+        MF.reset();
+        return true;
+      }
       return false;
     }
 
   };
 } // end anonymous namespace
 
 char ResetMachineFunction::ID = 0;
 INITIALIZE_PASS(ResetMachineFunction, DEBUG_TYPE,
                 "Reset machine function if ISel failed", false, false)
 
 MachineFunctionPass *
 llvm::createResetMachineFunctionPass(bool EmitFallbackDiag = false,
                                      bool AbortOnFailedISel = false) {
   return new ResetMachineFunction(EmitFallbackDiag, AbortOnFailedISel);
 }

llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/SchedulerRegistry.h"
 #include "llvm/CodeGen/SelectionDAG.h"
 #include "llvm/CodeGen/SelectionDAGNodes.h"
 #include "llvm/CodeGen/StackProtector.h"
 #include "llvm/CodeGen/SwiftErrorValueTracking.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetLowering.h"
+#include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
for (const auto &Op : I.operands()) {
         return;
       }
     }
   }
 }
 
 bool SelectionDAGISel::runOnMachineFunction(MachineFunction &mf) {
   // If we already selected that function, we do not need to run SDISel.
+  // FIXME: GlobalISel passes are scheduled before SelectionDAGISel, and
+  // they will set the Selected property even if they are being skipped.
   if (mf.getProperties().hasProperty(
           MachineFunctionProperties::Property::Selected))
     return false;
   // Do some sanity-checking on the command-line options.
   assert((!EnableFastISelAbort || TM.Options.EnableFastISel) &&
          "-fast-isel-abort > 0 requires -fast-isel");
 
+  LLVM_DEBUG(dbgs() << "SDISel selecting function: " << mf.getName() << '\n');
+
   const Function &Fn = mf.getFunction();
   MF = &mf;
 
   // Reset the target options before resetting the optimization
   // level below.
   // FIXME: This is a horrible hack and should be processed via
   // codegen looking at the optimization level explicitly when
   // it wants to look at it.

llvm/lib/CodeGen/TargetPassConfig.cpp

if (!DisableVerify)
     addPass(createVerifierPass());
 }
 
 bool TargetPassConfig::addCoreISelPasses() {
   // Enable FastISel with -fast-isel, but allow that to be overridden.
   TM->setO0WantsFastISel(EnableFastISelOption != cl::BOU_FALSE);
 
   // Determine an instruction selector.
-  enum class SelectorType { SelectionDAG, FastISel, GlobalISel };
   SelectorType Selector;
 
   if (EnableFastISelOption == cl::BOU_TRUE)
     Selector = SelectorType::FastISel;
   else if (EnableGlobalISelOption == cl::BOU_TRUE ||
            (TM->Options.EnableGlobalISel &&
             EnableGlobalISelOption != cl::BOU_FALSE))
     Selector = SelectorType::GlobalISel;
   else if (TM->getOptLevel() == CodeGenOpt::None && TM->getO0WantsFastISel())
     Selector = SelectorType::FastISel;
   else
     Selector = SelectorType::SelectionDAG;
 
   // Set consistently TM->Options.EnableFastISel and EnableGlobalISel.
   if (Selector == SelectorType::FastISel) {
     TM->setFastISel(true);
     TM->setGlobalISel(false);
   } else if (Selector == SelectorType::GlobalISel) {
     TM->setFastISel(false);
     TM->setGlobalISel(true);
+  } else {
+    TM->setGlobalISel(false);
   }
 
   // FIXME: Injecting into the DAGISel pipeline seems to cause issues with
   //        analyses needing to be re-run. This can result in being unable to
   //        schedule passes (particularly with 'Function Alias Analysis
   //        Results'). It's not entirely clear why but AFAICT this seems to be
   //        due to one FunctionPassManager not being able to use analyses from a
   //        previous one. As we're injecting a ModulePass we break the usual
   //        pass manager into two. GlobalISel with the fallback path disabled
   //        and -run-pass seem to be unaffected. The majority of GlobalISel
   //        testing uses -run-pass so this probably isn't too bad.
   SaveAndRestore<bool> SavedDebugifyIsSafe(DebugifyIsSafe);
-  if (Selector != SelectorType::GlobalISel || !isGlobalISelAbortEnabled())
+  if (Selector != SelectorType::GlobalISel || isGlobalISelAbortEnabled())
     DebugifyIsSafe = false;
 
+  if (TM->useUnifiedISelPipeline()) {
+    if (addUnifiedISelPipeline(Selector))
+      return true;
+  } else {
     // Add instruction selector passes.
     if (Selector == SelectorType::GlobalISel) {
       SaveAndRestore<bool> SavedAddingMachinePasses(AddingMachinePasses, true);
       if (addIRTranslator())
         return true;
 
       addPreLegalizeMachineIR();
 
       if (addLegalizeMachineIR())
         return true;
 
       // Before running the register bank selector, ask the target if it
       // wants to run some passes.
       addPreRegBankSelect();
 
       if (addRegBankSelect())
         return true;
 
       addPreGlobalInstructionSelect();
 
       if (addGlobalInstructionSelect())
         return true;
 
       // Pass to reset the MachineFunction if the ISel failed.
       addPass(createResetMachineFunctionPass(
-        reportDiagnosticWhenGlobalISelFallback(), isGlobalISelAbortEnabled()));
+          reportDiagnosticWhenGlobalISelFallback(),
+          isGlobalISelAbortEnabled()));
 
       // Provide a fallback path when we do not want to abort on
       // not-yet-supported input.
       if (!isGlobalISelAbortEnabled() && addInstSelector())
         return true;
 
     } else if (addInstSelector())
       return true;
+  }
 
   // Expand pseudo-instructions emitted by ISel. Don't run the verifier before
   // FinalizeISel.
   addPass(&FinalizeISelID);
 
   // Print the instruction selected machine code...
   printAndVerify("After Instruction Selection");
 
if (addRegAssignmentOptimized()) {
 
     // Run post-ra machine LICM to hoist reloads / remats.
     //
     // FIXME: can this move into MachineLateOptimization?
     addPass(&MachineLICMID);
   }
 }
 
+bool TargetPassConfig::addUnifiedISelPipeline(SelectorType Selector) {
+  SaveAndRestore<bool> SavedAddingMachinePasses(AddingMachinePasses, true);
+  // Add instruction selector passes. We add the GlobalISel passes first
+  // and then schedule the FastISel/SelectionDAG selector afterwards. We don't
+  // have two separate pass pipelines because with LTO we need to run one
+  // selection framework or another using function attributes to decide, rather
+  // a global option switch.
+  AddingMachinePasses = true;
+  if (addIRTranslator())
+    return true;
+
+  addPreLegalizeMachineIR();
+
+  if (addLegalizeMachineIR())
+    return true;
+
+  // Before running the register bank selector, ask the target if it
+  // wants to run some passes.
+  addPreRegBankSelect();
+
+  if (addRegBankSelect())
+    return true;
+
+  addPreGlobalInstructionSelect();
+
+  if (addGlobalInstructionSelect())
+    return true;
+
+  // Pass to reset the MachineFunction if the ISel failed.
+  addPass(createResetMachineFunctionPass(
+      reportDiagnosticWhenGlobalISelFallback(), isGlobalISelAbortEnabled()));
+
+  // Now add the SDAG selector.
+  return addInstSelector();
+}
+
 //===---------------------------------------------------------------------===//
 /// Post RegAlloc Pass Configuration
 //===---------------------------------------------------------------------===//
 
 /// Add passes that optimize machine instructions after register allocation.
 void TargetPassConfig::addMachineLateOptimization() {
   // Branch folding must be run after regalloc and prolog/epilog insertion.
   addPass(&BranchFolderPassID);

 
 bool TargetPassConfig::isGISelCSEEnabled() const {
   return true;
 }
 
 std::unique_ptr<CSEConfigBase> TargetPassConfig::getCSEConfig() const {
   return std::make_unique<CSEConfigBase>();
 }
+
+bool TargetPassConfig::skipGISelPassesForFn(MachineFunction &MF) {
+  // If we have been specifically requested to run by llc's run-pass option,
+  // then don't do any skipping.
+  Function &F = MF.getFunction();
+  if (F.getParent()->getNamedMetadata("llvm.llc.run.pass"))
+    return false;
+
+  // Even if TargetMachine doesn't say we need to use GlobalISel, check the
+  // attributes before we skip the GISel pipeline.
+  if (!MF.getTarget().getGlobalISel()) {
+    if (!F.hasFnAttribute("use-globalisel"))
+      return true;
+  }
+  return false;
+}

llvm/lib/Target/AArch64/AArch64PreLegalizerCombiner.cpp

AArch64PreLegalizerCombiner::AArch64PreLegalizerCombiner(bool IsOptNone)
     : MachineFunctionPass(ID), IsOptNone(IsOptNone) {
   initializeAArch64PreLegalizerCombinerPass(*PassRegistry::getPassRegistry());
 }
 
 bool AArch64PreLegalizerCombiner::runOnMachineFunction(MachineFunction &MF) {
   if (MF.getProperties().hasProperty(
           MachineFunctionProperties::Property::FailedISel))
     return false;
+  if (TargetPassConfig::skipGISelPassesForFn(MF))
+    return false;
   auto *TPC = &getAnalysis<TargetPassConfig>();
   const Function &F = MF.getFunction();
   bool EnableOpt =
       MF.getTarget().getOptLevel() != CodeGenOpt::None && !skipFunction(F);
   GISelKnownBits *KB = &getAnalysis<GISelKnownBitsAnalysis>().get(MF);
   MachineDominatorTree *MDT =
       IsOptNone ? nullptr : &getAnalysis<MachineDominatorTree>();
   AArch64PreLegalizerCombinerInfo PCInfo(EnableOpt, F.hasOptSize(),

llvm/lib/Target/AArch64/AArch64TargetMachine.h

public:
   yaml::MachineFunctionInfo *createDefaultFuncInfoYAML() const override;
   yaml::MachineFunctionInfo *
   convertFuncInfoToYAML(const MachineFunction &MF) const override;
   bool parseMachineFunctionInfo(const yaml::MachineFunctionInfo &,
                                 PerFunctionMIParsingState &PFS,
                                 SMDiagnostic &Error,
                                 SMRange &SourceRange) const override;
 
+  bool useUnifiedISelPipeline() const override {
+    return true;
+  }
+
 private:
   bool isLittle;
 };
 
 // AArch64 little endian target machine.
 //
 class AArch64leTargetMachine : public AArch64TargetMachine {
   virtual void anchor();

llvm/test/CodeGen/AArch64/GlobalISel/gisel-commandline-option-fastisel.ll

 ; RUN:   -debug-only=isel \
 ; RUN:   | FileCheck %s --check-prefixes=DISABLED,NOFASTISEL
 
 ; RUN: llc -mtriple=aarch64-- -debug-pass=Structure %s -o /dev/null 2>&1 \
 ; RUN:   -verify-machineinstrs=0 -O1 -fast-isel=false -global-isel=false \
 ; RUN:   -debug-only=isel \
 ; RUN:   | FileCheck %s --check-prefixes=DISABLED,NOFASTISEL
 
-; Check that the right instruction selector is chosen when using
-; -global-isel=false. FastISel should be used at -O0 (unless -fast-isel=false is
-; also present) and SelectionDAG otherwise.
-
-; DISABLED-NOT: IRTranslator
-
 ; DISABLED: AArch64 Instruction Selection
 ; DISABLED: Finalize ISel and expand pseudo-instructions
 
 ; FASTISEL: Enabling fast-isel
 ; NOFASTISEL-NOT: Enabling fast-isel
 
 define void @empty() {
   ret void
 }

llvm/test/CodeGen/AArch64/GlobalISel/gisel-commandline-option.ll

 ; VERIFY-NEXT:   Verify generated machine code
 ; ENABLED-NEXT:  Localizer
 ; VERIFY-O0-NEXT:   Verify generated machine code
 ; ENABLED-NEXT: Analysis for ComputingKnownBits
 ; ENABLED-NEXT:  InstructionSelect
 ; VERIFY-NEXT:   Verify generated machine code
 ; ENABLED-NEXT:  ResetMachineFunction
 
-; FALLBACK:       AArch64 Instruction Selection
-; NOFALLBACK-NOT: AArch64 Instruction Selection
-
-; DISABLED-NOT: IRTranslator
-
 ; DISABLED: AArch64 Instruction Selection
 ; DISABLED: Finalize ISel and expand pseudo-instructions
 
 define void @empty() {
   ret void
 }

llvm/test/CodeGen/AArch64/GlobalISel/use-globalisel-fn-attr.ll

+; RUN: llc -mtriple=aarch64-- %s -o - -debug 2>&1 | FileCheck %s
+; REQUIRES: asserts
+;
+; CHECK: GlobalISel selecting function: has_gisel_attr
+; CHECK-NOT: SDISel selecting function: has_gisel_attr
+define i32 @has_gisel_attr(i32 %a, i32 %b) #0 {
+  %res = add i32 %a, %b
+  ret i32 %res
+}
+
+; On AArch64 with opt enabled, we don't enable GISel by default.
+; Shouldn't run GISel with no attribute.
+; CHECK-NOT: GlobalISel selecting function: no_gisel_attr
+; CHECK: SDISel selecting function: no_gisel_attr
+define i32 @no_gisel_attr(i32 %a, i32 %b) {
+  %res = add i32 %a, %b
+  ret i32 %res
+}
+
+attributes #0 = { "use-globalisel" } 

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:     AArch64 Promote Constant
 ; CHECK-NEXT:       FunctionPass Manager
 ; CHECK-NEXT:         Dominator Tree Construction
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Merge internal globals
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors
 ; CHECK-NEXT:       Module Verifier
+; CHECK-NEXT:       Analysis containing CSE Info
+; CHECK-NEXT:       IRTranslator
+; CHECK-NEXT:       Analysis for ComputingKnownBits
+; CHECK-NEXT:       MachineDominator Tree Construction
+; CHECK-NEXT:       AArch64PreLegalizerCombiner
+; CHECK-NEXT:       Analysis containing CSE Info
+; CHECK-NEXT:       Legalizer
+; CHECK-NEXT:       RegBankSelect
+; CHECK-NEXT:       Localizer
+; CHECK-NEXT:       Analysis for ComputingKnownBits
+; CHECK-NEXT:       InstructionSelect
+; CHECK-NEXT:       ResetMachineFunction
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Analysis
 ; CHECK-NEXT:       Lazy Branch Probability Analysis
 ; CHECK-NEXT:       Lazy Block Frequency Analysis

llvm/test/CodeGen/AArch64/max-jump-table.ll

switch i32 %a, label %return [
     i32 12, label %bb12
     i32 13, label %bb13
     i32 14, label %bb14
     i32 15, label %bb15
     i32 16, label %bb16
     i32 17, label %bb17
   ]
 ; CHECK-LABEL: function jt1:
-; CHECK-NEXT: Jump Tables:
+; CHECK: Jump Tables:
 ; CHECK0-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK0-NOT:   %jump-table.1:
 ; CHECK4-NEXT:  %jump-table.0: %bb.2 %bb.3 %bb.4 %bb.5
 ; CHECK4-NEXT:  %jump-table.1: %bb.6 %bb.7 %bb.8 %bb.9
 ; CHECK4-NEXT:  %jump-table.2: %bb.10 %bb.11 %bb.12 %bb.13
 ; CHECK4-NEXT:  %jump-table.3: %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK4-NOT:   %jump-table.4:
 ; CHECK8-NEXT:  %jump-table.0: %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9
switch i32 %x, label %return [
     i32 2,  label %bb2
     i32 3,  label %bb3
     i32 4,  label %bb4
 
     i32 14, label %bb5
     i32 15, label %bb6
   ]
 ; CHECK-LABEL: function jt2:
-; CHECK-NEXT: Jump Tables:
+; CHECK: Jump Tables:
 ; CHECK0-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.5 %bb.6{{$}}
 ; CHECK0-NOT:   %jump-table.1:
 ; CHECK4-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4{{$}}
 ; CHECK4-NOT:   %jump-table.1:
 ; CHECK8-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4{{$}}
 ; CHECK8-NOT:   %jump-table.1:
 ; CHECK16-NEXT: %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.7 %bb.5 %bb.6{{$}}
 ; CHECK16-NOT:  %jump-table.1:
switch i32 %x, label %return [
 
     i32 19, label %bb9
     i32 20, label %bb10
 
     i32 22, label %bb11
     i32 23, label %bb12
   ]
 ; CHECK-LABEL: function jt3:
-; CHECK-NEXT: Jump Tables:
+; CHECK: Jump Tables:
 ; CHECK0-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.5 %bb.6 %bb.7 %bb.8 %bb.13 %bb.9 %bb.10 %bb.13 %bb.11 %bb.12
 ; CHECK0-NOT:   %jump-table.1:
 ; CHECK4-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4
 ; CHECK4-NEXT:  %jump-table.1: %bb.5 %bb.6 %bb.7 %bb.8
 ; CHECK4-NOT:   %jump-table.2:
 ; CHECK8-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4
 ; CHECK8-NEXT:  %jump-table.1: %bb.5 %bb.6 %bb.7 %bb.8 %bb.13 %bb.9 %bb.10
 ; CHECK8-NOT:   %jump-table.2:
switch i32 %x, label %default [
 
     i32 19, label %bb9
     i32 20, label %bb10
 
     i32 22, label %bb11
     i32 23, label %bb12
   ]
 ; CHECK-LABEL: function jt4:
-; CHECK-NEXT: Jump Tables:
+; CHECK: Jump Tables:
 ; CHECK0-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.13 %bb.5 %bb.6 %bb.7 %bb.8 %bb.13 %bb.9 %bb.10 %bb.13 %bb.11 %bb.12
 ; CHECK0-NOT:   %jump-table.1:
 ; CHECK4-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4
 ; CHECK4-NEXT:  %jump-table.1: %bb.5 %bb.6 %bb.7 %bb.8
 ; CHECK4-NOT:   %jump-table.2:
 ; CHECK8-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4
 ; CHECK8-NEXT:  %jump-table.1: %bb.5 %bb.6 %bb.7 %bb.8 %bb.13 %bb.9 %bb.10
 ; CHECK8-NOT:   %jump-table.2:
switch i32 %a, label %return [
     i32 12, label %bb12
     i32 13, label %bb13
     i32 14, label %bb14
     i32 15, label %bb15
     i32 16, label %bb16
     i32 17, label %bb17
   ]
 ; CHECK-LABEL: function jt1_optsize:
-; CHECK-NEXT: Jump Tables:
+; CHECK: Jump Tables:
 ; CHECK0-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK0-NOT:   %jump-table.1:
 ; CHECK4-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK4-NOT:   %jump-table.1:
 ; CHECK8-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK8-NOT:   %jump-table.1:
 ; CHECK16-NEXT: %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK16-NOT:  %jump-table.1:
switch i32 %a, label %return [
     i32 12, label %bb12
     i32 13, label %bb13
     i32 14, label %bb14
     i32 15, label %bb15
     i32 16, label %bb16
     i32 17, label %bb17
   ]
 ; CHECK-LABEL: function jt1_pgso:
-; CHECK-NEXT: Jump Tables:
+; CHECK: Jump Tables:
 ; CHECK0-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK0-NOT:   %jump-table.1:
 ; CHECK4-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK4-NOT:   %jump-table.1:
 ; CHECK8-NEXT:  %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK8-NOT:   %jump-table.1:
 ; CHECK16-NEXT: %jump-table.0: %bb.1 %bb.2 %bb.3 %bb.4 %bb.5 %bb.6 %bb.7 %bb.8 %bb.9 %bb.10 %bb.11 %bb.12 %bb.13 %bb.14 %bb.15 %bb.16 %bb.17
 ; CHECK16-NOT:  %jump-table.1:

llvm/test/CodeGen/AArch64/min-jump-table.ll

 ; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40 -min-jump-table-entries=0 -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECK0  < %t
 ; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40 -min-jump-table-entries=2 -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECK2  < %t
 ; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40 -min-jump-table-entries=4 -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECK4  < %t
 ; RUN: llc %s -O2 -print-machineinstrs -mtriple=aarch64-linux-gnu -jump-table-density=40 -min-jump-table-entries=8 -o /dev/null 2> %t; FileCheck %s --check-prefixes=CHECK,CHECK8  < %t
 
 declare void @ext(i32, i32)
 
 define i32 @jt2(i32 %a, i32 %b) {
 entry:
   switch i32 %a, label %return [
     i32 1, label %bb1
     i32 3, label %bb2
   ]
 ; CHECK-LABEL: function jt2:
-; CHECK0-NEXT: Jump Tables:
-; CHECK2-NEXT: Jump Tables:
+; CHECK0: Jump Tables:
+; CHECK2: Jump Tables:
 ; CHECK4-NOT: {{^}}Jump Tables:
 ; CHECK8-NOT: {{^}}Jump Tables:
 
 bb1: tail call void @ext(i32 1, i32 0) br label %return
 bb2: tail call void @ext(i32 2, i32 2) br label %return
 
 return: ret i32 %b
 }
 
 define i32 @jt4(i32 %a, i32 %b) {
 entry:
   switch i32 %a, label %return [
     i32 1, label %bb1
     i32 2, label %bb2
     i32 3, label %bb3
     i32 5, label %bb4
   ]
 ; CHECK-LABEL: function jt4:
-; CHECK0-NEXT: Jump Tables:
-; CHECK2-NEXT: Jump Tables:
-; CHECK4-NEXT: Jump Tables:
+; CHECK0: Jump Tables:
+; CHECK2: Jump Tables:
+; CHECK4: Jump Tables:
 ; CHECK8-NOT: {{^}}Jump Tables:
 
 bb1: tail call void @ext(i32 1, i32 0) br label %return
 bb2: tail call void @ext(i32 3, i32 2) br label %return
 bb3: tail call void @ext(i32 4, i32 4) br label %return
 bb4: tail call void @ext(i32 5, i32 6) br label %return
 
 return: ret i32 %b
 }
 
 define i32 @jt8(i32 %a, i32 %b) {
 entry:
   switch i32 %a, label %return [
     i32 1, label %bb1
     i32 2, label %bb2
     i32 3, label %bb3
     i32 4, label %bb4
     i32 5, label %bb5
     i32 6, label %bb6
     i32 7, label %bb7
     i32 9, label %bb8
   ]
 ; CHECK-LABEL: function jt8:
-; CHECK-NEXT: Jump Tables:
+; CHECK: Jump Tables:
 
 bb1: tail call void @ext(i32 1, i32 0) br label %return
 bb2: tail call void @ext(i32 2, i32 2) br label %return
 bb3: tail call void @ext(i32 3, i32 4) br label %return
 bb4: tail call void @ext(i32 4, i32 6) br label %return
 bb5: tail call void @ext(i32 5, i32 8) br label %return
 bb6: tail call void @ext(i32 6, i32 10) br label %return
 bb7: tail call void @ext(i32 7, i32 12) br label %return
 bb8: tail call void @ext(i32 8, i32 14) br label %return
 
 return: ret i32 %b
 }

llvm/test/CodeGen/AArch64/multi-vector-store-size.ll

-; RUN: llc -mtriple=aarch64-linux-gnu -stop-after=instruction-select < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -stop-after=finalize-isel < %s | FileCheck %s
 
 declare void @llvm.aarch64.neon.st2.v4f32.p0f32(<4 x float>, <4 x float>, float*)
 declare void @llvm.aarch64.neon.st3.v4f32.p0f32(<4 x float>, <4 x float>, <4 x float>, float*)
 declare void @llvm.aarch64.neon.st4.v4f32.p0f32(<4 x float>, <4 x float>, <4 x float>, <4 x float>, float*)
 
 declare void @llvm.aarch64.neon.st1x2.v4f32.p0f32(<4 x float>, <4 x float>, float*)
 declare void @llvm.aarch64.neon.st1x3.v4f32.p0f32(<4 x float>, <4 x float>, <4 x float>, float*)
 declare void @llvm.aarch64.neon.st1x4.v4f32.p0f32(<4 x float>, <4 x float>, <4 x float>, <4 x float>, float*)

llvm/tools/llc/llc.cpp

if (MIR) {
       assert(MMIWP && "Forgot to create MMIWP?");
       if (MIR->parseMachineFunctions(*M, MMIWP->getMMI()))
         return 1;
     }
 
     // Before executing passes, print the final values of the LLVM options.
     cl::PrintOptionValues();
 
+    // Some passes like GlobalISel ones need to know when -run-pass is being
+    // used, otherwise they may skip unless some function attributes are set.
+    // Communicate via some module metadata here that we have specifically
+    // requested a pass to run.
+    if (!RunPassNames->empty())
+      M->getOrInsertNamedMetadata(StringRef("llvm.llc.run.pass"));
+
     // If requested, run the pass manager over the same module again,
     // to catch any bugs due to persistent state in the passes. Note that
     // opt has the same functionality, so it may be worth abstracting this out
     // in the future.
     SmallVector<char, 0> CompileTwiceBuffer;
     if (CompileTwice) {
       std::unique_ptr<Module> M2(llvm::CloneModule(*M));
       PM.run(*M2);