[GlobalISel] Add a new G_INVOKE_REGION_START instruction to fix an EH bug

[aemerson]

Summary

We currently have a bug where the legalizer, when dealing with phi operands,
may create instructions in the phi's incoming blocks at points which are effectively
dead due to a possible exception throw.

Say we have:

throwbb:
EH_LABEL
x0 = %callarg1
BL @may_throw_call
EH_LABEL
B returnbb

bb:
%v = phi i1 %true, throwbb, %false....

When legalizing we may need to widen the i1 %true value, and to do that we need
to create new extension instructions in the incoming block. Our insertion point
currently is the MBB::getFirstTerminator() which puts the IP before the unconditional
branch terminator in throwbb. These extensions may never be executed if the call
throws, and therefore we need to emit them before the call (but not too early, since
our new instruction may need values defined within throwbb as well).

throwbb:
EH_LABEL
x0 = %callarg1
BL @may_throw_call
EH_LABEL
%true = G_CONSTANT i32 1 ; <<<-- ruh'roh, this never executes if may_throw_call() throws!
B returnbb

bb:
%v = phi i32 %true, throwbb, %false....

To fix this, I've added two new instructions. The main idea is that G_INVOKE_REGION_START
is a terminator, which tries to model the fact that in the IR, the original invoke inst
is actually a terminator as well. By using that as the new insertion point, we
make sure to place new instructions on always executing paths.

Unfortunately we still need to make the legalizer use a new insertion point API
that I've added, since the existing getFirstTerminator() method does a reverse
walk up the block, and any non-terminator instructions cause it to bail out. To
avoid impacting compile time for all getFirstTerminator() uses, I've added a new
method that does a forward walk instead.

[paquette]

llvm/include/llvm/Target/GenericOpcodes.td

1278

It'd be useful to note in a comment that this is intended to be a terminator.

llvm/lib/CodeGen/MachineBasicBlock.cpp

259

find_if?

llvm/lib/CodeGen/MachineVerifier.cpp

836

MachineVerifier test?

[arsenm]

Should the invoke just stay as a terminator instead? I'd hate to have more code placement mechanisms

llvm/docs/GlobalISel/GenericOpcode.rst

850–854

Don't see the point of adding an unused instruction

llvm/test/CodeGen/AArch64/GlobalISel/invoke-region.ll

61

Can you add a case with non-constant phi inputs

[aemerson]

In D137905#3928956, @arsenm wrote:

Should the invoke just stay as a terminator instead? I'd hate to have more code placement mechanisms

What do you mean stay as a terminator? Invoke is an IR only instruction that we no longer have after translation.

[aemerson]

llvm/lib/CodeGen/MachineVerifier.cpp

836

I couldn’t think of anything to verify since these have no operands or results

[paquette]

llvm/lib/CodeGen/MachineVerifier.cpp

836

probably just a test that shows that we don't get an error when this precedes a non-terminator

[arsenm]

I don't know anything about exceptions, but looking at translateInvoke, I'm confused by several things.

Why do you need this new thing, instead of just looking for EH_LABEL? Why is this specially handling inlineasm such that it may turn off the label if the called operand can't throw as some kind of optimization? If it can't throw, wouldn't it optimize to use a regular call in the first place?

[aemerson]

In D137905#3931413, @arsenm wrote:

I don't know anything about exceptions, but looking at translateInvoke, I'm confused by several things.

Why do you need this new thing, instead of just looking for EH_LABEL? Why is this specially handling inlineasm such that it may turn off the label if the called operand can't throw as some kind of optimization? If it can't throw, wouldn't it optimize to use a regular call in the first place?

I’m not sure about the inline asm thing since I’m new to this code too, but semantically EH_LABEL isn’t a terminator, it’s a label used for computing exception table information. It seems cleaner to have a dedicated barrier instruction. The reason the legalizations insert at the first terminator is because that guarantees the instruction will execute. If you specifically look for an EH label then it’s leaking implementation details about instruction placement to the API client. Granted, having to use a forward walking ‘getFirstIteratorForward()’ isn’t ideal either.

[paquette]

Does G_INVOKE_REGION_START work with the terminators() iterator?

llvm/lib/CodeGen/MachineBasicBlock.cpp

259

Should we check for debug instructions + skip them here like in getFirstTerminator(), or is that not necessary?

[aemerson]

llvm/lib/CodeGen/MachineBasicBlock.cpp

259

I don't think it's necessary since debug instructions would never be terminators.

[arsenm]

llvm/lib/CodeGen/MachineBasicBlock.cpp

259

getFirstTerminator does check for them, so I'm not sure what the rule is for debug instructions

[aemerson]

In D137905#3931866, @paquette wrote:

Does G_INVOKE_REGION_START work with the terminators() iterator?

It would not. terminators() uses getFirstTerminator() as the start iterator for the range, and therefore it only counts terminators which are adjacent instructions at the end of the block.

I'm open to other approaches to fix this, its just that a new terminator seemed like the least hacky approach, but it's not without it's own issues.

[aemerson]

llvm/lib/CodeGen/MachineBasicBlock.cpp

259

It's because getFirstTerminator() stops searching at the first non-debug-non-terminator instruction, walking up the block. That's how it determines whether something is the "first" terminator.

Unfortunately I can't really change that because it would make every call to it more expensive.

[aemerson]

Address comments.

[aemerson]

Forgot to remove the END instruction, I’ll do that and update.

[aemerson]

Remove G_INVOKE_REGION_END

[paquette]

I think this looks good at this point

llvm/docs/GlobalISel/GenericOpcode.rst

 codegen with GlobalISel.
 
 .. code-block:: none
 
   %dst:_(p0) = G_JUMP_TABLE %jump-table.0
 
 The above example generates a pointer to the source jump table index.
 
+G_INVOKE_REGION_START
+^^^^^^^^^^^^^^^^^^^^^
+
+A marker instruction that acts as a pseudo-terminator for regions of code that may
+throw exceptions. Being a terminator, it prevents code from being inserted after
+it during passes like legalization. This is needed because calls to exception
+throw routines do not return, so no code that must be on an executable path must
+be placed after throwing.
+
+G_INVOKE_REGION_END
+^^^^^^^^^^^^^^^^^^^
+
+Marks the end of an invoke region. Instructions generated after this may be
+unreachable.
+
 
 G_INTRINSIC, G_INTRINSIC_W_SIDE_EFFECTS
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Call an intrinsic
 
 The _W_SIDE_EFFECTS version is considered to have unknown side-effects and
 as such cannot be reordered across other side-effecting instructions.

llvm/include/llvm/CodeGen/MachineBasicBlock.h

public:
   const_iterator getFirstTerminator() const {
     return const_cast<MachineBasicBlock *>(this)->getFirstTerminator();
   }
 
   /// Same getFirstTerminator but it ignores bundles and return an
   /// instr_iterator instead.
   instr_iterator getFirstInstrTerminator();
 
+  /// Finds the first terminator in a block by scanning forward. This can handle
+  /// cases in GlobalISel where there may be non-terminator instructions between
+  /// terminators, for which getFirstTerminator() will not work correctly.
+  iterator getFirstTerminatorForward();
+
   /// Returns an iterator to the first non-debug instruction in the basic block,
   /// or end(). Skip any pseudo probe operation if \c SkipPseudoOp is true.
   /// Pseudo probes are like debug instructions which do not turn into real
   /// machine code. We try to use the function to skip both debug instructions
   /// and pseudo probe operations to avoid API proliferation. This should work
   /// most of the time when considering optimizing the rest of code in the
   /// block, except for certain cases where pseudo probes are designed to block
   /// the optimizations. For example, code merge like optimizations are supposed

llvm/include/llvm/Support/TargetOpcodes.def

 HANDLE_TARGET_OPCODE(G_FENCE)
 
 /// Generic conditional branch instruction.
 HANDLE_TARGET_OPCODE(G_BRCOND)
 
 /// Generic indirect branch instruction.
 HANDLE_TARGET_OPCODE(G_BRINDIRECT)
 
+/// Begin an invoke region marker.
+HANDLE_TARGET_OPCODE(G_INVOKE_REGION_START)
+
+/// End an invoke region marker.
+HANDLE_TARGET_OPCODE(G_INVOKE_REGION_END)
+
 /// Generic intrinsic use (without side effects).
 HANDLE_TARGET_OPCODE(G_INTRINSIC)
 
 /// Generic intrinsic use (with side effects).
 HANDLE_TARGET_OPCODE(G_INTRINSIC_W_SIDE_EFFECTS)
 
 /// Generic extension allowing rubbish in high bits.
 HANDLE_TARGET_OPCODE(G_ANYEXT)

llvm/include/llvm/Target/GenericOpcodes.td

def G_BRJT : GenericInstruction {
   let InOperandList = (ins ptype0:$tbl, unknown:$jti, type1:$idx);
   let hasSideEffects = false;
   let isBranch = true;
   let isTerminator = true;
   let isBarrier = true;
   let isIndirectBranch = true;
 }
 
+// A marker to signal the following code is an invoke region, that may throw
+// an exception and therefore not return.
+def G_INVOKE_REGION_START : GenericInstruction {
+  let OutOperandList = (outs);
+  let InOperandList = (ins);
+  let isTerminator = true; // This must be a terminator.
+  let hasSideEffects = false;
+}
+
+// End an invoke region.
+def G_INVOKE_REGION_END : GenericInstruction {
+  let OutOperandList = (outs);
+  let InOperandList = (ins);
+  let hasSideEffects = true;
+}
+
 def G_READ_REGISTER : GenericInstruction {
   let OutOperandList = (outs type0:$dst);
   let InOperandList = (ins unknown:$register);
   let hasSideEffects = true;
 
   // Assume convergent. It's probably not worth the effort of somehow
   // modeling convergent and nonconvergent register accesses.
   let isConvergent = true;

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

bool IRTranslator::translateInvoke(const User &U,
   // If it can't throw then use a fast-path without emitting EH labels.
   if (LowerInlineAsm)
     NeedEHLabel = (cast<InlineAsm>(I.getCalledOperand()))->canThrow();
 
   // Emit the actual call, bracketed by EH_LABELs so that the MF knows about
   // the region covered by the try.
   MCSymbol *BeginSymbol = nullptr;
   if (NeedEHLabel) {
+    MIRBuilder.buildInstr(TargetOpcode::G_INVOKE_REGION_START);
     BeginSymbol = Context.createTempSymbol();
     MIRBuilder.buildInstr(TargetOpcode::EH_LABEL).addSym(BeginSymbol);
   }
 
   if (LowerInlineAsm) {
     if (!translateInlineAsm(I, MIRBuilder))
       return false;
   } else if (!translateCallBase(I, MIRBuilder))
     return false;
 
   MCSymbol *EndSymbol = nullptr;
   if (NeedEHLabel) {
     EndSymbol = Context.createTempSymbol();
     MIRBuilder.buildInstr(TargetOpcode::EH_LABEL).addSym(EndSymbol);
+    MIRBuilder.buildInstr(TargetOpcode::G_INVOKE_REGION_END);
   }
 
   SmallVector<std::pair<MachineBasicBlock *, BranchProbability>, 1> UnwindDests;
   BranchProbabilityInfo *BPI = FuncInfo.BPI;
   MachineBasicBlock *InvokeMBB = &MIRBuilder.getMBB();
   BranchProbability EHPadBBProb =
       BPI ? BPI->getEdgeProbability(InvokeMBB->getBasicBlock(), EHPadBB)
           : BranchProbability::getZero();

llvm/lib/CodeGen/GlobalISel/InstructionSelect.cpp

#endif
           MRI.setRegClass(SrcReg, DstRC);
         assert(canReplaceReg(DstReg, SrcReg, MRI) &&
                "Must be able to replace dst with src!");
         MI.eraseFromParent();
         MRI.replaceRegWith(DstReg, SrcReg);
         continue;
       }
 
+      if (MI.getOpcode() == TargetOpcode::G_INVOKE_REGION_START ||
+          MI.getOpcode() == TargetOpcode::G_INVOKE_REGION_END) {
+        MI.eraseFromParent();
+        continue;
+      }
+
       if (!ISel->select(MI)) {
         // FIXME: It would be nice to dump all inserted instructions.  It's
         // not obvious how, esp. considering select() can insert after MI.
         reportGISelFailure(MF, TPC, MORE, "gisel-select", "cannot select", MI);
         return false;
       }
 
       // Dump the range of instructions that MI expanded into.

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

if (SizeOp0 % NarrowSize != 0)
       return UnableToLegalize;
 
     unsigned NumParts = SizeOp0 / NarrowSize;
     SmallVector<Register, 2> DstRegs(NumParts);
     SmallVector<SmallVector<Register, 2>, 2> SrcRegs(MI.getNumOperands() / 2);
     Observer.changingInstr(MI);
     for (unsigned i = 1; i < MI.getNumOperands(); i += 2) {
       MachineBasicBlock &OpMBB = *MI.getOperand(i + 1).getMBB();
-      MIRBuilder.setInsertPt(OpMBB, OpMBB.getFirstTerminator());
+      MIRBuilder.setInsertPt(OpMBB, OpMBB.getFirstTerminatorForward());
       extractParts(MI.getOperand(i).getReg(), NarrowTy, NumParts,
                    SrcRegs[i / 2]);
     }
     MachineBasicBlock &MBB = *MI.getParent();
     MIRBuilder.setInsertPt(MBB, MI);
     for (unsigned i = 0; i < NumParts; ++i) {
       DstRegs[i] = MRI.createGenericVirtualRegister(NarrowTy);
       MachineInstrBuilder MIB =
case TargetOpcode::G_PTR_ADD:
     return Legalized;
 
   case TargetOpcode::G_PHI: {
     assert(TypeIdx == 0 && "Expecting only Idx 0");
 
     Observer.changingInstr(MI);
     for (unsigned I = 1; I < MI.getNumOperands(); I += 2) {
       MachineBasicBlock &OpMBB = *MI.getOperand(I + 1).getMBB();
-      MIRBuilder.setInsertPt(OpMBB, OpMBB.getFirstTerminator());
+      MIRBuilder.setInsertPt(OpMBB, OpMBB.getFirstTerminatorForward());
       widenScalarSrc(MI, WideTy, I, TargetOpcode::G_ANYEXT);
     }
 
     MachineBasicBlock &MBB = *MI.getParent();
     MIRBuilder.setInsertPt(MBB, --MBB.getFirstNonPHI());
     widenScalarDst(MI, WideTy);
     Observer.changedInstr(MI);
     return Legalized;
LegalizerHelper::fewerElementsVectorPhi(GenericMachineInstr &MI,
   makeDstOps(OutputOpsPieces, MRI.getType(MI.getReg(0)), NumElts);
 
   // Instructions that perform register split will be inserted in basic block
   // where register is defined (basic block is in the next operand).
   SmallVector<SmallVector<Register, 8>, 3> InputOpsPieces(NumInputs / 2);
   for (unsigned UseIdx = NumDefs, UseNo = 0; UseIdx < MI.getNumOperands();
        UseIdx += 2, ++UseNo) {
     MachineBasicBlock &OpMBB = *MI.getOperand(UseIdx + 1).getMBB();
-    MIRBuilder.setInsertPt(OpMBB, OpMBB.getFirstTerminator());
+    MIRBuilder.setInsertPt(OpMBB, OpMBB.getFirstTerminatorForward());
     extractVectorParts(MI.getReg(UseIdx), NumElts, InputOpsPieces[UseNo]);
   }
 
   // Build PHIs with fewer elements.
   unsigned NumLeftovers = OrigNumElts % NumElts ? 1 : 0;
   MIRBuilder.setInsertPt(*MI.getParent(), MI);
   for (unsigned i = 0; i < OrigNumElts / NumElts + NumLeftovers; ++i) {
     auto Phi = MIRBuilder.buildInstr(TargetOpcode::G_PHI);

llvm/lib/CodeGen/MachineBasicBlock.cpp

 //===-- llvm/CodeGen/MachineBasicBlock.cpp ----------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Collect the sequence of machine instructions for a basic block.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/CodeGen/LiveIntervals.h"
 #include "llvm/CodeGen/LivePhysRegs.h"
 #include "llvm/CodeGen/LiveVariables.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
MachineBasicBlock::instr_iterator MachineBasicBlock::getFirstInstrTerminator() {
   instr_iterator B = instr_begin(), E = instr_end(), I = E;
   while (I != B && ((--I)->isTerminator() || I->isDebugInstr()))
     ; /*noop */
   while (I != E && !I->isTerminator())
     ++I;
   return I;
 }
 
+MachineBasicBlock::iterator MachineBasicBlock::getFirstTerminatorForward() {
+  return find_if(instrs(), [](auto &II) { return II.isTerminator(); });
+}
+
 MachineBasicBlock::iterator
 MachineBasicBlock::getFirstNonDebugInstr(bool SkipPseudoOp) {
   // Skip over begin-of-block dbg_value instructions.
   return skipDebugInstructionsForward(begin(), end(), SkipPseudoOp);
 }
 
 MachineBasicBlock::iterator
 MachineBasicBlock::getLastNonDebugInstr(bool SkipPseudoOp) {

llvm/lib/CodeGen/MachineVerifier.cpp

if (Indexes && Indexes->hasIndex(*MI)) {
     lastIndex = idx;
   }
 
   // Ensure non-terminators don't follow terminators.
   if (MI->isTerminator()) {
     if (!FirstTerminator)
       FirstTerminator = MI;
   } else if (FirstTerminator) {
+    // For GlobalISel, G_INVOKE_REGION_START is a terminator that we allow to
+    // precede non-terminators.
+    if (FirstTerminator->getOpcode() != TargetOpcode::G_INVOKE_REGION_START) {
       report("Non-terminator instruction after the first terminator", MI);
       errs() << "First terminator was:\t" << *FirstTerminator;
     }
   }
+}
 
 // The operands on an INLINEASM instruction must follow a template.
 // Verify that the flag operands make sense.
 void MachineVerifier::verifyInlineAsm(const MachineInstr *MI) {
   // The first two operands on INLINEASM are the asm string and global flags.
   if (MI->getNumOperands() < 2) {
     report("Too few operands on inline asm", MI);
     return;

llvm/test/CodeGen/AArch64/GlobalISel/invoke-region.ll

+; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+; RUN: llc -O0 -mtriple=aarch64-apple-ios -global-isel -stop-after=legalizer %s -o - | FileCheck %s
+
+@global_var = external global i32
+declare i32 @__gxx_personality_v0(...)
+declare void @may_throw()
+
+; This test checks that the widened G_CONSTANT operand to the phi in "continue" bb
+; is placed before the potentially throwing call in the entry block.
+define i1 @test_lpad_phi_widen_into_pred() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
+  ; CHECK-LABEL: name: test_lpad_phi_widen_into_pred
+  ; CHECK: bb.1 (%ir-block.0):
+  ; CHECK-NEXT:   successors: %bb.3(0x40000000), %bb.2(0x40000000)
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
+  ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @global_var
+  ; CHECK-NEXT:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 11
+  ; CHECK-NEXT:   G_STORE [[C]](s32), [[GV]](p0) :: (store (s32) into @global_var)
+  ; CHECK-NEXT:   [[C2:%[0-9]+]]:_(s16) = G_CONSTANT i16 1
+  ; CHECK-NEXT:   G_INVOKE_REGION_START
+  ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+  ; CHECK-NEXT:   BL @may_throw, csr_darwin_aarch64_aapcs, implicit-def $lr, implicit $sp
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_INVOKE_REGION_END
+  ; CHECK-NEXT:   G_BR %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.lpad (landing-pad):
+  ; CHECK-NEXT:   successors: %bb.3(0x80000000)
+  ; CHECK-NEXT:   liveins: $x0, $x1
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[PHI:%[0-9]+]]:_(s32) = G_PHI [[C1]](s32), %bb.1
+  ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_STORE [[PHI]](s32), [[GV]](p0) :: (store (s32) into @global_var)
+  ; CHECK-NEXT:   [[C3:%[0-9]+]]:_(s16) = G_CONSTANT i16 0
+  ; CHECK-NEXT:   G_BR %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.continue:
+  ; CHECK-NEXT:   [[PHI1:%[0-9]+]]:_(s16) = G_PHI [[C2]](s16), %bb.1, [[C3]](s16), %bb.2
+  ; CHECK-NEXT:   [[C4:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
+  ; CHECK-NEXT:   [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[PHI1]](s16)
+  ; CHECK-NEXT:   [[AND:%[0-9]+]]:_(s32) = G_AND [[ANYEXT]], [[C4]]
+  ; CHECK-NEXT:   $w0 = COPY [[AND]](s32)
+  ; CHECK-NEXT:   RET_ReallyLR implicit $w0
+  store i32 42, i32* @global_var
+  invoke void @may_throw()
+          to label %continue unwind label %lpad
+
+lpad:                                             ; preds = %entry
+  %p = phi i32 [ 11, %0 ]
+  %1 = landingpad { i8*, i32 }
+          catch i8* null
+  store i32 %p, i32* @global_var
+  br label %continue
+
+continue:                                         ; preds = %entry, %lpad
+  %r.0 = phi i1 [ 1, %0 ], [ 0, %lpad ]
+  ret i1 %r.0
+}
+
+; Same test but with extensions.
+define i1 @test_lpad_phi_widen_into_pred_ext(i1 *%ptr) personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
+  ; CHECK-LABEL: name: test_lpad_phi_widen_into_pred_ext
+  ; CHECK: bb.1 (%ir-block.0):
+  ; CHECK-NEXT:   successors: %bb.3(0x40000000), %bb.2(0x40000000)
+  ; CHECK-NEXT:   liveins: $x0
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0
+  ; CHECK-NEXT:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
+  ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @global_var
+  ; CHECK-NEXT:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 11
+  ; CHECK-NEXT:   G_STORE [[C]](s32), [[GV]](p0) :: (store (s32) into @global_var)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(s8) = G_LOAD [[COPY]](p0) :: (load (s8) from %ir.ptr)
+  ; CHECK-NEXT:   [[ASSERT_ZEXT:%[0-9]+]]:_(s8) = G_ASSERT_ZEXT [[LOAD]], 1
+  ; CHECK-NEXT:   [[ANYEXT:%[0-9]+]]:_(s16) = G_ANYEXT [[ASSERT_ZEXT]](s8)
+  ; CHECK-NEXT:   G_INVOKE_REGION_START
+  ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
+  ; CHECK-NEXT:   BL @may_throw, csr_darwin_aarch64_aapcs, implicit-def $lr, implicit $sp
+  ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
+  ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_INVOKE_REGION_END
+  ; CHECK-NEXT:   G_BR %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.2.lpad (landing-pad):
+  ; CHECK-NEXT:   successors: %bb.3(0x80000000)
+  ; CHECK-NEXT:   liveins: $x0, $x1
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT:   [[PHI:%[0-9]+]]:_(s32) = G_PHI [[C1]](s32), %bb.1
+  ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_STORE [[PHI]](s32), [[GV]](p0) :: (store (s32) into @global_var)
+  ; CHECK-NEXT:   [[C2:%[0-9]+]]:_(s16) = G_CONSTANT i16 0
+  ; CHECK-NEXT:   G_BR %bb.3
+  ; CHECK-NEXT: {{  $}}
+  ; CHECK-NEXT: bb.3.continue:
+  ; CHECK-NEXT:   [[PHI1:%[0-9]+]]:_(s16) = G_PHI [[ANYEXT]](s16), %bb.1, [[C2]](s16), %bb.2
+  ; CHECK-NEXT:   [[C3:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
+  ; CHECK-NEXT:   [[ANYEXT1:%[0-9]+]]:_(s32) = G_ANYEXT [[PHI1]](s16)
+  ; CHECK-NEXT:   [[AND:%[0-9]+]]:_(s32) = G_AND [[ANYEXT1]], [[C3]]
+  ; CHECK-NEXT:   $w0 = COPY [[AND]](s32)
+  ; CHECK-NEXT:   RET_ReallyLR implicit $w0
+  store i32 42, i32* @global_var
+  %v = load i1, i1* %ptr
+  invoke void @may_throw()
+          to label %continue unwind label %lpad
+
+lpad:                                             ; preds = %entry
+  %p = phi i32 [ 11, %0 ]
+  %1 = landingpad { i8*, i32 }
+          catch i8* null
+  store i32 %p, i32* @global_var
+  br label %continue
+
+continue:                                         ; preds = %entry, %lpad
+  %r.0 = phi i1 [ %v, %0 ], [ 0, %lpad ]
+  ret i1 %r.0
+}

llvm/test/CodeGen/AArch64/GlobalISel/irtranslator-exceptions.ll

 define { i8*, i32 } @bar() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
   ; CHECK-LABEL: name: bar
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK-NEXT:   successors: %bb.3(0x40000000), %bb.2(0x40000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p0) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[DEF1:%[0-9]+]]:_(s32) = G_IMPLICIT_DEF
+  ; CHECK-NEXT:   G_INVOKE_REGION_START
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   $w0 = COPY [[C]](s32)
   ; CHECK-NEXT:   BL @foo, csr_darwin_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $w0, implicit-def $w0
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:_(s32) = COPY $w0
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_INVOKE_REGION_END
   ; CHECK-NEXT:   G_BR %bb.3
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.broken (landing-pad):
   ; CHECK-NEXT:   liveins: $x0, $x1
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   [[DEF2:%[0-9]+]]:_(s128) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x0

 
 define void @test_invoke_indirect(void()* %callee) personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
   ; CHECK-LABEL: name: test_invoke_indirect
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK-NEXT:   successors: %bb.3(0x40000000), %bb.2(0x40000000)
   ; CHECK-NEXT:   liveins: $x0
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:gpr64(p0) = COPY $x0
+  ; CHECK-NEXT:   G_INVOKE_REGION_START
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   BLR [[COPY]](p0), csr_darwin_aarch64_aapcs, implicit-def $lr, implicit $sp
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_INVOKE_REGION_END
   ; CHECK-NEXT:   G_BR %bb.3
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.broken (landing-pad):
   ; CHECK-NEXT:   liveins: $x0, $x1
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(s128) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x0

 define void @test_invoke_varargs() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
   ; CHECK-LABEL: name: test_invoke_varargs
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK-NEXT:   successors: %bb.3(0x40000000), %bb.2(0x40000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[C:%[0-9]+]]:_(p0) = G_CONSTANT i64 0
   ; CHECK-NEXT:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
   ; CHECK-NEXT:   [[C2:%[0-9]+]]:_(s32) = G_FCONSTANT float 1.000000e+00
+  ; CHECK-NEXT:   G_INVOKE_REGION_START
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 16, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:_(p0) = COPY $sp
   ; CHECK-NEXT:   [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 0
   ; CHECK-NEXT:   [[PTR_ADD:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C3]](s64)
   ; CHECK-NEXT:   [[ANYEXT:%[0-9]+]]:_(s64) = G_ANYEXT [[C1]](s32)
   ; CHECK-NEXT:   G_STORE [[ANYEXT]](s64), [[PTR_ADD]](p0) :: (store (s64) into stack, align 1)
   ; CHECK-NEXT:   [[C4:%[0-9]+]]:_(s64) = G_CONSTANT i64 8
   ; CHECK-NEXT:   [[PTR_ADD1:%[0-9]+]]:_(p0) = G_PTR_ADD [[COPY]], [[C4]](s64)
   ; CHECK-NEXT:   G_STORE [[C2]](s32), [[PTR_ADD1]](p0) :: (store (s32) into stack + 8, align 1)
   ; CHECK-NEXT:   $x0 = COPY [[C]](p0)
   ; CHECK-NEXT:   BL @printf, csr_darwin_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $x0
   ; CHECK-NEXT:   ADJCALLSTACKUP 16, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_INVOKE_REGION_END
   ; CHECK-NEXT:   G_BR %bb.3
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.broken (landing-pad):
   ; CHECK-NEXT:   liveins: $x0, $x1
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(s128) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:_(p0) = COPY $x0
define i32 @test_lpad_phi() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
   ; CHECK-NEXT:   successors: %bb.3(0x40000000), %bb.2(0x40000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @global_var
   ; CHECK-NEXT:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 11
   ; CHECK-NEXT:   [[C2:%[0-9]+]]:_(s32) = G_CONSTANT i32 13
   ; CHECK-NEXT:   [[C3:%[0-9]+]]:_(s32) = G_CONSTANT i32 55
   ; CHECK-NEXT:   G_STORE [[C]](s32), [[GV]](p0) :: (store (s32) into @global_var)
+  ; CHECK-NEXT:   G_INVOKE_REGION_START
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   BL @may_throw, csr_darwin_aarch64_aapcs, implicit-def $lr, implicit $sp
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_INVOKE_REGION_END
   ; CHECK-NEXT:   G_BR %bb.3
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.lpad (landing-pad):
   ; CHECK-NEXT:   successors: %bb.3(0x80000000)
   ; CHECK-NEXT:   liveins: $x0, $x1
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[PHI:%[0-9]+]]:_(s32) = G_PHI [[C1]](s32), %bb.1
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >

llvm/test/CodeGen/AArch64/GlobalISel/irtranslator-unwind-inline-asm.ll

 
 define dso_local void @test() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
   ; CHECK-LABEL: name: test
   ; CHECK: bb.1.entry:
   ; CHECK-NEXT:   successors: %bb.2(0x40000000), %bb.3(0x40000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @.str.2
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:_(p0) = COPY [[GV]](p0)
+  ; CHECK-NEXT:   G_INVOKE_REGION_START
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   INLINEASM &"bl trap", 1 /* sideeffect attdialect */
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_INVOKE_REGION_END
   ; CHECK-NEXT:   G_BR %bb.2
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.invoke.cont:
   ; CHECK-NEXT:   RET_ReallyLR
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.3.lpad (landing-pad):
   ; CHECK-NEXT:   liveins: $x0, $x1
   ; CHECK-NEXT: {{  $}}

llvm/test/CodeGen/AArch64/GlobalISel/legalize-exceptions.ll

 define void @bar() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
   ; CHECK-LABEL: name: bar
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK-NEXT:   successors: %bb.3(0x40000000), %bb.2(0x40000000)
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
   ; CHECK-NEXT:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.0.exn.slot
   ; CHECK-NEXT:   [[FRAME_INDEX1:%[0-9]+]]:_(p0) = G_FRAME_INDEX %stack.1.ehselector.slot
+  ; CHECK-NEXT:   G_INVOKE_REGION_START
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   ADJCALLSTACKDOWN 0, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   $w0 = COPY [[C]](s32)
   ; CHECK-NEXT:   BL @foo, csr_darwin_aarch64_aapcs, implicit-def $lr, implicit $sp, implicit $w0, implicit-def $w0
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def $sp, implicit $sp
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
+  ; CHECK-NEXT:   G_INVOKE_REGION_END
   ; CHECK-NEXT:   G_BR %bb.3
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT: bb.2.cleanup (landing-pad):
   ; CHECK-NEXT:   successors: %bb.4(0x80000000)
   ; CHECK-NEXT:   liveins: $x0, $x1
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   EH_LABEL <mcsymbol >
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:_(p0) = COPY $x0

llvm/test/CodeGen/AArch64/GlobalISel/legalizer-info-validation.mir

 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_BRCOND (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. the first uncovered type index: 1, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
 # DEBUG-NEXT: G_BRINDIRECT (opcode {{[0-9]+}}): 1 type index, 0 imm indices
 # DEBUG-NEXT: .. the first uncovered type index: 1, OK
 # DEBUG-NEXT: .. the first uncovered imm index: 0, OK
+# DEBUG-NEXT: G_INVOKE_REGION_START (opcode {{[0-9]+}}): 0 type indices, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT:  .. imm index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT: G_INVOKE_REGION_END (opcode {{[0-9]+}}): 0 type indices, 0 imm indices
+# DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
+# DEBUG-NEXT:  .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_INTRINSIC (opcode {{[0-9]+}}): 0 type indices, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_INTRINSIC_W_SIDE_EFFECTS (opcode {{[0-9]+}}): 0 type indices, 0 imm indices
 # DEBUG-NEXT: .. type index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: .. imm index coverage check SKIPPED: no rules defined
 # DEBUG-NEXT: G_ANYEXT (opcode {{[0-9]+}}): 2 type indices, 0 imm indices
 # DEBUG-NEXT: .. opcode {{[0-9]+}} is aliased to {{[0-9]+}}

llvm/test/MachineVerifier/test_g_invoke_region_start.mir

+#RUN: llc -march=aarch64 -o - -global-isel -run-pass=none -verify-machineinstrs %s 2>&1 | FileCheck %s
+# REQUIRES: aarch64-registered-target
+
+---
+name:            test_invoke_start
+legalized:       true
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+liveins:
+body:             |
+  bb.0:
+
+    %0:_(s32) = G_CONSTANT i32 0
+
+    ; Check we don't error when preceding a non-terminator
+    ; CHECK: G_INVOKE_REGION_START
+    G_INVOKE_REGION_START
+
+    %1:_(s32) = G_ADD %0, %0
+
+...