Outline CFI Instructions in Tail Calls

[AndrewLitteken]

Summary

This patch outlines CFI Instructions if the outlined function is part of a tail call

[paquette]

llvm/lib/CodeGen/MachineOutliner.cpp

1222

Why is this out here?

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5867–5869

Typically in LLVM, we prefer omitting braces on ifs with a single line of code.

I think running clang-format should handle this automatically.

Also variable names are written with a capital letter in LLVM. (see: https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly)

6195–6200

Comment is outdated now

llvm/test/CodeGen/AArch64/machine-outliner-cfi.mir

6

This testcase is the same as the new one you're adding, right?

I don't think we need both testcases if so. I think it would be better to just keep this one and not add a new, identical test.

[AndrewLitteken]

llvm/test/CodeGen/AArch64/machine-outliner-cfi.mir

6

This is the old test so that it actually shows this. The previous test could outline the CFI instruction, this one should not. The comments are the same because I was copy pasting the file to get the testing framework correct.

[paquette]

llvm/test/CodeGen/AArch64/machine-outliner-cfi-tail.mir

5–7

Is this comment accurate? This test is to show that we do outline CFI instructions in tail calls, right?

llvm/test/CodeGen/AArch64/machine-outliner-cfi.mir

6

Ah now I see!

[paquette]

Francis, you had some concerns about unwinding here. Can you take a look as well?

llvm/lib/CodeGen/MachineOutliner.cpp

1173–1174

It looks like you copy over every CFI from the (first) original function into the outlined function here.

However, I don't think that there are any guarantees that *every* CFI instruction from the original function is actually present in the outlined sequence right now.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5865

I don't think you have to check any_of. They should all contain the same instructions.

6069–6070

We probably also want to check that the candidate has all of the original function's CFI instructions

[AndrewLitteken]

Taking more care to include all CFI instructions and general formatting, also making sure all the tests pass

[paquette]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

50

It looks like std::set isn't used in the patch anywhere, so you probably don't need this.

51

You shouldn't have to include vector here. It is already included by something else.

5866

Can this be a SmallVector?

5867–5870

I think comment is kind of confusing.

• Substring doesn't really imply equality. It would make more sense to say "every candidate has equivalent instructions".

• "size matches" doesn't way what is being referred to with respect to size. You could say "the number of CFI instructions found is equal to the number in the containing function's frame" or something similar.

• I think we should add some reasoning for why we don't want to allow candidates which do not include all of the CFIs.

5867–5872

I think we also have to check that the other Candidate's functions have the same number of CFI instructions as well.

e.g. we shouldn't have

foo:
  cfi 0
  cfi 1
  cfi 2
  ...
  ret

bar:
  cfi 1
  cfi 2
  ...
  ret

be outlined to

foo:
  cfi 0
  b outlined

bar:
  b outlined

I think that this could happen if for whatever reason bar ends up containing the first Candidate.

5868–5869

I think it would be good to express the intent of this loop here, rather than in the comment below.

e.g.

// To safely outline CFI instructions, we must outline every CFI instruction in the function.
// If CFI instructions are present in the candidate, store them in a vector. This can be used to check if we have collected all of the CFI instructions.

6196–6198

We don't fix up the offsets, so maybe we should just remove the first sentence in this comment.

[paquette]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5867

s/colllect/collect/

5871

Missing period at the end of sentence. (https://llvm.org/docs/CodingStandards.html#commenting)

5873

Use outliner::Candidate &C?

5876

Pull vector out of the loop and use clear()? (See: http://llvm.org/docs/ProgrammersManual.html#vector)

5878–5884

I don't think you have to do this for every candidate.

The part you have to do for every candidate is checking the number of CFI instructions in each function frame. This should be sufficient, because we know every instruction in every candidate is the same.

E.g. after collecting the CFI instructions in the first candidate, you can do something like this:

if (HasCFI) {
  for (outliner::Candidate &C : RepeatedSequenceLocs) {
    if (C.getMF()->getFrameInstructions().size() != CFIInstructionsPresent.size())
      return outliner::OutlinedFunction();
  }
}

or even

if (HasCFI && any_of(RepeatedSequenceLocs, ...))
  return outliner::OutlinedFunction()

Actually, is it even necessary to collect the CFI instructions? Since all we need to know is that the candidates actually contain all of them, it would be sufficient to just count how many are in the candidate versus how many are in the frame of each function, no?

[AndrewLitteken]

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5878–5884

I see now, I thought you reversed what you said in the first review about not having to check everything. I also agree, just finding the number should be sufficient, we shouldn't even need use a vector.

[AndrewLitteken]

Added a new test to make sure that we do not outline CFI Instruction if the outline would not capture all the CFI instruction in a function of one of the candidates

[paquette]

llvm/lib/CodeGen/MachineOutliner.cpp

1170–1172

Can we move this out of the loop?

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5866

s/Instruction/Instructions/
s/cadidates/candidates/

5869–5870

There are some extra spaces here which can be clang-formatted away

5881

extra space after "to"

[paquette]

LGTM!

[thegameg]

Thanks Andrew! A few coments inline and below:

• does this need an X86 test as well? I'm not sure what the state of the outliner is there.
• please grab std::vector<MCCFIInstruction> by const &.
• a MachineBasicBlock::iterator is usually called MBBI in codegen.

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

5866

s/Instructionz/Instructions/

llvm/lib/Target/X86/X86InstrInfo.cpp

8686

s/Instructionz/Instructions/

llvm/test/CodeGen/AArch64/machine-outliner-cfi-tail-some.mir

24

CHECK-NEXT might be even better here.

[AndrewLitteken]

• does this need an X86 test as well? I'm not sure what the state of the outliner is there.

I'm not sure, I'll try writing some analogous ones

[AndrewLitteken]

[vsk]

Hi all,

I'm seeing what looks like an unexpected use of $noreg when I run some of these tests with synthetic debug uses inserted into the MIR stream (with -debugify-and-strip-all-safe). This is what I see:

~/src/builds/llvm-project-master-RA (1) $ ./bin/llc -debugify-and-strip-all-safe -mtriple=aarch64 -run-pass=machine-outliner -verify-machineinstrs /Users/vsk/src/llvm-project-master/llvm/test/CodeGen/AArch64/machine-outliner-side-effect.mir -o - > bad2.mir
~/src/builds/llvm-project-master-RA (0) $ ./bin/llc -debugify-and-strip-all-safe=0 -mtriple=aarch64 -run-pass=machine-outliner -verify-machineinstrs /Users/vsk/src/llvm-project-master/llvm/test/CodeGen/AArch64/machine-outliner-side-effect.mir -o - > good2.mir
~/src/builds/llvm-project-master-RA (0) $ diff -u {good,bad}2.mir
--- good2.mir   2020-04-24 16:53:17.000000000 -0700
+++ bad2.mir    2020-04-24 16:53:08.000000000 -0700
@@ -62,11 +62,11 @@
   bb.0:
     liveins: $x0, $x20

-    BL @OUTLINED_FUNCTION_0, implicit-def $lr, implicit $sp, implicit-def $lr, implicit-def $sp, implicit-def $x0, implicit $x20, implicit $sp
+    BL @OUTLINED_FUNCTION_0, implicit-def $lr, implicit $sp, implicit-def $lr, implicit-def $sp, implicit-def $x0, implicit $noreg, implicit $sp, implicit $x20
     renamable $x21 = COPY $x0
-    BL @OUTLINED_FUNCTION_0, implicit-def $lr, implicit $sp, implicit-def $lr, implicit-def $sp, implicit-def $x0, implicit $x20, implicit $sp
+    BL @OUTLINED_FUNCTION_0, implicit-def $lr, implicit $sp, implicit-def $lr, implicit-def $sp, implicit-def $x0, implicit $noreg, implicit $sp, implicit $x20
     renamable $x22 = COPY $x0
-    BL @OUTLINED_FUNCTION_0, implicit-def $lr, implicit $sp, implicit-def $lr, implicit-def $sp, implicit-def $x0, implicit $x20, implicit $sp
+    BL @OUTLINED_FUNCTION_0, implicit-def $lr, implicit $sp, implicit-def $lr, implicit-def $sp, implicit-def $x0, implicit $noreg, implicit $sp, implicit $x20
     renamable $x3 = COPY $x0
     RET_ReallyLR

Not sure if that's something to worry about (or even whether this patch causes the difference), just thought I'd report the difference. Generally we don't expect tests to fail when -debugify-and-strip-all-safe is added, but there are exceptions of course.

[AndrewLitteken]

I think the issue is actually introduced by this patch: https://reviews.llvm.org/D78173

[AndrewLitteken]

After going through and reverting recent outliner commits, I don't think this change is related to any of them.

Commits:
1a78b0bd3829381e7be627b459c22083bf4671d4
1488bef8fc916fb5b563122bf64b949bb5c16464
8d5024f7fe721513c4332048e6836cd408581a61
66037b84cf5e2d432801fd0a3a0f2921c0860af3

llvm/lib/CodeGen/MachineOutliner.cpp

MachineFunction *MachineOutliner::createOutlinedFunction(
   MachineFunction &MF = MMI.getOrCreateMachineFunction(*F);
   MachineBasicBlock &MBB = *MF.CreateMachineBasicBlock();
   const TargetSubtargetInfo &STI = MF.getSubtarget();
   const TargetInstrInfo &TII = *STI.getInstrInfo();
 
   // Insert the new function into the module.
   MF.insert(MF.begin(), &MBB);
 
+  MachineFunction *OriginalMF = FirstCand.front()->getMF();
+  const std::vector<MCCFIInstruction> &Instrs =
+      OriginalMF->getFrameInstructions();
   for (auto I = FirstCand.front(), E = std::next(FirstCand.back()); I != E;
        ++I) {
     MachineInstr *NewMI = MF.CloneMachineInstr(&*I);
+    if (I->isCFIInstruction()) {
+      unsigned CFIIndex = NewMI->getOperand(0).getCFIIndex();
+      MCCFIInstruction CFI = Instrs[CFIIndex];
+      (void)MF.addFrameInst(CFI);
+    }
     NewMI->dropMemRefs(MF);
 
     // Don't keep debug information for outlined instructions.
     NewMI->setDebugLoc(DebugLoc());
     MBB.insert(MBB.end(), NewMI);
   }
 
   TII.buildOutlinedFrame(MBB, MF, OF);
 
if (DISubprogram *SP = getSubprogramOrNull(OF)) {
     F->setSubprogram(OutlinedSP);
     // We're done with the DIBuilder.
     DB.finalize();
   }
 
   return &MF;
 }
 
 bool MachineOutliner::outline(Module &M,
                               std::vector<OutlinedFunction> &FunctionList,
                               InstructionMapper &Mapper,
                               unsigned &OutlinedFunctionNum) {
 
   bool OutlinedSomething = false;
 
   // Sort by benefit. The most beneficial functions should be outlined first.
   llvm::stable_sort(FunctionList, [](const OutlinedFunction &LHS,
for (Candidate &C : OF.Candidates) {
       OutlinedSomething = true;
 
       // Statistics.
       NumOutlined++;
     }
   }
 
   LLVM_DEBUG(dbgs() << "OutlinedSomething = " << OutlinedSomething << "\n";);
-
   return OutlinedSomething;
 }
 
 void MachineOutliner::populateMapper(InstructionMapper &Mapper, Module &M,
                                      MachineModuleInfo &MMI) {
   // Build instruction mappings for each function in the module. Start by
   // iterating over each Function in M.
   for (Function &F : M) {

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
+#include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
-#include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/StackMaps.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/MC/MCInstrDesc.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
 #include <cassert>
 #include <cstdint>
 #include <iterator>
 #include <utility>
 
 using namespace llvm;
 
 #define GET_INSTRINFO_CTOR_DTOR
 #include "AArch64GenInstrInfo.inc"
 
 static cl::opt<unsigned> TBZDisplacementBits(
     "aarch64-tbz-offset-bits", cl::Hidden, cl::init(14),
     cl::desc("Restrict range of TB[N]Z instructions (DEBUG)"));
outliner::OutlinedFunction AArch64InstrInfo::getOutliningCandidateInfo(
 
   unsigned FrameID = MachineOutlinerDefault;
   NumBytesToCreateFrame += 4;
 
   bool HasBTI = any_of(RepeatedSequenceLocs, [](outliner::Candidate &C) {
     return C.getMF()->getFunction().hasFnAttribute("branch-target-enforcement");
   });
 
+  // We check to see if CFI Instructions are present, and if they are
+  // we find the number of CFI Instructions in the candidates.
+  unsigned CFICount = 0;
+  MachineBasicBlock::iterator MBBI = RepeatedSequenceLocs[0].front();
+  for (unsigned Loc = RepeatedSequenceLocs[0].getStartIdx();
+       Loc < RepeatedSequenceLocs[0].getEndIdx() + 1; Loc++) {
+    const std::vector<MCCFIInstruction> &CFIInstructions =
+        RepeatedSequenceLocs[0].getMF()->getFrameInstructions();
+    if (MBBI->isCFIInstruction()) {
+      unsigned CFIIndex = MBBI->getOperand(0).getCFIIndex();
+      MCCFIInstruction CFI = CFIInstructions[CFIIndex];
+      CFICount++;
+    }
+    MBBI++;
+  }
+
+  // We compare the number of found CFI Instructions to  the number of CFI
+  // instructions in the parent function for each candidate.  We must check this
+  // since if we outline one of the CFI instructions in a function, we have to
+  // outline them all for correctness. If we do not, the address offsets will be
+  // incorrect between the two sections of the program.
+  for (outliner::Candidate &C : RepeatedSequenceLocs) {
+    std::vector<MCCFIInstruction> CFIInstructions =
+        C.getMF()->getFrameInstructions();
+
+    if (CFICount > 0 && CFICount != CFIInstructions.size())
+      return outliner::OutlinedFunction();
+  }
+
   // Returns true if an instructions is safe to fix up, false otherwise.
   auto IsSafeToFixup = [this, &TRI](MachineInstr &MI) {
     if (MI.isCall())
       return true;
 
     if (!MI.modifiesRegister(AArch64::SP, &TRI) &&
         !MI.readsRegister(AArch64::SP, &TRI))
       return true;
if (ModStackToSaveLR) {
         return outliner::OutlinedFunction();
       }
 
       // Save + restore LR.
       NumBytesToCreateFrame += 8;
     }
   }
 
+  // If we have CFI instructions, we can only outline if the outlined section
+  // can be a tail call
+  if (FrameID != MachineOutlinerTailCall && CFICount > 0)
+    return outliner::OutlinedFunction();
+
   return outliner::OutlinedFunction(RepeatedSequenceLocs, SequenceSize,
                                     NumBytesToCreateFrame, FrameID);
 }
 
 bool AArch64InstrInfo::isFunctionSafeToOutlineFrom(
     MachineFunction &MF, bool OutlineFromLinkOnceODRs) const {
   const Function &F = MF.getFunction();
 
AArch64InstrInfo::getOutliningType(MachineBasicBlock::iterator &MIT,
   case AArch64::EMITBKEY:
     return outliner::InstrType::Illegal;
   }
 
   // Don't outline LOHs.
   if (FuncInfo->getLOHRelated().count(&MI))
     return outliner::InstrType::Illegal;
 
   // We can only outline these if we will tail call the outlined function, or
-  // fix up the CFI offsets. For the sake of safety, don't outline CFI
-  // instructions.
+  // fix up the CFI offsets. Currently, CFI instructions are outlined only if
+  // in a tail call.
   //
-  // FIXME: If the proper fixups are implemented, this should be possible.
+  // FIXME: If the proper fixups for the offset are implemented, this should be
+  // possible.
   if (MI.isCFIInstruction())
-    return outliner::InstrType::Illegal;
+    return outliner::InstrType::Legal;
 
   // Don't allow debug values to impact outlining type.
   if (MI.isDebugInstr() || MI.isIndirectDebugValue())
     return outliner::InstrType::Invisible;
 
   // At this point, KILL instructions don't really tell us much so we can go
   // ahead and skip over them.
   if (MI.isKill())

llvm/lib/Target/X86/X86InstrInfo.cpp

unsigned SequenceSize =
                         // FIXME: x86 doesn't implement getInstSizeInBytes, so
                         // we can't tell the cost.  Just assume each instruction
                         // is one byte.
                         if (MI.isDebugInstr() || MI.isKill())
                           return Sum;
                         return Sum + 1;
                       });
 
+  // We check to see if CFI Instructions are present, and if they are
+  // we find the number of CFI Instructions in the candidates.
+  unsigned CFICount = 0;
+  MachineBasicBlock::iterator MBBI = RepeatedSequenceLocs[0].front();
+  for (unsigned Loc = RepeatedSequenceLocs[0].getStartIdx();
+       Loc < RepeatedSequenceLocs[0].getEndIdx() + 1; Loc++) {
+    const std::vector<MCCFIInstruction> &CFIInstructions =
+        RepeatedSequenceLocs[0].getMF()->getFrameInstructions();
+    if (MBBI->isCFIInstruction()) {
+      unsigned CFIIndex = MBBI->getOperand(0).getCFIIndex();
+      MCCFIInstruction CFI = CFIInstructions[CFIIndex];
+      CFICount++;
+    }
+    MBBI++;
+  }
+
+  // We compare the number of found CFI Instructions to  the number of CFI
+  // instructions in the parent function for each candidate.  We must check this
+  // since if we outline one of the CFI instructions in a function, we have to
+  // outline them all for correctness. If we do not, the address offsets will be
+  // incorrect between the two sections of the program.
+  for (outliner::Candidate &C : RepeatedSequenceLocs) {
+    std::vector<MCCFIInstruction> CFIInstructions =
+        C.getMF()->getFrameInstructions();
+
+    if (CFICount > 0 && CFICount != CFIInstructions.size())
+      return outliner::OutlinedFunction();
+  }
+
   // FIXME: Use real size in bytes for call and ret instructions.
   if (RepeatedSequenceLocs[0].back()->isTerminator()) {
     for (outliner::Candidate &C : RepeatedSequenceLocs)
       C.setCallInfo(MachineOutlinerTailCall, 1);
 
     return outliner::OutlinedFunction(RepeatedSequenceLocs, SequenceSize,
                                       0, // Number of bytes to emit frame.
                                       MachineOutlinerTailCall // Type of frame.
     );
   }
 
+  if (CFICount > 0)
+    return outliner::OutlinedFunction();
+
   for (outliner::Candidate &C : RepeatedSequenceLocs)
     C.setCallInfo(MachineOutlinerDefault, 1);
 
   return outliner::OutlinedFunction(RepeatedSequenceLocs, SequenceSize, 1,
                                     MachineOutlinerDefault);
 }
 
 bool X86InstrInfo::isFunctionSafeToOutlineFrom(MachineFunction &MF,

llvm/test/CodeGen/AArch64/machine-outliner-cfi-tail-some.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=aarch64-apple-unknown -run-pass=machine-outliner -verify-machineinstrs %s -o - | FileCheck %s
+
+# Outlining CFI instructions is unsafe if we cannot outline all of the CFI
+# instructions from a function.  This shows that we choose not to outline the
+# CFI instructions since function foo has a CFI Instruction that would not
+# be caught.
+
+--- |
+  define void @foo() #0 { ret void }
+  define void @bar() #0 { ret void }
+  define void @baz() #0 { ret void }
+  attributes #0 = { noredzone }
+...
+---
+name:            foo
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  liveins: $lr
+    ; CHECK-LABEL: name: foo
+    ; CHECK: liveins: $lr
+    ; CHECK: frame-setup CFI_INSTRUCTION def_cfa $w28, 16
+    ; CHECK: $w9 = ORRWri $wzr, 1
+    ; CHECK: $w10 = ORRWri $wzr, 2
+    ; CHECK: $w11 = ORRWri $wzr, 3
+    ; CHECK: frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    frame-setup CFI_INSTRUCTION def_cfa $w28, 16
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
+    frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w12 = ORRWri $wzr, 1
+    $w13 = ORRWri $wzr, 2
+    $w14 = ORRWri $wzr, 3
+    $w15 = ORRWri $wzr, 4
+    RET undef $lr
+...
+---
+name:            bar
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  liveins: $lr
+    ; CHECK-LABEL: name: bar
+    ; CHECK: liveins: $lr
+    ; CHECK: $w9 = ORRWri $wzr, 1
+    ; CHECK: $w10 = ORRWri $wzr, 2
+    ; CHECK: $w11 = ORRWri $wzr, 3
+    ; CHECK: frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
+    frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w12 = ORRWri $wzr, 1
+    $w13 = ORRWri $wzr, 2
+    $w14 = ORRWri $wzr, 3
+    $w15 = ORRWri $wzr, 4
+    RET undef $lr
+...
+---
+name:            baz
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  liveins: $lr
+    ; CHECK-LABEL: name: baz
+    ; CHECK: liveins: $lr
+    ; CHECK: $w9 = ORRWri $wzr, 1
+    ; CHECK: $w10 = ORRWri $wzr, 2
+    ; CHECK: $w11 = ORRWri $wzr, 3
+    ; CHECK: frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
+    frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w12 = ORRWri $wzr, 1
+    $w13 = ORRWri $wzr, 2
+    $w14 = ORRWri $wzr, 3
+    $w15 = ORRWri $wzr, 4
+    RET undef $lr

llvm/test/CodeGen/AArch64/machine-outliner-cfi-tail.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=aarch64-apple-unknown -run-pass=machine-outliner -verify-machineinstrs %s -o - | FileCheck %s
+
+# Outlining CFI instructions is unsafe if it is not tail called, but otherwise,
+# it requires fixups. Show that we include CFI instructions in tail call 
+# outlined sequences right now.
+
+--- |
+  define void @foo() #0 { ret void }
+  define void @bar() #0 { ret void }
+  define void @baz() #0 { ret void }
+  attributes #0 = { noredzone }
+...
+---
+name:            foo
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  liveins: $lr
+    ; CHECK-LABEL: name: foo
+    ; CHECK: liveins: $lr
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w9, implicit-def $w10, implicit-def $w11, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
+    frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w12 = ORRWri $wzr, 1
+    $w13 = ORRWri $wzr, 2
+    $w14 = ORRWri $wzr, 3
+    $w15 = ORRWri $wzr, 4
+    RET undef $lr
+...
+---
+name:            bar
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  liveins: $lr
+    ; CHECK-LABEL: name: bar
+    ; CHECK: liveins: $lr
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w9, implicit-def $w10, implicit-def $w11, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
+    frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w12 = ORRWri $wzr, 1
+    $w13 = ORRWri $wzr, 2
+    $w14 = ORRWri $wzr, 3
+    $w15 = ORRWri $wzr, 4
+    RET undef $lr
+...
+---
+name:            baz
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  liveins: $lr
+    ; CHECK-LABEL: name: baz
+    ; CHECK: liveins: $lr
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w9, implicit-def $w10, implicit-def $w11, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
+    frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w12 = ORRWri $wzr, 1
+    $w13 = ORRWri $wzr, 2
+    $w14 = ORRWri $wzr, 3
+    $w15 = ORRWri $wzr, 4
+    RET undef $lr

llvm/test/CodeGen/AArch64/machine-outliner-cfi.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple=aarch64-apple-unknown -run-pass=machine-outliner -verify-machineinstrs %s -o - | FileCheck %s
 
-# Outlining CFI instructions is unsafe. It is possible if the call is tail
-# called, but otherwise, it requires fixups. Show that we don't include CFI
-# instructions in outlined sequences right now.
+# Outlining CFI instructions is unsafe if it is not tail called, but otherwise,
+# it requires fixups. Show that we don't include CFI instructions in non
+# tail call outlined sequences right now.
 
 --- |
   define void @foo() #0 { ret void }
   define void @bar() #0 { ret void }
   define void @baz() #0 { ret void }
   attributes #0 = { noredzone }
 ...
 ---
 name:            foo
 tracksRegLiveness: true
 body:             |
   bb.0:
   liveins: $lr
     ; CHECK-LABEL: name: foo
     ; CHECK: liveins: $lr
+    ; CHECK: $w9 = ORRWri $wzr, 1
+    ; CHECK: $w10 = ORRWri $wzr, 2
+    ; CHECK: $w11 = ORRWri $wzr, 3
     ; CHECK: frame-setup CFI_INSTRUCTION def_cfa $w29, 16
-    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15
+    ; CHECK: $w20 = ORRWri $wzr, 1
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
     frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w20 = ORRWri $wzr, 1
     $w12 = ORRWri $wzr, 1
     $w13 = ORRWri $wzr, 2
     $w14 = ORRWri $wzr, 3
     $w15 = ORRWri $wzr, 4
     RET undef $lr
 ...
 ---
 name:            bar
 tracksRegLiveness: true
 body:             |
   bb.0:
   liveins: $lr
     ; CHECK-LABEL: name: bar
     ; CHECK: liveins: $lr
+    ; CHECK: $w9 = ORRWri $wzr, 1
+    ; CHECK: $w10 = ORRWri $wzr, 2
+    ; CHECK: $w11 = ORRWri $wzr, 3
     ; CHECK: frame-setup CFI_INSTRUCTION def_cfa $w29, 16
-    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15
+    ; CHECK: $w21 = ORRWri $wzr, 1
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
     frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w21 = ORRWri $wzr, 1
     $w12 = ORRWri $wzr, 1
     $w13 = ORRWri $wzr, 2
     $w14 = ORRWri $wzr, 3
     $w15 = ORRWri $wzr, 4
     RET undef $lr
 ...
 ---
 name:            baz
 tracksRegLiveness: true
 body:             |
   bb.0:
   liveins: $lr
     ; CHECK-LABEL: name: baz
     ; CHECK: liveins: $lr
+    ; CHECK: $w9 = ORRWri $wzr, 1
+    ; CHECK: $w10 = ORRWri $wzr, 2
+    ; CHECK: $w11 = ORRWri $wzr, 3
     ; CHECK: frame-setup CFI_INSTRUCTION def_cfa $w29, 16
-    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15
+    ; CHECK: $w22 = ORRWri $wzr, 1
+    ; CHECK: TCRETURNdi @OUTLINED_FUNCTION_0, 0, implicit $sp, implicit-def $w12, implicit-def $w13, implicit-def $w14, implicit-def $w15, implicit $wzr, implicit $sp
+    $w9 = ORRWri $wzr, 1
+    $w10 = ORRWri $wzr, 2
+    $w11 = ORRWri $wzr, 3
     frame-setup CFI_INSTRUCTION def_cfa $w29, 16
+    $w22 = ORRWri $wzr, 1
     $w12 = ORRWri $wzr, 1
     $w13 = ORRWri $wzr, 2
     $w14 = ORRWri $wzr, 3
     $w15 = ORRWri $wzr, 4
     RET undef $lr

llvm/test/CodeGen/AArch64/machine-outliner-remarks.ll

 ; RUN: llc %s -enable-machine-outliner -mtriple=aarch64-unknown-unknown -pass-remarks=machine-outliner -pass-remarks-missed=machine-outliner -o /dev/null 2>&1 | FileCheck %s
 ; CHECK: <unknown>:0:0:
 ; CHECK-SAME: Did not outline 2 instructions from 2 locations.
 ; CHECK-SAME: Bytes from outlining all occurrences (16) >=
 ; CHECK-SAME: Unoutlined instruction bytes (16)
 ; CHECK-SAME: (Also found at: <UNKNOWN LOCATION>)
-; CHECK: remark: <unknown>:0:0: Saved 36 bytes by outlining 11 instructions
+; CHECK: remark: <unknown>:0:0: Saved 40 bytes by outlining 13 instructions
 ; CHECK-SAME: from 2 locations. (Found at: <UNKNOWN LOCATION>,
 ; CHECK-SAME: <UNKNOWN LOCATION>)
 ; RUN: llc %s -enable-machine-outliner -mtriple=aarch64-unknown-unknown -o /dev/null -pass-remarks-missed=machine-outliner -pass-remarks-output=%t.yaml
 ; RUN: cat %t.yaml | FileCheck %s -check-prefix=YAML
 
 ; For the YAML case, the function we pick depends on the order of the candidate
 ; list.
 ; YAML: --- !Missed

 ; YAML-NEXT:   - OtherStartLoc1:  '<UNKNOWN LOCATION>'
 ; YAML-NEXT:   - String:          ')'
 ; YAML: --- !Passed
 ; YAML-NEXT: Pass:            machine-outliner
 ; YAML-NEXT: Name:            OutlinedFunction
 ; YAML-NEXT: Function:        OUTLINED_FUNCTION_0
 ; YAML-NEXT: Args:
 ; YAML-NEXT:   - String:          'Saved '
-; YAML-NEXT:   - OutliningBenefit: '36'
+; YAML-NEXT:   - OutliningBenefit: '40'
 ; YAML-NEXT:   - String:          ' bytes by '
 ; YAML-NEXT:   - String:          'outlining '
-; YAML-NEXT:   - Length:          '11'
+; YAML-NEXT:   - Length:          '13'
 ; YAML-NEXT:   - String:          ' instructions '
 ; YAML-NEXT:   - String:          'from '
 ; YAML-NEXT:   - NumOccurrences:  '2'
 ; YAML-NEXT:   - String:          ' locations. '
 ; YAML-NEXT:   - String:          '(Found at: '
 ; YAML-NEXT:   - StartLoc0:       '<UNKNOWN LOCATION>'
 ; YAML-NEXT:   - String:          ', '
 ; YAML-NEXT:   - StartLoc1:       '<UNKNOWN LOCATION>'

llvm/test/CodeGen/X86/machine-outliner-cfi-tail-some.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=x86_64-apple-unknown -run-pass=machine-outliner -verify-machineinstrs %s -o - | FileCheck %s
+
+# Outlining CFI instructions is unsafe if we cannot outline all of the CFI
+# instructions from a function.  This shows that we choose not to outline the
+# CFI instructions since function foo has a CFI Instruction that would not
+# be caught.
+
+--- |
+  define void @foo() #0 { ret void }
+  define void @bar() #0 { ret void }
+  define void @baz() #0 { ret void }
+  attributes #0 = { noredzone }
+...
+---
+name:            foo
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    ; CHECK-LABEL: name: foo
+    ; CHECK: CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK: CFI_INSTRUCTION offset $rbp, -16
+    ; CHECK: CFI_INSTRUCTION def_cfa_register $rbp
+    ; CHECK: CALL64pcrel32 @OUTLINED_FUNCTION_1, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $ebx, implicit-def $ecx, implicit $rsp, implicit $ssp
+    ; CHECK: CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK: CFI_INSTRUCTION offset $rbp, -16
+    ; CHECK: CFI_INSTRUCTION def_cfa_register $rsp
+    ; CHECK: TAILJMPd64 @OUTLINED_FUNCTION_0, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $edi, implicit-def $edx, implicit-def $esi, implicit $rsp, implicit $ssp
+    CFI_INSTRUCTION def_cfa_offset 16
+    CFI_INSTRUCTION offset $rbp, -16
+    CFI_INSTRUCTION def_cfa_register $rbp
+    $eax =  MOV32ri 1
+    $ebx =  MOV32ri 2
+    $ecx =  MOV32ri 3
+    CFI_INSTRUCTION def_cfa_offset 16
+    CFI_INSTRUCTION offset $rbp, -16
+    CFI_INSTRUCTION def_cfa_register $rsp
+    $esi = MOV32ri 1
+    $edx = MOV32ri 2
+    $edi = MOV32ri 3
+    $eax = MOV32ri 4
+    RETQ
+...
+---
+name:            bar
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  ;liveins: $lr
+    ; CHECK-LABEL: name: bar
+    ; CHECK: CALL64pcrel32 @OUTLINED_FUNCTION_1, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $ebx, implicit-def $ecx, implicit $rsp, implicit $ssp
+    ; CHECK: CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK: CFI_INSTRUCTION offset $rbp, -16
+    ; CHECK: CFI_INSTRUCTION def_cfa_register $rsp
+    ; CHECK: TAILJMPd64 @OUTLINED_FUNCTION_0, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $edi, implicit-def $edx, implicit-def $esi, implicit $rsp, implicit $ssp
+    $eax =  MOV32ri 1
+    $ebx =  MOV32ri 2
+    $ecx =  MOV32ri 3
+    CFI_INSTRUCTION def_cfa_offset 16
+    CFI_INSTRUCTION offset $rbp, -16
+    CFI_INSTRUCTION def_cfa_register $rsp
+    $esi = MOV32ri 1
+    $edx = MOV32ri 2
+    $edi = MOV32ri 3
+    $eax = MOV32ri 4
+    RETQ
+...
+---
+name:            baz
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  ;liveins: $lr
+    ; CHECK-LABEL: name: baz
+    ; CHECK: CALL64pcrel32 @OUTLINED_FUNCTION_1, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $ebx, implicit-def $ecx, implicit $rsp, implicit $ssp
+    ; CHECK: CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK: CFI_INSTRUCTION offset $rbp, -16
+    ; CHECK: CFI_INSTRUCTION def_cfa_register $rsp
+    ; CHECK: TAILJMPd64 @OUTLINED_FUNCTION_0, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $edi, implicit-def $edx, implicit-def $esi, implicit $rsp, implicit $ssp
+    $eax =  MOV32ri 1
+    $ebx =  MOV32ri 2
+    $ecx =  MOV32ri 3
+    CFI_INSTRUCTION def_cfa_offset 16
+    CFI_INSTRUCTION offset $rbp, -16
+    CFI_INSTRUCTION def_cfa_register $rsp
+    $esi = MOV32ri 1
+    $edx = MOV32ri 2
+    $edi = MOV32ri 3
+    $eax = MOV32ri 4
+    RETQ

llvm/test/CodeGen/X86/machine-outliner-cfi-tail.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=x86_64-apple-unknown -run-pass=machine-outliner -verify-machineinstrs %s -o - | FileCheck %s
+
+# Outlining CFI instructions is unsafe if we cannot outline all of the CFI
+# instructions from a function.  This shows that we choose to outline the
+# CFI instructions if they can be included in a tail call.
+
+--- |
+  define void @foo() #0 { ret void }
+  define void @bar() #0 { ret void }
+  define void @baz() #0 { ret void }
+  attributes #0 = { noredzone }
+...
+---
+name:            foo
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    ; CHECK-LABEL: name: foo
+    ; CHECK: $ecx = MOV32ri 3
+    ; CHECK: CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK: CFI_INSTRUCTION offset $rbp, -16
+    ; CHECK: CFI_INSTRUCTION def_cfa_register $rsp
+    ; CHECK: TAILJMPd64 @OUTLINED_FUNCTION_0, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $edi, implicit-def $edx, implicit-def $esi, implicit $rsp, implicit $ssp
+    $ecx =  MOV32ri 3
+    CFI_INSTRUCTION def_cfa_offset 16
+    CFI_INSTRUCTION offset $rbp, -16
+    CFI_INSTRUCTION def_cfa_register $rsp
+    $esi = MOV32ri 1
+    $edx = MOV32ri 2
+    $edi = MOV32ri 3
+    $eax = MOV32ri 4
+    RETQ
+...
+---
+name:            bar
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  ;liveins: $lr
+    ; CHECK-LABEL: name: bar
+    ; CHECK: $ecx = MOV32ri 3
+    ; CHECK: CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK: CFI_INSTRUCTION offset $rbp, -16
+    ; CHECK: CFI_INSTRUCTION def_cfa_register $rsp
+    ; CHECK: TAILJMPd64 @OUTLINED_FUNCTION_0, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $edi, implicit-def $edx, implicit-def $esi, implicit $rsp, implicit $ssp
+    $ecx =  MOV32ri 3
+    CFI_INSTRUCTION def_cfa_offset 16
+    CFI_INSTRUCTION offset $rbp, -16
+    CFI_INSTRUCTION def_cfa_register $rsp
+    $esi = MOV32ri 1
+    $edx = MOV32ri 2
+    $edi = MOV32ri 3
+    $eax = MOV32ri 4
+    RETQ
+...
+---
+name:            baz
+tracksRegLiveness: true
+body:             |
+  bb.0:
+  ;liveins: $lr
+    ; CHECK-LABEL: name: baz
+    ; CHECK: $ecx = MOV32ri 3
+    ; CHECK: CFI_INSTRUCTION def_cfa_offset 16
+    ; CHECK: CFI_INSTRUCTION offset $rbp, -16
+    ; CHECK: CFI_INSTRUCTION def_cfa_register $rsp
+    ; CHECK: TAILJMPd64 @OUTLINED_FUNCTION_0, implicit $rsp, implicit $ssp, implicit-def $eax, implicit-def $edi, implicit-def $edx, implicit-def $esi, implicit $rsp, implicit $ssp
+    $ecx =  MOV32ri 3
+    CFI_INSTRUCTION def_cfa_offset 16
+    CFI_INSTRUCTION offset $rbp, -16
+    CFI_INSTRUCTION def_cfa_register $rsp
+    $esi = MOV32ri 1
+    $edx = MOV32ri 2
+    $edi = MOV32ri 3
+    $eax = MOV32ri 4
+    RETQ