Add TCOPY, a terminator form of the COPY instr

[void]

Summary

The INLINEASM_BR instruction's a terminator, and the code generator
doesn't allow non-terminator instructions after a terminator. This is an
issue when an INLINEASM_BR defines a physical register. We can't place
the copies of the physical registers into virtual registers in the
fallthrough block because physical registers can't be marked as
"live-in" until after register allocation.

To get around this issue, we introduce a new pseudo-instruction, TCOPY,
that's identical to the COPY instruction, but is a terminator. With it
we're able to copy the physical registers to virtual registers without
needing to place the copies in a fallthrough block:

bb.1:
  INLINEASM_BR &"" ..., implicit-def $esi, $1:[regdef],...
  %9:gr32 = TCOPY $esi

bb.2:
; predecessors: %bb.1
  successors: %bb.3(0x80000000); %bb.3(100.00%)

  %0:gr32 = COPY %9:gr32
  JMP_1 %bb.3

The TCOPY is converted to a normal COPY after register allocation, when
we have live variable information and live-ins are allowed on basic
blocks.

The fast register allocator behaves a bit differently because everything
is spilled before the end of a basic block. Therefore, we allow a store
of a physical register after the INLINEASM_BR when optimizations are
disabled.

[arsenm]

This will be useful for AMDGPU, we currently have a set of _term mov instructions for this purpose.

llvm/include/llvm/Support/TargetOpcodes.def

101

s/terminal/terminator

llvm/include/llvm/Target/Target.td

1124–1125

This shouldn't need isBranch or isIndirectBranch

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

179

I would expect this to not special case INLINEASM_BR. This should be any value defined by a terminator instruction

[arsenm]

llvm/include/llvm/Support/TargetOpcodes.def

101

COPY_TERM or COPY_TERMINATOR would be better since it isn't really a branch

[nickdesaulniers]

llvm/include/llvm/CodeGen/MachineInstr.h

1129

Does this save a few getOpcode() calls? The rest of this CL explicitly compares the opcode against COPY_BR? I just worry if this might mess up existing callsites of isCopy when getOpcode() == TargetOpcode::COPY_BR;.

llvm/include/llvm/Target/Target.td

1126

COPY also sets hasNoSchedulingInfo to 0. Should COPY_BR do that as well?

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

191

what? if we could use a range based for loop for the MBB's, surely we can for MI.

llvm/lib/CodeGen/MachineSink.cpp

1398

Does it make sense to "sink" register info invalidation into performSink()? (pun intended) Since it's already checking the MI's opcode? Or are the two call sites of performSink() problematic? The implementation of invalidateLiveness() looks pretty cheap, IMO.

llvm/lib/CodeGen/MachineVerifier.cpp

849

this is hard to read. Is there a nice way to simply this? Maybe negate, and fold into parent if?

[void]

llvm/include/llvm/CodeGen/MachineInstr.h

1129

It's more than just a few. isCopy() is used pretty extensively. I want the COPY_BR (or COPY_TERM) to be exactly like a normal COPY, with the only exception that it can come after a terminator.

llvm/include/llvm/Support/TargetOpcodes.def

101

How about TCOPY?

llvm/include/llvm/Target/Target.td

1126

Possibly, though I had trouble finding where to place its scheduling info. It seems like COPY is special and TableGen adds it by default. I couldn't find the place to do that for COPY_BR, but I'll take another look.

[void]

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

191

The ++mi below made me worried, and the fact that MI could be erased.

llvm/lib/CodeGen/MachineSink.cpp

1398

Possibly. If we're going to make COPY_TERM (or whatever name we settle on) not specific to INLINEASM_BR, then we'll probably need to modify this to accept any case where we sink copies after a terminator. We'll come back to this after other comments settle.

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

179

Does this include unconditional jump terminators?

[arsenm]

llvm/include/llvm/Target/Target.td

1126

Last time I looked at adding a copy variant, the right solution was just isAsCheapAsAMove = 1. Otherwise it would have required adding the COPY_BR to every target's COPY handling

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

179

I'm not sure how you could construct a sensible unconditional jump that would require a copy after it. Seems like something to check in the verifier

[nickdesaulniers]

Also, if INLINEASM_BR is a terminator, but we want TCOPY to be a terminator, should this patch make INLINEASM_BR NOT a terminator? Or would that be a follow up patch after creating TCOPY?

Also, none of the tests have TCOPY (or w/e). I feel like any patch adding a new instructions at w/e level of IR should have some test that the instruction exists.

Also, I don't quite follow what the changes to the tests are testing for with this change.

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

191

Ah sorry, I missed that. I'm not sure what's meant by erased, but I'd be curious if a continue could not be used instead of manual iterator advancement.

[void]

In D75098#1892171, @nickdesaulniers wrote:

Also, if INLINEASM_BR is a terminator, but we want TCOPY to be a terminator, should this patch make INLINEASM_BR NOT a terminator? Or would that be a follow up patch after creating TCOPY?

INLINEASM_BR being a terminator makes a lot of things "just work" for the code generator. I'm not 100% convinced that it *should* be a terminator, but if we decided to make it a normal instruction, then I think we'd have to touch way too much code to make it worthwhile.

Also, none of the tests have TCOPY (or w/e). I feel like any patch adding a new instructions at w/e level of IR should have some test that the instruction exists.

The tests are the current "asm goto with outputs" tests.

Also, I don't quite follow what the changes to the tests are testing for with this change.

The test changes are testing that we can compile at -O0.

[void]

Don't mark TCOPY as a branch. Also s/COPY_BR/TCOPY/.

[void]

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

179

My comment wasn't very good. When converting something to a TCOPY, we need to figure out which terminators are candidates for having a TCOPY after them. Or perhaps better, what condition requires a TCOPY instead of a regular COPY.

[void]

Liveness is handled by the PostRA machine sink pass, so there's no need to
invalidate it. Also simplify the logic in the machine verifier that allows
stack dumps after a terminator.

[nickdesaulniers]

llvm/lib/CodeGen/MachineSink.cpp

845

pass as const&, or make performSink a private method.

[void]

Fix accidental change inclusion in the verifier.

[void]

llvm/lib/CodeGen/MachineSink.cpp

845

Two different passes use this function so it can't be made a private method. I"m not sure why passing it as "const&" is better than a "const*"...

[void]

Reformat and satisfy clang-tidy.

[nickdesaulniers]

llvm/lib/CodeGen/MachineSink.cpp

845

Generally, passing by const reference indicates that a parameter is strictly an input, as opposed to both input AND output, which is why you don't see a mix of pointers and references in this function signature.

[void]

llvm/lib/CodeGen/MachineSink.cpp

845

A 'const*' doesn't allow modifications either. Also note that none of the references in this function signature are 'const', and are probably passed by reference because they aren't pointers (SuccToSinkto notwithstanding) in the originating function. Converting this to a reference is not useful.

[arsenm]

llvm/include/llvm/CodeGen/MachineInstr.h

1129

I think this is potentially hazardous. I'm worried about the peephole optimizer doing things like folding a copy into a tcopy and losing the terminator bit. Can you add some testcases with coalescable tcopy pairs for PeepholeOptimizer and the register coalescer?

[void]

• Add a verification step to ensure a COPY doesn't follow a TCOPY.
• Comment that the predicate for determining whether to use a TCOPY or COPY should be improved, but at this time we don't have enough information to determine the best criteria.

[void]

llvm/include/llvm/CodeGen/MachineInstr.h

1129

I'll craft some tests, though note that TCOPY should only be after a terminator, so a COPY should never be merged with it. (I'll add a verifier check to ensure that TCOPY doesn't happen before a terminator.)

[void]

Add TCOPY to some switch statements that handle COPY

[void]

llvm/include/llvm/CodeGen/MachineInstr.h

1129

@arsenm I'm struggling a bit to come up with tests that will exercise the peephole optimizer and register coalescer. Do you have any advice on how to do this?

[void]

Rebase and update testcase.

[void]

Friendly ping. :-)

[void]

Another friendly ping. :-)

[arsenm]

llvm/include/llvm/CodeGen/MachineInstr.h

1129

You shouldn't need a special verifier check, it should be caught by the normal terminator before non-terminator check.

I mean something like
%1 = COPY %0
%2 = TCOPY %1

and then maybe mix in some subregistsers? The worry is the COPY will somehow replace the TCOPY

[void]

llvm/include/llvm/CodeGen/MachineInstr.h

1129

This isn't an issue as far as I can see (at least the kind of example you're referring to). A TCOPY can become a COPY if appropriate. The only time it can't become a COPY is if it occurs after a terminator, in which case it should be caught by current verifier checks. I *don't* think a COPY should become a TCOPY though.

[void]

• Add support to MIR parsing.
• Rebase current change.

[arsenm]

llvm/lib/CodeGen/MIRParser/MIRParser.cpp

312–342 ↗

(On Diff #261771)

This is an unrelated change

llvm/lib/CodeGen/MachineSink.cpp

861–864

I don't see this captured in a test?

llvm/lib/CodeGen/MachineVerifier.cpp

629–630

This also looks like a separate change

[qcolombet]

I don't think introducing the TCOPY opcode is a valid solution.
Let's assume the copy doesn't get coalesced, that means it needs to be executed after the inlineasm branch. And this is not going to happen if it sits in the same basic block right after the jump.

What happens if you put the copy at the beginning of each successor?
You mentioned that this is not possible because the live-ins wouldn't be set, but I would expect that this information would be correctly computed when building the liveness information. I.e., I don't expect this to be a problem.

we allow a store of a physical register after the INLINEASM_BR when optimizations are disabled.

That sounds wrong.

Could we step back a little bit, what is the semantic of INLINEASM_BR?

[void]

In D75098#2017980, @qcolombet wrote:

I don't think introducing the TCOPY opcode is a valid solution.
Let's assume the copy doesn't get coalesced, that means it needs to be executed after the inlineasm branch. And this is not going to happen if it sits in the same basic block right after the jump.

What happens if you put the copy at the beginning of each successor?
You mentioned that this is not possible because the live-ins wouldn't be set, but I would expect that this information would be correctly computed when building the liveness information. I.e., I don't expect this to be a problem.

we allow a store of a physical register after the INLINEASM_BR when optimizations are disabled.

That sounds wrong.

Could we step back a little bit, what is the semantic of INLINEASM_BR?

[I'm sorry if some of this is basic. I just want to give a clear description of what's going on and why.]

INLINEASM_BR is an attempt to replicate the behavior of "asm goto". As such, it has one default destination (more-or-less the fallthrough block), and one or more indirect destinations. If the indirect destinations aren't taken, the resulting control flow is to fall out of the bottom of the ASM block. The callbr instruction is a terminator so that it can model this behavior as best as it can (which isn't all that great with LLVM's IR, but doable (see "invoke")). Once it's converted to INLINEASM_BR the behavior obviously doesn't change, but now that we have outputs, we need to figure out how to handle them given the constraints of MIR (i.e. you can't have non-terminals after a terminal, and live-ins are only expected to be correct after register allocation).

Because of how callbr is modeled, INLINEASM_BR is also a terminator. So during ISEL the values that are defined by INLINEASM_BR need to be spilled before jumping to the default block. Because of the constraint against having non-terminators after a terminator, we artificially modify the code after ISEL's finished building the block, placing any spills into a "copy" block (that's between the INLINEASM_BR and its default destination), and adding any registers defined by INLINEASM_BR as "live-ins" to the copy block. This is bad, because it violates the constraints, but was necessary at the time because we had no way to avoid it.

[Note that the outputs from an INLINEASM_BR are valid only on the default branch. It's very difficult to have them on the indirect branches without making the resulting code bad---both in style and in performance.]

Okay, so that's the behavior of INLINEASM_BR and some of the issues we came across that led us to here. I think TCOPY is a good solution because the default behavior of INLINEASM_BR is to fall out the bottom of the ASM block, so it would "naturally" execute the TCOPY instructions. And since we only ensure that the values are valid on the default branch, the COPYs will always be executed when needed (note a block containing an INLINEASM_BR should either exit to the default via a fallthrough or unconditional jump). It's still possible to split the block directly after the INLINEASM_BR and convert the TCOPYs into regular COPYs, in fact I would encourage us to do that at the appropriate point. It could remove a lot of the code in ScheduleDAGSDNodes.cpp.

What happens if you put the copy at the beginning of each successor?

I found that this didn't work. CodeGen wants registers to be spilled at the end of a block. The spill slot is recorded and used later on when accessing the value. If we just add a COPY into the default block it won't have the necessary information available to load the correct value. We would have to add the information as a "live-in" on the default block, but that's a constraint violation, wash-rinse-repeat.

[qcolombet]

Thanks Bill for the detailed explanations. A couple more questions.

CodeGen wants registers to be spilled at the end of a block.

That's fast reg alloc only. What is happening with the other allocators?

The spill slot is recorded and used later on when accessing the value. If we just add a COPY into the default block it won't have the necessary information available to load the correct value.

Why is that? (Why doesn't it have the necessary information to load the correct value.)

I feel that we are trying to work around a bug/limitation in the fast register allocator where physreg are assumed not to cross basic block boundaries. I wonder if it wouldn't be easier to fix fast reg alloc if that is the problem.

Out-of-curiosity, how do we deal with invoke? I would expect we have the exact same problems.

[void]

In D75098#2021573, @qcolombet wrote:

Thanks Bill for the detailed explanations. A couple more questions.

CodeGen wants registers to be spilled at the end of a block.

That's fast reg alloc only. What is happening with the other allocators?

They don't appear to use the correct registers for the defined values.

The spill slot is recorded and used later on when accessing the value. If we just add a COPY into the default block it won't have the necessary information available to load the correct value.

Why is that? (Why doesn't it have the necessary information to load the correct value.)

I feel that we are trying to work around a bug/limitation in the fast register allocator where physreg are assumed not to cross basic block boundaries. I wonder if it wouldn't be easier to fix fast reg alloc if that is the problem.

It's not so much working around a bug/limitation in any of the register allocators. What I'm most concerned about is having to specify live-ins to MBBs before register allocation. That invariant was a huge sticking point in the review process for this feature. We're currently violating it because it "seems to work", but I don't want to rely on that.

Because of the live-ins restriction, all live physical registers at the end of an MBB need to be "spilled", or at least recorded in a way so that other blocks can access the information without having their live-ins set. (This is my understanding, it may be inaccurate). The SelectionDAGBuilder::CopyToExportRegsIfNeeded() function is one of the ways this is done.

Out-of-curiosity, how do we deal with invoke? I would expect we have the exact same problems.

Invoke is turned into a regular call with exception handling stuff around it. E.g.:

$ cat ex.cpp:
void g();
int f() {
  try {
    g();
  } catch (int&) {
    return 37;
  }
  return 1;
}

$ clang++ -mllvm -print-after-all ex.cpp -c -o /dev/null
...
bb.0 (%ir-block.0):
  successors: %bb.1, %bb.2

  EH_LABEL <mcsymbol .Ltmp0>
  ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
  CALL64pcrel32 @_Z1gv, <regmask $bh $bl $bp $bph $bpl $bx $ebp $ebx $hbp $hbx $rbp $rbx $r12 $r13 $r14 $r15 $r12b $r13b $r14b $r15b $r12bh $r13bh $r14bh $r15bh $r12d $r13d $r14d $r15d $r12w $r13w $r14w $r15w $r12wh and 3 more...>, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
  ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
  EH_LABEL <mcsymbol .Ltmp1>
  JMP_1 %bb.1
...

[void]

Remove some of the hackery in the SDNode scheduler that splits below an
INLINEASM_BR instruction. It's not needed with TCOPY.

[void]

Allow a TCOPY to sink, since it's exactly like a COPY.

[void]

llvm/lib/CodeGen/MachineSink.cpp

861–864

This will happen now that we correctly mark TCOPY as sinkable. I'll see if I can craft an MIR test to explicitly do this.

[void]

Spill TCOPY in FastRA right before the TCOPY instr instead of before the
terminators.

[void]

Fix formatting and clang-tidy warnings.

[void]

Revert bad formatting that sneaked (snuck?) in.

[void]

In D75098#2026425, @void wrote:

In D75098#2021573, @qcolombet wrote:

Thanks Bill for the detailed explanations. A couple more questions.

CodeGen wants registers to be spilled at the end of a block.

That's fast reg alloc only. What is happening with the other allocators?

They don't appear to use the correct registers for the defined values.

I should probably explain a bit better. With a normal INLINEASM instruction, any defined registers are spilled directly afterwards. This is one reason why I want to do the same for INLINEASM_BR with the TCOPY. There could be other ways to achieve the same thing, but I think it would involve placing a larger burden on the code generation passes than needs be.

[jyknight]

I think the better fix here is to make INLINEASM_BR _not_ be a terminator -- just like the CALL involved in an invoke is not.

That change turned out to be a bit complicated, and while I had started taking a stab at that a couple months ago, I got distracted by many other things going on...However, I've now dusted off that experiment and got it into a useful state, and will send out a review for this tomorrow, so we can consider both the options.

[void]

In D75098#2030777, @jyknight wrote:

I think the better fix here is to make INLINEASM_BR _not_ be a terminator -- just like the CALL involved in an invoke is not.

That change turned out to be a bit complicated, and while I had started taking a stab at that a couple months ago, I got distracted by many other things going on...However, I've now dusted off that experiment and got it into a useful state, and will send out a review for this tomorrow, so we can consider both the options.

I've thought about the same thing. I'm right now running up against an issue that Nick pointed out. Having a non-branch as a terminator works until the register allocator wants to spill. Then it could insert a copy where the TCOPY is, which messes up analyzeBranch. I'm interested to see your patch.

[arsenm]

In D75098#2030777, @jyknight wrote:

I think the better fix here is to make INLINEASM_BR _not_ be a terminator -- just like the CALL involved in an invoke is not.

That change turned out to be a bit complicated, and while I had started taking a stab at that a couple months ago, I got distracted by many other things going on...However, I've now dusted off that experiment and got it into a useful state, and will send out a review for this tomorrow, so we can consider both the options.

In D75098#2032980, @void wrote:

In D75098#2030777, @jyknight wrote:

I think the better fix here is to make INLINEASM_BR _not_ be a terminator -- just like the CALL involved in an invoke is not.

That change turned out to be a bit complicated, and while I had started taking a stab at that a couple months ago, I got distracted by many other things going on...However, I've now dusted off that experiment and got it into a useful state, and will send out a review for this tomorrow, so we can consider both the options.

I've thought about the same thing. I'm right now running up against an issue that Nick pointed out. Having a non-branch as a terminator works until the register allocator wants to spill. Then it could insert a copy where the TCOPY is, which messes up analyzeBranch. I'm interested to see your patch.

Regardless of what happens to INLINEASM_BR, I think we still need a terminator copy

[qcolombet]

Regardless of what happens to INLINEASM_BR, I think we still need a terminator copy

What for?
The whole concept is bogus to me because if the previous instructions are really terminators (as in they branch to somewhere else), then this copy would never be executed.

[arsenm]

In D75098#2049787, @qcolombet wrote:

Regardless of what happens to INLINEASM_BR, I think we still need a terminator copy

What for?
The whole concept is bogus to me because if the previous instructions are really terminators (as in they branch to somewhere else), then this copy would never be executed.

Terminator does not imply a branch, it only means glued to the end of the block. It's the only way to get correct spill code placement around exec mask writes. We need a copy to the exec mask, but spill code needs to be inserted before that point. Currently we get unnecessary copies since we use a raw move instructions for this purpose, when we could have coalesced a copy

[void]

In D75098#2049787, @qcolombet wrote:

Regardless of what happens to INLINEASM_BR, I think we still need a terminator copy

What for?
The whole concept is bogus to me because if the previous instructions are really terminators (as in they branch to somewhere else), then this copy would never be executed.

That's not necessarily so, because you can have multiple branching terminators then there must be some that branch conditionally (allowing other code to exist between the jumps), otherwise there wouldn't be a need for multiple terminators.

[arsenm]

What's the status of this? I'm still interested in terminator copies

[void]

In D75098#2170382, @arsenm wrote:

What's the status of this? I'm still interested in terminator copies

When we last left our adventurers, there was an issue with live range splitting. In some cases it could emit a spill after terminators, which failed validation.

llvm/include/llvm/CodeGen/MachineInstr.h

bool isRegSequence() const {
     return getOpcode() == TargetOpcode::REG_SEQUENCE;
   }
 
   bool isBundle() const {
     return getOpcode() == TargetOpcode::BUNDLE;
   }
 
   bool isCopy() const {
-    return getOpcode() == TargetOpcode::COPY;
+    return getOpcode() == TargetOpcode::COPY ||
+        getOpcode() == TargetOpcode::TCOPY;
   }
 
   bool isFullCopy() const {
     return isCopy() && !getOperand(0).getSubReg() && !getOperand(1).getSubReg();
   }
 
   bool isExtractSubreg() const {
     return getOpcode() == TargetOpcode::EXTRACT_SUBREG;

llvm/include/llvm/Support/TargetOpcodes.def

 HANDLE_TARGET_OPCODE(SUBREG_TO_REG)
 
 /// COPY_TO_REGCLASS - This instruction is a placeholder for a plain
 /// register-to-register copy into a specific register class. This is only
 /// used between instruction selection and MachineInstr creation, before
 /// virtual registers have been created for all the instructions, and it's
 /// only needed in cases where the register classes implied by the
 /// instructions are insufficient. It is emitted as a COPY MachineInstr.
-  HANDLE_TARGET_OPCODE(COPY_TO_REGCLASS)
 
 /// DBG_VALUE - a mapping of the llvm.dbg.value intrinsic
 HANDLE_TARGET_OPCODE(DBG_VALUE)
 
 /// DBG_LABEL - a mapping of the llvm.dbg.label intrinsic
 HANDLE_TARGET_OPCODE(DBG_LABEL)
 
 /// REG_SEQUENCE - This variadic instruction is used to form a register that
 /// represents a consecutive sequence of sub-registers. It's used as a
 /// register coalescing / allocation aid and must be eliminated before code
 /// emission.
 // In SDNode form, the first operand encodes the register class created by
 // the REG_SEQUENCE, while each subsequent pair names a vreg + subreg index
 // pair.  Once it has been lowered to a MachineInstr, the regclass operand
 // is no longer present.
 /// e.g. v1027 = REG_SEQUENCE v1024, 3, v1025, 4, v1026, 5
 /// After register coalescing references of v1024 should be replace with
 /// v1027:3, v1025 with v1027:4, etc.
-  HANDLE_TARGET_OPCODE(REG_SEQUENCE)
 
 /// COPY - Target-independent register copy. This instruction can also be
 /// used to copy between subregisters of virtual registers.
-  HANDLE_TARGET_OPCODE(COPY)
 
+/// TCOPY - This instruction is the terminator version of COPY. The purpose
+/// is to allow copies from terminators to be properly represented (e.g. an
+/// INLINEASM_BR that defines a physical register) without having
+/// to introduce "live-ins" for physical registers before register allocation.
+HANDLE_TARGET_OPCODE(TCOPY)
+
 /// BUNDLE - This instruction represents an instruction bundle. Instructions
 /// which immediately follow a BUNDLE instruction which are marked with
 /// 'InsideBundle' flag are inside the bundle.
 HANDLE_TARGET_OPCODE(BUNDLE)
 
 /// Lifetime markers.
 HANDLE_TARGET_OPCODE(LIFETIME_START)
 HANDLE_TARGET_OPCODE(LIFETIME_END)

llvm/include/llvm/Target/Target.td

 def COPY : StandardPseudoInstruction {
   let OutOperandList = (outs unknown:$dst);
   let InOperandList = (ins unknown:$src);
   let AsmString = "";
   let hasSideEffects = 0;
   let isAsCheapAsAMove = 1;
   let hasNoSchedulingInfo = 0;
 }
+def TCOPY : StandardPseudoInstruction {
+  let OutOperandList = (outs unknown:$dst);
+  let InOperandList = (ins unknown:$src);
+  let AsmString = "";
+  let hasSideEffects = 0;
+  let isAsCheapAsAMove = 1;
+  let isTerminator = 1;
+}
 def BUNDLE : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins variable_ops);
   let AsmString = "BUNDLE";
   let hasSideEffects = 0;
 }
 def LIFETIME_START : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins i32imm:$id);
   let AsmString = "LIFETIME_START";

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

bool ExpandPostRA::LowerSubregToReg(MachineInstr *MI) {
   }
 
   LLVM_DEBUG(dbgs() << '\n');
   MBB->erase(MI);
   return true;
 }
 
 bool ExpandPostRA::LowerCopy(MachineInstr *MI) {
-
   if (MI->allDefsAreDead()) {
     LLVM_DEBUG(dbgs() << "dead copy: " << *MI);
     MI->setDesc(TII->get(TargetOpcode::KILL));
     LLVM_DEBUG(dbgs() << "replaced by: " << *MI);
     return true;
   }
 
   MachineOperand &DstMO = MI->getOperand(0);
bool ExpandPostRA::runOnMachineFunction(MachineFunction &MF) {
   LLVM_DEBUG(dbgs() << "Machine Function\n"
                     << "********** EXPANDING POST-RA PSEUDO INSTRS **********\n"
                     << "********** Function: " << MF.getName() << '\n');
   TRI = MF.getSubtarget().getRegisterInfo();
   TII = MF.getSubtarget().getInstrInfo();
 
   bool MadeChange = false;
 
-  for (MachineFunction::iterator mbbi = MF.begin(), mbbe = MF.end();
-       mbbi != mbbe; ++mbbi) {
-    for (MachineBasicBlock::iterator mi = mbbi->begin(), me = mbbi->end();
+  for (auto &MBB : MF) {
+    for (MachineBasicBlock::iterator mi = MBB.begin(), me = MBB.end();
          mi != me;) {
       MachineInstr &MI = *mi;
       // Advance iterator here because MI may be erased.
       ++mi;
 
       // Only expand pseudos.
       if (!MI.isPseudo())
         continue;
 
       // Give targets a chance to expand even standard pseudos.
       if (TII->expandPostRAPseudo(MI)) {
         MadeChange = true;
         continue;
       }
 
       // Expand standard pseudos.
       switch (MI.getOpcode()) {
       case TargetOpcode::SUBREG_TO_REG:
         MadeChange |= LowerSubregToReg(&MI);
         break;
       case TargetOpcode::COPY:
+      case TargetOpcode::TCOPY:
         MadeChange |= LowerCopy(&MI);
         break;
       case TargetOpcode::DBG_VALUE:
         continue;
       case TargetOpcode::INSERT_SUBREG:
       case TargetOpcode::EXTRACT_SUBREG:
         llvm_unreachable("Sub-register pseudos should have been eliminated.");
       }
     }
   }
 
   return MadeChange;
 }

llvm/lib/CodeGen/MachineSink.cpp

static bool attemptDebugCopyProp(MachineInstr &SinkInst, MachineInstr &DbgMI) {
   DbgMI.getOperand(0).setReg(SrcMO->getReg());
   DbgMI.getOperand(0).setSubReg(SrcMO->getSubReg());
   return true;
 }
 
 /// Sink an instruction and its associated debug instructions.
 static void performSink(MachineInstr &MI, MachineBasicBlock &SuccToSinkTo,
                         MachineBasicBlock::iterator InsertPos,
+                        const TargetInstrInfo *TII,
                         SmallVectorImpl<MachineInstr *> &DbgValuesToSink) {
-
   // If we cannot find a location to use (merge with), then we erase the debug
   // location to prevent debug-info driven tools from potentially reporting
   // wrong location information.
   if (!SuccToSinkTo.empty() && InsertPos != SuccToSinkTo.end())
     MI.setDebugLoc(DILocation::getMergedLocation(MI.getDebugLoc(),
                                                  InsertPos->getDebugLoc()));
   else
     MI.setDebugLoc(DebugLoc());
 
   // Move the instruction.
   MachineBasicBlock *ParentBlock = MI.getParent();
   SuccToSinkTo.splice(InsertPos, ParentBlock, MI,
                       ++MachineBasicBlock::iterator(MI));
 
+  // The copy no longer needs to be a terminator, so convert it to a normal
+  // COPY.
+  if (MI.getOpcode() == TargetOpcode::TCOPY)
+    MI.setDesc(TII->get(TargetOpcode::COPY));
+
   // Sink a copy of debug users to the insert position. Mark the original
   // DBG_VALUE location as 'undef', indicating that any earlier variable
   // location should be terminated as we've optimised away the value at this
   // point.
-  for (SmallVectorImpl<MachineInstr *>::iterator DBI = DbgValuesToSink.begin(),
-                                                 DBE = DbgValuesToSink.end();
-       DBI != DBE; ++DBI) {
-    MachineInstr *DbgMI = *DBI;
-    MachineInstr *NewDbgMI = DbgMI->getMF()->CloneMachineInstr(*DBI);
+  for (auto *DbgMI : DbgValuesToSink) {
+    MachineInstr *NewDbgMI = DbgMI->getMF()->CloneMachineInstr(DbgMI);
     SuccToSinkTo.insert(InsertPos, NewDbgMI);
 
     if (!attemptDebugCopyProp(MI, *DbgMI))
       DbgMI->getOperand(0).setReg(0);
   }
 }
 
 /// SinkInstruction - Determine whether it is safe to sink the specified machine
 /// instruction out of its current block into a successor.
 bool MachineSinking::SinkInstruction(MachineInstr &MI, bool &SawStore,
                                      AllSuccsCache &AllSuccessors) {
   // Don't sink instructions that the target prefers not to sink.
   if (!TII->shouldSink(MI))
     return false;
 
   // Check if it's safe to move the instruction.
   if (!MI.isSafeToMove(AA, SawStore))
     return false;
 
   // Convergent operations may not be made control-dependent on additional
   // values.
   if (MI.isConvergent())
     return false;
 
+  // Sink TCOPY instructions after register allocation to avoid mucking with
+  // live-ins.
+  if (MI.getOpcode() == TargetOpcode::TCOPY)
+    return false;
+
   // Don't break implicit null checks.  This is a performance heuristic, and not
   // required for correctness.
   if (SinkingPreventsImplicitNullCheck(MI, TII, TRI))
     return false;
 
   // FIXME: This should include support for sinking instructions within the
   // block they are currently in to shorten the live ranges.  We often get
   // instructions sunk into the top of a large block, but it would be better to
bool MachineSinking::SinkInstruction(MachineInstr &MI, bool &SawStore,
   }
 
   // After sinking, some debug users may not be dominated any more. If possible,
   // copy-propagate their operands. As it's expensive, don't do this if there's
   // no debuginfo in the program.
   if (MI.getMF()->getFunction().getSubprogram() && MI.isCopy())
     SalvageUnsunkDebugUsersOfCopy(MI, SuccToSinkTo);
 
-  performSink(MI, *SuccToSinkTo, InsertPos, DbgUsersToSink);
+  performSink(MI, *SuccToSinkTo, InsertPos, TII, DbgUsersToSink);
 
   // Conservatively, clear any kill flags, since it's possible that they are no
   // longer correct.
   // Note that we have to clear the kill flags for any register this instruction
   // uses as we may sink over another instruction which currently kills the
   // used registers.
   for (MachineOperand &MO : MI.operands()) {
     if (MO.isReg() && MO.isUse())
for (auto I = CurBB.rbegin(), E = CurBB.rend(); I != E;) {
     }
     DbgValsToSink.insert(DbgValsToSink.begin(), DbgValsToSinkSet.begin(),
                          DbgValsToSinkSet.end());
 
     // Clear the kill flag if SrcReg is killed between MI and the end of the
     // block.
     clearKillFlags(MI, CurBB, UsedOpsInCopy, UsedRegUnits, TRI);
     MachineBasicBlock::iterator InsertPos = SuccBB->getFirstNonPHI();
-    performSink(*MI, *SuccBB, InsertPos, DbgValsToSink);
+    performSink(*MI, *SuccBB, InsertPos, TII, DbgValsToSink);
     updateLiveIn(MI, SuccBB, UsedOpsInCopy, DefedRegsInCopy);
 
     Changed = true;
     ++NumPostRACopySink;
   }
+
   return Changed;
 }
 
 bool PostRAMachineSinking::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   bool Changed = false;
   const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
   const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo();
 
   ModifiedRegUnits.init(*TRI);
   UsedRegUnits.init(*TRI);
   for (auto &BB : MF)
     Changed |= tryToSinkCopy(BB, MF, TRI, TII);
 
   return Changed;
 }

llvm/lib/CodeGen/MachineVerifier.cpp

for (const auto *succ : MBB->successors()) {
     if (!MBBInfoMap[succ].Preds.count(MBB)) {
       report("Inconsistent CFG", MBB);
       errs() << "MBB is not in the predecessor list of the successor "
              << printMBBReference(*succ) << ".\n";
     }
   }
 
   // Count the number of INLINEASM_BR indirect target successors.
   SmallPtrSet<const MachineBasicBlock*, 4> IndirectTargetSuccs;
   for (const auto *succ : MBB->successors()) {
     if (MBB->isInlineAsmBrIndirectTarget(succ))
       IndirectTargetSuccs.insert(succ);
     if (!FunctionBlocks.count(succ))
       report("MBB has successor that isn't part of the function.", MBB);
     if (!MBBInfoMap[succ].Preds.count(MBB)) {
       report("Inconsistent CFG", MBB);
       errs() << "MBB is not in the predecessor list of the successor "
              << printMBBReference(*succ) << ".\n";
if (Indexes && Indexes->hasIndex(*MI)) {
     SlotIndex idx = Indexes->getInstructionIndex(*MI);
     if (!(idx > lastIndex)) {
       report("Instruction index out of order", MI);
       errs() << "Last instruction was at " << lastIndex << '\n';
     }
     lastIndex = idx;
   }
 
-  // Ensure non-terminators don't follow terminators.
-  // Ignore predicated terminators formed by if conversion.
+  // Ensure non-terminators don't follow terminators. Ignore predicated
+  // terminators formed by if conversion, and stack dumps after a terminator at
+  // -O0. These are most likely from a TCOPY.
+  //
   // FIXME: If conversion shouldn't need to violate this rule.
-  if (MI->isTerminator() && !TII->isPredicated(*MI)) {
+  if ((MI->isTerminator() && !TII->isPredicated(*MI)) ||
+      (TM->getOptLevel() == CodeGenOpt::None && MI->mayStore())) {
     if (!FirstTerminator)
       FirstTerminator = MI;
   } else if (FirstTerminator && !MI->isDebugEntryValue()) {
     report("Non-terminator instruction after the first terminator", MI);
     errs() << "First terminator was:\t" << *FirstTerminator;
   }
 }
 
 // The operands on an INLINEASM instruction must follow a template.
 // Verify that the flag operands make sense.
 void MachineVerifier::verifyInlineAsm(const MachineInstr *MI) {
   // The first two operands on INLINEASM are the asm string and global flags.
   if (MI->getNumOperands() < 2) {
     report("Too few operands on inline asm", MI);
     return;
   }

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

EmitCopyFromReg(SDNode *Node, unsigned ResNo, bool IsClone, bool IsCloned,
 
   // If all uses are reading from the src physical register and copying the
   // register is either impossible or very expensive, then don't create a copy.
   if (MatchReg && SrcRC->getCopyCost() < 0) {
     VRBase = SrcReg;
   } else {
     // Create the reg, emit the copy.
     VRBase = MRI->createVirtualRegister(DstRC);
-    BuildMI(*MBB, InsertPos, Node->getDebugLoc(), TII->get(TargetOpcode::COPY),
+    const MachineBasicBlock::iterator Term = MBB->getFirstTerminator();
+    unsigned TgtOpc =
+        Term != MBB->end() && Term->getOpcode() == TargetOpcode::INLINEASM_BR
+        ? TargetOpcode::TCOPY
+        : TargetOpcode::COPY;
+    BuildMI(*MBB, InsertPos, Node->getDebugLoc(), TII->get(TgtOpc),
             VRBase).addReg(SrcReg);
   }
 
   SDValue Op(Node, ResNo);
   if (IsClone)
     VRBaseMap.erase(Op);
   bool isNew = VRBaseMap.insert(std::make_pair(Op, VRBase)).second;
   (void)isNew; // Silence compiler warning.

llvm/lib/CodeGen/SelectionDAG/ScheduleDAGSDNodes.cpp

for (const auto &InstrOrder : Orders) {
 
       LastOrder = Order;
     }
   }
 
   // Split after an INLINEASM_BR block with outputs. This allows us to keep the
   // copy to/from register instructions from being between two terminator
   // instructions, which causes the machine instruction verifier agita.
-  auto TI = llvm::find_if(*BB, [](const MachineInstr &MI){
-    return MI.getOpcode() == TargetOpcode::INLINEASM_BR;
+  auto Iter = llvm::find_if(*BB, [](MachineInstr &MI){
+    MachineBasicBlock::iterator Next(MI);
+    Next = std::next(Next);
+    return Next != MI.getParent()->end() &&
+        (MI.getOpcode() == TargetOpcode::TCOPY ||
+         MI.getOpcode() == TargetOpcode::INLINEASM_BR) &&
+        Next->getOpcode() == TargetOpcode::COPY;
   });
-  auto SplicePt = TI != BB->end() ? std::next(TI) : BB->end();
-  if (TI != BB->end() && SplicePt != BB->end() &&
-      TI->getOpcode() == TargetOpcode::INLINEASM_BR &&
-      SplicePt->getOpcode() == TargetOpcode::COPY) {
-    MachineBasicBlock *FallThrough = BB->getFallThrough();
-    if (!FallThrough)
+  if (Iter != BB->end()) {
+    MachineBasicBlock *DefaultTarget = BB->getFallThrough();
+    if (!DefaultTarget)
       for (const MachineOperand &MO : BB->back().operands())
         if (MO.isMBB()) {
-          FallThrough = MO.getMBB();
+          DefaultTarget = MO.getMBB();
           break;
         }
-    assert(FallThrough && "Cannot find default dest block for callbr!");
+    assert(DefaultTarget && "Cannot find default dest block for callbr!");
 
     MachineBasicBlock *CopyBB = MF.CreateMachineBasicBlock(BB->getBasicBlock());
     MachineFunction::iterator BBI(*BB);
     MF.insert(++BBI, CopyBB);
 
-    CopyBB->splice(CopyBB->begin(), BB, SplicePt, BB->end());
+    CopyBB->splice(CopyBB->begin(), BB, std::next(Iter), BB->end());
     CopyBB->setInlineAsmBrDefaultTarget();
 
-    CopyBB->addSuccessor(FallThrough, BranchProbability::getOne());
+    CopyBB->addSuccessor(DefaultTarget, BranchProbability::getOne());
     BB->addSuccessor(CopyBB, BranchProbability::getOne());
 
-    // Mark all physical registers defined in the original block as being live
-    // on entry to the copy block.
-    for (const auto &MI : *CopyBB)
-      for (const MachineOperand &MO : MI.operands())
-        if (MO.isReg()) {
-          Register reg = MO.getReg();
-          if (Register::isPhysicalRegister(reg)) {
-            CopyBB->addLiveIn(reg);
-            break;
-          }
-        }
-
     // Bit of a hack: The copy block we created here exists only because we want
     // the CFG to work with the current system. However, the successors to the
     // block with the INLINEASM_BR instruction expect values to come from *that*
     // block, not this usurper block. Thus we steal its successors and add them
     // to the copy so that everyone is happy.
     for (auto *Succ : BB->successors())
       if (Succ != CopyBB && !CopyBB->isSuccessor(Succ))
         CopyBB->addSuccessor(Succ, BranchProbability::getZero());

llvm/test/CodeGen/AArch64/callbr-asm-label.ll

 ; RUN: llc < %s -mtriple=aarch64-linux-gnu | FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-linux-gnu -O0 | FileCheck %s --check-prefix=CHECK-O0
 
 @X = common local_unnamed_addr global i32 0, align 4
 
 define i32 @test1() {
 ; CHECK-LABEL: test1:
 ; CHECK:         .word b
 ; CHECK-NEXT:    .word .Ltmp0
 ; CHECK-LABEL: .Ltmp0:
 ; CHECK-LABEL: .LBB0_1: // %l_yes
 ; CHECK-LABEL: .LBB0_2: // %cleanup
+
+; CHECK-O0-LABEL:  test1:
+; CHECK-O0:          .word b
+; CHECK-O0-NEXT:     .word .Ltmp1
+; CHECK-O0-LABEL: .Ltmp1:
+; CHECK-O0-LABEL: .LBB0_1: // %l_yes
+; CHECK-O0-LABEL: .LBB0_2: // %cleanup
 entry:
   callbr void asm sideeffect "1:\0A\09.word b, ${0:l}\0A\09", "X"(i8* blockaddress(@test1, %l_yes))
           to label %cleanup [label %l_yes]
 
 l_yes:
   br label %cleanup
 
 cleanup:
   %retval.0 = phi i32 [ 1, %l_yes ], [ 0, %entry ]
   ret i32 %retval.0
 }
 
 define void @test2() {
-; CHECK-LABEL: test2:
+; CHECK-LABEL:    test2:
+; CHECK-O0-LABEL: test2:
 entry:
   %0 = load i32, i32* @X, align 4
   %and = and i32 %0, 1
   %tobool = icmp eq i32 %and, 0
   br i1 %tobool, label %if.end10, label %if.then
 
 if.then:
 ; CHECK:       .word b
 ; CHECK-NEXT:  .word .Ltmp2
 ; CHECK-LABEL: .Ltmp2:
 ; CHECK-NEXT:  .LBB1_3: // %if.end6
+
+; CHECK-O0:       .word b
+; CHECK-O0-NEXT:  .word .Ltmp3
+; CHECK-O0-LABEL: .Ltmp3:
+; CHECK-O0-NEXT:  .LBB1_3: // %if.end6
   callbr void asm sideeffect "1:\0A\09.word b, ${0:l}\0A\09", "X"(i8* blockaddress(@test2, %if.end6))
           to label %if.then4 [label %if.end6]
 
 if.then4:
   %call5 = tail call i32 bitcast (i32 (...)* @g to i32 ()*)()
   br label %if.end6
 
 if.end6:
   %.pre = load i32, i32* @X, align 4
   %.pre13 = and i32 %.pre, 1
   %phitmp = icmp eq i32 %.pre13, 0
   br i1 %phitmp, label %if.end10, label %if.then9
 
 if.then9:
 ; CHECK-LABEL: .Ltmp4:
 ; CHECK-NEXT:  .LBB1_5: // %l_yes
+
+; CHECK-O0-LABEL:  .Ltmp5:
+; CHECK-O0-NEXT:   .LBB1_6: // %l_yes
   callbr void asm sideeffect "", "X"(i8* blockaddress(@test2, %l_yes))
           to label %if.end10 [label %l_yes]
 
 if.end10:
   br label %l_yes
 
 l_yes:
   ret void
 }
 
 declare i32 @g(...)

llvm/test/CodeGen/SystemZ/asm-20.ll

 ; Test that asm goto can be compiled.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 -O0
 
 define i32 @c() {
 entry:
   callbr void asm sideeffect "j d", "X"(i8* blockaddress(@c, %d))
           to label %asm.fallthrough [label %d]
 
 asm.fallthrough:               ; preds = %entry
   br label %d
 
 d:                             ; preds = %asm.fallthrough, %entry
   ret i32 undef
 }

llvm/test/CodeGen/X86/callbr-asm-label-addr.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -O0 | FileCheck %s --check-prefix=CHECK-O0
 
 define i32 @test1(i32 %x) {
 ; CHECK-LABEL: test1:
 ; CHECK:         .quad .Ltmp0
 ; CHECK-NEXT:    .quad .Ltmp1
 ; CHECK-LABEL: .Ltmp1:
 ; CHECK-LABEL: .LBB0_1: # %bar
 ; CHECK-NEXT:    callq foo
 ; CHECK-LABEL: .Ltmp0:
 ; CHECK-NEXT:  # %bb.2: # %baz
+
+; CHECK-O0-LABEL: test1:
+; CHECK-O0:         .quad .Ltmp0
+; CHECK-O0-NEXT:    .quad .Ltmp1
+; CHECK-O0-LABEL: .Ltmp1:
+; CHECK-O0-LABEL: .LBB0_2: # %bar
+; CHECK-O0-NEXT:    movl
+; CHECK-O0-NEXT:    callq foo
+; CHECK-O0-LABEL: .Ltmp0:
+; CHECK-O0-NEXT:  # %bb.3: # %baz
 entry:
   callbr void asm sideeffect ".quad ${0:l}\0A\09.quad ${1:l}", "i,X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test1, %baz), i8* blockaddress(@test1, %bar))
           to label %asm.fallthrough [label %bar]
 
 asm.fallthrough:
   br label %bar
 
 bar: