Add TCOPY, a terminator form of the COPY instr

[void]

Summary

The INLINEASM_BR instruction's a terminator, and the code generator
doesn't allow non-terminator instructions after a terminator. This is an
issue when an INLINEASM_BR defines a physical register. We can't place
the copies of the physical registers into virtual registers in the
fallthrough block because physical registers can't be marked as
"live-in" until after register allocation.

To get around this issue, we introduce a new pseudo-instruction, TCOPY,
that's identical to the COPY instruction, but is a terminator. With it
we're able to copy the physical registers to virtual registers without
needing to place the copies in a fallthrough block:

bb.1:
  INLINEASM_BR &"" ..., implicit-def $esi, $1:[regdef],...
  %9:gr32 = TCOPY $esi

bb.2:
; predecessors: %bb.1
  successors: %bb.3(0x80000000); %bb.3(100.00%)

  %0:gr32 = COPY %9:gr32
  JMP_1 %bb.3

The TCOPY is converted to a normal COPY after register allocation, when
we have live variable information and live-ins are allowed on basic
blocks.

The fast register allocator behaves a bit differently because everything
is spilled before the end of a basic block. Therefore, we allow a store
of a physical register after the INLINEASM_BR when optimizations are
disabled.

[arsenm]

This will be useful for AMDGPU, we currently have a set of _term mov instructions for this purpose.

llvm/include/llvm/Support/TargetOpcodes.def

101

s/terminal/terminator

llvm/include/llvm/Target/Target.td

1131–1132

This shouldn't need isBranch or isIndirectBranch

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

179

I would expect this to not special case INLINEASM_BR. This should be any value defined by a terminator instruction

[arsenm]

llvm/include/llvm/Support/TargetOpcodes.def

101

COPY_TERM or COPY_TERMINATOR would be better since it isn't really a branch

[nickdesaulniers]

llvm/include/llvm/CodeGen/MachineInstr.h

1132

Does this save a few getOpcode() calls? The rest of this CL explicitly compares the opcode against COPY_BR? I just worry if this might mess up existing callsites of isCopy when getOpcode() == TargetOpcode::COPY_BR;.

llvm/include/llvm/Target/Target.td

1133

COPY also sets hasNoSchedulingInfo to 0. Should COPY_BR do that as well?

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

198

what? if we could use a range based for loop for the MBB's, surely we can for MI.

llvm/lib/CodeGen/MachineSink.cpp

1395

Does it make sense to "sink" register info invalidation into performSink()? (pun intended) Since it's already checking the MI's opcode? Or are the two call sites of performSink() problematic? The implementation of invalidateLiveness() looks pretty cheap, IMO.

llvm/lib/CodeGen/MachineVerifier.cpp

829

this is hard to read. Is there a nice way to simply this? Maybe negate, and fold into parent if?

[void]

llvm/include/llvm/CodeGen/MachineInstr.h

1132

It's more than just a few. isCopy() is used pretty extensively. I want the COPY_BR (or COPY_TERM) to be exactly like a normal COPY, with the only exception that it can come after a terminator.

llvm/include/llvm/Support/TargetOpcodes.def

101

How about TCOPY?

llvm/include/llvm/Target/Target.td

1133

Possibly, though I had trouble finding where to place its scheduling info. It seems like COPY is special and TableGen adds it by default. I couldn't find the place to do that for COPY_BR, but I'll take another look.

[void]

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

198

The ++mi below made me worried, and the fact that MI could be erased.

llvm/lib/CodeGen/MachineSink.cpp

1395

Possibly. If we're going to make COPY_TERM (or whatever name we settle on) not specific to INLINEASM_BR, then we'll probably need to modify this to accept any case where we sink copies after a terminator. We'll come back to this after other comments settle.

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

179

Does this include unconditional jump terminators?

[arsenm]

llvm/include/llvm/Target/Target.td

1133

Last time I looked at adding a copy variant, the right solution was just isAsCheapAsAMove = 1. Otherwise it would have required adding the COPY_BR to every target's COPY handling

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

179

I'm not sure how you could construct a sensible unconditional jump that would require a copy after it. Seems like something to check in the verifier

[nickdesaulniers]

Also, if INLINEASM_BR is a terminator, but we want TCOPY to be a terminator, should this patch make INLINEASM_BR NOT a terminator? Or would that be a follow up patch after creating TCOPY?

Also, none of the tests have TCOPY (or w/e). I feel like any patch adding a new instructions at w/e level of IR should have some test that the instruction exists.

Also, I don't quite follow what the changes to the tests are testing for with this change.

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

198

Ah sorry, I missed that. I'm not sure what's meant by erased, but I'd be curious if a continue could not be used instead of manual iterator advancement.

[void]

In D75098#1892171, @nickdesaulniers wrote:

Also, if INLINEASM_BR is a terminator, but we want TCOPY to be a terminator, should this patch make INLINEASM_BR NOT a terminator? Or would that be a follow up patch after creating TCOPY?

INLINEASM_BR being a terminator makes a lot of things "just work" for the code generator. I'm not 100% convinced that it *should* be a terminator, but if we decided to make it a normal instruction, then I think we'd have to touch way too much code to make it worthwhile.

Also, none of the tests have TCOPY (or w/e). I feel like any patch adding a new instructions at w/e level of IR should have some test that the instruction exists.

The tests are the current "asm goto with outputs" tests.

Also, I don't quite follow what the changes to the tests are testing for with this change.

The test changes are testing that we can compile at -O0.

[void]

Don't mark TCOPY as a branch. Also s/COPY_BR/TCOPY/.

[void]

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

179

My comment wasn't very good. When converting something to a TCOPY, we need to figure out which terminators are candidates for having a TCOPY after them. Or perhaps better, what condition requires a TCOPY instead of a regular COPY.

[void]

Liveness is handled by the PostRA machine sink pass, so there's no need to
invalidate it. Also simplify the logic in the machine verifier that allows
stack dumps after a terminator.

[nickdesaulniers]

llvm/lib/CodeGen/MachineSink.cpp

841

pass as const&, or make performSink a private method.

[void]

Fix accidental change inclusion in the verifier.

[void]

llvm/lib/CodeGen/MachineSink.cpp

841

Two different passes use this function so it can't be made a private method. I"m not sure why passing it as "const&" is better than a "const*"...

[void]

Reformat and satisfy clang-tidy.

[nickdesaulniers]

llvm/lib/CodeGen/MachineSink.cpp

841

Generally, passing by const reference indicates that a parameter is strictly an input, as opposed to both input AND output, which is why you don't see a mix of pointers and references in this function signature.

[void]

llvm/lib/CodeGen/MachineSink.cpp

841

A 'const*' doesn't allow modifications either. Also note that none of the references in this function signature are 'const', and are probably passed by reference because they aren't pointers (SuccToSinkto notwithstanding) in the originating function. Converting this to a reference is not useful.

[arsenm]

llvm/include/llvm/CodeGen/MachineInstr.h

1132

I think this is potentially hazardous. I'm worried about the peephole optimizer doing things like folding a copy into a tcopy and losing the terminator bit. Can you add some testcases with coalescable tcopy pairs for PeepholeOptimizer and the register coalescer?

[void]

• Add a verification step to ensure a COPY doesn't follow a TCOPY.
• Comment that the predicate for determining whether to use a TCOPY or COPY should be improved, but at this time we don't have enough information to determine the best criteria.

[void]

llvm/include/llvm/CodeGen/MachineInstr.h

1132

I'll craft some tests, though note that TCOPY should only be after a terminator, so a COPY should never be merged with it. (I'll add a verifier check to ensure that TCOPY doesn't happen before a terminator.)

[void]

Add TCOPY to some switch statements that handle COPY

[void]

llvm/include/llvm/CodeGen/MachineInstr.h

1132

@arsenm I'm struggling a bit to come up with tests that will exercise the peephole optimizer and register coalescer. Do you have any advice on how to do this?

[void]

Rebase and update testcase.

[void]

Friendly ping. :-)

[void]

Another friendly ping. :-)

[arsenm]

llvm/include/llvm/CodeGen/MachineInstr.h

1132

You shouldn't need a special verifier check, it should be caught by the normal terminator before non-terminator check.

I mean something like
%1 = COPY %0
%2 = TCOPY %1

and then maybe mix in some subregistsers? The worry is the COPY will somehow replace the TCOPY

[void]

llvm/include/llvm/CodeGen/MachineInstr.h

1132

This isn't an issue as far as I can see (at least the kind of example you're referring to). A TCOPY can become a COPY if appropriate. The only time it can't become a COPY is if it occurs after a terminator, in which case it should be caught by current verifier checks. I *don't* think a COPY should become a TCOPY though.

[void]

• Add support to MIR parsing.
• Rebase current change.

[arsenm]

llvm/lib/CodeGen/MIRParser/MIRParser.cpp

312–342 ↗

(On Diff #261771)

This is an unrelated change

llvm/lib/CodeGen/MachineSink.cpp

858–861

I don't see this captured in a test?

llvm/lib/CodeGen/MachineVerifier.cpp

615–616

This also looks like a separate change

[qcolombet]

I don't think introducing the TCOPY opcode is a valid solution.
Let's assume the copy doesn't get coalesced, that means it needs to be executed after the inlineasm branch. And this is not going to happen if it sits in the same basic block right after the jump.

What happens if you put the copy at the beginning of each successor?
You mentioned that this is not possible because the live-ins wouldn't be set, but I would expect that this information would be correctly computed when building the liveness information. I.e., I don't expect this to be a problem.

we allow a store of a physical register after the INLINEASM_BR when optimizations are disabled.

That sounds wrong.

Could we step back a little bit, what is the semantic of INLINEASM_BR?

[void]

In D75098#2017980, @qcolombet wrote:

I don't think introducing the TCOPY opcode is a valid solution.
Let's assume the copy doesn't get coalesced, that means it needs to be executed after the inlineasm branch. And this is not going to happen if it sits in the same basic block right after the jump.

What happens if you put the copy at the beginning of each successor?
You mentioned that this is not possible because the live-ins wouldn't be set, but I would expect that this information would be correctly computed when building the liveness information. I.e., I don't expect this to be a problem.

we allow a store of a physical register after the INLINEASM_BR when optimizations are disabled.

That sounds wrong.

Could we step back a little bit, what is the semantic of INLINEASM_BR?

[I'm sorry if some of this is basic. I just want to give a clear description of what's going on and why.]

INLINEASM_BR is an attempt to replicate the behavior of "asm goto". As such, it has one default destination (more-or-less the fallthrough block), and one or more indirect destinations. If the indirect destinations aren't taken, the resulting control flow is to fall out of the bottom of the ASM block. The callbr instruction is a terminator so that it can model this behavior as best as it can (which isn't all that great with LLVM's IR, but doable (see "invoke")). Once it's converted to INLINEASM_BR the behavior obviously doesn't change, but now that we have outputs, we need to figure out how to handle them given the constraints of MIR (i.e. you can't have non-terminals after a terminal, and live-ins are only expected to be correct after register allocation).

Because of how callbr is modeled, INLINEASM_BR is also a terminator. So during ISEL the values that are defined by INLINEASM_BR need to be spilled before jumping to the default block. Because of the constraint against having non-terminators after a terminator, we artificially modify the code after ISEL's finished building the block, placing any spills into a "copy" block (that's between the INLINEASM_BR and its default destination), and adding any registers defined by INLINEASM_BR as "live-ins" to the copy block. This is bad, because it violates the constraints, but was necessary at the time because we had no way to avoid it.

[Note that the outputs from an INLINEASM_BR are valid only on the default branch. It's very difficult to have them on the indirect branches without making the resulting code bad---both in style and in performance.]

Okay, so that's the behavior of INLINEASM_BR and some of the issues we came across that led us to here. I think TCOPY is a good solution because the default behavior of INLINEASM_BR is to fall out the bottom of the ASM block, so it would "naturally" execute the TCOPY instructions. And since we only ensure that the values are valid on the default branch, the COPYs will always be executed when needed (note a block containing an INLINEASM_BR should either exit to the default via a fallthrough or unconditional jump). It's still possible to split the block directly after the INLINEASM_BR and convert the TCOPYs into regular COPYs, in fact I would encourage us to do that at the appropriate point. It could remove a lot of the code in ScheduleDAGSDNodes.cpp.

What happens if you put the copy at the beginning of each successor?

I found that this didn't work. CodeGen wants registers to be spilled at the end of a block. The spill slot is recorded and used later on when accessing the value. If we just add a COPY into the default block it won't have the necessary information available to load the correct value. We would have to add the information as a "live-in" on the default block, but that's a constraint violation, wash-rinse-repeat.

[qcolombet]

Thanks Bill for the detailed explanations. A couple more questions.

CodeGen wants registers to be spilled at the end of a block.

That's fast reg alloc only. What is happening with the other allocators?

The spill slot is recorded and used later on when accessing the value. If we just add a COPY into the default block it won't have the necessary information available to load the correct value.

Why is that? (Why doesn't it have the necessary information to load the correct value.)

I feel that we are trying to work around a bug/limitation in the fast register allocator where physreg are assumed not to cross basic block boundaries. I wonder if it wouldn't be easier to fix fast reg alloc if that is the problem.

Out-of-curiosity, how do we deal with invoke? I would expect we have the exact same problems.

[void]

In D75098#2021573, @qcolombet wrote:

Thanks Bill for the detailed explanations. A couple more questions.

CodeGen wants registers to be spilled at the end of a block.

That's fast reg alloc only. What is happening with the other allocators?

They don't appear to use the correct registers for the defined values.

The spill slot is recorded and used later on when accessing the value. If we just add a COPY into the default block it won't have the necessary information available to load the correct value.

Why is that? (Why doesn't it have the necessary information to load the correct value.)

I feel that we are trying to work around a bug/limitation in the fast register allocator where physreg are assumed not to cross basic block boundaries. I wonder if it wouldn't be easier to fix fast reg alloc if that is the problem.

It's not so much working around a bug/limitation in any of the register allocators. What I'm most concerned about is having to specify live-ins to MBBs before register allocation. That invariant was a huge sticking point in the review process for this feature. We're currently violating it because it "seems to work", but I don't want to rely on that.

Because of the live-ins restriction, all live physical registers at the end of an MBB need to be "spilled", or at least recorded in a way so that other blocks can access the information without having their live-ins set. (This is my understanding, it may be inaccurate). The SelectionDAGBuilder::CopyToExportRegsIfNeeded() function is one of the ways this is done.

Out-of-curiosity, how do we deal with invoke? I would expect we have the exact same problems.

Invoke is turned into a regular call with exception handling stuff around it. E.g.:

$ cat ex.cpp:
void g();
int f() {
  try {
    g();
  } catch (int&) {
    return 37;
  }
  return 1;
}

$ clang++ -mllvm -print-after-all ex.cpp -c -o /dev/null
...
bb.0 (%ir-block.0):
  successors: %bb.1, %bb.2

  EH_LABEL <mcsymbol .Ltmp0>
  ADJCALLSTACKDOWN64 0, 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
  CALL64pcrel32 @_Z1gv, <regmask $bh $bl $bp $bph $bpl $bx $ebp $ebx $hbp $hbx $rbp $rbx $r12 $r13 $r14 $r15 $r12b $r13b $r14b $r15b $r12bh $r13bh $r14bh $r15bh $r12d $r13d $r14d $r15d $r12w $r13w $r14w $r15w $r12wh and 3 more...>, implicit $rsp, implicit $ssp, implicit-def $rsp, implicit-def $ssp
  ADJCALLSTACKUP64 0, 0, implicit-def dead $rsp, implicit-def dead $eflags, implicit-def dead $ssp, implicit $rsp, implicit $ssp
  EH_LABEL <mcsymbol .Ltmp1>
  JMP_1 %bb.1
...

[void]

Remove some of the hackery in the SDNode scheduler that splits below an
INLINEASM_BR instruction. It's not needed with TCOPY.

[void]

Allow a TCOPY to sink, since it's exactly like a COPY.

[void]

llvm/lib/CodeGen/MachineSink.cpp

858–861

This will happen now that we correctly mark TCOPY as sinkable. I'll see if I can craft an MIR test to explicitly do this.

[void]

Spill TCOPY in FastRA right before the TCOPY instr instead of before the
terminators.

[void]

Fix formatting and clang-tidy warnings.

[void]

Revert bad formatting that sneaked (snuck?) in.

[void]

In D75098#2026425, @void wrote:

In D75098#2021573, @qcolombet wrote:

Thanks Bill for the detailed explanations. A couple more questions.

CodeGen wants registers to be spilled at the end of a block.

That's fast reg alloc only. What is happening with the other allocators?

They don't appear to use the correct registers for the defined values.

I should probably explain a bit better. With a normal INLINEASM instruction, any defined registers are spilled directly afterwards. This is one reason why I want to do the same for INLINEASM_BR with the TCOPY. There could be other ways to achieve the same thing, but I think it would involve placing a larger burden on the code generation passes than needs be.

[jyknight]

I think the better fix here is to make INLINEASM_BR _not_ be a terminator -- just like the CALL involved in an invoke is not.

That change turned out to be a bit complicated, and while I had started taking a stab at that a couple months ago, I got distracted by many other things going on...However, I've now dusted off that experiment and got it into a useful state, and will send out a review for this tomorrow, so we can consider both the options.

[void]

In D75098#2030777, @jyknight wrote:

I think the better fix here is to make INLINEASM_BR _not_ be a terminator -- just like the CALL involved in an invoke is not.

That change turned out to be a bit complicated, and while I had started taking a stab at that a couple months ago, I got distracted by many other things going on...However, I've now dusted off that experiment and got it into a useful state, and will send out a review for this tomorrow, so we can consider both the options.

I've thought about the same thing. I'm right now running up against an issue that Nick pointed out. Having a non-branch as a terminator works until the register allocator wants to spill. Then it could insert a copy where the TCOPY is, which messes up analyzeBranch. I'm interested to see your patch.

[arsenm]

In D75098#2030777, @jyknight wrote:

I think the better fix here is to make INLINEASM_BR _not_ be a terminator -- just like the CALL involved in an invoke is not.

That change turned out to be a bit complicated, and while I had started taking a stab at that a couple months ago, I got distracted by many other things going on...However, I've now dusted off that experiment and got it into a useful state, and will send out a review for this tomorrow, so we can consider both the options.

In D75098#2032980, @void wrote:

In D75098#2030777, @jyknight wrote:

I think the better fix here is to make INLINEASM_BR _not_ be a terminator -- just like the CALL involved in an invoke is not.

That change turned out to be a bit complicated, and while I had started taking a stab at that a couple months ago, I got distracted by many other things going on...However, I've now dusted off that experiment and got it into a useful state, and will send out a review for this tomorrow, so we can consider both the options.

I've thought about the same thing. I'm right now running up against an issue that Nick pointed out. Having a non-branch as a terminator works until the register allocator wants to spill. Then it could insert a copy where the TCOPY is, which messes up analyzeBranch. I'm interested to see your patch.

Regardless of what happens to INLINEASM_BR, I think we still need a terminator copy

[qcolombet]

Regardless of what happens to INLINEASM_BR, I think we still need a terminator copy

What for?
The whole concept is bogus to me because if the previous instructions are really terminators (as in they branch to somewhere else), then this copy would never be executed.

[arsenm]

In D75098#2049787, @qcolombet wrote:

Regardless of what happens to INLINEASM_BR, I think we still need a terminator copy

What for?
The whole concept is bogus to me because if the previous instructions are really terminators (as in they branch to somewhere else), then this copy would never be executed.

Terminator does not imply a branch, it only means glued to the end of the block. It's the only way to get correct spill code placement around exec mask writes. We need a copy to the exec mask, but spill code needs to be inserted before that point. Currently we get unnecessary copies since we use a raw move instructions for this purpose, when we could have coalesced a copy

[void]

In D75098#2049787, @qcolombet wrote:

Regardless of what happens to INLINEASM_BR, I think we still need a terminator copy

What for?
The whole concept is bogus to me because if the previous instructions are really terminators (as in they branch to somewhere else), then this copy would never be executed.

That's not necessarily so, because you can have multiple branching terminators then there must be some that branch conditionally (allowing other code to exist between the jumps), otherwise there wouldn't be a need for multiple terminators.

[arsenm]

What's the status of this? I'm still interested in terminator copies

[void]

In D75098#2170382, @arsenm wrote:

What's the status of this? I'm still interested in terminator copies

When we last left our adventurers, there was an issue with live range splitting. In some cases it could emit a spill after terminators, which failed validation.

llvm/include/llvm/CodeGen/MachineBasicBlock.h

private:
   MBBSectionID SectionID{0};
 
   // Indicate that this basic block begins a section.
   bool IsBeginSection = false;
 
   // Indicate that this basic block ends a section.
   bool IsEndSection = false;
 
-  /// Default target of the callbr of a basic block.
-  bool InlineAsmBrDefaultTarget = false;
-
   /// List of indirect targets of the callbr of a basic block.
   SmallPtrSet<const MachineBasicBlock*, 4> InlineAsmBrIndirectTargets;
 
   /// since getSymbol is a relatively heavy-weight operation, the symbol
   /// is only computed once and is cached.
   mutable MCSymbol *CachedMCSymbol = nullptr;
 
   /// Used during basic block sections to mark the end of a basic block.
bool isInlineAsmBrIndirectTarget(const MachineBasicBlock *Tgt) const {
     return InlineAsmBrIndirectTargets.count(Tgt);
   }
 
   /// Indicates if this is the indirect dest of an INLINEASM_BR.
   void addInlineAsmBrIndirectTarget(const MachineBasicBlock *Tgt) {
     InlineAsmBrIndirectTargets.insert(Tgt);
   }
 
-  /// Transfers indirect targets to INLINEASM_BR's copy block.
-  void transferInlineAsmBrIndirectTargets(MachineBasicBlock *CopyBB) {
-    for (auto *Target : InlineAsmBrIndirectTargets)
-      CopyBB->addInlineAsmBrIndirectTarget(Target);
-    return InlineAsmBrIndirectTargets.clear();
-  }
-
-  /// Returns true if this is the default dest of an INLINEASM_BR.
-  bool isInlineAsmBrDefaultTarget() const {
-    return InlineAsmBrDefaultTarget;
-  }
-
-  /// Indicates if this is the default deft of an INLINEASM_BR.
-  void setInlineAsmBrDefaultTarget() {
-    InlineAsmBrDefaultTarget = true;
-  }
+  /// Returns the default destination of an INLINEASM_BR instruction.
+  MachineBasicBlock *getInlineAsmBrDefaultTarget();
 
   /// Returns true if it is legal to hoist instructions into this block.
   bool isLegalToHoistInto() const;
 
   // Code Layout methods.
 
   /// Move 'this' block before or after the specified block.  This only moves
   /// the block, it does not modify the CFG or adjust potential fall-throughs at

llvm/include/llvm/CodeGen/MachineInstr.h

bool isRegSequence() const {
     return getOpcode() == TargetOpcode::REG_SEQUENCE;
   }
 
   bool isBundle() const {
     return getOpcode() == TargetOpcode::BUNDLE;
   }
 
   bool isCopy() const {
-    return getOpcode() == TargetOpcode::COPY;
+    return getOpcode() == TargetOpcode::COPY ||
+           getOpcode() == TargetOpcode::TCOPY;
   }
 
   bool isFullCopy() const {
     return isCopy() && !getOperand(0).getSubReg() && !getOperand(1).getSubReg();
   }
 
   bool isExtractSubreg() const {
     return getOpcode() == TargetOpcode::EXTRACT_SUBREG;

llvm/include/llvm/Support/TargetOpcodes.def

 HANDLE_TARGET_OPCODE(SUBREG_TO_REG)
 
 /// COPY_TO_REGCLASS - This instruction is a placeholder for a plain
 /// register-to-register copy into a specific register class. This is only
 /// used between instruction selection and MachineInstr creation, before
 /// virtual registers have been created for all the instructions, and it's
 /// only needed in cases where the register classes implied by the
 /// instructions are insufficient. It is emitted as a COPY MachineInstr.
-  HANDLE_TARGET_OPCODE(COPY_TO_REGCLASS)
 
 /// DBG_VALUE - a mapping of the llvm.dbg.value intrinsic
 HANDLE_TARGET_OPCODE(DBG_VALUE)
 
 /// DBG_LABEL - a mapping of the llvm.dbg.label intrinsic
 HANDLE_TARGET_OPCODE(DBG_LABEL)
 
 /// REG_SEQUENCE - This variadic instruction is used to form a register that
 /// represents a consecutive sequence of sub-registers. It's used as a
 /// register coalescing / allocation aid and must be eliminated before code
 /// emission.
 // In SDNode form, the first operand encodes the register class created by
 // the REG_SEQUENCE, while each subsequent pair names a vreg + subreg index
 // pair.  Once it has been lowered to a MachineInstr, the regclass operand
 // is no longer present.
 /// e.g. v1027 = REG_SEQUENCE v1024, 3, v1025, 4, v1026, 5
 /// After register coalescing references of v1024 should be replace with
 /// v1027:3, v1025 with v1027:4, etc.
-  HANDLE_TARGET_OPCODE(REG_SEQUENCE)
 
 /// COPY - Target-independent register copy. This instruction can also be
 /// used to copy between subregisters of virtual registers.
-  HANDLE_TARGET_OPCODE(COPY)
 
+/// TCOPY - This instruction is the terminator version of COPY. The purpose
+/// is to allow copies from terminators to be properly represented (e.g. an
+/// INLINEASM_BR that defines a physical register) without having
+/// to introduce "live-ins" for physical registers before register allocation.
+HANDLE_TARGET_OPCODE(TCOPY)
+
 /// BUNDLE - This instruction represents an instruction bundle. Instructions
 /// which immediately follow a BUNDLE instruction which are marked with
 /// 'InsideBundle' flag are inside the bundle.
 HANDLE_TARGET_OPCODE(BUNDLE)
 
 /// Lifetime markers.
 HANDLE_TARGET_OPCODE(LIFETIME_START)
 HANDLE_TARGET_OPCODE(LIFETIME_END)

llvm/include/llvm/Target/Target.td

 def COPY : StandardPseudoInstruction {
   let OutOperandList = (outs unknown:$dst);
   let InOperandList = (ins unknown:$src);
   let AsmString = "";
   let hasSideEffects = 0;
   let isAsCheapAsAMove = 1;
   let hasNoSchedulingInfo = 0;
 }
+def TCOPY : StandardPseudoInstruction {
+  let OutOperandList = (outs unknown:$dst);
+  let InOperandList = (ins unknown:$src);
+  let AsmString = "";
+  let hasSideEffects = 0;
+  let isAsCheapAsAMove = 1;
+  let isTerminator = 1;
+}
 def BUNDLE : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins variable_ops);
   let AsmString = "BUNDLE";
   let hasSideEffects = 0;
 }
 def LIFETIME_START : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins i32imm:$id);
   let AsmString = "LIFETIME_START";

llvm/lib/CodeGen/DetectDeadLanes.cpp

 /// Returns true if \p MI will get lowered to a series of COPY instructions.
 /// We call this a COPY-like instruction.
 static bool lowersToCopies(const MachineInstr &MI) {
   // Note: We could support instructions with MCInstrDesc::isRegSequenceLike(),
   // isExtractSubRegLike(), isInsertSubregLike() in the future even though they
   // are not lowered to a COPY.
   switch (MI.getOpcode()) {
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::PHI:
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::EXTRACT_SUBREG:
     return true;
   }
   return false;
 }
LaneBitmask DetectDeadLanes::transferUsedLanes(const MachineInstr &MI,
                                                LaneBitmask UsedLanes,
                                                const MachineOperand &MO) const {
   unsigned OpNum = MI.getOperandNo(&MO);
   assert(lowersToCopies(MI) &&
          DefinedByCopy[Register::virtReg2Index(MI.getOperand(0).getReg())]);
 
   switch (MI.getOpcode()) {
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::PHI:
     return UsedLanes;
   case TargetOpcode::REG_SEQUENCE: {
     assert(OpNum % 2 == 1);
     unsigned SubIdx = MI.getOperand(OpNum + 1).getImm();
     return TRI->reverseComposeSubRegIndexLaneMask(SubIdx, UsedLanes);
   }
   case TargetOpcode::INSERT_SUBREG: {
LaneBitmask DetectDeadLanes::transferDefinedLanes(const MachineOperand &Def,
   }
   case TargetOpcode::EXTRACT_SUBREG: {
     unsigned SubIdx = MI.getOperand(2).getImm();
     assert(OpNum == 1 && "EXTRACT_SUBREG must have one register operand only");
     DefinedLanes = TRI->reverseComposeSubRegIndexLaneMask(SubIdx, DefinedLanes);
     break;
   }
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::PHI:
     break;
   default:
     llvm_unreachable("function must be called with COPY-like instruction");
   }
 
   assert(Def.getSubReg() == 0 &&
          "Should not have subregister defs in machine SSA phase");

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

 #define DEBUG_TYPE "postrapseudos"
 
 namespace {
 struct ExpandPostRA : public MachineFunctionPass {
 private:
   const TargetRegisterInfo *TRI;
   const TargetInstrInfo *TII;
 
+  MachineBasicBlock *TCopyDestBlock;
+
 public:
   static char ID; // Pass identification, replacement for typeid
   ExpandPostRA() : MachineFunctionPass(ID) {}
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
     AU.addPreservedID(MachineLoopInfoID);
     AU.addPreservedID(MachineDominatorsID);
bool ExpandPostRA::LowerSubregToReg(MachineInstr *MI) {
   }
 
   LLVM_DEBUG(dbgs() << '\n');
   MBB->erase(MI);
   return true;
 }
 
 bool ExpandPostRA::LowerCopy(MachineInstr *MI) {
-
   if (MI->allDefsAreDead()) {
     LLVM_DEBUG(dbgs() << "dead copy: " << *MI);
     MI->setDesc(TII->get(TargetOpcode::KILL));
     LLVM_DEBUG(dbgs() << "replaced by: " << *MI);
     return true;
   }
 
   MachineOperand &DstMO = MI->getOperand(0);
if (SrcMO.isUndef() || MI->getNumOperands() > 2) {
       return true;
     }
     // Vanilla identity copy.
     MI->eraseFromParent();
     return true;
   }
 
   LLVM_DEBUG(dbgs() << "real copy:   " << *MI);
-  TII->copyPhysReg(*MI->getParent(), MI, MI->getDebugLoc(),
-                   DstMO.getReg(), SrcMO.getReg(), SrcMO.isKill());
+  MachineBasicBlock *CopyBlock = MI->getParent();
+  MachineBasicBlock::iterator MII(MI);
+  if (MI->getOpcode() == TargetOpcode::TCOPY) {
+    CopyBlock = TCopyDestBlock;
+    MII = TCopyDestBlock->getFirstTerminator();
+  }
+  TII->copyPhysReg(*CopyBlock, MII, MI->getDebugLoc(), DstMO.getReg(),
+                   SrcMO.getReg(), SrcMO.isKill());
 
   if (MI->getNumOperands() > 2)
     TransferImplicitOperands(MI);
   LLVM_DEBUG({
     MachineBasicBlock::iterator dMI = MI;
     dbgs() << "replaced by: " << *(--dMI);
   });
   MI->eraseFromParent();
   return true;
 }
 
 /// runOnMachineFunction - Reduce subregister inserts and extracts to register
 /// copies.
 ///
 bool ExpandPostRA::runOnMachineFunction(MachineFunction &MF) {
   LLVM_DEBUG(dbgs() << "Machine Function\n"
                     << "********** EXPANDING POST-RA PSEUDO INSTRS **********\n"
                     << "********** Function: " << MF.getName() << '\n');
   TRI = MF.getSubtarget().getRegisterInfo();
   TII = MF.getSubtarget().getInstrInfo();
 
   bool MadeChange = false;
 
-  for (MachineFunction::iterator mbbi = MF.begin(), mbbe = MF.end();
-       mbbi != mbbe; ++mbbi) {
-    for (MachineBasicBlock::iterator mi = mbbi->begin(), me = mbbi->end();
-         mi != me;) {
-      MachineInstr &MI = *mi;
+  for (auto &MBB : MF) {
+    for (auto MII = MBB.begin(), ME = MBB.end(); MII != ME;) {
+      MachineInstr &MI = *MII;
       // Advance iterator here because MI may be erased.
-      ++mi;
+      ++MII;
 
       // Only expand pseudos.
       if (!MI.isPseudo())
         continue;
 
       // Give targets a chance to expand even standard pseudos.
       if (TII->expandPostRAPseudo(MI)) {
         MadeChange = true;
         continue;
       }
 
       // Expand standard pseudos.
       switch (MI.getOpcode()) {
+      case TargetOpcode::INLINEASM_BR: {
+        MachineBasicBlock::iterator Next(MI);
+        Next = detail::next_or_end(Next, MBB.end());
+        if (Next == MBB.end() || Next->getOpcode() != TargetOpcode::TCOPY)
+          break;
+
+        // Find the destination for any TCOPY instructions to sink into.
+        TCopyDestBlock = MBB.getInlineAsmBrDefaultTarget();
+        assert(TCopyDestBlock && "Cannot find default dest block for callbr!");
+        break;
+      }
       case TargetOpcode::SUBREG_TO_REG:
         MadeChange |= LowerSubregToReg(&MI);
         break;
       case TargetOpcode::COPY:
+      case TargetOpcode::TCOPY:
         MadeChange |= LowerCopy(&MI);
         break;
       case TargetOpcode::DBG_VALUE:
         continue;
       case TargetOpcode::INSERT_SUBREG:
       case TargetOpcode::EXTRACT_SUBREG:
         llvm_unreachable("Sub-register pseudos should have been eliminated.");
       }
     }
   }
 
   return MadeChange;
 }

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

 bool CombinerHelper::tryCombineCopy(MachineInstr &MI) {
   if (matchCombineCopy(MI)) {
     applyCombineCopy(MI);
     return true;
   }
   return false;
 }
 bool CombinerHelper::matchCombineCopy(MachineInstr &MI) {
-  if (MI.getOpcode() != TargetOpcode::COPY)
+  if (!MI.isCopy())
     return false;
   Register DstReg = MI.getOperand(0).getReg();
   Register SrcReg = MI.getOperand(1).getReg();
   return canReplaceReg(DstReg, SrcReg, MRI);
 }
 void CombinerHelper::applyCombineCopy(MachineInstr &MI) {
   Register DstReg = MI.getOperand(0).getReg();
   Register SrcReg = MI.getOperand(1).getReg();

llvm/lib/CodeGen/GlobalISel/GISelKnownBits.cpp

void GISelKnownBits::computeKnownBitsImpl(Register R, KnownBits &Known,
   KnownBits Known2;
 
   switch (Opcode) {
   default:
     TL.computeKnownBitsForTargetInstr(*this, R, Known, DemandedElts, MRI,
                                       Depth);
     break;
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::G_PHI:
   case TargetOpcode::PHI: {
     Known.One = APInt::getAllOnesValue(BitWidth);
     Known.Zero = APInt::getAllOnesValue(BitWidth);
     // Destination registers should not have subregisters at this
     // point of the pipeline, otherwise the main live-range will be
     // defined more than once, which is against SSA.
     assert(MI.getOperand(0).getSubReg() == 0 && "Is this code in SSA?");
for (unsigned Idx = 1; Idx < MI.getNumOperands(); Idx += 2) {
       // register class.
       //
       // We can't use NoSubRegister by name as it's defined by each target but
       // it's always defined to be 0 by tablegen.
       if (SrcReg.isVirtual() && Src.getSubReg() == 0 /*NoSubRegister*/ &&
           MRI.getType(SrcReg).isValid()) {
         // For COPYs we don't do anything, don't increase the depth.
         computeKnownBitsImpl(SrcReg, Known2, DemandedElts,
-                             Depth + (Opcode != TargetOpcode::COPY));
+                             Depth + !MI.isCopy());
         Known.One &= Known2.One;
         Known.Zero &= Known2.Zero;
         // If we reach a point where we don't know anything
         // just stop looking through the operands.
         if (Known.One == 0 && Known.Zero == 0)
           break;
       } else {
         // We know nothing.
unsigned GISelKnownBits::computeNumSignBits(Register R,
   // type constraint. This is unlikely to occur except by looking through copies
   // but it is possible for the initial register being queried to be in this
   // state.
   if (!DstTy.isValid())
     return 1;
 
   unsigned FirstAnswer = 1;
   switch (Opcode) {
-  case TargetOpcode::COPY: {
+  case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY: {
     MachineOperand &Src = MI.getOperand(1);
     if (Src.getReg().isVirtual() && Src.getSubReg() == 0 &&
         MRI.getType(Src.getReg()).isValid()) {
       // Don't increment Depth for this one since we didn't do any work.
       return computeNumSignBits(Src.getReg(), DemandedElts, Depth);
     }
 
     return 1;

llvm/lib/CodeGen/GlobalISel/InstructionSelect.cpp

for (auto MII = std::prev(MBB.end()), Begin = MBB.begin(); !ReachedBegin;) {
       // Select this instruction.
       MachineInstr &MI = *MII;
 
       // And have our iterator point to the next instruction, if there is one.
       if (MII == Begin)
         ReachedBegin = true;
       else
         --MII;
-      if (MI.getOpcode() != TargetOpcode::COPY)
+      if (!MI.isCopy())
         continue;
       Register SrcReg = MI.getOperand(1).getReg();
       Register DstReg = MI.getOperand(0).getReg();
       if (Register::isVirtualRegister(SrcReg) &&
           Register::isVirtualRegister(DstReg)) {
         auto SrcRC = MRI.getRegClass(SrcReg);
         auto DstRC = MRI.getRegClass(DstReg);
         if (SrcRC == DstRC) {

llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp

MachineInstrBuilder MachineIRBuilder::buildInstr(unsigned Opc,
   case TargetOpcode::G_BITCAST: {
     assert(DstOps.size() == 1 && "Invalid Dst");
     assert(SrcOps.size() == 1 && "Invalid Srcs");
     assert(DstOps[0].getLLTTy(*getMRI()).getSizeInBits() ==
            SrcOps[0].getLLTTy(*getMRI()).getSizeInBits() && "invalid bitcast");
     break;
   }
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
     assert(DstOps.size() == 1 && "Invalid Dst");
     // If the caller wants to add a subreg source it has to be done separately
     // so we may not have any SrcOps at this point yet.
     break;
   case TargetOpcode::G_FCMP:
   case TargetOpcode::G_ICMP: {
     assert(DstOps.size() == 1 && "Invalid Dst Operands");
     assert(SrcOps.size() == 3 && "Invalid Src Operands");

llvm/lib/CodeGen/GlobalISel/Utils.cpp

while ((MI = MRI.getVRegDef(VReg)) && !IsConstantOpcode(MI->getOpcode()) &&
     case TargetOpcode::G_SEXT:
     case TargetOpcode::G_ZEXT:
       SeenOpcodes.push_back(std::make_pair(
           MI->getOpcode(),
           MRI.getType(MI->getOperand(0).getReg()).getSizeInBits()));
       VReg = MI->getOperand(1).getReg();
       break;
     case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY:
       VReg = MI->getOperand(1).getReg();
       if (Register::isPhysicalRegister(VReg))
         return None;
       break;
     case TargetOpcode::G_INTTOPTR:
       VReg = MI->getOperand(1).getReg();
       break;
     default:

 
 static llvm::Optional<DefinitionAndSourceRegister>
 getDefSrcRegIgnoringCopies(Register Reg, const MachineRegisterInfo &MRI) {
   Register DefSrcReg = Reg;
   auto *DefMI = MRI.getVRegDef(Reg);
   auto DstTy = MRI.getType(DefMI->getOperand(0).getReg());
   if (!DstTy.isValid())
     return None;
-  while (DefMI->getOpcode() == TargetOpcode::COPY) {
+  while (DefMI->isCopy()) {
     Register SrcReg = DefMI->getOperand(1).getReg();
     auto SrcTy = MRI.getType(SrcReg);
     if (!SrcTy.isValid() || SrcTy != DstTy)
       break;
     DefMI = MRI.getVRegDef(SrcReg);
     DefSrcReg = SrcReg;
   }
   return DefinitionAndSourceRegister{DefMI, DefSrcReg};

llvm/lib/CodeGen/MachineBasicBlock.cpp

 
 bool MachineBasicBlock::hasEHPadSuccessor() const {
   for (const_succ_iterator I = succ_begin(), E = succ_end(); I != E; ++I)
     if ((*I)->isEHPad())
       return true;
   return false;
 }
 
+MachineBasicBlock *MachineBasicBlock::getInlineAsmBrDefaultTarget() {
+  if (llvm::none_of(terminators(), [](const MachineInstr &Term) {
+        return Term.getOpcode() == TargetOpcode::INLINEASM_BR;
+      }))
+    return nullptr;
+
+  MachineBasicBlock *DefaultTarget = nullptr;
+  for (auto Succ : successors())
+    if (!isInlineAsmBrIndirectTarget(Succ)) {
+      DefaultTarget = Succ;
+      break;
+    }
+  if (!DefaultTarget) {
+    const auto &Br = back();
+    if (Br.isUnconditionalBranch()) {
+      for (const MachineOperand &MO : Br.operands())
+        if (MO.isMBB()) {
+          DefaultTarget = MO.getMBB();
+          break;
+        }
+    }
+  }
+  return DefaultTarget;
+}
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 LLVM_DUMP_METHOD void MachineBasicBlock::dump() const {
   print(dbgs());
 }
 #endif
 
 bool MachineBasicBlock::isLegalToHoistInto() const {
   if (isReturnBlock() || hasEHPadSuccessor())

llvm/lib/CodeGen/MachineInstr.cpp

bool MachineInstr::isSafeToMove(AAResults *AA, bool &SawStore) const {
   // volatiles, but it is required for atomic loads. It is not allowed to move
   // a load across an atomic load with Ordering > Monotonic.
   if (mayStore() || isCall() || isPHI() ||
       (mayLoad() && hasOrderedMemoryRef())) {
     SawStore = true;
     return false;
   }
 
-  if (isPosition() || isDebugInstr() || isTerminator() ||
+  if (isPosition() || isDebugInstr() || (isTerminator() && !isCopy()) ||
       mayRaiseFPException() || hasUnmodeledSideEffects())
     return false;
 
   // See if this instruction does a load.  If so, we have to guarantee that the
   // loaded value doesn't change between the load and the its intended
   // destination. The check for isInvariantLoad gives the targe the chance to
   // classify the load as always returning a constant, e.g. a constant pool
   // load.

llvm/lib/CodeGen/MachineSink.cpp

static bool attemptDebugCopyProp(MachineInstr &SinkInst, MachineInstr &DbgMI) {
   DbgMI.getOperand(0).setReg(SrcMO->getReg());
   DbgMI.getOperand(0).setSubReg(SrcMO->getSubReg());
   return true;
 }
 
 /// Sink an instruction and its associated debug instructions.
 static void performSink(MachineInstr &MI, MachineBasicBlock &SuccToSinkTo,
                         MachineBasicBlock::iterator InsertPos,
+                        const TargetInstrInfo *TII,
                         SmallVectorImpl<MachineInstr *> &DbgValuesToSink) {
 
   // If we cannot find a location to use (merge with), then we erase the debug
   // location to prevent debug-info driven tools from potentially reporting
   // wrong location information.
   if (!SuccToSinkTo.empty() && InsertPos != SuccToSinkTo.end())
     MI.setDebugLoc(DILocation::getMergedLocation(MI.getDebugLoc(),
                                                  InsertPos->getDebugLoc()));
   else
     MI.setDebugLoc(DebugLoc());
 
   // Move the instruction.
   MachineBasicBlock *ParentBlock = MI.getParent();
   SuccToSinkTo.splice(InsertPos, ParentBlock, MI,
                       ++MachineBasicBlock::iterator(MI));
 
+  // The copy no longer needs to be a terminator, so convert it to a normal
+  // COPY.
+  if (MI.getOpcode() == TargetOpcode::TCOPY)
+    MI.setDesc(TII->get(TargetOpcode::COPY));
+
   // Sink a copy of debug users to the insert position. Mark the original
   // DBG_VALUE location as 'undef', indicating that any earlier variable
   // location should be terminated as we've optimised away the value at this
   // point.
-  for (SmallVectorImpl<MachineInstr *>::iterator DBI = DbgValuesToSink.begin(),
-                                                 DBE = DbgValuesToSink.end();
-       DBI != DBE; ++DBI) {
-    MachineInstr *DbgMI = *DBI;
-    MachineInstr *NewDbgMI = DbgMI->getMF()->CloneMachineInstr(*DBI);
+  for (auto *DbgMI : DbgValuesToSink) {
+    MachineInstr *NewDbgMI = DbgMI->getMF()->CloneMachineInstr(DbgMI);
     SuccToSinkTo.insert(InsertPos, NewDbgMI);
 
     if (!attemptDebugCopyProp(MI, *DbgMI))
       DbgMI->getOperand(0).setReg(0);
   }
 }
 
 /// SinkInstruction - Determine whether it is safe to sink the specified machine
 /// instruction out of its current block into a successor.
 bool MachineSinking::SinkInstruction(MachineInstr &MI, bool &SawStore,
                                      AllSuccsCache &AllSuccessors) {
   // Don't sink instructions that the target prefers not to sink.
   if (!TII->shouldSink(MI))
     return false;
 
   // Check if it's safe to move the instruction.
   if (!MI.isSafeToMove(AA, SawStore))
     return false;
 
   // Convergent operations may not be made control-dependent on additional
   // values.
   if (MI.isConvergent())
     return false;
 
+  // Sink TCOPY instructions after register allocation to avoid mucking with
+  // live-ins.
+  if (MI.getOpcode() == TargetOpcode::TCOPY)
+    return false;
+
   // Don't break implicit null checks.  This is a performance heuristic, and not
   // required for correctness.
   if (SinkingPreventsImplicitNullCheck(MI, TII, TRI))
     return false;
 
   // FIXME: This should include support for sinking instructions within the
   // block they are currently in to shorten the live ranges.  We often get
   // instructions sunk into the top of a large block, but it would be better to
bool MachineSinking::SinkInstruction(MachineInstr &MI, bool &SawStore,
   }
 
   // After sinking, some debug users may not be dominated any more. If possible,
   // copy-propagate their operands. As it's expensive, don't do this if there's
   // no debuginfo in the program.
   if (MI.getMF()->getFunction().getSubprogram() && MI.isCopy())
     SalvageUnsunkDebugUsersOfCopy(MI, SuccToSinkTo);
 
-  performSink(MI, *SuccToSinkTo, InsertPos, DbgUsersToSink);
+  performSink(MI, *SuccToSinkTo, InsertPos, TII, DbgUsersToSink);
 
   // Conservatively, clear any kill flags, since it's possible that they are no
   // longer correct.
   // Note that we have to clear the kill flags for any register this instruction
   // uses as we may sink over another instruction which currently kills the
   // used registers.
   for (MachineOperand &MO : MI.operands()) {
     if (MO.isReg() && MO.isUse())
for (auto I = CurBB.rbegin(), E = CurBB.rend(); I != E;) {
     }
     DbgValsToSink.insert(DbgValsToSink.begin(), DbgValsToSinkSet.begin(),
                          DbgValsToSinkSet.end());
 
     // Clear the kill flag if SrcReg is killed between MI and the end of the
     // block.
     clearKillFlags(MI, CurBB, UsedOpsInCopy, UsedRegUnits, TRI);
     MachineBasicBlock::iterator InsertPos = SuccBB->getFirstNonPHI();
-    performSink(*MI, *SuccBB, InsertPos, DbgValsToSink);
+    performSink(*MI, *SuccBB, InsertPos, TII, DbgValsToSink);
     updateLiveIn(MI, SuccBB, UsedOpsInCopy, DefedRegsInCopy);
 
     Changed = true;
     ++NumPostRACopySink;
   }
   return Changed;
 }
 
 bool PostRAMachineSinking::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   bool Changed = false;
   const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
   const TargetInstrInfo *TII = MF.getSubtarget().getInstrInfo();
 
   ModifiedRegUnits.init(*TRI);
   UsedRegUnits.init(*TRI);
   for (auto &BB : MF)
     Changed |= tryToSinkCopy(BB, MF, TRI, TII);
 
   return Changed;
 }

llvm/lib/CodeGen/MachineVerifier.cpp

MachineVerifier::visitMachineBasicBlockBefore(const MachineBasicBlock *MBB) {
   FirstNonPHI = nullptr;
 
   if (!MF->getProperties().hasProperty(
       MachineFunctionProperties::Property::NoPHIs) && MRI->tracksLiveness()) {
     // If this block has allocatable physical registers live-in, check that
     // it is an entry block or landing pad.
     for (const auto &LI : MBB->liveins()) {
       if (isAllocatable(LI.PhysReg) && !MBB->isEHPad() &&
-          !MBB->isInlineAsmBrDefaultTarget() &&
           MBB->getIterator() != MBB->getParent()->begin()) {
         report("MBB has allocatable live-in, but isn't entry or landing-pad.", MBB);
         report_context(LI.PhysReg);
       }
     }
   }
 
   // Count the number of landing pad successors.
   SmallPtrSet<const MachineBasicBlock*, 4> LandingPadSuccs;
   for (const auto *succ : MBB->successors()) {
     if (succ->isEHPad())
       LandingPadSuccs.insert(succ);
     if (!FunctionBlocks.count(succ))
       report("MBB has successor that isn't part of the function.", MBB);
     if (!MBBInfoMap[succ].Preds.count(MBB)) {
       report("Inconsistent CFG", MBB);
       errs() << "MBB is not in the predecessor list of the successor "
              << printMBBReference(*succ) << ".\n";
     }
   }
 
   // Count the number of INLINEASM_BR indirect target successors.
   SmallPtrSet<const MachineBasicBlock*, 4> IndirectTargetSuccs;
   for (const auto *succ : MBB->successors()) {
     if (MBB->isInlineAsmBrIndirectTarget(succ))
       IndirectTargetSuccs.insert(succ);
     if (!FunctionBlocks.count(succ))
       report("MBB has successor that isn't part of the function.", MBB);
     if (!MBBInfoMap[succ].Preds.count(MBB)) {
       report("Inconsistent CFG", MBB);
       errs() << "MBB is not in the predecessor list of the successor "
              << printMBBReference(*succ) << ".\n";
void MachineVerifier::visitMachineBundleBefore(const MachineInstr *MI) {
   if (MI->isTerminator() && !TII->isPredicated(*MI)) {
     if (!FirstTerminator)
       FirstTerminator = MI;
   } else if (FirstTerminator) {
     report("Non-terminator instruction after the first terminator", MI);
     errs() << "First terminator was:\t" << *FirstTerminator;
   }
 }
 
 // The operands on an INLINEASM instruction must follow a template.
 // Verify that the flag operands make sense.
 void MachineVerifier::verifyInlineAsm(const MachineInstr *MI) {
   // The first two operands on INLINEASM are the asm string and global flags.
   if (MI->getNumOperands() < 2) {
     report("Too few operands on inline asm", MI);
     return;
   }
void MachineVerifier::visitMachineInstrBefore(const MachineInstr *MI) {
   }
 
   StringRef ErrorInfo;
   if (!TII->verifyInstruction(*MI, ErrorInfo))
     report(ErrorInfo.data(), MI);
 
   // Verify properties of various specific instruction types
   switch (MI->getOpcode()) {
+  case TargetOpcode::TCOPY: {
+    MachineBasicBlock::const_iterator MII(MI), MIE = MI->getParent()->end();
+    for (; MII != MIE; ++MII) {
+      if (MII->getOpcode() != TargetOpcode::COPY)
+        continue;
+      report("TCOPY and COPY instructions are intermixed", &*MII);
+      errs() << "- TCOPY instruction: ";
+      if (Indexes && Indexes->hasIndex(*MI))
+        errs() << Indexes->getInstructionIndex(*MI) << '\t';
+      MI->print(errs(), /*SkipOpers=*/true);
+    }
+    LLVM_FALLTHROUGH;
+  }
   case TargetOpcode::COPY: {
     if (foundErrors)
       break;
     const MachineOperand &DstOp = MI->getOperand(0);
     const MachineOperand &SrcOp = MI->getOperand(1);
     LLT DstTy = MRI->getType(DstOp.getReg());
     LLT SrcTy = MRI->getType(SrcOp.getReg());
     if (SrcTy.isValid() && DstTy.isValid()) {

llvm/lib/CodeGen/PeepholeOptimizer.cpp

static Rewriter *getCopyRewriter(MachineInstr &MI, const TargetInstrInfo &TII) {
   if (MI.isBitcast() || MI.isRegSequenceLike() || MI.isInsertSubregLike() ||
       MI.isExtractSubregLike())
     return new UncoalescableRewriter(MI);
 
   switch (MI.getOpcode()) {
   default:
     return nullptr;
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
     return new CopyRewriter(MI);
   case TargetOpcode::INSERT_SUBREG:
     return new InsertSubregRewriter(MI);
   case TargetOpcode::EXTRACT_SUBREG:
     return new ExtractSubregRewriter(MI, TII);
   case TargetOpcode::REG_SEQUENCE:
     return new RegSequenceRewriter(MI);
   }

llvm/lib/CodeGen/ReachingDefAnalysis.cpp

for (auto &MO : Last->operands())
     if (isValidRegDefOf(MO, PhysReg))
       return Last;
 
   return Def < 0 ? nullptr : getInstFromId(MBB, Def);
 }
 
 static bool mayHaveSideEffects(MachineInstr &MI) {
   return MI.mayLoadOrStore() || MI.mayRaiseFPException() ||
-         MI.hasUnmodeledSideEffects() || MI.isTerminator() ||
+         MI.hasUnmodeledSideEffects() || (MI.isTerminator() && !MI.isCopy()) ||
          MI.isCall() || MI.isBarrier() || MI.isBranch() || MI.isReturn();
 }
 
 // Can we safely move 'From' to just before 'To'? To satisfy this, 'From' must
 // not define a register that is used by any instructions, after and including,
 // 'To'. These instructions also must not redefine any of Froms operands.
 template<typename Iterator>
 bool ReachingDefAnalysis::isSafeToMove(MachineInstr *From,

llvm/lib/CodeGen/RegAllocFast.cpp

private:
     /// Basic block currently being allocated.
     MachineBasicBlock *MBB;
 
     /// Maps virtual regs to the frame index where these values are spilled.
     IndexedMap<int, VirtReg2IndexFunctor> StackSlotForVirtReg;
 
     /// Everything we know about a live virtual register.
     struct LiveReg {
+      MachineInstr *OrigMI = nullptr;  ///< Instr the LiveReg comes from.
       MachineInstr *LastUse = nullptr; ///< Last instr to use reg.
       Register VirtReg;                ///< Virtual register number.
       MCPhysReg PhysReg = 0;           ///< Currently held here.
       unsigned short LastOpNum = 0;    ///< OpNum on LastUse.
       bool Dirty = false;              ///< Register needs spill.
 
       explicit LiveReg(Register VirtReg) : VirtReg(VirtReg) {}
 
if (LiveVirtRegs.empty())
     return;
   // The LiveRegMap is keyed by an unsigned (the virtreg number), so the order
   // of spilling here is deterministic, if arbitrary.
   for (LiveReg &LR : LiveVirtRegs) {
     if (!LR.PhysReg)
       continue;
     if (OnlyLiveOut && !mayLiveOut(LR.VirtReg))
       continue;
-    spillVirtReg(MI, LR);
+    spillVirtReg(LR.OrigMI->getOpcode() == TargetOpcode::TCOPY ? LR.OrigMI : MI,
+                 LR);
   }
   LiveVirtRegs.clear();
 }
 
 /// Handle the direct use of a physical register.  Check that the register is
 /// not used by a virtreg. Kill the physreg, marking it free. This may add
 /// implicit kills to MO->getParent() and invalidate MO.
 void RegAllocFast::usePhysReg(MachineOperand &MO) {
if (!LRI->PhysReg) {
     allocVirtReg(MI, *LRI, Hint);
   } else if (LRI->LastUse) {
     // Redefining a live register - kill at the last use, unless it is this
     // instruction defining VirtReg multiple times.
     if (LRI->LastUse != &MI || LRI->LastUse->getOperand(LRI->LastOpNum).isUse())
       addKillFlag(*LRI);
   }
   assert(LRI->PhysReg && "Register not assigned");
+  LRI->OrigMI = &MI;
   LRI->LastUse = &MI;
   LRI->LastOpNum = OpNum;
   LRI->Dirty = true;
   markRegUsedInInstr(LRI->PhysReg);
   return LRI->PhysReg;
 }
 
 /// Make sure VirtReg is available in a physreg and return it.
if (!LRI->PhysReg) {
     // This would cause a second reload of %x into a different register.
     LLVM_DEBUG(dbgs() << "Clearing clean kill: " << MO << '\n');
     MO.setIsKill(false);
   } else if (MO.isDead()) {
     LLVM_DEBUG(dbgs() << "Clearing clean dead: " << MO << '\n');
     MO.setIsDead(false);
   }
   assert(LRI->PhysReg && "Register not assigned");
+  LRI->OrigMI = &MI;
   LRI->LastUse = &MI;
   LRI->LastOpNum = OpNum;
   markRegUsedInInstr(LRI->PhysReg);
   return *LRI;
 }
 
 /// Changes operand OpNum in MI the refer the PhysReg, considering subregs. This
 /// may invalidate any operand pointers.  Return true if the operand kills its

llvm/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

EmitCopyFromReg(SDNode *Node, unsigned ResNo, bool IsClone, bool IsCloned,
 
   // If all uses are reading from the src physical register and copying the
   // register is either impossible or very expensive, then don't create a copy.
   if (MatchReg && SrcRC->getCopyCost() < 0) {
     VRBase = SrcReg;
   } else {
     // Create the reg, emit the copy.
     VRBase = MRI->createVirtualRegister(DstRC);
-    BuildMI(*MBB, InsertPos, Node->getDebugLoc(), TII->get(TargetOpcode::COPY),
-            VRBase).addReg(SrcReg);
+
+    // FIXME: The predicate to determine whether an instruction is a COPY or
+    // TCOPY should be generic. At this time though the criteria isn't
+    // well-known except for INLINEASM_BR instructions.
+    unsigned TgtOpc =
+        llvm::any_of(MBB->terminators(),
+                     [](const MachineInstr &Term) {
+                       return Term.getOpcode() == TargetOpcode::INLINEASM_BR;
+                     })
+            ? TargetOpcode::TCOPY
+            : TargetOpcode::COPY;
+    BuildMI(*MBB, InsertPos, Node->getDebugLoc(), TII->get(TgtOpc), VRBase)
+        .addReg(SrcReg);
   }
 
   SDValue Op(Node, ResNo);
   if (IsClone)
     VRBaseMap.erase(Op);
   bool isNew = VRBaseMap.insert(std::make_pair(Op, VRBase)).second;
   (void)isNew; // Silence compiler warning.
   assert(isNew && "Node emitted out of order - early");
case ISD::CopyToReg: {
     if (RegisterSDNode *R = dyn_cast<RegisterSDNode>(SrcVal))
       SrcReg = R->getReg();
     else
       SrcReg = getVR(SrcVal, VRBaseMap);
 
     if (SrcReg == DestReg) // Coalesced away the copy? Ignore.
       break;
 
-    BuildMI(*MBB, InsertPos, Node->getDebugLoc(), TII->get(TargetOpcode::COPY),
-            DestReg).addReg(SrcReg);
+    unsigned TgtOpc =
+        llvm::any_of(MBB->terminators(),
+                     [](const MachineInstr &Term) {
+                       return Term.getOpcode() == TargetOpcode::INLINEASM_BR;
+                     })
+            ? TargetOpcode::TCOPY
+            : TargetOpcode::COPY;
+    BuildMI(*MBB, InsertPos, Node->getDebugLoc(), TII->get(TgtOpc), DestReg)
+        .addReg(SrcReg);
     break;
   }
   case ISD::CopyFromReg: {
     unsigned SrcReg = cast<RegisterSDNode>(Node->getOperand(1))->getReg();
     EmitCopyFromReg(Node, 0, IsClone, IsCloned, SrcReg, VRBaseMap);
     break;
   }
   case ISD::EH_LABEL:

llvm/lib/CodeGen/SelectionDAG/ScheduleDAGSDNodes.cpp

for (const auto &InstrOrder : Orders) {
       }
       if (DLI == DLE)
         break;
 
       LastOrder = Order;
     }
   }
 
-  // Split after an INLINEASM_BR block with outputs. This allows us to keep the
-  // copy to/from register instructions from being between two terminator
-  // instructions, which causes the machine instruction verifier agita.
-  auto TI = llvm::find_if(*BB, [](const MachineInstr &MI){
-    return MI.getOpcode() == TargetOpcode::INLINEASM_BR;
+  // Split after an INLINEASM_BR block with outputs. This gives us a place to
+  // store output values.
+  auto InlineAsmBr = llvm::find_if(BB->terminators(), [](MachineInstr &term) {
+    return term.getOpcode() == TargetOpcode::INLINEASM_BR;
   });
-  auto SplicePt = TI != BB->end() ? std::next(TI) : BB->end();
-  if (TI != BB->end() && SplicePt != BB->end() &&
-      TI->getOpcode() == TargetOpcode::INLINEASM_BR &&
-      SplicePt->getOpcode() == TargetOpcode::COPY) {
-    MachineBasicBlock *FallThrough = BB->getFallThrough();
-    if (!FallThrough)
-      for (const MachineOperand &MO : BB->back().operands())
-        if (MO.isMBB()) {
-          FallThrough = MO.getMBB();
-          break;
-        }
-    assert(FallThrough && "Cannot find default dest block for callbr!");
+  auto TermIter = detail::next_or_end(InlineAsmBr, BB->end());
+  if (InlineAsmBr != BB->end() && TermIter != BB->end() &&
+      TermIter->getOpcode() == TargetOpcode::TCOPY) {
+    do {
+      ++TermIter;
+    } while (TermIter != BB->end() &&
+             TermIter->getOpcode() == TargetOpcode::TCOPY);
+
+    MachineBasicBlock *DefaultTarget = BB->getInlineAsmBrDefaultTarget();
+    assert(DefaultTarget && "Cannot find default dest block for callbr!");
 
     MachineBasicBlock *CopyBB = MF.CreateMachineBasicBlock(BB->getBasicBlock());
     MachineFunction::iterator BBI(*BB);
     MF.insert(++BBI, CopyBB);
+    if (TermIter != BB->end())
+      CopyBB->splice(CopyBB->begin(), BB, TermIter, BB->end());
 
-    CopyBB->splice(CopyBB->begin(), BB, SplicePt, BB->end());
-    CopyBB->setInlineAsmBrDefaultTarget();
-
-    CopyBB->addSuccessor(FallThrough, BranchProbability::getOne());
-    BB->removeSuccessor(FallThrough);
+    CopyBB->addSuccessor(DefaultTarget, BranchProbability::getOne());
+    BB->removeSuccessor(DefaultTarget);
     BB->addSuccessor(CopyBB, BranchProbability::getOne());
 
-    // Mark all physical registers defined in the original block as being live
-    // on entry to the copy block.
-    for (const auto &MI : *CopyBB)
-      for (const MachineOperand &MO : MI.operands())
-        if (MO.isReg()) {
-          Register reg = MO.getReg();
-          if (Register::isPhysicalRegister(reg)) {
-            CopyBB->addLiveIn(reg);
-            break;
-          }
-        }
-
     CopyBB->normalizeSuccProbs();
     BB->normalizeSuccProbs();
 
-    BB->transferInlineAsmBrIndirectTargets(CopyBB);
-
     InsertPos = CopyBB->end();
     return CopyBB;
   }
 
   InsertPos = Emitter.getInsertPos();
   return Emitter.getBlock();
 }
 
 /// Return the basic block label.
 std::string ScheduleDAGSDNodes::getDAGName() const {
   return "sunit-dag." + BB->getFullName();
 }

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

assert(!I.hasOperandBundlesOtherThan(
          "Cannot lower callbrs with arbitrary operand bundles yet!");
 
   assert(I.isInlineAsm() && "Only know how to handle inlineasm callbr");
   visitInlineAsm(I);
   CopyToExportRegsIfNeeded(&I);
 
   // Retrieve successors.
   MachineBasicBlock *Return = FuncInfo.MBBMap[I.getDefaultDest()];
-  Return->setInlineAsmBrDefaultTarget();
 
   // Update successor info.
   addSuccessorWithProb(CallBrMBB, Return, BranchProbability::getOne());
   for (unsigned i = 0, e = I.getNumIndirectDests(); i < e; ++i) {
     MachineBasicBlock *Target = FuncInfo.MBBMap[I.getIndirectDest(i)];
     addSuccessorWithProb(CallBrMBB, Target, BranchProbability::getZero());
     CallBrMBB->addInlineAsmBrIndirectTarget(Target);
   }

llvm/lib/Target/AArch64/AArch64CallLowering.cpp

if (OutInfo.Regs.size() > 1) {
           dbgs() << "... Cannot handle arguments in multiple registers.\n");
       return false;
     }
 
     // Check if we copy the register, walking through copies from virtual
     // registers. Note that getDefIgnoringCopies does not ignore copies from
     // physical registers.
     MachineInstr *RegDef = getDefIgnoringCopies(OutInfo.Regs[0], MRI);
-    if (!RegDef || RegDef->getOpcode() != TargetOpcode::COPY) {
+    if (!RegDef || !RegDef->isCopy()) {
       LLVM_DEBUG(
           dbgs()
           << "... Parameter was not copied into a VReg, cannot tail call.\n");
       return false;
     }
 
     // Got a copy. Verify that it's the same as the register we want.
     Register CopyRHS = RegDef->getOperand(1).getReg();

llvm/lib/Target/AArch64/AArch64FastISel.cpp

bool AArch64FastISel::optimizeIntExtLoad(const Instruction *I, MVT RetVT,
   MachineInstr *MI = MRI.getUniqueVRegDef(Reg);
   if (!MI)
     return false;
 
   // Check if the correct load instruction has been emitted - SelectionDAG might
   // have emitted a zero-extending load, but we need a sign-extending load.
   bool IsZExt = isa<ZExtInst>(I);
   const auto *LoadMI = MI;
-  if (LoadMI->getOpcode() == TargetOpcode::COPY &&
+  if (LoadMI->isCopy() &&
       LoadMI->getOperand(1).getSubReg() == AArch64::sub_32) {
     Register LoadReg = MI->getOperand(1).getReg();
     LoadMI = MRI.getUniqueVRegDef(LoadReg);
     assert(LoadMI && "Expected valid instruction");
   }
   if (!(IsZExt && isZExtLoad(LoadMI)) && !(!IsZExt && isSExtLoad(LoadMI)))
     return false;
 
   // Nothing to be done.
   if (RetVT != MVT::i64 || SrcVT > MVT::i32) {
     updateValueMap(I, Reg);
     return true;
   }
 
   if (IsZExt) {
     unsigned Reg64 = createResultReg(&AArch64::GPR64RegClass);
     BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, DbgLoc,
             TII.get(AArch64::SUBREG_TO_REG), Reg64)
         .addImm(0)
         .addReg(Reg, getKillRegState(true))
         .addImm(AArch64::sub_32);
     Reg = Reg64;
   } else {
-    assert((MI->getOpcode() == TargetOpcode::COPY &&
-            MI->getOperand(1).getSubReg() == AArch64::sub_32) &&
+    assert((MI->isCopy() && MI->getOperand(1).getSubReg() == AArch64::sub_32) &&
            "Expected copy instruction");
     Reg = MI->getOperand(1).getReg();
     MachineBasicBlock::iterator I(MI);
     removeDeadCode(I, std::next(I));
   }
   updateValueMap(I, Reg);
   return true;
 }

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

bool AArch64InstrInfo::isAsCheapAsAMove(const MachineInstr &MI) const {
   if (Subtarget.hasZeroCycleZeroingFP()) {
     if (Opcode == AArch64::FMOVH0 ||
         Opcode == AArch64::FMOVS0 ||
         Opcode == AArch64::FMOVD0)
       return true;
   }
 
   if (Subtarget.hasZeroCycleZeroingGP()) {
-    if (Opcode == TargetOpcode::COPY &&
-        (MI.getOperand(1).getReg() == AArch64::WZR ||
+    if (MI.isCopy() && (MI.getOperand(1).getReg() == AArch64::WZR ||
                         MI.getOperand(1).getReg() == AArch64::XZR))
       return true;
   }
 
   // Secondly, check cases specific to sub-targets.
 
   if (Subtarget.hasExynosCheapAsMoveHandling()) {
     if (isExynosCheapAsMove(MI))
       return true;
if (MI.getOperand(1).isImm() && MI.getOperand(1).getImm() == 0) {
       return true;
     }
     break;
   case AArch64::ANDWri: // and Rd, Rzr, #imm
     return MI.getOperand(1).getReg() == AArch64::WZR;
   case AArch64::ANDXri:
     return MI.getOperand(1).getReg() == AArch64::XZR;
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
     return MI.getOperand(1).getReg() == AArch64::WZR;
   }
   return false;
 }
 
 // Return true if this instruction simply renames a general register without
 // modifying bits.
 bool AArch64InstrInfo::isGPRCopy(const MachineInstr &MI) {
   switch (MI.getOpcode()) {
   default:
     break;
-  case TargetOpcode::COPY: {
+  case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY: {
     // GPR32 copies will by lowered to ORRXrs
     Register DstReg = MI.getOperand(0).getReg();
     return (AArch64::GPR32RegClass.contains(DstReg) ||
             AArch64::GPR64RegClass.contains(DstReg));
   }
   case AArch64::ORRXrs: // orr Xd, Xzr, Xm (LSL #0)
     if (MI.getOperand(1).getReg() == AArch64::XZR) {
       assert(MI.getDesc().getNumOperands() == 4 &&

 }
 
 // Return true if this instruction simply renames a general register without
 // modifying bits.
 bool AArch64InstrInfo::isFPRCopy(const MachineInstr &MI) {
   switch (MI.getOpcode()) {
   default:
     break;
-  case TargetOpcode::COPY: {
+  case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY: {
     // FPR64 copies will by lowered to ORR.16b
     Register DstReg = MI.getOperand(0).getReg();
     return (AArch64::FPR64RegClass.contains(DstReg) ||
             AArch64::FPR128RegClass.contains(DstReg));
   }
   case AArch64::ORRv16i8:
     if (MI.getOperand(1).getReg() == MI.getOperand(2).getReg()) {
       assert(MI.getDesc().getNumOperands() == 3 && MI.getOperand(0).isReg() &&

llvm/lib/Target/AArch64/AArch64InstructionSelector.cpp

if (!MRI.hasOneNonDBGUse(CondDefReg)) {
         if (UI.getOpcode() != TargetOpcode::G_SELECT)
           return false;
       }
     }
 
     // We can skip over G_TRUNC since the condition is 1-bit.
     // Truncating/extending can have no impact on the value.
     unsigned Opc = CondDef->getOpcode();
-    if (Opc != TargetOpcode::COPY && Opc != TargetOpcode::G_TRUNC)
+    if (!CondDef->isCopy() && Opc != TargetOpcode::G_TRUNC)
       break;
 
     // Can't see past copies from physregs.
-    if (Opc == TargetOpcode::COPY &&
+    if (CondDef->isCopy() &&
         Register::isPhysicalRegister(CondDef->getOperand(1).getReg()))
       return false;
 
     CondDef = MRI.getVRegDef(CondDef->getOperand(1).getReg());
   }
 
   // Is the condition defined by a compare?
   if (!CondDef)
bool AArch64InstructionSelector::isDef32(const MachineInstr &MI) const {
   // Only return true if we know the operation will zero-out the high half of
   // the 64-bit register. Truncates can be subregister copies, which don't
   // zero out the high bits. Copies and other copy-like instructions can be
   // fed by truncates, or could be lowered as subregister copies.
   switch (MI.getOpcode()) {
   default:
     return true;
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::G_BITCAST:
   case TargetOpcode::G_TRUNC:
   case TargetOpcode::G_PHI:
     return false;
   }
 }
 
 

llvm/lib/Target/AArch64/AArch64RegisterBankInfo.cpp

#endif // End NDEBUG.
 
   return getInstructionMapping(DefaultMappingID, 1,
                                getValueMapping(RBIdx, Size), NumOperands);
 }
 
 bool AArch64RegisterBankInfo::hasFPConstraints(
     const MachineInstr &MI, const MachineRegisterInfo &MRI,
     const TargetRegisterInfo &TRI) const {
-  unsigned Op = MI.getOpcode();
-
   // Do we have an explicit floating point instruction?
-  if (isPreISelGenericFloatingPointOpcode(Op))
+  if (isPreISelGenericFloatingPointOpcode(MI.getOpcode()))
     return true;
 
   // No. Check if we have a copy-like instruction. If we do, then we could
   // still be fed by floating point instructions.
-  if (Op != TargetOpcode::COPY && !MI.isPHI())
+  if (!MI.isCopy() && !MI.isPHI())
     return false;
 
   // MI is copy-like. Return true if it outputs an FPR.
   return getRegBank(MI.getOperand(0).getReg(), MRI, TRI) ==
          &AArch64::FPRRegBank;
 }
 
 bool AArch64RegisterBankInfo::onlyUsesFP(const MachineInstr &MI,

 }
 
 const RegisterBankInfo::InstructionMapping &
 AArch64RegisterBankInfo::getInstrMapping(const MachineInstr &MI) const {
   const unsigned Opc = MI.getOpcode();
 
   // Try the default logic for non-generic instructions that are either copies
   // or already have some operands assigned to banks.
-  if ((Opc != TargetOpcode::COPY && !isPreISelGenericOpcode(Opc)) ||
+  if ((!MI.isCopy() && !isPreISelGenericOpcode(Opc)) ||
       Opc == TargetOpcode::G_PHI) {
     const RegisterBankInfo::InstructionMapping &Mapping =
         getInstrMappingImpl(MI);
     if (Mapping.isValid())
       return Mapping;
   }
 
   const MachineFunction &MF = *MI.getParent()->getParent();
AArch64RegisterBankInfo::getInstrMapping(const MachineInstr &MI) const {
   case TargetOpcode::G_ASHR: {
     LLT ShiftAmtTy = MRI.getType(MI.getOperand(2).getReg());
     LLT SrcTy = MRI.getType(MI.getOperand(1).getReg());
     if (ShiftAmtTy.getSizeInBits() == 64 && SrcTy.getSizeInBits() == 32)
       return getInstructionMapping(DefaultMappingID, 1,
                                    &ValMappings[Shift64Imm], 3);
     return getSameKindOfOperandsMapping(MI);
   }
-  case TargetOpcode::COPY: {
+  case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY: {
     Register DstReg = MI.getOperand(0).getReg();
     Register SrcReg = MI.getOperand(1).getReg();
     // Check if one of the register is not a generic register.
     if ((Register::isPhysicalRegister(DstReg) ||
          !MRI.getType(DstReg).isValid()) ||
         (Register::isPhysicalRegister(SrcReg) ||
          !MRI.getType(SrcReg).isValid())) {
       const RegisterBank *DstRB = getRegBank(DstReg, MRI, TRI);

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

case AMDGPU::SI_INIT_EXEC_FROM_INPUT: {
     MachineInstr *FirstMI = &*BB->begin();
     MachineRegisterInfo &MRI = MF->getRegInfo();
     Register InputReg = MI.getOperand(0).getReg();
     Register CountReg = MRI.createVirtualRegister(&AMDGPU::SGPR_32RegClass);
     bool Found = false;
 
     // Move the COPY of the input reg to the beginning, so that we can use it.
     for (auto I = BB->begin(); I != &MI; I++) {
-      if (I->getOpcode() != TargetOpcode::COPY ||
-          I->getOperand(0).getReg() != InputReg)
+      if (!I->isCopy() || I->getOperand(0).getReg() != InputReg)
         continue;
 
       if (I == FirstMI) {
         FirstMI = &*++BB->begin();
       } else {
         I->removeFromParent();
         BB->insert(FirstMI, &*I);
       }

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

void SIInstrInfo::copyPhysReg(MachineBasicBlock &MBB,
 
   for (unsigned Idx = 0; Idx < SubIndices.size(); ++Idx) {
     unsigned SubIdx;
     if (Forward)
       SubIdx = SubIndices[Idx];
     else
       SubIdx = SubIndices[SubIndices.size() - Idx - 1];
 
-    if (Opcode == TargetOpcode::COPY) {
+    if (Opcode == TargetOpcode::COPY || Opcode == TargetOpcode::TCOPY) {
       copyPhysReg(MBB, MI, DL, RI.getSubReg(DestReg, SubIdx),
                   RI.getSubReg(SrcReg, SubIdx), KillSrc);
       continue;
     }
 
     MachineInstrBuilder Builder = BuildMI(MBB, MI, DL,
       get(Opcode), RI.getSubReg(DestReg, SubIdx));
 

llvm/lib/Target/Hexagon/BitTracker.cpp

case TargetOpcode::REG_SEQUENCE: {
       uint16_t W = getRegBitWidth(RD);
       RegisterCell Res(W);
       Res.insert(RegisterCell::ref(getCell(RS, Inputs)), mask(RD.Reg, SS));
       Res.insert(RegisterCell::ref(getCell(RT, Inputs)), mask(RD.Reg, ST));
       putCell(RD, Res, Outputs);
       break;
     }
 
-    case TargetOpcode::COPY: {
+    case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY: {
       // COPY can transfer a smaller register into a wider one.
       // If that is the case, fill the remaining high bits with 0.
       RegisterRef RD = MI.getOperand(0);
       RegisterRef RS = MI.getOperand(1);
       assert(RD.Sub == 0);
       uint16_t WD = getRegBitWidth(RD);
       uint16_t WS = getRegBitWidth(RS);
       assert(WD >= WS);

llvm/lib/Target/Hexagon/HexagonBitSimplify.cpp

bool RedundantInstrElimination::processBlock(MachineBasicBlock &B,
   if (!BT.reached(&B))
     return false;
   bool Changed = false;
 
   for (auto I = B.begin(), E = B.end(), NextI = I; I != E; ++I) {
     NextI = std::next(I);
     MachineInstr *MI = &*I;
 
-    if (MI->getOpcode() == TargetOpcode::COPY)
+    if (MI->isCopy())
       continue;
     if (MI->isPHI() || MI->hasUnmodeledSideEffects() || MI->isInlineAsm())
       continue;
     unsigned NumD = MI->getDesc().getNumDefs();
     if (NumD != 1)
       continue;
 
     BitTracker::RegisterRef RD = MI->getOperand(0);
bool CopyGeneration::processBlock(MachineBasicBlock &B,
   }
 
   return Changed;
 }
 
 bool CopyPropagation::isCopyReg(unsigned Opc, bool NoConv) {
   switch (Opc) {
     case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY:
     case TargetOpcode::REG_SEQUENCE:
     case Hexagon::A4_combineir:
     case Hexagon::A4_combineri:
       return true;
     case Hexagon::A2_tfr:
     case Hexagon::A2_tfrp:
     case Hexagon::A2_combinew:
     case Hexagon::V6_vcombine:
       return NoConv;
     default:
       break;
   }
   return false;
 }
 
 bool CopyPropagation::propagateRegCopy(MachineInstr &MI) {
   bool Changed = false;
   unsigned Opc = MI.getOpcode();
   BitTracker::RegisterRef RD = MI.getOperand(0);
   assert(MI.getOperand(0).getSubReg() == 0);
 
   switch (Opc) {
     case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY:
     case Hexagon::A2_tfr:
     case Hexagon::A2_tfrp: {
       BitTracker::RegisterRef RS = MI.getOperand(1);
       if (!HBS::isTransparentCopy(RD, RS, MRI))
         break;
       if (RS.Sub != 0)
         Changed = HBS::replaceRegWithSub(RD.Reg, RS.Reg, RS.Sub, MRI);
       else
bool BitSimplification::processBlock(MachineBasicBlock &B,
   RegisterSet AVB = AVs;
   RegisterSet Defs;
 
   for (auto I = B.begin(), E = B.end(); I != E; ++I, AVB.insert(Defs)) {
     MachineInstr *MI = &*I;
     Defs.clear();
     HBS::getInstrDefs(*MI, Defs);
 
-    unsigned Opc = MI->getOpcode();
-    if (Opc == TargetOpcode::COPY || Opc == TargetOpcode::REG_SEQUENCE)
+    if (MI->isCopy() || MI->getOpcode() == TargetOpcode::REG_SEQUENCE)
       continue;
 
     if (MI->mayStore()) {
       bool T = genStoreUpperHalf(MI);
       T = T || genStoreImmediate(MI);
       Changed |= T;
       continue;
     }
bool HexagonLoopRescheduling::isConst(unsigned Reg) const {
   return true;
 }
 
 bool HexagonLoopRescheduling::isBitShuffle(const MachineInstr *MI,
       unsigned DefR) const {
   unsigned Opc = MI->getOpcode();
   switch (Opc) {
     case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY:
     case Hexagon::S2_lsr_i_r:
     case Hexagon::S2_asr_i_r:
     case Hexagon::S2_asl_i_r:
     case Hexagon::S2_lsr_i_p:
     case Hexagon::S2_asr_i_p:
     case Hexagon::S2_asl_i_p:
     case Hexagon::S2_insert:
     case Hexagon::A2_or:

llvm/lib/Target/Hexagon/HexagonFrameLowering.cpp

for (auto &B : MF) {
     MachineBasicBlock::iterator NextI;
     for (auto I = B.begin(), E = B.end(); I != E; I = NextI) {
       MachineInstr *MI = &*I;
       NextI = std::next(I);
       unsigned Opc = MI->getOpcode();
 
       switch (Opc) {
         case TargetOpcode::COPY:
+        case TargetOpcode::TCOPY:
           Changed |= expandCopy(B, I, MRI, HII, NewRegs);
           break;
         case Hexagon::STriw_pred:
         case Hexagon::STriw_ctr:
           Changed |= expandStoreInt(B, I, MRI, HII, NewRegs);
           break;
         case Hexagon::LDriw_pred:
         case Hexagon::LDriw_ctr:

llvm/lib/Target/Hexagon/HexagonGenPredicate.cpp

void HexagonGenPredicate::collectPredicateGPR(MachineFunction &MF) {
   for (MachineFunction::iterator A = MF.begin(), Z = MF.end(); A != Z; ++A) {
     MachineBasicBlock &B = *A;
     for (MachineBasicBlock::iterator I = B.begin(), E = B.end(); I != E; ++I) {
       MachineInstr *MI = &*I;
       unsigned Opc = MI->getOpcode();
       switch (Opc) {
         case Hexagon::C2_tfrpr:
         case TargetOpcode::COPY:
+        case TargetOpcode::TCOPY:
           if (isPredReg(MI->getOperand(1).getReg())) {
             RegisterSubReg RD = MI->getOperand(0);
             if (Register::isVirtualRegister(RD.R))
               PredGPRs.insert(RD);
           }
           break;
       }
     }
RegisterSubReg HexagonGenPredicate::getPredRegFor(const RegisterSubReg &Reg) {
   RegToRegMap::iterator F = G2P.find(Reg);
   if (F != G2P.end())
     return F->second;
 
   LLVM_DEBUG(dbgs() << __func__ << ": " << PrintRegister(Reg, *TRI));
   MachineInstr *DefI = MRI->getVRegDef(Reg.R);
   assert(DefI);
   unsigned Opc = DefI->getOpcode();
-  if (Opc == Hexagon::C2_tfrpr || Opc == TargetOpcode::COPY) {
+  if (Opc == Hexagon::C2_tfrpr || DefI->isCopy()) {
     assert(DefI->getOperand(0).isDef() && DefI->getOperand(1).isUse());
     RegisterSubReg PR = DefI->getOperand(1);
     G2P.insert(std::make_pair(Reg, PR));
     LLVM_DEBUG(dbgs() << " -> " << PrintRegister(PR, *TRI) << '\n');
     return PR;
   }
 
   MachineBasicBlock &B = *DefI->getParent();
bool HexagonGenPredicate::isScalarPred(RegisterSubReg PredReg) {
   while (!WorkQ.empty()) {
     RegisterSubReg PR = WorkQ.front();
     WorkQ.pop();
     const MachineInstr *DefI = MRI->getVRegDef(PR.R);
     if (!DefI)
       return false;
     unsigned DefOpc = DefI->getOpcode();
     switch (DefOpc) {
-      case TargetOpcode::COPY: {
+      case TargetOpcode::COPY:
+      case TargetOpcode::TCOPY: {
         const TargetRegisterClass *PredRC = &Hexagon::PredRegsRegClass;
         if (MRI->getRegClass(PR.R) != PredRC)
           return false;
         // If it is a copy between two predicate registers, fall through.
         LLVM_FALLTHROUGH;
       }
       case Hexagon::C2_and:
       case Hexagon::C2_andn:
bool HexagonGenPredicate::eliminatePredCopies(MachineFunction &MF) {
   //   PredR2 = PredR1
   // Such sequences can be generated when a copy-into-pred is generated from
   // a gpr register holding a result of a convertible instruction. After
   // the convertible instruction is converted, its predicate result will be
   // copied back into the original gpr.
 
   for (MachineBasicBlock &MBB : MF) {
     for (MachineInstr &MI : MBB) {
-      if (MI.getOpcode() != TargetOpcode::COPY)
+      if (!MI.isCopy())
         continue;
       RegisterSubReg DR = MI.getOperand(0);
       RegisterSubReg SR = MI.getOperand(1);
       if (!Register::isVirtualRegister(DR.R))
         continue;
       if (!Register::isVirtualRegister(SR.R))
         continue;
       if (MRI->getRegClass(DR.R) != PredRC)

llvm/lib/Target/Hexagon/HexagonHardwareLoops.cpp

bool HexagonHardwareLoops::checkForImmediate(const MachineOperand &MO,
 
   Register R = MO.getReg();
   if (!Register::isVirtualRegister(R))
     return false;
   MachineInstr *DI = MRI->getVRegDef(R);
   unsigned DOpc = DI->getOpcode();
   switch (DOpc) {
     case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY:
     case Hexagon::A2_tfrsi:
     case Hexagon::A2_tfrpi:
     case Hexagon::CONST32:
     case Hexagon::CONST64:
       // Call recursively to avoid an extra check whether operand(1) is
       // indeed an immediate (it could be a global address, for example),
       // plus we can handle COPY at the same time.
       if (!checkForImmediate(DI->getOperand(1), TV))

llvm/lib/Target/Hexagon/HexagonISelDAGToDAGHVX.cpp

for (const OpRef &R : Node.Ops) {
         unsigned Sub = (Part == OpRef::LoHalf) ? Hexagon::vsub_lo
                                                : Hexagon::vsub_hi;
         Op = DAG.getTargetExtractSubreg(Sub, dl, HalfTy, Op);
       }
       Ops.push_back(Op);
     } // for (Node : Results)
 
     assert(Node.Ty != MVT::Other);
-    SDNode *ResN = (Node.Opc == TargetOpcode::COPY)
-                      ? Ops.front().getNode()
-                      : DAG.getMachineNode(Node.Opc, dl, Node.Ty, Ops);
+    SDNode *ResN =
+        (Node.Opc == TargetOpcode::COPY || Node.Opc == TargetOpcode::TCOPY)
     Output.push_back(SDValue(ResN, 0));
   }
 
   SDNode *OutN = Output.back().getNode();
   SDNode *InpN = Results.InpNode;
   DEBUG_WITH_TYPE("isel", {
     dbgs() << "Generated node:\n";
     OutN->dumpr(&DAG);

llvm/lib/Target/Hexagon/HexagonInstrInfo.cpp

auto UseAligned = [&] (const MachineInstr &MI, unsigned NeedAlign) {
     if (MI.memoperands().empty())
       return false;
     return all_of(MI.memoperands(), [NeedAlign](const MachineMemOperand *MMO) {
       return MMO->getAlign() >= NeedAlign;
     });
   };
 
   switch (Opc) {
-    case TargetOpcode::COPY: {
+    case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY: {
       MachineOperand &MD = MI.getOperand(0);
       MachineOperand &MS = MI.getOperand(1);
       MachineBasicBlock::iterator MBBI = MI.getIterator();
       if (MD.getReg() != MS.getReg() && !MS.isUndef()) {
         copyPhysReg(MBB, MI, DL, MD.getReg(), MS.getReg(), MS.isKill());
         std::prev(MBBI)->copyImplicitOps(*MBB.getParent(), MI);
       }
       MBB.erase(MBBI);
       return true;
     }
     case Hexagon::PS_aligna:
       BuildMI(MBB, MI, DL, get(Hexagon::A2_andir), MI.getOperand(0).getReg())
           .addReg(HRI.getFrameRegister())
           .addImm(-MI.getOperand(1).getImm());
       MBB.erase(MI);
       return true;
     case Hexagon::V6_vassignp: {
       Register SrcReg = MI.getOperand(1).getReg();

 bool HexagonInstrInfo::isLateResultInstr(const MachineInstr &MI) const {
   switch (MI.getOpcode()) {
   case TargetOpcode::EXTRACT_SUBREG:
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::SUBREG_TO_REG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::INLINEASM:
   case TargetOpcode::PHI:
     return false;
   default:
     break;
   }
 
   unsigned SchedClass = MI.getDesc().getSchedClass();

llvm/lib/Target/Hexagon/HexagonMachineScheduler.cpp

if (!ResourcesModel->canReserveResources(*SU->getInstr()))
       return false;
     break;
   case TargetOpcode::EXTRACT_SUBREG:
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::SUBREG_TO_REG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::INLINEASM:
   case TargetOpcode::INLINEASM_BR:
     break;
   }
 
   MachineBasicBlock *MBB = SU->getInstr()->getParent();
   auto &QST = MBB->getParent()->getSubtarget<HexagonSubtarget>();
   const auto &QII = *QST.getInstrInfo();
bool VLIWResourceModel::reserveResources(SUnit *SU, bool IsTop) {
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::SUBREG_TO_REG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::KILL:
   case TargetOpcode::CFI_INSTRUCTION:
   case TargetOpcode::EH_LABEL:
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::INLINEASM:
   case TargetOpcode::INLINEASM_BR:
     break;
   }
   Packet.push_back(SU);
 
 #ifndef NDEBUG
   LLVM_DEBUG(dbgs() << "Packet[" << TotalPackets << "]:\n");

llvm/lib/Target/Hexagon/HexagonNewValueJump.cpp

if (!afterRA) {
     // KILL sets kill flag on the opcode. It also sets up a
     // single register, out of pair.
     //    %d0 = S2_lsr_r_p killed %d0, killed %r2
     //    %r0 = KILL %r0, implicit killed %d0
     //    %p0 = C2_cmpeqi killed %r0, 0
     // PHI can be anything after RA.
     // COPY can remateriaze things in between feeder, compare and nvj.
     if (MII->getOpcode() == TargetOpcode::KILL ||
-        MII->getOpcode() == TargetOpcode::PHI ||
-        MII->getOpcode() == TargetOpcode::COPY)
+        MII->getOpcode() == TargetOpcode::PHI || MII->isCopy())
       return false;
 
     // The following pseudo Hexagon instructions sets "use" and "def"
     // of registers by individual passes in the backend. At this time,
     // we don't know the scope of usage and definitions of these
     // instructions.
     if (MII->getOpcode() == Hexagon::LDriw_pred ||
         MII->getOpcode() == Hexagon::STriw_pred)
if (cmpReg1 == cmpOp2)
       return false;
 
     // Make sure that the second register is not from COPY
     // at machine code level, we don't need this, but if we decide
     // to move new value jump prior to RA, we would be needing this.
     MachineRegisterInfo &MRI = MF.getRegInfo();
     if (secondReg && !Register::isPhysicalRegister(cmpOp2)) {
       MachineInstr *def = MRI.getVRegDef(cmpOp2);
-      if (def->getOpcode() == TargetOpcode::COPY)
+      if (def->isCopy())
         return false;
     }
   }
 
   // Walk the instructions after the compare (predicate def) to the jump,
   // and satisfy the following conditions.
   ++II;
   for (MachineBasicBlock::iterator localII = II; localII != end; ++localII) {

llvm/lib/Target/Hexagon/HexagonSplitDouble.cpp

bool HexagonSplitDoubleRegs::isFixedInstr(const MachineInstr *MI) const {
 
   unsigned Opc = MI->getOpcode();
   switch (Opc) {
     default:
       return true;
 
     case TargetOpcode::PHI:
     case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY:
       break;
 
     case Hexagon::L2_loadrd_io:
       // Not handling stack stores (only reg-based addresses).
       if (MI->getOperand(1).isReg())
         break;
       return true;
     case Hexagon::S2_storerd_io:
int32_t HexagonSplitDoubleRegs::profit(const MachineInstr *MI) const {
   unsigned Opc = MI->getOpcode();
   switch (Opc) {
     case TargetOpcode::PHI:
       for (const auto &Op : MI->operands())
         if (!Op.getSubReg())
           return 0;
       return 10;
     case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY:
       if (MI->getOperand(1).getSubReg() != 0)
         return 10;
       return 0;
 
     case Hexagon::L2_loadrd_io:
     case Hexagon::S2_storerd_io:
       return -1;
     case Hexagon::L2_loadrd_pi:
bool HexagonSplitDoubleRegs::splitInstr(MachineInstr *MI,
   using namespace Hexagon;
 
   LLVM_DEBUG(dbgs() << "Splitting: " << *MI);
   bool Split = false;
   unsigned Opc = MI->getOpcode();
 
   switch (Opc) {
     case TargetOpcode::PHI:
-    case TargetOpcode::COPY: {
+    case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY: {
       Register DstR = MI->getOperand(0).getReg();
       if (MRI->getRegClass(DstR) == DoubleRC) {
         createHalfInstr(Opc, MI, PairMap, isub_lo);
         createHalfInstr(Opc, MI, PairMap, isub_hi);
         Split = true;
       }
       break;
     }

llvm/lib/Target/Hexagon/RDFCopy.cpp

 #ifndef NDEBUG
 static cl::opt<unsigned> CpLimit("rdf-cp-limit", cl::init(0), cl::Hidden);
 static unsigned CpCount = 0;
 #endif
 
 bool CopyPropagation::interpretAsCopy(const MachineInstr *MI, EqualityMap &EM) {
   unsigned Opc = MI->getOpcode();
   switch (Opc) {
-    case TargetOpcode::COPY: {
+    case TargetOpcode::COPY:
+    case TargetOpcode::TCOPY: {
       const MachineOperand &Dst = MI->getOperand(0);
       const MachineOperand &Src = MI->getOperand(1);
       RegisterRef DstR = DFG.makeRegRef(Dst.getReg(), Dst.getSubReg());
       RegisterRef SrcR = DFG.makeRegRef(Src.getReg(), Src.getSubReg());
       assert(Register::isPhysicalRegister(DstR.Reg));
       assert(Register::isPhysicalRegister(SrcR.Reg));
       const TargetRegisterInfo &TRI = DFG.getTRI();
       if (TRI.getMinimalPhysRegClass(DstR.Reg) !=

llvm/lib/Target/Mips/MipsRegisterBankInfo.cpp

 
 void MipsRegisterBankInfo::AmbiguousRegDefUseContainer::addDefUses(
     Register Reg, const MachineRegisterInfo &MRI) {
   assert(!MRI.getType(Reg).isPointer() &&
          "Pointers are gprb, they should not be considered as ambiguous.\n");
   for (MachineInstr &UseMI : MRI.use_instructions(Reg)) {
     MachineInstr *NonCopyInstr = skipCopiesOutgoing(&UseMI);
     // Copy with many uses.
-    if (NonCopyInstr->getOpcode() == TargetOpcode::COPY &&
+    if (NonCopyInstr->isCopy() &&
         !Register::isPhysicalRegister(NonCopyInstr->getOperand(0).getReg()))
       addDefUses(NonCopyInstr->getOperand(0).getReg(), MRI);
     else
       DefUses.push_back(skipCopiesOutgoing(&UseMI));
   }
 }
 
 void MipsRegisterBankInfo::AmbiguousRegDefUseContainer::addUseDef(
     Register Reg, const MachineRegisterInfo &MRI) {
   assert(!MRI.getType(Reg).isPointer() &&
          "Pointers are gprb, they should not be considered as ambiguous.\n");
   MachineInstr *DefMI = MRI.getVRegDef(Reg);
   UseDefs.push_back(skipCopiesIncoming(DefMI));
 }
 
 MachineInstr *
 MipsRegisterBankInfo::AmbiguousRegDefUseContainer::skipCopiesOutgoing(
     MachineInstr *MI) const {
   const MachineFunction &MF = *MI->getParent()->getParent();
   const MachineRegisterInfo &MRI = MF.getRegInfo();
   MachineInstr *Ret = MI;
-  while (Ret->getOpcode() == TargetOpcode::COPY &&
+  while (Ret->isCopy() &&
          !Register::isPhysicalRegister(Ret->getOperand(0).getReg()) &&
          MRI.hasOneUse(Ret->getOperand(0).getReg())) {
     Ret = &(*MRI.use_instr_begin(Ret->getOperand(0).getReg()));
   }
   return Ret;
 }
 
 MachineInstr *
 MipsRegisterBankInfo::AmbiguousRegDefUseContainer::skipCopiesIncoming(
     MachineInstr *MI) const {
   const MachineFunction &MF = *MI->getParent()->getParent();
   const MachineRegisterInfo &MRI = MF.getRegInfo();
   MachineInstr *Ret = MI;
-  while (Ret->getOpcode() == TargetOpcode::COPY &&
+  while (Ret->isCopy() &&
          !Register::isPhysicalRegister(Ret->getOperand(1).getReg()))
     Ret = MRI.getVRegDef(Ret->getOperand(1).getReg());
   return Ret;
 }
 
 MipsRegisterBankInfo::AmbiguousRegDefUseContainer::AmbiguousRegDefUseContainer(
     const MachineInstr *MI) {
   assert(isAmbiguous(MI->getOpcode()) &&
while (!AdjacentInstrs.empty()) {
     if (isDefUse ? isFloatingPointOpcodeUse(AdjMI->getOpcode())
                  : isFloatingPointOpcodeDef(AdjMI->getOpcode())) {
       setTypes(MI, InstType::FloatingPoint);
       return true;
     }
 
     // Determine InstType from register bank of phys register that is
     // 'isDefUse ? def : use' of this copy.
-    if (AdjMI->getOpcode() == TargetOpcode::COPY) {
+    if (AdjMI->isCopy()) {
       setTypesAccordingToPhysicalRegister(MI, AdjMI, isDefUse ? 0 : 1);
       return true;
     }
 
     // Defaults to integer instruction. Small registers in G_MERGE (uses) and
     // G_UNMERGE (defs) will always be gprb.
     if ((!isDefUse && AdjMI->getOpcode() == TargetOpcode::G_UNMERGE_VALUES) ||
         (isDefUse && AdjMI->getOpcode() == TargetOpcode::G_MERGE_VALUES) ||

llvm/lib/Target/Mips/MipsSEFrameLowering.cpp

case Mips::ExtractElementF64:
     if (expandExtractElementF64(MBB, I, false))
       MBB.erase(I);
     return false;
   case Mips::ExtractElementF64_64:
     if (expandExtractElementF64(MBB, I, true))
       MBB.erase(I);
     return false;
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
     if (!expandCopy(MBB, I))
       return false;
     break;
   default:
     return false;
   }
 
   MBB.erase(I);

llvm/lib/Target/NVPTX/NVPTXReplaceImageHandles.cpp

case NVPTX::texsurf_handles: {
     assert(TexHandleDef.getOperand(1).isGlobal() && "Load is not a global!");
     const GlobalValue *GV = TexHandleDef.getOperand(1).getGlobal();
     assert(GV->hasName() && "Global sampler must be named!");
     InstrsToRemove.insert(&TexHandleDef);
     Idx = MFI->getImageHandleSymbolIndex(GV->getName().data());
     return true;
   }
   case NVPTX::nvvm_move_i64:
-  case TargetOpcode::COPY: {
+  case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY: {
     bool Res = findIndexForHandle(TexHandleDef.getOperand(1), MF, Idx);
     if (Res) {
       InstrsToRemove.insert(&TexHandleDef);
     }
     return Res;
   }
   default:
     llvm_unreachable("Unknown instruction operating on handle");
   }
 }
 
 MachineFunctionPass *llvm::createNVPTXReplaceImageHandlesPass() {
   return new NVPTXReplaceImageHandles();
 }

llvm/lib/Target/X86/X86DomainReassignment.cpp

if (Register::isPhysicalRegister(SrcReg) &&
          X86::GR16RegClass.contains(SrcReg)))
       return false;
 
     return true;
   }
 
   double getExtraCost(const MachineInstr *MI,
                       MachineRegisterInfo *MRI) const override {
-    assert(MI->getOpcode() == TargetOpcode::COPY && "Expected a COPY");
+    assert(MI->isCopy() && "Expected a COPY");
 
     for (auto &MO : MI->operands()) {
       // Physical registers will not be converted. Assume that converting the
       // COPY to the destination domain will eventually result in a actual
       // instruction.
       if (Register::isPhysicalRegister(MO.getReg()))
         return 1;
 

llvm/lib/Target/X86/X86FlagsCopyLowering.cpp

bool X86FlagsCopyLoweringPass::runOnMachineFunction(MachineFunction &MF) {
   // Collect the copies in RPO so that when there are chains where a copy is in
   // turn copied again we visit the first one first. This ensures we can find
   // viable locations for testing the original EFLAGS that dominate all the
   // uses across complex CFGs.
   SmallVector<MachineInstr *, 4> Copies;
   ReversePostOrderTraversal<MachineFunction *> RPOT(&MF);
   for (MachineBasicBlock *MBB : RPOT)
     for (MachineInstr &MI : *MBB)
-      if (MI.getOpcode() == TargetOpcode::COPY &&
-          MI.getOperand(0).getReg() == X86::EFLAGS)
+      if (MI.isCopy() && MI.getOperand(0).getReg() == X86::EFLAGS)
         Copies.push_back(&MI);
 
   for (MachineInstr *CopyI : Copies) {
     MachineBasicBlock &MBB = *CopyI->getParent();
 
     MachineOperand &VOp = CopyI->getOperand(1);
     assert(VOp.isReg() &&
            "The input to the copy for EFLAGS should always be a register!");
     MachineInstr &CopyDefI = *MRI->getVRegDef(VOp.getReg());
-    if (CopyDefI.getOpcode() != TargetOpcode::COPY) {
+    if (!CopyDefI.isCopy()) {
       // FIXME: The big likely candidate here are PHI nodes. We could in theory
       // handle PHI nodes, but it gets really, really hard. Insanely hard. Hard
       // enough that it is probably better to change every other part of LLVM
       // to avoid creating them. The issue is that once we have PHIs we won't
       // know which original EFLAGS value we need to capture with our setCCs
       // below. The end result will be computing a complete set of setCCs that
       // we *might* want, computing them in every place where we copy *out* of
       // EFLAGS and then doing SSA formation on all of them to insert necessary
do {
 
         // Otherwise we can just rewrite in-place.
         if (X86::getCondFromCMov(MI) != X86::COND_INVALID) {
           rewriteCMov(*TestMBB, TestPos, TestLoc, MI, *FlagUse, CondRegs);
         } else if (getCondFromFCMOV(MI.getOpcode()) != X86::COND_INVALID) {
           rewriteFCMov(*TestMBB, TestPos, TestLoc, MI, *FlagUse, CondRegs);
         } else if (X86::getCondFromSETCC(MI) != X86::COND_INVALID) {
           rewriteSetCC(*TestMBB, TestPos, TestLoc, MI, *FlagUse, CondRegs);
-        } else if (MI.getOpcode() == TargetOpcode::COPY) {
+        } else if (MI.isCopy()) {
           rewriteCopy(MI, *FlagUse, CopyDefI);
         } else {
           // We assume all other instructions that use flags also def them.
           assert(MI.findRegisterDefOperand(X86::EFLAGS) &&
                  "Expected a def of EFLAGS for this instruction!");
 
           // NB!!! Several arithmetic instructions only *partially* update
           // flags. Theoretically, we could generate MI code sequences that
for (MachineInstr *CopyI : Copies) {
 
     // FIXME: Mark the last use of EFLAGS before the copy's def as a kill if
     // the copy's def operand is itself a kill.
   }
 
 #ifndef NDEBUG
   for (MachineBasicBlock &MBB : MF)
     for (MachineInstr &MI : MBB)
-      if (MI.getOpcode() == TargetOpcode::COPY &&
-          (MI.getOperand(0).getReg() == X86::EFLAGS ||
+      if (MI.isCopy() && (MI.getOperand(0).getReg() == X86::EFLAGS ||
                           MI.getOperand(1).getReg() == X86::EFLAGS)) {
         LLVM_DEBUG(dbgs() << "ERROR: Found a COPY involving EFLAGS: ";
                    MI.dump());
         llvm_unreachable("Unlowered EFLAGS copy!");
       }
 #endif
 
   return true;
 }

llvm/lib/Target/X86/X86FloatingPoint.cpp

void FPS::handleSpecialFP(MachineBasicBlock::iterator &Inst) {
 
   if (MI.isReturn()) {
     handleReturn(Inst);
     return;
   }
 
   switch (MI.getOpcode()) {
   default: llvm_unreachable("Unknown SpecialFP instruction!");
-  case TargetOpcode::COPY: {
+  case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY: {
     // We handle three kinds of copies: FP <- FP, FP <- ST, and ST <- FP.
     const MachineOperand &MO1 = MI.getOperand(1);
     const MachineOperand &MO0 = MI.getOperand(0);
     bool KillsSrc = MI.killsRegister(MO1.getReg());
 
     // FP <- FP copy.
     unsigned DstFP = getFPReg(MO0);
     unsigned SrcFP = getFPReg(MO1);

llvm/lib/Target/X86/X86InstrInfo.cpp

bool X86InstrInfo::isDataInvariant(MachineInstr &MI) {
   switch (MI.getOpcode()) {
   default:
     // By default, assume that the instruction is not data invariant.
     return false;
 
     // Some target-independent operations that trivially lower to data-invariant
     // instructions.
   case TargetOpcode::COPY:
+  case TargetOpcode::TCOPY:
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::SUBREG_TO_REG:
     return true;
 
   // On x86 it is believed that imul is constant time w.r.t. the loaded data.
   // However, they set flags and are perhaps the most surprisingly constant
   // time operations so we call them out here separately.
   case X86::IMUL16rr:

llvm/test/CodeGen/AArch64/callbr-asm-label.ll

 ; RUN: llc < %s -mtriple=aarch64-linux-gnu | FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-linux-gnu -O0 | FileCheck %s --check-prefix=CHECK-O0
 
 @X = common local_unnamed_addr global i32 0, align 4
 
 define i32 @test1() {
 ; CHECK-LABEL: test1:
 ; CHECK:         .word b
 ; CHECK-NEXT:    .word .Ltmp0
 ; CHECK-LABEL: .LBB0_1: // %cleanup
 ; CHECK-LABEL: .Ltmp0:
 ; CHECK-LABEL: .LBB0_2: // %indirect
+
+; CHECK-O0-LABEL:  test1:
+; CHECK-O0:          .word b
+; CHECK-O0-NEXT:     .word .Ltmp1
+; CHECK-O0-LABEL: .Ltmp1:
+; CHECK-O0-LABEL: .LBB0_1: // %indirect
+; CHECK-O0-LABEL: .LBB0_2: // %cleanup
 entry:
   callbr void asm sideeffect "1:\0A\09.word b, ${0:l}\0A\09", "X"(i8* blockaddress(@test1, %indirect))
           to label %cleanup [label %indirect]
 
 indirect:
   br label %cleanup
 
 cleanup:
   %retval.0 = phi i32 [ 1, %indirect ], [ 0, %entry ]
   ret i32 %retval.0
 }
 
 define void @test2() {
-; CHECK-LABEL: test2:
+; CHECK-LABEL:    test2:
+; CHECK-O0-LABEL: test2:
 entry:
   %0 = load i32, i32* @X, align 4
   %and = and i32 %0, 1
   %tobool = icmp eq i32 %and, 0
   br i1 %tobool, label %if.end10, label %if.then
 
 if.then:
 ; CHECK:       .word b
 ; CHECK-NEXT:  .word .Ltmp2
 ; CHECK-LABEL: .Ltmp2:
 ; CHECK-NEXT:  .LBB1_3: // %if.end6
+
+; CHECK-O0:       .word b
+; CHECK-O0-NEXT:  .word .Ltmp3
+; CHECK-O0-LABEL: .Ltmp3:
+; CHECK-O0-NEXT:  .LBB1_3: // %if.end6
   callbr void asm sideeffect "1:\0A\09.word b, ${0:l}\0A\09", "X"(i8* blockaddress(@test2, %if.end6))
           to label %if.then4 [label %if.end6]
 
 if.then4:
   %call5 = tail call i32 bitcast (i32 (...)* @g to i32 ()*)()
   br label %if.end6
 
 if.end6:
   %.pre = load i32, i32* @X, align 4
   %.pre13 = and i32 %.pre, 1
   %phitmp = icmp eq i32 %.pre13, 0
   br i1 %phitmp, label %if.end10, label %if.then9
 
 if.then9:
 ; CHECK-LABEL: .Ltmp4:
 ; CHECK-NEXT:  .LBB1_5: // %l_yes
+
+; CHECK-O0-LABEL:  .Ltmp5:
+; CHECK-O0-NEXT:   .LBB1_6: // %l_yes
   callbr void asm sideeffect "", "X"(i8* blockaddress(@test2, %l_yes))
           to label %if.end10 [label %l_yes]
 
 if.end10:
   br label %l_yes
 
 l_yes:
   ret void
 }
 
 declare i32 @g(...)

llvm/test/CodeGen/SystemZ/asm-20.ll

 ; Test that asm goto can be compiled.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 -O0
 
 define i32 @c() {
 entry:
   callbr void asm sideeffect "j d", "X"(i8* blockaddress(@c, %d))
           to label %asm.fallthrough [label %d]
 
 asm.fallthrough:               ; preds = %entry
   br label %d
 
 d:                             ; preds = %asm.fallthrough, %entry
   ret i32 undef
 }

llvm/test/CodeGen/X86/callbr-asm-label-addr.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -O0 | FileCheck %s --check-prefix=CHECK-O0
 
 define i32 @test1(i32 %x) {
 ; CHECK-LABEL: test1:
 ; CHECK:         .quad .Ltmp0
 ; CHECK-NEXT:    .quad .Ltmp1
 ; CHECK-LABEL: .Ltmp1:
 ; CHECK-LABEL: .LBB0_1: # %bar
 ; CHECK-NEXT:    callq foo
 ; CHECK-LABEL: .Ltmp0:
 ; CHECK-NEXT:  # %bb.2: # %baz
+
+; CHECK-O0-LABEL: test1:
+; CHECK-O0:         .quad .Ltmp0
+; CHECK-O0-NEXT:    .quad .Ltmp1
+; CHECK-O0-LABEL: .Ltmp1:
+; CHECK-O0-LABEL: .LBB0_2: # %bar
+; CHECK-O0-NEXT:    movl
+; CHECK-O0-NEXT:    callq foo
+; CHECK-O0-LABEL: .Ltmp0:
+; CHECK-O0-NEXT:  # %bb.3: # %baz
 entry:
   callbr void asm sideeffect ".quad ${0:l}\0A\09.quad ${1:l}", "i,X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test1, %baz), i8* blockaddress(@test1, %bar))
           to label %asm.fallthrough [label %bar]
 
 asm.fallthrough:
   br label %bar
 
 bar:

llvm/test/CodeGen/X86/callbr-asm-outputs-tcopy-spilling.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -O0 | FileCheck %s --check-prefix=CHECK
+
+%struct.kernel_rseq = type { i32, i32, i8*, i32, [12 x i8] }
+
+@__rseq_abi = external thread_local global %struct.kernel_rseq, align 32
+
+define i32 @test1(i8* %percpu_data, i64 %lock_value) #0 {
+; CHECK-LABEL: test1:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    movq %rdi, -16(%rbp)
+; CHECK-NEXT:    movq %rsi, -24(%rbp)
+; CHECK-NEXT:    movq -16(%rbp), %rax
+; CHECK-NEXT:    movq -24(%rbp), %rcx
+; CHECK-NEXT:    movq __rseq_abi@{{.*}}(%rip), %rdx
+; CHECK-NEXT:    movq %fs:0, %rsi
+; CHECK-NEXT:    leaq 8(%rsi,%rdx), %rdi
+; CHECK-NEXT:    leaq 4(%rsi,%rdx), %rdx
+; CHECK-NEXT:    #APP
+; CHECK-NEXT: .Ltmp1:
+; CHECK-NEXT:    leaq __rseq_cs_RseqFunction_PerCpuTryLock_0(%rip), %rsi
+; CHECK-NEXT:    movq %rsi, (%rdi)
+; CHECK-NEXT: .Ltmp2:
+; CHECK-NEXT:    movl (%rdx), %r8d
+; CHECK-NEXT:    movl %r8d, %esi
+; CHECK-NEXT:    shlq $12, %rsi
+; CHECK-NEXT:    addq %rax, %rsi
+; CHECK-NEXT:    cmpq $0, (%rsi)
+; CHECK-NEXT:    jne .Ltmp0
+; CHECK-NEXT:    movq %rcx, (%rsi)
+; CHECK-NEXT: .Ltmp3:
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    movl %r8d, -40(%rbp) # 4-byte Spill
+; CHECK-NEXT:    movq %rsi, -48(%rbp) # 8-byte Spill
+; CHECK-NEXT: # %bb.4: # %entry
+; CHECK-NEXT:    jmp .LBB0_1
+entry:
+  %retval = alloca i32, align 4
+  %percpu_data.addr = alloca i8*, align 8
+  %lock_value.addr = alloca i64, align 8
+  %scratch = alloca i64, align 8
+  %cpu = alloca i32, align 4
+  store i8* %percpu_data, i8** %percpu_data.addr, align 8
+  store i64 %lock_value, i64* %lock_value.addr, align 8
+  %0 = load i8*, i8** %percpu_data.addr, align 8
+  %1 = load i64, i64* %lock_value.addr, align 8
+  %2 = callbr { i64, i32 } asm "3:\0Alea __rseq_cs_RseqFunction_PerCpuTryLock_${:uid}(%rip), $0\0Amov $0, ($2)\0A4:\0Amov ($3), $1\0Amov $1, ${0:k}\0Ashl $5, $0\0Aadd $6, $0\0Acmpq $$0, ($0)\0Ajne ${8:l}\0Amov $7, ($0)\0A5:", "=&r,=&r,r,r,n,n,r,r,X,~{cc},~{memory},~{dirflag},~{fpsr},~{flags}"(i8** getelementptr inbounds (%struct.kernel_rseq, %struct.kernel_rseq* @__rseq_abi, i32 0, i32 2), i32* getelementptr inbounds (%struct.kernel_rseq, %struct.kernel_rseq* @__rseq_abi, i32 0, i32 1), i32 1392848979, i32 12, i8* %0, i64 %1, i8* blockaddress(@test1, %fail_contended)) #1
+          to label %asm.fallthrough [label %fail_contended]
+
+asm.fallthrough:                                  ; preds = %entry
+  %asmresult = extractvalue { i64, i32 } %2, 0
+  %asmresult1 = extractvalue { i64, i32 } %2, 1
+  store i64 %asmresult, i64* %scratch, align 8
+  store i32 %asmresult1, i32* %cpu, align 4
+  %3 = load i32, i32* %cpu, align 4
+  store i32 %3, i32* %retval, align 4
+  br label %return
+
+fail_contended:                                   ; preds = %entry
+  store i32 -1, i32* %retval, align 4
+  br label %return
+
+return:                                           ; preds = %fail_contended, %asm.fallthrough
+  %4 = load i32, i32* %retval, align 4
+  ret i32 %4
+}
+
+attributes #0 = { noinline nounwind optnone uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="all" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #1 = { nounwind }

llvm/test/CodeGen/X86/callbr-asm-outputs.ll

 ; CHECK-NEXT:    jge .LBB1_3
 ; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    testl %esi, %esi
 ; CHECK-NEXT:    testl %edi, %esi
 ; CHECK-NEXT:    jne .Ltmp1
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:  .LBB1_2: # %if.then
+; CHECK-NEXT:    addl %esi, %edi
 ; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    addl %esi, %eax
 ; CHECK-NEXT:  .Ltmp2: # Block address taken
 ; CHECK-NEXT:  .LBB1_6: # %return
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    popl %edi
 ; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 ; CHECK-NEXT:  .LBB1_3: # %if.else