Give option to use isCopyInstr to determine which MI is treated as Copy instruction in MCP

[adriantong1024]

Summary

This is then used in AArch64 to remove copy instructions after taildup
ran in machine block placement

For example.

$w9 = ORRWrs $wzr, $w8, 0 is used to do copy in AArch64. This is handled by isCopyInstr() implemented in Target/AArch64.

[adriantong1024]

Pass lamba to MachineCopyProp to determine which MI is treated as Copy instruction.

[adriantong1024]

Pass lamba to MachineCopyProp to determine which MI is treated as Copy instruction.

[adriantong1024]

Update a test case.

[craig.topper]

llvm/test/CodeGen/Thumb2/bti-indirect-branches.ll

12 ↗

(On Diff #428486)

This isn't related to this patch.

[craig.topper]

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

784

This feels a bit heavy to do for each instruction. The subtarget lookup has to query a hash table. MachineCopyProp already has TII. Why can't it call a virtual method on TII that AArch64 implements and have a default implementation that does whatever the previous behavior was.

Or you could pass a bool to the MachineCopyPropagation that tells it to use TII->isCopyInstr() and default the bool to false for other targets.

[adriantong1024]

Address Craig comment. Thanks for taking a look.

[adriantong1024]

Thanks for the review @craig.topper. I've updated the patch.

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

784

It seems getting subtarget is returning a pointer to STI.

https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/CodeGen/MachineFunction.h#L640

[craig.topper]

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

784

So it does. I guess we call getSubtargetImpl and cache it earlier. Guess I forgot how that works.

It's still very unusual to pass a lamba to a pass constructor. Are there other examples? We usually use TII to implement target behavior.

[adriantong1024]

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

784

https://github.com/llvm/llvm-project/blob/main/llvm/lib/CodeGen/IfConversion.cpp#L2364 is one example. There are another example in CodeGen/ I could not find now.

[dmgreen]

I thought you said you were just going to write a new pass for this? :)

It's still very unusual to pass a lamba to a pass constructor. Are there other examples? We usually use TII to implement target behavior.

fConversion.cpp is one example. There are another example in CodeGen/ I could not find now.

Yeah there are a few other examples of passing a lambda to a pass if you look at the ARMPassConfig. They usually take a Function though I think.

Would it be simpler, given the lambda that is passed from AArch64 at the moment, to just pass a boolean instead to change the behaviour between using TTI->isCopyInstr and MI.isCopy? Similar to the UseCopyInstr option.

llvm/test/CodeGen/AArch64/copyprop.mir

106

It might be worth adding an end-to-end .ll file test too, to show that the propagation happen later after MachineBlockPlacement. To show that part is working, and the make sure it doesn't regress again in the future.

[adriantong1024]

Thanks for the review @dmgreen.

In D125335#3505465, @dmgreen wrote:

I thought you said you were just going to write a new pass for this? :)

Feels like MCP can do the job and most likely has a better (than a new peephole) capability to do copy propagation.

It's still very unusual to pass a lamba to a pass constructor. Are there other examples? We usually use TII to implement target behavior.

fConversion.cpp is one example. There are another example in CodeGen/ I could not find now.

Yeah there are a few other examples of passing a lambda to a pass if you look at the ARMPassConfig. They usually take a Function though I think.

Would it be simpler, given the lambda that is passed from AArch64 at the moment, to just pass a boolean instead to change the behaviour between using TTI->isCopyInstr and MI.isCopy? Similar to the UseCopyInstr option.

Sure. I agree its a bit simpler.

llvm/test/CodeGen/AArch64/copyprop.mir

106

Make sense. Will do.

[adriantong1024]

Address comments

[craig.topper]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

310

Why do you need a lambda inside the pass? Can't you pass around the bool and TII? And have a static function that takes MI, TII, and the bool and does the right thing?

[craig.topper]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

310

Or make more of the static functions in the pass, member functions so you don't have to pass around TII and the bool/predicate.

[arsenm]

I don't understand the problem this is supposed to solve

llvm/lib/CodeGen/MachineCopyPropagation.cpp

309

I don't understand the meaning of this pass parameter

llvm/test/CodeGen/AArch64/O3-pipeline.ll

203

Was this not already run? Is this adding a second instance?

[adriantong1024]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

310

Yes. I thought about it and that has the cost of passing 2 arguments around. while this has the cost of retrieving TII from MI.

llvm/test/CodeGen/AArch64/O3-pipeline.ll

203

Hi @arsenm, Yes this is running the 2nd instance. We noticed some redundant copy instructions after tail duplication in machine block placement, so we want to run MCP to get rid of them.

[craig.topper]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

310

I think the 2 arguments wins since the predicate function is an argument and you're already passing TII to every function that gets the predicate. So replacing the predicate with a bool doesn't change the number of arguments.

[adriantong1024]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

310

Sure. let me get rid of the predicate. Thanks for the suggestion.

[adriantong1024]

Address comments.

[craig.topper]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

94

Drop curly braces

98

Drop curly braces

140–146

Can you write this assert using CopyOperands instead of calling isCopyInstr twice?

183

Use CopyOperands in the assert instead of calling isCopyInstr twice

249

Put this blank line back

425–428

Check CopyOperands instead of calling isCopyInstr twice

659

Don't call isCopyInstr twice

828–829

This assert is useless. You already accessed the Source and Destination above, which would assert if the isCopyInstr had returned None.

849

Don't call isCopyInstr twice

930

Don't call isCopyInstr twice

[adriantong1024]

Address comment. Thanks @craig.topper for the time and review.

[craig.topper]

Please update the title. There’s no lambda anymore

[lkail]

Could you please describe briefly what copy-like instructions MCP is missing in the summary?

llvm/lib/CodeGen/MachineCopyPropagation.cpp

89

Why do we need this option? Can we extend MCP to check TII.isCopyInstr by default?

[adriantong1024]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

89

I've tried to use isCopyInstr by default, but there are cases which it does not work for some architecture.

e.g. ARM
$lr = tMOVr killed $r1, 14, $noreg
tBX_RET 14, $noreg, implicit $r0

isCopyInstr sees tMOVr as a COPY instruction, but the use of $lr is not represented in tBX_RET, so the tMOVr is eliminated.

So far such problems have not been seen in AArch64.

[lkail]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

89

I mean first we can check MI.isCopy(), if MI.isCopy() returns false, we do further check via TII->isCopyInstr(MI). For the initial draft, we can do simple feasibility check(sufficient to cover your case) on MI which TII->isCopyInstr(MI) returns true.

[adriantong1024]

llvm/lib/CodeGen/MachineCopyPropagation.cpp

89

This is what isCopyInstr does (https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/CodeGen/TargetInstrInfo.h#L1025)

The problem (my concern) here is some architecture may not be ready to use isCopyInstr to identify what a COPY is. We could slowly enable it for more and more architectures once this option in place.

[lkail]

The problem (my concern) here is some architecture may not be ready to use isCopyInstr to identify what a COPY is.

IIUC, isCopyInstrImpl hasn't been override by many targets and returns None, I think it doesn't affect these targets in MCP. Even when MI.isCopy() returns false, and then isCopyInstr returns true, you still have chance to decide if it's feasible to perform propagation based on this MI's operands' flags and etc. As I have said, you can implement conservative checks in your first patch(sufficient to cover your cases) and enhance it in following patches.

[adriantong1024]

In D125335#3519829, @lkail wrote:

The problem (my concern) here is some architecture may not be ready to use isCopyInstr to identify what a COPY is.

IIUC, isCopyInstrImpl hasn't been override by many targets and returns None, I think it doesn't affect these targets in MCP. Even when MI.isCopy() returns false, and then isCopyInstr returns true, you still have chance to decide if it's feasible to perform propagation based on this MI's operands' flags and etc. As I have said, you can implement conservative checks in your first patch(sufficient to cover your cases) and enhance it in following patches.

Hi Kai

I agree that isCopyInstrImpl has not been overriden by many targets. However, in case of ARM, I have observed a problem calling MCP with isCopyInstr(). While this could be fixed by checking additional things in MCP, but this would make MCP more complex and target specific.

Another way is to implement specific AArch64 optimizations to handle my case, but I feel MCP is pretty well written and would be great if we can re-use its code.

Right now MCP+MI.isCopy() runs for all architecture, but MCP+isCopyInstr only runs for AArch64. This could be extended to run on other architectures once tested on the architectures.

[dmgreen]

I was taking a look at some of the ARM/Thumb tests that were failing if this used TII.isCopyInstr. There are definitely some problems happening in there, with undefined register.

llvm/include/llvm/CodeGen/Passes.h

29

Is this not needed any more?

llvm/lib/CodeGen/MachineCopyPropagation.cpp

86

I don't think this is used either.

657–666

I would use Optional<DestSourcePair> as opposed to auto.

llvm/test/CodeGen/AArch64/copyprop.ll

2

Can you use update_llc_test_checks on the file, to show the full output.

[adriantong1024]

Address comment from @dmgreen. Thanks for the review.

[arsenm]

Can you avoid adding a second run by moving the current run? The ad-hoc physreg liveness tracking after allocation is really expensive

llvm/lib/CodeGen/MachineCopyPropagation.cpp

89

I am against adding any kind of option here. These are totally undiscoverable and we have too many random flags and hooks out of fear of changing anything on any other architecture

[adriantong1024]

In D125335#3522676, @arsenm wrote:

Can you avoid adding a second run by moving the current run? The ad-hoc physreg liveness tracking after allocation is really expensive

Hi @arsenm

Thanks for the review. I tried to move the current pass after machine block placement. The problem with this is machine block placement runs after Post-RA pseudo instruction expansion pass and postrapseudos changes COPY instruction to real instructions, MCP cant handle real instructions with MI.isCopy(). e.g. test/CodeGen/ARM/sadd_sat_plus.ll.

Before postrapseudos
bb.4 (%ir-block.0):
liveins: $r0, $r3
renamable $r2 = COPY killed renamable $r3
$r1 = COPY killed renamable $r2
tPOP_RET 14, $noreg, def $r4, def $pc, implicit-def $sp, implicit $sp, implicit $r0, implicit $r1

After postrapseudos
bb.4 (%ir-block.0):
liveins: $r0, $r3
$r2 = tMOVr killed $r3, 14, $noreg
$r1 = tMOVr killed $r2, 14, $noreg
tPOP_RET 14, $noreg, def $r4, def $pc, implicit-def $sp, implicit $sp, implicit $r0, implicit $r1

After MCP
bb.4 (%ir-block.0):
liveins: $r0, $r3
$r2 = tMOVr killed $r3, 14, $noreg
$r1 = tMOVr killed $r2, 14, $noreg
tPOP_RET 14, $noreg, def $r4, def $pc, implicit-def $sp, implicit $sp, implicit $r0, implicit $r1

Right now this additional pass only happens on AArch64 and its removing things that could not be done with the first pass of MCP.

[adriantong1024]

@craig.topper @dmgreen @arsenm Please advise if there is anything else we want to change ?

Thanks for the review!

[dmgreen]

I hadn't expected MachineCopyPropgation would be too expensive - it sounds like a simple concept and seems to already be run twice in the default pipeline. This is only adding an extra at -O3, so hopefully won't be too bad for compile time.

Perhaps it is worth running some quick tests though? And if it is a problem adding disablePass(&MachineCopyPropagationID); for AArch64, to run just at the new pass location, not the old. I'm not sure if that would have any knock-on effects, disabling the two existing runs of the pass earlier in the pipeline.

[adriantong1024]

In D125335#3531767, @dmgreen wrote:

I hadn't expected MachineCopyPropgation would be too expensive - it sounds like a simple concept and seems to already be run twice in the default pipeline. This is only adding an extra at -O3, so hopefully won't be too bad for compile time.

Perhaps it is worth running some quick tests though? And if it is a problem adding disablePass(&MachineCopyPropagationID); for AArch64, to run just at the new pass location, not the old. I'm not sure if that would have any knock-on effects, disabling the two existing runs of the pass earlier in the pipeline.

Thanks for the comment @dmgreen. I will run some workload with and without the additional pass in AArch64 and report compilation time here.

[adriantong1024]

Hi

I compiled gcc.c from https://people.csail.mit.edu/smcc/projects/single-file-programs/ and here are the results. This is compiling for ARM64 on an ARM64 machine.

Overall I feel MCP is taking a very small percentage of the total compilation time.

TRUNK:

ytbcp30:~# time ./clang.trunk -O3 gcc.i -c -ftime-report |& grep "Machine Copy Propagation Pass"

0.2181 (  0.5%)   0.0087 (  0.2%)   0.2267 (  0.5%)   0.2279 (  0.5%)  Machine Copy Propagation Pass
0.1829 (  0.4%)   0.0092 (  0.2%)   0.1921 (  0.4%)   0.1928 (  0.4%)  Machine Copy Propagation Pass #2

real 2m7.288s
user 2m0.292s
sys 0m5.951s

PATCHED:

ytbcp30:~# time ./clang.patched -O3 gcc.i -c -ftime-report |& grep "Machine Copy Propagation Pass"

0.2162 (  0.5%)   0.0099 (  0.2%)   0.2261 (  0.5%)   0.2272 (  0.5%)  Machine Copy Propagation Pass
0.2071 (  0.5%)   0.0081 (  0.2%)   0.2153 (  0.5%)   0.2161 (  0.5%)  Machine Copy Propagation Pass #3
0.1811 (  0.4%)   0.0098 (  0.2%)   0.1909 (  0.4%)   0.1915 (  0.4%)  Machine Copy Propagation Pass #2

real 2m6.928s
user 2m0.060s
sys 0m6.070s

[dmgreen]

So it looks like roughly a 0.2/126 = 0.16% increase in compile time? I think that is OK for -O3, even though the expected gains will likely not be very large. It did come up as an improvement in the tests I ran though.

The patch looks OK to me. LGTM, thanks

[mstorsjo]

This change caused a regression; when compiling with -mbranch-protection=standard, this change causes the generated code to crash, due to using registers that have been clobbered by a call to _setjmp. See https://github.com/llvm/llvm-project/issues/73787 for more detailed analysis of the issue.

llvm/include/llvm/CodeGen/Passes.h

 #include <functional>
 #include <string>
 
 namespace llvm {
 
 class FunctionPass;
 class MachineFunction;
 class MachineFunctionPass;
 class MemoryBuffer;
 class ModulePass;
 class Pass;
 class TargetMachine;
 class raw_ostream;
 
 } // End llvm namespace
 
 // List of target independent CodeGen pass IDs.
namespace llvm {
 
   /// MachineSinking - This pass performs sinking on machine instructions.
   extern char &MachineSinkingID;
 
   /// MachineCopyPropagation - This pass performs copy propagation on
   /// machine instructions.
   extern char &MachineCopyPropagationID;
 
+  MachineFunctionPass *createMachineCopyPropagationPass(bool UseCopyInstr);
+
   /// PeepholeOptimizer - This pass performs peephole optimizations -
   /// like extension and comparison eliminations.
   extern char &PeepholeOptimizerID;
 
   /// OptimizePHIs - This pass optimizes machine instruction PHIs
   /// to take advantage of opportunities created during DAG legalization.
   extern char &OptimizePHIsID;
 

llvm/lib/CodeGen/MachineCopyPropagation.cpp

 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugCounter.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <iterator>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "machine-cp"
 
 STATISTIC(NumDeletes, "Number of dead copies deleted");
 STATISTIC(NumCopyForwards, "Number of copy uses forwarded");
 STATISTIC(NumCopyBackwardPropagated, "Number of copy defs backward propagated");
 DEBUG_COUNTER(FwdCounter, "machine-cp-fwd",
               "Controls which register COPYs are forwarded");
 
+static cl::opt<bool> MCPUseCopyInstr("mcp-use-is-copy-instr", cl::init(false),
+                                     cl::Hidden);
+
 namespace {
 
+static Optional<DestSourcePair> isCopyInstr(const MachineInstr &MI,
+                                            const TargetInstrInfo &TII,
+                                            bool UseCopyInstr) {
+  if (UseCopyInstr)
+    return TII.isCopyInstr(MI);
+
+  if (MI.isCopy())
+    return Optional<DestSourcePair>(
+        DestSourcePair{MI.getOperand(0), MI.getOperand(1)});
+
+  return None;
+}
+
 class CopyTracker {
   struct CopyInfo {
     MachineInstr *MI;
     SmallVector<MCRegister, 4> DefRegs;
     bool Avail;
   };
 
   DenseMap<MCRegister, CopyInfo> Copies;
for (MCRegister Reg : Regs) {
         auto CI = Copies.find(*RUI);
         if (CI != Copies.end())
           CI->second.Avail = false;
       }
     }
   }
 
   /// Remove register from copy maps.
-  void invalidateRegister(MCRegister Reg, const TargetRegisterInfo &TRI) {
+  void invalidateRegister(MCRegister Reg, const TargetRegisterInfo &TRI,
+                          const TargetInstrInfo &TII, bool UseCopyInstr) {
     // Since Reg might be a subreg of some registers, only invalidate Reg is not
     // enough. We have to find the COPY defines Reg or registers defined by Reg
     // and invalidate all of them.
     SmallSet<MCRegister, 8> RegsToInvalidate;
     RegsToInvalidate.insert(Reg);
     for (MCRegUnitIterator RUI(Reg, &TRI); RUI.isValid(); ++RUI) {
       auto I = Copies.find(*RUI);
       if (I != Copies.end()) {
         if (MachineInstr *MI = I->second.MI) {
-          RegsToInvalidate.insert(MI->getOperand(0).getReg().asMCReg());
-          RegsToInvalidate.insert(MI->getOperand(1).getReg().asMCReg());
+          Optional<DestSourcePair> CopyOperands =
+              isCopyInstr(*MI, TII, UseCopyInstr);
+          assert(CopyOperands && "Expect copy");
+
+          RegsToInvalidate.insert(
+              CopyOperands->Destination->getReg().asMCReg());
+          RegsToInvalidate.insert(CopyOperands->Source->getReg().asMCReg());
         }
         RegsToInvalidate.insert(I->second.DefRegs.begin(),
                                 I->second.DefRegs.end());
       }
     }
     for (MCRegister InvalidReg : RegsToInvalidate)
       for (MCRegUnitIterator RUI(InvalidReg, &TRI); RUI.isValid(); ++RUI)
         Copies.erase(*RUI);
   }
 
   /// Clobber a single register, removing it from the tracker's copy maps.
-  void clobberRegister(MCRegister Reg, const TargetRegisterInfo &TRI) {
+  void clobberRegister(MCRegister Reg, const TargetRegisterInfo &TRI,
+                       const TargetInstrInfo &TII, bool UseCopyInstr) {
     for (MCRegUnitIterator RUI(Reg, &TRI); RUI.isValid(); ++RUI) {
       auto I = Copies.find(*RUI);
       if (I != Copies.end()) {
         // When we clobber the source of a copy, we need to clobber everything
         // it defined.
         markRegsUnavailable(I->second.DefRegs, TRI);
         // When we clobber the destination of a copy, we need to clobber the
         // whole register it defined.
-        if (MachineInstr *MI = I->second.MI)
-          markRegsUnavailable({MI->getOperand(0).getReg().asMCReg()}, TRI);
+        if (MachineInstr *MI = I->second.MI) {
+          Optional<DestSourcePair> CopyOperands =
+              isCopyInstr(*MI, TII, UseCopyInstr);
+          markRegsUnavailable({CopyOperands->Destination->getReg().asMCReg()},
+                              TRI);
+        }
         // Now we can erase the copy.
         Copies.erase(I);
       }
     }
   }
 
   /// Add this copy's registers into the tracker's copy maps.
-  void trackCopy(MachineInstr *MI, const TargetRegisterInfo &TRI) {
-    assert(MI->isCopy() && "Tracking non-copy?");
+  void trackCopy(MachineInstr *MI, const TargetRegisterInfo &TRI,
+                 const TargetInstrInfo &TII, bool UseCopyInstr) {
+    Optional<DestSourcePair> CopyOperands = isCopyInstr(*MI, TII, UseCopyInstr);
+    assert(CopyOperands && "Tracking non-copy?");
 
-    MCRegister Def = MI->getOperand(0).getReg().asMCReg();
-    MCRegister Src = MI->getOperand(1).getReg().asMCReg();
+    MCRegister Src = CopyOperands->Source->getReg().asMCReg();
+    MCRegister Def = CopyOperands->Destination->getReg().asMCReg();
 
     // Remember Def is defined by the copy.
     for (MCRegUnitIterator RUI(Def, &TRI); RUI.isValid(); ++RUI)
       Copies[*RUI] = {MI, {}, true};
 
     // Remember source that's copied to Def. Once it's clobbered, then
     // it's no longer available for copy propagation.
     for (MCRegUnitIterator RUI(Src, &TRI); RUI.isValid(); ++RUI) {
if (CI == Copies.end())
       return nullptr;
     if (CI->second.DefRegs.size() != 1)
       return nullptr;
     MCRegUnitIterator RUI(CI->second.DefRegs[0], &TRI);
     return findCopyForUnit(*RUI, TRI, true);
   }
 
   MachineInstr *findAvailBackwardCopy(MachineInstr &I, MCRegister Reg,
-                                      const TargetRegisterInfo &TRI) {
+                                      const TargetRegisterInfo &TRI,
+                                      const TargetInstrInfo &TII,
+                                      bool UseCopyInstr) {
     MCRegUnitIterator RUI(Reg, &TRI);
     MachineInstr *AvailCopy = findCopyDefViaUnit(*RUI, TRI);
-    if (!AvailCopy ||
-        !TRI.isSubRegisterEq(AvailCopy->getOperand(1).getReg(), Reg))
+
+    if (!AvailCopy)
+      return nullptr;
+
+    Optional<DestSourcePair> CopyOperands =
+        isCopyInstr(*AvailCopy, TII, UseCopyInstr);
+    Register AvailSrc = CopyOperands->Source->getReg();
+    Register AvailDef = CopyOperands->Destination->getReg();
+    if (!TRI.isSubRegisterEq(AvailSrc, Reg))
       return nullptr;
 
-    Register AvailSrc = AvailCopy->getOperand(1).getReg();
-    Register AvailDef = AvailCopy->getOperand(0).getReg();
     for (const MachineInstr &MI :
          make_range(AvailCopy->getReverseIterator(), I.getReverseIterator()))
       for (const MachineOperand &MO : MI.operands())
         if (MO.isRegMask())
           // FIXME: Shall we simultaneously invalidate AvailSrc or AvailDef?
           if (MO.clobbersPhysReg(AvailSrc) || MO.clobbersPhysReg(AvailDef))
             return nullptr;
 
     return AvailCopy;
   }
 
   MachineInstr *findAvailCopy(MachineInstr &DestCopy, MCRegister Reg,
-                              const TargetRegisterInfo &TRI) {
+                              const TargetRegisterInfo &TRI,
+                              const TargetInstrInfo &TII, bool UseCopyInstr) {
     // We check the first RegUnit here, since we'll only be interested in the
     // copy if it copies the entire register anyway.
     MCRegUnitIterator RUI(Reg, &TRI);
     MachineInstr *AvailCopy =
         findCopyForUnit(*RUI, TRI, /*MustBeAvailable=*/true);
-    if (!AvailCopy ||
-        !TRI.isSubRegisterEq(AvailCopy->getOperand(0).getReg(), Reg))
+
+    if (!AvailCopy)
+      return nullptr;
+
+    Optional<DestSourcePair> CopyOperands =
+        isCopyInstr(*AvailCopy, TII, UseCopyInstr);
+    Register AvailSrc = CopyOperands->Source->getReg();
+    Register AvailDef = CopyOperands->Destination->getReg();
+    if (!TRI.isSubRegisterEq(AvailDef, Reg))
       return nullptr;
 
     // Check that the available copy isn't clobbered by any regmasks between
     // itself and the destination.
-    Register AvailSrc = AvailCopy->getOperand(1).getReg();
-    Register AvailDef = AvailCopy->getOperand(0).getReg();
     for (const MachineInstr &MI :
          make_range(AvailCopy->getIterator(), DestCopy.getIterator()))
       for (const MachineOperand &MO : MI.operands())
         if (MO.isRegMask())
           if (MO.clobbersPhysReg(AvailSrc) || MO.clobbersPhysReg(AvailDef))
             return nullptr;
 
     return AvailCopy;
   }
 
   void clear() {
     Copies.clear();
   }
 };
 
 class MachineCopyPropagation : public MachineFunctionPass {
   const TargetRegisterInfo *TRI;
   const TargetInstrInfo *TII;
   const MachineRegisterInfo *MRI;
 
+  // Return true if this is a copy instruction and false otherwise.
+  bool UseCopyInstr;
+
 public:
   static char ID; // Pass identification, replacement for typeid
 
-  MachineCopyPropagation() : MachineFunctionPass(ID) {
+  MachineCopyPropagation(bool CopyInstr = false)
+      : MachineFunctionPass(ID), UseCopyInstr(CopyInstr || MCPUseCopyInstr) {
     initializeMachineCopyPropagationPass(*PassRegistry::getPassRegistry());
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
     MachineFunctionPass::getAnalysisUsage(AU);
   }
 
   bool runOnMachineFunction(MachineFunction &MF) override;
 
   MachineFunctionProperties getRequiredProperties() const override {
     return MachineFunctionProperties().set(
         MachineFunctionProperties::Property::NoVRegs);

 
 /// Return true if \p PreviousCopy did copy register \p Src to register \p Def.
 /// This fact may have been obscured by sub register usage or may not be true at
 /// all even though Src and Def are subregisters of the registers used in
 /// PreviousCopy. e.g.
 /// isNopCopy("ecx = COPY eax", AX, CX) == true
 /// isNopCopy("ecx = COPY eax", AH, CL) == false
 static bool isNopCopy(const MachineInstr &PreviousCopy, MCRegister Src,
-                      MCRegister Def, const TargetRegisterInfo *TRI) {
-  MCRegister PreviousSrc = PreviousCopy.getOperand(1).getReg().asMCReg();
-  MCRegister PreviousDef = PreviousCopy.getOperand(0).getReg().asMCReg();
+                      MCRegister Def, const TargetRegisterInfo *TRI,
+                      const TargetInstrInfo *TII, bool UseCopyInstr) {
+
+  Optional<DestSourcePair> CopyOperands =
+      isCopyInstr(PreviousCopy, *TII, UseCopyInstr);
+  MCRegister PreviousSrc = CopyOperands->Source->getReg().asMCReg();
+  MCRegister PreviousDef = CopyOperands->Destination->getReg().asMCReg();
   if (Src == PreviousSrc && Def == PreviousDef)
     return true;
   if (!TRI->isSubRegister(PreviousSrc, Src))
     return false;
   unsigned SubIdx = TRI->getSubRegIndex(PreviousSrc, Src);
   return SubIdx == TRI->getSubRegIndex(PreviousDef, Def);
 }
 
 /// Remove instruction \p Copy if there exists a previous copy that copies the
 /// register \p Src to the register \p Def; This may happen indirectly by
 /// copying the super registers.
 bool MachineCopyPropagation::eraseIfRedundant(MachineInstr &Copy,
                                               MCRegister Src, MCRegister Def) {
   // Avoid eliminating a copy from/to a reserved registers as we cannot predict
   // the value (Example: The sparc zero register is writable but stays zero).
   if (MRI->isReserved(Src) || MRI->isReserved(Def))
     return false;
 
   // Search for an existing copy.
-  MachineInstr *PrevCopy = Tracker.findAvailCopy(Copy, Def, *TRI);
+  MachineInstr *PrevCopy =
+      Tracker.findAvailCopy(Copy, Def, *TRI, *TII, UseCopyInstr);
   if (!PrevCopy)
     return false;
 
+  auto PrevCopyOperands = isCopyInstr(*PrevCopy, *TII, UseCopyInstr);
   // Check that the existing copy uses the correct sub registers.
-  if (PrevCopy->getOperand(0).isDead())
+  if (PrevCopyOperands->Destination->isDead())
     return false;
-  if (!isNopCopy(*PrevCopy, Src, Def, TRI))
+  if (!isNopCopy(*PrevCopy, Src, Def, TRI, TII, UseCopyInstr))
     return false;
 
   LLVM_DEBUG(dbgs() << "MCP: copy is a NOP, removing: "; Copy.dump());
 
   // Copy was redundantly redefining either Src or Def. Remove earlier kill
   // flags between Copy and PrevCopy because the value will be reused now.
-  assert(Copy.isCopy());
-  Register CopyDef = Copy.getOperand(0).getReg();
+  Optional<DestSourcePair> CopyOperands = isCopyInstr(Copy, *TII, UseCopyInstr);
+  assert(CopyOperands);
+
+  Register CopyDef = CopyOperands->Destination->getReg();
   assert(CopyDef == Src || CopyDef == Def);
   for (MachineInstr &MI :
        make_range(PrevCopy->getIterator(), Copy.getIterator()))
     MI.clearRegisterKills(CopyDef, TRI);
 
   Copy.eraseFromParent();
   Changed = true;
   ++NumDeletes;
   return true;
 }
 
 bool MachineCopyPropagation::isBackwardPropagatableRegClassCopy(
     const MachineInstr &Copy, const MachineInstr &UseI, unsigned UseIdx) {
-  Register Def = Copy.getOperand(0).getReg();
+
+  Optional<DestSourcePair> CopyOperands = isCopyInstr(Copy, *TII, UseCopyInstr);
+  Register Def = CopyOperands->Destination->getReg();
 
   if (const TargetRegisterClass *URC =
           UseI.getRegClassConstraint(UseIdx, TII, TRI))
     return URC->contains(Def);
 
   // We don't process further if UseI is a COPY, since forward copy propagation
   // should handle that.
   return false;
 }
 
 /// Decide whether we should forward the source of \param Copy to its use in
 /// \param UseI based on the physical register class constraints of the opcode
 /// and avoiding introducing more cross-class COPYs.
 bool MachineCopyPropagation::isForwardableRegClassCopy(const MachineInstr &Copy,
                                                        const MachineInstr &UseI,
                                                        unsigned UseIdx) {
 
-  Register CopySrcReg = Copy.getOperand(1).getReg();
+  Optional<DestSourcePair> CopyOperands = isCopyInstr(Copy, *TII, UseCopyInstr);
+  Register CopySrcReg = CopyOperands->Source->getReg();
 
   // If the new register meets the opcode register constraints, then allow
   // forwarding.
   if (const TargetRegisterClass *URC =
           UseI.getRegClassConstraint(UseIdx, TII, TRI))
     return URC->contains(CopySrcReg);
 
-  if (!UseI.isCopy())
+  auto UseICopyOperands = isCopyInstr(UseI, *TII, UseCopyInstr);
+  if (!UseICopyOperands)
     return false;
 
   /// COPYs don't have register class constraints, so if the user instruction
   /// is a COPY, we just try to avoid introducing additional cross-class
   /// COPYs.  For example:
   ///
   ///   RegClassA = COPY RegClassB  // Copy parameter
   ///   ...
   ///   RegClassB = COPY RegClassA  // UseI parameter
   ///
   /// which after forwarding becomes
   ///
   ///   RegClassA = COPY RegClassB
   ///   ...
   ///   RegClassB = COPY RegClassB
   ///
   /// so we have reduced the number of cross-class COPYs and potentially
   /// introduced a nop COPY that can be removed.
 
   // Allow forwarding if src and dst belong to any common class, so long as they
   // don't belong to any (possibly smaller) common class that requires copies to
   // go via a different class.
-  Register UseDstReg = UseI.getOperand(0).getReg();
+  Register UseDstReg = UseICopyOperands->Destination->getReg();
   bool Found = false;
   bool IsCrossClass = false;
   for (const TargetRegisterClass *RC : TRI->regclasses()) {
     if (RC->contains(CopySrcReg) && RC->contains(UseDstReg)) {
       Found = true;
       if (TRI->getCrossCopyRegClass(RC) != RC) {
         IsCrossClass = true;
         break;
       }
     }
   }
   if (!Found)
     return false;
   if (!IsCrossClass)
     return true;
   // The forwarded copy would be cross-class. Only do this if the original copy
   // was also cross-class.
-  Register CopyDstReg = Copy.getOperand(0).getReg();
+  Register CopyDstReg = CopyOperands->Destination->getReg();
   for (const TargetRegisterClass *RC : TRI->regclasses()) {
     if (RC->contains(CopySrcReg) && RC->contains(CopyDstReg) &&
         TRI->getCrossCopyRegClass(RC) != RC)
       return true;
   }
   return false;
 }
 
if (!MOUse.getReg())
       continue;
 
     // Check that the register is marked 'renamable' so we know it is safe to
     // rename it without violating any constraints that aren't expressed in the
     // IR (e.g. ABI or opcode requirements).
     if (!MOUse.isRenamable())
       continue;
 
-    MachineInstr *Copy =
-        Tracker.findAvailCopy(MI, MOUse.getReg().asMCReg(), *TRI);
+    MachineInstr *Copy = Tracker.findAvailCopy(MI, MOUse.getReg().asMCReg(),
+                                               *TRI, *TII, UseCopyInstr);
     if (!Copy)
       continue;
 
-    Register CopyDstReg = Copy->getOperand(0).getReg();
-    const MachineOperand &CopySrc = Copy->getOperand(1);
+    Optional<DestSourcePair> CopyOperands =
+        isCopyInstr(*Copy, *TII, UseCopyInstr);
+    Register CopyDstReg = CopyOperands->Destination->getReg();
+    const MachineOperand &CopySrc = *CopyOperands->Source;
     Register CopySrcReg = CopySrc.getReg();
 
     // FIXME: Don't handle partial uses of wider COPYs yet.
     if (MOUse.getReg() != CopyDstReg) {
       LLVM_DEBUG(
           dbgs() << "MCP: FIXME! Not forwarding COPY to sub-register use:\n  "
                  << MI);
       continue;
     }
 
     // Don't forward COPYs of reserved regs unless they are constant.
     if (MRI->isReserved(CopySrcReg) && !MRI->isConstantPhysReg(CopySrcReg))
       continue;
 
     if (!isForwardableRegClassCopy(*Copy, MI, OpIdx))
       continue;
 
     if (hasImplicitOverlap(MI, MOUse))
       continue;
 
     // Check that the instruction is not a copy that partially overwrites the
     // original copy source that we are about to use. The tracker mechanism
     // cannot cope with that.
-    if (MI.isCopy() && MI.modifiesRegister(CopySrcReg, TRI) &&
+    if (isCopyInstr(MI, *TII, UseCopyInstr) &&
+        MI.modifiesRegister(CopySrcReg, TRI) &&
         !MI.definesRegister(CopySrcReg)) {
       LLVM_DEBUG(dbgs() << "MCP: Copy source overlap with dest in " << MI);
       continue;
     }
 
     if (!DebugCounter::shouldExecute(FwdCounter)) {
       LLVM_DEBUG(dbgs() << "MCP: Skipping forwarding due to debug counter:\n  "
                         << MI);

 }
 
 void MachineCopyPropagation::ForwardCopyPropagateBlock(MachineBasicBlock &MBB) {
   LLVM_DEBUG(dbgs() << "MCP: ForwardCopyPropagateBlock " << MBB.getName()
                     << "\n");
 
   for (MachineInstr &MI : llvm::make_early_inc_range(MBB)) {
     // Analyze copies (which don't overlap themselves).
-    if (MI.isCopy() && !TRI->regsOverlap(MI.getOperand(0).getReg(),
-                                         MI.getOperand(1).getReg())) {
-      assert(MI.getOperand(0).getReg().isPhysical() &&
-             MI.getOperand(1).getReg().isPhysical() &&
+    Optional<DestSourcePair> CopyOperands = isCopyInstr(MI, *TII, UseCopyInstr);
+    if (CopyOperands) {
+
+      Register RegSrc = CopyOperands->Source->getReg();
+      Register RegDef = CopyOperands->Destination->getReg();
+
+      if (TRI->regsOverlap(RegDef, RegSrc))
+        continue;
+
+      assert(RegDef.isPhysical() && RegSrc.isPhysical() &&
              "MachineCopyPropagation should be run after register allocation!");
 
-      MCRegister Def = MI.getOperand(0).getReg().asMCReg();
-      MCRegister Src = MI.getOperand(1).getReg().asMCReg();
+      MCRegister Def = RegDef.asMCReg();
+      MCRegister Src = RegSrc.asMCReg();
 
       // The two copies cancel out and the source of the first copy
       // hasn't been overridden, eliminate the second one. e.g.
       //  %ecx = COPY %eax
       //  ... nothing clobbered eax.
       //  %eax = COPY %ecx
       // =>
       //  %ecx = COPY %eax
       //
       // or
       //
       //  %ecx = COPY %eax
       //  ... nothing clobbered eax.
       //  %ecx = COPY %eax
       // =>
       //  %ecx = COPY %eax
       if (eraseIfRedundant(MI, Def, Src) || eraseIfRedundant(MI, Src, Def))
         continue;
 
       forwardUses(MI);
 
       // Src may have been changed by forwardUses()
-      Src = MI.getOperand(1).getReg().asMCReg();
+      CopyOperands = isCopyInstr(MI, *TII, UseCopyInstr);
+      Src = CopyOperands->Source->getReg().asMCReg();
 
       // If Src is defined by a previous copy, the previous copy cannot be
       // eliminated.
       ReadRegister(Src, MI, RegularUse);
       for (const MachineOperand &MO : MI.implicit_operands()) {
         if (!MO.isReg() || !MO.readsReg())
           continue;
         MCRegister Reg = MO.getReg().asMCReg();
if (CopyOperands) {
 
       // If 'Def' is previously source of another copy, then this earlier copy's
       // source is no longer available. e.g.
       // %xmm9 = copy %xmm2
       // ...
       // %xmm2 = copy %xmm0
       // ...
       // %xmm2 = copy %xmm9
-      Tracker.clobberRegister(Def, *TRI);
+      Tracker.clobberRegister(Def, *TRI, *TII, UseCopyInstr);
       for (const MachineOperand &MO : MI.implicit_operands()) {
         if (!MO.isReg() || !MO.isDef())
           continue;
         MCRegister Reg = MO.getReg().asMCReg();
         if (!Reg)
           continue;
-        Tracker.clobberRegister(Reg, *TRI);
+        Tracker.clobberRegister(Reg, *TRI, *TII, UseCopyInstr);
       }
 
-      Tracker.trackCopy(&MI, *TRI);
+      Tracker.trackCopy(&MI, *TRI, *TII, UseCopyInstr);
 
       continue;
     }
 
     // Clobber any earlyclobber regs first.
     for (const MachineOperand &MO : MI.operands())
       if (MO.isReg() && MO.isEarlyClobber()) {
         MCRegister Reg = MO.getReg().asMCReg();
         // If we have a tied earlyclobber, that means it is also read by this
         // instruction, so we need to make sure we don't remove it as dead
         // later.
         if (MO.isTied())
           ReadRegister(Reg, MI, RegularUse);
-        Tracker.clobberRegister(Reg, *TRI);
+        Tracker.clobberRegister(Reg, *TRI, *TII, UseCopyInstr);
       }
 
     forwardUses(MI);
 
     // Not a copy.
     SmallVector<Register, 2> Defs;
     const MachineOperand *RegMask = nullptr;
     for (const MachineOperand &MO : MI.operands()) {
for (MachineInstr &MI : llvm::make_early_inc_range(MBB)) {
     // a large set of registers.  Treat clobbered registers the same way as
     // defined registers.
     if (RegMask) {
       // Erase any MaybeDeadCopies whose destination register is clobbered.
       for (SmallSetVector<MachineInstr *, 8>::iterator DI =
                MaybeDeadCopies.begin();
            DI != MaybeDeadCopies.end();) {
         MachineInstr *MaybeDead = *DI;
-        MCRegister Reg = MaybeDead->getOperand(0).getReg().asMCReg();
+        Optional<DestSourcePair> CopyOperands =
+            isCopyInstr(*MaybeDead, *TII, UseCopyInstr);
+        MCRegister Reg = CopyOperands->Destination->getReg().asMCReg();
         assert(!MRI->isReserved(Reg));
 
         if (!RegMask->clobbersPhysReg(Reg)) {
           ++DI;
           continue;
         }
 
         LLVM_DEBUG(dbgs() << "MCP: Removing copy due to regmask clobbering: ";
                    MaybeDead->dump());
 
         // Make sure we invalidate any entries in the copy maps before erasing
         // the instruction.
-        Tracker.clobberRegister(Reg, *TRI);
+        Tracker.clobberRegister(Reg, *TRI, *TII, UseCopyInstr);
 
         // erase() will return the next valid iterator pointing to the next
         // element after the erased one.
         DI = MaybeDeadCopies.erase(DI);
         MaybeDead->eraseFromParent();
         Changed = true;
         ++NumDeletes;
       }
     }
 
     // Any previous copy definition or reading the Defs is no longer available.
     for (MCRegister Reg : Defs)
-      Tracker.clobberRegister(Reg, *TRI);
+      Tracker.clobberRegister(Reg, *TRI, *TII, UseCopyInstr);
   }
 
   // If MBB doesn't have successors, delete the copies whose defs are not used.
   // If MBB does have successors, then conservative assume the defs are live-out
   // since we don't want to trust live-in lists.
   if (MBB.succ_empty()) {
     for (MachineInstr *MaybeDead : MaybeDeadCopies) {
       LLVM_DEBUG(dbgs() << "MCP: Removing copy due to no live-out succ: ";
                  MaybeDead->dump());
-      assert(!MRI->isReserved(MaybeDead->getOperand(0).getReg()));
+
+      Optional<DestSourcePair> CopyOperands =
+          isCopyInstr(*MaybeDead, *TII, UseCopyInstr);
+      assert(CopyOperands);
+
+      Register SrcReg = CopyOperands->Source->getReg();
+      Register DestReg = CopyOperands->Destination->getReg();
+      assert(!MRI->isReserved(DestReg));
 
       // Update matching debug values, if any.
-      assert(MaybeDead->isCopy());
-      Register SrcReg = MaybeDead->getOperand(1).getReg();
-      Register DestReg = MaybeDead->getOperand(0).getReg();
       SmallVector<MachineInstr *> MaybeDeadDbgUsers(
           CopyDbgUsers[MaybeDead].begin(), CopyDbgUsers[MaybeDead].end());
       MRI->updateDbgUsersToReg(DestReg.asMCReg(), SrcReg.asMCReg(),
                                MaybeDeadDbgUsers);
 
       MaybeDead->eraseFromParent();
       Changed = true;
       ++NumDeletes;
     }
   }
 
   MaybeDeadCopies.clear();
   CopyDbgUsers.clear();
   Tracker.clear();
 }
 
 static bool isBackwardPropagatableCopy(MachineInstr &MI,
-                                       const MachineRegisterInfo &MRI) {
-  assert(MI.isCopy() && "MI is expected to be a COPY");
-  Register Def = MI.getOperand(0).getReg();
-  Register Src = MI.getOperand(1).getReg();
+                                       const MachineRegisterInfo &MRI,
+                                       const TargetInstrInfo &TII,
+                                       bool UseCopyInstr) {
+  Optional<DestSourcePair> CopyOperands = isCopyInstr(MI, TII, UseCopyInstr);
+  assert(CopyOperands && "MI is expected to be a COPY");
+
+  Register Def = CopyOperands->Destination->getReg();
+  Register Src = CopyOperands->Source->getReg();
 
   if (!Def || !Src)
     return false;
 
   if (MRI.isReserved(Def) || MRI.isReserved(Src))
     return false;
 
-  return MI.getOperand(1).isRenamable() && MI.getOperand(1).isKill();
+  return CopyOperands->Source->isRenamable() && CopyOperands->Source->isKill();
 }
 
 void MachineCopyPropagation::propagateDefs(MachineInstr &MI) {
   if (!Tracker.hasAnyCopies())
     return;
 
   for (unsigned OpIdx = 0, OpEnd = MI.getNumOperands(); OpIdx != OpEnd;
        ++OpIdx) {
     MachineOperand &MODef = MI.getOperand(OpIdx);
 
     if (!MODef.isReg() || MODef.isUse())
       continue;
 
     // Ignore non-trivial cases.
     if (MODef.isTied() || MODef.isUndef() || MODef.isImplicit())
       continue;
 
     if (!MODef.getReg())
       continue;
 
     // We only handle if the register comes from a vreg.
     if (!MODef.isRenamable())
       continue;
 
-    MachineInstr *Copy =
-        Tracker.findAvailBackwardCopy(MI, MODef.getReg().asMCReg(), *TRI);
+    MachineInstr *Copy = Tracker.findAvailBackwardCopy(
+        MI, MODef.getReg().asMCReg(), *TRI, *TII, UseCopyInstr);
     if (!Copy)
       continue;
 
-    Register Def = Copy->getOperand(0).getReg();
-    Register Src = Copy->getOperand(1).getReg();
+    Optional<DestSourcePair> CopyOperands =
+        isCopyInstr(*Copy, *TII, UseCopyInstr);
+    Register Def = CopyOperands->Destination->getReg();
+    Register Src = CopyOperands->Source->getReg();
 
     if (MODef.getReg() != Src)
       continue;
 
     if (!isBackwardPropagatableRegClassCopy(*Copy, MI, OpIdx))
       continue;
 
     if (hasImplicitOverlap(MI, MODef))
       continue;
 
     if (hasOverlappingMultipleDef(MI, MODef, Def))
       continue;
 
     LLVM_DEBUG(dbgs() << "MCP: Replacing " << printReg(MODef.getReg(), TRI)
                       << "\n     with " << printReg(Def, TRI) << "\n     in "
                       << MI << "     from " << *Copy);
 
     MODef.setReg(Def);
-    MODef.setIsRenamable(Copy->getOperand(0).isRenamable());
+    MODef.setIsRenamable(CopyOperands->Destination->isRenamable());
 
     LLVM_DEBUG(dbgs() << "MCP: After replacement: " << MI << "\n");
     MaybeDeadCopies.insert(Copy);
     Changed = true;
     ++NumCopyBackwardPropagated;
   }
 }
 
 void MachineCopyPropagation::BackwardCopyPropagateBlock(
     MachineBasicBlock &MBB) {
   LLVM_DEBUG(dbgs() << "MCP: BackwardCopyPropagateBlock " << MBB.getName()
                     << "\n");
 
   for (MachineInstr &MI : llvm::make_early_inc_range(llvm::reverse(MBB))) {
     // Ignore non-trivial COPYs.
-    if (MI.isCopy() && MI.getNumOperands() == 2 &&
-        !TRI->regsOverlap(MI.getOperand(0).getReg(),
-                          MI.getOperand(1).getReg())) {
+    Optional<DestSourcePair> CopyOperands = isCopyInstr(MI, *TII, UseCopyInstr);
+    if (CopyOperands) {
+      Register DefReg = CopyOperands->Destination->getReg();
+      Register SrcReg = CopyOperands->Source->getReg();
 
-      MCRegister Def = MI.getOperand(0).getReg().asMCReg();
-      MCRegister Src = MI.getOperand(1).getReg().asMCReg();
+      if (TRI->regsOverlap(DefReg, SrcReg))
+        continue;
+
+      MCRegister Def = DefReg.asMCReg();
+      MCRegister Src = SrcReg.asMCReg();
 
       // Unlike forward cp, we don't invoke propagateDefs here,
       // just let forward cp do COPY-to-COPY propagation.
-      if (isBackwardPropagatableCopy(MI, *MRI)) {
-        Tracker.invalidateRegister(Src, *TRI);
-        Tracker.invalidateRegister(Def, *TRI);
-        Tracker.trackCopy(&MI, *TRI);
+      if (isBackwardPropagatableCopy(MI, *MRI, *TII, UseCopyInstr)) {
+        Tracker.invalidateRegister(Src, *TRI, *TII, UseCopyInstr);
+        Tracker.invalidateRegister(Def, *TRI, *TII, UseCopyInstr);
+        Tracker.trackCopy(&MI, *TRI, *TII, UseCopyInstr);
         continue;
       }
     }
 
     // Invalidate any earlyclobber regs first.
     for (const MachineOperand &MO : MI.operands())
       if (MO.isReg() && MO.isEarlyClobber()) {
         MCRegister Reg = MO.getReg().asMCReg();
         if (!Reg)
           continue;
-        Tracker.invalidateRegister(Reg, *TRI);
+        Tracker.invalidateRegister(Reg, *TRI, *TII, UseCopyInstr);
       }
 
     propagateDefs(MI);
     for (const MachineOperand &MO : MI.operands()) {
       if (!MO.isReg())
         continue;
 
       if (!MO.getReg())
         continue;
 
       if (MO.isDef())
-        Tracker.invalidateRegister(MO.getReg().asMCReg(), *TRI);
+        Tracker.invalidateRegister(MO.getReg().asMCReg(), *TRI, *TII,
+                                   UseCopyInstr);
 
       if (MO.readsReg()) {
         if (MO.isDebug()) {
           //  Check if the register in the debug instruction is utilized
           // in a copy instruction, so we can update the debug info if the
           // register is changed.
           for (MCRegUnitIterator RUI(MO.getReg().asMCReg(), TRI); RUI.isValid();
                ++RUI) {
             if (auto *Copy = Tracker.findCopyDefViaUnit(*RUI, *TRI)) {
               CopyDbgUsers[Copy].insert(&MI);
             }
           }
         } else {
-          Tracker.invalidateRegister(MO.getReg().asMCReg(), *TRI);
+          Tracker.invalidateRegister(MO.getReg().asMCReg(), *TRI, *TII,
+                                     UseCopyInstr);
         }
       }
     }
   }
 
   for (auto *Copy : MaybeDeadCopies) {
 
-    Register Src = Copy->getOperand(1).getReg();
-    Register Def = Copy->getOperand(0).getReg();
+    Optional<DestSourcePair> CopyOperands =
+        isCopyInstr(*Copy, *TII, UseCopyInstr);
+    Register Src = CopyOperands->Source->getReg();
+    Register Def = CopyOperands->Destination->getReg();
     SmallVector<MachineInstr *> MaybeDeadDbgUsers(CopyDbgUsers[Copy].begin(),
                                                   CopyDbgUsers[Copy].end());
 
     MRI->updateDbgUsersToReg(Src.asMCReg(), Def.asMCReg(), MaybeDeadDbgUsers);
     Copy->eraseFromParent();
     ++NumDeletes;
   }
 
bool MachineCopyPropagation::runOnMachineFunction(MachineFunction &MF) {
 
   for (MachineBasicBlock &MBB : MF) {
     BackwardCopyPropagateBlock(MBB);
     ForwardCopyPropagateBlock(MBB);
   }
 
   return Changed;
 }
+
+MachineFunctionPass *
+llvm::createMachineCopyPropagationPass(bool UseCopyInstr = false) {
+  return new MachineCopyPropagation(UseCopyInstr);
+}

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

 #include "llvm/CodeGen/GlobalISel/InstructionSelect.h"
 #include "llvm/CodeGen/GlobalISel/Legalizer.h"
 #include "llvm/CodeGen/GlobalISel/LoadStoreOpt.h"
 #include "llvm/CodeGen/GlobalISel/Localizer.h"
 #include "llvm/CodeGen/GlobalISel/RegBankSelect.h"
 #include "llvm/CodeGen/MIRParser/MIParser.h"
 #include "llvm/CodeGen/MachineScheduler.h"
 #include "llvm/CodeGen/Passes.h"
+#include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/TargetRegistry.h"
 #include "llvm/Pass.h"
static cl::opt<bool> EnableCCMP("aarch64-enable-ccmp",
                                 cl::desc("Enable the CCMP formation pass"),
                                 cl::init(true), cl::Hidden);
 
 static cl::opt<bool>
     EnableCondBrTuning("aarch64-enable-cond-br-tune",
                        cl::desc("Enable the conditional branch tuning pass"),
                        cl::init(true), cl::Hidden);
 
+static cl::opt<bool> EnableAArch64CopyPropagation(
+    "aarch64-enable-copy-propagation",
+    cl::desc("Enable the copy propagation with AArch64 copy instr"),
+    cl::init(true), cl::Hidden);
+
 static cl::opt<bool> EnableMCR("aarch64-enable-mcr",
                                cl::desc("Enable the machine combiner pass"),
                                cl::init(true), cl::Hidden);
 
 static cl::opt<bool> EnableStPairSuppress("aarch64-enable-stp-suppress",
                                           cl::desc("Suppress STP for AArch64"),
                                           cl::init(true), cl::Hidden);
 

 
 void AArch64PassConfig::addPreEmitPass() {
   // Machine Block Placement might have created new opportunities when run
   // at O3, where the Tail Duplication Threshold is set to 4 instructions.
   // Run the load/store optimizer once more.
   if (TM->getOptLevel() >= CodeGenOpt::Aggressive && EnableLoadStoreOpt)
     addPass(createAArch64LoadStoreOptimizationPass());
 
+  if (TM->getOptLevel() >= CodeGenOpt::Aggressive &&
+      EnableAArch64CopyPropagation)
+    addPass(createMachineCopyPropagationPass(true));
+
   addPass(createAArch64A53Fix835769());
 
   if (EnableBranchTargets)
     addPass(createAArch64BranchTargetsPass());
 
   // Relax conditional branch instructions if they're otherwise out of
   // range of their destination.
   if (BranchRelaxation)

llvm/test/CodeGen/AArch64/O3-pipeline.ll

 ; CHECK-NEXT:       Analyze Machine Code For Garbage Collection
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       MachinePostDominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Basic Block Placement
 ; CHECK-NEXT:       Insert fentry calls
 ; CHECK-NEXT:       Insert XRay ops
 ; CHECK-NEXT:       Implement the 'patchable-function' attribute
 ; CHECK-NEXT:       AArch64 load / store optimization pass
+; CHECK-NEXT:       Machine Copy Propagation Pass
 ; CHECK-NEXT:       Workaround A53 erratum 835769 pass
 ; CHECK-NEXT:       AArch64 Branch Targets
 ; CHECK-NEXT:       Branch relaxation pass
 ; CHECK-NEXT:       AArch64 Compress Jump Tables
 ; CHECK-NEXT:       Contiguously Lay Out Funclets
 ; CHECK-NEXT:       StackMap Liveness Analysis
 ; CHECK-NEXT:       Live DEBUG_VALUE analysis
 ; CHECK-NEXT:     Machine Outliner

llvm/test/CodeGen/AArch64/copyprop.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -O3 -mtriple=aarch64-- | FileCheck %s
+
+define void @copyprop_after_mbp(i32 %v, i32* %a, i32* %b, i32* %c, i32* %d) {
+; CHECK-LABEL: copyprop_after_mbp:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    cmp w0, #10
+; CHECK-NEXT:    b.ne .LBB0_2
+; CHECK-NEXT:  // %bb.1: // %bb.0
+; CHECK-NEXT:    mov w9, #15
+; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    str w9, [x2]
+; CHECK-NEXT:    mov w9, #12
+; CHECK-NEXT:    str w8, [x1]
+; CHECK-NEXT:    str w9, [x4]
+; CHECK-NEXT:    ret
+; CHECK-NEXT:  .LBB0_2: // %bb.1
+; CHECK-NEXT:    mov w9, #25
+; CHECK-NEXT:    str w9, [x3]
+; CHECK-NEXT:    mov w9, #12
+; CHECK-NEXT:    str wzr, [x1]
+; CHECK-NEXT:    str w9, [x4]
+; CHECK-NEXT:    ret
+  %1 = icmp eq i32 %v, 10
+  br i1 %1, label %bb.0, label %bb.1
+
+bb.0:
+  store i32 15, i32* %b, align 4
+  br label %bb.2
+
+bb.1:
+  store i32 25, i32* %c, align 4
+  br label %bb.2
+
+bb.2:
+  %2 = phi i32 [ 1, %bb.0 ], [ 0, %bb.1 ]
+  store i32 %2, i32* %a, align 4
+  store i32 12, i32* %d, align 4
+  ret void
+}

llvm/test/CodeGen/AArch64/copyprop.mir

-# RUN: llc -mtriple=aarch64-linux-gnu -run-pass machine-cp -o - %s | FileCheck %s
+# RUN: llc -mtriple=aarch64-linux-gnu -run-pass machine-cp -mcp-use-is-copy-instr -o - %s | FileCheck %s
 # Tests for MachineCopyPropagation copy forwarding.
 ---
 # Simple forwarding.
 # CHECK-LABEL: name: test1
 # CHECK: $x0 = SUBXri $x0, 1, 0
 name:            test1
 tracksRegLiveness: true
 body: |

 tracksRegLiveness: true
 body: |
   bb.0:
     liveins: $x0
     renamable $x1 = COPY $x0
     $x0 = ADDXri $x0, 1, 0
     $x2 = SUBXri renamable $x1, 1, 0
 ...
+---
+# CHECK-LABEL: name: test1_orr_as_copy
+# CHECK: STRBBui $wzr, killed renamable $x8, 36
+name:            test1_orr_as_copy
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $x8
+    $w9 = ORRWrs $wzr, $wzr, 0
+    STRBBui killed renamable $w9, killed renamable $x8, 36
+...