[MachineInstr] Introduce generic predicated copy opcode

[cdevadas]

Summary

The copy operation with a predicated dependency is currently
marked with an implicit operand. It is good to have a separate
opcode for such instances to distinguish them from the regular
copies that have no dependencies at all. This will allow targets
more control while lowering the copies. AMDGPU, for instance,
have the implicit dependency with exec register for all vector
copies and no such dependency for scalar copies and there is a
custom pass currently adds the implicit exec operand to every
vector copy and that doesn't look right.

This patch introduces a new opcode PRED_COPY, called the
predicated equivalent of the existing COPY opcode. Targets
can define the default copy opcode and the getCopyOpcode
callback would return the chosen opcode appropriately. Every
interface that needs TargetOpcode::COPY should now be replaced
with a call to getCopyOpcode. Targets have the control over how
and when to lower the PRED_COPY instructions.

[cdevadas]

D143757 in the review chain enables predicated copy for AMDGPU target.
The application of pred-copy is here: D143762.
Testcase llvm/test/CodeGen/AMDGPU/whole-wave-register-copy.ll added along with D124196 demonstrates the need for pred-copy.

[kparzysz]

Please post an RFC for this. So far it looks to me like something that can be dealt with in AMDGPU codegen.

[arsenm]

In D143754#4119171, @kparzysz wrote:

Please post an RFC for this. So far it looks to me like something that can be dealt with in AMDGPU codegen.

Whatever we end up with needs to be deeply embedded in everything related to register allocation, so it can't be handled just in AMDGPU code

[cdevadas]

RFC posted https://discourse.llvm.org/t/rfc-introduce-generic-predicated-copy-opcode/68494

[bjope]

llvm/include/llvm/Support/TargetOpcodes.def

114

So what is the result when the predicate is false?.

(That kind of question does not exist for any existing generic opcodes afaik, because this would be the first predicable generic opcode.)

llvm/lib/CodeGen/GlobalISel/GISelKnownBits.cpp

204

Here we have a typical example where I think it gets complicated to say that PRED_COPY is predicable. If the predicate is false, then you can't guarantee anything about the known bits. OTOH, if the predicate is guaranteed to be true, then there is no need to have a PRED_COPY.

llvm/lib/CodeGen/TwoAddressInstructionPass.cpp

1927

Here is a general comment to about all these places that now may or may not create a predicated instruction.

In our downstream target all predicated instructions also have extra operands. So whenever we create a predicated instruction those predicate operands should be added to the instruction. But that won't happen for any of these places where you use getCopyOpcode(). That is of course not a problem as long as we avoid using PRED_COPY for our target. I'm just saying that this probably wouldn't work for any target, depending on how isPredicable=1 is handled.

[cdevadas]

The IsPredicable field will always be turned to 1 for Pred_Copy to indicate that it is the Predicated version of the copy.

I actually wanted to have the Pred_Copy without these implicit operands. We should be able to model the dependency using an additional operand.
But I don't know how to model this operand and hence decided to continue using the implicit operand itself.

May be an immediate operand that takes const values (0, 1, 2, etc.)?
The immediate values can be mapped to enumerations defined via target files to represent these predicated dependencies.
When there is no dependency, use the value 0. It means, there is no dependency and can change to the regular COPY.
If required, the Pred_Copy with Imm_opnd value 0 can also be modeled for the "predicate is false" scenario.

For example. the following can be defined for AMDGPU target in llvm/lib/Target/AMDGPU/SIDefines.h

enum Copy_Predication {

None = 0,    // No dependency. It is for SGPR copies. At some point, convert the instruction into regular COPY.
Lane_Copy,  // copy only the active lanes of VGPRs.
Wave_Copy // copy all lanes of VGPRs

}

Any thoughts?

llvm/lib/CodeGen/GlobalISel/GISelKnownBits.cpp

204

The PRED_COPY here was added (like many other instances) for the codegen transformations to work as similar to the COPY opcode. Is there a need to consider the Predicable field here?

[MatzeB]

I wrote more details on the RFC on discourse. But this feels pretty AMDGPU specific, why wouldn't it used a target specific opcode then?

[MatzeB]

llvm/lib/CodeGen/MachineVerifier.cpp

1837

As I still don't fully understand the point of the new opcode... I was hoping to see some extra rules and constraints here, but this feels like it allows the exact same operands and constraints as a normal COPY if there's no extra verified rules here... But then why do we need a new opcode in the first place?

[bjope]

In D143754#4132515, @cdevadas wrote:

The IsPredicable field will always be turned to 1 for Pred_Copy to indicate that it is the Predicated version of the copy.

I actually wanted to have the Pred_Copy without these implicit operands. We should be able to model the dependency using an additional operand.
But I don't know how to model this operand and hence decided to continue using the implicit operand itself.

May be an immediate operand that takes const values (0, 1, 2, etc.)?
The immediate values can be mapped to enumerations defined via target files to represent these predicated dependencies.
When there is no dependency, use the value 0. It means, there is no dependency and can change to the regular COPY.
If required, the Pred_Copy with Imm_opnd value 0 can also be modeled for the "predicate is false" scenario.

For example. the following can be defined for AMDGPU target in llvm/lib/Target/AMDGPU/SIDefines.h

enum Copy_Predication {

None = 0,    // No dependency. It is for SGPR copies. At some point, convert the instruction into regular COPY.
Lane_Copy,  // copy only the active lanes of VGPRs.
Wave_Copy // copy all lanes of VGPRs

}

Any thoughts?

This would still be very target specific. And since all instructions with isPredicable=1 in my target takes 3 extra operands, then I assume other targets can have similar rules that should be fulfilled for such instructions. So if target generic opcodes starts to use isPredicable=1 it might end up a bit messy as such target sepcific expectations on TargetInstrInfo::isPredicable() isn't fulfilled.

I also agree with others that this feels quite target specific for AMDGPU (but that also makes me feel that maybe I've misunderstood why you want to make this generic).

[bjope]

In the description you have written "The copy operation with a predicated dependency is currently marked with an implicit operand". I don't really understand what that is referring to. Is that something AMDGPU specific, or what is this referring to?

[cdevadas]

This patch is no longer required after the introduction of liverange split target opcode with D150388

llvm/include/llvm/CodeGen/GlobalISel/LegalizationArtifactCombiner.h

while (!UpdatedDefs.empty()) {
         case TargetOpcode::G_SEXT:
         case TargetOpcode::G_UNMERGE_VALUES:
         case TargetOpcode::G_EXTRACT:
         case TargetOpcode::G_TRUNC:
         case TargetOpcode::G_BUILD_VECTOR:
           // Adding Use to ArtifactList.
           WrapperObserver.changedInstr(Use);
           break;
-        case TargetOpcode::COPY: {
+        case TargetOpcode::COPY:
+        case TargetOpcode::PRED_COPY: {
           Register Copy = Use.getOperand(0).getReg();
           if (Copy.isVirtual())
             UpdatedDefs.push_back(Copy);
           break;
         }
         default:
           // If we do not have an artifact combine for the opcode, there is no
           // point in adding it to the ArtifactList as nothing interesting will
           // be done to it anyway.
           break;
         }
       }
     }
     return Changed;
   }
 
 private:
   static Register getArtifactSrcReg(const MachineInstr &MI) {
     switch (MI.getOpcode()) {
     case TargetOpcode::COPY:
+    case TargetOpcode::PRED_COPY:
     case TargetOpcode::G_TRUNC:
     case TargetOpcode::G_ZEXT:
     case TargetOpcode::G_ANYEXT:
     case TargetOpcode::G_SEXT:
     case TargetOpcode::G_EXTRACT:
       return MI.getOperand(1).getReg();
     case TargetOpcode::G_UNMERGE_VALUES:
       return MI.getOperand(MI.getNumOperands() - 1).getReg();
private:
   }
 
   /// Looks through copy instructions and returns the actual
   /// source register.
   Register lookThroughCopyInstrs(Register Reg) {
     using namespace llvm::MIPatternMatch;
 
     Register TmpReg;
-    while (mi_match(Reg, MRI, m_Copy(m_Reg(TmpReg)))) {
+    while (mi_match(Reg, MRI, m_Copy(m_Reg(TmpReg))) ||
+           mi_match(Reg, MRI, m_Pred_Copy(m_Reg(TmpReg)))) {
       if (MRI.getType(TmpReg).isValid())
         Reg = TmpReg;
       else
         break;
     }
     return Reg;
   }
 };
 
 } // namespace llvm
 
 #endif // LLVM_CODEGEN_GLOBALISEL_LEGALIZATIONARTIFACTCOMBINER_H

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

 }
 
 template <typename SrcTy>
 inline UnaryOp_match<SrcTy, TargetOpcode::COPY> m_Copy(SrcTy &&Src) {
   return UnaryOp_match<SrcTy, TargetOpcode::COPY>(std::forward<SrcTy>(Src));
 }
 
 template <typename SrcTy>
+inline UnaryOp_match<SrcTy, TargetOpcode::PRED_COPY> m_Pred_Copy(SrcTy &&Src) {
+  return UnaryOp_match<SrcTy, TargetOpcode::PRED_COPY>(
+      std::forward<SrcTy>(Src));
+}
+
+template <typename SrcTy>
 inline UnaryOp_match<SrcTy, TargetOpcode::G_FSQRT> m_GFSqrt(const SrcTy &Src) {
   return UnaryOp_match<SrcTy, TargetOpcode::G_FSQRT>(Src);
 }
 
 // General helper for generic MI compares, i.e. G_ICMP and G_FCMP
 // TODO: Allow checking a specific predicate.
 template <typename Pred_P, typename LHS_P, typename RHS_P, unsigned Opcode,
           bool Commutable = false>

llvm/include/llvm/CodeGen/GlobalISel/RegBankSelect.h

 class MachineBlockFrequencyInfo;
 class MachineBranchProbabilityInfo;
 class MachineOperand;
 class MachineRegisterInfo;
 class Pass;
 class raw_ostream;
 class TargetPassConfig;
 class TargetRegisterInfo;
+class TargetInstrInfo;
 
 /// This pass implements the reg bank selector pass used in the GlobalISel
 /// pipeline. At the end of this pass, all register operands have been assigned
 class RegBankSelect : public MachineFunctionPass {
 public:
   static char ID;
 
   /// List of the modes supported by the RegBankSelect pass.
protected:
 
   /// MRI contains all the register class/bank information that this
   /// pass uses and updates.
   MachineRegisterInfo *MRI = nullptr;
 
   /// Information on the register classes for the current function.
   const TargetRegisterInfo *TRI = nullptr;
 
+  /// Information used to access the description of the opcodes.
+  const TargetInstrInfo *TII = nullptr;
+
   /// Get the frequency of blocks.
   /// This is required for non-fast mode.
   MachineBlockFrequencyInfo *MBFI = nullptr;
 
   /// Get the frequency of the edges.
   /// This is required for non-fast mode.
   MachineBranchProbabilityInfo *MBPI = nullptr;
 

llvm/include/llvm/CodeGen/MachineInstr.h

bool isRegSequence() const {
     return getOpcode() == TargetOpcode::REG_SEQUENCE;
   }
 
   bool isBundle() const {
     return getOpcode() == TargetOpcode::BUNDLE;
   }
 
   bool isCopy() const {
-    return getOpcode() == TargetOpcode::COPY;
+    return getOpcode() == TargetOpcode::COPY ||
+           getOpcode() == TargetOpcode::PRED_COPY;
   }
 
   bool isFullCopy() const {
     return isCopy() && !getOperand(0).getSubReg() && !getOperand(1).getSubReg();
   }
 
   bool isExtractSubreg() const {
     return getOpcode() == TargetOpcode::EXTRACT_SUBREG;

   bool isTransient() const {
     switch (getOpcode()) {
     default:
       return isMetaInstruction();
     // Copy-like instructions are usually eliminated during register allocation.
     case TargetOpcode::PHI:
     case TargetOpcode::G_PHI:
     case TargetOpcode::COPY:
+    case TargetOpcode::PRED_COPY:
     case TargetOpcode::INSERT_SUBREG:
     case TargetOpcode::SUBREG_TO_REG:
     case TargetOpcode::REG_SEQUENCE:
       return true;
     }
   }
 
   /// Return the number of instructions inside the MI bundle, excluding the

llvm/include/llvm/CodeGen/TargetInstrInfo.h

 
 //---------------------------------------------------------------------------
 ///
 /// TargetInstrInfo - Interface to description of machine instruction set
 ///
 class TargetInstrInfo : public MCInstrInfo {
 public:
   TargetInstrInfo(unsigned CFSetupOpcode = ~0u, unsigned CFDestroyOpcode = ~0u,
-                  unsigned CatchRetOpcode = ~0u, unsigned ReturnOpcode = ~0u)
+                  unsigned CatchRetOpcode = ~0u, unsigned ReturnOpcode = ~0u,
+                  unsigned CopyOpcode = TargetOpcode::COPY)
       : CallFrameSetupOpcode(CFSetupOpcode),
         CallFrameDestroyOpcode(CFDestroyOpcode), CatchRetOpcode(CatchRetOpcode),
-        ReturnOpcode(ReturnOpcode) {}
+        ReturnOpcode(ReturnOpcode), CopyOpcode(CopyOpcode) {}
   TargetInstrInfo(const TargetInstrInfo &) = delete;
   TargetInstrInfo &operator=(const TargetInstrInfo &) = delete;
   virtual ~TargetInstrInfo();
 
   static bool isGenericOpcode(unsigned Opc) {
     return Opc <= TargetOpcode::GENERIC_OP_END;
   }
 
if (isFrameSetup(I)) {
              "Frame size must not be negative");
       return getFrameSize(I) + I.getOperand(1).getImm();
     }
     return getFrameSize(I);
   }
 
   unsigned getCatchReturnOpcode() const { return CatchRetOpcode; }
   unsigned getReturnOpcode() const { return ReturnOpcode; }
+  unsigned getCopyOpcode() const { return CopyOpcode; }
 
   /// Returns the actual stack pointer adjustment made by an instruction
   /// as part of a call sequence. By default, only call frame setup/destroy
   /// instructions adjust the stack, but targets may want to override this
   /// to enable more fine-grained adjustment, or adjust by a different value.
   virtual int getSPAdjust(const MachineInstr &MI) const;
 
   /// Return true if the instruction is a "coalescable" extension instruction.
virtual bool isBasicBlockPrologue(const MachineInstr &MI) const {
     return false;
   }
 
   /// Helper function for inserting a COPY to \p Dst at insertion point \p InsPt
   /// in \p MBB block.
   MachineInstr *buildCopy(MachineBasicBlock &MBB,
                           MachineBasicBlock::iterator InsPt, const DebugLoc &DL,
                           Register Dst) const {
-    return BuildMI(MBB, InsPt, DL, get(TargetOpcode::COPY), Dst);
+    return BuildMI(MBB, InsPt, DL, get(getCopyOpcode()), Dst);
   }
 
   /// Helper function for inserting a COPY to \p Dst from \p Src at insertion
   /// point \p InsPt in \p MBB block.
   MachineInstr *buildCopy(MachineBasicBlock &MBB,
                           MachineBasicBlock::iterator InsPt, const DebugLoc &DL,
                           Register Dst, Register Src, unsigned Flags = 0,
                           unsigned SubReg = 0) const {
-    return BuildMI(MBB, InsPt, DL, get(TargetOpcode::COPY), Dst)
+    return BuildMI(MBB, InsPt, DL, get(getCopyOpcode()), Dst)
         .addReg(Src, Flags, SubReg);
   }
 
   /// Helper function for inserting a COPY to \p Dst from \p Src at insertion
   /// point \p InsPt in \p MBB block. Get the Debug Location from \p MIMD.
   MachineInstrBuilder buildCopy(MachineBasicBlock &MBB,
                                 MachineBasicBlock::iterator InsPt,
                                 const MIMetadata &MIMD, Register Dst,
                                 Register Src, unsigned Flags = 0,
                                 unsigned SubReg = 0) const {
     MachineFunction &MF = *MBB.getParent();
     MachineInstr *MI =
-        MF.CreateMachineInstr(get(TargetOpcode::COPY), MIMD.getDL());
+        MF.CreateMachineInstr(get(getCopyOpcode()), MIMD.getDL());
     MBB.insert(InsPt, MI);
     return MachineInstrBuilder(MF, MI)
         .setPCSections(MIMD.getPCSections())
         .addReg(Dst, RegState::Define)
         .addReg(Src, Flags, SubReg);
   }
 
   /// During PHI eleimination lets target to make necessary checks and
virtual bool isExplicitTargetIndexDef(const MachineInstr &MI, int &Index,
     return false;
   }
 
 private:
   mutable std::unique_ptr<MIRFormatter> Formatter;
   unsigned CallFrameSetupOpcode, CallFrameDestroyOpcode;
   unsigned CatchRetOpcode;
   unsigned ReturnOpcode;
+  unsigned CopyOpcode;
 };
 
 /// Provide DenseMapInfo for TargetInstrInfo::RegSubRegPair.
 template <> struct DenseMapInfo<TargetInstrInfo::RegSubRegPair> {
   using RegInfo = DenseMapInfo<unsigned>;
 
   static inline TargetInstrInfo::RegSubRegPair getEmptyKey() {
     return TargetInstrInfo::RegSubRegPair(RegInfo::getEmptyKey(),

llvm/include/llvm/Support/TargetOpcodes.def

 /// After register coalescing references of v1024 should be replace with
 /// v1027:3, v1025 with v1027:4, etc.
   HANDLE_TARGET_OPCODE(REG_SEQUENCE)
 
 /// COPY - Target-independent register copy. This instruction can also be
 /// used to copy between subregisters of virtual registers.
   HANDLE_TARGET_OPCODE(COPY)
 
+/// PRED_COPY - Target-independent register copy with a predication.
+/// Some targets require it for special handling certain register copies.
+  HANDLE_TARGET_OPCODE(PRED_COPY)
+
 /// BUNDLE - This instruction represents an instruction bundle. Instructions
 /// which immediately follow a BUNDLE instruction which are marked with
 /// 'InsideBundle' flag are inside the bundle.
 HANDLE_TARGET_OPCODE(BUNDLE)
 
 /// Lifetime markers.
 HANDLE_TARGET_OPCODE(LIFETIME_START)
 HANDLE_TARGET_OPCODE(LIFETIME_END)

llvm/include/llvm/Target/GlobalISel/Combine.td

 def register_matchinfo: GIDefMatchData<"Register">;
 def int64_matchinfo: GIDefMatchData<"int64_t">;
 def apint_matchinfo : GIDefMatchData<"APInt">;
 def build_fn_matchinfo :
 GIDefMatchData<"std::function<void(MachineIRBuilder &)>">;
 def unsigned_matchinfo: GIDefMatchData<"unsigned">;
 
 def copy_prop : GICombineRule<
-  (defs root:$d),
-  (match (COPY $d, $s):$mi,
+  (defs root:$mi),
+  (match (wip_match_opcode COPY, PRED_COPY):$mi,
          [{ return Helper.matchCombineCopy(*${mi}); }]),
   (apply [{ Helper.applyCombineCopy(*${mi}); }])>;
 
 // idempotent operations
 // Fold (freeze (freeze x)) -> (freeze x).
 // Fold (fabs (fabs x)) -> (fabs x).
 // Fold (fcanonicalize (fcanonicalize x)) -> (fcanonicalize x).
 def idempotent_prop : GICombineRule<

llvm/include/llvm/Target/Target.td

 def COPY : StandardPseudoInstruction {
   let OutOperandList = (outs unknown:$dst);
   let InOperandList = (ins unknown:$src);
   let AsmString = "";
   let hasSideEffects = false;
   let isAsCheapAsAMove = true;
   let hasNoSchedulingInfo = false;
 }
+def PRED_COPY : StandardPseudoInstruction {
+  let OutOperandList = (outs unknown:$dst);
+  let InOperandList = (ins unknown:$src);
+  let AsmString = "PRED_COPY";
+  let hasSideEffects = false;
+  let isAsCheapAsAMove = true;
+  let hasNoSchedulingInfo = false;
+  let isPredicable = true;
+}
 def BUNDLE : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins variable_ops);
   let AsmString = "BUNDLE";
   let hasSideEffects = false;
 }
 def LIFETIME_START : StandardPseudoInstruction {
   let OutOperandList = (outs);

llvm/lib/CodeGen/CodeGenCommonISel.cpp

 
 static MachineOperand *salvageDebugInfoImpl(const MachineRegisterInfo &MRI,
                                             MachineInstr &MI,
                                             SmallVectorImpl<uint64_t> &Ops) {
   switch (MI.getOpcode()) {
   case TargetOpcode::G_TRUNC:
     return getSalvageOpsForTrunc(MRI, MI, Ops);
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
     return getSalvageOpsForCopy(MRI, MI);
   default:
     return nullptr;
   }
 }
 
 void llvm::salvageDebugInfoForDbgValue(const MachineRegisterInfo &MRI,
                                        MachineInstr &MI,

llvm/lib/CodeGen/DetectDeadLanes.cpp

 /// Returns true if \p MI will get lowered to a series of COPY instructions.
 /// We call this a COPY-like instruction.
 static bool lowersToCopies(const MachineInstr &MI) {
   // Note: We could support instructions with MCInstrDesc::isRegSequenceLike(),
   // isExtractSubRegLike(), isInsertSubregLike() in the future even though they
   // are not lowered to a COPY.
   switch (MI.getOpcode()) {
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
   case TargetOpcode::PHI:
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::EXTRACT_SUBREG:
     return true;
   }
   return false;
 }
DeadLaneDetector::transferUsedLanes(const MachineInstr &MI,
                                     LaneBitmask UsedLanes,
                                     const MachineOperand &MO) const {
   unsigned OpNum = MO.getOperandNo();
   assert(lowersToCopies(MI) &&
          DefinedByCopy[Register::virtReg2Index(MI.getOperand(0).getReg())]);
 
   switch (MI.getOpcode()) {
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
   case TargetOpcode::PHI:
     return UsedLanes;
   case TargetOpcode::REG_SEQUENCE: {
     assert(OpNum % 2 == 1);
     unsigned SubIdx = MI.getOperand(OpNum + 1).getImm();
     return TRI->reverseComposeSubRegIndexLaneMask(SubIdx, UsedLanes);
   }
   case TargetOpcode::INSERT_SUBREG: {
LaneBitmask DeadLaneDetector::transferDefinedLanes(
   }
   case TargetOpcode::EXTRACT_SUBREG: {
     unsigned SubIdx = MI.getOperand(2).getImm();
     assert(OpNum == 1 && "EXTRACT_SUBREG must have one register operand only");
     DefinedLanes = TRI->reverseComposeSubRegIndexLaneMask(SubIdx, DefinedLanes);
     break;
   }
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
   case TargetOpcode::PHI:
     break;
   default:
     llvm_unreachable("function must be called with COPY-like instruction");
   }
 
   assert(Def.getSubReg() == 0 &&
          "Should not have subregister defs in machine SSA phase");

llvm/lib/CodeGen/ExpandPostRAPseudos.cpp

for (MachineInstr &MI : llvm::make_early_inc_range(MBB)) {
       }
 
       // Expand standard pseudos.
       switch (MI.getOpcode()) {
       case TargetOpcode::SUBREG_TO_REG:
         MadeChange |= LowerSubregToReg(&MI);
         break;
       case TargetOpcode::COPY:
+      case TargetOpcode::PRED_COPY:
         MadeChange |= LowerCopy(&MI);
         break;
       case TargetOpcode::DBG_VALUE:
         continue;
       case TargetOpcode::INSERT_SUBREG:
       case TargetOpcode::EXTRACT_SUBREG:
         llvm_unreachable("Sub-register pseudos should have been eliminated.");
       }
     }
   }
 
   return MadeChange;
 }

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

bool CombinerHelper::matchICmpToLHSKnownBits(
   Register LHS = MI.getOperand(2).getReg();
   auto KnownLHS = KB->getKnownBits(LHS);
   if (KnownLHS.getMinValue() != 0 || KnownLHS.getMaxValue() != 1)
     return false;
   // Make sure replacing Dst with the LHS is a legal operation.
   LLT LHSTy = MRI.getType(LHS);
   unsigned LHSSize = LHSTy.getSizeInBits();
   unsigned DstSize = DstTy.getSizeInBits();
-  unsigned Op = TargetOpcode::COPY;
+  unsigned Op = Builder.getTII().getCopyOpcode();
   if (DstSize != LHSSize)
     Op = DstSize < LHSSize ? TargetOpcode::G_TRUNC : TargetOpcode::G_ZEXT;
   if (!isLegalOrBeforeLegalizer({Op, {DstTy, LHSTy}}))
     return false;
   MatchInfo = [=](MachineIRBuilder &B) { B.buildInstr(Op, {Dst}, {LHS}); };
   return true;
 }
 

llvm/lib/CodeGen/GlobalISel/GISelKnownBits.cpp

 GISelKnownBits::GISelKnownBits(MachineFunction &MF, unsigned MaxDepth)
     : MF(MF), MRI(MF.getRegInfo()), TL(*MF.getSubtarget().getTargetLowering()),
       DL(MF.getFunction().getParent()->getDataLayout()), MaxDepth(MaxDepth) {}
 
 Align GISelKnownBits::computeKnownAlignment(Register R, unsigned Depth) {
   const MachineInstr *MI = MRI.getVRegDef(R);
   switch (MI->getOpcode()) {
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
     return computeKnownAlignment(MI->getOperand(1).getReg(), Depth);
   case TargetOpcode::G_ASSERT_ALIGN: {
     // TODO: Min with source
     return Align(MI->getOperand(2).getImm());
   }
   case TargetOpcode::G_FRAME_INDEX: {
     int FrameIdx = MI->getOperand(1).getIndex();
     return MF.getFrameInfo().getObjectAlign(FrameIdx);
for (unsigned i = 0, e = MI.getNumOperands() - 1; i < e; ++i) {
 
       // If we don't know any bits, early out.
       if (Known.isUnknown())
         break;
     }
     break;
   }
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
   case TargetOpcode::G_PHI:
   case TargetOpcode::PHI: {
     Known.One = APInt::getAllOnes(BitWidth);
     Known.Zero = APInt::getAllOnes(BitWidth);
     // Destination registers should not have subregisters at this
     // point of the pipeline, otherwise the main live-range will be
     // defined more than once, which is against SSA.
     assert(MI.getOperand(0).getSubReg() == 0 && "Is this code in SSA?");
unsigned GISelKnownBits::computeNumSignBits(Register R,
   // type constraint. This is unlikely to occur except by looking through copies
   // but it is possible for the initial register being queried to be in this
   // state.
   if (!DstTy.isValid())
     return 1;
 
   unsigned FirstAnswer = 1;
   switch (Opcode) {
-  case TargetOpcode::COPY: {
+  case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY: {
     MachineOperand &Src = MI.getOperand(1);
     if (Src.getReg().isVirtual() && Src.getSubReg() == 0 &&
         MRI.getType(Src.getReg()).isValid()) {
       // Don't increment Depth for this one since we didn't do any work.
       return computeNumSignBits(Src.getReg(), DemandedElts, Depth);
     }
 
     return 1;

llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp

LegalizerHelper::lowerVectorReduction(MachineInstr &MI) {
   LLT DstTy = MRI.getType(SrcReg);
 
   // The source could be a scalar if the IR type was <1 x sN>.
   if (SrcTy.isScalar()) {
     if (DstTy.getSizeInBits() > SrcTy.getSizeInBits())
       return UnableToLegalize; // FIXME: handle extension.
     // This can be just a plain copy.
     Observer.changingInstr(MI);
-    MI.setDesc(MIRBuilder.getTII().get(TargetOpcode::COPY));
+    const TargetInstrInfo &TII = MIRBuilder.getTII();
+    MI.setDesc(TII.get(TII.getCopyOpcode()));
     Observer.changedInstr(MI);
     return Legalized;
   }
   return UnableToLegalize;;
 }
 
 static bool shouldLowerMemFuncForSize(const MachineFunction &MF) {
   // On Darwin, -Os means optimize for size without hurting performance, so

llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp

MachineInstrBuilder MachineIRBuilder::buildBrJT(Register TablePtr,
   return buildInstr(TargetOpcode::G_BRJT)
       .addUse(TablePtr)
       .addJumpTableIndex(JTI)
       .addUse(IndexReg);
 }
 
 MachineInstrBuilder MachineIRBuilder::buildCopy(const DstOp &Res,
                                                 const SrcOp &Op) {
-  return buildInstr(TargetOpcode::COPY, Res, Op);
+  return buildInstr(getTII().getCopyOpcode(), Res, Op);
 }
 
 MachineInstrBuilder MachineIRBuilder::buildConstant(const DstOp &Res,
                                                     const ConstantInt &Val) {
   LLT Ty = Res.getLLTTy(*getMRI());
   LLT EltTy = Ty.getScalarType();
   assert(EltTy.getScalarSizeInBits() == Val.getBitWidth() &&
          "creating constant with the wrong size");
MachineInstrBuilder MachineIRBuilder::buildExtOrTrunc(unsigned ExtOpc,
   assert((TargetOpcode::G_ANYEXT == ExtOpc || TargetOpcode::G_ZEXT == ExtOpc ||
           TargetOpcode::G_SEXT == ExtOpc) &&
          "Expecting Extending Opc");
   assert(Res.getLLTTy(*getMRI()).isScalar() ||
          Res.getLLTTy(*getMRI()).isVector());
   assert(Res.getLLTTy(*getMRI()).isScalar() ==
          Op.getLLTTy(*getMRI()).isScalar());
 
-  unsigned Opcode = TargetOpcode::COPY;
+  unsigned Opcode = getTII().getCopyOpcode();
   if (Res.getLLTTy(*getMRI()).getSizeInBits() >
       Op.getLLTTy(*getMRI()).getSizeInBits())
     Opcode = ExtOpc;
   else if (Res.getLLTTy(*getMRI()).getSizeInBits() <
            Op.getLLTTy(*getMRI()).getSizeInBits())
     Opcode = TargetOpcode::G_TRUNC;
   else
     assert(Res.getLLTTy(*getMRI()) == Op.getLLTTy(*getMRI()));
MachineIRBuilder::buildInstr(unsigned Opc, ArrayRef<DstOp> DstOps,
   case TargetOpcode::G_BITCAST: {
     assert(DstOps.size() == 1 && "Invalid Dst");
     assert(SrcOps.size() == 1 && "Invalid Srcs");
     assert(DstOps[0].getLLTTy(*getMRI()).getSizeInBits() ==
            SrcOps[0].getLLTTy(*getMRI()).getSizeInBits() && "invalid bitcast");
     break;
   }
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
     assert(DstOps.size() == 1 && "Invalid Dst");
     // If the caller wants to add a subreg source it has to be done separately
     // so we may not have any SrcOps at this point yet.
     break;
   case TargetOpcode::G_FCMP:
   case TargetOpcode::G_ICMP: {
     assert(DstOps.size() == 1 && "Invalid Dst Operands");
     assert(SrcOps.size() == 3 && "Invalid Src Operands");

llvm/lib/CodeGen/GlobalISel/RegBankSelect.cpp

 #include "llvm/CodeGen/MachineBranchProbabilityInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineOptimizationRemarkEmitter.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/RegisterBank.h"
 #include "llvm/CodeGen/RegisterBankInfo.h"
+#include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetOpcodes.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Function.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
RegBankSelect::RegBankSelect(char &PassID, Mode RunningMode)
   }
 }
 
 void RegBankSelect::init(MachineFunction &MF) {
   RBI = MF.getSubtarget().getRegBankInfo();
   assert(RBI && "Cannot work without RegisterBankInfo");
   MRI = &MF.getRegInfo();
   TRI = MF.getSubtarget().getRegisterInfo();
+  TII = MF.getSubtarget().getInstrInfo();
   TPC = &getAnalysis<TargetPassConfig>();
   if (OptMode != Mode::Fast) {
     MBFI = &getAnalysis<MachineBlockFrequencyInfo>();
     MBPI = &getAnalysis<MachineBranchProbabilityInfo>();
   } else {
     MBFI = nullptr;
     MBPI = nullptr;
   }
if (MO.isDef())
       std::swap(Src, Dst);
 
     assert((RepairPt.getNumInsertPoints() == 1 || Dst.isPhysical()) &&
            "We are about to create several defs for Dst");
 
     // Build the instruction used to repair, then clone it at the right
     // places. Avoiding buildCopy bypasses the check that Src and Dst have the
     // same types because the type is a placeholder when this function is called.
-    MI = MIRBuilder.buildInstrNoInsert(TargetOpcode::COPY)
+    MI = MIRBuilder.buildInstrNoInsert(TII->getCopyOpcode())
              .addDef(Dst)
              .addUse(Src);
     LLVM_DEBUG(dbgs() << "Copy: " << printReg(Src) << ':'
                       << printRegClassOrBank(Src, *MRI, TRI)
                       << " to: " << printReg(Dst) << ':'
                       << printRegClassOrBank(Dst, *MRI, TRI) << '\n');
   } else {
     // TODO: Support with G_IMPLICIT_DEF + G_INSERT sequence or G_EXTRACT
     // sequence.
     assert(ValMapping.partsAllUniform() && "irregular breakdowns not supported");

llvm/lib/CodeGen/GlobalISel/Utils.cpp

while ((MI = MRI.getVRegDef(VReg)) && !IsConstantOpcode(MI) &&
     case TargetOpcode::G_SEXT:
     case TargetOpcode::G_ZEXT:
       SeenOpcodes.push_back(std::make_pair(
           MI->getOpcode(),
           MRI.getType(MI->getOperand(0).getReg()).getSizeInBits()));
       VReg = MI->getOperand(1).getReg();
       break;
     case TargetOpcode::COPY:
+    case TargetOpcode::PRED_COPY:
       VReg = MI->getOperand(1).getReg();
       if (VReg.isPhysical())
         return std::nullopt;
       break;
     case TargetOpcode::G_INTTOPTR:
       VReg = MI->getOperand(1).getReg();
       break;
     default:

llvm/lib/CodeGen/MachineVerifier.cpp

void MachineVerifier::visitMachineInstrBefore(const MachineInstr *MI) {
   }
 
   StringRef ErrorInfo;
   if (!TII->verifyInstruction(*MI, ErrorInfo))
     report(ErrorInfo.data(), MI);
 
   // Verify properties of various specific instruction types
   switch (MI->getOpcode()) {
-  case TargetOpcode::COPY: {
+  case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY: {
     const MachineOperand &DstOp = MI->getOperand(0);
     const MachineOperand &SrcOp = MI->getOperand(1);
     const Register SrcReg = SrcOp.getReg();
     const Register DstReg = DstOp.getReg();
 
     LLT DstTy = MRI->getType(DstReg);
     LLT SrcTy = MRI->getType(SrcReg);
     if (SrcTy.isValid() && DstTy.isValid()) {

llvm/lib/CodeGen/PeepholeOptimizer.cpp

if (!NewSubReg) {
       // Move the current index to an invalid position.
       // We do not want another call to this method to be able
       // to do any change.
       CurrentSrcIdx = -1;
       // Rewrite the operation as a COPY.
       // Get rid of the sub-register index.
       CopyLike.removeOperand(2);
       // Morph the operation into a COPY.
-      CopyLike.setDesc(TII.get(TargetOpcode::COPY));
+      CopyLike.setDesc(TII.get(TII.getCopyOpcode()));
       return true;
     }
     CopyLike.getOperand(CurrentSrcIdx + 1).setImm(NewSubReg);
     return true;
   }
 };
 
 /// Specialized rewriter for REG_SEQUENCE instruction.
static Rewriter *getCopyRewriter(MachineInstr &MI, const TargetInstrInfo &TII) {
   if (MI.isBitcast() || MI.isRegSequenceLike() || MI.isInsertSubregLike() ||
       MI.isExtractSubregLike())
     return new UncoalescableRewriter(MI);
 
   switch (MI.getOpcode()) {
   default:
     return nullptr;
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
     return new CopyRewriter(MI);
   case TargetOpcode::INSERT_SUBREG:
     return new InsertSubregRewriter(MI);
   case TargetOpcode::EXTRACT_SUBREG:
     return new ExtractSubregRewriter(MI, TII);
   case TargetOpcode::REG_SEQUENCE:
     return new RegSequenceRewriter(MI);
   }

llvm/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

 void SelectionDAGISel::Select_UNDEF(SDNode *N) {
   CurDAG->SelectNodeTo(N, TargetOpcode::IMPLICIT_DEF, N->getValueType(0));
 }
 
 void SelectionDAGISel::Select_FREEZE(SDNode *N) {
   // TODO: We don't have FREEZE pseudo-instruction in MachineInstr-level now.
   // If FREEZE instruction is added later, the code below must be changed as
   // well.
-  CurDAG->SelectNodeTo(N, TargetOpcode::COPY, N->getValueType(0),
+  CurDAG->SelectNodeTo(N, TII->getCopyOpcode(), N->getValueType(0),
                        N->getOperand(0));
 }
 
 void SelectionDAGISel::Select_ARITH_FENCE(SDNode *N) {
   CurDAG->SelectNodeTo(N, TargetOpcode::ARITH_FENCE, N->getValueType(0),
                        N->getOperand(0));
 }
 

llvm/lib/CodeGen/SplitKit.cpp

void SplitEditor::forceRecompute(unsigned RegIdx, const VNInfo &ParentVNI) {
 
   // Mark as complex mapped, forced.
   VFP = ValueForcePair(nullptr, true);
 }
 
 SlotIndex SplitEditor::buildSingleSubRegCopy(Register FromReg, Register ToReg,
     MachineBasicBlock &MBB, MachineBasicBlock::iterator InsertBefore,
     unsigned SubIdx, LiveInterval &DestLI, bool Late, SlotIndex Def) {
-  const MCInstrDesc &Desc = TII.get(TargetOpcode::COPY);
+  const MCInstrDesc &Desc = TII.get(TII.getCopyOpcode());
   bool FirstCopy = !Def.isValid();
   MachineInstr *CopyMI = BuildMI(MBB, InsertBefore, DebugLoc(), Desc)
       .addReg(ToReg, RegState::Define | getUndefRegState(FirstCopy)
               | getInternalReadRegState(!FirstCopy), SubIdx)
       .addReg(FromReg, 0, SubIdx);
 
   SlotIndexes &Indexes = *LIS.getSlotIndexes();
   if (FirstCopy) {

llvm/lib/CodeGen/TwoAddressInstructionPass.cpp

for (MachineBasicBlock::iterator mi = MBB->begin(), me = MBB->end();
         // From %reg = INSERT_SUBREG %reg, %subreg, subidx
         // To   %reg:subidx = COPY %subreg
         unsigned SubIdx = mi->getOperand(3).getImm();
         mi->removeOperand(3);
         assert(mi->getOperand(0).getSubReg() == 0 && "Unexpected subreg idx");
         mi->getOperand(0).setSubReg(SubIdx);
         mi->getOperand(0).setIsUndef(mi->getOperand(1).isUndef());
         mi->removeOperand(1);
-        mi->setDesc(TII->get(TargetOpcode::COPY));
+        mi->setDesc(TII->get(TII->getCopyOpcode()));
         LLVM_DEBUG(dbgs() << "\t\tconvert to:\t" << *mi);
 
         // Update LiveIntervals.
         if (LIS) {
           Register Reg = mi->getOperand(0).getReg();
           LiveInterval &LI = LIS->getInterval(Reg);
           if (LI.hasSubRanges()) {
             // The COPY no longer defines subregs of %reg except for
if (isKill)
         if (MI.getOperand(j).getReg() == SrcReg) {
           MI.getOperand(j).setIsKill();
           UseMO.setIsKill(false);
           isKill = false;
           break;
         }
 
     // Insert the sub-register copy.
     MachineInstr *CopyMI = BuildMI(*MI.getParent(), MI, MI.getDebugLoc(),
-                                   TII->get(TargetOpcode::COPY))
+                                   TII->get(TII->getCopyOpcode()))
                                .addReg(DstReg, RegState::Define, SubIdx)
                                .add(UseMO);
 
     // The first def needs an undef flag because there is no live register
     // before it.
     if (!DefEmitted) {
       CopyMI->getOperand(0).setIsUndef(true);
       // Return an iterator pointing to the first inserted instr.

llvm/lib/CodeGen/VLIWMachineScheduler.cpp

if (!ResourcesModel->canReserveResources(*SU->getInstr()))
       return false;
     break;
   case TargetOpcode::EXTRACT_SUBREG:
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::SUBREG_TO_REG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
   case TargetOpcode::INLINEASM:
   case TargetOpcode::INLINEASM_BR:
     break;
   }
 
   // Now see if there are no other dependencies to instructions already
   // in the packet.
   if (IsTop) {
bool VLIWResourceModel::reserveResources(SUnit *SU, bool IsTop) {
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::SUBREG_TO_REG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::KILL:
   case TargetOpcode::CFI_INSTRUCTION:
   case TargetOpcode::EH_LABEL:
   case TargetOpcode::COPY:
+  case TargetOpcode::PRED_COPY:
   case TargetOpcode::INLINEASM:
   case TargetOpcode::INLINEASM_BR:
     break;
   }
   Packet.push_back(SU);
 
 #ifndef NDEBUG
   LLVM_DEBUG(dbgs() << "Packet[" << TotalPackets << "]:\n");

llvm/lib/Target/AArch64/AArch64SchedA53.td

 
 //===----------------------------------------------------------------------===//
 // Subtarget-specific InstRWs.
 
 //---
 // Miscellaneous
 //---
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 //---
 // Vector Loads
 //---
 def : InstRW<[A53WriteVLD1], (instregex "LD1i(8|16|32|64)$")>;
 def : InstRW<[A53WriteVLD1], (instregex "LD1Rv(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
 def : InstRW<[A53WriteVLD1], (instregex "LD1Onev(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
 def : InstRW<[A53WriteVLD2], (instregex "LD1Twov(8b|4h|2s|1d|16b|8h|4s|2d)$")>;

llvm/lib/Target/AArch64/AArch64SchedA55.td

 def : InstRW<[CortexA55WriteVLD1,CortexA55WriteLDP1], (instregex "LDPSi")>;
 def : InstRW<[CortexA55WriteVLD1,CortexA55WriteLDP2], (instregex "LDP(X|D)i")>;
 def : InstRW<[CortexA55WriteVLD1,CortexA55WriteLDP4], (instregex "LDPQi")>;
 def : InstRW<[WriteAdr, CortexA55WriteVLD1SI,CortexA55WriteLDP1], (instregex "LDPS?W(pre|post)")>;
 def : InstRW<[WriteAdr, CortexA55WriteVLD1,CortexA55WriteLDP1], (instregex "LDPS(pre|post)")>;
 def : InstRW<[WriteAdr, CortexA55WriteVLD1,CortexA55WriteLDP2], (instregex "LDP(X|D)(pre|post)")>;
 def : InstRW<[WriteAdr, CortexA55WriteVLD1,CortexA55WriteLDP4], (instregex "LDPQ(pre|post)")>;
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 //---
 // Vector Loads - 64-bit per cycle
 //---
 //   1-element structures
 def : InstRW<[CortexA55WriteVLD1], (instregex "LD1i(8|16|32|64)$")>;                // single element
 def : InstRW<[CortexA55WriteVLD1], (instregex "LD1Rv(8b|4h|2s|1d|16b|8h|4s|2d)$")>; // replicate
 def : InstRW<[CortexA55WriteVLD1], (instregex "LD1Onev(8b|4h|2s|1d)$")>;
 def : InstRW<[CortexA55WriteVLD2], (instregex "LD1Onev(16b|8h|4s|2d)$")>;

llvm/lib/Target/AArch64/AArch64SchedA57.td

 // Specialize the coarse model by associating instruction groups with the
 // subtarget-defined types. As the modeled is refined, this will override most
 // of the above ShchedAlias mappings.
 
 // Miscellaneous
 // -----------------------------------------------------------------------------
 
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 
 // Branch Instructions
 // -----------------------------------------------------------------------------
 
 def : InstRW<[A57Write_1cyc_1B_1I], (instrs BL)>;
 def : InstRW<[A57Write_2cyc_1B_1I], (instrs BLR)>;
 

llvm/lib/Target/AArch64/AArch64SchedA64FX.td

def : InstRW<[WriteI],
                        "ORR?(W|X)r(i|r|s|x)",   "SUB?(W|X)r(i|r|s|x)",
                        "SUBS?(W|X)r(i|r|s|x)",  "SBC(W|X)r",
                        "SBCS(W|X)r",            "CCMN(W|X)(i|r)",
                        "CCMP(W|X)(i|r)",        "CSEL(W|X)r",
                        "CSINC(W|X)r",           "CSINV(W|X)r",
                        "CSNEG(W|X)r")>;
 
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 // ALU, extend and/or shift
 def : WriteRes<WriteISReg,   [A64FXGI2456]> {
   let Latency = 2;
 }
 
 def : InstRW<[WriteISReg],
             (instregex "ADD?(W|X)r(i|r|s|x)",   "ADDS?(W|X)r(i|r|s|x)(64)?",

llvm/lib/Target/AArch64/AArch64SchedAmpere1.td

 def : ReadAdvance<ReadST,      0>;
 def : ReadAdvance<ReadAdrBase, 0>;
 def : ReadAdvance<ReadVLD,     0>;
 
 //===----------------------------------------------------------------------===//
 // Specialising the scheduling model further for Ampere-1.
 
 def : InstRW<[Ampere1Write_1cyc_1AB], (instrs COPY)>;
+def : InstRW<[Ampere1Write_1cyc_1AB], (instrs PRED_COPY)>;
 
 // Branch instructions
 def : InstRW<[Ampere1Write_1cyc_1A], (instrs Bcc, BL, RET)>;
 def : InstRW<[Ampere1Write_1cyc_1A],
         (instrs CBZW, CBZX, CBNZW, CBNZX, TBZW, TBZX, TBNZW, TBNZX)>;
 def : InstRW<[Ampere1Write_1cyc_2A], (instrs BLR)>;
 
 // Cryptography instructions

llvm/lib/Target/AArch64/AArch64SchedCyclone.td

 // ORR Xd, XZR, Xm
 // ADD Xd, Xn, #0
 def WriteIMovPred : SchedPredicate<[{TII->isGPRCopy(*MI)}]>;
 def WriteVMovPred : SchedPredicate<[{TII->isFPRCopy(*MI)}]>;
 def WriteMov      : SchedWriteVariant<[
                       SchedVar<WriteIMovPred, [WriteX]>,
                       SchedVar<WriteVMovPred, [WriteX]>,
                       SchedVar<NoSchedPred,   [WriteI]>]>;
-def : InstRW<[WriteMov], (instrs COPY,ORRXrr,ADDXrr)>;
+def : InstRW<[WriteMov], (instrs COPY,PRED_COPY,ORRXrr,ADDXrr)>;
 
 // Move non-zero immediate is an integer ALU op.
 // MOVN,MOVZ,MOVK
 def : WriteRes<WriteImm, [CyUnitI]>;
 
 //---
 // 7.8.2-7.8.5. Arithmetic and Logical, Comparison, Conditional,
 //              Shifts and Bitfield Operations

llvm/lib/Target/AArch64/AArch64SchedExynosM3.td

 def : InstRW<[M3WriteAU], (instrs ORRWrs, ORRXrs)>;
 def : InstRW<[M3WriteAX], (instregex "^(ADD|SUB)S?[WX]rx(64)?$")>;
 def : InstRW<[M3WriteAX], (instregex "^(ADD|AND|BIC|SUB)S[WX]rs$")>;
 def : InstRW<[M3WriteAV], (instrs ADDWri, ADDXri)>;
 def : InstRW<[M3WriteAW], (instrs ORRWri, ORRXri)>;
 
 // Move instructions.
 def : InstRW<[M3WriteCOPY], (instrs COPY)>;
+def : InstRW<[M3WriteCOPY], (instrs PRED_COPY)>;
 def : InstRW<[M3WriteZ0],   (instrs ADR, ADRP)>;
 def : InstRW<[M3WriteZ0],   (instregex "^MOV[NZ][WX]i")>;
 
 // Divide and multiply instructions.
 
 // Miscellaneous instructions.
 
 // Load instructions.

llvm/lib/Target/AArch64/AArch64SchedExynosM4.td

 def : InstRW<[M4WriteAX], (instregex "^(ADD|AND|BIC|EON|EOR|ORN|SUB)[WX]rs$")>;
 def : InstRW<[M4WriteAU], (instrs ORRWrs, ORRXrs)>;
 def : InstRW<[M4WriteAX], (instregex "^(ADD|AND|BIC|SUB)S[WX]rs$")>;
 def : InstRW<[M4WriteAX], (instregex "^(ADD|SUB)S?[WX]rx(64)?$")>;
 def : InstRW<[M4WriteAV], (instrs ADDWri, ADDXri, ORRWri, ORRXri)>;
 
 // Move instructions.
 def : InstRW<[M4WriteCOPY], (instrs COPY)>;
+def : InstRW<[M4WriteCOPY], (instrs PRED_COPY)>;
 def : InstRW<[M4WriteZ0],   (instrs ADR, ADRP)>;
 def : InstRW<[M4WriteZ0],   (instregex "^MOV[NZ][WX]i")>;
 
 // Divide and multiply instructions.
 
 // Miscellaneous instructions.
 
 // Load instructions.

llvm/lib/Target/AArch64/AArch64SchedExynosM5.td

 def : InstRW<[M5WriteAVX], (instrs ADDXri, ORRXri)>;
 def : InstRW<[M5WriteA1W], (instregex "^CCM[NP]W[ir]$")>;
 def : InstRW<[M5WriteA1X], (instregex "^CCM[NP]X[ir]$")>;
 def : InstRW<[M5WriteA1W], (instrs CSELWr, CSINCWr, CSINVWr, CSNEGWr)>;
 def : InstRW<[M5WriteA1X], (instrs CSELXr, CSINCXr, CSINVXr, CSNEGXr)>;
 
 // Move instructions.
 def : InstRW<[M5WriteCOPY], (instrs COPY)>;
+def : InstRW<[M5WriteCOPY], (instrs PRED_COPY)>;
 def : InstRW<[M5WriteZ0],   (instrs ADR, ADRP)>;
 def : InstRW<[M5WriteZ0],   (instregex "^MOV[NZ][WX]i$")>;
 
 // Shift instructions.
 def : InstRW<[M5WriteA1W], (instrs ASRVWr, LSLVWr, LSRVWr, RORVWr)>;
 def : InstRW<[M5WriteA1X], (instrs ASRVXr, LSLVXr, LSRVXr, RORVXr)>;
 
 // Miscellaneous instructions.

llvm/lib/Target/AArch64/AArch64SchedFalkorDetails.td

 // subtarget-defined types. As the modeled is refined, this will override most
 // of the earlier mappings.
 
 // Miscellaneous
 // -----------------------------------------------------------------------------
 
 // FIXME: This could be better modeled by looking at the regclasses of the operands.
 def : InstRW<[FalkorWr_1XYZ_1cyc], (instrs COPY)>;
+def : InstRW<[FalkorWr_1XYZ_1cyc], (instrs PRED_COPY)>;
 
 // SIMD Floating-point Instructions
 // -----------------------------------------------------------------------------
 def : InstRW<[FalkorWr_1VXVY_1cyc],   (instregex "^(FABS|FNEG)v2f32$")>;
 
 def : InstRW<[FalkorWr_1VXVY_2cyc],   (instregex "^(F(MAX|MIN)(NM)?P?|FAC(GE|GT))(v2f32|v2i32p)$")>;
 def : InstRW<[FalkorWr_1VXVY_2cyc],   (instregex "^FAC(GE|GT)(32|64)$")>;
 def : InstRW<[FalkorWr_1VXVY_2cyc],   (instregex "^FCM(EQ|GE|GT)(32|64|v2f32|v2i32)$")>;

llvm/lib/Target/AArch64/AArch64SchedKryo.td

 // Specialize the coarse model by associating instruction groups with the
 // subtarget-defined types. As the modeled is refined, this will override most
 // of the above SchedWriteRes and SchedAlias mappings.
 
 // Miscellaneous
 // -----------------------------------------------------------------------------
 
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 
 // Detailed Refinedments
 // -----------------------------------------------------------------------------
 include "AArch64SchedKryoDetails.td"
 
 
 } // SchedModel = KryoModel

llvm/lib/Target/AArch64/AArch64SchedNeoverseN2.td

def N2Write_11cyc_9L01_9S_9V : SchedWriteRes<[N2UnitL01, N2UnitL01, N2UnitL01,
   let Latency     = 11;
   let NumMicroOps = 27;
 }
 
 // Miscellaneous
 // -----------------------------------------------------------------------------
 
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 // Branch Instructions
 // -----------------------------------------------------------------------------
 
 // Branch, immed
 // Compare and branch
 def : SchedAlias<WriteBr,    N2Write_1cyc_1B>;
 

llvm/lib/Target/AArch64/AArch64SchedTSV110.td

 def : ReadAdvance<ReadIMA,     2, [WriteIM32, WriteIM64]>;
 def : ReadAdvance<ReadID,      0>;
 def : ReadAdvance<ReadExtrHi,  0>;
 def : ReadAdvance<ReadAdrBase, 0>;
 def : ReadAdvance<ReadVLD,     0>;
 def : ReadAdvance<ReadST,      0>;
 
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 // Detailed Refinements
 //===----------------------------------------------------------------------===//
 
 // Contains all of the TSV110 specific SchedWriteRes types. The approach
 // below is to define a generic SchedWriteRes for every combination of
 // latency and microOps. The naming conventions is to use a prefix, one field
 // for latency, and one or more microOp count/type designators.

llvm/lib/Target/AArch64/AArch64SchedThunderX.td

 // Ret
 //---
 def : InstRW<[THXT8XWriteRET], (instregex "^RET$")>;
 
 //---
 // Miscellaneous
 //---
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 //---
 // Vector Loads
 //---
 def : InstRW<[THXT8XWriteVLD1], (instregex "LD1i(8|16|32|64)$")>;
 def : InstRW<[THXT8XWriteVLD1], (instregex "LD1Rv(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
 def : InstRW<[THXT8XWriteVLD1], (instregex "LD1Onev(8b|4h|2s|1d|16b|8h|4s|2d)$")>;
 def : InstRW<[THXT8XWriteVLD2], (instregex "LD1Twov(8b|4h|2s|1d|16b|8h|4s|2d)$")>;

llvm/lib/Target/AArch64/AArch64SchedThunderX2T99.td

def : InstRW<[WriteI],
                        "ORR?(W|X)r(i|r|s|x)",   "SUB?(W|X)r(i|r|s|x)",
                        "SUBS?(W|X)r(i|r|s|x)",  "SBC(W|X)r",
                        "SBCS(W|X)r",            "CCMN(W|X)(i|r)",
                        "CCMP(W|X)(i|r)",        "CSEL(W|X)r",
                        "CSINC(W|X)r",           "CSINV(W|X)r",
                        "CSNEG(W|X)r")>;
 
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 // ALU, extend and/or shift
 def : WriteRes<WriteISReg,   [THX2T99I012]> {
   let Latency = 2;
   let ResourceCycles = [2];
   let NumMicroOps = 2;
 }
 

llvm/lib/Target/AArch64/AArch64SchedThunderX3T110.td

def : InstRW<[WriteI],
                        "ORR?(W|X)r(i|r|s|x)",   "SUB?(W|X)r(i|r|s|x)",
                        "SUBS?(W|X)r(i|r|s|x)",  "SBC(W|X)r",
                        "SBCS(W|X)r",            "CCMN(W|X)(i|r)",
                        "CCMP(W|X)(i|r)",        "CSEL(W|X)r",
                        "CSINC(W|X)r",           "CSINV(W|X)r",
                        "CSNEG(W|X)r")>;
 
 def : InstRW<[WriteI], (instrs COPY)>;
+def : InstRW<[WriteI], (instrs PRED_COPY)>;
 
 // ALU, extend and/or shift
 def : WriteRes<WriteISReg,   [THX3T110I0123]> {
   let Latency = 2;
   let ResourceCycles = [2];
   let NumMicroOps = 2;
 }
 

llvm/lib/Target/AMDGPU/SISchedule.td

 def : HWVALUWriteRes<WriteFloatFMA,    1>;
 def : HWVALUWriteRes<WriteDouble,      4>;
 def : HWVALUWriteRes<WriteDoubleAdd,   2>;
 def : HWVALUWriteRes<WriteDoubleCvt,   4>;
 def : HWVALUWriteRes<WriteTrans64,     4>;
 } // End RetireOOO = 1
 
 def : InstRW<[WriteCopy], (instrs COPY)>;
+def : InstRW<[WriteCopy], (instrs PRED_COPY)>;
 
 } // End SchedModel = SIFullSpeedModel
 
 let SchedModel = SIQuarterSpeedModel in {
 
 defm : SICommonWriteRes;
 
 let RetireOOO = 1 in { // llvm-mca specific flag
 def : HWVALUWriteRes<Write64Bit,       2>;
 def : HWVALUWriteRes<WriteIntMul,      4>;
 def : HWVALUWriteRes<WriteFloatFMA,    16>;
 def : HWVALUWriteRes<WriteDouble,      16>;
 def : HWVALUWriteRes<WriteDoubleAdd,    8>;
 def : HWVALUWriteRes<WriteDoubleCvt,    4>;
 def : HWVALUWriteRes<WriteTrans64,     16>;
 } // End RetireOOO = 1
 
 def : InstRW<[WriteCopy], (instrs COPY)>;
+def : InstRW<[WriteCopy], (instrs PRED_COPY)>;
 def : InstRW<[Write64Bit, MIReadVGPR], (instregex "^V_ACCVGPR_WRITE_B32_e64$")>;
 def : InstRW<[Write2PassMAI,  MIMFMARead], (instregex "^V_MFMA_..._4X4X")>;
 def : InstRW<[Write8PassMAI,  MIMFMARead], (instregex "^V_MFMA_..._16X16X")>;
 def : InstRW<[Write16PassMAI, MIMFMARead], (instregex "^V_MFMA_..._32X32X")>;
 
 }  // End SchedModel = SIQuarterSpeedModel
 
 let SchedModel = SIDPFullSpeedModel in {
 
 defm : SICommonWriteRes;
 
 let RetireOOO = 1 in { // llvm-mca specific flag
 def : HWVALUWriteRes<WriteFloatFMA,    1>;
 def : HWVALUWriteRes<WriteDouble,      1>;
 def : HWVALUWriteRes<WriteDoubleAdd,   1>;
 def : HWVALUWriteRes<WriteDoubleCvt,   1>;
 def : HWVALUWriteRes<WriteTrans64,     4>;
 def : HWVALUWriteRes<WriteIntMul,      1>;
 def : HWVALUWriteRes<Write64Bit,       1>;
 } // End RetireOOO = 1
 
 def : InstRW<[WriteCopy], (instrs COPY)>;
+def : InstRW<[WriteCopy], (instrs PRED_COPY)>;
 def : InstRW<[Write64Bit], (instregex "^V_ACCVGPR_WRITE_B32_e64$")>;
 def : InstRW<[Write2PassMAI,   MIMFMARead], (instregex "^V_MFMA_.32_4X4X")>;
 def : InstRW<[Write8PassMAI,   MIMFMARead], (instregex "^V_MFMA_.32_16X16X")>;
 def : InstRW<[Write16PassMAI,  MIMFMARead], (instregex "^V_MFMA_.32_32X32X")>;
 def : InstRW<[Write4PassDGEMM, MIMFMARead], (instregex "^V_MFMA_.64_4X4X")>;
 def : InstRW<[Write8PassDGEMM, MIMFMARead], (instregex "^V_MFMA_.64_16X16X")>;
 
 } // End SchedModel = SIDPFullSpeedModel
 
 let SchedModel = SIDPGFX940FullSpeedModel in {
 
 defm : SICommonWriteRes;
 
 def : HWVALUWriteRes<WriteFloatFMA,    1>;
 def : HWVALUWriteRes<WriteDouble,      1>;
 def : HWVALUWriteRes<WriteDoubleAdd,   1>;
 def : HWVALUWriteRes<WriteDoubleCvt,   1>;
 def : HWVALUWriteRes<WriteTrans64,     4>;
 def : HWVALUWriteRes<WriteIntMul,      1>;
 def : HWVALUWriteRes<Write64Bit,       1>;
 
 def : InstRW<[WriteCopy], (instrs COPY)>;
+def : InstRW<[WriteCopy], (instrs PRED_COPY)>;
 def : InstRW<[Write64Bit], (instregex "^V_ACCVGPR_WRITE_B32_e64$")>;
 def : InstRW<[Write2PassMAI,   MIMFMARead], (instregex "^V_MFMA_.32_4X4X")>;
 
 def : InstRW<[Write4PassMAI,   MIMFMARead], (instregex "^V_MFMA_.32_16X16X8X")>;
 def : InstRW<[Write4PassMAI,   MIMFMARead], (instregex "^V_MFMA_.32_16X16X16")>;
 def : InstRW<[Write4PassMAI,   MIMFMARead], (instregex "^V_MFMA_.32_16X16X32")>;
 def : InstRW<[Write8PassMAI,   MIMFMARead], (instregex "^V_MFMA_.32_16X16X[14][FBI]")>;
 

 def : HWWriteRes<WriteLDS,           [HWLGKM,   HWRC], 20>;
 def : HWWriteRes<WriteSALU,          [HWSALU,   HWRC], 2>;
 def : HWWriteRes<WriteSMEM,          [HWLGKM,   HWRC], 20>;
 def : HWWriteRes<WriteVMEM,          [HWVMEM,   HWRC], 320>;
 def : HWWriteRes<WriteBarrier,       [HWBranch],       2000>;
 } // End RetireOOO = 1
 
 def : InstRW<[WriteCopy], (instrs COPY)>;
+def : InstRW<[WriteCopy], (instrs PRED_COPY)>;
 
 }  // End SchedModel = GFX10SpeedModel
 
 let SchedModel = GFX11SpeedModel in {
 
 def : HWWriteRes<Write32Bit,         [HWVALU, HWRC],   5>;
 def : HWWriteRes<WriteFloatCvt,      [HWVALU, HWRC],   5>;
 def : HWWriteRes<Write64Bit,         [HWVALU, HWRC],   6>;

 def : HWWriteRes<WriteExport,        [HWExport, HWRC], 16>;
 def : HWWriteRes<WriteLDS,           [HWLGKM,   HWRC], 20>;
 def : HWWriteRes<WriteSALU,          [HWSALU,   HWRC], 2>;
 def : HWWriteRes<WriteSMEM,          [HWLGKM,   HWRC], 20>;
 def : HWWriteRes<WriteVMEM,          [HWVMEM,   HWRC], 320>;
 def : HWWriteRes<WriteBarrier,       [HWBranch],       2000>;
 
 def : InstRW<[WriteCopy], (instrs COPY)>;
+def : InstRW<[WriteCopy], (instrs PRED_COPY)>;
 
 }  // End SchedModel = GFX11SpeedModel

llvm/lib/Target/AMDGPU/VOP3Instructions.td

[{
      if (auto *C = dyn_cast<ConstantSDNode>(N->getOperand(1))) {
        return C->getZExtValue() <= 4;
      }
      return false;
    }]> {
   let GISelPredicateCode = [{
     int64_t Imm = 0;
     if (!mi_match(MI.getOperand(2).getReg(), MRI, m_ICst(Imm)) &&
-        !mi_match(MI.getOperand(2).getReg(), MRI, m_Copy(m_ICst(Imm))))
+        !mi_match(MI.getOperand(2).getReg(), MRI, m_Copy(m_ICst(Imm))) &&
+        !mi_match(MI.getOperand(2).getReg(), MRI, m_Pred_Copy(m_ICst(Imm))))
       return false;
     return (uint64_t)Imm <= 4;
   }];
 }
 
 def VOP3_CVT_PK_F8_F32_Profile : VOP3_Profile<VOP_I32_F32_F32, VOP3_OPSEL> {
   let InsVOP3OpSel = (ins FP32InputMods:$src0_modifiers, Src0RC64:$src0,
                           FP32InputMods:$src1_modifiers, Src1RC64:$src1,

llvm/lib/Target/ARM/ARMScheduleA57.td

def : InstRW<[WriteNoop], (instregex "(t2)?ABS$",
   "VST(1|2)LN(d|q)(WB_fixed_|WB_register_)?Asm",
   "VST(3|4)(DUP|LN)?(d|q)(WB_fixed_|WB_register_)?Asm",
   "WIN__CHKSTK", "WIN__DBZCHK")>;
 
 // Miscellaneous
 // -----------------------------------------------------------------------------
 
 def : InstRW<[A57Write_1cyc_1I], (instrs COPY)>;
+def : InstRW<[A57Write_1cyc_1I], (instrs PRED_COPY)>;
 
 // --- 3.2 Branch Instructions ---
 // B, BX, BL, BLX (imm, reg != LR, reg == LR), CBZ, CBNZ
 
 def : InstRW<[A57Write_1cyc_1B], (instregex "(t2|t)?B$", "t?BX", "(t2|t)?Bcc$",
   "t?TAILJMP(d|r)", "TCRETURN(d|r)i", "tBfar", "tCBN?Z")>;
 def : InstRW<[A57Write_1cyc_1B_1I],
   (instregex "t?BL$", "BL_pred$", "t?BLXi", "t?TPsoft")>;

llvm/lib/Target/ARM/ARMScheduleM4.td

 def : M4UnitL1<WriteMUL32>;
 def : M4UnitL1<WriteMUL64Hi>;
 def : M4UnitL1<WriteMUL64Lo>;
 def : M4UnitL1<WriteMUL16>;
 def : M4UnitL1<WriteNoop>;
 def : M4UnitL1<WritePreLd>;
 def : M4UnitL1I<(instregex "(t|t2)MOV")>;
 def : M4UnitL1I<(instrs COPY)>;
+def : M4UnitL1I<(instrs PRED_COPY)>;
 def : M4UnitL1I<(instregex "t2IT", "t2MSR", "t2MRS")>;
 def : M4UnitL1I<(instregex "t2CLREX")>;
 def : M4UnitL1I<(instregex "t2SEL", "t2USAD8", "t2SML[AS]",
     "t2(S|Q|SH|U|UQ|UH|QD)(ADD|ASX|SAX|SUB)", "t2USADA8", "(t|t2)REV")>;
 
 // These instructions are not of much interest to scheduling as they will not
 // be generated or it is not very useful to schedule them. They are here to make
 // the model more complete.

llvm/lib/Target/ARM/ARMScheduleR52.td

 def : SchedAlias<WriteFPMOV, R52WriteFPALU_F3>;
 def : SchedAlias<WriteFPALU32, R52WriteFPALU_F5>;
 def : SchedAlias<WriteFPALU64, R52WriteFPALU_F5>;
 
 //===----------------------------------------------------------------------===//
 // Subtarget-specific overrides. Map opcodes to list of SchedReadWrites types.
 //
 def : InstRW<[WriteALU], (instrs COPY)>;
+def : InstRW<[WriteALU], (instrs PRED_COPY)>;
 
 def : InstRW<[R52WriteALU_EX2, R52Read_EX1, R52Read_ISS],
       (instregex "SXTB", "SXTH", "SXTB16", "UXTB", "UXTH", "UXTB16",
       "t2SXTB", "t2SXTH", "t2SXTB16", "t2UXTB", "t2UXTH", "t2UXTB16")>;
 
 def : InstRW<[R52WriteALU_EX1, R52Read_ISS],
       (instregex "MOVCCi32imm", "MOVi32imm", "t2MOVCCi", "t2MOVi")>;
 def : InstRW<[R52WriteALU_EX2, R52Read_EX1],

llvm/lib/Target/Mips/MipsScheduleGeneric.td

 // srlv, ssnop, sub, subu, wsbh, xor, xori
 def : InstRW<[GenericWriteALU], (instrs ADD, ADDi, ADDiu, ADDu, AND, ANDi,
                                  CLO, CLZ, EXT, INS, LEA_ADDiu, LUi, NOP,
                                  NOR, OR, ORi, ROTR, ROTRV, SEB, SEH, SLL,
                                  SLLV, SLT, SLTi, SLTiu, SLTu, SRA, SRAV, SRL,
                                  SRLV, SSNOP, SUB, SUBu, WSBH, XOR, XORi)>;
 
 def : InstRW<[GenericWriteALU], (instrs COPY)>;
+def : InstRW<[GenericWriteALU], (instrs PRED_COPY)>;
 
 // MIPSR6
 // ======
 
 // addiupc, align, aluipc, aui, auipc, bitswap, clo, clz, lsa, seleqz, selnez
 def : InstRW<[GenericWriteALU], (instrs ADDIUPC, ALIGN, ALUIPC, AUI,
                                  AUIPC, BITSWAP, CLO_R6, CLZ_R6, LSA_R6,
                                  SELEQZ, SELNEZ)>;

llvm/lib/Target/Mips/MipsScheduleP5600.td

def P5600WriteEitherALU : SchedWriteVariant<
   [SchedVar<SchedPredicate<[{1}]>, [P5600WriteALU]>,
    SchedVar<SchedPredicate<[{0}]>, [P5600WriteAL2]>
   ]>;
 
 // add, addi, addiu, addu, andi, ori, rotr, se[bh], sllv?, sr[al]v?, slt, sltu,
 // xori
 def : InstRW<[P5600WriteEitherALU], (instrs ADD, ADDi, ADDiu, ANDi, ORi, ROTR,
                                      SEB, SEH, SLT, SLTu, SLL, SRA, SRL, XORi,
-                                     ADDu, SLLV, SRAV, SRLV, LSA, COPY)>;
+                                     ADDu, SLLV, SRAV, SRLV, LSA, COPY, PRED_COPY)>;
 
 // FPU Pipelines
 // =============
 
 def P5600FPQ : ProcResource<3> { let BufferSize = 16; }
 def P5600IssueFPUS : ProcResource<1> { let Super = P5600FPQ; }
 def P5600IssueFPUL : ProcResource<1> { let Super = P5600FPQ; }
 def P5600IssueFPULoad : ProcResource<1> { let Super = P5600FPQ; }

llvm/lib/Target/PowerPC/P10InstrResources.td

def : InstRW<[P10W_FX_3C, P10W_DISP_ANY, P10FX_Read, P10FX_Read],
     EQV, EQV8,
     EQV8_rec, EQV_rec,
     EXTSWSLI, EXTSWSLI_32_64,
     FCPSGND, FCPSGNS,
     NAND, NAND8,
     NAND8_rec, NAND_rec,
     NOR, NOR8,
     NOR8_rec, NOR_rec,
-    COPY, OR, OR8,
+    COPY, PRED_COPY, OR, OR8,
     OR8_rec, OR_rec,
     ORC, ORC8,
     ORC8_rec, ORC_rec,
     ORIS, ORIS8,
     SLD,
     SLW, SLW8,
     SRAD,
     SRADI, SRADI_32,

llvm/lib/Target/PowerPC/P9InstrResources.td

def : InstRW<[P9_ALU_2C, IP_EXEC_1C, DISP_1C],
     (instregex "OR(C)?(8)?(_rec)?$"),
     (instregex "EQV(8)?(_rec)?$"),
     (instregex "EXTS(B|H|W)(8)?(_32)?(_64)?(_rec)?$"),
     (instregex "ADD(4|8)(TLS)?(_)?$"),
     (instregex "NEG(8)?(O)?$"),
     (instregex "ADDI(S)?toc(HA|L)(8)?$"),
     (instregex "LA(8)?$"),
     COPY,
+    PRED_COPY,
     MCRF,
     MCRXRX,
     XSNABSDP,
     XSNABSDPs,
     XSXEXPDP,
     XSABSDP,
     XSNEGDP,
     XSCPSGNDP,

llvm/lib/Target/RISCV/RISCVSchedRocket.td

 def : WriteRes<WriteFSqrt64, [RocketUnitFPDivSqrt]> { let Latency = 25;
                                                       let ResourceCycles = [25]; }
 
 // Others
 def : WriteRes<WriteCSR, []>;
 def : WriteRes<WriteNop, []>;
 
 def : InstRW<[WriteIALU], (instrs COPY)>;
+def : InstRW<[WriteIALU], (instrs PRED_COPY)>;
 
 //===----------------------------------------------------------------------===//
 // Bypass and advance
 def : ReadAdvance<ReadJmp, 0>;
 def : ReadAdvance<ReadJalr, 0>;
 def : ReadAdvance<ReadCSR, 0>;
 def : ReadAdvance<ReadStoreData, 0>;
 def : ReadAdvance<ReadMemBase, 0>;

llvm/lib/Target/RISCV/RISCVSchedSiFive7.td

 def : WriteRes<WriteFMovF64ToI64, [SiFive7PipeB]>;
 }
 
 // Others
 def : WriteRes<WriteCSR, [SiFive7PipeB]>;
 def : WriteRes<WriteNop, []>;
 
 def : InstRW<[WriteIALU], (instrs COPY)>;
+def : InstRW<[WriteIALU], (instrs PRED_COPY)>;
 
 //===----------------------------------------------------------------------===//
 // Bypass and advance
 def : ReadAdvance<ReadJmp, 0>;
 def : ReadAdvance<ReadJalr, 0>;
 def : ReadAdvance<ReadCSR, 0>;
 def : ReadAdvance<ReadStoreData, 0>;
 def : ReadAdvance<ReadMemBase, 0>;

llvm/lib/Target/SystemZ/SystemZScheduleZ13.td

 def : InstRW<[WLat1, FXb, LSU, NormalGr], (instregex "MVI(Y)?$")>;
 
 // Move character
 def : InstRW<[WLat1, FXb, LSU3, GroupAlone], (instregex "MVC$")>;
 def : InstRW<[WLat30, WLat30, WLat30, MCD], (instregex "MVCL(E|U)?$")>;
 
 // Pseudo -> reg move
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "COPY(_TO_REGCLASS)?$")>;
+def : InstRW<[WLat1, FXa, NormalGr], (instregex "PRED_COPY$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "EXTRACT_SUBREG$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "INSERT_SUBREG$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "REG_SEQUENCE$")>;
 
 // Loads
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "L(Y|FH|RL|Mux)?$")>;
 def : InstRW<[LSULatency, LSULatency, LSU, NormalGr], (instregex "LCBB$")>;
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "LG(RL)?$")>;

llvm/lib/Target/SystemZ/SystemZScheduleZ14.td

 def : InstRW<[WLat1, FXb, LSU, NormalGr], (instregex "MVI(Y)?$")>;
 
 // Move character
 def : InstRW<[WLat1, FXb, LSU3, GroupAlone], (instregex "MVC$")>;
 def : InstRW<[WLat30, WLat30, WLat30, MCD], (instregex "MVCL(E|U)?$")>;
 
 // Pseudo -> reg move
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "COPY(_TO_REGCLASS)?$")>;
+def : InstRW<[WLat1, FXa, NormalGr], (instregex "PRED_COPY$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "EXTRACT_SUBREG$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "INSERT_SUBREG$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "REG_SEQUENCE$")>;
 
 // Loads
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "L(Y|FH|RL|Mux)?$")>;
 def : InstRW<[LSULatency, LSULatency, LSU, NormalGr], (instregex "LCBB$")>;
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "LG(RL)?$")>;

llvm/lib/Target/SystemZ/SystemZScheduleZ15.td

 
 // Move character
 def : InstRW<[WLat1, FXb, LSU3, GroupAlone], (instregex "MVC$")>;
 def : InstRW<[WLat30, WLat30, WLat30, MCD], (instregex "MVCL(E|U)?$")>;
 def : InstRW<[WLat1, LSU2, GroupAlone], (instregex "MVCRL$")>;
 
 // Pseudo -> reg move
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "COPY(_TO_REGCLASS)?$")>;
+def : InstRW<[WLat1, FXa, NormalGr], (instregex "PRED_COPY$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "EXTRACT_SUBREG$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "INSERT_SUBREG$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "REG_SEQUENCE$")>;
 
 // Loads
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "L(Y|FH|RL|Mux)?$")>;
 def : InstRW<[LSULatency, LSULatency, LSU, NormalGr], (instregex "LCBB$")>;
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "LG(RL)?$")>;

llvm/lib/Target/SystemZ/SystemZScheduleZ16.td

 
 // Move character
 def : InstRW<[WLat1, FXb, LSU3, GroupAlone], (instregex "MVC$")>;
 def : InstRW<[WLat30, WLat30, WLat30, MCD], (instregex "MVCL(E|U)?$")>;
 def : InstRW<[WLat1, LSU2, GroupAlone], (instregex "MVCRL$")>;
 
 // Pseudo -> reg move
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "COPY(_TO_REGCLASS)?$")>;
+def : InstRW<[WLat1, FXa, NormalGr], (instregex "PRED_COPY$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "EXTRACT_SUBREG$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "INSERT_SUBREG$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "REG_SEQUENCE$")>;
 
 // Loads
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "L(Y|FH|RL|Mux)?$")>;
 def : InstRW<[LSULatency, LSULatency, LSU, NormalGr], (instregex "LCBB$")>;
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "LG(RL)?$")>;

llvm/lib/Target/SystemZ/SystemZScheduleZ196.td

 def : InstRW<[WLat1, FXU, LSU, NormalGr], (instregex "MVI(Y)?$")>;
 
 // Move character
 def : InstRW<[WLat1, FXU, LSU3, GroupAlone], (instregex "MVC$")>;
 def : InstRW<[WLat30, WLat30, WLat30, MCD], (instregex "MVCL(E|U)?$")>;
 
 // Pseudo -> reg move
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "COPY(_TO_REGCLASS)?$")>;
+def : InstRW<[WLat1, FXU, NormalGr], (instregex "PRED_COPY$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "EXTRACT_SUBREG$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "INSERT_SUBREG$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "REG_SEQUENCE$")>;
 
 // Loads
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "L(Y|FH|RL|Mux)?$")>;
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "LG(RL)?$")>;
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "L128$")>;

llvm/lib/Target/SystemZ/SystemZScheduleZEC12.td

 def : InstRW<[WLat1, FXU, LSU, NormalGr], (instregex "MVI(Y)?$")>;
 
 // Move character
 def : InstRW<[WLat1, FXU, LSU3, GroupAlone], (instregex "MVC$")>;
 def : InstRW<[WLat30, WLat30, WLat30, MCD], (instregex "MVCL(E|U)?$")>;
 
 // Pseudo -> reg move
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "COPY(_TO_REGCLASS)?$")>;
+def : InstRW<[WLat1, FXU, NormalGr], (instregex "PRED_COPY$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "EXTRACT_SUBREG$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "INSERT_SUBREG$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "REG_SEQUENCE$")>;
 
 // Loads
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "L(Y|FH|RL|Mux)?$")>;
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "LG(RL)?$")>;
 def : InstRW<[LSULatency, LSU, NormalGr], (instregex "L128$")>;

llvm/lib/Target/X86/X86SchedBroadwell.td

 // Store_data on 4.
 defm : X86WriteRes<WriteStore,   [BWPort237, BWPort4], 1, [1,1], 1>;
 defm : X86WriteRes<WriteStoreNT, [BWPort237, BWPort4], 1, [1,1], 2>;
 defm : X86WriteRes<WriteLoad,    [BWPort23], 5, [1], 1>;
 defm : X86WriteRes<WriteMove,    [BWPort0156], 1, [1], 1>;
 
 // Treat misc copies as a move.
 def  : InstRW<[WriteMove], (instrs COPY)>;
+def  : InstRW<[WriteMove], (instrs PRED_COPY)>;
 
 // Idioms that clear a register, like xorps %xmm0, %xmm0.
 // These can often bypass execution ports completely.
 def  : WriteRes<WriteZero,       []>;
 
 // Model the effect of clobbering the read-write mask operand of the GATHER operation.
 // Does not cost anything by itself, only has latency, matching that of the WriteLoad,
 defm : X86WriteRes<WriteVecMaskedGatherWriteback, [], 5, [], 0>;

llvm/lib/Target/X86/X86SchedSandyBridge.td

 
 def : WriteRes<WriteStore,   [SBPort23, SBPort4]>;
 def : WriteRes<WriteStoreNT, [SBPort23, SBPort4]>;
 def : WriteRes<WriteLoad,    [SBPort23]> { let Latency = 5; }
 def : WriteRes<WriteMove,    [SBPort015]>;
 
 // Treat misc copies as a move.
 def : InstRW<[WriteMove], (instrs COPY)>;
+def : InstRW<[WriteMove], (instrs PRED_COPY)>;
 
 // Idioms that clear a register, like xorps %xmm0, %xmm0.
 // These can often bypass execution ports completely.
 def : WriteRes<WriteZero,    []>;
 
 // Model the effect of clobbering the read-write mask operand of the GATHER operation.
 // Does not cost anything by itself, only has latency, matching that of the WriteLoad,
 defm : X86WriteRes<WriteVecMaskedGatherWriteback, [], 5, [], 0>;

llvm/lib/Target/X86/X86ScheduleAtom.td

 def : WriteRes<WriteLoad,    [AtomPort0]>;
 def : WriteRes<WriteStore,   [AtomPort0]>;
 def : WriteRes<WriteStoreNT, [AtomPort0]>;
 def : WriteRes<WriteMove,    [AtomPort01]>;
 defm : X86WriteResUnsupported<WriteVecMaskedGatherWriteback>;
 
 // Treat misc copies as a move.
 def : InstRW<[WriteMove], (instrs COPY)>;
+def : InstRW<[WriteMove], (instrs PRED_COPY)>;
 
 ////////////////////////////////////////////////////////////////////////////////
 // Idioms that clear a register, like xorps %xmm0, %xmm0.
 // These can often bypass execution ports completely.
 ////////////////////////////////////////////////////////////////////////////////
 
 def : WriteRes<WriteZero,  []>;
 

llvm/lib/Target/X86/X86ScheduleBdVer2.td

 
 // Load/store MXCSR.
 // FIXME: These are copy and pasted from WriteLoad/Store.
 def : WriteRes<WriteLDMXCSR, [PdLoad]> { let Latency = 5; }
 def : WriteRes<WriteSTMXCSR, [PdStore]> { let NumMicroOps = 2; let ResourceCycles = [18]; }
 
 // Treat misc copies as a move.
 def : InstRW<[WriteMove], (instrs COPY)>;
+def : InstRW<[WriteMove], (instrs PRED_COPY)>;
 
 ////////////////////////////////////////////////////////////////////////////////
 // Idioms that clear a register, like xorps %xmm0, %xmm0.
 // These can often bypass execution ports completely.
 ////////////////////////////////////////////////////////////////////////////////
 
 def : WriteRes<WriteZero, [/*No ExePorts*/]>;
 

llvm/lib/Target/X86/X86ScheduleSLM.td

 defm : X86WriteResUnsupported<WriteVecMaskedGatherWriteback>;
 
 // Load/store MXCSR.
 defm : X86WriteRes<WriteSTMXCSR, [SLM_MEC_RSV], 12,[11], 4>;
 defm : X86WriteRes<WriteLDMXCSR, [SLM_MEC_RSV], 10, [8], 5>;
 
 // Treat misc copies as a move.
 def : InstRW<[WriteMove], (instrs COPY)>;
+def : InstRW<[WriteMove], (instrs PRED_COPY)>;
 
 defm : SLMWriteResPair<WriteALU,    [SLM_IEC_RSV01], 1>;
 defm : SLMWriteResPair<WriteADC,    [SLM_IEC_RSV01], 1>;
 
 defm : SLMWriteResPair<WriteIMul8,     [SLM_IEC_RSV1],  5, [5], 3>;
 defm : SLMWriteResPair<WriteIMul16,    [SLM_IEC_RSV1],  5, [5], 4, 1>;
 defm : SLMWriteResPair<WriteIMul16Imm, [SLM_IEC_RSV1],  4, [4], 2, 1>;
 defm : SLMWriteResPair<WriteIMul16Reg, [SLM_IEC_RSV1],  4, [4], 2, 1>;

llvm/lib/Target/X86/X86ScheduleZnver1.td

 defm : ZnWriteResPair<WriteBSF, [ZnALU], 3, [12], 6, 4, 2>;
 defm : ZnWriteResPair<WriteBSR, [ZnALU], 4, [16], 6, 4, 2>;
 defm : ZnWriteResPair<WriteLZCNT,          [ZnALU], 2>;
 defm : ZnWriteResPair<WriteTZCNT,          [ZnALU], 2, [2], 2, 4, 0>;
 defm : ZnWriteResPair<WritePOPCNT,         [ZnALU], 1>;
 
 // Treat misc copies as a move.
 def : InstRW<[WriteMove], (instrs COPY)>;
+def : InstRW<[WriteMove], (instrs PRED_COPY)>;
 
 // BMI1 BEXTR, BMI2 BZHI
 defm : ZnWriteResPair<WriteBEXTR, [ZnALU], 1, [1], 1, 4, 1>;
 defm : ZnWriteResPair<WriteBLS,   [ZnALU], 2, [2], 2, 4, 1>;
 defm : ZnWriteResPair<WriteBZHI,  [ZnALU], 1>;
 
 // IDIV
 defm : ZnWriteResPair<WriteDiv8,   [ZnALU2, ZnDivider], 15, [1,15], 1>;

llvm/lib/Target/X86/X86ScheduleZnver2.td

 defm : Zn2WriteResPair<WriteBSF, [Zn2ALU], 3, [12], 6, 4, 2>;
 defm : Zn2WriteResPair<WriteBSR, [Zn2ALU], 4, [16], 6, 4, 2>;
 defm : Zn2WriteResPair<WriteLZCNT,          [Zn2ALU], 1>;
 defm : Zn2WriteResPair<WriteTZCNT,          [Zn2ALU], 2, [2], 2, 4, 0>;
 defm : Zn2WriteResPair<WritePOPCNT,         [Zn2ALU], 1>;
 
 // Treat misc copies as a move.
 def : InstRW<[WriteMove], (instrs COPY)>;
+def : InstRW<[WriteMove], (instrs PRED_COPY)>;
 
 // BMI1 BEXTR, BMI2 BZHI
 defm : Zn2WriteResPair<WriteBEXTR, [Zn2ALU], 1, [1], 1, 4, 1>;
 defm : Zn2WriteResPair<WriteBLS,   [Zn2ALU], 2, [2], 2, 4, 1>;
 defm : Zn2WriteResPair<WriteBZHI,  [Zn2ALU], 1>;
 
 // IDIV
 defm : Zn2WriteResPair<WriteDiv8,   [Zn2ALU2, Zn2Divider], 15, [1,15], 1>;

llvm/lib/Target/X86/X86ScheduleZnver3.td

 def : InstRW<[Zn3WriteMOVSlow], (instrs MOV8rm, MOV8rm_NOREX, MOV16rm, MOVSX16rm16, MOVSX16rm32, MOVZX16rm16, MOVSX16rm8, MOVZX16rm8)>;
 
 defm : Zn3WriteResInt<WriteStore, [Zn3AGU012, Zn3Store], Znver3Model.StoreLatency, [1, 2], 1>;
 defm : Zn3WriteResInt<WriteStoreNT, [Zn3AGU012, Zn3Store], Znver3Model.StoreLatency, [1, 2], 1>;
 defm : Zn3WriteResInt<WriteMove, [Zn3ALU0123], 1, [4], 1>;
 
 // Treat misc copies as a move.
 def : InstRW<[WriteMove], (instrs COPY)>;
+def : InstRW<[WriteMove], (instrs PRED_COPY)>;
 
 def Zn3WriteMOVBE16rm : SchedWriteRes<[Zn3AGU012, Zn3Load, Zn3ALU0123]> {
   let Latency = Znver3Model.LoadLatency;
   let ResourceCycles = [1, 1, 4];
   let NumMicroOps = 1;
 }
 def : InstRW<[Zn3WriteMOVBE16rm], (instrs MOVBE16rm)>;
 

llvm/utils/TableGen/InstrInfoEmitter.cpp

void InstrInfoEmitter::run(raw_ostream &OS) {
   OS << "#ifdef GET_INSTRINFO_HEADER\n";
   OS << "#undef GET_INSTRINFO_HEADER\n";
 
   std::string ClassName = TargetName + "GenInstrInfo";
   OS << "namespace llvm {\n";
   OS << "struct " << ClassName << " : public TargetInstrInfo {\n"
      << "  explicit " << ClassName
      << "(unsigned CFSetupOpcode = ~0u, unsigned CFDestroyOpcode = ~0u, "
-        "unsigned CatchRetOpcode = ~0u, unsigned ReturnOpcode = ~0u);\n"
+        "unsigned CatchRetOpcode = ~0u, unsigned ReturnOpcode = ~0u, "
+        "unsigned CopyOpcode = TargetOpcode::COPY);\n"
      << "  ~" << ClassName << "() override = default;\n";
 
-
   OS << "\n};\n} // end namespace llvm\n";
 
   OS << "#endif // GET_INSTRINFO_HEADER\n\n";
 
   OS << "#ifdef GET_INSTRINFO_HELPER_DECLS\n";
   OS << "#undef GET_INSTRINFO_HELPER_DECLS\n\n";
   emitTIIHelperMethods(OS, TargetName, /* ExpandDefinition = */ false);
   OS << "\n";
void InstrInfoEmitter::run(raw_ostream &OS) {
   if (HasDeprecationFeatures)
     OS << "extern const uint8_t " << TargetName
        << "InstrDeprecationFeatures[];\n";
   if (HasComplexDeprecationInfos)
     OS << "extern const MCInstrInfo::ComplexDeprecationPredicate " << TargetName
        << "InstrComplexDeprecationInfos[];\n";
   OS << ClassName << "::" << ClassName
      << "(unsigned CFSetupOpcode, unsigned CFDestroyOpcode, unsigned "
-        "CatchRetOpcode, unsigned ReturnOpcode)\n"
+        "CatchRetOpcode, unsigned ReturnOpcode, unsigned CopyOpcode)\n"
      << "  : TargetInstrInfo(CFSetupOpcode, CFDestroyOpcode, CatchRetOpcode, "
-        "ReturnOpcode) {\n"
+        "ReturnOpcode, CopyOpcode) {\n"
      << "  InitMCInstrInfo(" << TargetName << "Insts, " << TargetName
      << "InstrNameIndices, " << TargetName << "InstrNameData, ";
   if (HasDeprecationFeatures)
     OS << TargetName << "InstrDeprecationFeatures, ";
   else
     OS << "nullptr, ";
   if (HasComplexDeprecationInfos)
     OS << TargetName << "InstrComplexDeprecationInfos, ";