[SystemZ] Utilize Compare/Add/Sub "High" instructions

[jonpa]

Summary

Use GRX32 Mux pseudo instructions for compare, add and sub.

Use regalloc hints and a new target hook to control copy propagation after (and during) regalloc to minimize the number of pseudos ending up with unsupported register combinations.

[jonpa]

The patch as of now is a work in progress (experimental).

Only one pseudo right now ends up being unsupported :-)

This seems generally beneficial:

Opcode counts differences. master <> patch
chlr           :                    0                 1874    +1874
ar             :                18873                19564     +691
chhr           :                    0                  617     +617
srk            :                 3742                 3252     -490
ahhlr          :                    0                  461     +461
ark            :                 4985                 4554     -431
clhlr          :                    0                  385     +385
shhlr          :                    0                  313     +313
sr             :                 6059                 6349     +290
cr             :                 4560                 4279     -281
clr            :                 1125                 1047      -78
shhhr          :                    0                   47      +47
ahhhr          :                    0                   34      +34
clhhr          :                    0                   31      +31
alr            :                    4                    3       -1
alhhlr         :                    0                    1       +1
Spill|Reload   :               189169               187154    -2015

Number of register moves instructions per benchark:
Reg moves         spec-llvm_A_master/              spec-llvm
447.dealII     :                67389                  66960     -429
454.calculix   :                19298                  19131     -167
444.namd       :                 6464                   6572     +108
435.gromacs    :                16386                  16283     -103
403.gcc        :                73292                  73204      -88
450.soplex     :                 7779                   7706      -73
483.xalancbmk  :               109978                 109932      -46
453.povray     :                15298                  15263      -35
400.perlbench  :                21702                  21731      +29
482.sphinx3    :                 3024                   3001      -23
445.gobmk      :                16052                  16030      -22
436.cactusADM  :                11225                  11235      +10
464.h264ref    :                 7998                   7988      -10
456.hmmer      :                 5741                   5734       -7
401.bzip2      :                 1161                   1155       -6
... (<= 5)
Reg moves      :               415172               414309       -863

[jonpa]

Still INCOMPLETE! SPEC builds, but this patch is still experimental.

The handling of hintRecoloring() / MachineCopyPropagation has been simplified. By using a new pass SystemZSelectMux immediately after virtregrewrite but before MCP. The selected legal instructions with High/Low register classes can not get bad registers from later passes. The Mux pseudos that did not get a legal allocation get handled as before by SystemZExpandPseudo, to let passes after regalloc, such as machine sinking optimize them.

Minispec performance measurements show some varying results.

Stats from SPEC builds:

Master (z13)
------------

Number of LOCRMux jump-sequences         :         1
Number of load-and-test instructions     :     65516
Number of Mux pseudo compares CC live out:         0

Number of spilled live ranges                 :     57661
Number of spills inserted                     :     53601
Number of copies between allocatable physregs :    412658


"Hard hints only to avoid illegal muxes" (z13)
----------------------------------------------

Number of LOCRs                          :      7649
Number of LOCRMux jump-sequences         :         9
Number of load-and-test instructions     :     65535
Number of Muxes using only low parts     :     70026
Number of Muxes using only high parts    :       704
Number of Muxes using high and low parts :      2828

Number of Muxes illegal registers        :         8
Number of Muxes neeeding copy to dst     :         3
Number of Muxes neeeding two rotates     :         0
Number of Mux pseudo compares swapped    :         5
Num Mux pseudo cmps not locally updatable:         0
Number of Mux pseudo compares CC live out:         0

Number of spilled live ranges                 :     56884
Number of spills inserted                     :     52783
Number of copies between allocatable physregs :    412981


Same, plus "Hard hints for All-high or All-low" (z13)
-----------------------------------------------------

Number of LOCRs                          :      7649
Number of LOCRMux jump-sequences         :         9
Number of load-and-test instructions     :     65536
Number of Muxes using only low parts     :     72184
Number of Muxes using only high parts    :      1352
Number of Muxes using high and low parts :        23

Number of Muxes illegal registers        :         7
Number of Muxes neeeding copy to dst     :         2
Number of Muxes neeeding two rotates     :         0
Number of Mux pseudo compares swapped    :         5
Num Mux pseudo cmps not locally updatable:         0
Number of Mux pseudo compares CC live out:         0

Number of spilled live ranges                 :     57248
Number of spills inserted                     :     53190
Number of copies between allocatable physregs :    413166


Same, but "Soft hints for All-high or All-low" (z13)
----------------------------------------------------

Number of LOCRs                          :      7649
Number of LOCRMux jump-sequences         :         9
Number of load-and-test instructions     :     65562
Number of Muxes using only low parts     :     71497
Number of Muxes using only high parts    :      1407
Number of Muxes using high and low parts :       654

Number of Muxes illegal registers        :         8
Number of Muxes neeeding copy to dst     :         3
Number of Muxes neeeding two rotates     :         0
Number of Mux pseudo compares swapped    :         5
Num Mux pseudo cmps not locally updatable:         0
Number of Mux pseudo compares CC live out:         0

Number of spilled live ranges                 :     57016
Number of spills inserted                     :     52924
Number of copies between allocatable physregs :    412950

[jonpa]

Patch is still experimental, but updated with the latest improvements.

• Fall backs implemented in SystemZExpandPseudo.cpp, using register copys, rotates, commutation, negation...
• Compare muxes not getting regalloc hints except when CC is live out, which improves things a bit. Extra CC liveness check added in SystemZExpandPseudo.

...

[jonpa]

Patch refined / improved.

[jonpa]

Regalloc hints for llc/llh added as well, which handled two regressions.

[qcolombet]

The generic changes look sensible to me.
I would just suggest to add a comment on what VRM is for on the modified methods.

lib/Target/SystemZ/SystemZRegisterInfo.cpp

304

Either make VRM a reference on the prototype, or check that it is not nullptr.
In theory a caller could pass nullptr.

[jonpa]

Thank you Quentin for the review! Patch updated per your suggestions. (I am not sure why LIS and VRM are optional parameters to foldMemoryOperandImpl(). It would have simplified my patch somewhat if they were not, but I instead added a check to see that VRM is available.)

Plus some minor fixes.

[uweigand]

Not a full review yet, but a couple of comments (in the test case) where the resulting code could be simplified.

test/CodeGen/SystemZ/expand-mux-pseudos.mir

61 ↗

(On Diff #193878)

This doesn't need to have a four-instruction sequence. The first two instructions simply duplicate the r2h value into both r2l and r2h; this can be done in a single risblg instruction.

137 ↗

(On Diff #193878)

Same as above.

148 ↗

(On Diff #193878)

This is a bit of a silly case since it implements X - X, so it doesn't matter what X was. This means the risblg is just dead code. (OTOH one would hope that the optimizers would have removed that already anyway ...)

156 ↗

(On Diff #193878)

Again a case of X - X, where the risbhg is dead.

[jonpa]

Don't build a new MI in selectAddSubMux(), but instead update the registers, MCInstrDesc and TiedTo flag. This avoids the need to update SlotIndexes. This is needed to not loose any extra ("regalloc") operands or instruction flags (such as "nsw") . Question: Are there any such flags that should be modified when commutation is performed on a sub and it becomes an add?

Some minor fixing:

• Don't pass TRI or TII to MuxInfo member functions (for readability).
• Remove confusing MuxInfo::PseudoOpcode member - might as well just set the MCInstrDesc temporarily in negateRHSIfSub().

[jonpa]

Patch refactored, with very near NFC.

• Merged SystemZSelectMux and SystemZExpandPseudo passes (the latter removed). This seems to mean that "MachineDominator Tree Construction" and "Machine Natural Loop Construction" is run one extra time (5 instead of 4 times). This is because they are not marked as preserved, since it is theoretically possible that LOCRMuxes get expanded and change CFG. Is it this ok, or is there a simple way to mark those analysises as preserved, and then only recomputing them if needed? (something like releaseMemory() + runOnMachineFunction() for both MLI and MDT, maybe?).

• MuxInfo simplified. Instead of Regs[], MachineOperands are updated directly, with hopefully correct flags.

Todo: Tests to verify that all cases are correctly handled, with correct register flags.

[jonpa]

• SystemZSelectMux.cpp merged into SystemZPostRewrite.cpp.

• Mux instructions now only 3-address, with simplified instruction classes. Instruction mappings for getTwoOperandOpcode() and getMemOpcode() added.

• foldMemoryOperandImpl() improved.

• Bugfix in MuxInfo::RotateReg0(): Only add an implicit use of the 32-bit register if it is *used* (not if it is just defined).

• getRegAllocationHints()
  • don't give 2-address hints for a GRH32 reg, since it would be useless. (This strangely seemed to increse spilling a bit at least on that day).
  • for GRX32 regs:
    • instead of iterative searches, revisit constrained regs immediately (push to front of worklist).
    • put a limit on the number of visited regs (excluding any revisits). This also influences the aggressiveness of the algorithm.
    • break out of loop over MIs if current Reg will be revisited.

On SPEC 2006, with unlimited hints search, I saw with -time-passes a very slight increase in compile time with the patch compared to master:
Average Wall Time for Greedy Register Allocator: 1.41% -> 1.42%
Average User Time for Greedy Register Allocator: 1.42% -> 1.45%

As well, it seems that the worst cases of relative compile time are not getting worse.

It seems that even with unlimited search, compile time is not an issue, so a high limit like 50 seems to be acceptable. See below for some stats with different values of "maxregs".

• Exhaustive testing in expand-mux-pseudos.mir: all combinations of low/high as well as same/different GR64 regs.

TODO:

• ThreeAddr: - Should all of the 2-address conversions be done in PostRewrite?
• Swap compare operands in foldMemoryOperandImpl()? Can we trust live-in lists for checking if CC is live-out?
• return false from allowHintRecoloring() for MemFoldPseudos to avoid the few cases of copying?
• SystemZISelLowering.cpp::reverseCCMask() merge with new code for swapping compare operands?
• SystemZPostRewrite::runOnMachineFunction() should return true even if only updateLiveInLists_CC() changed MF?
• Update CFG when expanding LOCRMUXr and give setPreservesAll()?

Before commit:

• remove some/most experimental stats.

Here is a summary of the effects of different values of -maxregs (experimental option in SystemZRegisterInfo.cpp).

• regalloc/systemz statistics as reported by the llvm option "-stats"
• "Spill|Reload" is the number of times the comment "Spill" or "Reload" is seen in .s files.
• "Copies" is the number of instructions built by copyPhysReg() seen in .s files.

This was done on SPEC 2006, but also previously on 2017, where there seems to be similar results, although it seemed like the value of maxregs=50 was
needed to not need any mux fallbacks at all (for 2006 it is maxregs=40, see below).

master
     57380                       regalloc - Number of spilled live ranges
         6                     systemz-II - Number of LOCRMux jump-sequences (lower is better)

master <> master
Spill|Reload   :               188230               188230       +0
Copies         :               423366               423366       +0

HiMuxes: maxregs=100
     56046                       regalloc - Number of spilled live ranges

master <> patch
Spill|Reload   :               188230               183270    -4960
Copies         :               423366               426449    +3083

HiMuxes: maxregs=50
     56046                       regalloc - Number of spilled live ranges

master <> patch
Spill|Reload   :               188230               183270    -4960
Copies         :               423366               426449    +3083

HiMuxes: maxregs=40
     56046                       regalloc - Number of spilled live ranges

master <> patch
Spill|Reload   :               188230               183270    -4960
Copies         :               423366               426449    +3083

HiMuxes: maxregs=30
     56030                       regalloc - Number of spilled live ranges
         9            systemz-postrewrite - Number of Mux pseudos needing a copy to dst.
         9            systemz-postrewrite - Number of Mux pseudos needing two rotates of a reg.

master <> patch
Spill|Reload   :               188230               183238    -4992
Copies         :               423366               426457    +3091

HiMuxes: maxregs=20
     56029                       regalloc - Number of spilled live ranges
         8            systemz-postrewrite - Number of Mux pseudos needing a copy to dst.
        11            systemz-postrewrite - Number of Mux pseudos needing two rotates of a reg.

master <> patch
Spill|Reload   :               188230               183236    -4994
Copies         :               423366               426456    +3090

HiMuxes: maxregs=15
     56019                       regalloc - Number of spilled live ranges
         9            systemz-postrewrite - Number of Mux pseudos needing a copy to dst.
        14            systemz-postrewrite - Number of Mux pseudos needing two rotates of a reg.

master <> patch
Spill|Reload   :               188230               183224    -5006
Copies         :               423366               426458    +3092

HiMuxes: maxregs=10
     55989                       regalloc - Number of spilled live ranges
        22            systemz-postrewrite - Number of Mux pseudos needing a copy to dst.
        43            systemz-postrewrite - Number of Mux pseudos needing two rotates of a reg.
         1            systemz-postrewrite - Number of Mux pseudos negation of RHS (subtractions).

master <> patch
Spill|Reload   :               188230               183159    -5071
Copies         :               423366               426473    +3107

HiMuxes: maxregs=5
     55988                       regalloc - Number of spilled live ranges
        27            systemz-postrewrite - Number of LOCRMux jump-sequences (lower is better)
        29            systemz-postrewrite - Number of Mux pseudos needing a copy to dst.
        63            systemz-postrewrite - Number of Mux pseudos needing two rotates of a reg.
         1            systemz-postrewrite - Number of Mux pseudos negation of RHS (subtractions).

master <> patch
Spill|Reload   :               188230               183277    -4953
Copies         :               423366               426557    +3191

Without any GRX32 hints at all:

     53931                       regalloc - Number of spilled live ranges
      1483            systemz-postrewrite - Number of LOCRMux jump-sequences (lower is better)
        49            systemz-postrewrite - Number of Mux pseudo compares with live out CC.
      3283            systemz-postrewrite - Number of Mux pseudos needing a copy to dst.
      3969            systemz-postrewrite - Number of Mux pseudos needing two rotates of a reg.
       117            systemz-postrewrite - Number of Mux pseudos negation of RHS (subtractions).

master <> patch
Spill|Reload   :               188230               180090    -8140
Copies         :               423366               431998    +8632

I also tried to use soft hints instead of hard hints to avoid fallbacks once, and it seemed to give a result somewhere in between "no GRX32 hints at all" and the patch.

[jonpa]

test/CodeGen/SystemZ/expand-mux-pseudos.mir

61 ↗

(On Diff #193878)

OK, I added a special handling for that case in expandAddSubMux().

148 ↗

(On Diff #193878)

Since this is just a MIR test anyway I would like to test all possible combinations even if they do not make much sense.

[jonpa]

Don't skip all 2-address hints in getRegallocationHints() just because it's a GRH32 register: AHIMuxK actually can expand to a 2-address instruction using a high register.

[jonpa]

stats.txt3 KBDownload

Here's an updated list of stats for SPEC 2006, with the latest fixes applied, and also with "soft hints" at the bottom.

[jonpa]

• Don't use VRM for VirtReg in analyzeOperands(). In case of an eviction (where VirtReg already has a physreg in VRM), this previous regclass should not be reused initially.

• Rename new Mux:es to ...MuxK.

include/llvm/CodeGen/TargetRegisterInfo.h

public:
   virtual bool getRegAllocationHints(unsigned VirtReg,
                                      ArrayRef<MCPhysReg> Order,
                                      SmallVectorImpl<MCPhysReg> &Hints,
                                      const MachineFunction &MF,
                                      const VirtRegMap *VRM = nullptr,
                                      const LiveRegMatrix *Matrix = nullptr)
     const;
 
+  /// Let target disallow a replacement of a physical register. This would
+  /// typically be useful during copy propagation for the cases where
+  /// getRegAllocationHints() returned hard hints. This could either be
+  /// called with for a virtual register (with non-null VReg and VRM
+  /// arguments), or for a physical register at OpIdx to be replaced.
+  virtual bool isRegSuitableReplacement(const MachineInstr *MI,
+                                        unsigned VReg,
+                                        unsigned NewPhysReg,
+                                        const VirtRegMap *VRM = nullptr,
+                                        unsigned OpIdx = UINT_MAX) const {
+    assert(((OpIdx == UINT_MAX) != (VReg == 0)) &&
+           ((VReg == 0) == (VRM == nullptr)) && "Bad arguments.");
+    return true;
+  }
+
   /// A callback to allow target a chance to update register allocation hints
   /// when a register is "changed" (e.g. coalesced) to another register.
   /// e.g. On ARM, some virtual registers should target register pairs,
   /// if one of pair is coalesced to another register, the allocation hint of
   /// the other half of the pair should be changed to point to the new register.
   virtual void updateRegAllocHint(unsigned Reg, unsigned NewReg,
                                   MachineFunction &MF) const {
     // Do nothing.

lib/CodeGen/MachineCopyPropagation.cpp

 /// \param UseI based on the physical register class constraints of the opcode
 /// and avoiding introducing more cross-class COPYs.
 bool MachineCopyPropagation::isForwardableRegClassCopy(const MachineInstr &Copy,
                                                        const MachineInstr &UseI,
                                                        unsigned UseIdx) {
 
   unsigned CopySrcReg = Copy.getOperand(1).getReg();
 
+  // Let target disallow a replacement of a physical register
+  if (!TRI->isRegSuitableReplacement(&UseI, 0/*NoVReg*/, CopySrcReg,
+                                     nullptr/*VRM*/, UseIdx))
+    return false;
+
   // If the new register meets the opcode register constraints, then allow
   // forwarding.
   if (const TargetRegisterClass *URC =
           UseI.getRegClassConstraint(UseIdx, TII, TRI))
     return URC->contains(CopySrcReg);
 
   if (!UseI.isCopy())
     return false;

lib/CodeGen/RegAllocGreedy.cpp

if (CurrPhys != PhysReg) {
       BlockFrequency NewCopiesCost = getBrokenHintFreq(Info, PhysReg);
       LLVM_DEBUG(dbgs() << "Old Cost: " << OldCopiesCost.getFrequency()
                         << "\nNew Cost: " << NewCopiesCost.getFrequency()
                         << '\n');
       if (OldCopiesCost < NewCopiesCost) {
         LLVM_DEBUG(dbgs() << "=> Not profitable.\n");
         continue;
       }
+
+      // Check with target for any special replacements to avoid.
+      bool SuitableReplacement = true;
+      for (const MachineInstr &Instr : MRI->reg_nodbg_instructions(Reg))
+        if (!TRI->isRegSuitableReplacement(&Instr, Reg, PhysReg, VRM)) {
+            LLVM_DEBUG(dbgs() << "=> Not suitable.\n");
+            SuitableReplacement = false;
+            break;
+          }
+      if (!SuitableReplacement)
+        continue;
+
       // At this point, the cost is either cheaper or equal. If it is
       // equal, we consider this is profitable because it may expose
       // more recoloring opportunities.
       LLVM_DEBUG(dbgs() << "=> Profitable.\n");
       // Recolor the live-range.
       Matrix->unassign(LI);
       Matrix->assign(LI, PhysReg);
     }

lib/Target/SystemZ/SystemZInstrFormats.td

 class UnaryRRPseudo<string key, SDPatternOperator operator,
                     RegisterOperand cls1, RegisterOperand cls2>
   : Pseudo<(outs cls1:$R1), (ins cls2:$R2),
            [(set cls1:$R1, (operator cls2:$R2))]> {
   let OpKey = key#cls1;
   let OpType = "reg";
 }
 
+// Like BinaryRRAndK, but expanded after RA depending on the choice of register.
+multiclass BinaryRRAndKPseudo<string key, SDPatternOperator operator,
+            RegisterOperand cls1, RegisterOperand cls2, RegisterOperand cls3> {
+  let NumOpsKey = key, Predicates = [FeatureHighWord] in {
+    let NumOpsValue = "3" in
+      def K : Pseudo<(outs cls1:$R1), (ins cls2:$R2, cls3:$R3), []>;
+    let NumOpsValue = "2", isConvertibleToThreeAddress = 1,
+        Constraints = "$R1 = $R1src" in
+      def "" : Pseudo<(outs cls1:$R1), (ins cls2:$R1src, cls3:$R3),
+               [(set cls1:$R1, (operator cls2:$R1src, cls3:$R3))]>;
+  }
+}
+
 // Like BinaryRI, but expanded after RA depending on the choice of register.
 class BinaryRIPseudo<SDPatternOperator operator, RegisterOperand cls,
                      Immediate imm>
   : Pseudo<(outs cls:$R1), (ins cls:$R1src, imm:$I2),
            [(set cls:$R1, (operator cls:$R1src, imm:$I2))]> {
   let Constraints = "$R1 = $R1src";
 }
 
class CompareRXYPseudo<SDPatternOperator operator, RegisterOperand cls,
   : Pseudo<(outs), (ins cls:$R1, mode:$XBD2),
            [(set CC, (operator cls:$R1, (load mode:$XBD2)))]> {
   let mayLoad = 1;
   let Has20BitOffset = 1;
   let HasIndex = 1;
   let AccessBytes = bytes;
 }
 
+// Like CompareRRE, but expanded after RA depending on the choice of
+// register.
+class CompareRREPseudo<SDPatternOperator operator, RegisterOperand cls1,
+                       RegisterOperand cls2>
+  : Pseudo<(outs), (ins cls1:$R1, cls2:$R2),
+           [(set CC, (operator cls1:$R1, cls2:$R2))]> {
+  let isCompare = 1;
+}
+
 // Like TestBinarySIL, but expanded later.
 class TestBinarySILPseudo<SDPatternOperator operator, Immediate imm>
   : Pseudo<(outs), (ins bdaddr12only:$BD1, imm:$I2),
            [(set CC, (operator bdaddr12only:$BD1, imm:$I2))]>;
 
 // Like CondBinaryRRF, but expanded after RA depending on the choice of
 // register.
 class CondBinaryRRFPseudo<RegisterOperand cls1, RegisterOperand cls2>

lib/Target/SystemZ/SystemZInstrInfo.h

class SystemZInstrInfo : public SystemZGenInstrInfo {
   void splitMove(MachineBasicBlock::iterator MI, unsigned NewOpcode) const;
   void splitAdjDynAlloc(MachineBasicBlock::iterator MI) const;
   void expandRIPseudo(MachineInstr &MI, unsigned LowOpcode, unsigned HighOpcode,
                       bool ConvertHigh) const;
   void expandRIEPseudo(MachineInstr &MI, unsigned LowOpcode,
                        unsigned LowOpcodeK, unsigned HighOpcode) const;
   void expandRXYPseudo(MachineInstr &MI, unsigned LowOpcode,
                        unsigned HighOpcode) const;
+  void expandRRAndKPseudo(MachineInstr &MI, unsigned LowOpcode,
+                          unsigned HiOpcode, unsigned HiHiLoOpcode) const;
   void expandLOCPseudo(MachineInstr &MI, unsigned LowOpcode,
                        unsigned HighOpcode) const;
   void expandLOCRPseudo(MachineInstr &MI, unsigned LowOpcode,
                         unsigned HighOpcode) const;
+  void expandCmpRMux(MachineInstr &MI, unsigned LowLowOpcode,
+                     unsigned HiHiOpcode, unsigned HiLowOpcode) const;
   void expandZExtPseudo(MachineInstr &MI, unsigned LowOpcode,
                         unsigned Size) const;
   void expandLoadStackGuard(MachineInstr *MI) const;
 
   MachineInstrBuilder
   emitGRX32Move(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                 const DebugLoc &DL, unsigned DestReg, unsigned SrcReg,
                 unsigned LowLowOpcode, unsigned Size, bool KillSrc,

lib/Target/SystemZ/SystemZInstrInfo.cpp

 void SystemZInstrInfo::expandRXYPseudo(MachineInstr &MI, unsigned LowOpcode,
                                        unsigned HighOpcode) const {
   unsigned Reg = MI.getOperand(0).getReg();
   unsigned Opcode = getOpcodeForOffset(isHighReg(Reg) ? HighOpcode : LowOpcode,
                                        MI.getOperand(2).getImm());
   MI.setDesc(get(Opcode));
 }
 
+// TODO: comment
+void SystemZInstrInfo::expandRRAndKPseudo(MachineInstr &MI, unsigned LowOpcode,
+                                          unsigned HiOpcode,
+                                          unsigned HiHiLoOpcode) const {
+  unsigned DstReg = MI.getOperand(0).getReg();
+  unsigned LHSReg = MI.getOperand(1).getReg();
+  unsigned RHSReg = MI.getOperand(2).getReg();
+  bool LHSisKill = MI.getOperand(1).isKill();
+  bool RHSisKill = MI.getOperand(2).isKill();
+  bool DstIsHigh = isHighReg(DstReg);
+  bool LHSIsHigh = isHighReg(LHSReg);
+  bool RHSIsHigh = isHighReg(RHSReg);
+  bool IsAdd = MI.isCommutable();
+  bool Commute = false;
+  unsigned Opcode;
+  if (!DstIsHigh && !LHSIsHigh && !RHSIsHigh)
+    Opcode = (DstReg == LHSReg ?
+              LowOpcode : SystemZ::getThreeOperandOpcode(LowOpcode));
+  else if (DstIsHigh && LHSIsHigh && RHSIsHigh)
+    Opcode = HiOpcode;
+  else if (DstIsHigh && (LHSIsHigh || RHSIsHigh)) {
+    Opcode = HiHiLoOpcode;
+    if (!LHSIsHigh && RHSIsHigh) { // HLH is unsupported.
+      if (IsAdd)
+        Commute = true;
+      else
+        Opcode = SystemZ::SUBCOMMUX;  // EXPERIMENTAL
+    }
+  }
+  else
+    Opcode = SystemZ::ADDSUBMUX;  // EXPERIMENTAL
+
+  MachineBasicBlock *MBB = MI.getParent();
+  if (!Commute)
+    BuildMI(*MBB, MI, MI.getDebugLoc(), get(Opcode), DstReg)
+      .addReg(LHSReg, getKillRegState(LHSisKill))
+      .addReg(RHSReg, getKillRegState(RHSisKill));
+  else
+    BuildMI(*MBB, MI, MI.getDebugLoc(), get(Opcode), DstReg)
+      .addReg(RHSReg, getKillRegState(RHSisKill))
+      .addReg(LHSReg, getKillRegState(LHSisKill));
+  MI.eraseFromParent();
+}
+
 // MI is a load-on-condition pseudo instruction with a single register
 // (source or destination) operand.  Replace it with LowOpcode if the
 // register is a low GR32 and HighOpcode if the register is a high GR32.
 void SystemZInstrInfo::expandLOCPseudo(MachineInstr &MI, unsigned LowOpcode,
                                        unsigned HighOpcode) const {
   unsigned Reg = MI.getOperand(0).getReg();
   unsigned Opcode = isHighReg(Reg) ? HighOpcode : LowOpcode;
   MI.setDesc(get(Opcode));
else
     LOCRMuxJumps++;
 
   // If we were unable to implement the pseudo with a single instruction, we
   // need to convert it back into a branch sequence.  This cannot be done here
   // since the caller of expandPostRAPseudo does not handle changes to the CFG
   // correctly.  This change is defered to the SystemZExpandPseudo pass.
 }
 
+// MI is a 32 bit compare with register pseudo instruction. Replace it with
+// one of the three possible opcodes. "LowHi" is not supported and must
+// therefore be inverted.
+void SystemZInstrInfo::expandCmpRMux(MachineInstr &MI, unsigned LowLowOpcode,
+                                     unsigned HiHiOpcode,
+                                     unsigned HiLowOpcode) const {
+  unsigned Src0Reg = MI.getOperand(0).getReg();
+  unsigned Src1Reg = MI.getOperand(1).getReg();
+  bool Src0IsHigh = isHighReg(Src0Reg);
+  bool Src1IsHigh = isHighReg(Src1Reg);
+
+  if (!Src0IsHigh && !Src1IsHigh)
+    MI.setDesc(get(LowLowOpcode));
+  else if (Src0IsHigh && Src1IsHigh)
+    MI.setDesc(get(HiHiOpcode));
+  else if (Src0IsHigh && !Src1IsHigh)
+    MI.setDesc(get(HiLowOpcode));
+  else  // LowHi
+    MI.setDesc(get(SystemZ::CMPMUX)); // EXPERIMENTAL
+}
+
 // MI is an RR-style pseudo instruction that zero-extends the low Size bits
 // of one GRX32 into another.  Replace it with LowOpcode if both operands
 // are low registers, otherwise use RISB[LH]G.
 void SystemZInstrInfo::expandZExtPseudo(MachineInstr &MI, unsigned LowOpcode,
                                         unsigned Size) const {
   MachineInstrBuilder MIB =
     emitGRX32Move(*MI.getParent(), MI, MI.getDebugLoc(),
                MI.getOperand(0).getReg(), MI.getOperand(1).getReg(), LowOpcode,
bool SystemZInstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
   case SystemZ::TMLMux:
     expandRIPseudo(MI, SystemZ::TMLL, SystemZ::TMHL, false);
     return true;
 
   case SystemZ::TMHMux:
     expandRIPseudo(MI, SystemZ::TMLH, SystemZ::TMHH, false);
     return true;
 
+  case SystemZ::ARMux:
+  case SystemZ::ARMuxK:
+    expandRRAndKPseudo(MI, SystemZ::AR, SystemZ::AHHHR, SystemZ::AHHLR);
+    return true;
+
+  case SystemZ::SRMux:
+  case SystemZ::SRMuxK:
+    expandRRAndKPseudo(MI, SystemZ::SR, SystemZ::SHHHR, SystemZ::SHHLR);
+    return true;
+
+  case SystemZ::ALRMux:
+  case SystemZ::ALRMuxK:
+    expandRRAndKPseudo(MI, SystemZ::ALR, SystemZ::ALHHHR, SystemZ::ALHHLR);
+    return true;
+
+  case SystemZ::SLRMux:
+  case SystemZ::SLRMuxK:
+    expandRRAndKPseudo(MI, SystemZ::SLR, SystemZ::SLHHHR, SystemZ::SLHHLR);
+    return true;
+
   case SystemZ::AHIMux:
     expandRIPseudo(MI, SystemZ::AHI, SystemZ::AIH, false);
     return true;
 
   case SystemZ::AHIMuxK:
     expandRIEPseudo(MI, SystemZ::AHI, SystemZ::AHIK, SystemZ::AIH);
     return true;
 
bool SystemZInstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
   case SystemZ::CMux:
     expandRXYPseudo(MI, SystemZ::C, SystemZ::CHF);
     return true;
 
   case SystemZ::CLMux:
     expandRXYPseudo(MI, SystemZ::CL, SystemZ::CLHF);
     return true;
 
+  case SystemZ::CRMux:
+    expandCmpRMux(MI, SystemZ::CR, SystemZ::CHHR, SystemZ::CHLR);
+    return true;
+
+  case SystemZ::CLRMux:
+    expandCmpRMux(MI, SystemZ::CLR, SystemZ::CLHHR, SystemZ::CLHLR);
+    return true;
+
   case SystemZ::RISBMux: {
     bool DestIsHigh = isHighReg(MI.getOperand(0).getReg());
     bool SrcIsHigh = isHighReg(MI.getOperand(2).getReg());
     if (SrcIsHigh == DestIsHigh)
       MI.setDesc(get(DestIsHigh ? SystemZ::RISBHH : SystemZ::RISBLL));
     else {
       MI.setDesc(get(DestIsHigh ? SystemZ::RISBHL : SystemZ::RISBLH));
       MI.getOperand(5).setImm(MI.getOperand(5).getImm() ^ 32);

lib/Target/SystemZ/SystemZInstrInfo.td

 //===----------------------------------------------------------------------===//
 // Addition
 //===----------------------------------------------------------------------===//
 
 // Addition producing a signed overflow flag.
 let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8 in {
   // Addition of a register.
   let isCommutable = 1 in {
+    defm ARMux : BinaryRRAndKPseudo<"armux", z_sadd, GRX32, GRX32, GRX32>;
     defm AR : BinaryRRAndK<"ar", 0x1A, 0xB9F8, z_sadd, GR32, GR32>;
     defm AGR : BinaryRREAndK<"agr", 0xB908, 0xB9E8, z_sadd, GR64, GR64>;
   }
   def AGFR : BinaryRRE<"agfr", 0xB918, null_frag, GR64, GR32>;
 
+  // EXPERIMENTAL!  -- just to be able to compile and see the occurences.
+  let hasNoSchedulingInfo = 1, isCodeGenOnly = 1 in {
+    def ADDSUBMUX : BinaryRRFa<"# addsubmux", 0x0, null_frag, GRX32, GRX32, GRX32>;
+    def SUBCOMMUX : BinaryRRFa<"# commsub", 0x0, null_frag, GRX32, GRX32, GRX32>;
+    def CMPMUX : CompareRRE<"# cmpmux", 0x0, null_frag, GRX32, GRX32>;
+  }
+
   // Addition to a high register.
   def AHHHR : BinaryRRFa<"ahhhr", 0xB9C8, null_frag, GRH32, GRH32, GRH32>,
               Requires<[FeatureHighWord]>;
   def AHHLR : BinaryRRFa<"ahhlr", 0xB9D8, null_frag, GRH32, GRH32, GR32>,
               Requires<[FeatureHighWord]>;
 
   // Addition of signed 16-bit immediates.
   defm AHIMux : BinaryRIAndKPseudo<"ahimux", z_sadd, GRX32, imm32sx16>;
let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8 in {
   def AGSI : BinarySIY<"agsi", 0xEB7A, add, imm64sx8>;
 }
 defm : SXB<z_sadd, GR64, AGFR>;
 
 // Addition producing a carry.
 let Defs = [CC] in {
   // Addition of a register.
   let isCommutable = 1 in {
+    defm ALRMux : BinaryRRAndKPseudo<"alrmux", z_uadd, GRX32, GRX32, GRX32>;
     defm ALR : BinaryRRAndK<"alr", 0x1E, 0xB9FA, z_uadd, GR32, GR32>;
     defm ALGR : BinaryRREAndK<"algr", 0xB90A, 0xB9EA, z_uadd, GR64, GR64>;
   }
   def ALGFR : BinaryRRE<"algfr", 0xB91A, null_frag, GR64, GR32>;
 
   // Addition to a high register.
   def ALHHHR : BinaryRRFa<"alhhhr", 0xB9CA, null_frag, GRH32, GRH32, GRH32>,
                Requires<[FeatureHighWord]>;

 
 //===----------------------------------------------------------------------===//
 // Subtraction
 //===----------------------------------------------------------------------===//
 
 // Subtraction producing a signed overflow flag.
 let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8 in {
   // Subtraction of a register.
+  defm SRMux : BinaryRRAndKPseudo<"srmux", z_ssub, GRX32, GRX32, GRX32>;
   defm SR : BinaryRRAndK<"sr", 0x1B, 0xB9F9, z_ssub, GR32, GR32>;
   def SGFR : BinaryRRE<"sgfr", 0xB919, null_frag, GR64, GR32>;
   defm SGR : BinaryRREAndK<"sgr", 0xB909, 0xB9E9, z_ssub, GR64, GR64>;
 
   // Subtraction from a high register.
   def SHHHR : BinaryRRFa<"shhhr", 0xB9C9, null_frag, GRH32, GRH32, GRH32>,
               Requires<[FeatureHighWord]>;
   def SHHLR : BinaryRRFa<"shhlr", 0xB9D9, null_frag, GRH32, GRH32, GR32>,
def : Pat<(z_ssub GR64:$src1, imm64sx16n:$src2),
             (AGHI GR64:$src1, imm64sx16n:$src2)>;
   def : Pat<(z_ssub GR64:$src1, imm64sx32n:$src2),
             (AGFI GR64:$src1, imm64sx32n:$src2)>;
 }
 
 // Subtraction producing a carry.
 let Defs = [CC] in {
   // Subtraction of a register.
+  defm SLRMux : BinaryRRAndKPseudo<"slrmux", z_usub, GRX32, GRX32, GRX32>;
   defm SLR : BinaryRRAndK<"slr", 0x1F, 0xB9FB, z_usub, GR32, GR32>;
   def SLGFR : BinaryRRE<"slgfr", 0xB91B, null_frag, GR64, GR32>;
   defm SLGR : BinaryRREAndK<"slgr", 0xB90B, 0xB9EB, z_usub, GR64, GR64>;
 
   // Subtraction from a high register.
   def SLHHHR : BinaryRRFa<"slhhhr", 0xB9CB, null_frag, GRH32, GRH32, GRH32>,
                Requires<[FeatureHighWord]>;
   def SLHHLR : BinaryRRFa<"slhhlr", 0xB9DB, null_frag, GRH32, GRH32, GR32>,

 //===----------------------------------------------------------------------===//
 // Comparison
 //===----------------------------------------------------------------------===//
 
 // Signed comparisons.  We put these before the unsigned comparisons because
 // some of the signed forms have COMPARE AND BRANCH equivalents whereas none
 // of the unsigned forms do.
 let Defs = [CC], CCValues = 0xE in {
-  // Comparison with a register.
+  // Comparison with a register.  CRMux expands to CR, CHHR or CHLR,
+  // depending on the choice of register.
+  def CRMux : CompareRREPseudo<z_scmp, GRX32, GRX32>,
+              Requires<[FeatureHighWord]>;
   def CR   : CompareRR <"cr",   0x19,   z_scmp,    GR32, GR32>;
   def CGFR : CompareRRE<"cgfr", 0xB930, null_frag, GR64, GR32>;
   def CGR  : CompareRRE<"cgr",  0xB920, z_scmp,    GR64, GR64>;
 
   // Comparison with a high register.
   def CHHR : CompareRRE<"chhr", 0xB9CD, null_frag, GRH32, GRH32>,
              Requires<[FeatureHighWord]>;
   def CHLR : CompareRRE<"chlr", 0xB9DD, null_frag, GRH32, GR32>,
let Defs = [CC], CCValues = 0xE in {
   def CHHSI : CompareSIL<"chhsi", 0xE554, z_scmp, asextloadi16, imm32sx16>;
   def CHSI  : CompareSIL<"chsi",  0xE55C, z_scmp, load, imm32sx16>;
   def CGHSI : CompareSIL<"cghsi", 0xE558, z_scmp, load, imm64sx16>;
 }
 defm : SXB<z_scmp, GR64, CGFR>;
 
 // Unsigned comparisons.
 let Defs = [CC], CCValues = 0xE, IsLogical = 1 in {
-  // Comparison with a register.
+  // Comparison with a register.  CLRMux expands to CLR, CLHHR or CLHLR,
+  // depending on the choice of register.
+  def CLRMux : CompareRREPseudo<z_ucmp, GRX32, GRX32>,
+               Requires<[FeatureHighWord]>;
   def CLR   : CompareRR <"clr",   0x15,   z_ucmp,    GR32, GR32>;
   def CLGFR : CompareRRE<"clgfr", 0xB931, null_frag, GR64, GR32>;
   def CLGR  : CompareRRE<"clgr",  0xB921, z_ucmp,    GR64, GR64>;
 
   // Comparison with a high register.
   def CLHHR : CompareRRE<"clhhr", 0xB9CF, null_frag, GRH32, GRH32>,
               Requires<[FeatureHighWord]>;
   def CLHLR : CompareRRE<"clhlr", 0xB9DF, null_frag, GRH32, GR32>,

lib/Target/SystemZ/SystemZRegisterInfo.h

public:
 
   bool getRegAllocationHints(unsigned VirtReg,
                              ArrayRef<MCPhysReg> Order,
                              SmallVectorImpl<MCPhysReg> &Hints,
                              const MachineFunction &MF,
                              const VirtRegMap *VRM,
                              const LiveRegMatrix *Matrix) const override;
 
+  bool isRegSuitableReplacement(const MachineInstr *MI,
+                                unsigned VReg,
+                                unsigned NewPhysReg,
+                                const VirtRegMap *VRM = nullptr,
+                                unsigned OpIdx = UINT_MAX) const override;
+
   // Override TargetRegisterInfo.h.
   bool requiresRegisterScavenging(const MachineFunction &MF) const override {
     return true;
   }
   bool requiresFrameIndexScavenging(const MachineFunction &MF) const override {
     return true;
   }
   bool trackLivenessAfterRegAlloc(const MachineFunction &MF) const override {

lib/Target/SystemZ/SystemZRegisterInfo.cpp

 #include "llvm/CodeGen/TargetFrameLowering.h"
 #include "llvm/CodeGen/VirtRegMap.h"
 
 using namespace llvm;
 
 #define GET_REGINFO_TARGET_DESC
 #include "SystemZGenRegisterInfo.inc"
 
+#define DEBUG_TYPE "regalloc"
+
 SystemZRegisterInfo::SystemZRegisterInfo()
     : SystemZGenRegisterInfo(SystemZ::R14D) {}
 
+static const TargetRegisterClass *getRCFromPhysReg(unsigned PhysReg) {
+  if (SystemZ::GR32BitRegClass.contains(PhysReg))
+    return &SystemZ::GR32BitRegClass;
+  assert (SystemZ::GRH32BitRegClass.contains(PhysReg) &&
+          "Phys reg not in GR32 or GRH32?");
+  return &SystemZ::GRH32BitRegClass;
+}
+
 // Given that MO is a GRX32 operand, return either GR32 or GRH32 if MO
 // somehow belongs in it. Otherwise, return GRX32.
 static const TargetRegisterClass *getRC32(MachineOperand &MO,
                                           const VirtRegMap *VRM,
                                           const MachineRegisterInfo *MRI) {
   const TargetRegisterClass *RC = MRI->getRegClass(MO.getReg());
 
   if (SystemZ::GR32BitRegClass.hasSubClassEq(RC) ||
       MO.getSubReg() == SystemZ::subreg_l32 ||
       MO.getSubReg() == SystemZ::subreg_hl32)
     return &SystemZ::GR32BitRegClass;
   if (SystemZ::GRH32BitRegClass.hasSubClassEq(RC) ||
       MO.getSubReg() == SystemZ::subreg_h32 ||
       MO.getSubReg() == SystemZ::subreg_hh32)
     return &SystemZ::GRH32BitRegClass;
 
-  if (VRM && VRM->hasPhys(MO.getReg())) {
-    unsigned PhysReg = VRM->getPhys(MO.getReg());
-    if (SystemZ::GR32BitRegClass.contains(PhysReg))
-      return &SystemZ::GR32BitRegClass;
-    assert (SystemZ::GRH32BitRegClass.contains(PhysReg) &&
-            "Phys reg not in GR32 or GRH32?");
-    return &SystemZ::GRH32BitRegClass;
-  }
+  if (VRM && VRM->hasPhys(MO.getReg()))
+    return getRCFromPhysReg(VRM->getPhys(MO.getReg()));
 
   assert (RC == &SystemZ::GRX32BitRegClass);
   return RC;
 }
 
 // Pass the registers of RC as hints while making sure that if any of these
 // registers are copy hints (and therefore already in Hints), hint them
 // first.
SystemZRegisterInfo::getRegAllocationHints(unsigned VirtReg,
                                            const LiveRegMatrix *Matrix) const {
   const MachineRegisterInfo *MRI = &MF.getRegInfo();
   const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
 
   bool BaseImplRetVal = TargetRegisterInfo::getRegAllocationHints(
       VirtReg, Order, Hints, MF, VRM, Matrix);
 
   if (MRI->getRegClass(VirtReg) == &SystemZ::GRX32BitRegClass) {
+    // Compare/Add/Sub "High" Muxes. Not all combinations are supported by
+    // the target.
+    struct Reg2RCMap : std::map<MCPhysReg, const TargetRegisterClass *> {
+      unsigned VirtReg_;   // XXX
+      bool Change;
+      unsigned RequiredHigh;  // USEFUL?
+      unsigned RequiredLow;
+      void reset(unsigned Reg) {
+        VirtReg_ = Reg;
+        Change = false;
+        RequiredHigh = 0;
+        RequiredLow = 0;
+      }
+      void findRC(MachineOperand &MO,
+                 const VirtRegMap *VRM,
+                 const MachineRegisterInfo *MRI) {
+        if (find(MO.getReg()) != end())
+          return;
+        const TargetRegisterClass *RC = getRC32(MO, VRM, MRI);
+        (*this)[MO.getReg()] = RC;
+      }
+      void constrainReg(unsigned Reg, const TargetRegisterClass *RC) {
+        if ((*this)[Reg] != &SystemZ::GRX32BitRegClass)
+          return;
+        if (Reg == VirtReg_) {
+          if (RC == &SystemZ::GR32BitRegClass)
+            RequiredLow++;
+          else
+            RequiredHigh++;
+        } else {
+          (*this)[Reg] = RC;
+          Change = true;
+        }
+      }
+    } Reg2RC;
+    // Search all use/def connected instructions iteratively to propagate the
+    // requirements of registers being in either GR32 or GR32H. At the end,
+    // pass hints for VirtReg if either GR32 or GR32H seems preferred.
+    // XXX: Make sure not to revisit same instruction multiple times?
     SmallVector<unsigned, 8> Worklist;
     SmallSet<unsigned, 4> DoneRegs;
+    Reg2RC.Change = true;
+    while (Reg2RC.Change) {
+      Reg2RC.reset(VirtReg);
+      Worklist.clear();
       Worklist.push_back(VirtReg);
+      DoneRegs.clear();
       while (Worklist.size()) {
         unsigned Reg = Worklist.pop_back_val();
         if (!DoneRegs.insert(Reg).second)
           continue;
+        for (MachineRegisterInfo::reg_instr_nodbg_iterator
+               RI = MRI->reg_instr_nodbg_begin(Reg),
+               E = MRI->reg_instr_nodbg_end(); RI != E; ) {
+          MachineInstr &MI = *RI++;
+          unsigned Regs[3];
+          unsigned OpI = 0;
+          auto processOperands = [&]() -> void {
+            for (; OpI < MI.getNumOperands(); ++OpI) {
+              MachineOperand &MO = MI.getOperand(OpI);
+              if (MO.isImplicit())
+                break;
+              Regs[OpI] = MO.getReg();
+              Reg2RC.findRC(MO, VRM, MRI);
+              // Add GRX32 registers to worklist
+              if (Reg2RC[MO.getReg()] == &SystemZ::GRX32BitRegClass)
+                Worklist.push_back(MO.getReg());
+            }
+          };
+
+          const TargetRegisterClass *LOW = &SystemZ::GR32BitRegClass;
+          const TargetRegisterClass *HIGH = &SystemZ::GRH32BitRegClass;
+          switch (MI.getOpcode()) {
+          case SystemZ::CRMux:
+          case SystemZ::CLRMux:
+            processOperands();
+            // Avoid LH
+            if (Reg2RC[Regs[0]] == LOW)
+              Reg2RC.constrainReg(Regs[1], LOW);
+            else if (Reg2RC[Regs[1]] == HIGH)
+              Reg2RC.constrainReg(Regs[0], HIGH);
+            break;
+
+          case SystemZ::SRMux:
+          case SystemZ::SRMuxK:
+          case SystemZ::SLRMux:
+          case SystemZ::SLRMuxK:
+            processOperands();
+            // H__ -> HH_
+            if (Reg2RC[Regs[0]] == HIGH) {
+              Reg2RC.constrainReg(Regs[1], HIGH);
+              break;
+            }
+            // _L_ -> LLL
+            if (Reg2RC[Regs[1]] == LOW) {
+              Reg2RC.constrainReg(Regs[0], LOW);
+              Reg2RC.constrainReg(Regs[2], LOW);
+              break;
+            }
+            LLVM_FALLTHROUGH;
+          case SystemZ::ARMux:
+          case SystemZ::ARMuxK:
+          case SystemZ::ALRMux:
+          case SystemZ::ALRMuxK:
+            processOperands();
+            // L__ -> LLL
+            if (Reg2RC[Regs[0]] == LOW) {
+              Reg2RC.constrainReg(Regs[1], LOW);
+              Reg2RC.constrainReg(Regs[2], LOW);
+            }
+            // _LL -> LLL
+            else if (Reg2RC[Regs[1]] == LOW && Reg2RC[Regs[2]] == LOW)
+              Reg2RC.constrainReg(Regs[0], LOW);
+            else if (Reg2RC[Regs[0]] == HIGH) {
+              // HL_ -> HLH
+              if (Reg2RC[Regs[1]] == LOW)
+                Reg2RC.constrainReg(Regs[2], HIGH);
+              // H_L -> HHL
+              else if (Reg2RC[Regs[2]] == LOW)
+                Reg2RC.constrainReg(Regs[1], HIGH);
+              // Hrr -> HHH
+              else if (Regs[1] == Regs[2])
+                Reg2RC.constrainReg(Regs[1], HIGH);
+            }
+            // _H? -> HH?
+            else if (Reg2RC[Regs[1]] == HIGH || Reg2RC[Regs[2]] == HIGH)
+              Reg2RC.constrainReg(Regs[0], HIGH);
+            break;
+          case SystemZ::COPY:
+            if (MI.getOperand(1).getReg() == Reg)
+              Worklist.push_back(MI.getOperand(0).getReg()); // USEFUL?
+            break;
+          }
+        } // Reg
+      } // Worklist
+    } // Change
+    if (Reg2RC.RequiredHigh > Reg2RC.RequiredLow) {
+      addHints(Order, Hints, &SystemZ::GRH32BitRegClass, MRI);
+      LLVM_DEBUG(dbgs() << "SystemZ: Cmp/Add/Sub Mux hints\n");
+      return true;
+    } else if (Reg2RC.RequiredLow > Reg2RC.RequiredHigh) {
+      addHints(Order, Hints, &SystemZ::GR32BitRegClass, MRI);
+      LLVM_DEBUG(dbgs() << "SystemZ: Cmp/Add/Sub Mux hints\n");
+      return true;
+    }
 
+    // XXX: Incorporate rest of hints above? Or do LOCRMux first?
+    // LOCRMux
+    Worklist.clear();
+    DoneRegs.clear();
+    Worklist.push_back(VirtReg);
+    while (Worklist.size()) {
+      unsigned Reg = Worklist.pop_back_val();
+      if (!DoneRegs.insert(Reg).second)
+        continue;
       for (auto &Use : MRI->use_instructions(Reg)) {
         // For LOCRMux, see if the other operand is already a high or low
         // register, and in that case give the correpsonding hints for
         // VirtReg. LOCR instructions need both operands in either high or
         // low parts.
         if (Use.getOpcode() == SystemZ::LOCRMux) {
           MachineOperand &TrueMO = Use.getOperand(1);
           MachineOperand &FalseMO = Use.getOperand(2);
           const TargetRegisterClass *RC =
             TRI->getCommonSubClass(getRC32(FalseMO, VRM, MRI),
                                    getRC32(TrueMO, VRM, MRI));
           if (RC && RC != &SystemZ::GRX32BitRegClass) {
             addHints(Order, Hints, RC, MRI);
+            LLVM_DEBUG(dbgs() << "SystemZ: LOCRMux hints\n");
             // Return true to make these hints the only regs available to
             // RA. This may mean extra spilling but since the alternative is
             // a jump sequence expansion of the LOCRMux, it is preferred.
             return true;
           }
 
           // Add the other operand of the LOCRMux to the worklist.
           unsigned OtherReg =
             (TrueMO.getReg() == Reg ? FalseMO.getReg() : TrueMO.getReg());
           if (MRI->getRegClass(OtherReg) == &SystemZ::GRX32BitRegClass)
             Worklist.push_back(OtherReg);
-        } // end LOCRMux
-        else if (Use.getOpcode() == SystemZ::CHIMux ||
-                 Use.getOpcode() == SystemZ::CFIMux) {
-          if (Use.getOperand(1).getImm() == 0) {
-            bool OnlyLMuxes = true;
-            for (MachineInstr &DefMI : MRI->def_instructions(VirtReg))
-              if (DefMI.getOpcode() != SystemZ::LMux)
-                OnlyLMuxes = false;
-            if (OnlyLMuxes) {
-              addHints(Order, Hints, &SystemZ::GR32BitRegClass, MRI);
-              // Return false to make these hints preferred but not obligatory.
-              return false;
-            }
-          }
-        } // end CHIMux / CFIMux
-      }
+        }
+      }
+
+    // CHIMux / CFIMux
+    for (auto &Use : MRI->use_instructions(VirtReg))
+      if ((Use.getOpcode() == SystemZ::CHIMux ||
+           Use.getOpcode() == SystemZ::CFIMux) &&
+          Use.getOperand(1).getImm() == 0) {
+          LLVM_DEBUG(dbgs() << "SystemZ: CHIMux/CFIMux hints\n");
+
+  return BaseImplRetVal;
+}
+
+bool SystemZRegisterInfo::isRegSuitableReplacement(const MachineInstr *MI,
+                                                   unsigned VReg,
+                                                   unsigned NewPhysReg,
+                                                   const VirtRegMap *VRM,
+                                                   unsigned OpIdx) const {
+  if (!SystemZ::GRX32BitRegClass.contains(NewPhysReg))
+    return true;
+  const TargetRegisterClass *NewRC = getRCFromPhysReg(NewPhysReg);
+  unsigned PhysRegs[3];
+  const TargetRegisterClass *RCs[3];
+  unsigned PRegToReplace = 0;
+  unsigned OpI = 0;
+  auto processOperands = [&]() -> bool {
+    for (; OpI < MI->getNumOperands(); ++OpI) {
+      const MachineOperand &MO = MI->getOperand(OpI);
+      if (MO.isImplicit())
+        break;
+      unsigned Reg = MO.getReg();
+      unsigned PhysReg = (isPhysicalRegister(Reg) ? Reg : VRM->getPhys(Reg));
+      if ((VReg && VReg == Reg) || (!VReg && OpIdx == OpI)) {
+        PRegToReplace = PhysReg;
+        if (NewRC == getRCFromPhysReg(PRegToReplace))
+          return false;
       }
+      assert(PhysReg && "Expected phys reg for operand");
+      if (MO.getSubReg() != SystemZ::NoSubRegister)
+        PhysReg = getSubReg(PhysReg, MO.getSubReg());
+      PhysRegs[OpI] = PhysReg;
+      RCs[OpI] = getRCFromPhysReg(PhysReg);
     }
+    return true;
+  };
 
-  return BaseImplRetVal;
+  bool Unsupported = false;
+  switch (MI->getOpcode()) {
+  case SystemZ::CRMux:
+  case SystemZ::CLRMux: {
+    if (!processOperands())
+      break;
+    bool LHS = (PRegToReplace == PhysRegs[0]);
+    if ((LHS && NewRC == &SystemZ::GR32BitRegClass &&    // Avoid LH
+         RCs[1] == &SystemZ::GRH32BitRegClass) ||
+        (!LHS && NewRC == &SystemZ::GRH32BitRegClass &&
+         RCs[0] == &SystemZ::GR32BitRegClass))
+      Unsupported = true;
+    break;
+  }
+  case SystemZ::SRMuxK:
+  case SystemZ::SLRMuxK:
+    if (!processOperands())
+      break;
+    if (RCs[0] == &SystemZ::GRH32BitRegClass &&   // Avoid HLH
+        PRegToReplace == PhysRegs[1] && NewRC == &SystemZ::GR32BitRegClass) {
+      Unsupported = true;
+      break;
+    }
+    LLVM_FALLTHROUGH;
+  case SystemZ::SRMux:
+  case SystemZ::SLRMux:
+  case SystemZ::ARMux:
+  case SystemZ::ARMuxK:
+  case SystemZ::ALRMux:
+  case SystemZ::ALRMuxK:
+    if (!processOperands())
+      break;
+    if ((RCs[0] == &SystemZ::GR32BitRegClass &&   // Avoid LLH / LHL / HLL
+         RCs[1] == &SystemZ::GR32BitRegClass &&
+         RCs[2] == &SystemZ::GR32BitRegClass) ||
+        ((PRegToReplace == PhysRegs[0]) &&        // Avoid LHH
+         RCs[0] == &SystemZ::GRH32BitRegClass &&
+         RCs[1] == &SystemZ::GRH32BitRegClass &&
+         RCs[2] == &SystemZ::GRH32BitRegClass))
+      Unsupported = true;
+    break;
+  default: break;
+  }
+  if (Unsupported) {
+    LLVM_DEBUG(dbgs() << "SystemZ: Avoiding replacing "
+               << getName(PRegToReplace) << " with "
+               << getName(NewPhysReg) << " in "; MI->dump());
+    return false;
+  }
+  return true;
 }
 
 const MCPhysReg *
 SystemZRegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   const SystemZSubtarget &Subtarget = MF->getSubtarget<SystemZSubtarget>();
   if (MF->getFunction().getCallingConv() == CallingConv::AnyReg)
     return Subtarget.hasVector()? CSR_SystemZ_AllRegs_Vector_SaveList
                                 : CSR_SystemZ_AllRegs_SaveList;

lib/Target/SystemZ/SystemZScheduleZ13.td

def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXa, LSU, NormalGr],
              (instregex "ALG(F)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALGHSIK$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALGF(I|R)$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALGR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "AR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "A(L)?HHHR$")>;
 def : InstRW<[WLat2, WLat2, FXa, NormalGr], (instregex "A(L)?HHLR$")>;
+def : InstRW<[WLat1, FXa, NormalGr], (instregex "A(L)?RMux(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALSIH(N)?$")>;
 def : InstRW<[WLat2LSU, FXb, LSU, NormalGr], (instregex "A(L)?(G)?SI$")>;
 
 // Logical addition with carry
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXa, LSU, GroupAlone],
              (instregex "ALC(G)?$")>;
 def : InstRW<[WLat2, WLat2, FXa, GroupAlone], (instregex "ALC(G)?R$")>;
 

 def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXa, LSU, NormalGr],
              (instregex "SL(G|GF|Y)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "SLGF(I|R)$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "SLGR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "SLR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "SR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "S(L)?HHHR$")>;
 def : InstRW<[WLat2, WLat2, FXa, NormalGr], (instregex "S(L)?HHLR$")>;
+def : InstRW<[WLat1, FXa, NormalGr], (instregex "S(L)?RMux(K)?$")>;
 
 // Subtraction with borrow
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXa, LSU, GroupAlone],
              (instregex "SLB(G)?$")>;
 def : InstRW<[WLat2, WLat2, FXa, GroupAlone], (instregex "SLB(G)?R$")>;
 
 // Subtraction with sign extension (32 -> 64)
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXa, LSU, NormalGr],

 def : InstRW<[WLat1LSU, RegReadAdv, FXb, LSU, NormalGr], (instregex "CLHF$")>;
 def : InstRW<[WLat1LSU, FXb, LSU, NormalGr], (instregex "CLH(RL|HSI)$")>;
 def : InstRW<[WLat1, FXb, NormalGr], (instregex "CLIH$")>;
 def : InstRW<[WLat1LSU, FXb, LSU, NormalGr], (instregex "CLI(Y)?$")>;
 def : InstRW<[WLat1, FXb, NormalGr], (instregex "CLR$")>;
 def : InstRW<[WLat1LSU, FXb, LSU, NormalGr], (instregex "CLRL$")>;
 def : InstRW<[WLat1, FXb, NormalGr], (instregex "C(L)?HHR$")>;
 def : InstRW<[WLat2, FXb, NormalGr], (instregex "C(L)?HLR$")>;
+def : InstRW<[WLat1, FXb, NormalGr], (instregex "C(L)?RMux")>;
 
 // Compare halfword
 def : InstRW<[WLat2LSU, RegReadAdv, FXb, LSU, NormalGr], (instregex "CH(Y)?$")>;
 def : InstRW<[WLat2LSU, FXb, LSU, NormalGr], (instregex "CHRL$")>;
 def : InstRW<[WLat2LSU, RegReadAdv, FXb, LSU, NormalGr], (instregex "CGH$")>;
 def : InstRW<[WLat2LSU, FXb, LSU, NormalGr], (instregex "CGHRL$")>;
 def : InstRW<[WLat2LSU, FXa, FXb, LSU, Cracked], (instregex "CHHSI$")>;
 

lib/Target/SystemZ/SystemZScheduleZ14.td

def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXa, LSU, NormalGr],
              (instregex "ALG(F)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALGHSIK$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALGF(I|R)$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALGR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "AR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "A(L)?HHHR$")>;
 def : InstRW<[WLat2, WLat2, FXa, NormalGr], (instregex "A(L)?HHLR$")>;
+def : InstRW<[WLat1, FXa, NormalGr], (instregex "A(L)?RMux(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "ALSIH(N)?$")>;
 def : InstRW<[WLat2LSU, FXb, LSU, NormalGr], (instregex "A(L)?(G)?SI$")>;
 
 // Logical addition with carry
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXa, LSU, GroupAlone],
              (instregex "ALC(G)?$")>;
 def : InstRW<[WLat2, WLat2, FXa, GroupAlone], (instregex "ALC(G)?R$")>;
 

 def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXa, LSU, NormalGr],
              (instregex "SL(G|GF|Y)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "SLGF(I|R)$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "SLGR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "SLR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "SR(K)?$")>;
 def : InstRW<[WLat1, FXa, NormalGr], (instregex "S(L)?HHHR$")>;
 def : InstRW<[WLat2, WLat2, FXa, NormalGr], (instregex "S(L)?HHLR$")>;
+def : InstRW<[WLat1, FXa, NormalGr], (instregex "S(L)?RMux(K)?$")>;
 
 // Subtraction with borrow
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXa, LSU, GroupAlone],
              (instregex "SLB(G)?$")>;
 def : InstRW<[WLat2, WLat2, FXa, GroupAlone], (instregex "SLB(G)?R$")>;
 
 // Subtraction with sign extension (16/32 -> 64)
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXa, LSU, NormalGr],

 def : InstRW<[WLat1LSU, RegReadAdv, FXb, LSU, NormalGr], (instregex "CLHF$")>;
 def : InstRW<[WLat1LSU, FXb, LSU, NormalGr], (instregex "CLH(RL|HSI)$")>;
 def : InstRW<[WLat1, FXb, NormalGr], (instregex "CLIH$")>;
 def : InstRW<[WLat1LSU, FXb, LSU, NormalGr], (instregex "CLI(Y)?$")>;
 def : InstRW<[WLat1, FXb, NormalGr], (instregex "CLR$")>;
 def : InstRW<[WLat1LSU, FXb, LSU, NormalGr], (instregex "CLRL$")>;
 def : InstRW<[WLat1, FXb, NormalGr], (instregex "C(L)?HHR$")>;
 def : InstRW<[WLat2, FXb, NormalGr], (instregex "C(L)?HLR$")>;
+def : InstRW<[WLat1, FXb, NormalGr], (instregex "C(L)?RMux")>;
 
 // Compare halfword
 def : InstRW<[WLat2LSU, RegReadAdv, FXb, LSU, NormalGr], (instregex "CH(Y)?$")>;
 def : InstRW<[WLat2LSU, FXb, LSU, NormalGr], (instregex "CHRL$")>;
 def : InstRW<[WLat2LSU, RegReadAdv, FXb, LSU, NormalGr], (instregex "CGH$")>;
 def : InstRW<[WLat2LSU, FXb, LSU, NormalGr], (instregex "CGHRL$")>;
 def : InstRW<[WLat2LSU, FXa, FXb, LSU, Cracked], (instregex "CHHSI$")>;
 

lib/Target/SystemZ/SystemZScheduleZ196.td

def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXU, LSU, NormalGr],
              (instregex "ALGF$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALGHSIK$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALGF(I|R)$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALGR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "AR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "A(L)?HHHR$")>;
 def : InstRW<[WLat2, WLat2, FXU2, GroupAlone], (instregex "A(L)?HHLR$")>;
+def : InstRW<[WLat1, FXU, NormalGr], (instregex "A(L)?RMux(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALSIH(N)?$")>;
 def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXU, LSU, NormalGr],
              (instregex "A(L)?G$")>;
 def : InstRW<[WLat1LSU, FXU, LSU, NormalGr], (instregex "A(L)?GSI$")>;
 
 // Logical addition with carry
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXU, LSU, GroupAlone],
              (instregex "ALC(G)?$")>;

 def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXU, LSU, NormalGr],
              (instregex "SL(G|GF|Y)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "SLGF(I|R)$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "SLGR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "SLR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "SR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "S(L)?HHHR$")>;
 def : InstRW<[WLat2, WLat2, FXU2, GroupAlone], (instregex "S(L)?HHLR$")>;
+def : InstRW<[WLat1, FXU, NormalGr], (instregex "S(L)?RMux(K)?$")>;
 
 // Subtraction with borrow
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXU, LSU, GroupAlone],
              (instregex "SLB(G)?$")>;
 def : InstRW<[WLat2, WLat2, FXU, GroupAlone], (instregex "SLB(G)?R$")>;
 
 // Subtraction with sign extension (32 -> 64)
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXU2, LSU, GroupAlone],

 def : InstRW<[WLat1LSU, RegReadAdv, FXU, LSU, NormalGr], (instregex "CLHF$")>;
 def : InstRW<[WLat1LSU, FXU, LSU, NormalGr], (instregex "CLH(RL|HSI)$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "CLIH$")>;
 def : InstRW<[WLat1LSU, FXU, LSU, NormalGr], (instregex "CLI(Y)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "CLR$")>;
 def : InstRW<[WLat1LSU, FXU, LSU, NormalGr], (instregex "CLRL$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "C(L)?HHR$")>;
 def : InstRW<[WLat2, FXU2, GroupAlone], (instregex "C(L)?HLR$")>;
+def : InstRW<[WLat1, FXU, NormalGr], (instregex "C(L)?RMux")>;
 
 // Compare halfword
 def : InstRW<[WLat2LSU, RegReadAdv, FXU2, LSU, GroupAlone],
              (instregex "CH(Y)?$")>;
 def : InstRW<[WLat2LSU, FXU2, LSU, GroupAlone], (instregex "CHRL$")>;
 def : InstRW<[WLat2LSU, RegReadAdv, FXU2, LSU, GroupAlone], (instregex "CGH$")>;
 def : InstRW<[WLat2LSU, FXU2, LSU, GroupAlone], (instregex "CGHRL$")>;
 def : InstRW<[WLat2LSU, FXU2, LSU, GroupAlone], (instregex "CHHSI$")>;

lib/Target/SystemZ/SystemZScheduleZEC12.td

def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXU, LSU, NormalGr],
              (instregex "ALGF$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALGHSIK$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALGF(I|R)$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALGR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "AR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "A(L)?HHHR$")>;
 def : InstRW<[WLat2, WLat2, FXU, NormalGr], (instregex "A(L)?HHLR$")>;
+def : InstRW<[WLat1, FXU, NormalGr], (instregex "A(L)?RMux(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "ALSIH(N)?$")>;
 def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXU, LSU, NormalGr],
              (instregex "A(L)?G$")>;
 def : InstRW<[WLat1LSU, FXU, LSU, NormalGr], (instregex "A(L)?GSI$")>;
 
 // Logical addition with carry
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXU, LSU, GroupAlone],
              (instregex "ALC(G)?$")>;

 def : InstRW<[WLat1LSU, WLat1LSU, RegReadAdv, FXU, LSU, NormalGr],
              (instregex "SL(G|GF|Y)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "SLGF(I|R)$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "SLGR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "SLR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "SR(K)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "S(L)?HHHR$")>;
 def : InstRW<[WLat2, WLat2, FXU, NormalGr], (instregex "S(L)?HHLR$")>;
+def : InstRW<[WLat1, FXU, NormalGr], (instregex "S(L)?RMux(K)?$")>;
 
 // Subtraction with borrow
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXU, LSU, GroupAlone],
              (instregex "SLB(G)?$")>;
 def : InstRW<[WLat2, WLat2, FXU, GroupAlone], (instregex "SLB(G)?R$")>;
 
 // Subtraction with sign extension (32 -> 64)
 def : InstRW<[WLat2LSU, WLat2LSU, RegReadAdv, FXU, LSU, NormalGr],

 def : InstRW<[WLat1LSU, RegReadAdv, FXU, LSU, NormalGr], (instregex "CLHF$")>;
 def : InstRW<[WLat1LSU, FXU, LSU, NormalGr], (instregex "CLH(RL|HSI)$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "CLIH$")>;
 def : InstRW<[WLat1LSU, FXU, LSU, NormalGr], (instregex "CLI(Y)?$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "CLR$")>;
 def : InstRW<[WLat1LSU, FXU, LSU, NormalGr], (instregex "CLRL$")>;
 def : InstRW<[WLat1, FXU, NormalGr], (instregex "C(L)?HHR$")>;
 def : InstRW<[WLat2, FXU, NormalGr], (instregex "C(L)?HLR$")>;
+def : InstRW<[WLat1, FXU, NormalGr], (instregex "C(L)?RMux")>;
 
 // Compare halfword
 def : InstRW<[WLat2LSU, RegReadAdv, FXU, LSU, NormalGr], (instregex "CH(Y)?$")>;
 def : InstRW<[WLat2LSU, FXU, LSU, NormalGr], (instregex "CHRL$")>;
 def : InstRW<[WLat2LSU, RegReadAdv, FXU, LSU, NormalGr], (instregex "CGH$")>;
 def : InstRW<[WLat2LSU, FXU, LSU, NormalGr], (instregex "CGHRL$")>;
 def : InstRW<[WLat2LSU, FXU2, LSU, GroupAlone], (instregex "CHHSI$")>;