[SystemZ] Handle 3-address instructions in foldMemoryOperandImpl()

[jonpa]

Summary

This patch is to be applied on top of the recent 3-address patch (https://reviews.llvm.org/D60888). Without this, there were a dozen of tests that failed with D60888, but with this patch they do not.

TargetPassConfig::addPostRewrite() method and VirtRegMap argument to foldMemoryOperand() approved already on a separate patch by Quentin Colombet (https://reviews.llvm.org/D58923).

Statistics:

"regalloc - Number of folded stack accesses":

master: 17762
3-address patch without handling in foldMemoryOperand(): 14081
3-address patch + this patch without commutation: 15285
with commutation: 17395

"systemz-postrewrite - Number of copies inserted before folded mem ops."
with commuation: 9

[uweigand]

I'm a bit confused about the mapping logic. For the case of e.g. ADD, we today have

AR ---> gets mapped to A by getMemOpcode

and

ARK --> no mapping via getMemOpcode

I would have expected the first mapping to stay as is, and a new second mapping of ARK to some pseudo A_MemFoldPseudo.
Instead, it seems you're redirecting the mapping of AR to A_MemFoldPseudo, if I'm reading the patch correctly. Why is this? If you already have an AR, you already have a two-operand form, so it can just be modified to A. The problem is as long as you have an ARK ...

I may have forgot, but could you explain again why we need (yet another) new pass for this, instead of just expanding the pseudo in one of the existing pseudo-expansion passes?

[jonpa]

merged into D60888.

include/llvm/CodeGen/TargetInstrInfo.h

 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineCombinerPattern.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineOutliner.h"
 #include "llvm/CodeGen/PseudoSourceValue.h"
+#include "llvm/CodeGen/VirtRegMap.h"
 #include "llvm/MC/MCInstrInfo.h"
 #include "llvm/Support/BranchProbability.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <utility>
 #include <vector>
public:
   virtual bool isSubregFoldable() const { return false; }
 
   /// Attempt to fold a load or store of the specified stack
   /// slot into the specified machine instruction for the specified operand(s).
   /// If this is possible, a new instruction is returned with the specified
   /// operand folded, otherwise NULL is returned.
   /// The new instruction is inserted before MI, and the client is responsible
   /// for removing the old instruction.
+  /// If VRM is passed, the assigned physregs can be inspected by target to
+  /// decide on using an opcode (note that those assignments can still change).
   MachineInstr *foldMemoryOperand(MachineInstr &MI, ArrayRef<unsigned> Ops,
                                   int FI,
-                                  LiveIntervals *LIS = nullptr) const;
+                                  LiveIntervals *LIS = nullptr,
+                                  VirtRegMap *VRM = nullptr) const;
 
   /// Same as the previous version except it allows folding of any load and
   /// store from / to any address, not just from a specific stack slot.
   MachineInstr *foldMemoryOperand(MachineInstr &MI, ArrayRef<unsigned> Ops,
                                   MachineInstr &LoadMI,
                                   LiveIntervals *LIS = nullptr) const;
 
   /// Return true when there is potentially a faster code sequence
protected:
   /// Target-independent code in foldMemoryOperand will
   /// take care of adding a MachineMemOperand to the newly created instruction.
   /// The instruction and any auxiliary instructions necessary will be inserted
   /// at InsertPt.
   virtual MachineInstr *
   foldMemoryOperandImpl(MachineFunction &MF, MachineInstr &MI,
                         ArrayRef<unsigned> Ops,
                         MachineBasicBlock::iterator InsertPt, int FrameIndex,
-                        LiveIntervals *LIS = nullptr) const {
+                        LiveIntervals *LIS = nullptr,
+                        VirtRegMap *VRM = nullptr) const {
     return nullptr;
   }
 
   /// Target-dependent implementation for foldMemoryOperand.
   /// Target-independent code in foldMemoryOperand will
   /// take care of adding a MachineMemOperand to the newly created instruction.
   /// The instruction and any auxiliary instructions necessary will be inserted
   /// at InsertPt.

include/llvm/CodeGen/TargetPassConfig.h

protected:
   ///
   /// Note if the target overloads addRegAssignAndRewriteOptimized, this may not
   /// be honored. This is also not generally used for the the fast variant,
   /// where the allocation and rewriting are done in one pass.
   virtual bool addPreRewrite() {
     return false;
   }
 
+  /// Add passes to be run immediately after virtual registers are rewritten
+  /// to physical registers. These passes may replace an MI with a new one,
+  /// but should preserve SlotIndexes while doing so.
+  virtual void addPostRewrite() { }
+
   /// This method may be implemented by targets that want to run passes after
   /// register allocation pass pipeline but before prolog-epilog insertion.
   virtual void addPostRegAlloc() { }
 
   /// Add passes that optimize machine instructions after register allocation.
   virtual void addMachineLateOptimization();
 
   /// This method may be implemented by targets that want to run passes after

lib/CodeGen/InlineSpiller.cpp

foldMemoryOperand(ArrayRef<std::pair<MachineInstr *, unsigned>> Ops,
   // Moreover, TargetInstrInfo::foldMemoryOperand will assert if we try!
   if (FoldOps.empty())
     return false;
 
   MachineInstrSpan MIS(MI);
 
   MachineInstr *FoldMI =
       LoadMI ? TII.foldMemoryOperand(*MI, FoldOps, *LoadMI, &LIS)
-             : TII.foldMemoryOperand(*MI, FoldOps, StackSlot, &LIS);
+             : TII.foldMemoryOperand(*MI, FoldOps, StackSlot, &LIS, &VRM);
   if (!FoldMI)
     return false;
 
   // Remove LIS for any dead defs in the original MI not in FoldMI.
   for (MIBundleOperands MO(*MI); MO.isValid(); ++MO) {
     if (!MO->isReg())
       continue;
     unsigned Reg = MO->getReg();

lib/CodeGen/TargetInstrInfo.cpp

for (unsigned i = StartIdx; i < MI.getNumOperands(); ++i) {
     else
       MIB.add(MO);
   }
   return NewMI;
 }
 
 MachineInstr *TargetInstrInfo::foldMemoryOperand(MachineInstr &MI,
                                                  ArrayRef<unsigned> Ops, int FI,
-                                                 LiveIntervals *LIS) const {
+                                                 LiveIntervals *LIS,
+                                                 VirtRegMap *VRM) const {
   auto Flags = MachineMemOperand::MONone;
   for (unsigned OpIdx : Ops)
     Flags |= MI.getOperand(OpIdx).isDef() ? MachineMemOperand::MOStore
                                           : MachineMemOperand::MOLoad;
 
   MachineBasicBlock *MBB = MI.getParent();
   assert(MBB && "foldMemoryOperand needs an inserted instruction");
   MachineFunction &MF = *MBB->getParent();
if (MI.getOpcode() == TargetOpcode::STACKMAP ||
       MI.getOpcode() == TargetOpcode::PATCHPOINT ||
       MI.getOpcode() == TargetOpcode::STATEPOINT) {
     // Fold stackmap/patchpoint.
     NewMI = foldPatchpoint(MF, MI, Ops, FI, *this);
     if (NewMI)
       MBB->insert(MI, NewMI);
   } else {
     // Ask the target to do the actual folding.
-    NewMI = foldMemoryOperandImpl(MF, MI, Ops, MI, FI, LIS);
+    NewMI = foldMemoryOperandImpl(MF, MI, Ops, MI, FI, LIS, VRM);
   }
 
   if (NewMI) {
     NewMI->setMemRefs(MF, MI.memoperands());
     // Add a memory operand, foldMemoryOperandImpl doesn't do that.
     assert((!(Flags & MachineMemOperand::MOStore) ||
             NewMI->mayStore()) &&
            "Folded a def to a non-store!");

lib/CodeGen/TargetPassConfig.cpp

void TargetPassConfig::addOptimizedRegAlloc() {
   // when moving subregister definitions around, avoid this by splitting them to
   // separate vregs before. Splitting can also improve reg. allocation quality.
   addPass(&RenameIndependentSubregsID);
 
   // PreRA instruction scheduling.
   addPass(&MachineSchedulerID);
 
   if (addRegAssignmentOptimized()) {
+    // Allow targets to expand pseudo instructions depending on the choice of
+    // registers before MachineCopyPropagation.
+    addPostRewrite();
+
     // Copy propagate to forward register uses and try to eliminate COPYs that
     // were not coalesced.
     addPass(&MachineCopyPropagationID);
 
     // Run post-ra machine LICM to hoist reloads / remats.
     //
     // FIXME: can this move into MachineLateOptimization?
     addPass(&MachineLICMID);

lib/Target/AArch64/AArch64InstrInfo.h

public:
   // with subreg operands to foldMemoryOperandImpl.
   bool isSubregFoldable() const override { return true; }
 
   using TargetInstrInfo::foldMemoryOperandImpl;
   MachineInstr *
   foldMemoryOperandImpl(MachineFunction &MF, MachineInstr &MI,
                         ArrayRef<unsigned> Ops,
                         MachineBasicBlock::iterator InsertPt, int FrameIndex,
-                        LiveIntervals *LIS = nullptr) const override;
+                        LiveIntervals *LIS = nullptr,
+                        VirtRegMap *VRM = nullptr) const override;
 
   /// \returns true if a branch from an instruction with opcode \p BranchOpc
   ///  bytes is capable of jumping to a position \p BrOffset bytes away.
   bool isBranchOffsetInRange(unsigned BranchOpc,
                              int64_t BrOffset) const override;
 
   MachineBasicBlock *getBranchDestBlock(const MachineInstr &MI) const override;
 

lib/Target/AArch64/AArch64InstrInfo.cpp

if ((DestReg == AArch64::FP && SrcReg == AArch64::SP) ||
               addImm(Offset).setMIFlag(Flag);
     }
   }
 }
 
 MachineInstr *AArch64InstrInfo::foldMemoryOperandImpl(
     MachineFunction &MF, MachineInstr &MI, ArrayRef<unsigned> Ops,
     MachineBasicBlock::iterator InsertPt, int FrameIndex,
-    LiveIntervals *LIS) const {
+    LiveIntervals *LIS, VirtRegMap *VRM) const {
   // This is a bit of a hack. Consider this instruction:
   //
   //   %0 = COPY %sp; GPR64all:%0
   //
   // We explicitly chose GPR64all for the virtual register so such a copy might
   // be eliminated by RegisterCoalescer. However, that may not be possible, and
   // %0 may even spill. We can't spill %sp, and since it is in the GPR64all
   // register class, TargetInstrInfo::foldMemoryOperand() is going to try.

lib/Target/SystemZ/CMakeLists.txt

add_llvm_target(SystemZCodeGen
   SystemZInstrInfo.cpp
   SystemZLDCleanup.cpp
   SystemZLongBranch.cpp
   SystemZMachineFunctionInfo.cpp
   SystemZMachineScheduler.cpp
   SystemZMCInstLower.cpp
   SystemZRegisterInfo.cpp
   SystemZSelectionDAGInfo.cpp
+  SystemZPostRewrite.cpp
   SystemZShortenInst.cpp
   SystemZSubtarget.cpp
   SystemZTargetMachine.cpp
   SystemZTargetTransformInfo.cpp
   SystemZTDC.cpp
   )
 
 add_subdirectory(AsmParser)
 add_subdirectory(Disassembler)
 add_subdirectory(MCTargetDesc)
 add_subdirectory(TargetInfo)

lib/Target/SystemZ/SystemZ.h

 
 FunctionPass *createSystemZISelDag(SystemZTargetMachine &TM,
                                    CodeGenOpt::Level OptLevel);
 FunctionPass *createSystemZElimComparePass(SystemZTargetMachine &TM);
 FunctionPass *createSystemZExpandPseudoPass(SystemZTargetMachine &TM);
 FunctionPass *createSystemZShortenInstPass(SystemZTargetMachine &TM);
 FunctionPass *createSystemZLongBranchPass(SystemZTargetMachine &TM);
 FunctionPass *createSystemZLDCleanupPass(SystemZTargetMachine &TM);
+FunctionPass *createSystemZPostRewritePass(SystemZTargetMachine &TM);
 FunctionPass *createSystemZTDCPass();
 } // end namespace llvm
 
 #endif

lib/Target/SystemZ/SystemZInstrFormats.td

class InstSystemZ<int size, dag outs, dag ins, string asmstr,
   string DispSize = "none";
 
   // Many register-based <INSN>R instructions have a memory-based <INSN>
   // counterpart.  OpKey uniquely identifies <INSN>R, while OpType is
   // "reg" for <INSN>R and "mem" for <INSN>.
   string OpKey = "";
   string OpType = "none";
 
+  // MemKey identifies a targe reg-mem opcode, while MemType can be either
+  // "pseudo" or "target". This is used to map a pseduo memory instruction to
+  // its corresponding target opcode. See comment at MemFoldPseudo.
+  string MemKey = "";
+  string MemType = "none";
+
   // Many distinct-operands instructions have older 2-operand equivalents.
   // NumOpsKey uniquely identifies one of these 2-operand and 3-operand pairs,
   // with NumOpsValue being "2" or "3" as appropriate.
   string NumOpsKey = "";
   string NumOpsValue = "none";
 
   // True if this instruction is a simple D(X,B) load of a register
   // (with no sign or zero extension).
class InstSystemZ<int size, dag outs, dag ins, string asmstr,
   let TSFlags{9-5}   = AccessBytes;
   let TSFlags{13-10} = CCValues;
   let TSFlags{17-14} = CompareZeroCCMask;
   let TSFlags{18}    = CCMaskFirst;
   let TSFlags{19}    = CCMaskLast;
   let TSFlags{20}    = IsLogical;
 }
 
+class Pseudo<dag outs, dag ins, list<dag> pattern>
+  : InstSystemZ<0, outs, ins, "", pattern> {
+  let isPseudo = 1;
+  let isCodeGenOnly = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // Mappings between instructions
 //===----------------------------------------------------------------------===//
 
 // Return the version of an instruction that has an unsigned 12-bit
 // displacement.
 def getDisp12Opcode : InstrMapping {
   let FilterClass = "InstSystemZ";
   let RowFields = ["DispKey"];
   let ColFields = ["DispSize"];
   let KeyCol = ["20"];
   let ValueCols = [["12"]];
 }
 
 // Return the version of an instruction that has a signed 20-bit displacement.
 def getDisp20Opcode : InstrMapping {
   let FilterClass = "InstSystemZ";
   let RowFields = ["DispKey"];
   let ColFields = ["DispSize"];
   let KeyCol = ["12"];
   let ValueCols = [["20"]];
 }
 
-// Return the memory form of a register instruction.
+// Return the memory form of a register instruction. Note that this may
+// return a MemFoldPseudo instruction (see below).
 def getMemOpcode : InstrMapping {
   let FilterClass = "InstSystemZ";
   let RowFields = ["OpKey"];
   let ColFields = ["OpType"];
   let KeyCol = ["reg"];
   let ValueCols = [["mem"]];
 }
 
+// Return the target memory instruction for a MemFoldPseudo.
+def getTargetMemOpcode : InstrMapping {
+  let FilterClass = "InstSystemZ";
+  let RowFields = ["MemKey"];
+  let ColFields = ["MemType"];
+  let KeyCol = ["pseudo"];
+  let ValueCols = [["target"]];
+}
+
 // Return the 2-operand form of a 3-operand instruction.
 def getTwoOperandOpcode : InstrMapping {
   let FilterClass = "InstSystemZ";
   let RowFields = ["NumOpsKey"];
   let ColFields = ["NumOpsValue"];
   let KeyCol = ["3"];
   let ValueCols = [["2"]];
 }

 
 class BinaryRSL<string mnemonic, bits<16> opcode, RegisterOperand cls>
   : InstRSLb<opcode, (outs cls:$R1),
              (ins bdladdr12onlylen8:$BDL2, imm32zx4:$M3),
              mnemonic#"\t$R1, $BDL2, $M3", []> {
   let mayLoad = 1;
 }
 
-class BinaryRX<string mnemonic, bits<8> opcode, SDPatternOperator operator,
+class BinaryRX_base<string mnemonic, bits<8> opcode, SDPatternOperator operator,
                RegisterOperand cls, SDPatternOperator load, bits<5> bytes,
                AddressingMode mode = bdxaddr12only>
   : InstRXa<opcode, (outs cls:$R1), (ins cls:$R1src, mode:$XBD2),
             mnemonic#"\t$R1, $XBD2",
             [(set cls:$R1, (operator cls:$R1src, (load mode:$XBD2)))]> {
-  let OpKey = mnemonic#"r"#cls;
-  let OpType = "mem";
   let Constraints = "$R1 = $R1src";
   let DisableEncoding = "$R1src";
   let mayLoad = 1;
   let AccessBytes = bytes;
 }
 
+class BinaryRX<string mnemonic, bits<8> opcode, SDPatternOperator operator,
+               RegisterOperand cls, SDPatternOperator load, bits<5> bytes,
+               AddressingMode mode = bdxaddr12only>
+  : BinaryRX_base<mnemonic, opcode, operator, cls, load, bytes, mode> {
+  let OpKey = mnemonic#"r"#cls;
+  let OpType = "mem";
+}
+
 class BinaryRXE<string mnemonic, bits<16> opcode, SDPatternOperator operator,
                   RegisterOperand cls, SDPatternOperator load, bits<5> bytes>
   : InstRXE<opcode, (outs cls:$R1), (ins cls:$R1src, bdxaddr12only:$XBD2),
             mnemonic#"\t$R1, $XBD2",
             [(set cls:$R1, (operator cls:$R1src,
                                      (load bdxaddr12only:$XBD2)))]> {
   let OpKey = mnemonic#"r"#cls;
   let OpType = "mem";
: InstRXF<opcode, (outs cls1:$R1), (ins cls2:$R3, bdxaddr12only:$XBD2),
             mnemonic#"\t$R1, $R3, $XBD2",
             [(set cls1:$R1, (operator cls2:$R3, (load bdxaddr12only:$XBD2)))]> {
   let OpKey = mnemonic#"r"#cls;
   let OpType = "mem";
   let mayLoad = 1;
   let AccessBytes = bytes;
 }
 
-class BinaryRXY<string mnemonic, bits<16> opcode, SDPatternOperator operator,
-                RegisterOperand cls, SDPatternOperator load, bits<5> bytes,
+class BinaryRXY_base<string mnemonic, bits<16> opcode,
+                     SDPatternOperator operator, RegisterOperand cls,
+                     SDPatternOperator load, bits<5> bytes,
                      AddressingMode mode = bdxaddr20only>
   : InstRXYa<opcode, (outs cls:$R1), (ins cls:$R1src, mode:$XBD2),
              mnemonic#"\t$R1, $XBD2",
              [(set cls:$R1, (operator cls:$R1src, (load mode:$XBD2)))]> {
-  let OpKey = mnemonic#"r"#cls;
-  let OpType = "mem";
   let Constraints = "$R1 = $R1src";
   let DisableEncoding = "$R1src";
   let mayLoad = 1;
   let AccessBytes = bytes;
 }
 
+class BinaryRXY<string mnemonic, bits<16> opcode, SDPatternOperator operator,
+                RegisterOperand cls, SDPatternOperator load, bits<5> bytes,
+                AddressingMode mode = bdxaddr20only>
+  : BinaryRXY_base<mnemonic, opcode, operator, cls, load, bytes, mode> {
+  let OpKey = mnemonic#"r"#cls;
+  let OpType = "mem";
+}
+
+// A pseudo that is used during register allocation when folding a memory
+// operand. The 3-address register instruction with a spilled source cannot
+// be converted directly to a target 2-address reg/mem instruction.
+// Mapping:  <INSN>R  ->  MemFoldPseudo  ->  <INSN>
+class MemFoldPseudo<string mnemonic, RegisterOperand cls, bits<5> bytes,
+                    AddressingMode mode>
+  : Pseudo<(outs cls:$R1), (ins cls:$R2, mode:$XBD2), []> {
+    let OpKey = mnemonic#"r"#cls;
+    let OpType = "mem";
+    let MemKey = mnemonic#cls;
+    let MemType = "pseudo";
+    let mayLoad = 1;
+    let AccessBytes = bytes;
+    let HasIndex = 1;
+    let hasNoSchedulingInfo = 1;
+}
+
+multiclass BinaryRXYAndPseudo<string mnemonic, bits<16> opcode,
+                              SDPatternOperator operator, RegisterOperand cls,
+                              SDPatternOperator load, bits<5> bytes,
+                              AddressingMode mode = bdxaddr20only> {
+
+  def "" : BinaryRXY_base<mnemonic, opcode, operator, cls, load, bytes, mode> {
+    let MemKey = mnemonic#cls;
+    let MemType = "target";
+  }
+  let Has20BitOffset = 1 in
+    def _MemFoldPseudo : MemFoldPseudo<mnemonic, cls, bytes, mode>;
+}
+
 multiclass BinaryRXPair<string mnemonic, bits<8> rxOpcode, bits<16> rxyOpcode,
                         SDPatternOperator operator, RegisterOperand cls,
                         SDPatternOperator load, bits<5> bytes> {
   let DispKey = mnemonic ## #cls in {
     let DispSize = "12" in
       def "" : BinaryRX<mnemonic, rxOpcode, operator, cls, load, bytes,
                         bdxaddr12pair>;
     let DispSize = "20" in
       def Y  : BinaryRXY<mnemonic#"y", rxyOpcode, operator, cls, load, bytes,
                          bdxaddr20pair>;
   }
 }
 
+multiclass BinaryRXPairAndPseudo<string mnemonic, bits<8> rxOpcode,
+                                 bits<16> rxyOpcode, SDPatternOperator operator,
+                                 RegisterOperand cls,
+                                 SDPatternOperator load, bits<5> bytes> {
+  let DispKey = mnemonic ## #cls in {
+    def "" : BinaryRX_base<mnemonic, rxOpcode, operator, cls, load, bytes,
+                      bdxaddr12pair> {
+      let DispSize = "12";
+      let MemKey = mnemonic#cls;
+      let MemType = "target";
+    }
+    let DispSize = "20" in
+      def Y  : BinaryRXY_base<mnemonic#"y", rxyOpcode, operator, cls, load,
+                              bytes, bdxaddr20pair>;
+  }
+  def _MemFoldPseudo : MemFoldPseudo<mnemonic, cls, bytes, bdxaddr12pair>;
+}
+
 class BinarySI<string mnemonic, bits<8> opcode, SDPatternOperator operator,
                Operand imm, AddressingMode mode = bdaddr12only>
   : InstSI<opcode, (outs), (ins mode:$BD1, imm:$I2),
            mnemonic#"\t$BD1, $I2",
            [(store (operator (load mode:$BD1), imm:$I2), mode:$BD1)]> {
   let mayLoad = 1;
   let mayStore = 1;
 }

 //===----------------------------------------------------------------------===//
 //
 // Convenience instructions that get lowered to real instructions
 // by either SystemZTargetLowering::EmitInstrWithCustomInserter()
 // or SystemZInstrInfo::expandPostRAPseudo().
 //
 //===----------------------------------------------------------------------===//
 
-class Pseudo<dag outs, dag ins, list<dag> pattern>
-  : InstSystemZ<0, outs, ins, "", pattern> {
-  let isPseudo = 1;
-  let isCodeGenOnly = 1;
-}
-
 // Like UnaryRI, but expanded after RA depending on the choice of register.
 class UnaryRIPseudo<SDPatternOperator operator, RegisterOperand cls,
                     Immediate imm>
   : Pseudo<(outs cls:$R1), (ins imm:$I2),
            [(set cls:$R1, (operator imm:$I2))]>;
 
 // Like UnaryRXY, but expanded after RA depending on the choice of register.
 class UnaryRXYPseudo<string key, SDPatternOperator operator,

lib/Target/SystemZ/SystemZInstrInfo.h

enum FusedCompareType {
   // Trap
   CompareAndTrap
 };
 
 } // end namespace SystemZII
 
 namespace SystemZ {
 int getTwoOperandOpcode(uint16_t Opcode);
+int getTargetMemOpcode(uint16_t Opcode);
 }
 
 class SystemZInstrInfo : public SystemZGenInstrInfo {
   const SystemZRegisterInfo RI;
   SystemZSubtarget &STI;
 
   void splitMove(MachineBasicBlock::iterator MI, unsigned NewOpcode) const;
   void splitAdjDynAlloc(MachineBasicBlock::iterator MI) const;
void loadRegFromStackSlot(MachineBasicBlock &MBB,
                             const TargetRegisterInfo *TRI) const override;
   MachineInstr *convertToThreeAddress(MachineFunction::iterator &MFI,
                                       MachineInstr &MI,
                                       LiveVariables *LV) const override;
   MachineInstr *
   foldMemoryOperandImpl(MachineFunction &MF, MachineInstr &MI,
                         ArrayRef<unsigned> Ops,
                         MachineBasicBlock::iterator InsertPt, int FrameIndex,
-                        LiveIntervals *LIS = nullptr) const override;
+                        LiveIntervals *LIS = nullptr,
+                        VirtRegMap *VRM = nullptr) const override;
   MachineInstr *foldMemoryOperandImpl(
       MachineFunction &MF, MachineInstr &MI, ArrayRef<unsigned> Ops,
       MachineBasicBlock::iterator InsertPt, MachineInstr &LoadMI,
       LiveIntervals *LIS = nullptr) const override;
   bool expandPostRAPseudo(MachineInstr &MBBI) const override;
   bool reverseBranchCondition(SmallVectorImpl<MachineOperand> &Cond) const
     override;
 

lib/Target/SystemZ/SystemZInstrInfo.cpp

if (LogicOp And = interpretAndImmediate(MI.getOpcode())) {
     }
   }
   return nullptr;
 }
 
 MachineInstr *SystemZInstrInfo::foldMemoryOperandImpl(
     MachineFunction &MF, MachineInstr &MI, ArrayRef<unsigned> Ops,
     MachineBasicBlock::iterator InsertPt, int FrameIndex,
-    LiveIntervals *LIS) const {
+    LiveIntervals *LIS, VirtRegMap *VRM) const {
   const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   unsigned Size = MFI.getObjectSize(FrameIndex);
   unsigned Opcode = MI.getOpcode();
 
   if (Ops.size() == 2 && Ops[0] == 0 && Ops[1] == 1) {
     if (LIS != nullptr && (Opcode == SystemZ::LA || Opcode == SystemZ::LAY) &&
         isInt<8>(MI.getOperand(2).getImm()) && !MI.getOperand(3).getReg()) {
if (MMO->getSize() == Size && !MMO->isVolatile() && !MMO->isAtomic()) {
             .addImm(Size)
             .addFrameIndex(FrameIndex)
             .addImm(0)
             .addMemOperand(MMO);
       }
     }
   }
 
-  // If the spilled operand is the final one, try to change <INSN>R
-  // into <INSN>.
-  int MemOpcode = SystemZ::getMemOpcode(Opcode);
-  if (MemOpcode >= 0) {
-    unsigned NumOps = MI.getNumExplicitOperands();
-    if (OpNum == NumOps - 1) {
+  // If the spilled operand is the final one or the instruction is
+  // commutable, try to change <INSN>R into <INSN>.
+  int MemOpcode = -1;
+
+  // See if this is a 3-address instruction that is convertible to 2-address
+  // and suitable for folding below.  Only try this whith virtual registers
+  // and a provided VRM (during regalloc).
+  bool NeedsCommute = false;
+  int TwoOperandOpcode = SystemZ::getTwoOperandOpcode(Opcode);
+  int MemFoldPseudoOpc =
+      (TwoOperandOpcode == -1 ? -1 : SystemZ::getMemOpcode(TwoOperandOpcode));
+  if (VRM != nullptr && MemFoldPseudoOpc != -1) {
+    assert(NumOps == 3 && "Expected two source registers.");
+    unsigned DstReg = MI.getOperand(0).getReg();
+    unsigned DstPhys =
+        (TRI->isVirtualRegister(DstReg) ? VRM->getPhys(DstReg) : DstReg);
+    if (DstPhys && !SystemZ::GRH32BitRegClass.contains(DstPhys)) {
+      unsigned SrcReg = (OpNum == 2 ? MI.getOperand(1).getReg()
+                                    : ((OpNum == 1 && MI.isCommutable())
+                                           ? MI.getOperand(2).getReg()
+                                           : 0));
+      if (SrcReg && TRI->isVirtualRegister(SrcReg) &&
+          DstPhys == VRM->getPhys(SrcReg)) {
+        MemOpcode = MemFoldPseudoOpc;
+        NeedsCommute = (OpNum == 1);
+      }
+    }
+  } else
+    MemOpcode = SystemZ::getMemOpcode(Opcode);
+
+  if (MemOpcode >= 0) {
+    if ((OpNum == NumOps - 1) || NeedsCommute) {
       const MCInstrDesc &MemDesc = get(MemOpcode);
       uint64_t AccessBytes = SystemZII::getAccessSize(MemDesc.TSFlags);
       assert(AccessBytes != 0 && "Size of access should be known");
       assert(AccessBytes <= Size && "Access outside the frame index");
       uint64_t Offset = Size - AccessBytes;
       MachineInstrBuilder MIB = BuildMI(*InsertPt->getParent(), InsertPt,
                                         MI.getDebugLoc(), get(MemOpcode));
-      for (unsigned I = 0; I < OpNum; ++I)
+      MIB.add(MI.getOperand(0));
+      if (NeedsCommute)
+        MIB.add(MI.getOperand(2));
+      else
+        for (unsigned I = 1; I < OpNum; ++I)
           MIB.add(MI.getOperand(I));
       MIB.addFrameIndex(FrameIndex).addImm(Offset);
       if (MemDesc.TSFlags & SystemZII::HasIndex)
         MIB.addReg(0);
       transferDeadCC(&MI, MIB);
       return MIB;
     }
   }
 

lib/Target/SystemZ/SystemZInstrInfo.td

def AFIMux : BinaryRIPseudo<z_sadd, GRX32, simm32>,
                Requires<[FeatureHighWord]>;
   def AFI  : BinaryRIL<"afi",  0xC29, z_sadd, GR32, simm32>;
   def AIH  : BinaryRIL<"aih",  0xCC8, z_sadd, GRH32, simm32>,
              Requires<[FeatureHighWord]>;
   def AGFI : BinaryRIL<"agfi", 0xC28, z_sadd, GR64, imm64sx32>;
 
   // Addition of memory.
   defm AH  : BinaryRXPair<"ah", 0x4A, 0xE37A, z_sadd, GR32, asextloadi16, 2>;
-  defm A   : BinaryRXPair<"a",  0x5A, 0xE35A, z_sadd, GR32, load, 4>;
+  defm A   : BinaryRXPairAndPseudo<"a",  0x5A, 0xE35A, z_sadd, GR32, load, 4>;
   def  AGH : BinaryRXY<"agh", 0xE338, z_sadd, GR64, asextloadi16, 2>,
              Requires<[FeatureMiscellaneousExtensions2]>;
   def  AGF : BinaryRXY<"agf", 0xE318, z_sadd, GR64, asextloadi32, 4>;
-  def  AG  : BinaryRXY<"ag",  0xE308, z_sadd, GR64, load, 8>;
+  defm AG  : BinaryRXYAndPseudo<"ag",  0xE308, z_sadd, GR64, load, 8>;
 
   // Addition to memory.
   def ASI  : BinarySIY<"asi",  0xEB6A, add, imm32sx8>;
   def AGSI : BinarySIY<"agsi", 0xEB7A, add, imm64sx8>;
 }
 defm : SXB<z_sadd, GR64, AGFR>;
 
 // Addition producing a carry.
let Defs = [CC] in {
   def ALFI  : BinaryRIL<"alfi",  0xC2B, z_uadd, GR32, uimm32>;
   def ALGFI : BinaryRIL<"algfi", 0xC2A, z_uadd, GR64, imm64zx32>;
 
   // Addition of signed 32-bit immediates.
   def ALSIH : BinaryRIL<"alsih", 0xCCA, null_frag, GRH32, simm32>,
               Requires<[FeatureHighWord]>;
 
   // Addition of memory.
-  defm AL   : BinaryRXPair<"al", 0x5E, 0xE35E, z_uadd, GR32, load, 4>;
+  defm AL   : BinaryRXPairAndPseudo<"al", 0x5E, 0xE35E, z_uadd, GR32, load, 4>;
   def  ALGF : BinaryRXY<"algf", 0xE31A, z_uadd, GR64, azextloadi32, 4>;
-  def  ALG  : BinaryRXY<"alg",  0xE30A, z_uadd, GR64, load, 8>;
+  defm ALG  : BinaryRXYAndPseudo<"alg",  0xE30A, z_uadd, GR64, load, 8>;
 
   // Addition to memory.
   def ALSI  : BinarySIY<"alsi",  0xEB6E, null_frag, imm32sx8>;
   def ALGSI : BinarySIY<"algsi", 0xEB7E, null_frag, imm64sx8>;
 }
 defm : ZXB<z_uadd, GR64, ALGFR>;
 
 // Addition producing and using a carry.
let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8 in {
   // Subtraction from a high register.
   def SHHHR : BinaryRRFa<"shhhr", 0xB9C9, null_frag, GRH32, GRH32, GRH32>,
               Requires<[FeatureHighWord]>;
   def SHHLR : BinaryRRFa<"shhlr", 0xB9D9, null_frag, GRH32, GRH32, GR32>,
               Requires<[FeatureHighWord]>;
 
   // Subtraction of memory.
   defm SH  : BinaryRXPair<"sh", 0x4B, 0xE37B, z_ssub, GR32, asextloadi16, 2>;
-  defm S   : BinaryRXPair<"s", 0x5B, 0xE35B, z_ssub, GR32, load, 4>;
+  defm S   : BinaryRXPairAndPseudo<"s", 0x5B, 0xE35B, z_ssub, GR32, load, 4>;
   def  SGH : BinaryRXY<"sgh", 0xE339, z_ssub, GR64, asextloadi16, 2>,
              Requires<[FeatureMiscellaneousExtensions2]>;
   def  SGF : BinaryRXY<"sgf", 0xE319, z_ssub, GR64, asextloadi32, 4>;
-  def  SG  : BinaryRXY<"sg",  0xE309, z_ssub, GR64, load, 8>;
+  defm SG  : BinaryRXYAndPseudo<"sg",  0xE309, z_ssub, GR64, load, 8>;
 }
 defm : SXB<z_ssub, GR64, SGFR>;
 
 // Subtracting an immediate is the same as adding the negated immediate.
 let AddedComplexity = 1 in {
   def : Pat<(z_ssub GR32:$src1, imm32sx16n:$src2),
             (AHIMux GR32:$src1, imm32sx16n:$src2)>,
         Requires<[FeatureHighWord]>;
let Defs = [CC] in {
   def SLHHLR : BinaryRRFa<"slhhlr", 0xB9DB, null_frag, GRH32, GRH32, GR32>,
                Requires<[FeatureHighWord]>;
 
   // Subtraction of unsigned 32-bit immediates.
   def SLFI  : BinaryRIL<"slfi",  0xC25, z_usub, GR32, uimm32>;
   def SLGFI : BinaryRIL<"slgfi", 0xC24, z_usub, GR64, imm64zx32>;
 
   // Subtraction of memory.
-  defm SL   : BinaryRXPair<"sl", 0x5F, 0xE35F, z_usub, GR32, load, 4>;
+  defm SL   : BinaryRXPairAndPseudo<"sl", 0x5F, 0xE35F, z_usub, GR32, load, 4>;
   def  SLGF : BinaryRXY<"slgf", 0xE31B, z_usub, GR64, azextloadi32, 4>;
-  def  SLG  : BinaryRXY<"slg",  0xE30B, z_usub, GR64, load, 8>;
+  defm SLG  : BinaryRXYAndPseudo<"slg",  0xE30B, z_usub, GR64, load, 8>;
 }
 defm : ZXB<z_usub, GR64, SLGFR>;
 
 // Subtracting an immediate is the same as adding the negated immediate.
 let AddedComplexity = 1 in {
   def : Pat<(z_usub GR32:$src1, imm32sx16n:$src2),
             (ALHSIK GR32:$src1, imm32sx16n:$src2)>,
         Requires<[FeatureDistinctOps]>;
let CCValues = 0xC, CompareZeroCCMask = 0x8 in {
       def NIHF : BinaryRIL<"nihf", 0xC0A, and, GRH32, uimm32>;
     }
     def NILF64 : BinaryAliasRIL<and, GR64, imm64lf32c>;
     def NIHF64 : BinaryAliasRIL<and, GR64, imm64hf32c>;
   }
 
   // ANDs of memory.
   let CCValues = 0xC, CompareZeroCCMask = 0x8 in {
-    defm N  : BinaryRXPair<"n", 0x54, 0xE354, and, GR32, load, 4>;
-    def  NG : BinaryRXY<"ng", 0xE380, and, GR64, load, 8>;
+    defm N  : BinaryRXPairAndPseudo<"n", 0x54, 0xE354, and, GR32, load, 4>;
+    defm NG : BinaryRXYAndPseudo<"ng", 0xE380, and, GR64, load, 8>;
   }
 
   // AND to memory
   defm NI : BinarySIPair<"ni", 0x94, 0xEB54, null_frag, imm32zx8>;
 
   // Block AND.
   let mayLoad = 1, mayStore = 1 in
     defm NC : MemorySS<"nc", 0xD4, z_nc, z_nc_loop>;
let CCValues = 0xC, CompareZeroCCMask = 0x8 in {
     def OILF : BinaryRIL<"oilf", 0xC0D, or, GR32, uimm32>;
     def OIHF : BinaryRIL<"oihf", 0xC0C, or, GRH32, uimm32>;
   }
   def OILF64 : BinaryAliasRIL<or, GR64, imm64lf32>;
   def OIHF64 : BinaryAliasRIL<or, GR64, imm64hf32>;
 
   // ORs of memory.
   let CCValues = 0xC, CompareZeroCCMask = 0x8 in {
-    defm O  : BinaryRXPair<"o", 0x56, 0xE356, or, GR32, load, 4>;
-    def  OG : BinaryRXY<"og", 0xE381, or, GR64, load, 8>;
+    defm O  : BinaryRXPairAndPseudo<"o", 0x56, 0xE356, or, GR32, load, 4>;
+    defm OG : BinaryRXYAndPseudo<"og", 0xE381, or, GR64, load, 8>;
   }
 
   // OR to memory
   defm OI : BinarySIPair<"oi", 0x96, 0xEB56, null_frag, imm32zx8>;
 
   // Block OR.
   let mayLoad = 1, mayStore = 1 in
     defm OC : MemorySS<"oc", 0xD6, z_oc, z_oc_loop>;
let CCValues = 0xC, CompareZeroCCMask = 0x8 in {
     def XILF : BinaryRIL<"xilf", 0xC07, xor, GR32, uimm32>;
     def XIHF : BinaryRIL<"xihf", 0xC06, xor, GRH32, uimm32>;
   }
   def XILF64 : BinaryAliasRIL<xor, GR64, imm64lf32>;
   def XIHF64 : BinaryAliasRIL<xor, GR64, imm64hf32>;
 
   // XORs of memory.
   let CCValues = 0xC, CompareZeroCCMask = 0x8 in {
-    defm X  : BinaryRXPair<"x",0x57, 0xE357, xor, GR32, load, 4>;
-    def  XG : BinaryRXY<"xg", 0xE382, xor, GR64, load, 8>;
+    defm X  : BinaryRXPairAndPseudo<"x",0x57, 0xE357, xor, GR32, load, 4>;
+    defm XG : BinaryRXYAndPseudo<"xg", 0xE382, xor, GR64, load, 8>;
   }
 
   // XOR to memory
   defm XI : BinarySIPair<"xi", 0x97, 0xEB57, null_frag, imm32zx8>;
 
   // Block XOR.
   let mayLoad = 1, mayStore = 1 in
     defm XC : MemorySS<"xc", 0xD7, z_xc, z_xc_loop>;

lib/Target/SystemZ/SystemZPostRewrite.cpp

+//==---- SystemZPostRewrite.cpp - Select pseudos after RegAlloc ---*- C++ -*-=//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file contains a pass that is run immediately after VirtRegRewriter
+// but before MachineCopyPropagation. The purpose is to lower pseudos to
+// target instructions before any later pass might substitute a register for
+// another.
+//
+//===----------------------------------------------------------------------===//
+
+#include "SystemZ.h"
+#include "SystemZInstrInfo.h"
+#include "SystemZSubtarget.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+using namespace llvm;
+
+#define SYSTEMZ_POSTREWRITE_NAME "SystemZ Post Rewrite pass"
+
+#define DEBUG_TYPE "systemz-postrewrite"
+STATISTIC(MemFoldCopies, "Number of copies inserted before folded mem ops.");
+
+namespace llvm {
+  void initializeSystemZPostRewritePass(PassRegistry&);
+}
+
+namespace {
+
+class SystemZPostRewrite : public MachineFunctionPass {
+public:
+  static char ID;
+  SystemZPostRewrite() : MachineFunctionPass(ID) {
+    initializeSystemZPostRewritePass(*PassRegistry::getPassRegistry());
+  }
+
+  const SystemZInstrInfo *TII;
+
+  bool runOnMachineFunction(MachineFunction &Fn) override;
+
+  StringRef getPassName() const override { return SYSTEMZ_POSTREWRITE_NAME; }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesAll();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+
+private:
+  bool selectMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
+                MachineBasicBlock::iterator &NextMBBI);
+  bool selectMBB(MachineBasicBlock &MBB);
+};
+
+char SystemZPostRewrite::ID = 0;
+
+} // end anonymous namespace
+
+INITIALIZE_PASS(SystemZPostRewrite, "systemz-post-rewrite",
+                SYSTEMZ_POSTREWRITE_NAME, false, false)
+
+/// Returns an instance of the Post Rewrite pass.
+FunctionPass *llvm::createSystemZPostRewritePass(SystemZTargetMachine &TM) {
+  return new SystemZPostRewrite();
+}
+
+/// If MBBI references a pseudo instruction that should be selected here,
+/// do it and return true.  Otherwise return false.
+bool SystemZPostRewrite::selectMI(MachineBasicBlock &MBB,
+                                MachineBasicBlock::iterator MBBI,
+                                MachineBasicBlock::iterator &NextMBBI) {
+  MachineInstr &MI = *MBBI;
+  unsigned Opcode = MI.getOpcode();
+
+  // Note: If this could be done during regalloc in foldMemoryOperandImpl()
+  // while also updating the LiveIntervals, there would be no need for the
+  // MemFoldPseudo to begin with.
+  int TargetMemOpcode = SystemZ::getTargetMemOpcode(Opcode);
+  if (TargetMemOpcode != -1) {
+    MI.setDesc(TII->get(TargetMemOpcode));
+    MI.tieOperands(0, 1);
+    unsigned DstReg = MI.getOperand(0).getReg();
+    MachineOperand &SrcMO = MI.getOperand(1);
+    if (DstReg != SrcMO.getReg()) {
+      BuildMI(MBB, &MI, MI.getDebugLoc(), TII->get(SystemZ::COPY), DstReg)
+        .addReg(SrcMO.getReg());
+      SrcMO.setReg(DstReg);
+      MemFoldCopies++;
+    }
+    return true;
+  }
+
+  return false;
+}
+
+/// Iterate over the instructions in basic block MBB and select any
+/// pseudo instructions.  Return true if anything was modified.
+bool SystemZPostRewrite::selectMBB(MachineBasicBlock &MBB) {
+  bool Modified = false;
+
+  MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
+  while (MBBI != E) {
+    MachineBasicBlock::iterator NMBBI = std::next(MBBI);
+    Modified |= selectMI(MBB, MBBI, NMBBI);
+    MBBI = NMBBI;
+  }
+
+  return Modified;
+}
+
+bool SystemZPostRewrite::runOnMachineFunction(MachineFunction &MF) {
+  TII = static_cast<const SystemZInstrInfo *>(MF.getSubtarget().getInstrInfo());
+
+  bool Modified = false;
+  for (auto &MBB : MF)
+    Modified |= selectMBB(MBB);
+
+  return Modified;
+}
+

lib/Target/SystemZ/SystemZTargetMachine.cpp

createPostMachineScheduler(MachineSchedContext *C) const override {
     return new ScheduleDAGMI(C,
                              llvm::make_unique<SystemZPostRASchedStrategy>(C),
                              /*RemoveKillFlags=*/true);
   }
 
   void addIRPasses() override;
   bool addInstSelector() override;
   bool addILPOpts() override;
+  void addPostRewrite() override;
   void addPreSched2() override;
   void addPreEmitPass() override;
 };
 
 } // end anonymous namespace
 
 void SystemZPassConfig::addIRPasses() {
   if (getOptLevel() != CodeGenOpt::None) {
if (getOptLevel() != CodeGenOpt::None)
   return false;
 }
 
 bool SystemZPassConfig::addILPOpts() {
   addPass(&EarlyIfConverterID);
   return true;
 }
 
+void SystemZPassConfig::addPostRewrite() {
+  addPass(createSystemZPostRewritePass(getSystemZTargetMachine()));
+}
+
 void SystemZPassConfig::addPreSched2() {
+  // PostRewrite needs to be run at -O0 also (in which case addPostRewrite()
+  // is not called).
+  if (getOptLevel() == CodeGenOpt::None)
+    addPass(createSystemZPostRewritePass(getSystemZTargetMachine()));
+
   addPass(createSystemZExpandPseudoPass(getSystemZTargetMachine()));
 
   if (getOptLevel() != CodeGenOpt::None)
     addPass(&IfConverterID);
 }
 
 void SystemZPassConfig::addPreEmitPass() {
   // Do instruction shortening before compare elimination because some

lib/Target/X86/X86InstrInfo.h

public:
   /// specified operand(s).  If this is possible, the target should perform the
   /// folding and return true, otherwise it should return false.  If it folds
   /// the instruction, it is likely that the MachineInstruction the iterator
   /// references has been changed.
   MachineInstr *
   foldMemoryOperandImpl(MachineFunction &MF, MachineInstr &MI,
                         ArrayRef<unsigned> Ops,
                         MachineBasicBlock::iterator InsertPt, int FrameIndex,
-                        LiveIntervals *LIS = nullptr) const override;
+                        LiveIntervals *LIS = nullptr,
+                        VirtRegMap *VRM = nullptr) const override;
 
   /// foldMemoryOperand - Same as the previous version except it allows folding
   /// of any load and store from / to any address, not just from a specific
   /// stack slot.
   MachineInstr *foldMemoryOperandImpl(
       MachineFunction &MF, MachineInstr &MI, ArrayRef<unsigned> Ops,
       MachineBasicBlock::iterator InsertPt, MachineInstr &LoadMI,
       LiveIntervals *LIS = nullptr) const override;

lib/Target/X86/X86InstrInfo.cpp

if (PrintFailedFusing && !MI.isCopy())
     dbgs() << "We failed to fuse operand " << OpNum << " in " << MI;
   return nullptr;
 }
 
 MachineInstr *
 X86InstrInfo::foldMemoryOperandImpl(MachineFunction &MF, MachineInstr &MI,
                                     ArrayRef<unsigned> Ops,
                                     MachineBasicBlock::iterator InsertPt,
-                                    int FrameIndex, LiveIntervals *LIS) const {
+                                    int FrameIndex, LiveIntervals *LIS,
+                                    VirtRegMap *VRM) const {
   // Check switch flag
   if (NoFusing)
     return nullptr;
 
   // Avoid partial and undef register update stalls unless optimizing for size.
   if (!MF.getFunction().hasOptSize() &&
       (hasPartialRegUpdate(MI.getOpcode(), Subtarget, /*ForLoadFold*/true) ||
        shouldPreventUndefRegUpdateMemFold(MF, MI)))

test/CodeGen/SystemZ/and-01.ll

; CHECK: br %r14
   %and = and i32 %a, %b
   ret i32 %and
 }
 
 ; Check that ANDs of spilled values can use N rather than NR.
 define i32 @f12(i32 *%ptr0) {
 ; CHECK-LABEL: f12:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; _CHECK: n %r2, 16{{[04]}}(%r15)
+; CHECK: n %r2, 16{{[04]}}(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i32, i32 *%ptr0, i64 2
   %ptr2 = getelementptr i32, i32 *%ptr0, i64 4
   %ptr3 = getelementptr i32, i32 *%ptr0, i64 6
   %ptr4 = getelementptr i32, i32 *%ptr0, i64 8
   %ptr5 = getelementptr i32, i32 *%ptr0, i64 10
   %ptr6 = getelementptr i32, i32 *%ptr0, i64 12
   %ptr7 = getelementptr i32, i32 *%ptr0, i64 14

test/CodeGen/SystemZ/and-03.ll

; CHECK: br %r14
   %and = and i64 %a, %b
   ret i64 %and
 }
 
 ; Check that ANDs of spilled values can use NG rather than NGR.
 define i64 @f9(i64 *%ptr0) {
 ; CHECK-LABEL: f9:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: ng %r2, 160(%r15)
+; CHECK: ng %r2, 160(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i64, i64 *%ptr0, i64 2
   %ptr2 = getelementptr i64, i64 *%ptr0, i64 4
   %ptr3 = getelementptr i64, i64 *%ptr0, i64 6
   %ptr4 = getelementptr i64, i64 *%ptr0, i64 8
   %ptr5 = getelementptr i64, i64 *%ptr0, i64 10
   %ptr6 = getelementptr i64, i64 *%ptr0, i64 12
   %ptr7 = getelementptr i64, i64 *%ptr0, i64 14

test/CodeGen/SystemZ/int-add-02.ll

; CHECK: br %r14
   %add = add i32 %a, %b
   ret i32 %add
 }
 
 ; Check that additions of spilled values can use A rather than AR.
 define i32 @f12(i32 *%ptr0) {
 ; CHECK-LABEL: f12:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: a %r2, 16{{[04]}}(%r15)
+; CHECK: a %r2, 16{{[04]}}(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i32, i32 *%ptr0, i64 2
   %ptr2 = getelementptr i32, i32 *%ptr0, i64 4
   %ptr3 = getelementptr i32, i32 *%ptr0, i64 6
   %ptr4 = getelementptr i32, i32 *%ptr0, i64 8
   %ptr5 = getelementptr i32, i32 *%ptr0, i64 10
   %ptr6 = getelementptr i32, i32 *%ptr0, i64 12
   %ptr7 = getelementptr i32, i32 *%ptr0, i64 14

test/CodeGen/SystemZ/int-add-05.ll

 }
 
 ; Check that additions of spilled values can use AG rather than AGR.
 ; Note: Z196 is suboptimal with one unfolded reload.
 define i64 @f9(i64 *%ptr0) {
 ; CHECK-LABEL: f9:
 ; CHECK: brasl %r14, foo@PLT
 ; Z10:  ag %r2, 168(%r15)
-; TODO: This test is temporarily failing due to D60888.
-; Z196_: ag %r0, 168(%r15)
+; Z196: ag %r0, 168(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i64, i64 *%ptr0, i64 2
   %ptr2 = getelementptr i64, i64 *%ptr0, i64 4
   %ptr3 = getelementptr i64, i64 *%ptr0, i64 6
   %ptr4 = getelementptr i64, i64 *%ptr0, i64 8
   %ptr5 = getelementptr i64, i64 *%ptr0, i64 10
   %ptr6 = getelementptr i64, i64 *%ptr0, i64 12
   %ptr7 = getelementptr i64, i64 *%ptr0, i64 14

test/CodeGen/SystemZ/int-add-08.ll

; CHECK: br %r14
   ret void
 }
 
 ; Check that additions of spilled values can use ALG and ALCG rather than
 ; ALGR and ALCGR.
 define void @f8(i128 *%ptr0) {
 ; CHECK-LABEL: f8:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: alg {{%r[0-9]+}}, {{[0-9]+}}(%r15)
+; CHECK: alg {{%r[0-9]+}}, {{[0-9]+}}(%r15)
 ; CHECK: alcg {{%r[0-9]+}}, {{[0-9]+}}(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i128, i128 *%ptr0, i128 2
   %ptr2 = getelementptr i128, i128 *%ptr0, i128 4
   %ptr3 = getelementptr i128, i128 *%ptr0, i128 6
   %ptr4 = getelementptr i128, i128 *%ptr0, i128 8
   %ptr5 = getelementptr i128, i128 *%ptr0, i128 10
 

test/CodeGen/SystemZ/int-sub-01.ll

; CHECK: br %r14
   %sub = sub i32 %a, %b
   ret i32 %sub
 }
 
 ; Check that subtractions of spilled values can use S rather than SR.
 define i32 @f12(i32 *%ptr0) {
 ; CHECK-LABEL: f12:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: s %r2, 16{{[04]}}(%r15)
+; CHECK: s %r2, 16{{[04]}}(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i32, i32 *%ptr0, i64 2
   %ptr2 = getelementptr i32, i32 *%ptr0, i64 4
   %ptr3 = getelementptr i32, i32 *%ptr0, i64 6
   %ptr4 = getelementptr i32, i32 *%ptr0, i64 8
   %ptr5 = getelementptr i32, i32 *%ptr0, i64 10
   %ptr6 = getelementptr i32, i32 *%ptr0, i64 12
   %ptr7 = getelementptr i32, i32 *%ptr0, i64 14

test/CodeGen/SystemZ/int-sub-04.ll

; CHECK: br %r14
   %sub = sub i64 %a, %b
   ret i64 %sub
 }
 
 ; Check that subtractions of spilled values can use SG rather than SGR.
 define i64 @f9(i64 *%ptr0) {
 ; CHECK-LABEL: f9:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: sg %r2, 160(%r15)
+; CHECK: sg %r2, 160(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i64, i64 *%ptr0, i64 2
   %ptr2 = getelementptr i64, i64 *%ptr0, i64 4
   %ptr3 = getelementptr i64, i64 *%ptr0, i64 6
   %ptr4 = getelementptr i64, i64 *%ptr0, i64 8
   %ptr5 = getelementptr i64, i64 *%ptr0, i64 10
   %ptr6 = getelementptr i64, i64 *%ptr0, i64 12
   %ptr7 = getelementptr i64, i64 *%ptr0, i64 14

test/CodeGen/SystemZ/int-sub-05.ll

; CHECK: br %r14
   ret void
 }
 
 ; Check that subtractions of spilled values can use SLG and SLBG rather than
 ; SLGR and SLBGR.
 define void @f8(i128 *%ptr0) {
 ; CHECK-LABEL: f8:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: slg {{%r[0-9]+}}, {{[0-9]+}}(%r15)
+; CHECK: slg {{%r[0-9]+}}, {{[0-9]+}}(%r15)
 ; CHECK: slbg {{%r[0-9]+}}, {{[0-9]+}}(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i128, i128 *%ptr0, i128 2
   %ptr2 = getelementptr i128, i128 *%ptr0, i128 4
   %ptr3 = getelementptr i128, i128 *%ptr0, i128 6
   %ptr4 = getelementptr i128, i128 *%ptr0, i128 8
   %ptr5 = getelementptr i128, i128 *%ptr0, i128 10
 

test/CodeGen/SystemZ/or-01.ll

; CHECK: br %r14
   %or = or i32 %a, %b
   ret i32 %or
 }
 
 ; Check that ORs of spilled values can use O rather than OR.
 define i32 @f12(i32 *%ptr0) {
 ; CHECK-LABEL: f12:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: o %r2, 16{{[04]}}(%r15)
+; CHECK: o %r2, 16{{[04]}}(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i32, i32 *%ptr0, i64 2
   %ptr2 = getelementptr i32, i32 *%ptr0, i64 4
   %ptr3 = getelementptr i32, i32 *%ptr0, i64 6
   %ptr4 = getelementptr i32, i32 *%ptr0, i64 8
   %ptr5 = getelementptr i32, i32 *%ptr0, i64 10
   %ptr6 = getelementptr i32, i32 *%ptr0, i64 12
   %ptr7 = getelementptr i32, i32 *%ptr0, i64 14

test/CodeGen/SystemZ/or-03.ll

; CHECK: br %r14
   %or = or i64 %a, %b
   ret i64 %or
 }
 
 ; Check that ORs of spilled values can use OG rather than OGR.
 define i64 @f9(i64 *%ptr0) {
 ; CHECK-LABEL: f9:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: og %r2, 160(%r15)
+; CHECK: og %r2, 160(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i64, i64 *%ptr0, i64 2
   %ptr2 = getelementptr i64, i64 *%ptr0, i64 4
   %ptr3 = getelementptr i64, i64 *%ptr0, i64 6
   %ptr4 = getelementptr i64, i64 *%ptr0, i64 8
   %ptr5 = getelementptr i64, i64 *%ptr0, i64 10
   %ptr6 = getelementptr i64, i64 *%ptr0, i64 12
   %ptr7 = getelementptr i64, i64 *%ptr0, i64 14

test/CodeGen/SystemZ/xor-01.ll

; CHECK: br %r14
   %xor = xor i32 %a, %b
   ret i32 %xor
 }
 
 ; Check that XORs of spilled values can use X rather than XR.
 define i32 @f12(i32 *%ptr0) {
 ; CHECK-LABEL: f12:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: x %r2, 16{{[04]}}(%r15)
+; CHECK: x %r2, 16{{[04]}}(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i32, i32 *%ptr0, i64 2
   %ptr2 = getelementptr i32, i32 *%ptr0, i64 4
   %ptr3 = getelementptr i32, i32 *%ptr0, i64 6
   %ptr4 = getelementptr i32, i32 *%ptr0, i64 8
   %ptr5 = getelementptr i32, i32 *%ptr0, i64 10
   %ptr6 = getelementptr i32, i32 *%ptr0, i64 12
   %ptr7 = getelementptr i32, i32 *%ptr0, i64 14

test/CodeGen/SystemZ/xor-03.ll

; CHECK: br %r14
   %xor = xor i64 %a, %b
   ret i64 %xor
 }
 
 ; Check that XORs of spilled values can use OG rather than OGR.
 define i64 @f9(i64 *%ptr0) {
 ; CHECK-LABEL: f9:
 ; CHECK: brasl %r14, foo@PLT
-; TODO: This test is temporarily failing due to D60888.
-; CHECK_: xg %r2, 160(%r15)
+; CHECK: xg %r2, 160(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i64, i64 *%ptr0, i64 2
   %ptr2 = getelementptr i64, i64 *%ptr0, i64 4
   %ptr3 = getelementptr i64, i64 *%ptr0, i64 6
   %ptr4 = getelementptr i64, i64 *%ptr0, i64 8
   %ptr5 = getelementptr i64, i64 *%ptr0, i64 10
   %ptr6 = getelementptr i64, i64 *%ptr0, i64 12
   %ptr7 = getelementptr i64, i64 *%ptr0, i64 14