[SystemZ] Improve codegen for memset

[jonpa]

Summary

Memset with a constant length was implemented with a single store followed by series of MVC:s. This patch changes this so that one store of the byte is emitted for each MVC, which avoids data dependencies between the MVCs. An MVI/STC + MVC(len-1) is done for each block.

(A few somewhat unfortunate changes in memset-01.ll (extra STC/MVC) - hopefully not important. E.g. STC;MVC(256) -> STC;MVC(255);STC)

Memset with a variable is now also handled without a libcall. Since the byte is first stored and then MVC is used from that address, a length of two must now be subtracted instead of one for the loop. There needs to be an extra check for the case of length one is handled in a special block with just a single MVI/STC (this is per what GCC does).

I put the MBB for doing the memset of length 1 at the end of MF and also set a low probability on that edge since otherwise the Branch Probability Basic Block Placement pass would move it back into the sequence. When that happens the common path (length > 1) then involves a taken branch across the length-1 block which should be suboptimal.

I see that GCC prefetches these loops 4 iterations ahead, while clang does 3...

Side note: Unfortunately it seems that memset loops suffer a bit currently from poor register coalescing in cases where the address is used after the memset. This is not just for this patch, but also for the memset-0 case (XC loop). I guess that should be handled if possible...

char fun(char *dst, char byte, int len)
{
  memset(dst, byte, len);
  return dst[14];
}

=>

fun:                                    # @fun
# %bb.0:                                # %entry
	aghi	%r5, -2
	cgije	%r5, -2, .LBB0_6
# %bb.1:                                # %entry
	cgije	%r5, -1, .LBB0_7
# %bb.2:                                # %entry
	srlg	%r0, %r5, 8
	lgr	%r1, %r2                ##### <<<<<<<<<<<<<<<
	cgije	%r0, 0, .LBB0_5
# %bb.3:
	lgr	%r1, %r2                ##### <<<<<<<<<<<<<<< REDUNDANT
.LBB0_4:                                # %entry
                                        # =>This Inner Loop Header: Depth=1
	pfd	2, 768(%r1)
	stc	%r4, 0(%r1)
	mvc	1(255,%r1), 0(%r1)
	la	%r1, 256(%r1)
	brctg	%r0, .LBB0_4
...

So far, I know that Early Machine LICM inserts a pre-header for the loop since there is none (a preheader can only have one successor: the loop header). This causes the PHI node using the original address register in the loop to have a different predecessor than the one after the loop (EXRL block), so there are two COPYs inserted into different MBBs by the PHI lowering pass. Not sure yet why this doesn't get coalesced in the memset cases, when it is handled for memcpy...

[jonpa]

https://reviews.llvm.org/D112065 is a proposed patch for the register coalescing problem.

[jonpa]

Refactored so that MEMSET_MVC and associated pseudos only carry one address. NFC.

[uweigand]

I think it would be a bit simpler and clearer to handle the Src vs. Dest address special case for memset in a single place at the beginning. Something along the lines of:

if (!isMemSet)
{
  DestBase = earlyUseOperand(MI.getOperand(0));
  DestDisp = MI.getOperand(1).getImm();
  SrcBase = earlyUseOperand(MI.getOperand(2));
  SrcDisp = MI.getOperand(3).getImm();
}
else
{
  SrcBase = earlyUseOperand(MI.getOperand(0));
  SrcDisp = MI.getOperand(1).getImm();
  DestBase = SrcBase;
  DestDisp = SrcDisp + 1;
  foldDisplIfNeeded(DestBase, DestDisp);
}

There should be no need for any subsequent +1 displacement arithmetic then.

[jonpa]

Patch updated per suggenstion.

I tried to also avoid folding just Dest so that two registers would not be used when SrcDisp is 4095, like:

+  auto foldDisplIfNeeded = [&](MachineOperand &Base, uint64_t &Disp,
+                               bool Force = false) -> void {
+    if (Disp > 0xfff || Force) {
...
+      if (DestDisp > 0xfff) {
+        // Make sure only one base register is used.
+        foldDisplIfNeeded(SrcBase, SrcDisp, true/*Force*/);
+        DestBase = SrcBase;
+        DestDisp = 1;
+      }

This however did not change benchmarks at all, so I removed it (it did however improve memset-01.ll:f17,f18...). I also tried generalizing this idea to all operations, but again there was no cases of this in benchmarks.

Also some other minor fixing:

• Changed the variable names in insertMemMemOp() to avoid using the same names already used (DestBase, SrcBase, ...)
• add ByteMO directly also with STC in insertMemMemOp() and make sure to call earlyUseOperand() to clear any kill flag.
• For the PFD I decided it was simplest to subtract IsMemset compared to using both ThisSrcReg and SrcDisp.

[uweigand]

LGTM, just a few minor nits inline. We also should do a performance validation run before committing this.

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

7881

Minor nit: this should use !isUInt<12>(Disp) like elsewhere.

7980

Was this line moved deliberately?

llvm/test/CodeGen/SystemZ/tail-call-mem-intrinsics.ll

12

Maybe we can still find a way to test emission of the call? E.g. using a volatile pointer?

[jonpa]

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

7980

yes, I thought that looked a little better, but it doesn't matter that much...

[uweigand]

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

7980

OK, that's fine with me then.

[jonpa]

llvm/test/CodeGen/SystemZ/tail-call-mem-intrinsics.ll

12

ah, yes - the fourth argument is actually 'isvolatile'...

[jonpa]

Updated per review.

[uweigand]

LGTM, thanks!

llvm/lib/Target/SystemZ/SystemZISelLowering.h

enum NodeType : unsigned {
   NC,
   OC,
   XC,
 
   // Use CLC to compare two blocks of memory, with the same comments
   // as for MVC.
   CLC,
 
+  // Use MVC to set a block of memory after storing the first byte.
+  MEMSET_MVC,
+
   // Use an MVST-based sequence to implement stpcpy().
   STPCPY,
 
   // Use a CLST-based sequence to implement strcmp().  The two input operands
   // are the addresses of the strings to compare.
   STRCMP,
 
   // Use an SRST-based sequence to search a block of memory.  The first
private:
   MachineBasicBlock *emitAtomicLoadMinMax(MachineInstr &MI,
                                           MachineBasicBlock *MBB,
                                           unsigned CompareOpcode,
                                           unsigned KeepOldMask,
                                           unsigned BitSize) const;
   MachineBasicBlock *emitAtomicCmpSwapW(MachineInstr &MI,
                                         MachineBasicBlock *BB) const;
   MachineBasicBlock *emitMemMemWrapper(MachineInstr &MI, MachineBasicBlock *BB,
-                                       unsigned Opcode) const;
+                                       unsigned Opcode,
+                                       bool IsMemset = false) const;
   MachineBasicBlock *emitStringWrapper(MachineInstr &MI, MachineBasicBlock *BB,
                                        unsigned Opcode) const;
   MachineBasicBlock *emitTransactionBegin(MachineInstr &MI,
                                           MachineBasicBlock *MBB,
                                           unsigned Opcode, bool NoFloat) const;
   MachineBasicBlock *emitLoadAndTestCmp0(MachineInstr &MI,
                                          MachineBasicBlock *MBB,
                                          unsigned Opcode) const;

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

switch ((SystemZISD::NodeType)Opcode) {
     OPCODE(ADDCARRY);
     OPCODE(SUBCARRY);
     OPCODE(GET_CCMASK);
     OPCODE(MVC);
     OPCODE(NC);
     OPCODE(OC);
     OPCODE(XC);
     OPCODE(CLC);
+    OPCODE(MEMSET_MVC);
     OPCODE(STPCPY);
     OPCODE(STRCMP);
     OPCODE(SEARCH_STRING);
     OPCODE(IPM);
     OPCODE(MEMBARRIER);
     OPCODE(TBEGIN);
     OPCODE(TBEGIN_NOFLOAT);
     OPCODE(TEND);
MachineBasicBlock *SystemZTargetLowering::emitExt128(MachineInstr &MI,
   }
   BuildMI(*MBB, MI, DL, TII->get(TargetOpcode::INSERT_SUBREG), Dest)
     .addReg(In128).addReg(Src).addImm(SystemZ::subreg_l64);
 
   MI.eraseFromParent();
   return MBB;
 }
 
-MachineBasicBlock *SystemZTargetLowering::emitMemMemWrapper(
-    MachineInstr &MI, MachineBasicBlock *MBB, unsigned Opcode) const {
+MachineBasicBlock *
+SystemZTargetLowering::emitMemMemWrapper(MachineInstr &MI,
+                                         MachineBasicBlock *MBB,
+                                         unsigned Opcode, bool IsMemset) const {
   MachineFunction &MF = *MBB->getParent();
   const SystemZInstrInfo *TII =
       static_cast<const SystemZInstrInfo *>(Subtarget.getInstrInfo());
   MachineRegisterInfo &MRI = MF.getRegInfo();
   DebugLoc DL = MI.getDebugLoc();
 
   MachineOperand DestBase = earlyUseOperand(MI.getOperand(0));
   uint64_t DestDisp = MI.getOperand(1).getImm();
-  MachineOperand SrcBase = earlyUseOperand(MI.getOperand(2));
-  uint64_t SrcDisp = MI.getOperand(3).getImm();
-  MachineOperand &LengthMO = MI.getOperand(4);
+  MachineOperand SrcBase = MachineOperand::CreateReg(0U, false);
+  uint64_t SrcDisp;
+
+  // Fold the displacement Disp if it is out of range.
+  auto foldDisplIfNeeded = [&](MachineOperand &Base, uint64_t &Disp) -> void {
+    if (!isUInt<12>(Disp)) {
+      Register Reg = MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
+      unsigned Opcode = TII->getOpcodeForOffset(SystemZ::LA, Disp);
+      BuildMI(*MI.getParent(), MI, MI.getDebugLoc(), TII->get(Opcode), Reg)
+        .add(Base).addImm(Disp).addReg(0);
+      Base = MachineOperand::CreateReg(Reg, false);
+      Disp = 0;
+    }
+  };
+
+  if (!IsMemset) {
+    SrcBase = earlyUseOperand(MI.getOperand(2));
+    SrcDisp = MI.getOperand(3).getImm();
+  } else {
+    SrcBase = DestBase;
+    SrcDisp = DestDisp++;
+    foldDisplIfNeeded(DestBase, DestDisp);
+  }
+
+  MachineOperand &LengthMO = MI.getOperand(IsMemset ? 2 : 4);
   bool IsImmForm = LengthMO.isImm();
   bool IsRegForm = !IsImmForm;
 
+  // Build and insert one Opcode of Length, with special treatment for memset.
+  auto insertMemMemOp = [&](MachineBasicBlock *InsMBB,
+                            MachineBasicBlock::iterator InsPos,
+                            MachineOperand DBase, uint64_t DDisp,
+                            MachineOperand SBase, uint64_t SDisp,
+                            unsigned Length) -> void {
+    assert(Length > 0 && Length <= 256 && "Building memory op with bad length.");
+    if (IsMemset) {
+      MachineOperand ByteMO = earlyUseOperand(MI.getOperand(3));
+      if (ByteMO.isImm())
+        BuildMI(*InsMBB, InsPos, DL, TII->get(SystemZ::MVI))
+          .add(SBase).addImm(SDisp).add(ByteMO);
+      else
+        BuildMI(*InsMBB, InsPos, DL, TII->get(SystemZ::STC))
+          .add(ByteMO).add(SBase).addImm(SDisp).addReg(0);
+      if (--Length == 0)
+        return;
+    }
+    BuildMI(*MBB, InsPos, DL, TII->get(Opcode))
+      .add(DBase).addImm(DDisp).addImm(Length)
+      .add(SBase).addImm(SDisp)
+      .setMemRefs(MI.memoperands());
+  };
+
   bool NeedsLoop = false;
   uint64_t ImmLength = 0;
-  Register LenMinus1Reg = SystemZ::NoRegister;
+  Register LenAdjReg = SystemZ::NoRegister;
   if (IsImmForm) {
     ImmLength = LengthMO.getImm();
-    ImmLength++; // Add back the '1' subtracted originally.
+    ImmLength += IsMemset ? 2 : 1; // Add back the subtracted adjustment.
     if (ImmLength == 0) {
       MI.eraseFromParent();
       return MBB;
     }
     if (Opcode == SystemZ::CLC) {
       if (ImmLength > 3 * 256)
         // A two-CLC sequence is a clear win over a loop, not least because
         // it needs only one branch.  A three-CLC sequence needs the same
         // number of branches as a loop (i.e. 2), but is shorter.  That
         // brings us to lengths greater than 768 bytes.  It seems relatively
         // likely that a difference will be found within the first 768 bytes,
         // so we just optimize for the smallest number of branch
         // instructions, in order to avoid polluting the prediction buffer
         // too much.
         NeedsLoop = true;
     } else if (ImmLength > 6 * 256)
       // The heuristic we use is to prefer loops for anything that would
       // require 7 or more MVCs.  With these kinds of sizes there isn't much
       // to choose between straight-line code and looping code, since the
       // time will be dominated by the MVCs themselves.
       NeedsLoop = true;
   } else {
     NeedsLoop = true;
-    LenMinus1Reg = LengthMO.getReg();
+    LenAdjReg = LengthMO.getReg();
   }
 
   // When generating more than one CLC, all but the last will need to
   // branch to the end when a difference is found.
   MachineBasicBlock *EndMBB =
       (Opcode == SystemZ::CLC && (ImmLength > 256 || NeedsLoop)
            ? SystemZ::splitBlockAfter(MI, MBB)
            : nullptr);
 
   if (NeedsLoop) {
     Register StartCountReg =
       MRI.createVirtualRegister(&SystemZ::GR64BitRegClass);
     if (IsImmForm) {
       TII->loadImmediate(*MBB, MI, StartCountReg, ImmLength / 256);
       ImmLength &= 255;
     } else {
       BuildMI(*MBB, MI, DL, TII->get(SystemZ::SRLG), StartCountReg)
-        .addReg(LenMinus1Reg)
+        .addReg(LenAdjReg)
         .addReg(0)
         .addImm(8);
     }
 
+    bool HaveSingleBase = DestBase.isIdenticalTo(SrcBase);
     auto loadZeroAddress = [&]() -> MachineOperand {
       Register Reg = MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
       BuildMI(*MBB, MI, DL, TII->get(SystemZ::LGHI), Reg).addImm(0);
       return MachineOperand::CreateReg(Reg, false);
     };
-    bool HaveSingleBase = DestBase.isIdenticalTo(SrcBase);
     if (DestBase.isReg() && DestBase.getReg() == SystemZ::NoRegister)
       DestBase = loadZeroAddress();
     if (SrcBase.isReg() && SrcBase.getReg() == SystemZ::NoRegister)
       SrcBase = HaveSingleBase ? DestBase : loadZeroAddress();
 
     MachineBasicBlock *StartMBB = nullptr;
     MachineBasicBlock *LoopMBB = nullptr;
     MachineBasicBlock *NextMBB = nullptr;
if (NeedsLoop) {
     if (IsRegForm) {
       AllDoneMBB = SystemZ::splitBlockBefore(MI, MBB);
       StartMBB = SystemZ::emitBlockAfter(MBB);
       LoopMBB = SystemZ::emitBlockAfter(StartMBB);
       NextMBB = (EndMBB ? SystemZ::emitBlockAfter(LoopMBB) : LoopMBB);
       DoneMBB = SystemZ::emitBlockAfter(NextMBB);
 
       //  MBB:
-      //   # Jump to AllDoneMBB if LenMinus1Reg is -1, or fall thru to StartMBB.
+      //   # Jump to AllDoneMBB if LenAdjReg means 0, or fall thru to StartMBB.
       BuildMI(MBB, DL, TII->get(SystemZ::CGHI))
-        .addReg(LenMinus1Reg).addImm(-1);
+        .addReg(LenAdjReg).addImm(IsMemset ? -2 : -1);
       BuildMI(MBB, DL, TII->get(SystemZ::BRC))
         .addImm(SystemZ::CCMASK_ICMP).addImm(SystemZ::CCMASK_CMP_EQ)
         .addMBB(AllDoneMBB);
       MBB->addSuccessor(AllDoneMBB);
+      if (!IsMemset)
         MBB->addSuccessor(StartMBB);
+      else {
+        // MemsetOneCheckMBB:
+        // # Jump to MemsetOneMBB for a memset of length 1, or
+        // # fall thru to StartMBB.
+        MachineBasicBlock *MemsetOneCheckMBB = SystemZ::emitBlockAfter(MBB);
+        MachineBasicBlock *MemsetOneMBB = SystemZ::emitBlockAfter(&*MF.rbegin());
+        MBB->addSuccessor(MemsetOneCheckMBB);
+        MBB = MemsetOneCheckMBB;
+        BuildMI(MBB, DL, TII->get(SystemZ::CGHI))
+          .addReg(LenAdjReg).addImm(-1);
+        BuildMI(MBB, DL, TII->get(SystemZ::BRC))
+          .addImm(SystemZ::CCMASK_ICMP).addImm(SystemZ::CCMASK_CMP_EQ)
+          .addMBB(MemsetOneMBB);
+        MBB->addSuccessor(MemsetOneMBB, {10, 100});
+        MBB->addSuccessor(StartMBB, {90, 100});
+
+        // MemsetOneMBB:
+        // # Jump back to AllDoneMBB after a single MVI or STC.
+        MBB = MemsetOneMBB;
+        insertMemMemOp(MBB, MBB->end(),
+                       MachineOperand::CreateReg(StartDestReg, false), DestDisp,
+                       MachineOperand::CreateReg(StartSrcReg, false), SrcDisp,
+                       1);
+        BuildMI(MBB, DL, TII->get(SystemZ::J)).addMBB(AllDoneMBB);
+        MBB->addSuccessor(AllDoneMBB);
+      }
 
       // StartMBB:
       // # Jump to DoneMBB if %StartCountReg is zero, or fall through to LoopMBB.
       MBB = StartMBB;
       BuildMI(MBB, DL, TII->get(SystemZ::CGHI))
         .addReg(StartCountReg).addImm(0);
       BuildMI(MBB, DL, TII->get(SystemZ::BRC))
         .addImm(SystemZ::CCMASK_ICMP).addImm(SystemZ::CCMASK_CMP_EQ)
if (!HaveSingleBase)
         .addReg(StartSrcReg).addMBB(StartMBB)
         .addReg(NextSrcReg).addMBB(NextMBB);
     BuildMI(MBB, DL, TII->get(SystemZ::PHI), ThisCountReg)
       .addReg(StartCountReg).addMBB(StartMBB)
       .addReg(NextCountReg).addMBB(NextMBB);
     if (Opcode == SystemZ::MVC)
       BuildMI(MBB, DL, TII->get(SystemZ::PFD))
         .addImm(SystemZ::PFD_WRITE)
-        .addReg(ThisDestReg).addImm(DestDisp + 768).addReg(0);
-    BuildMI(MBB, DL, TII->get(Opcode))
-      .addReg(ThisDestReg).addImm(DestDisp).addImm(256)
-      .addReg(ThisSrcReg).addImm(SrcDisp);
+        .addReg(ThisDestReg).addImm(DestDisp - IsMemset + 768).addReg(0);
+    insertMemMemOp(MBB, MBB->end(),
+                   MachineOperand::CreateReg(ThisDestReg, false), DestDisp,
+                   MachineOperand::CreateReg(ThisSrcReg, false), SrcDisp, 256);
     if (EndMBB) {
       BuildMI(MBB, DL, TII->get(SystemZ::BRC))
         .addImm(SystemZ::CCMASK_ICMP).addImm(SystemZ::CCMASK_CMP_NE)
         .addMBB(EndMBB);
       MBB->addSuccessor(EndMBB);
       MBB->addSuccessor(NextMBB);
     }
 
if (NeedsLoop) {
     MBB->addSuccessor(DoneMBB);
 
     MBB = DoneMBB;
     if (IsRegForm) {
       // DoneMBB:
       // # Make PHIs for RemDestReg/RemSrcReg as the loop may or may not run.
       // # Use EXecute Relative Long for the remainder of the bytes. The target
       //   instruction of the EXRL will have a length field of 1 since 0 is an
-      //   illegal value. The number of bytes processed becomes (%LenMinus1Reg &
+      //   illegal value. The number of bytes processed becomes (%LenAdjReg &
       //   0xff) + 1.
       // # Fall through to AllDoneMBB.
       Register RemSrcReg  = MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
       Register RemDestReg = HaveSingleBase ? RemSrcReg
         : MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
       BuildMI(MBB, DL, TII->get(SystemZ::PHI), RemDestReg)
         .addReg(StartDestReg).addMBB(StartMBB)
         .addReg(NextDestReg).addMBB(NextMBB);
       if (!HaveSingleBase)
         BuildMI(MBB, DL, TII->get(SystemZ::PHI), RemSrcReg)
           .addReg(StartSrcReg).addMBB(StartMBB)
           .addReg(NextSrcReg).addMBB(NextMBB);
+      if (IsMemset)
+        insertMemMemOp(MBB, MBB->end(),
+                       MachineOperand::CreateReg(RemDestReg, false), DestDisp,
+                       MachineOperand::CreateReg(RemSrcReg, false), SrcDisp, 1);
       MachineInstrBuilder EXRL_MIB =
         BuildMI(MBB, DL, TII->get(SystemZ::EXRL_Pseudo))
           .addImm(Opcode)
-          .addReg(LenMinus1Reg)
+          .addReg(LenAdjReg)
           .addReg(RemDestReg).addImm(DestDisp)
           .addReg(RemSrcReg).addImm(SrcDisp);
       MBB->addSuccessor(AllDoneMBB);
       MBB = AllDoneMBB;
       if (EndMBB) {
         EXRL_MIB.addReg(SystemZ::CC, RegState::ImplicitDefine);
         MBB->addLiveIn(SystemZ::CC);
       }
     }
   }
 
   // Handle any remaining bytes with straight-line code.
   while (ImmLength > 0) {
     uint64_t ThisLength = std::min(ImmLength, uint64_t(256));
     // The previous iteration might have created out-of-range displacements.
-    // Apply them using LAY if so.
-    if (!isUInt<12>(DestDisp)) {
-      Register Reg = MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
-      BuildMI(*MBB, MI, MI.getDebugLoc(), TII->get(SystemZ::LAY), Reg)
-          .add(DestBase)
-          .addImm(DestDisp)
-          .addReg(0);
-      DestBase = MachineOperand::CreateReg(Reg, false);
-      DestDisp = 0;
-    }
-    if (!isUInt<12>(SrcDisp)) {
-      Register Reg = MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
-      BuildMI(*MBB, MI, MI.getDebugLoc(), TII->get(SystemZ::LAY), Reg)
-          .add(SrcBase)
-          .addImm(SrcDisp)
-          .addReg(0);
-      SrcBase = MachineOperand::CreateReg(Reg, false);
-      SrcDisp = 0;
-    }
-    BuildMI(*MBB, MI, DL, TII->get(Opcode))
-        .add(DestBase)
-        .addImm(DestDisp)
-        .addImm(ThisLength)
-        .add(SrcBase)
-        .addImm(SrcDisp)
-        .setMemRefs(MI.memoperands());
+    // Apply them using LA/LAY if so.
+    foldDisplIfNeeded(DestBase, DestDisp);
+    foldDisplIfNeeded(SrcBase, SrcDisp);
+    insertMemMemOp(MBB, MI, DestBase, DestDisp, SrcBase, SrcDisp, ThisLength);
     DestDisp += ThisLength;
     SrcDisp += ThisLength;
     ImmLength -= ThisLength;
     // If there's another CLC to go, branch to the end if a difference
     // was found.
     if (EndMBB && ImmLength > 0) {
       MachineBasicBlock *NextMBB = SystemZ::splitBlockBefore(MI, MBB);
       BuildMI(MBB, DL, TII->get(SystemZ::BRC))
MachineBasicBlock *SystemZTargetLowering::EmitInstrWithCustomInserter(
   case SystemZ::OCImm:
     return emitMemMemWrapper(MI, MBB, SystemZ::OC);
   case SystemZ::XCImm:
   case SystemZ::XCReg:
     return emitMemMemWrapper(MI, MBB, SystemZ::XC);
   case SystemZ::CLCImm:
   case SystemZ::CLCReg:
     return emitMemMemWrapper(MI, MBB, SystemZ::CLC);
+  case SystemZ::MemsetImmImm:
+  case SystemZ::MemsetImmReg:
+  case SystemZ::MemsetRegImm:
+  case SystemZ::MemsetRegReg:
+    return emitMemMemWrapper(MI, MBB, SystemZ::MVC, true/*IsMemset*/);
   case SystemZ::CLSTLoop:
     return emitStringWrapper(MI, MBB, SystemZ::CLST);
   case SystemZ::MVSTLoop:
     return emitStringWrapper(MI, MBB, SystemZ::MVST);
   case SystemZ::SRSTLoop:
     return emitStringWrapper(MI, MBB, SystemZ::SRST);
   case SystemZ::TBEGIN:
     return emitTransactionBegin(MI, MBB, SystemZ::TBEGIN, false);

llvm/lib/Target/SystemZ/SystemZInstrFormats.td

 // An alias of a RotateSelectRIEf, but with different register sizes.
 class RotateSelectAliasRIEf<RegisterOperand cls1, RegisterOperand cls2>
   : Alias<6, (outs cls1:$R1),
           (ins cls1:$R1src, cls2:$R2, imm32zx8:$I3, imm32zx8:$I4,
                imm32zx6:$I5), []> {
   let Constraints = "$R1 = $R1src";
 }
 
+class MemsetPseudo<DAGOperand lenop, DAGOperand byteop>
+  : Pseudo<(outs), (ins bdaddr12only:$dest, lenop:$length, byteop:$B),
+           [(z_memset_mvc bdaddr12only:$dest, lenop:$length, byteop:$B)]> {
+  let Defs = [CC];
+  let mayLoad = 1;
+  let mayStore = 1;
+  let usesCustomInserter = 1;
+  let hasNoSchedulingInfo = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // Multiclasses that emit both real and pseudo instructions
 //===----------------------------------------------------------------------===//
 
 multiclass BinaryRXYAndPseudo<string mnemonic, bits<16> opcode,
                               SDPatternOperator operator, RegisterOperand cls,
                               SDPatternOperator load, bits<5> bytes,
                               AddressingMode mode = bdxaddr20only> {

llvm/lib/Target/SystemZ/SystemZInstrInfo.td

 let mayLoad = 1, mayStore = 1 in
   defm MVC : MemorySS<"mvc", 0xD2, z_mvc>;
 let mayLoad = 1, mayStore = 1, Defs = [CC] in {
   def MVCL  : SideEffectBinaryMemMemRR<"mvcl", 0x0E, GR128, GR128>;
   def MVCLE : SideEffectTernaryMemMemRS<"mvcle", 0xA8, GR128, GR128>;
   def MVCLU : SideEffectTernaryMemMemRSY<"mvclu", 0xEB8E, GR128, GR128>;
 }
 
+// Memset[Length][Byte] pseudos.
+def MemsetImmImm : MemsetPseudo<imm64, imm32zx8trunc>;
+def MemsetImmReg : MemsetPseudo<imm64, GR32>;
+def MemsetRegImm : MemsetPseudo<ADDR64, imm32zx8trunc>;
+def MemsetRegReg : MemsetPseudo<ADDR64, GR32>;
+
 // Move right.
 let Predicates = [FeatureMiscellaneousExtensions3],
     mayLoad = 1, mayStore = 1, Uses = [R0L] in
   def MVCRL : SideEffectBinarySSE<"mvcrl", 0xE50A>;
 
 // String moves.
 let mayLoad = 1, mayStore = 1, Defs = [CC] in
   defm MVST : StringRRE<"mvst", 0xB255, z_stpcpy>;

llvm/lib/Target/SystemZ/SystemZOperators.td

def SDT_ZMemMemLength       : SDTypeProfile<0, 3,
                                             [SDTCisPtrTy<0>,
                                              SDTCisPtrTy<1>,
                                              SDTCisVT<2, i64>]>;
 def SDT_ZMemMemLengthCC     : SDTypeProfile<1, 3,
                                             [SDTCisVT<0, i32>,
                                              SDTCisPtrTy<1>,
                                              SDTCisPtrTy<2>,
                                              SDTCisVT<3, i64>]>;
+def SDT_ZMemsetMVC          : SDTypeProfile<0, 3,
+                                            [SDTCisPtrTy<0>,
+                                             SDTCisVT<1, i64>,
+                                             SDTCisVT<2, i32>]>;
 def SDT_ZString             : SDTypeProfile<1, 3,
                                             [SDTCisPtrTy<0>,
                                              SDTCisPtrTy<1>,
                                              SDTCisPtrTy<2>,
                                              SDTCisVT<3, i32>]>;
 def SDT_ZStringCC           : SDTypeProfile<2, 3,
                                             [SDTCisPtrTy<0>,
                                              SDTCisVT<1, i32>,

 def z_nc                : SDNode<"SystemZISD::NC", SDT_ZMemMemLength,
                                   [SDNPHasChain, SDNPMayStore, SDNPMayLoad]>;
 def z_oc                : SDNode<"SystemZISD::OC", SDT_ZMemMemLength,
                                   [SDNPHasChain, SDNPMayStore, SDNPMayLoad]>;
 def z_xc                : SDNode<"SystemZISD::XC", SDT_ZMemMemLength,
                                   [SDNPHasChain, SDNPMayStore, SDNPMayLoad]>;
 def z_clc               : SDNode<"SystemZISD::CLC", SDT_ZMemMemLengthCC,
                                  [SDNPHasChain, SDNPMayLoad]>;
+def z_memset_mvc        : SDNode<"SystemZISD::MEMSET_MVC", SDT_ZMemsetMVC,
+                                 [SDNPHasChain, SDNPMayStore, SDNPMayLoad]>;
 def z_strcmp            : SDNode<"SystemZISD::STRCMP", SDT_ZStringCC,
                                  [SDNPHasChain, SDNPMayLoad]>;
 def z_stpcpy            : SDNode<"SystemZISD::STPCPY", SDT_ZString,
                                  [SDNPHasChain, SDNPMayStore, SDNPMayLoad]>;
 def z_search_string     : SDNode<"SystemZISD::SEARCH_STRING", SDT_ZStringCC,
                                  [SDNPHasChain, SDNPMayLoad]>;
 def z_prefetch          : SDNode<"SystemZISD::PREFETCH", SDT_ZPrefetch,
                                  [SDNPHasChain, SDNPMayLoad, SDNPMayStore,

llvm/lib/Target/SystemZ/SystemZSelectionDAGInfo.cpp

 
 #include "SystemZTargetMachine.h"
 #include "llvm/CodeGen/SelectionDAG.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "systemz-selectiondag-info"
 
-static SDVTList getMemMemVTs(unsigned Op, SelectionDAG &DAG) {
-  return Op == SystemZISD::CLC ? DAG.getVTList(MVT::i32, MVT::Other)
+static unsigned getMemMemLenAdj(unsigned Op) {
+  return Op == SystemZISD::MEMSET_MVC ? 2 : 1;
+}
+
+static SDValue createMemMemNode(SelectionDAG &DAG, const SDLoc &DL, unsigned Op,
+                                SDValue Chain, SDValue Dst, SDValue Src,
+                                SDValue LenAdj, SDValue Byte) {
+  SDVTList VTs = Op == SystemZISD::CLC ? DAG.getVTList(MVT::i32, MVT::Other)
                                        : DAG.getVTList(MVT::Other);
+  SmallVector<SDValue, 6> Ops;
+  if (Op == SystemZISD::MEMSET_MVC)
+    Ops = { Chain, Dst, LenAdj, Byte };
+  else
+    Ops = { Chain, Dst, Src, LenAdj };
+  return DAG.getNode(Op, DL, VTs, Ops);
 }
 
-// Emit a mem-mem operation after subtracting one from size, which will be
-// added back during pseudo expansion. As the Reg case emitted here may be
-// converted by DAGCombiner into having an Imm length, they are both emitted
-// the same way.
+// Emit a mem-mem operation after subtracting one (or two for memset) from
+// size, which will be added back during pseudo expansion. As the Reg case
+// emitted here may be converted by DAGCombiner into having an Imm length,
+// they are both emitted the same way.
 static SDValue emitMemMemImm(SelectionDAG &DAG, const SDLoc &DL, unsigned Op,
                              SDValue Chain, SDValue Dst, SDValue Src,
-                             uint64_t Size) {
-  return DAG.getNode(Op, DL, getMemMemVTs(Op, DAG), Chain, Dst, Src,
-                     DAG.getConstant(Size - 1, DL, Src.getValueType()));
+                             uint64_t Size, SDValue Byte = SDValue()) {
+  unsigned Adj = getMemMemLenAdj(Op);
+  assert(Size >= Adj && "Adjusted length overflow.");
+  SDValue LenAdj = DAG.getConstant(Size - Adj, DL, Dst.getValueType());
+  return createMemMemNode(DAG, DL, Op, Chain, Dst, Src, LenAdj, Byte);
 }
 
 static SDValue emitMemMemReg(SelectionDAG &DAG, const SDLoc &DL, unsigned Op,
                              SDValue Chain, SDValue Dst, SDValue Src,
-                             SDValue Size) {
-  SDValue LenMinus1 = DAG.getNode(ISD::ADD, DL, MVT::i64,
-                                  DAG.getZExtOrTrunc(Size, DL, MVT::i64),
-                                  DAG.getConstant(-1, DL, MVT::i64));
-  return DAG.getNode(Op, DL, getMemMemVTs(Op, DAG), Chain, Dst, Src, LenMinus1);
+                             SDValue Size, SDValue Byte = SDValue()) {
+  int64_t Adj = getMemMemLenAdj(Op);
+  SDValue LenAdj = DAG.getNode(ISD::ADD, DL, MVT::i64,
+                               DAG.getConstant(0 - Adj, DL, MVT::i64));
+  return createMemMemNode(DAG, DL, Op, Chain, Dst, Src, LenAdj, Byte);
 }
 
 SDValue SystemZSelectionDAGInfo::EmitTargetCodeForMemcpy(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Dst, SDValue Src,
     SDValue Size, Align Alignment, bool IsVolatile, bool AlwaysInline,
     MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
   if (IsVolatile)
     return SDValue();
if (CByte) {
       }
     }
     assert(Bytes >= 2 && "Should have dealt with 0- and 1-byte cases already");
 
     // Handle the special case of a memset of 0, which can use XC.
     if (CByte && CByte->getZExtValue() == 0)
       return emitMemMemImm(DAG, DL, SystemZISD::XC, Chain, Dst, Dst, Bytes);
 
-    // Copy the byte to the first location and then use MVC to copy
-    // it to the rest.
-    Chain = DAG.getStore(Chain, DL, Byte, Dst, DstPtrInfo, Alignment);
-    SDValue DstPlus1 = DAG.getNode(ISD::ADD, DL, PtrVT, Dst,
-                                   DAG.getConstant(1, DL, PtrVT));
-    return emitMemMemImm(DAG, DL, SystemZISD::MVC, Chain, DstPlus1, Dst,
-                         Bytes - 1);
+    return emitMemMemImm(DAG, DL, SystemZISD::MEMSET_MVC, Chain, Dst, SDValue(),
+                         Bytes, DAG.getAnyExtOrTrunc(Byte, DL, MVT::i32));
   }
 
   // Variable length
   if (CByte && CByte->getZExtValue() == 0)
     // Handle the special case of a variable length memset of 0 with XC.
     return emitMemMemReg(DAG, DL, SystemZISD::XC, Chain, Dst, Dst, Size);
 
-  return SDValue();
+  return emitMemMemReg(DAG, DL, SystemZISD::MEMSET_MVC, Chain, Dst, SDValue(),
+                       Size, DAG.getAnyExtOrTrunc(Byte, DL, MVT::i32));
 }
 
 // Convert the current CC value into an integer that is 0 if CC == 0,
 // greater than zero if CC == 1 and less than zero if CC >= 2.
 // The sequence starts with IPM, which puts CC into bits 29 and 28
 // of an integer and clears bits 30 and 31.
 static SDValue addIPMSequence(const SDLoc &DL, SDValue CCReg,
                               SelectionDAG &DAG) {

llvm/test/CodeGen/SystemZ/memset-01.ll

; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 3, i1 false)
   ret void
 }
 
 ; 257 bytes, i32 version.
 define void @f9(i8* %dest, i8 %val) {
 ; CHECK-LABEL: f9:
 ; CHECK: stc %r3, 0(%r2)
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: stc %r3, 256(%r2)
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i32(i8* %dest, i8 %val, i32 257, i1 false)
   ret void
 }
 
 ; 257 bytes, i64 version.
 define void @f10(i8* %dest, i8 %val) {
 ; CHECK-LABEL: f10:
 ; CHECK: stc %r3, 0(%r2)
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: stc %r3, 256(%r2)
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 257, i1 false)
   ret void
 }
 
 ; 258 bytes, i32 version.  We need two MVCs.
 define void @f11(i8* %dest, i8 %val) {
 ; CHECK-LABEL: f11:
 ; CHECK: stc %r3, 0(%r2)
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: stc %r3, 256(%r2)
 ; CHECK: mvc 257(1,%r2), 256(%r2)
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i32(i8* %dest, i8 %val, i32 258, i1 false)
   ret void
 }
 
 ; 258 bytes, i64 version.
 define void @f12(i8* %dest, i8 %val) {
 ; CHECK-LABEL: f12:
 ; CHECK: stc %r3, 0(%r2)
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: stc %r3, 256(%r2)
 ; CHECK: mvc 257(1,%r2), 256(%r2)
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 258, i1 false)
   ret void
 }
 
 ; Test the largest case for which straight-line code is used.
 define void @f13(i8* %dest, i8 %val) {
 ; CHECK-LABEL: f13:
 ; CHECK: stc %r3, 0(%r2)
-; CHECK: mvc 1(256,%r2), 0(%r2)
-; CHECK: mvc 257(256,%r2), 256(%r2)
-; CHECK: mvc 513(256,%r2), 512(%r2)
-; CHECK: mvc 769(256,%r2), 768(%r2)
-; CHECK: mvc 1025(256,%r2), 1024(%r2)
-; CHECK: mvc 1281(256,%r2), 1280(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: stc %r3, 256(%r2)
+; CHECK: mvc 257(255,%r2), 256(%r2)
+; CHECK: stc %r3, 512(%r2)
+; CHECK: mvc 513(255,%r2), 512(%r2)
+; CHECK: stc %r3, 768(%r2)
+; CHECK: mvc 769(255,%r2), 768(%r2)
+; CHECK: stc %r3, 1024(%r2)
+; CHECK: mvc 1025(255,%r2), 1024(%r2)
+; CHECK: stc %r3, 1280(%r2)
+; CHECK: mvc 1281(255,%r2), 1280(%r2)
 ; CHECK: br %r14
-  call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 1537, i1 false)
+  call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 1536, i1 false)
   ret void
 }
 
 ; Test the next size up, which uses a loop.  We leave the other corner
-; cases to memcpy-01.ll.
+; cases to memcpy-01.ll and memset-07.ll.
 define void @f14(i8* %dest, i8 %val) {
 ; CHECK-LABEL: f14:
-; CHECK: stc %r3, 0(%r2)
 ; CHECK: lghi [[COUNT:%r[0-5]]], 6
 ; CHECK: [[LABEL:\.L[^:]*]]:
-; CHECK: pfd 2, 769(%r2)
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: pfd 2, 768(%r2)
+; CHECK: stc %r3, 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
 ; CHECK: la %r2, 256(%r2)
 ; CHECK: brctg [[COUNT]], [[LABEL]]
-; CHECK: mvc 1(1,%r2), 0(%r2)
-; CHECK: br %r14
-  call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 1538, i1 false)
+; CHECK: stc %r3, 0(%r2)
+; CHECK-NEXT: br %r14
+  call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 1537, i1 false)
   ret void
 }
+
+; Test (no) folding of displacement: Begins with max(uint12) - 1.
+define void @f15(i8* %dest, i8 %val) {
+; CHECK-LABEL: f15:
+; CHECK-NOT: la {{.*}}%r2
+  %addr = getelementptr i8, i8* %dest, i64 4094
+  call void @llvm.memset.p0i8.i64(i8* %addr, i8 %val, i64 256, i1 false)
+  ret void
+}
+
+; Test folding of displacement: Begins with max(uint12).
+define void @f16(i8* %dest, i8 %val) {
+; CHECK-LABEL: f16:
+; CHECK-DAG: lay %r1, 4096(%r2)
+; CHECK-DAG: stc %r3, 4095(%r2)
+  %addr = getelementptr i8, i8* %dest, i64 4095
+  call void @llvm.memset.p0i8.i64(i8* %addr, i8 %val, i64 256, i1 false)
+  ret void
+}
+
+; Test folding of displacement with LA: First two ops are in range.
+define void @f17(i8* %dest, i8 %val) {
+; CHECK-LABEL: f17:
+; CHECK:      stc %r3, 3583(%r2)
+; CHECK-NEXT: mvc 3584(255,%r2), 3583(%r2)
+; CHECK-NEXT: stc %r3, 3839(%r2)
+; CHECK-NEXT: mvc 3840(255,%r2), 3839(%r2)
+; CHECK-NEXT: lay %r1, 4096(%r2)
+; CHECK-NEXT: stc %r3, 4095(%r2)
+; CHECK-NEXT: mvc 0(1,%r1), 4095(%r2)
+; CHECK-NEXT: br %r14
+  %addr = getelementptr i8, i8* %dest, i64 3583
+  call void @llvm.memset.p0i8.i64(i8* %addr, i8 %val, i64 514, i1 false)
+  ret void
+}
+
+; Test folding of displacement with LAY: First two ops are in range.
+define void @f18(i8* %dest, i8 %val) {
+; CHECK-LABEL: f18:
+; CHECK:      stc %r3, 3584(%r2)
+; CHECK-NEXT: mvc 3585(255,%r2), 3584(%r2)
+; CHECK-NEXT: stc %r3, 3840(%r2)
+; CHECK-NEXT: mvc 3841(255,%r2), 3840(%r2)
+; CHECK-NEXT: lay %r1, 4097(%r2)
+; CHECK-NEXT: lay %r2, 4096(%r2)
+; CHECK-NEXT: stc %r3, 0(%r2)
+; CHECK-NEXT: mvc 0(1,%r1), 0(%r2)
+; CHECK-NEXT: br %r14
+  %addr = getelementptr i8, i8* %dest, i64 3584
+  call void @llvm.memset.p0i8.i64(i8* %addr, i8 %val, i64 514, i1 false)
+  ret void
+}
+

llvm/test/CodeGen/SystemZ/memset-02.ll

; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 128, i64 5, i1 false)
   ret void
 }
 
 ; 257 bytes, i32 version.
 define void @f13(i8* %dest) {
 ; CHECK-LABEL: f13:
 ; CHECK: mvi 0(%r2), 128
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: mvi 256(%r2), 128
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i32(i8* %dest, i8 128, i32 257, i1 false)
   ret void
 }
 
 ; 257 bytes, i64 version.
 define void @f14(i8* %dest) {
 ; CHECK-LABEL: f14:
 ; CHECK: mvi 0(%r2), 128
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: mvi 256(%r2), 128
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 128, i64 257, i1 false)
   ret void
 }
 
 ; 258 bytes, i32 version.  We need two MVCs.
 define void @f15(i8* %dest) {
 ; CHECK-LABEL: f15:
 ; CHECK: mvi 0(%r2), 128
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: mvi 256(%r2), 128
 ; CHECK: mvc 257(1,%r2), 256(%r2)
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i32(i8* %dest, i8 128, i32 258, i1 false)
   ret void
 }
 
 ; 258 bytes, i64 version.
 define void @f16(i8* %dest) {
 ; CHECK-LABEL: f16:
 ; CHECK: mvi 0(%r2), 128
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: mvi 256(%r2), 128
 ; CHECK: mvc 257(1,%r2), 256(%r2)
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 128, i64 258, i1 false)
   ret void
 }

llvm/test/CodeGen/SystemZ/memset-04.ll

; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 -1, i64 17, i1 false)
   ret void
 }
 
 ; 257 bytes, i32 version.
 define void @f37(i8* %dest) {
 ; CHECK-LABEL: f37:
 ; CHECK: mvi 0(%r2), 255
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: mvi 256(%r2), 255
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i32(i8* %dest, i8 -1, i32 257, i1 false)
   ret void
 }
 
 ; 257 bytes, i64 version.
 define void @f38(i8* %dest) {
 ; CHECK-LABEL: f38:
 ; CHECK: mvi 0(%r2), 255
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: mvi 256(%r2), 255
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 -1, i64 257, i1 false)
   ret void
 }
 
 ; 258 bytes, i32 version.  We need two MVCs.
 define void @f39(i8* %dest) {
 ; CHECK-LABEL: f39:
 ; CHECK: mvi 0(%r2), 255
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: mvi 256(%r2), 255
 ; CHECK: mvc 257(1,%r2), 256(%r2)
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i32(i8* %dest, i8 -1, i32 258, i1 false)
   ret void
 }
 
 ; 258 bytes, i64 version.
 define void @f40(i8* %dest) {
 ; CHECK-LABEL: f40:
 ; CHECK: mvi 0(%r2), 255
-; CHECK: mvc 1(256,%r2), 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: mvi 256(%r2), 255
 ; CHECK: mvc 257(1,%r2), 256(%r2)
 ; CHECK: br %r14
   call void @llvm.memset.p0i8.i64(i8* %dest, i8 -1, i64 258, i1 false)
   ret void
 }

llvm/test/CodeGen/SystemZ/memset-07.ll

+; Test memset in cases where a loop is used.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
+
+declare void @llvm.memset.p0i8.i32(i8 *nocapture, i8, i32, i1) nounwind
+declare void @llvm.memset.p0i8.i64(i8 *nocapture, i8, i64, i1) nounwind
+
+; Constant length: 6 iterations and 2 bytes remainder.
+define void @f1(i8* %dest, i8 %val) {
+; CHECK-LABEL: f1:
+; CHECK: lghi [[COUNT:%r[0-5]]], 6
+; CHECK: [[LABEL:\.L[^:]*]]:
+; CHECK: pfd 2, 768(%r2)
+; CHECK: stc %r3, 0(%r2)
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: la %r2, 256(%r2)
+; CHECK: brctg [[COUNT]], [[LABEL]]
+; CHECK: stc %r3, 0(%r2)
+; CHECK-NEXT: mvc 1(1,%r2), 0(%r2)
+; CHECK-NEXT: br %r14
+  call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 1538, i1 false)
+  ret void
+}
+
+; Constant length: 6 iterations and 255 bytes remainder.
+define void @f2(i8* %dest) {
+; CHECK-LABEL: f2:
+; CHECK: lghi [[COUNT:%r[0-5]]], 6
+; CHECK: [[LABEL:\.L[^:]*]]:
+; CHECK: pfd 2, 768(%r2)
+; CHECK: mvi  0(%r2), 1
+; CHECK: mvc 1(255,%r2), 0(%r2)
+; CHECK: la %r2, 256(%r2)
+; CHECK: brctg [[COUNT]], [[LABEL]]
+; CHECK: mvi  0(%r2), 1
+; CHECK-NEXT: mvc 1(254,%r2), 0(%r2)
+; CHECK-NEXT: br %r14
+  call void @llvm.memset.p0i8.i32(i8* %dest, i8 1, i32 1791, i1 false)
+  ret void
+}
+
+; Variable length, byte in register.
+define void @f3(i8* %dest, i8 %val, i64 %Len) {
+; CHECK-LABEL: f3:
+; CHECK: # %bb.0:
+; CHECK-NEXT:   aghi  %r4, -2
+; CHECK-NEXT:   cgibe %r4, -2, 0(%r14)
+; CHECK-NEXT: .LBB2_1:
+; CHECK-NEXT: cgije %r4, -1, .LBB2_5
+; CHECK-NEXT:# %bb.2:
+; CHECK-NEXT: srlg  %r0, %r4, 8
+; CHECK-NEXT: cgije %r0, 0, .LBB2_4
+; CHECK-NEXT:.LBB2_3:                   # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: pfd 2, 768(%r2)
+; CHECK-NEXT: stc %r3, 0(%r2)
+; CHECK-NEXT: mvc 1(255,%r2), 0(%r2)
+; CHECK-NEXT: la  %r2, 256(%r2)
+; CHECK-NEXT: brctg %r0, .LBB2_3
+; CHECK-NEXT:.LBB2_4:
+; CHECK-NEXT: stc %r3, 0(%r2)
+; CHECK-NEXT: exrl  %r4, .Ltmp0
+; CHECK-NEXT: br  %r14
+; CHECK-NEXT:.LBB2_5:
+; CHECK-NEXT: stc %r3, 0(%r2)
+; CHECK-NEXT: br  %r14
+  call void @llvm.memset.p0i8.i64(i8* %dest, i8 %val, i64 %Len, i1 false)
+  ret void
+}
+
+; Variable length, immediate byte.
+define void @f4(i8* %dest, i32 %Len) {
+; CHECK-LABEL: f4:
+; CHECK: # %bb.0:
+; CHECK-NEXT: llgfr %r1, %r3
+; CHECK-NEXT: aghi  %r1, -2
+; CHECK-NEXT: cgibe %r1, -2, 0(%r14)
+; CHECK-NEXT:.LBB3_1:
+; CHECK-NEXT: cgije %r1, -1, .LBB3_5
+; CHECK-NEXT:# %bb.2:
+; CHECK-NEXT: srlg  %r0, %r1, 8
+; CHECK-NEXT: cgije %r0, 0, .LBB3_4
+; CHECK-NEXT:.LBB3_3:                   # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: pfd 2, 768(%r2)
+; CHECK-NEXT: mvi 0(%r2), 1
+; CHECK-NEXT: mvc 1(255,%r2), 0(%r2)
+; CHECK-NEXT: la  %r2, 256(%r2)
+; CHECK-NEXT: brctg %r0, .LBB3_3
+; CHECK-NEXT:.LBB3_4:
+; CHECK-NEXT: mvi 0(%r2), 1
+; CHECK-NEXT: exrl  %r1, .Ltmp0
+; CHECK-NEXT: br  %r14
+; CHECK-NEXT:.LBB3_5:
+; CHECK-NEXT: mvi 0(%r2), 1
+; CHECK-NEXT: br  %r14
+  call void @llvm.memset.p0i8.i32(i8* %dest, i8 1, i32 %Len, i1 false)
+  ret void
+}
+
+; CHECK: .Ltmp0:
+; CHECK-NEXT: mvc 1(1,%r2), 0(%r2)

llvm/test/CodeGen/SystemZ/tail-call-mem-intrinsics.ll

 ; RUN: llc -mtriple=s390x-linux-gnu < %s | FileCheck %s
 
 ; CHECK-LABEL: tail_memmove:
 ; CHECK: jg memmove
 define void @tail_memmove(i8* nocapture %p, i8* nocapture readonly %q, i32 %n) #0 {
 entry:
   tail call void @llvm.memmove.p0i8.p0i8.i32(i8* %p, i8* %q, i32 %n, i1 false)
   ret void
 }
 
 ; CHECK-LABEL: tail_memset:
 ; CHECK: jg memset
 define void @tail_memset(i8* nocapture %p, i8 %c, i32 %n) #0 {
 entry:
-  tail call void @llvm.memset.p0i8.i32(i8* %p, i8 %c, i32 %n, i1 false)
+  tail call void @llvm.memset.p0i8.i32(i8* %p, i8 %c, i32 %n, i1 true)
   ret void
 }
 
 declare void @llvm.memcpy.p0i8.p0i8.i32(i8* nocapture, i8* nocapture readonly, i32, i1) #0
 declare void @llvm.memmove.p0i8.p0i8.i32(i8* nocapture, i8* nocapture readonly, i32, i1) #0
 declare void @llvm.memset.p0i8.i32(i8* nocapture, i8, i32, i1) #0
 
 attributes #0 = { nounwind }