[SystemZ] Generate XC loop for memset 0 of variable length.

[jonpa]

Summary

Tried first inserting new MBBs for the EXRL and target instruction, but that has some problems:

• Need a guarantee that the target instruction is not moved by later passes. hasSideEffects flag and surrounding the target instruction with MemBarrier:s might should do the trick but that is a bit clumsy. And to add hasSideEffects on all MemorySS instructions will cause many files to change since that will cause a new scheduler region to be created.
• The MBB of the target instruction should be reachable, or I think it might be simply removed by some optimizer. In order to maintain CFG edges between the EXRL MBB and TargetIns MBB, analyzeBranch() needs to detect these blocks and return them as unanalyzable, since EXRL is not really a branch or terminator. This is also a bit of extra work...

I Instead tried using an EXRL_Pseudo that is kept intact all the way to AsmPrinter. My newbie approach of creating symbols and MCInsts and then emit them at the end of the function seems to work fine both for assembly and obj streaming. Not sure if there is another more correct way...

A new method emitMemMemLoopVarLen() which shares code with emitMemMemWrapper() via new class MemMemBuilder (since there is no public default constructor for MachineOperand the DestBase and SrcBase must be passed to its constructor rather than via a single MI argument).

The target instruction will never be out-of-range for the EXRL instruction with a 32 bit signed range, right?

Pardon the typo-fixes mixed in here (MMB -> MBB). Maybe pre-commit them?

[jonpa]

There is something wrong somewhere here.. some benchmarks are seg-faulting. investigating

[jonpa]

I think the problem was that EXRL cannot use %r0 as that means no or:ing will take place.

[jonpa]

Benchmarking:

Z14:

reduced:

Improvements:
0.984: f511.povray_r 
0.988: f510.parest_r 
0.992: i523.xalancbmk_r 
0.993: i531.deepsjeng_r 
0.993: f508.namd_r 
...

Regressions:
1.027: i557.xz_r 
1.006: f507.cactuBSSN_r 
...

full (4):

Improvements:
0.990: f511.povray_r 
0.991: i557.xz_r 
0.991: f510.parest_r 
0.998: i523.xalancbmk_r

Z15:

reduced:

Improvements:
0.823: i531.deepsjeng_r 
0.985: f538.imagick_r 
0.990: i500.perlbench_r 
0.993: f510.parest_r 

Regressions:
(none)

full (4):

Improvements:
0.994: i531.deepsjeng_r 
0.995: f510.parest_r 
0.997: f538.imagick_r 

Regressions:
1.002: i500.perlbench_r

[uweigand]

This looks pretty good to me. One thing that is odd is the extra always-zero parameter to the LoopVarLen insns. This really should be removed.

As a suggestion that might remove a bit of the duplication between emitMemMemWrapper and emitMemMemLoopVarLen: the (old) routine is currently called for both the Sequence and Loop insns, and it distinguishes between them based on the operand count. However, for the LoopVarLen case, your patch now introduces a separate new routine. I think it might be simpler to continue to use the same routine, and just internally distinguish between the cases based on whether the length operand is an immediate or a register.

Finally, I'm wondering if it wouldn't make sense to extend support for the variable-length case to the other block operations, now that we already have the EXRL_Pseudo infrastructure. (I guess that could also be done in a separate patch.)

Pardon the typo-fixes mixed in here (MMB -> MBB). Maybe pre-commit them?

Yes, please pre-commit those.

llvm/lib/Target/SystemZ/SystemZAsmPrinter.h

14

The above clang-format check looks valid (includes should be kept sorted).

29

This as well.

[jonpa]

One thing that is odd is the extra always-zero parameter to the LoopVarLen insns. This really should be removed.

Done - it was easy to remove on the MachineInstr, but on the SystemZISD node it seems reasonable to keep a dummy zero there and reuse the existing opcode, or?

As a suggestion that might remove a bit of the duplication between emitMemMemWrapper and emitMemMemLoopVarLen: the (old) routine is currently called for both the Sequence and Loop insns, and it distinguishes between them based on the operand count. However, for the LoopVarLen case, your patch now introduces a separate new routine. I think it might be simpler to continue to use the same routine, and just internally distinguish between the cases based on whether the length operand is an immediate or a register.

OK - I merged the two functions instead.

Finally, I'm wondering if it wouldn't make sense to extend support for the variable-length case to the other block operations, now that we already have the EXRL_Pseudo infrastructure. (I guess that could also be done in a separate patch.)

I would like to do that afterwards then with some more benchmarking as well...

Pardon the typo-fixes mixed in here (MMB -> MBB). Maybe pre-commit them?

Yes, please pre-commit those.

b2cd98d

llvm/lib/Target/SystemZ/SystemZAsmPrinter.h
The above clang-format check looks valid (includes should be kept sorted).
This as well.

done

Also added handling/test for i32 case (test/CodeGen/Generic/ForceStackAlign.ll was actually failing) - if the type of length is <64 bits, it needs zero-extension.

[jonpa]

On spec, I see now 8869 exrl instructions, each with its own xc target instruction. I wonder if we should try to avoid duplicating target instructions. I see in some files hundreds of identical xc targets (worst case is 845 identical ones in one file: f510.parest_r/build/vectors.s !). 67 files have more than 20 identical xc targets...

[uweigand]

In D103865#2831600, @jonpa wrote:

One thing that is odd is the extra always-zero parameter to the LoopVarLen insns. This really should be removed.

Done - it was easy to remove on the MachineInstr, but on the SystemZISD node it seems reasonable to keep a dummy zero there and reuse the existing opcode, or?

Aren't there already two opcodes, one with 4 operands and one with 5? Can't you just use the one with 4 operands instead?

That said, maybe it actually is better to use the version with 5 operands, and actually pass in two values: the length (minus one), and the loop trip count (length / 256). These two values can easily be computed at the SelectionDAG level already, and if they are, that might open opportunities to optimize / reuse the values at that level. (Also, then it is more obvious to re-use the same opcode, since the operands do in fact have the same semantics then, it's just that in one case they're immediates and in the other they are registers.)

As a suggestion that might remove a bit of the duplication between emitMemMemWrapper and emitMemMemLoopVarLen: the (old) routine is currently called for both the Sequence and Loop insns, and it distinguishes between them based on the operand count. However, for the LoopVarLen case, your patch now introduces a separate new routine. I think it might be simpler to continue to use the same routine, and just internally distinguish between the cases based on whether the length operand is an immediate or a register.

OK - I merged the two functions instead.

Hmm, that's not quite what I had in mind, sorry. This code still has two completely separate code paths, just now in one function. What I had been thinking of is to actually share the same code path that e.g. emits the loop -- they should be identical except that the trip count is an immediate vs. a register.

The point is to merge the two paths to the extent where there is no longer any point in extracting buildMemMemLoop as a subroutine as it actually is used only once.

On spec, I see now 8869 exrl instructions, each with its own xc target instruction. I wonder if we should try to avoid duplicating target instructions. I see in some files hundreds of identical xc targets (worst case is 845 identical ones in one file: f510.parest_r/build/vectors.s !). 67 files have more than 20 identical xc targets...

That makes sense. It should be straightforward to sort and de-duplicate the target instructions at final emission time.

[jonpa]

Aren't there already two opcodes, one with 4 operands and one with 5? Can't you just use the one with 4 operands instead?

Ah, yes, I guess I was looking at the names: 'Length' vs 'Loop'...

That said, maybe it actually is better to use the version with 5 operands, and actually pass in two values: the length (minus one), and the loop trip count (length / 256). These two values can easily be computed at the SelectionDAG level already, and if they are, that might open opportunities to optimize / reuse the values at that level. (Also, then it is more obvious to re-use the same opcode, since the operands do in fact have the same semantics then, it's just that in one case they're immediates and in the other they are registers.)

I updated the patch to create the length-1 and trip count on the DAG instead.

I see that this means that they are initially placed before the zero-length check, while before they were put in the preheader of the loop. The SRLG is actually moved to the preheader by MachineSink later, and this could probably be done for the AGHI as well if the comparison would be made with the AGHI source against 0.

To keep things simple, I just changed the comparison to be against -1 instead for now. What do you think about that - is the zero-length case rare enough to ignore where the AGHI is placed? We could do the extra work of finding the AGHI/-1 and comparing with 0 of that source reg instead, and expect the AGHI will be sunk later if there are no other users I would hope.

Hmm, that's not quite what I had in mind, sorry. This code still has two completely separate code paths, just now in one function. What I had been thinking of is to actually share the same code path that e.g. emits the loop -- they should be identical except that the trip count is an immediate vs. a register.

The point is to merge the two paths to the extent where there is no longer any point in extracting buildMemMemLoop as a subroutine as it actually is used only once.

I gave it another try: First let the reg/immediate cases set up the needed MBBs and then build the loop afterwards for both cases in one place. I guess that seems more simple than having a separate class for creating the loop...

That makes sense. It should be straightforward to sort and de-duplicate the target instructions at final emission time.

AsmPrinter::EmitToStreamer() calls getSubtargetInfo() which needs an MF, so it did not seem quite simple to emit all the EXRL targets in emitEndOfAsmFile(). That method seems to be intended for other things than emitting instructions. Does it seem right to try to create a new section at that point after all else and emit the target instructions?

For now, I tried just improving the output per function. This saves 4110 XC target instructions (prior count was ~9000). Worst case is now 168 in one file, and 38 files have >20, so it is an improvement, but still some room for improvement if reusing per file...

[uweigand]

In D103865#2837530, @jonpa wrote:

I updated the patch to create the length-1 and trip count on the DAG instead.

Thanks!

I see that this means that they are initially placed before the zero-length check, while before they were put in the preheader of the loop. The SRLG is actually moved to the preheader by MachineSink later, and this could probably be done for the AGHI as well if the comparison would be made with the AGHI source against 0.

To keep things simple, I just changed the comparison to be against -1 instead for now. What do you think about that - is the zero-length case rare enough to ignore where the AGHI is placed? We could do the extra work of finding the AGHI/-1 and comparing with 0 of that source reg instead, and expect the AGHI will be sunk later if there are no other users I would hope.

I think this is probably OK. GCC does the compare with 0, but there's probably not much difference. It's a pity we cannot move that check up to SystemZSelectionDAGInfo::EmitTargetCodeForMemset, but I believe we cannot create new basic blocks on the DAG level.

Hmm, that's not quite what I had in mind, sorry. This code still has two completely separate code paths, just now in one function. What I had been thinking of is to actually share the same code path that e.g. emits the loop -- they should be identical except that the trip count is an immediate vs. a register.

The point is to merge the two paths to the extent where there is no longer any point in extracting buildMemMemLoop as a subroutine as it actually is used only once.

I gave it another try: First let the reg/immediate cases set up the needed MBBs and then build the loop afterwards for both cases in one place. I guess that seems more simple than having a separate class for creating the loop...

That's better, thanks! I think readability might be even better if you continue to create the MBBs in order, i.e. emit the EXRL at the end instead of at the beginning. (That means a duplicated LengthMO.isReg() test, but that should still be better.)

That makes sense. It should be straightforward to sort and de-duplicate the target instructions at final emission time.

AsmPrinter::EmitToStreamer() calls getSubtargetInfo() which needs an MF, so it did not seem quite simple to emit all the EXRL targets in emitEndOfAsmFile(). That method seems to be intended for other things than emitting instructions. Does it seem right to try to create a new section at that point after all else and emit the target instructions?

Ah, that's unfortunate, but I guess understandable. The assumption is that code emission needs subtarget-specific info, and the subtarget is defined by attributes (like target CPU) that change between functions. At the end of the asm file, we're outside of all functions ...

There are ways around that. We do have a TargetMachine in the AsmPrinter, and there used to be a function getSubtargetImpl() that returns a "default" subtarget for the compilation unit. This has been deprecated (for the above reasons), but could probably be re-activated, and then be used as the subtarget to pass to emitInstruction during emitEndOfAsmFile.

Another option would be to simply "assemble" those instructions by hand, i.e. compute the 6-byte integer value from the constituent fields. Something like:

Opcode =  TargetInsOpc << 40 | LenMinus1Reg << 32 | DestReg << 28 | DestDisp << 16 | SrcReg << 12 | SrcDisp;

and then simply emit that value as 6 bytes of "data" (but still into the .text section, of course). This is also somewhat ugly, but maybe less ugly than the default Subtarget ...

llvm/lib/Target/SystemZ/SystemZAsmPrinter.cpp

568

Ah, if you do it that way, you don't actually have to have multiple labels for the same instruction. Just compute the instruction (or just its opcode, see discussion in the main comment) first, and look it up in the table. If found, simply return its (one) associated symbol; if not, create a new symbol and add the pair of symbol, instruction to the table.

Actually, that table can then just be a std::map (from instruction to symbol) instead of a vector, this implements the lookup in a more efficient way ...

[jonpa]

I think readability might be even better if you continue to create the MBBs in order, i.e. emit the EXRL at the end instead of at the beginning. (That means a duplicated LengthMO.isReg() test, but that should still be better.)

Makes sense... fixed.

Ah, if you do it that way, you don't actually have to have multiple labels for the same instruction. Just compute the instruction (or just its opcode, see discussion in the main comment) first, and look it up in the table. If found, simply return its (one) associated symbol; if not, create a new symbol and add the pair of symbol, instruction to the table.

Actually, that table can then just be a std::map (from instruction to symbol) instead of a vector, this implements the lookup in a more efficient way ...

ah, of course - only one label emitted now. They by this get sorted per the operands in the comparator for the std::map object, but a little detail then is that the labels now get emitted in any order (non-sorted) - but I suppose that's ok, right?

There are ways around that. We do have a TargetMachine in the AsmPrinter, and there used to be a function getSubtargetImpl() that returns a "default" subtarget for the compilation unit. This has been deprecated (for the above reasons), but could probably be re-activated, and then be used as the subtarget to pass to emitInstruction during emitEndOfAsmFile.

Another option would be to simply "assemble" those instructions by hand, i.e. compute the 6-byte integer value from the constituent fields.

I started out with trying to "assemble" by hand per your suggestion, but it seemed like quite some work, especially considering that we also have to be able to print it as text. So I tried instead with adding back the GenericSubtarget, which was simple enough.

This then eliminates another 3786 XC targets: there are no duplicates in any file, and at most I see 13 different XC targets in a single file :-)

SPEC master <> patch

cgije          :               115009               132379   +17370
xc             :                13334                23198    +9864
exrl           :                    0                 8873    +8873
brctg          :                 8356                17228    +8872
srlg           :                54407                63229    +8822
brasl          :               640463               631658    -8805
...

[uweigand]

In D103865#2848831, @jonpa wrote:

ah, of course - only one label emitted now. They by this get sorted per the operands in the comparator for the std::map object, but a little detail then is that the labels now get emitted in any order (non-sorted) - but I suppose that's ok, right?

That shouldn't matter.

I started out with trying to "assemble" by hand per your suggestion, but it seemed like quite some work, especially considering that we also have to be able to print it as text.

I was thinking of just using emitInt16 for the three words, that will show as .word in the textual assembler output, which may not be pretty but is certainly correct. (You could in addition use an emitRawComment to print a textual representation to make the file more readable).

So I tried instead with adding back the GenericSubtarget, which was simple enough.

See the comments inline on that option.

In the meantime I thought of yet another option: the instructions used via EXRL are really part of the function containing the EXRL, and should therefore be emitted using the same Subtarget that is in effect for that function (this is not *currently* a problem, but it might be a potential issue if we want to use an EXRL target instruction that is only available in some ISA levels).

To correctly model this, one option might be to not have a single EXRLT2SymMap, but one per Subtarget, and sort the target instructions into the appropriate subtarget for the current function as they're added to the map. (Or, even simpler, just make the Subtarget part of the "key" for the map.)

In the common case where all functions use the same Subtarget, this would actually lead to the same result as now.

This then eliminates another 3786 XC targets: there are no duplicates in any file, and at most I see 13 different XC targets in a single file :-)

Excellent!

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

7828

All these changes seem unnecessary now.

llvm/lib/Target/SystemZ/SystemZSelectionDAGInfo.cpp

142

The above suggestion actually makes sense, we don't need a else here.

llvm/lib/Target/SystemZ/SystemZTargetMachine.h

48 ↗

(On Diff #355400)

So your new "getGenericSubtarget" does exactly the same that the no-argument getSubtargetImpl used to do. This is a bit odd, in particular given the comment here.

If we do want to go that route, I think we should actually use the getSubtargetImpl name, but change the comment to explain what the "default" subtarget is, and what we use it for.

[jonpa]

Updated per review.

I was thinking of just using emitInt16 for the three words, that will show as .word in the textual assembler output, which may not be pretty but is certainly correct. (You could in addition use an emitRawComment to print a textual representation to make the file more readable).

Ah, ok, I did not realize at the time I could just print the instruction as numbers... I tried this instead now, since we don't really want to have a generic subtarget around. Is there any reason to print 3 x Int16? Would it be better to print it as one (or three) hex if possible?

In the meantime I thought of yet another option: the instructions used via EXRL are really part of the function containing the EXRL, and should therefore be emitted using the same Subtarget that is in effect for that function (this is not *currently* a problem, but it might be a potential issue if we want to use an EXRL target instruction that is only available in some ISA levels).

This seems like the logical solution, but it is a bit unfortunate to have to go through all of that trouble just for this case where it is actually not really needed at all.

Not sure if the best thing is to do it the manual way, or to have an extra subtarget map on the side in the SystemZAsmPrinter...

I wonder if perhaps the MC layer could have an emitInstruction() / assembleInstruction() method without the STI argument, that was only very basic that could work in cases like this... It would be nice to reuse the bits encoding already available... On the other hand if we only have EXRL targets of this one format, maybe it's ok to do it on the side...

[uweigand]

In D103865#2851145, @jonpa wrote:

Updated per review.

I was thinking of just using emitInt16 for the three words, that will show as .word in the textual assembler output, which may not be pretty but is certainly correct. (You could in addition use an emitRawComment to print a textual representation to make the file more readable).

Ah, ok, I did not realize at the time I could just print the instruction as numbers... I tried this instead now, since we don't really want to have a generic subtarget around. Is there any reason to print 3 x Int16? Would it be better to print it as one (or three) hex if possible?

Well, there is no emitInt48 ... You could use one emitInt16 and one emitInt32, but that's not really better I think.

I must admin that manual encoding code is longer and uglier than I had initially thought ...

In the meantime I thought of yet another option: the instructions used via EXRL are really part of the function containing the EXRL, and should therefore be emitted using the same Subtarget that is in effect for that function (this is not *currently* a problem, but it might be a potential issue if we want to use an EXRL target instruction that is only available in some ISA levels).

This seems like the logical solution, but it is a bit unfortunate to have to go through all of that trouble just for this case where it is actually not really needed at all.

I think it wouldn't be all that much effort. You should simply change EXRLT2SymMap to be a map from a pair of (const SystemZSubtarget *, MCInst) to MCSymbol *, add the current subtarget pointer to the key when processing SystemZ::EXRL_Pseudo in the AsmPrinter, and in emitEXRLTargetInstructions just iterate over the map (as now) -- you'll get the pairs of (const SystemZSubtarget *, MCInst) back as key, and simply emit that instruction using that subtarget.

Otherwise, this is now looking good to me.

[jonpa]

Well, there is no emitInt48 ...

(I saw emitIntValueInHex(uint64_t Value, unsigned Size), but haven't tried it...)

I must admin that manual encoding code is longer and uglier than I had initially thought ...
I think it wouldn't be all that much effort. You should simply change EXRLT2SymMap to be a map from a pair of (const SystemZSubtarget *, MCInst) to MCSymbol *, add the current subtarget pointer to the key when processing SystemZ::EXRL_Pseudo in the AsmPrinter, and in emitEXRLTargetInstructions just iterate over the map (as now) -- you'll get the pairs of (const SystemZSubtarget *, MCInst) back as key, and simply emit that instruction using that subtarget.

OK, I tried this now. It is in fact the identical number of target instructions to before, so using different Subtarget do not seem to play a role on SPEC...

[uweigand]

See the minor comment inside. Otherwise, this LGTM now. Thanks!

llvm/lib/Target/SystemZ/SystemZAsmPrinter.h

37

This comparison might involve undefined behavior given that these aren't pointers into the same array. Probably best to convert to uintptr_t before comparing.

[sylvestre.ledru]

@jonpa @uweigand
It seems that this change causes clang to crash on s390x on Ubuntu bionic:
https://bugs.llvm.org/show_bug.cgi?id=51026

llvm/lib/Target/SystemZ/SystemZAsmPrinter.h

 //===-- SystemZAsmPrinter.h - SystemZ LLVM assembly printer ----*- C++ -*--===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_SYSTEMZ_SYSTEMZASMPRINTER_H
 #define LLVM_LIB_TARGET_SYSTEMZ_SYSTEMZASMPRINTER_H
 
 #include "SystemZTargetMachine.h"
 #include "SystemZMCInstLower.h"
 #include "llvm/CodeGen/AsmPrinter.h"
 #include "llvm/CodeGen/StackMaps.h"
+#include "llvm/MC/MCInstBuilder.h"
 #include "llvm/Support/Compiler.h"
 
 namespace llvm {
 class MCStreamer;
 class MachineBasicBlock;
 class MachineInstr;
 class Module;
 class raw_ostream;
 
 class LLVM_LIBRARY_VISIBILITY SystemZAsmPrinter : public AsmPrinter {
 private:
   StackMaps SM;
+  typedef std::pair<MCSymbol*, MCInstBuilder> SymbInstPair;
+  std::vector<SymbInstPair> EXRL_Targets;
 
 public:
   SystemZAsmPrinter(TargetMachine &TM, std::unique_ptr<MCStreamer> Streamer)
       : AsmPrinter(TM, std::move(Streamer)), SM(*this) {}
 
   // Override AsmPrinter.
   StringRef getPassName() const override { return "SystemZ Assembly Printer"; }
   void emitInstruction(const MachineInstr *MI) override;
   void emitMachineConstantPoolValue(MachineConstantPoolValue *MCPV) override;
   void emitEndOfAsmFile(Module &M) override;
+  void emitFunctionBodyEnd() override;
   bool PrintAsmOperand(const MachineInstr *MI, unsigned OpNo,
                        const char *ExtraCode, raw_ostream &OS) override;
   bool PrintAsmMemoryOperand(const MachineInstr *MI, unsigned OpNo,
                              const char *ExtraCode, raw_ostream &OS) override;
 
   bool doInitialization(Module &M) override {
     SM.reset();
     return AsmPrinter::doInitialization(M);
   }
 
 private:
   void LowerFENTRY_CALL(const MachineInstr &MI, SystemZMCInstLower &MCIL);
   void LowerSTACKMAP(const MachineInstr &MI);
   void LowerPATCHPOINT(const MachineInstr &MI, SystemZMCInstLower &Lower);
+  void emitEXRLTargetInstructions();
 };
 } // end namespace llvm
 
 #endif

llvm/lib/Target/SystemZ/SystemZAsmPrinter.cpp

#undef LOWER_HIGH
   case TargetOpcode::STACKMAP:
     LowerSTACKMAP(*MI);
     return;
 
   case TargetOpcode::PATCHPOINT:
     LowerPATCHPOINT(*MI, Lower);
     return;
 
+  case SystemZ::EXRL_Pseudo: {
+    unsigned TargetInsOpc = MI->getOperand(0).getImm();
+    Register LenMinus1Reg = MI->getOperand(1).getReg();
+    Register DestReg = MI->getOperand(2).getReg();
+    uint64_t DestDisp = MI->getOperand(3).getImm();
+    Register SrcReg = MI->getOperand(4).getReg();
+    uint64_t SrcDisp = MI->getOperand(5).getImm();
+
+    MCSymbol *DotSym = OutContext.createTempSymbol();
+    const MCSymbolRefExpr *Dot = MCSymbolRefExpr::create(DotSym, OutContext);
+    EmitToStreamer(*OutStreamer, MCInstBuilder(SystemZ::EXRL)
+                   .addReg(LenMinus1Reg).addExpr(Dot));
+    EXRL_Targets.push_back(SymbInstPair(DotSym,
+      MCInstBuilder(TargetInsOpc).addReg(DestReg).addImm(DestDisp).addImm(1)
+                                 .addReg(SrcReg).addImm(SrcDisp)));
+    }
+    return;
+
   default:
     Lower.lower(MI, LoweredMI);
     break;
   }
   EmitToStreamer(*OutStreamer, LoweredMI);
 }
 
 // Emit the largest nop instruction smaller than or equal to NumBytes
 // bytes.  Return the size of nop emitted.
 static unsigned EmitNop(MCContext &OutContext, MCStreamer &OutStreamer,
                         unsigned NumBytes, const MCSubtargetInfo &STI) {
   if (NumBytes < 2) {
     llvm_unreachable("Zero nops?");
     return 0;
   }
assert(NumBytes >= EncodedBytes &&
          "Patchpoint can't request size less than the length of a call.");
   assert((NumBytes - EncodedBytes) % 2 == 0 &&
          "Invalid number of NOP bytes requested!");
   while (EncodedBytes < NumBytes)
     EncodedBytes += EmitNop(OutContext, *OutStreamer, NumBytes - EncodedBytes,
                             getSubtargetInfo());
 }
 
+void SystemZAsmPrinter::emitEXRLTargetInstructions() {
+  for (auto I : EXRL_Targets) {
+    MCSymbol *DotSym = I.first;
+    MCInstBuilder &MCI = I.second;
+    OutStreamer->emitLabel(DotSym);
+    OutStreamer->emitInstruction(MCI, getSubtargetInfo());
+  }
+  EXRL_Targets.clear();
+}
+
 // Convert a SystemZ-specific constant pool modifier into the associated
 // MCSymbolRefExpr variant kind.
 static MCSymbolRefExpr::VariantKind
 getModifierVariantKind(SystemZCP::SystemZCPModifier Modifier) {
   switch (Modifier) {
   case SystemZCP::TLSGD: return MCSymbolRefExpr::VK_TLSGD;
   case SystemZCP::TLSLDM: return MCSymbolRefExpr::VK_TLSLDM;
   case SystemZCP::DTPOFF: return MCSymbolRefExpr::VK_DTPOFF;
SystemZInstPrinter::printAddress(MAI, MI->getOperand(OpNo).getReg(),
                                    MI->getOperand(OpNo + 2).getReg(), OS);
   return false;
 }
 
 void SystemZAsmPrinter::emitEndOfAsmFile(Module &M) {
   emitStackMaps(SM);
 }
 
+void SystemZAsmPrinter::emitFunctionBodyEnd() {
+  emitEXRLTargetInstructions();
+}
+
 // Force static initialization.
 extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeSystemZAsmPrinter() {
   RegisterAsmPrinter<SystemZAsmPrinter> X(getTheSystemZTarget());
 }

llvm/lib/Target/SystemZ/SystemZISelLowering.h

MachineBasicBlock *emitAtomicLoadMinMax(MachineInstr &MI,
                                           MachineBasicBlock *MBB,
                                           unsigned CompareOpcode,
                                           unsigned KeepOldMask,
                                           unsigned BitSize) const;
   MachineBasicBlock *emitAtomicCmpSwapW(MachineInstr &MI,
                                         MachineBasicBlock *BB) const;
   MachineBasicBlock *emitMemMemWrapper(MachineInstr &MI, MachineBasicBlock *BB,
                                        unsigned Opcode) const;
+  MachineBasicBlock *emitMemMemLoopVarLen(MachineInstr &MI,
+                                          MachineBasicBlock *BB,
+                                          unsigned Opcode) const;
   MachineBasicBlock *emitStringWrapper(MachineInstr &MI, MachineBasicBlock *BB,
                                        unsigned Opcode) const;
   MachineBasicBlock *emitTransactionBegin(MachineInstr &MI,
                                           MachineBasicBlock *MBB,
                                           unsigned Opcode, bool NoFloat) const;
   MachineBasicBlock *emitLoadAndTestCmp0(MachineInstr &MI,
                                          MachineBasicBlock *MBB,
                                          unsigned Opcode) const;

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

SystemZTargetLowering::getStackProbeSize(MachineFunction &MF) const {
   StackProbeSize &= ~(StackAlign - 1);
   return StackProbeSize ? StackProbeSize : StackAlign;
 }
 
 //===----------------------------------------------------------------------===//
 // Custom insertion
 //===----------------------------------------------------------------------===//
 
-// Force base value Base into a register before MI.  Return the register.
-static Register forceReg(MachineInstr &MI, MachineOperand &Base,
-                         const SystemZInstrInfo *TII) {
-  if (Base.isReg())
-    return Base.getReg();
-
-  MachineBasicBlock *MBB = MI.getParent();
-  MachineFunction &MF = *MBB->getParent();
-  MachineRegisterInfo &MRI = MF.getRegInfo();
-
-  Register Reg = MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
-  BuildMI(*MBB, MI, MI.getDebugLoc(), TII->get(SystemZ::LA), Reg)
-      .add(Base)
-      .addImm(0)
-      .addReg(0);
-  return Reg;
-}
-
 // The CC operand of MI might be missing a kill marker because there
 // were multiple uses of CC, and ISel didn't know which to mark.
 // Figure out whether MI should have had a kill marker.
 static bool checkCCKill(MachineInstr &MI, MachineBasicBlock *MBB) {
   // Scan forward through BB for a use/def of CC.
   MachineBasicBlock::iterator miI(std::next(MachineBasicBlock::iterator(MI)));
   for (MachineBasicBlock::iterator miE = MBB->end(); miI != miE; ++miI) {
     const MachineInstr& mi = *miI;
MachineBasicBlock *SystemZTargetLowering::emitAtomicLoadBinary(
   // Insert a basic block for the main loop.
   MachineBasicBlock *StartMBB = MBB;
   MachineBasicBlock *DoneMBB  = SystemZ::splitBlockBefore(MI, MBB);
   MachineBasicBlock *LoopMBB  = SystemZ::emitBlockAfter(StartMBB);
 
   //  StartMBB:
   //   ...
   //   %OrigVal = L Disp(%Base)
-  //   # fall through to LoopMMB
+  //   # fall through to LoopMBB
   MBB = StartMBB;
   BuildMI(MBB, DL, TII->get(LOpcode), OrigVal).add(Base).addImm(Disp).addReg(0);
   MBB->addSuccessor(LoopMBB);
 
   //  LoopMBB:
   //   %OldVal        = phi [ %OrigVal, StartMBB ], [ %Dest, LoopMBB ]
   //   %RotatedOldVal = RLL %OldVal, 0(%BitShift)
   //   %RotatedNewVal = OP %RotatedOldVal, %Src2
   //   %NewVal        = RLL %RotatedNewVal, 0(%NegBitShift)
   //   %Dest          = CS %OldVal, %NewVal, Disp(%Base)
   //   JNE LoopMBB
-  //   # fall through to DoneMMB
+  //   # fall through to DoneMBB
   MBB = LoopMBB;
   BuildMI(MBB, DL, TII->get(SystemZ::PHI), OldVal)
     .addReg(OrigVal).addMBB(StartMBB)
     .addReg(Dest).addMBB(LoopMBB);
   if (IsSubWord)
     BuildMI(MBB, DL, TII->get(SystemZ::RLL), RotatedOldVal)
       .addReg(OldVal).addReg(BitShift).addImm(0);
   if (Invert) {
MachineBasicBlock *SystemZTargetLowering::emitAtomicLoadMinMax(
   MachineBasicBlock *DoneMBB   = SystemZ::splitBlockBefore(MI, MBB);
   MachineBasicBlock *LoopMBB   = SystemZ::emitBlockAfter(StartMBB);
   MachineBasicBlock *UseAltMBB = SystemZ::emitBlockAfter(LoopMBB);
   MachineBasicBlock *UpdateMBB = SystemZ::emitBlockAfter(UseAltMBB);
 
   //  StartMBB:
   //   ...
   //   %OrigVal     = L Disp(%Base)
-  //   # fall through to LoopMMB
+  //   # fall through to LoopMBB
   MBB = StartMBB;
   BuildMI(MBB, DL, TII->get(LOpcode), OrigVal).add(Base).addImm(Disp).addReg(0);
   MBB->addSuccessor(LoopMBB);
 
   //  LoopMBB:
   //   %OldVal        = phi [ %OrigVal, StartMBB ], [ %Dest, UpdateMBB ]
   //   %RotatedOldVal = RLL %OldVal, 0(%BitShift)
   //   CompareOpcode %RotatedOldVal, %Src2
BuildMI(MBB, DL, TII->get(CompareOpcode))
     .addReg(RotatedOldVal).addReg(Src2);
   BuildMI(MBB, DL, TII->get(SystemZ::BRC))
     .addImm(SystemZ::CCMASK_ICMP).addImm(KeepOldMask).addMBB(UpdateMBB);
   MBB->addSuccessor(UpdateMBB);
   MBB->addSuccessor(UseAltMBB);
 
   //  UseAltMBB:
   //   %RotatedAltVal = RISBG %RotatedOldVal, %Src2, 32, 31 + BitSize, 0
-  //   # fall through to UpdateMMB
+  //   # fall through to UpdateMBB
   MBB = UseAltMBB;
   if (IsSubWord)
     BuildMI(MBB, DL, TII->get(SystemZ::RISBG32), RotatedAltVal)
       .addReg(RotatedOldVal).addReg(Src2)
       .addImm(32).addImm(31 + BitSize).addImm(0);
   MBB->addSuccessor(UpdateMBB);
 
   //  UpdateMBB:
   //   %RotatedNewVal = PHI [ %RotatedOldVal, LoopMBB ],
   //                        [ %RotatedAltVal, UseAltMBB ]
   //   %NewVal        = RLL %RotatedNewVal, 0(%NegBitShift)
   //   %Dest          = CS %OldVal, %NewVal, Disp(%Base)
   //   JNE LoopMBB
-  //   # fall through to DoneMMB
+  //   # fall through to DoneMBB
   MBB = UpdateMBB;
   BuildMI(MBB, DL, TII->get(SystemZ::PHI), RotatedNewVal)
     .addReg(RotatedOldVal).addMBB(LoopMBB)
     .addReg(RotatedAltVal).addMBB(UseAltMBB);
   if (IsSubWord)
     BuildMI(MBB, DL, TII->get(SystemZ::RLL), NewVal)
       .addReg(RotatedNewVal).addReg(NegBitShift).addImm(0);
   BuildMI(MBB, DL, TII->get(CSOpcode), Dest)
SystemZTargetLowering::emitAtomicCmpSwapW(MachineInstr &MI,
   MachineBasicBlock *StartMBB = MBB;
   MachineBasicBlock *DoneMBB  = SystemZ::splitBlockBefore(MI, MBB);
   MachineBasicBlock *LoopMBB  = SystemZ::emitBlockAfter(StartMBB);
   MachineBasicBlock *SetMBB   = SystemZ::emitBlockAfter(LoopMBB);
 
   //  StartMBB:
   //   ...
   //   %OrigOldVal     = L Disp(%Base)
-  //   # fall through to LoopMMB
+  //   # fall through to LoopMBB
   MBB = StartMBB;
   BuildMI(MBB, DL, TII->get(LOpcode), OrigOldVal)
       .add(Base)
       .addImm(Disp)
       .addReg(0);
   MBB->addSuccessor(LoopMBB);
 
   //  LoopMBB:
SystemZTargetLowering::emitAtomicCmpSwapW(MachineInstr &MI,
   MBB->addSuccessor(DoneMBB);
   MBB->addSuccessor(SetMBB);
 
   //  SetMBB:
   //   %StoreVal     = RLL %RetrySwapVal, -BitSize(%NegBitShift)
   //                      ^^ Rotate the new field to its proper position.
   //   %RetryOldVal  = CS %OldVal, %StoreVal, Disp(%Base)
   //   JNE LoopMBB
-  //   # fall through to ExitMMB
+  //   # fall through to ExitMBB
   MBB = SetMBB;
   BuildMI(MBB, DL, TII->get(SystemZ::RLL), StoreVal)
     .addReg(RetrySwapVal).addReg(NegBitShift).addImm(-BitSize);
   BuildMI(MBB, DL, TII->get(CSOpcode), RetryOldVal)
       .addReg(OldVal)
       .addReg(StoreVal)
       .add(Base)
       .addImm(Disp);
MachineBasicBlock *SystemZTargetLowering::emitExt128(MachineInstr &MI,
   }
   BuildMI(*MBB, MI, DL, TII->get(TargetOpcode::INSERT_SUBREG), Dest)
     .addReg(In128).addReg(Src).addImm(SystemZ::subreg_l64);
 
   MI.eraseFromParent();
   return MBB;
 }
 
-MachineBasicBlock *SystemZTargetLowering::emitMemMemWrapper(
-    MachineInstr &MI, MachineBasicBlock *MBB, unsigned Opcode) const {
-  MachineFunction &MF = *MBB->getParent();
-  const SystemZInstrInfo *TII =
-      static_cast<const SystemZInstrInfo *>(Subtarget.getInstrInfo());
-  MachineRegisterInfo &MRI = MF.getRegInfo();
+struct MemMemBuilder {
+  MachineOperand &DestBase;
+  uint64_t &DestDisp;
+  MachineOperand &SrcBase;
+  uint64_t &SrcDisp;
+
+  bool HaveSingleBase;
+  Register StartCountReg, ThisCountReg, NextCountReg;
+  Register StartSrcReg,   ThisSrcReg,   NextSrcReg;
+  Register StartDestReg,  ThisDestReg,  NextDestReg;
+  MemMemBuilder(MachineOperand &DestBase, uint64_t &DestDisp,
+                MachineOperand &SrcBase, uint64_t &SrcDisp) :
+    DestBase(DestBase), DestDisp(DestDisp), SrcBase(SrcBase), SrcDisp(SrcDisp) {}
+
+  void buildMemMemLoop(Register TripCountReg, unsigned Opcode, MachineInstr &MI,
+                       MachineBasicBlock *StartMBB, MachineBasicBlock *LoopMBB,
+                       MachineBasicBlock *NextMBB, MachineBasicBlock *EndMBB,
+                       MachineBasicBlock *DoneMBB, const SystemZInstrInfo *TII) {
+    MachineRegisterInfo &MRI = StartMBB->getParent()->getRegInfo();
     DebugLoc DL = MI.getDebugLoc();
 
-  MachineOperand DestBase = earlyUseOperand(MI.getOperand(0));
-  uint64_t DestDisp = MI.getOperand(1).getImm();
-  MachineOperand SrcBase = earlyUseOperand(MI.getOperand(2));
-  uint64_t SrcDisp = MI.getOperand(3).getImm();
-  uint64_t Length = MI.getOperand(4).getImm();
-
-  // When generating more than one CLC, all but the last will need to
-  // branch to the end when a difference is found.
-  MachineBasicBlock *EndMBB = (Length > 256 && Opcode == SystemZ::CLC ?
-                               SystemZ::splitBlockAfter(MI, MBB) : nullptr);
-
-  // Check for the loop form, in which operand 5 is the trip count.
-  if (MI.getNumExplicitOperands() > 5) {
-    bool HaveSingleBase = DestBase.isIdenticalTo(SrcBase);
+    // Force base value Base into a register before MI.  Return the register.
+    auto forceReg = [&](MachineOperand &Base) -> Register {
+      if (Base.isReg())
+        return Base.getReg();
+      Register Reg = MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
+      BuildMI(*StartMBB, StartMBB->getFirstTerminator(), DL, TII->get(SystemZ::LA), Reg)
+        .add(Base).addImm(0).addReg(0);
+      return Reg;
+    };
 
-    Register StartCountReg = MI.getOperand(5).getReg();
-    Register StartSrcReg   = forceReg(MI, SrcBase, TII);
-    Register StartDestReg  = (HaveSingleBase ? StartSrcReg :
-                              forceReg(MI, DestBase, TII));
+    StartCountReg = TripCountReg;
+    HaveSingleBase = DestBase.isIdenticalTo(SrcBase);
+    StartSrcReg  = forceReg(SrcBase);
+    StartDestReg = (HaveSingleBase ? StartSrcReg : forceReg(DestBase));
 
     const TargetRegisterClass *RC = &SystemZ::ADDR64BitRegClass;
-    Register ThisSrcReg  = MRI.createVirtualRegister(RC);
-    Register ThisDestReg = (HaveSingleBase ? ThisSrcReg :
-                            MRI.createVirtualRegister(RC));
-    Register NextSrcReg  = MRI.createVirtualRegister(RC);
-    Register NextDestReg = (HaveSingleBase ? NextSrcReg :
-                            MRI.createVirtualRegister(RC));
+    ThisSrcReg  = MRI.createVirtualRegister(RC);
+    ThisDestReg = (HaveSingleBase ? ThisSrcReg : MRI.createVirtualRegister(RC));
+    NextSrcReg  = MRI.createVirtualRegister(RC);
+    NextDestReg = (HaveSingleBase ? NextSrcReg : MRI.createVirtualRegister(RC));
 
     RC = &SystemZ::GR64BitRegClass;
-    Register ThisCountReg = MRI.createVirtualRegister(RC);
-    Register NextCountReg = MRI.createVirtualRegister(RC);
-
-    MachineBasicBlock *StartMBB = MBB;
-    MachineBasicBlock *DoneMBB = SystemZ::splitBlockBefore(MI, MBB);
-    MachineBasicBlock *LoopMBB = SystemZ::emitBlockAfter(StartMBB);
-    MachineBasicBlock *NextMBB =
-        (EndMBB ? SystemZ::emitBlockAfter(LoopMBB) : LoopMBB);
-
-    //  StartMBB:
-    //   # fall through to LoopMMB
-    MBB->addSuccessor(LoopMBB);
+    ThisCountReg = MRI.createVirtualRegister(RC);
+    NextCountReg = MRI.createVirtualRegister(RC);
 
     //  LoopMBB:
     //   %ThisDestReg = phi [ %StartDestReg, StartMBB ],
     //                      [ %NextDestReg, NextMBB ]
     //   %ThisSrcReg = phi [ %StartSrcReg, StartMBB ],
     //                     [ %NextSrcReg, NextMBB ]
     //   %ThisCountReg = phi [ %StartCountReg, StartMBB ],
     //                       [ %NextCountReg, NextMBB ]
     //   ( PFD 2, 768+DestDisp(%ThisDestReg) )
     //   Opcode DestDisp(256,%ThisDestReg), SrcDisp(%ThisSrcReg)
     //   ( JLH EndMBB )
     //
     // The prefetch is used only for MVC.  The JLH is used only for CLC.
-    MBB = LoopMBB;
+    MachineBasicBlock *MBB = LoopMBB;
 
     BuildMI(MBB, DL, TII->get(SystemZ::PHI), ThisDestReg)
       .addReg(StartDestReg).addMBB(StartMBB)
       .addReg(NextDestReg).addMBB(NextMBB);
     if (!HaveSingleBase)
       BuildMI(MBB, DL, TII->get(SystemZ::PHI), ThisSrcReg)
         .addReg(StartSrcReg).addMBB(StartMBB)
         .addReg(NextSrcReg).addMBB(NextMBB);
     BuildMI(MBB, DL, TII->get(SystemZ::PHI), ThisCountReg)
       .addReg(StartCountReg).addMBB(StartMBB)
void buildMemMemLoop(Register TripCountReg, unsigned Opcode, MachineInstr &MI,
     }
 
     // NextMBB:
     //   %NextDestReg = LA 256(%ThisDestReg)
     //   %NextSrcReg = LA 256(%ThisSrcReg)
     //   %NextCountReg = AGHI %ThisCountReg, -1
     //   CGHI %NextCountReg, 0
     //   JLH LoopMBB
-    //   # fall through to DoneMMB
+    //   # fall through to DoneMBB
     //
     // The AGHI, CGHI and JLH should be converted to BRCTG by later passes.
     MBB = NextMBB;
 
     BuildMI(MBB, DL, TII->get(SystemZ::LA), NextDestReg)
       .addReg(ThisDestReg).addImm(256).addReg(0);
     if (!HaveSingleBase)
       BuildMI(MBB, DL, TII->get(SystemZ::LA), NextSrcReg)
         .addReg(ThisSrcReg).addImm(256).addReg(0);
     BuildMI(MBB, DL, TII->get(SystemZ::AGHI), NextCountReg)
       .addReg(ThisCountReg).addImm(-1);
     BuildMI(MBB, DL, TII->get(SystemZ::CGHI))
       .addReg(NextCountReg).addImm(0);
     BuildMI(MBB, DL, TII->get(SystemZ::BRC))
       .addImm(SystemZ::CCMASK_ICMP).addImm(SystemZ::CCMASK_CMP_NE)
       .addMBB(LoopMBB);
     MBB->addSuccessor(LoopMBB);
     MBB->addSuccessor(DoneMBB);
+  }
+};
 
-    DestBase = MachineOperand::CreateReg(NextDestReg, false);
-    SrcBase = MachineOperand::CreateReg(NextSrcReg, false);
+MachineBasicBlock *SystemZTargetLowering::emitMemMemWrapper(
+    MachineInstr &MI, MachineBasicBlock *MBB, unsigned Opcode) const {
+  MachineFunction &MF = *MBB->getParent();
+  const SystemZInstrInfo *TII =
+      static_cast<const SystemZInstrInfo *>(Subtarget.getInstrInfo());
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+  DebugLoc DL = MI.getDebugLoc();
+
+  MachineOperand DestBase = earlyUseOperand(MI.getOperand(0));
+  uint64_t DestDisp = MI.getOperand(1).getImm();
+  MachineOperand SrcBase = earlyUseOperand(MI.getOperand(2));
+  uint64_t SrcDisp = MI.getOperand(3).getImm();
+  uint64_t Length = MI.getOperand(4).getImm();
+  MemMemBuilder MemMB(DestBase, DestDisp, SrcBase, SrcDisp);
+
+  // When generating more than one CLC, all but the last will need to
+  // branch to the end when a difference is found.
+  MachineBasicBlock *EndMBB = (Length > 256 && Opcode == SystemZ::CLC ?
+                               SystemZ::splitBlockAfter(MI, MBB) : nullptr);
+
+  // Check for the loop form, in which operand 5 is the trip count.
+  if (MI.getNumExplicitOperands() > 5) {
+    MachineBasicBlock *StartMBB = MBB;
+    MachineBasicBlock *DoneMBB = SystemZ::splitBlockBefore(MI, MBB);
+    MachineBasicBlock *LoopMBB = SystemZ::emitBlockAfter(StartMBB);
+    MachineBasicBlock *NextMBB =
+        (EndMBB ? SystemZ::emitBlockAfter(LoopMBB) : LoopMBB);
+
+    //  StartMBB:
+    //   # fall through to LoopMBB
+    MBB->addSuccessor(LoopMBB);
+
+    Register StartCountReg = MI.getOperand(5).getReg();
+    MemMB.buildMemMemLoop(StartCountReg, Opcode, MI, StartMBB, LoopMBB, NextMBB,
+                          EndMBB, DoneMBB, TII);
+
+    DestBase = MachineOperand::CreateReg(MemMB.NextDestReg, false);
+    SrcBase = MachineOperand::CreateReg(MemMB.NextSrcReg, false);
     Length &= 255;
     if (EndMBB && !Length)
       // If the loop handled the whole CLC range, DoneMBB will be empty with
       // CC live-through into EndMBB, so add it as live-in.
       DoneMBB->addLiveIn(SystemZ::CC);
     MBB = DoneMBB;
   }
   // Handle any remaining bytes with straight-line code.
if (EndMBB) {
     MBB = EndMBB;
     MBB->addLiveIn(SystemZ::CC);
   }
 
   MI.eraseFromParent();
   return MBB;
 }
 
+MachineBasicBlock *SystemZTargetLowering::emitMemMemLoopVarLen(
+    MachineInstr &MI, MachineBasicBlock *MBB, unsigned Opcode) const {
+  MachineFunction &MF = *MBB->getParent();
+  const SystemZInstrInfo *TII =
+      static_cast<const SystemZInstrInfo *>(Subtarget.getInstrInfo());
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+  DebugLoc DL = MI.getDebugLoc();
+
+  MachineOperand DestBase = earlyUseOperand(MI.getOperand(0));
+  uint64_t DestDisp = MI.getOperand(1).getImm();
+  MachineOperand SrcBase = earlyUseOperand(MI.getOperand(2));
+  uint64_t SrcDisp = MI.getOperand(3).getImm();
+  Register Length = MI.getOperand(4).getReg();
+  MemMemBuilder MemMB(DestBase, DestDisp, SrcBase, SrcDisp);
+  assert(MI.getOperand(5).getImm() == 0 && "This operand is zero and unused.");
+
+  MachineBasicBlock *DoneMBB = SystemZ::splitBlockBefore(MI, MBB);
+  MachineBasicBlock *PreheaderMBB = SystemZ::emitBlockAfter(MBB);
+  MachineBasicBlock *LoopMBB = SystemZ::emitBlockAfter(PreheaderMBB);
+  MachineBasicBlock *ExeMBB = SystemZ::emitBlockAfter(LoopMBB);
+
+  //  MBB:
+  //   # Jump to DoneMBB if Length is zero, or fall through to PreheaderMBB.
+  BuildMI(MBB, DL, TII->get(SystemZ::CGHI))
+    .addReg(Length).addImm(0);
+  BuildMI(MBB, DL, TII->get(SystemZ::BRC))
+    .addImm(SystemZ::CCMASK_ICMP).addImm(SystemZ::CCMASK_CMP_EQ)
+    .addMBB(DoneMBB);
+  MBB->addSuccessor(DoneMBB);
+  MBB->addSuccessor(PreheaderMBB);
+
+  // PreheaderMBB:
+  // %LenMinus1 = Length - 1 (see below)
+  // %TripC = %LenMinus1 / 256
+  // # Jump to ExeMBB if %TripC is zero, or fall through to LoopMBB.
+  Register LenMinus1 = MRI.createVirtualRegister(&SystemZ::GR64BitRegClass);
+  Register TripC = MRI.createVirtualRegister(&SystemZ::GR64BitRegClass);
+  MBB = PreheaderMBB;
+  BuildMI(MBB, DL, TII->get(SystemZ::AGHI), LenMinus1)
+    .addReg(Length).addImm(-1);
+  BuildMI(MBB, DL, TII->get(SystemZ::SRLG), TripC)
+    .addReg(LenMinus1).addReg(0).addImm(8);
+  BuildMI(MBB, DL, TII->get(SystemZ::CGHI))
+    .addReg(TripC).addImm(0);
+  BuildMI(MBB, DL, TII->get(SystemZ::BRC))
+    .addImm(SystemZ::CCMASK_ICMP).addImm(SystemZ::CCMASK_CMP_EQ)
+    .addMBB(ExeMBB);
+  MBB->addSuccessor(ExeMBB);
+  MBB->addSuccessor(LoopMBB);
+
+  // LoopMBB:
+  // Do %TripC iterations of 256-byte chunks.
+  MemMB.buildMemMemLoop(TripC, Opcode, MI, PreheaderMBB, LoopMBB, LoopMBB,
+                        nullptr, ExeMBB, TII);
+
+  // ExeMBB:
+  // # Make PHIs for RemDestReg/RemSrcReg since the loop may or may not run.
+  // # Use EXecute Relative Long for the remainder of the bytes. The target
+  //   instruction of the EXRL will have a length field of 1 since 0 is an
+  //   illegal value. The number of bytes processed becomes (%LenMinus1 &
+  //   0xff) + 1.
+  // # Fall through to DoneMBB.
+  Register RemSrcReg  = MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
+  Register RemDestReg = MemMB.HaveSingleBase ? RemSrcReg :
+    MRI.createVirtualRegister(&SystemZ::ADDR64BitRegClass);
+  MBB = ExeMBB;
+  BuildMI(MBB, DL, TII->get(SystemZ::PHI), RemDestReg)
+    .addReg(MemMB.StartDestReg).addMBB(PreheaderMBB)
+    .addReg(MemMB.NextDestReg).addMBB(LoopMBB);
+  if (!MemMB.HaveSingleBase)
+    BuildMI(MBB, DL, TII->get(SystemZ::PHI), RemSrcReg)
+      .addReg(MemMB.StartSrcReg).addMBB(PreheaderMBB)
+      .addReg(MemMB.NextSrcReg).addMBB(LoopMBB);
+  BuildMI(MBB, DL, TII->get(SystemZ::EXRL_Pseudo))
+    .addImm(Opcode)
+    .addReg(LenMinus1)
+    .addReg(RemDestReg).addImm(DestDisp)
+    .addReg(RemSrcReg).addImm(SrcDisp);
+  MBB->addSuccessor(DoneMBB);
+
+  MI.eraseFromParent();
+  return DoneMBB;
+}
+
 // Decompose string pseudo-instruction MI into a loop that continually performs
 // Opcode until CC != 3.
 MachineBasicBlock *SystemZTargetLowering::emitStringWrapper(
     MachineInstr &MI, MachineBasicBlock *MBB, unsigned Opcode) const {
   MachineFunction &MF = *MBB->getParent();
   const SystemZInstrInfo *TII =
       static_cast<const SystemZInstrInfo *>(Subtarget.getInstrInfo());
   MachineRegisterInfo &MRI = MF.getRegInfo();
MachineBasicBlock *SystemZTargetLowering::emitStringWrapper(
   uint64_t This2Reg = MRI.createVirtualRegister(RC);
   uint64_t End2Reg  = MRI.createVirtualRegister(RC);
 
   MachineBasicBlock *StartMBB = MBB;
   MachineBasicBlock *DoneMBB = SystemZ::splitBlockBefore(MI, MBB);
   MachineBasicBlock *LoopMBB = SystemZ::emitBlockAfter(StartMBB);
 
   //  StartMBB:
-  //   # fall through to LoopMMB
+  //   # fall through to LoopMBB
   MBB->addSuccessor(LoopMBB);
 
   //  LoopMBB:
   //   %This1Reg = phi [ %Start1Reg, StartMBB ], [ %End1Reg, LoopMBB ]
   //   %This2Reg = phi [ %Start2Reg, StartMBB ], [ %End2Reg, LoopMBB ]
   //   R0L = %CharReg
   //   %End1Reg, %End2Reg = CLST %This1Reg, %This2Reg -- uses R0L
   //   JO LoopMBB
-  //   # fall through to DoneMMB
+  //   # fall through to DoneMBB
   //
   // The load of R0L can be hoisted by post-RA LICM.
   MBB = LoopMBB;
 
   BuildMI(MBB, DL, TII->get(SystemZ::PHI), This1Reg)
     .addReg(Start1Reg).addMBB(StartMBB)
     .addReg(End1Reg).addMBB(LoopMBB);
   BuildMI(MBB, DL, TII->get(SystemZ::PHI), This2Reg)
MachineBasicBlock *SystemZTargetLowering::EmitInstrWithCustomInserter(
   case SystemZ::NCLoop:
     return emitMemMemWrapper(MI, MBB, SystemZ::NC);
   case SystemZ::OCSequence:
   case SystemZ::OCLoop:
     return emitMemMemWrapper(MI, MBB, SystemZ::OC);
   case SystemZ::XCSequence:
   case SystemZ::XCLoop:
     return emitMemMemWrapper(MI, MBB, SystemZ::XC);
+  case SystemZ::XCLoopVarLen:
+    return emitMemMemLoopVarLen(MI, MBB, SystemZ::XC);
   case SystemZ::CLCSequence:
   case SystemZ::CLCLoop:
     return emitMemMemWrapper(MI, MBB, SystemZ::CLC);
   case SystemZ::CLSTLoop:
     return emitStringWrapper(MI, MBB, SystemZ::CLST);
   case SystemZ::MVSTLoop:
     return emitStringWrapper(MI, MBB, SystemZ::MVST);
   case SystemZ::SRSTLoop:

llvm/lib/Target/SystemZ/SystemZInstrFormats.td

multiclass CondUnaryRSYPseudoAndMemFold<string mnemonic,
   def _MemFoldPseudo : MemFoldPseudo_CondMove<mnemonic, cls, bytes, mode>;
 }
 
 // Define an instruction that operates on two fixed-length blocks of memory,
 // and associated pseudo instructions for operating on blocks of any size.
 // The Sequence form uses a straight-line sequence of instructions and
 // the Loop form uses a loop of length-256 instructions followed by
 // another instruction to handle the excess.
+// The LoopVarLen form is for a loop with a non-constant length parameter.
 multiclass MemorySS<string mnemonic, bits<8> opcode,
                     SDPatternOperator sequence, SDPatternOperator loop> {
   def "" : SideEffectBinarySSa<mnemonic, opcode>;
   let usesCustomInserter = 1, hasNoSchedulingInfo = 1, Defs = [CC] in {
     def Sequence : Pseudo<(outs), (ins bdaddr12only:$dest, bdaddr12only:$src,
                                        imm64:$length),
                            [(sequence bdaddr12only:$dest, bdaddr12only:$src,
                                       imm64:$length)]>;
     def Loop : Pseudo<(outs), (ins bdaddr12only:$dest, bdaddr12only:$src,
                                    imm64:$length, GR64:$count256),
                       [(loop bdaddr12only:$dest, bdaddr12only:$src,
                              imm64:$length, GR64:$count256)]>;
+    def LoopVarLen : Pseudo<(outs), (ins bdaddr12only:$dest, bdaddr12only:$src,
+                                         GR64:$length, imm64:$zero),
+                            [(loop bdaddr12only:$dest, bdaddr12only:$src,
+                                   GR64:$length, imm64:$zero)]>;
   }
 }
 
 // The same, but setting a CC result as comparison operator.
 multiclass CompareMemorySS<string mnemonic, bits<8> opcode,
                           SDPatternOperator sequence, SDPatternOperator loop> {
   def "" : SideEffectBinarySSa<mnemonic, opcode>;
   let usesCustomInserter = 1, hasNoSchedulingInfo = 1 in {

llvm/lib/Target/SystemZ/SystemZInstrInfo.td

let Predicates = [FeatureDeflateConversion],
     mayLoad = 1, mayStore = 1, Defs = [CC], Uses = [R0L, R1D] in
   def DFLTCC : SideEffectTernaryMemMemRRFa<"dfltcc", 0xB939,
                                            GR128, GR128, GR64>;
 
 // Execute.
 let hasSideEffects = 1 in {
   def EX   : SideEffectBinaryRX<"ex", 0x44, GR64>;
   def EXRL : SideEffectBinaryRILPC<"exrl", 0xC60, GR64>;
+  let hasNoSchedulingInfo = 1 in
+    def EXRL_Pseudo : Pseudo<(outs), (ins i64imm:$TargetOpc, GR64:$lenMinus1,
+                                          bdaddr12only:$bdl1, bdaddr12only:$bd2),
+                                          []>;
 }
 
 //===----------------------------------------------------------------------===//
 // .insn directive instructions
 //===----------------------------------------------------------------------===//
 
 let isCodeGenOnly = 1, hasSideEffects = 1 in {
   def InsnE   : DirectiveInsnE<(outs), (ins imm64zx16:$enc), ".insn e,$enc", []>;

llvm/lib/Target/SystemZ/SystemZSelectionDAGInfo.cpp

SDValue SystemZSelectionDAGInfo::EmitTargetCodeForMemset(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Dst,
     SDValue Byte, SDValue Size, Align Alignment, bool IsVolatile,
     MachinePointerInfo DstPtrInfo) const {
   EVT PtrVT = Dst.getValueType();
 
   if (IsVolatile)
     return SDValue();
 
+  auto *CByte = dyn_cast<ConstantSDNode>(Byte);
   if (auto *CSize = dyn_cast<ConstantSDNode>(Size)) {
     uint64_t Bytes = CSize->getZExtValue();
     if (Bytes == 0)
       return SDValue();
-    if (auto *CByte = dyn_cast<ConstantSDNode>(Byte)) {
+    if (CByte) {
       // Handle cases that can be done using at most two of
       // MVI, MVHI, MVHHI and MVGHI.  The latter two can only be
       // used if ByteVal is all zeros or all ones; in other casees,
       // we can move at most 2 halfwords.
       uint64_t ByteVal = CByte->getZExtValue();
       if (ByteVal == 0 || ByteVal == 255 ?
           Bytes <= 16 && countPopulation(Bytes) <= 2 :
           Bytes <= 4) {
if (CByte) {
         SDValue Chain2 = DAG.getStore(Chain, DL, Byte, Dst2,
                                       DstPtrInfo.getWithOffset(1), Align(1));
         return DAG.getNode(ISD::TokenFactor, DL, MVT::Other, Chain1, Chain2);
       }
     }
     assert(Bytes >= 2 && "Should have dealt with 0- and 1-byte cases already");
 
     // Handle the special case of a memset of 0, which can use XC.
-    auto *CByte = dyn_cast<ConstantSDNode>(Byte);
     if (CByte && CByte->getZExtValue() == 0)
       return emitMemMem(DAG, DL, SystemZISD::XC, SystemZISD::XC_LOOP,
                         Chain, Dst, Dst, Bytes);
 
     // Copy the byte to the first location and then use MVC to copy
     // it to the rest.
     Chain = DAG.getStore(Chain, DL, Byte, Dst, DstPtrInfo, Alignment);
     SDValue DstPlus1 = DAG.getNode(ISD::ADD, DL, PtrVT, Dst,
                                    DAG.getConstant(1, DL, PtrVT));
     return emitMemMem(DAG, DL, SystemZISD::MVC, SystemZISD::MVC_LOOP,
                       Chain, DstPlus1, Dst, Bytes - 1);
   }
+  else {  // Variable length
+    if (CByte && CByte->getZExtValue() == 0)
+      // Handle the special case of a variable length memset of 0 with XC.
+      return DAG.getNode(SystemZISD::XC_LOOP, DL, MVT::Other, Chain, Dst, Dst,
+                         Size, DAG.getConstant(0, DL, PtrVT));
+
+  }
   return SDValue();
 }
 
 // Use CLC to compare [Src1, Src1 + Size) with [Src2, Src2 + Size),
 // deciding whether to use a loop or straight-line code.
 static SDValue emitCLC(SelectionDAG &DAG, const SDLoc &DL, SDValue Chain,
                        SDValue Src1, SDValue Src2, uint64_t Size) {
   SDVTList VTs = DAG.getVTList(MVT::i32, MVT::Other);

llvm/test/CodeGen/SystemZ/memset-05.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; Test memset 0 with variable length
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
+
+define void @fun(i8* %Addr, i64 %Len) {
+; CHECK-LABEL: fun:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    cgibe %r3, 0, 0(%r14)
+; CHECK-NEXT:  .LBB0_1:
+; CHECK-NEXT:    aghi %r3, -1
+; CHECK-NEXT:    srlg %r0, %r3, 8
+; CHECK-NEXT:    cgije %r0, 0, .LBB0_3
+; CHECK-NEXT:  .LBB0_2: # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    xc 0(256,%r2), 0(%r2)
+; CHECK-NEXT:    la %r2, 256(%r2)
+; CHECK-NEXT:    brctg %r0, .LBB0_2
+; CHECK-NEXT:  .LBB0_3:
+; CHECK-NEXT:    exrl %r3, .Ltmp0
+; CHECK-NEXT:    br %r14
+; CHECK-NEXT:  .Ltmp0:
+; CHECK-NEXT:    xc 0(1,%r2), 0(%r2)
+  tail call void @llvm.memset.p0i8.i64(i8* %Addr, i8 0, i64 %Len, i1 false)
+  ret void
+}
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg)