[SystemZ] Implement shouldCoalesce() to help regalloc to avoid running out of registers with GR128 regs

[jonpa]

Summary

Fix for https://bugs.llvm.org/show_bug.cgi?id=34610

Regalloc runs out of registers in a test case with GR128 register. This can be remedied by not allowing coalescing copys of subregs from GR128 regs in shouldCoalesce().

[uweigand]

Thanks for trying this suggestion! See a couple of inline comments.

Also, I'm wondering doing less coalescing here may lead to reduced performance somewhere? It would be good to verify at least some benchmarks to make sure we aren't too restrictive now.

lib/Target/SystemZ/SystemZRegisterInfo.cpp

163

While turning subreg liveness on fixes the simpler test case you have here, the even more restricted test case I've added to the bugzilla actually still fails even then. So I'm not sure we should mention subreg liveness here as the reason why this is needed.

166

Should this be symmetrical? I.e. something like

if (NewRC->hasSubClassEq(&SystemZ::GR128BitRegClass) &&
    (getRegSizeInBits(*SrcRC) <= 64 || getRegSizeInBits(*DstRC) <= 64))
  return false;

test/CodeGen/SystemZ/regalloc-GR128.ll

7

Wouldn't just running llc and checking its exit code already test that there is no LLVM ERROR?

10

Maybe better to use the more restricted test from the bugzilla.

[jonpa]

lib/Target/SystemZ/SystemZRegisterInfo.cpp

163

I see - removed.

166

Do we need a test case for this also?

test/CodeGen/SystemZ/regalloc-GR128.ll

7

OK - I thought it was a nice "comment", but I guess we already have a comment at the top of the file.

10

yes.

[jonpa]

Updated per review.

[jonpa]

Test cases updated. As expected, there are now more register moves whenever the coalescer is disabled by this patch.

[uweigand]

In D37899#875075, @jonpa wrote:

Test cases updated. As expected, there are now more register moves whenever the coalescer is disabled by this patch.

Huh. This seems more extreme than I expected, it looks like just about *every* load of a 128-bit value from memory now reloads into another set of registers, even in those quite simple test cases ... Is this just the coalescing or is there something else going on?

[jonpa]

In D37899#875173, @uweigand wrote:

In D37899#875075, @jonpa wrote:

Test cases updated. As expected, there are now more register moves whenever the coalescer is disabled by this patch.

Huh. This seems more extreme than I expected, it looks like just about *every* load of a 128-bit value from memory now reloads into another set of registers, even in those quite simple test cases ... Is this just the coalescing or is there something else going on?

Yes, this is indeed more than needed.

For atomic-load-05.ll, I see during isel:

Optimized lowered selection DAG: BB#0 'f1:'
SelectionDAG has 12 nodes:
  t0: ch = EntryToken
      t2: i64,ch = CopyFromReg t0, Register:i64 %vreg0
    t6: ch = CopyToReg t0, Register:i64 %vreg2, t2
    t4: i64,ch = CopyFromReg t0, Register:i64 %vreg1
  t7: i128,ch = AtomicLoad<Volatile LD16[%src]> t6, t4
      t8: i64,ch = CopyFromReg t0, Register:i64 %vreg2
    t11: ch = store<ST16[<unknown>](align=8)> t7:1, t7, t8, undef:i64
  t12: ch = SystemZISD::RET_FLAG t11


Type-legalized selection DAG: BB#0 'f1:'
SelectionDAG has 20 nodes:
  t0: ch = EntryToken
  t8: i64,ch = CopyFromReg t0, Register:i64 %vreg2
      t2: i64,ch = CopyFromReg t0, Register:i64 %vreg0
    t6: ch = CopyToReg t0, Register:i64 %vreg2, t2
    t4: i64,ch = CopyFromReg t0, Register:i64 %vreg1
  t13: Untyped,ch = SystemZISD::ATOMIC_LOAD_128<Volatile LD16[%src]> t6, t4
        t15: i64 = EXTRACT_SUBREG t13, TargetConstant:i32<2>
      t19: ch = store<ST8[<unknown>]> t13:1, t15, t8, undef:i64
        t17: i64 = EXTRACT_SUBREG t13, TargetConstant:i32<7>
        t21: i64 = add t8, Constant:i64<8>
      t22: ch = store<ST8[<unknown>]> t13:1, t17, t21, undef:i64
    t23: ch = TokenFactor t19, t22
  t12: ch = SystemZISD::RET_FLAG t23

It looks to me that SystemZ does not have i128 as a legal type, and therefore the 128 bit store gets expanded. If the coalescing is then turned off, one of the subregs will need an extra COPY as the GR128 reg is live past the first store.

Maybe this would be a good time to try and make i128 legal as we have discussed earlier?

1. untyped -> i128
2. Add pseudos that expands after register allocation, so that the coalescing problems we see here go away.
3. Make sure that all the i128 operations that actually does not have support still gets expanded per current behavior.

anything else? Any hacks relating to untyped that we know should go away then?

[uweigand]

I looked into making i128 legal when I recently added the 128-bit atomics, but in the end decided against it. Not only would it be a lot of work (since you basically would have to reimplement all the operations on i128 that are currently handled by common code), but it is not clear that we actually always want it.

i128 lives in a even/odd register pair, but this is really only needed for the few special instructions that operate on those pairs. If you had just a normal load followed by a normal store of a i128 value, you *want* to break it into two independent i64 copies -- there is no reason at all to constrain the register allocator to use a even/odd pair here.

I've been thinking about alternative solutions to this problem. One way might be to prevent coalescing (as this patch does) and then try to *encourage* (but not enforce) use of the register pair earlier, e.g. by providing register allocator hints for those registers that would have been coalesced with the pair if we had not prevented this - sort of like the ARM register pair hints. But it turns out this is no so easy, because the allocator will assume those hints conflict with the original register pair (unless we switch on enableSubRegLiveness, but this seems to have other problems, it looks like it causes a number of internal compiler errors in the test suite).

Another option might be to not *completely* prevent coalescing a 128-bit pair with its component registers, but allow it in cases where there is no excessive register pressure. But I'm not sure exactly how to get at this information at the time we make the coalescing decision ...

[jonpa]

Patch updated.

I looked into making i128 legal when I recently added the 128-bit atomics, but in the end decided against it. Not only would it be a lot of work (since you basically would have to reimplement all the operations on i128 that are currently handled by common code), but it is not clear that we actually always want it.

I thought one could get away with adding calling setOperationAction(..., i128, Expand) on all operations that we do not support...

i128 lives in a even/odd register pair, but this is really only needed for the few special instructions that operate on those pairs. If you had just a normal load followed by a normal store of a i128 value, you *want* to break it into two independent i64 copies -- there is no reason at all to constrain the register allocator to use a even/odd pair here.

I would have expected that for a tiny live range, it is always better to avoid extra COPYs.

Another option might be to not *completely* prevent coalescing a 128-bit pair with its component registers, but allow it in cases where there is no excessive register pressure. But I'm not sure exactly how to get at this information at the time we make the coalescing decision ...

There is a current discussion relating to the mischeduler register pressure tracking about the fact that there is no real support to query global liveness yet. It may be that this will get added soon, and perhaps then some cheap register pressure estimate might be available.

Until then, I thought it might serve well to at least allow coalescing in cases of tiny, local live-ranges, e.g. loading and then storing 128 bits.

The updated patch uses LiveIntervals to quickly check if the two connected vregs are local to MBB and what the first and last instructions are (it may be possible to do this without LIS if we are willing to scan the whole MBB). I merely tried some values of 10 instructions and 8 phys-reg operands in order to detect a non-safe live-range. It may be that we want to check for vreg operands as well.

I don't have adequate tests to tune this further, but this might hopefully be a good starting point. At least all the regressions we saw before are now gone, while still handling your test case.

On SPEC, I see

Current patch:

               master                 patched
lg             : 320549               318515                -2034
stg            : 125311               124230                -1081
lr             :  25927                26632                 +705
lgr            : 343901               343602                 -299
llill          :    982                  869                 -113
c              :  10989                11061                  +72

"Spill|Reload"   168062               165145                -2917

[uweigand]

In D37899#879892, @jonpa wrote:

Patch updated.

I looked into making i128 legal when I recently added the 128-bit atomics, but in the end decided against it. Not only would it be a lot of work (since you basically would have to reimplement all the operations on i128 that are currently handled by common code), but it is not clear that we actually always want it.

I thought one could get away with adding calling setOperationAction(..., i128, Expand) on all operations that we do not support...

It turns out this doesn't work. E.g. in the case of a 128-bit add, this is split into a sequence of 64-bit add-with-carry operations by the *type legalization* code, not the normal expander. If you declare i128 a legal type, the type legalizer doesn't do this any more. But then if you mark the ADD as Expand, the normal expander doesn't know what to do with it either, and -I think- just falls back to a libcall.

i128 lives in a even/odd register pair, but this is really only needed for the few special instructions that operate on those pairs. If you had just a normal load followed by a normal store of a i128 value, you *want* to break it into two independent i64 copies -- there is no reason at all to constrain the register allocator to use a even/odd pair here.

I would have expected that for a tiny live range, it is always better to avoid extra COPYs.

My point was for a 128-bit load feeding into a 128-bit store, both the load and the store gets expanded to two 64-bit loads / stores anyway. So we end up with two completely independent 64-bit data flow paths, which require 64-bit registers --- but there is absolutely no benefit to require that these registers form an even/odd pair, they might as well be arbitrary registers (it might even be the same one if the sequence ends up scheduled as LG / STG / LG / STG). Actually the same applies to most of the other cases where the 128-bit operation ends up split into 64-bit operations by the type legalizer.

The updated patch uses LiveIntervals to quickly check if the two connected vregs are local to MBB and what the first and last instructions are (it may be possible to do this without LIS if we are willing to scan the whole MBB). I merely tried some values of 10 instructions and 8 phys-reg operands in order to detect a non-safe live-range. It may be that we want to check for vreg operands as well.

I don't have adequate tests to tune this further, but this might hopefully be a good starting point. At least all the regressions we saw before are now gone, while still handling your test case.

This looks promising. Not sure if anybody would object to changing the interface to pass in the LiveIntervals, but it seems reasonable to me.

Not sure about the magic numbers -- does it matter whether we stop after 10 MIs? What kind of compile-time performance impact would it have to scan the whole range?

It looks like your check for > 8 occurrences of physical registers would trigger even if it is always the same register. Should we instead keep a bitmap of the used registers, so we know how many *different* ones are used? Also, it should probably only check for GPRs, not any random physical register.

[jonpa]

I thought one could get away with adding calling setOperationAction(..., i128, Expand) on all operations that we do not support...

It turns out this doesn't work. E.g. in the case of a 128-bit add, this is split into a sequence of 64-bit add-with-carry operations by the *type legalization* code, not the normal expander. If you declare i128 a legal type, the type legalizer doesn't do this any more. But then if you mark the ADD as Expand, the normal expander doesn't know what to do with it either, and -I think- just falls back to a libcall.

i128 lives in a even/odd register pair, but this is really only needed for the few special instructions that operate on those pairs. If you had just a normal load followed by a normal store of a i128 value, you *want* to break it into two independent i64 copies -- there is no reason at all to constrain the register allocator to use a even/odd pair here.

I would have expected that for a tiny live range, it is always better to avoid extra COPYs.

My point was for a 128-bit load feeding into a 128-bit store, both the load and the store gets expanded to two 64-bit loads / stores anyway. So we end up with two completely independent 64-bit data flow paths, which require 64-bit registers --- but there is absolutely no benefit to require that these registers form an even/odd pair, they might as well be arbitrary registers (it might even be the same one if the sequence ends up scheduled as LG / STG / LG / STG). Actually the same applies to most of the other cases where the 128-bit operation ends up split into 64-bit operations by the type legalizer.

I see.

The updated patch uses LiveIntervals to quickly check if the two connected vregs are local to MBB and what the first and last instructions are (it may be possible to do this without LIS if we are willing to scan the whole MBB). I merely tried some values of 10 instructions and 8 phys-reg operands in order to detect a non-safe live-range. It may be that we want to check for vreg operands as well.

I don't have adequate tests to tune this further, but this might hopefully be a good starting point. At least all the regressions we saw before are now gone, while still handling your test case.

This looks promising. Not sure if anybody would object to changing the interface to pass in the LiveIntervals, but it seems reasonable to me.

Not sure about the magic numbers -- does it matter whether we stop after 10 MIs? What kind of compile-time performance impact would it have to scan the whole range?

I don't know, I just thought that we basically wanted to at least handle the type of case where there is a GR128 op and then some subreg COPY(s) very close. My reason for having the instruction limit is that we can't know the register pressure, but it seems safe to think that tiny live ranges checked in this way could be coalesced.

It looks like your check for > 8 occurrences of physical registers would trigger even if it is always the same register. Should we instead keep a bitmap of the used registers, so we know how many *different* ones are used? Also, it should probably only check for GPRs, not any random physical register.

Checking for GPRs is something I seem to have missed, but however the different reg check is something I actually took away after first using a SmallSet. I thought that if we would to this properly, we should in addition also check aliases after finding the set of different regs. I then changed my mind since we are merely trying to do an estimate, and we don't really know for sure what the right limit is, etc and may rather just keep it simple (BTW, I also see that it should do +=2 for a GR128 reg).

I guess it depends on the code and if we are trying to handle "any" type of mixed code with an as-good-as-possible estimate, or are we merely trying to keep coalescing the simple and small sequences?

Taking this a step further, if regalloc can only run out of registers when enough physreg defs overlap the GR128 live range so that there is no single 128-bit register pair to be allocated, then we should basically allow coalescing if there is at least one non-clobbered i128 register pair in the region, right?

Even if there were many such GR128 virtregs overlapping, one would hope that regalloc would be able to split them so that one such free pair would be enough. Would we want to stop coalescing at that some point in order to reduce spilling? We could perhaps try to by looking for GR128 virtual register operands in the region.

[uweigand]

In D37899#880029, @jonpa wrote:

Not sure about the magic numbers -- does it matter whether we stop after 10 MIs? What kind of compile-time performance impact would it have to scan the whole range?

I don't know, I just thought that we basically wanted to at least handle the type of case where there is a GR128 op and then some subreg COPY(s) very close. My reason for having the instruction limit is that we can't know the register pressure, but it seems safe to think that tiny live ranges checked in this way could be coalesced.

Ah, I misread the code -- if we have more than 10 instructions, you default to *not* coalese (I somehow thought it was the opposite). That seems more reasonable. The interesting question will then be how this interacts with the scheduler, i.e. how frequently it happens that unrelated code is scheduled in between, and thus breaking the coalescing. Not sure if that is a real concern ...

It looks like your check for > 8 occurrences of physical registers would trigger even if it is always the same register. Should we instead keep a bitmap of the used registers, so we know how many *different* ones are used? Also, it should probably only check for GPRs, not any random physical register.

Checking for GPRs is something I seem to have missed, but however the different reg check is something I actually took away after first using a SmallSet. I thought that if we would to this properly, we should in addition also check aliases after finding the set of different regs. I then changed my mind since we are merely trying to do an estimate, and we don't really know for sure what the right limit is, etc and may rather just keep it simple (BTW, I also see that it should do +=2 for a GR128 reg).

Yes, that's why I was thinking of a simple bitmap of the 16 actual GPRs, and just set a bit if this GPR is touched (no matter via which super/subreg or alias).

Taking this a step further, if regalloc can only run out of registers when enough physreg defs overlap the GR128 live range so that there is no single 128-bit register pair to be allocated, then we should basically allow coalescing if there is at least one non-clobbered i128 register pair in the region, right?

Even if there were many such GR128 virtregs overlapping, one would hope that regalloc would be able to split them so that one such free pair would be enough. Would we want to stop coalescing at that some point in order to reduce spilling? We could perhaps try to by looking for GR128 virtual register operands in the region.

Strictly speaking, yes. But I think it also doesn't matter to avoid coalescing even if the register pressure isn't *that* extreme -- it is true that as long as at least one GPR pair is available, the allocator should be able to fix up anything via spilling, but in general spilling is more expensive than extra copying due to non-coalescing would have been ...

(B.t.w. if you do go for the one-pair check, it needs to check for one pair except 0/1, because 0/1 isn't usable for ADDR128.)

[jonpa]

Improved to accurately keep track of which registers are actually clobbered in the region by means of a BitVector.

Ah, I misread the code -- if we have more than 10 instructions, you default to *not* coalese (I somehow thought it was the opposite). That seems more reasonable. The interesting question will then be how this interacts with the scheduler, i.e. how frequently it happens that unrelated code is scheduled in between, and thus breaking the coalescing. Not sure if that is a real concern ...

See table below.

Yes, that's why I was thinking of a simple bitmap of the 16 actual GPRs, and just set a bit if this GPR is touched (no matter via which super/subreg or alias).

I did this for the 128 bit regs instead since that is what seems mostly relevant in the end.

(B.t.w. if you do go for the one-pair check, it needs to check for one pair except 0/1, because 0/1 isn't usable for ADDR128.)

I am hoping that by using NewRC (which should be either GR128 or ADDR128), this should work as expected.

Considering other types of code with more heavy use of GR128, would we perhaps want to also give a count for a virtual register definition of a GR128? That should avoid being too optimistic when several of these overlap.

SPEC

                 master               patched
lg             : 320549               318463                -2086
stg            : 125311               124234                -1077
lr             :  25927                26482                 +555
lgr            : 343901               343499                 -402
llill          :    982                  870                 -112
st             :  51272                51343                  +71
stfh           :   2037                 2095                  +58
j              : 100186               100129                  -57
c              :  10989                11045                  +56
"Spill|Reload"   168062               165106                -2956

Number of regions / GR128 pairs clobbered

6070 PhysClob: 0
 206 PhysClob: 1
 187 PhysClob: 2
  17 PhysClob: 3
   8 PhysClob: 4

(Never more than 4 clobbered GR128 phys reg pairs. I therefore took "3" out of the blue as a margin.)

Number of regions / MI count

1305 MIs: 3
2489 MIs: 4
 628 MIs: 5
 581 MIs: 6
 258 MIs: 7
 119 MIs: 8
 103 MIs: 9
  94 MIs: 10
  61 MIs: 11
 129 MIs: 12
 220 MIs: 13
  47 MIs: 14
  49 MIs: 15
  31 MIs: 16
  39 MIs: 17
  23 MIs: 18
  42 MIs: 19
  12 MIs: 20
  46 MIs: 21
  55 MIs: 22
   6 MIs: 23
  13 MIs: 24
  14 MIs: 25
  14 MIs: 26
   3 MIs: 27
   6 MIs: 28
   7 MIs: 29
   7 MIs: 30
   1 MIs: 31
  41 MIs: 32
   3 MIs: 33
   3 MIs: 34
   2 MIs: 36
   1 MIs: 37
   1 MIs: 38
   1 MIs: 40
   1 MIs: 41
   1 MIs: 42
   3 MIs: 44
   4 MIs: 45
   2 MIs: 46
   2 MIs: 47
   2 MIs: 48
   2 MIs: 49
   2 MIs: 50
   1 MIs: 51
   1 MIs: 52
   1 MIs: 55
   2 MIs: 56
   1 MIs: 63
   1 MIs: 65
   2 MIs: 67
   1 MIs: 69
   1 MIs: 70
   1 MIs: 71
   1 MIs: 73
   1 MIs: 78
   1 MIs: 103

Great majority of these cases are then as expected small regions, so I am thinking there's no real need for an instruction limit - at least not on SPEC

[jonpa]

@Quentin, Matthias: This patch introduces an extra parameter to shouldCoalesce() (LiveIntervals*). Does this look acceptable?

[uweigand]

In D37899#881218, @jonpa wrote:

Considering other types of code with more heavy use of GR128, would we perhaps want to also give a count for a virtual register definition of a GR128? That should avoid being too optimistic when several of these overlap.

Not sure, hard to say if this is going to be overall beneficial or not. I'd say to just do the physreg check for now.

Great majority of these cases are then as expected small regions, so I am thinking there's no real need for an instruction limit - at least not on SPEC

OK, makes sense to me.

The code now looks reasonable to me, just a few inline comments.

lib/Target/SystemZ/SystemZRegisterInfo.cpp

165

Hmm, does this still hit if we get ADDR128BitRegClass? Probably needs to check for that instead of GR128BitRegClass here ...

We then ideally should also have a test case that uses ADDR128BitRegClass (one way to achieve this is to add a use of "remainder + 1", which gets selected to an LA instruction).

198

Unused now?

208

Is this really necessary / useful? Why not do the whole test just via the SuperRegIterator loop (using IncludeSelf = true)?

[jonpa]

Updated per review. Not much change on SPEC compared to last version of the patch:

lg             :               318552               318542      -10
st             :                51343                51348       +5
cgijlh         :                17962                17958       -4
...
Spill|Reload   :               165200               165200       +0

Test case for ADDR128 still missing.

[qcolombet]

This patch introduces an extra parameter to shouldCoalesce() (LiveIntervals*). Does this look acceptable?

Looks fine to me.

I would use a reference if we don't allow it to be null or update the comment to say that this may be null.

[jonpa]

I would use a reference if we don't allow it to be null or update the comment to say that this may be null.

Patch updated to use reference instead of pointer for LiveIntervals argument.

[jonpa]

lib/Target/SystemZ/SystemZRegisterInfo.cpp

165

It should be NewRC->hasSuperClassEq(), which I believe returns true only for ADDR128 and GR128. I verified that this works as expected in gdb.

198

removed

208

removed

[jonpa]

Add test that exposes an ADDR128 virtual register.

[uweigand]

Given that:

• Quentin approved the interface change
• The SystemZ parts now look good to me
• We have performance runs confirming this does not introduce regressions

this patch now LGTM.

[jonpa]

Thanks for review.
r314516

lib/Target/SystemZ/SystemZRegisterInfo.h

public:
   }
   const MCPhysReg *getCalleeSavedRegs(const MachineFunction *MF) const override;
   const uint32_t *getCallPreservedMask(const MachineFunction &MF,
                                        CallingConv::ID CC) const override;
   BitVector getReservedRegs(const MachineFunction &MF) const override;
   void eliminateFrameIndex(MachineBasicBlock::iterator MI,
                            int SPAdj, unsigned FIOperandNum,
                            RegScavenger *RS) const override;
+ bool shouldCoalesce(MachineInstr *MI,
+                      const TargetRegisterClass *SrcRC,
+                      unsigned SubReg,
+                      const TargetRegisterClass *DstRC,
+                      unsigned DstSubReg,
+                      const TargetRegisterClass *NewRC) const override;
+
+  /// \brief SrcRC and DstRC will be morphed into NewRC if this returns true.
   unsigned getFrameRegister(const MachineFunction &MF) const override;
 };
 
 } // end namespace llvm
 
 #endif

lib/Target/SystemZ/SystemZRegisterInfo.cpp

if (MI->getDesc().TSFlags & SystemZII::HasIndex
       MI->getOperand(FIOperandNum).ChangeToRegister(ScratchReg,
                                                     false, false, true);
     }
   }
   MI->setDesc(TII->get(OpcodeForOffset));
   MI->getOperand(FIOperandNum + 1).ChangeToImmediate(Offset);
 }
 
+bool SystemZRegisterInfo::shouldCoalesce(MachineInstr *MI,
+                                  const TargetRegisterClass *SrcRC,
+                                  unsigned SubReg,
+                                  const TargetRegisterClass *DstRC,
+                                  unsigned DstSubReg,
+                                  const TargetRegisterClass *NewRC) const {
+  // Avoid coalescing COPYs of subregs of a GR128 register to prevent
+  // regalloc running out of registers.
+  if (NewRC->hasSubClassEq(&SystemZ::GR128BitRegClass) &&
+      (getRegSizeInBits(*SrcRC) <= 64 || getRegSizeInBits(*DstRC) <= 64))
+    return false;
+
+  return true;
+}
+
 unsigned
 SystemZRegisterInfo::getFrameRegister(const MachineFunction &MF) const {
   const SystemZFrameLowering *TFI = getFrameLowering(MF);
   return TFI->hasFP(MF) ? SystemZ::R11D : SystemZ::R15D;
 }

test/CodeGen/SystemZ/atomic-load-05.ll

 ; Test 128-bit atomic loads.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 define i128 @f1(i128 *%src) {
 ; CHECK-LABEL: f1:
 ; CHECK: lpq %r0, 0(%r3)
-; CHECK-DAG: stg %r1, 8(%r2)
+; CHECK-DAG: stg %r3, 8(%r2)
 ; CHECK-DAG: stg %r0, 0(%r2)
 ; CHECK: br %r14
   %val = load atomic i128, i128 *%src seq_cst, align 16
   ret i128 %val
 }

test/CodeGen/SystemZ/atomic-store-05.ll

 ; Test 128-bit atomic stores.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 define void @f1(i128 %val, i128 *%src) {
 ; CHECK-LABEL: f1:
-; CHECK-DAG: lg %r1, 8(%r2)
-; CHECK-DAG: lg %r0, 0(%r2)
-; CHECK: stpq %r0, 0(%r3)
+; CHECK-DAG: lg %r0, 8(%r2)
+; CHECK-DAG: lg %r1, 0(%r2)
+; CHECK: stpq %r4, 0(%r3)
 ; CHECK: bcr 1{{[45]}}, %r0
 ; CHECK: br %r14
   store atomic i128 %val, i128 *%src seq_cst, align 16
   ret void
 }
 
 define void @f2(i128 %val, i128 *%src) {
 ; CHECK-LABEL: f2:
-; CHECK-DAG: lg %r1, 8(%r2)
-; CHECK-DAG: lg %r0, 0(%r2)
-; CHECK: stpq %r0, 0(%r3)
+; CHECK-DAG: lg %r0, 8(%r2)
+; CHECK-DAG: lg %r1, 0(%r2)
+; CHECK: stpq %r4, 0(%r3)
 ; CHECK-NOT: bcr 1{{[45]}}, %r0
 ; CHECK: br %r14
   store atomic i128 %val, i128 *%src monotonic, align 16
   ret void
 }

test/CodeGen/SystemZ/cmpxchg-06.ll

 ; Test 64-bit compare and swap.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 ; Check CDSG without a displacement.
 define i128 @f1(i128 %cmp, i128 %swap, i128 *%src) {
 ; CHECK-LABEL: f1:
-; CHECK-DAG: lg %r1, 8(%r4)
-; CHECK-DAG: lg %r0, 0(%r4)
-; CHECK-DAG: lg %r13, 8(%r3)
-; CHECK-DAG: lg %r12, 0(%r3)
-; CHECK:     cdsg %r12, %r0, 0(%r5)
-; CHECK-DAG: stg %r13, 8(%r2)
-; CHECK-DAG: stg %r12, 0(%r2)
+; CHECK-DAG: lg %r0, 8(%r4)
+; CHECK-DAG: lg %r1, 0(%r4)
+; CHECK-DAG: lg %r0, 8(%r3)
+; CHECK-DAG: lg %r1, 0(%r3)
+; CHECK:     cdsg %r10, %r12, 0(%r5)
+; CHECK-DAG: stg %r0, 8(%r2)
+; CHECK-DAG: stg %r0, 0(%r2)
 ; CHECK:     br %r14
   %pairval = cmpxchg i128 *%src, i128 %cmp, i128 %swap seq_cst seq_cst
   %val = extractvalue { i128, i1 } %pairval, 0
   ret i128 %val
 }
 
 ; Check the high end of the aligned CDSG range.
 define i128 @f2(i128 %cmp, i128 %swap, i128 *%src) {

test/CodeGen/SystemZ/int-div-01.ll

 ; Test 32-bit signed division and remainder.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 declare i32 @foo()
 
 ; Test register division.  The result is in the second of the two registers.
 define void @f1(i32 *%dest, i32 %a, i32 %b) {
 ; CHECK-LABEL: f1:
-; CHECK: lgfr %r1, %r3
+; CHECK: lgfr %r0, %r3
 ; CHECK: dsgfr %r0, %r4
-; CHECK: st %r1, 0(%r2)
+; CHECK: st %r0, 0(%r2)
 ; CHECK: br %r14
   %div = sdiv i32 %a, %b
   store i32 %div, i32 *%dest
   ret void
 }
 
 ; Test register remainder.  The result is in the first of the two registers.
 define void @f2(i32 *%dest, i32 %a, i32 %b) {
 ; CHECK-LABEL: f2:
-; CHECK: lgfr %r1, %r3
+; CHECK: lgfr %r0, %r3
 ; CHECK: dsgfr %r0, %r4
 ; CHECK: st %r0, 0(%r2)
 ; CHECK: br %r14
   %rem = srem i32 %a, %b
   store i32 %rem, i32 *%dest
   ret void
 }
 
 ; Test that division and remainder use a single instruction.
 define i32 @f3(i32 %dummy, i32 %a, i32 %b) {
 ; CHECK-LABEL: f3:
 ; CHECK-NOT: %r2
-; CHECK: lgfr %r3, %r3
+; CHECK: lgfr %r0, %r3
 ; CHECK-NOT: %r2
-; CHECK: dsgfr %r2, %r4
+; CHECK: dsgfr %r0, %r4
 ; CHECK-NOT: dsgfr
 ; CHECK: or %r2, %r3
 ; CHECK: br %r14
   %div = sdiv i32 %a, %b
   %rem = srem i32 %a, %b
   %or = or i32 %rem, %div
   ret i32 %or
 }
 
 ; Check that the sign extension of the dividend is elided when the argument
 ; is already sign-extended.
 define i32 @f4(i32 %dummy, i32 signext %a, i32 %b) {
 ; CHECK-LABEL: f4:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgfr %r2, %r4
+; CHECK: dsgfr %r0, %r4
 ; CHECK-NOT: dsgfr
 ; CHECK: or %r2, %r3
 ; CHECK: br %r14
   %div = sdiv i32 %a, %b
   %rem = srem i32 %a, %b
   %or = or i32 %rem, %div
   ret i32 %or
 }
 
 ; Test that memory dividends are loaded using sign extension (LGF).
 define i32 @f5(i32 %dummy, i32 *%src, i32 %b) {
 ; CHECK-LABEL: f5:
 ; CHECK-NOT: %r2
-; CHECK: lgf %r3, 0(%r3)
+; CHECK: lgf %r0, 0(%r3)
 ; CHECK-NOT: %r2
-; CHECK: dsgfr %r2, %r4
+; CHECK: dsgfr %r0, %r4
 ; CHECK-NOT: dsgfr
 ; CHECK: or %r2, %r3
 ; CHECK: br %r14
   %a = load i32 , i32 *%src
   %div = sdiv i32 %a, %b
   %rem = srem i32 %a, %b
   %or = or i32 %rem, %div
   ret i32 %or
 }
 
 ; Test memory division with no displacement.
 define void @f6(i32 *%dest, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f6:
-; CHECK: lgfr %r1, %r3
+; CHECK: lgfr %r0, %r3
 ; CHECK: dsgf %r0, 0(%r4)
-; CHECK: st %r1, 0(%r2)
+; CHECK: st %r0, 0(%r2)
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %div = sdiv i32 %a, %b
   store i32 %div, i32 *%dest
   ret void
 }
 
 ; Test memory remainder with no displacement.
 define void @f7(i32 *%dest, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f7:
-; CHECK: lgfr %r1, %r3
+; CHECK: lgfr %r0, %r3
 ; CHECK: dsgf %r0, 0(%r4)
 ; CHECK: st %r0, 0(%r2)
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %rem = srem i32 %a, %b
   store i32 %rem, i32 *%dest
   ret void
 }
 
 ; Test both memory division and memory remainder.
 define i32 @f8(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f8:
 ; CHECK-NOT: %r2
-; CHECK: lgfr %r3, %r3
+; CHECK: lgfr %r0, %r3
 ; CHECK-NOT: %r2
-; CHECK: dsgf %r2, 0(%r4)
+; CHECK: dsgf %r0, 0(%r4)
 ; CHECK-NOT: {{dsgf|dsgfr}}
 ; CHECK: or %r2, %r3
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %div = sdiv i32 %a, %b
   %rem = srem i32 %a, %b
   %or = or i32 %rem, %div
   ret i32 %or
 }
 
 ; Check the high end of the DSGF range.
 define i32 @f9(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f9:
-; CHECK: dsgf %r2, 524284(%r4)
+; CHECK: dsgf %r0, 524284(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 131071
   %b = load i32 , i32 *%ptr
   %rem = srem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check the next word up, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i32 @f10(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f10:
 ; CHECK: agfi %r4, 524288
-; CHECK: dsgf %r2, 0(%r4)
+; CHECK: dsgf %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 131072
   %b = load i32 , i32 *%ptr
   %rem = srem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check the high end of the negative aligned DSGF range.
 define i32 @f11(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f11:
-; CHECK: dsgf %r2, -4(%r4)
+; CHECK: dsgf %r0, -4(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -1
   %b = load i32 , i32 *%ptr
   %rem = srem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check the low end of the DSGF range.
 define i32 @f12(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f12:
-; CHECK: dsgf %r2, -524288(%r4)
+; CHECK: dsgf %r0, -524288(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -131072
   %b = load i32 , i32 *%ptr
   %rem = srem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check the next word down, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i32 @f13(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f13:
 ; CHECK: agfi %r4, -524292
-; CHECK: dsgf %r2, 0(%r4)
+; CHECK: dsgf %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -131073
   %b = load i32 , i32 *%ptr
   %rem = srem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check that DSGF allows an index.
 define i32 @f14(i32 %dummy, i32 %a, i64 %src, i64 %index) {
 ; CHECK-LABEL: f14:
-; CHECK: dsgf %r2, 524287(%r5,%r4)
+; CHECK: dsgf %r0, 524287(%r5,%r4)
 ; CHECK: br %r14
   %add1 = add i64 %src, %index
   %add2 = add i64 %add1, 524287
   %ptr = inttoptr i64 %add2 to i32 *
   %b = load i32 , i32 *%ptr
   %rem = srem i32 %a, %b
   ret i32 %rem
 }
 
 ; Make sure that we still use DSGFR rather than DSGR in cases where
 ; a load and division cannot be combined.
 define void @f15(i32 *%dest, i32 *%src) {
 ; CHECK-LABEL: f15:
 ; CHECK: l [[B:%r[0-9]+]], 0(%r3)
 ; CHECK: brasl %r14, foo@PLT
-; CHECK: lgfr %r1, %r2
+; CHECK: lgfr %r0, %r2
 ; CHECK: dsgfr %r0, [[B]]
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %a = call i32 @foo()
   %div = sdiv i32 %a, %b
   store i32 %div, i32 *%dest
   ret void
 }

test/CodeGen/SystemZ/int-div-02.ll

 ; Test 32-bit unsigned division and remainder.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 declare i32 @foo()
 
 ; Test register division.  The result is in the second of the two registers.
 define void @f1(i32 %dummy, i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f1:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lhi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlr %r2, %r4
-; CHECK: st %r3, 0(%r5)
+; CHECK: {{llill|lhi}} %r0, 0
+; CHECK: dlr %r0, %r4
+; CHECK: st %r0, 0(%r5)
 ; CHECK: br %r14
   %div = udiv i32 %a, %b
   store i32 %div, i32 *%dest
   ret void
 }
 
 ; Test register remainder.  The result is in the first of the two registers.
 define void @f2(i32 %dummy, i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f2:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lhi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlr %r2, %r4
-; CHECK: st %r2, 0(%r5)
+; CHECK: {{llill|lhi}} %r0, 0
+; CHECK: dlr %r0, %r4
+; CHECK: st %r0, 0(%r5)
 ; CHECK: br %r14
   %rem = urem i32 %a, %b
   store i32 %rem, i32 *%dest
   ret void
 }
 
 ; Test that division and remainder use a single instruction.
 define i32 @f3(i32 %dummy1, i32 %a, i32 %b) {
 ; CHECK-LABEL: f3:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lhi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlr %r2, %r4
+; CHECK: {{llill|lhi}} %r0, 0
+; CHECK: dlr %r0, %r4
 ; CHECK-NOT: dlr
 ; CHECK: or %r2, %r3
 ; CHECK: br %r14
   %div = udiv i32 %a, %b
   %rem = urem i32 %a, %b
   %or = or i32 %rem, %div
   ret i32 %or
 }
 
 ; Test memory division with no displacement.
 define void @f4(i32 %dummy, i32 %a, i32 *%src, i32 *%dest) {
 ; CHECK-LABEL: f4:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lhi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dl %r2, 0(%r4)
-; CHECK: st %r3, 0(%r5)
+; CHECK: {{llill|lhi}} %r0, 0
+; CHECK: dl %r0, 0(%r4)
+; CHECK: st %r0, 0(%r5)
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %div = udiv i32 %a, %b
   store i32 %div, i32 *%dest
   ret void
 }
 
 ; Test memory remainder with no displacement.
 define void @f5(i32 %dummy, i32 %a, i32 *%src, i32 *%dest) {
 ; CHECK-LABEL: f5:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lhi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dl %r2, 0(%r4)
-; CHECK: st %r2, 0(%r5)
+; CHECK: {{llill|lhi}} %r0, 0
+; CHECK: dl %r0, 0(%r4)
+; CHECK: st %r0, 0(%r5)
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %rem = urem i32 %a, %b
   store i32 %rem, i32 *%dest
   ret void
 }
 
 ; Test both memory division and memory remainder.
 define i32 @f6(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f6:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lhi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dl %r2, 0(%r4)
+; CHECK: {{llill|lhi}} %r0, 0
+; CHECK: dl %r0, 0(%r4)
 ; CHECK-NOT: {{dl|dlr}}
 ; CHECK: or %r2, %r3
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %div = udiv i32 %a, %b
   %rem = urem i32 %a, %b
   %or = or i32 %rem, %div
   ret i32 %or
 }
 
 ; Check the high end of the DL range.
 define i32 @f7(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f7:
-; CHECK: dl %r2, 524284(%r4)
+; CHECK: dl %r0, 524284(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 131071
   %b = load i32 , i32 *%ptr
   %rem = urem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check the next word up, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i32 @f8(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f8:
 ; CHECK: agfi %r4, 524288
-; CHECK: dl %r2, 0(%r4)
+; CHECK: dl %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 131072
   %b = load i32 , i32 *%ptr
   %rem = urem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check the high end of the negative aligned DL range.
 define i32 @f9(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f9:
-; CHECK: dl %r2, -4(%r4)
+; CHECK: dl %r0, -4(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -1
   %b = load i32 , i32 *%ptr
   %rem = urem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check the low end of the DL range.
 define i32 @f10(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f10:
-; CHECK: dl %r2, -524288(%r4)
+; CHECK: dl %r0, -524288(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -131072
   %b = load i32 , i32 *%ptr
   %rem = urem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check the next word down, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i32 @f11(i32 %dummy, i32 %a, i32 *%src) {
 ; CHECK-LABEL: f11:
 ; CHECK: agfi %r4, -524292
-; CHECK: dl %r2, 0(%r4)
+; CHECK: dl %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -131073
   %b = load i32 , i32 *%ptr
   %rem = urem i32 %a, %b
   ret i32 %rem
 }
 
 ; Check that DL allows an index.
 define i32 @f12(i32 %dummy, i32 %a, i64 %src, i64 %index) {
 ; CHECK-LABEL: f12:
-; CHECK: dl %r2, 524287(%r5,%r4)
+; CHECK: dl %r0, 524287(%r5,%r4)
 ; CHECK: br %r14
   %add1 = add i64 %src, %index
   %add2 = add i64 %add1, 524287
   %ptr = inttoptr i64 %add2 to i32 *
   %b = load i32 , i32 *%ptr
   %rem = urem i32 %a, %b
   ret i32 %rem
 }

test/CodeGen/SystemZ/int-div-03.ll

 ; Test 64-bit signed division and remainder when the divisor is
 ; a signed-extended i32.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 declare i64 @foo()
 
 ; Test register division.  The result is in the second of the two registers.
 define void @f1(i64 %dummy, i64 %a, i32 %b, i64 *%dest) {
 ; CHECK-LABEL: f1:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgfr %r2, %r4
-; CHECK: stg %r3, 0(%r5)
+; CHECK: dsgfr %r0, %r4
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %bext = sext i32 %b to i64
   %div = sdiv i64 %a, %bext
   store i64 %div, i64 *%dest
   ret void
 }
 
 ; Test register remainder.  The result is in the first of the two registers.
 define void @f2(i64 %dummy, i64 %a, i32 %b, i64 *%dest) {
 ; CHECK-LABEL: f2:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgfr %r2, %r4
-; CHECK: stg %r2, 0(%r5)
+; CHECK: dsgfr %r0, %r4
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %bext = sext i32 %b to i64
   %rem = srem i64 %a, %bext
   store i64 %rem, i64 *%dest
   ret void
 }
 
 ; Test that division and remainder use a single instruction.
 define i64 @f3(i64 %dummy, i64 %a, i32 %b) {
 ; CHECK-LABEL: f3:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgfr %r2, %r4
+; CHECK: dsgfr %r0, %r4
 ; CHECK: ogr %r2, %r3
 ; CHECK: br %r14
   %bext = sext i32 %b to i64
   %div = sdiv i64 %a, %bext
   %rem = srem i64 %a, %bext
   %or = or i64 %rem, %div
   ret i64 %or
 }
; CHECK: br %r14
   %rem = srem i64 %a, %bext
   store i64 %rem, i64 *%dest
   ret void
 }
 
 ; Test memory division with no displacement.
 define void @f6(i64 %dummy, i64 %a, i32 *%src, i64 *%dest) {
 ; CHECK-LABEL: f6:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgf %r2, 0(%r4)
-; CHECK: stg %r3, 0(%r5)
+; CHECK: dsgf %r0, 0(%r4)
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %bext = sext i32 %b to i64
   %div = sdiv i64 %a, %bext
   store i64 %div, i64 *%dest
   ret void
 }
 
 ; Test memory remainder with no displacement.
 define void @f7(i64 %dummy, i64 %a, i32 *%src, i64 *%dest) {
 ; CHECK-LABEL: f7:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgf %r2, 0(%r4)
-; CHECK: stg %r2, 0(%r5)
+; CHECK: dsgf %r0, 0(%r4)
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %bext = sext i32 %b to i64
   %rem = srem i64 %a, %bext
   store i64 %rem, i64 *%dest
   ret void
 }
 
 ; Test both memory division and memory remainder.
 define i64 @f8(i64 %dummy, i64 %a, i32 *%src) {
 ; CHECK-LABEL: f8:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgf %r2, 0(%r4)
+; CHECK: dsgf %r0, 0(%r4)
 ; CHECK-NOT: {{dsgf|dsgfr}}
 ; CHECK: ogr %r2, %r3
 ; CHECK: br %r14
   %b = load i32 , i32 *%src
   %bext = sext i32 %b to i64
   %div = sdiv i64 %a, %bext
   %rem = srem i64 %a, %bext
   %or = or i64 %rem, %div
   ret i64 %or
 }
 
 ; Check the high end of the DSGF range.
 define i64 @f9(i64 %dummy, i64 %a, i32 *%src) {
 ; CHECK-LABEL: f9:
-; CHECK: dsgf %r2, 524284(%r4)
+; CHECK: dsgf %r0, 524284(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 131071
   %b = load i32 , i32 *%ptr
   %bext = sext i32 %b to i64
   %rem = srem i64 %a, %bext
   ret i64 %rem
 }
 
 ; Check the next word up, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f10(i64 %dummy, i64 %a, i32 *%src) {
 ; CHECK-LABEL: f10:
 ; CHECK: agfi %r4, 524288
-; CHECK: dsgf %r2, 0(%r4)
+; CHECK: dsgf %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 131072
   %b = load i32 , i32 *%ptr
   %bext = sext i32 %b to i64
   %rem = srem i64 %a, %bext
   ret i64 %rem
 }
 
 ; Check the high end of the negative aligned DSGF range.
 define i64 @f11(i64 %dummy, i64 %a, i32 *%src) {
 ; CHECK-LABEL: f11:
-; CHECK: dsgf %r2, -4(%r4)
+; CHECK: dsgf %r0, -4(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -1
   %b = load i32 , i32 *%ptr
   %bext = sext i32 %b to i64
   %rem = srem i64 %a, %bext
   ret i64 %rem
 }
 
 ; Check the low end of the DSGF range.
 define i64 @f12(i64 %dummy, i64 %a, i32 *%src) {
 ; CHECK-LABEL: f12:
-; CHECK: dsgf %r2, -524288(%r4)
+; CHECK: dsgf %r0, -524288(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -131072
   %b = load i32 , i32 *%ptr
   %bext = sext i32 %b to i64
   %rem = srem i64 %a, %bext
   ret i64 %rem
 }
 
 ; Check the next word down, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f13(i64 %dummy, i64 %a, i32 *%src) {
 ; CHECK-LABEL: f13:
 ; CHECK: agfi %r4, -524292
-; CHECK: dsgf %r2, 0(%r4)
+; CHECK: dsgf %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i32, i32 *%src, i64 -131073
   %b = load i32 , i32 *%ptr
   %bext = sext i32 %b to i64
   %rem = srem i64 %a, %bext
   ret i64 %rem
 }
 
 ; Check that DSGF allows an index.
 define i64 @f14(i64 %dummy, i64 %a, i64 %src, i64 %index) {
 ; CHECK-LABEL: f14:
-; CHECK: dsgf %r2, 524287(%r5,%r4)
+; CHECK: dsgf %r0, 524287(%r5,%r4)
 ; CHECK: br %r14
   %add1 = add i64 %src, %index
   %add2 = add i64 %add1, 524287
   %ptr = inttoptr i64 %add2 to i32 *
   %b = load i32 , i32 *%ptr
   %bext = sext i32 %b to i64
   %rem = srem i64 %a, %bext
   ret i64 %rem

test/CodeGen/SystemZ/int-div-04.ll

 ; Testg 64-bit signed division and remainder.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 declare i64 @foo()
 
 ; Testg register division.  The result is in the second of the two registers.
 define void @f1(i64 %dummy, i64 %a, i64 %b, i64 *%dest) {
 ; CHECK-LABEL: f1:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgr %r2, %r4
-; CHECK: stg %r3, 0(%r5)
+; CHECK: dsgr %r0, %r4
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %div = sdiv i64 %a, %b
   store i64 %div, i64 *%dest
   ret void
 }
 
 ; Testg register remainder.  The result is in the first of the two registers.
 define void @f2(i64 %dummy, i64 %a, i64 %b, i64 *%dest) {
 ; CHECK-LABEL: f2:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgr %r2, %r4
-; CHECK: stg %r2, 0(%r5)
+; CHECK: dsgr %r0, %r4
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %rem = srem i64 %a, %b
   store i64 %rem, i64 *%dest
   ret void
 }
 
 ; Testg that division and remainder use a single instruction.
 define i64 @f3(i64 %dummy1, i64 %a, i64 %b) {
 ; CHECK-LABEL: f3:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsgr %r2, %r4
+; CHECK: dsgr %r0, %r4
 ; CHECK-NOT: dsgr
 ; CHECK: ogr %r2, %r3
 ; CHECK: br %r14
   %div = sdiv i64 %a, %b
   %rem = srem i64 %a, %b
   %or = or i64 %rem, %div
   ret i64 %or
 }
 
 ; Testg memory division with no displacement.
 define void @f4(i64 %dummy, i64 %a, i64 *%src, i64 *%dest) {
 ; CHECK-LABEL: f4:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsg %r2, 0(%r4)
-; CHECK: stg %r3, 0(%r5)
+; CHECK: dsg %r0, 0(%r4)
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %b = load i64 , i64 *%src
   %div = sdiv i64 %a, %b
   store i64 %div, i64 *%dest
   ret void
 }
 
 ; Testg memory remainder with no displacement.
 define void @f5(i64 %dummy, i64 %a, i64 *%src, i64 *%dest) {
 ; CHECK-LABEL: f5:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsg %r2, 0(%r4)
-; CHECK: stg %r2, 0(%r5)
+; CHECK: dsg %r0, 0(%r4)
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %b = load i64 , i64 *%src
   %rem = srem i64 %a, %b
   store i64 %rem, i64 *%dest
   ret void
 }
 
 ; Testg both memory division and memory remainder.
 define i64 @f6(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f6:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: dsg %r2, 0(%r4)
+; CHECK: dsg %r0, 0(%r4)
 ; CHECK-NOT: {{dsg|dsgr}}
 ; CHECK: ogr %r2, %r3
 ; CHECK: br %r14
   %b = load i64 , i64 *%src
   %div = sdiv i64 %a, %b
   %rem = srem i64 %a, %b
   %or = or i64 %rem, %div
   ret i64 %or
 }
 
 ; Check the high end of the DSG range.
 define i64 @f7(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f7:
-; CHECK: dsg %r2, 524280(%r4)
+; CHECK: dsg %r0, 524280(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 65535
   %b = load i64 , i64 *%ptr
   %rem = srem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check the next doubleword up, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f8(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f8:
 ; CHECK: agfi %r4, 524288
-; CHECK: dsg %r2, 0(%r4)
+; CHECK: dsg %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 65536
   %b = load i64 , i64 *%ptr
   %rem = srem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check the high end of the negative aligned DSG range.
 define i64 @f9(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f9:
-; CHECK: dsg %r2, -8(%r4)
+; CHECK: dsg %r0, -8(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -1
   %b = load i64 , i64 *%ptr
   %rem = srem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check the low end of the DSG range.
 define i64 @f10(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f10:
-; CHECK: dsg %r2, -524288(%r4)
+; CHECK: dsg %r0, -524288(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -65536
   %b = load i64 , i64 *%ptr
   %rem = srem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check the next doubleword down, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f11(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f11:
 ; CHECK: agfi %r4, -524296
-; CHECK: dsg %r2, 0(%r4)
+; CHECK: dsg %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -65537
   %b = load i64 , i64 *%ptr
   %rem = srem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check that DSG allows an index.
 define i64 @f12(i64 %dummy, i64 %a, i64 %src, i64 %index) {
 ; CHECK-LABEL: f12:
-; CHECK: dsg %r2, 524287(%r5,%r4)
+; CHECK: dsg %r0, 524287(%r5,%r4)
 ; CHECK: br %r14
   %add1 = add i64 %src, %index
   %add2 = add i64 %add1, 524287
   %ptr = inttoptr i64 %add2 to i64 *
   %b = load i64 , i64 *%ptr
   %rem = srem i64 %a, %b
   ret i64 %rem
 }

test/CodeGen/SystemZ/int-div-05.ll

 ; Testg 64-bit unsigned division and remainder.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 declare i64 @foo()
 
 ; Testg register division.  The result is in the second of the two registers.
 define void @f1(i64 %dummy, i64 %a, i64 %b, i64 *%dest) {
 ; CHECK-LABEL: f1:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lghi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlgr %r2, %r4
-; CHECK: stg %r3, 0(%r5)
+; CHECK: {{llill|lghi}} %r0, 0
+; CHECK: dlgr %r0, %r4
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %div = udiv i64 %a, %b
   store i64 %div, i64 *%dest
   ret void
 }
 
 ; Testg register remainder.  The result is in the first of the two registers.
 define void @f2(i64 %dummy, i64 %a, i64 %b, i64 *%dest) {
 ; CHECK-LABEL: f2:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lghi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlgr %r2, %r4
-; CHECK: stg %r2, 0(%r5)
+; CHECK: {{llill|lghi}} %r0, 0
+; CHECK: dlgr %r0, %r4
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %rem = urem i64 %a, %b
   store i64 %rem, i64 *%dest
   ret void
 }
 
 ; Testg that division and remainder use a single instruction.
 define i64 @f3(i64 %dummy1, i64 %a, i64 %b) {
 ; CHECK-LABEL: f3:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lghi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlgr %r2, %r4
+; CHECK: {{llill|lghi}} %r0, 0
+; CHECK: dlgr %r0, %r4
 ; CHECK-NOT: dlgr
 ; CHECK: ogr %r2, %r3
 ; CHECK: br %r14
   %div = udiv i64 %a, %b
   %rem = urem i64 %a, %b
   %or = or i64 %rem, %div
   ret i64 %or
 }
 
 ; Testg memory division with no displacement.
 define void @f4(i64 %dummy, i64 %a, i64 *%src, i64 *%dest) {
 ; CHECK-LABEL: f4:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lghi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlg %r2, 0(%r4)
-; CHECK: stg %r3, 0(%r5)
+; CHECK: {{llill|lghi}} %r0, 0
+; CHECK: dlg %r0, 0(%r4)
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %b = load i64 , i64 *%src
   %div = udiv i64 %a, %b
   store i64 %div, i64 *%dest
   ret void
 }
 
 ; Testg memory remainder with no displacement.
 define void @f5(i64 %dummy, i64 %a, i64 *%src, i64 *%dest) {
 ; CHECK-LABEL: f5:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lghi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlg %r2, 0(%r4)
-; CHECK: stg %r2, 0(%r5)
+; CHECK: {{llill|lghi}} %r0, 0
+; CHECK: dlg %r0, 0(%r4)
+; CHECK: stg %r0, 0(%r5)
 ; CHECK: br %r14
   %b = load i64 , i64 *%src
   %rem = urem i64 %a, %b
   store i64 %rem, i64 *%dest
   ret void
 }
 
 ; Testg both memory division and memory remainder.
 define i64 @f6(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f6:
 ; CHECK-NOT: %r3
-; CHECK: {{llill|lghi}} %r2, 0
-; CHECK-NOT: %r3
-; CHECK: dlg %r2, 0(%r4)
+; CHECK: {{llill|lghi}} %r0, 0
+; CHECK: dlg %r0, 0(%r4)
 ; CHECK-NOT: {{dlg|dlgr}}
 ; CHECK: ogr %r2, %r3
 ; CHECK: br %r14
   %b = load i64 , i64 *%src
   %div = udiv i64 %a, %b
   %rem = urem i64 %a, %b
   %or = or i64 %rem, %div
   ret i64 %or
 }
 
 ; Check the high end of the DLG range.
 define i64 @f7(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f7:
-; CHECK: dlg %r2, 524280(%r4)
+; CHECK: dlg %r0, 524280(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 65535
   %b = load i64 , i64 *%ptr
   %rem = urem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check the next doubleword up, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f8(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f8:
 ; CHECK: agfi %r4, 524288
-; CHECK: dlg %r2, 0(%r4)
+; CHECK: dlg %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 65536
   %b = load i64 , i64 *%ptr
   %rem = urem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check the high end of the negative aligned DLG range.
 define i64 @f9(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f9:
-; CHECK: dlg %r2, -8(%r4)
+; CHECK: dlg %r0, -8(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -1
   %b = load i64 , i64 *%ptr
   %rem = urem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check the low end of the DLG range.
 define i64 @f10(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f10:
-; CHECK: dlg %r2, -524288(%r4)
+; CHECK: dlg %r0, -524288(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -65536
   %b = load i64 , i64 *%ptr
   %rem = urem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check the next doubleword down, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f11(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f11:
 ; CHECK: agfi %r4, -524296
-; CHECK: dlg %r2, 0(%r4)
+; CHECK: dlg %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -65537
   %b = load i64 , i64 *%ptr
   %rem = urem i64 %a, %b
   ret i64 %rem
 }
 
 ; Check that DLG allows an index.
 define i64 @f12(i64 %dummy, i64 %a, i64 %src, i64 %index) {
 ; CHECK-LABEL: f12:
-; CHECK: dlg %r2, 524287(%r5,%r4)
+; CHECK: dlg %r0, 524287(%r5,%r4)
 ; CHECK: br %r14
   %add1 = add i64 %src, %index
   %add2 = add i64 %add1, 524287
   %ptr = inttoptr i64 %add2 to i64 *
   %b = load i64 , i64 *%ptr
   %rem = urem i64 %a, %b
   ret i64 %rem
 }

test/CodeGen/SystemZ/int-div-06.ll

 ; Check signed 64-bit division.
 define i64 @f3(i64 %dummy, i64 %a) {
 ; CHECK-LABEL: f3:
 ; CHECK-DAG: llihf [[CONST:%r[0-5]]], 1005497601
 ; CHECK-DAG: oilf [[CONST]], 4251762321
 ; CHECK-DAG: srag [[REG:%r[0-5]]], %r3, 63
 ; CHECK-DAG: ngr [[REG]], [[CONST]]
 ; CHECK-DAG: mlgr %r2, [[CONST]]
-; CHECK: sgr %r2, [[REG]]
-; CHECK: srlg [[RES1:%r[0-5]]], %r2, 63
-; CHECK: srag %r2, %r2, 15
+; CHECK: sgr [[REG2:%r[0-5]]], [[REG]]
+; CHECK: srlg [[RES1:%r[0-5]]], [[REG2]], 63
+; CHECK: srag %r2, [[REG2]], 15
 ; CHECK: agr %r2, [[RES1]]
 ; CHECK: br %r14
   %b = sdiv i64 %a, 139968
   ret i64 %b
 }
 
 ; Check unsigned 64-bit division.
 define i64 @f4(i64 %dummy, i64 %a) {
 ; CHECK-LABEL: f4:
 ; CHECK: llihf [[CONST:%r[0-5]]], 1005497601
 ; CHECK: oilf [[CONST]], 4251762321
 ; CHECK: mlgr %r2, [[CONST]]
-; CHECK: srlg %r2, %r2, 15
+; CHECK: srlg %r2, %r0, 15
 ; CHECK: br %r14
   %b = udiv i64 %a, 139968
   ret i64 %b
 }

test/CodeGen/SystemZ/int-mul-08.ll

 ; Test high-part i64->i128 multiplications.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu | FileCheck %s
 
 declare i64 @foo()
 
 ; Check zero-extended multiplication in which only the high part is used.
 define i64 @f1(i64 %dummy, i64 %a, i64 %b) {
 ; CHECK-LABEL: f1:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: mlgr %r2, %r4
-; CHECK: br %r14
+; CHECK:    mlgr %r0, %r4
+; CHECK:    br %r14
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
; CHECK: br %r14
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check zero-extended multiplication in which only part of the high half
 ; is used.
 define i64 @f3(i64 %dummy, i64 %a, i64 %b) {
 ; CHECK-LABEL: f3:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: mlgr %r2, %r4
-; CHECK: srlg %r2, %r2, 3
-; CHECK: br %r14
+; CHECK:    mlgr %r0, %r4
+; CHECK:    srlg %r2, %r0, 3
+; CHECK:    br %r14
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 67
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check zero-extended multiplication in which the result is split into
 ; high and low halves.
 define i64 @f4(i64 %dummy, i64 %a, i64 %b) {
 ; CHECK-LABEL: f4:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: mlgr %r2, %r4
-; CHECK: ogr %r2, %r3
-; CHECK: br %r14
+; CHECK:    mlgr %r0, %r4
+; CHECK:    ogr %r2, %r3
+; CHECK:    br %r14
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   %low = trunc i128 %mulx to i64
   %or = or i64 %high, %low
   ret i64 %or
 }
 
 ; Check division by a constant, which should use multiplication instead.
 define i64 @f5(i64 %dummy, i64 %a) {
 ; CHECK-LABEL: f5:
-; CHECK: mlgr %r2,
-; CHECK: srlg %r2, %r2,
-; CHECK: br %r14
+; CHECK:    mlgr %r2, %r0
+; CHECK:    srlg %r2, %r0, 9
+; CHECK:    br %r14
   %res = udiv i64 %a, 1234
   ret i64 %res
 }
 
 ; Check MLG with no displacement.
 define i64 @f6(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f6:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: mlg %r2, 0(%r4)
-; CHECK: br %r14
+; CHECK:    mlg %r0, 0(%r4)
+; CHECK:    br %r14
   %b = load i64 , i64 *%src
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the high end of the aligned MLG range.
 define i64 @f7(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f7:
-; CHECK: mlg %r2, 524280(%r4)
-; CHECK: br %r14
+; CHECK:    mlg %r0, 524280(%r4)
+; CHECK:    br %r14
   %ptr = getelementptr i64, i64 *%src, i64 65535
   %b = load i64 , i64 *%ptr
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the next doubleword up, which requires separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f8(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f8:
 ; CHECK: agfi %r4, 524288
-; CHECK: mlg %r2, 0(%r4)
+; CHECK: mlg %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 65536
   %b = load i64 , i64 *%ptr
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the high end of the negative aligned MLG range.
 define i64 @f9(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f9:
-; CHECK: mlg %r2, -8(%r4)
+; CHECK: mlg %r0, -8(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -1
   %b = load i64 , i64 *%ptr
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the low end of the MLG range.
 define i64 @f10(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f10:
-; CHECK: mlg %r2, -524288(%r4)
+; CHECK: mlg %r0, -524288(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -65536
   %b = load i64 , i64 *%ptr
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the next doubleword down, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f11(i64 *%dest, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f11:
 ; CHECK: agfi %r4, -524296
-; CHECK: mlg %r2, 0(%r4)
+; CHECK: mlg %r0, 0(%r4)
 ; CHECK: br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -65537
   %b = load i64 , i64 *%ptr
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check that MLG allows an index.
 define i64 @f12(i64 *%dest, i64 %a, i64 %src, i64 %index) {
 ; CHECK-LABEL: f12:
-; CHECK: mlg %r2, 524287(%r5,%r4)
+; CHECK: mlg %r0, 524287(%r5,%r4)
 ; CHECK: br %r14
   %add1 = add i64 %src, %index
   %add2 = add i64 %add1, 524287
   %ptr = inttoptr i64 %add2 to i64 *
   %b = load i64 , i64 *%ptr
   %ax = zext i64 %a to i128
   %bx = zext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check that multiplications of spilled values can use MLG rather than MLGR.
 define i64 @f13(i64 *%ptr0) {
+
 ; CHECK-LABEL: f13:
 ; CHECK: brasl %r14, foo@PLT
 ; CHECK: mlg {{%r[0-9]+}}, 160(%r15)
 ; CHECK: br %r14
   %ptr1 = getelementptr i64, i64 *%ptr0, i64 2
   %ptr2 = getelementptr i64, i64 *%ptr0, i64 4
   %ptr3 = getelementptr i64, i64 *%ptr0, i64 6
   %ptr4 = getelementptr i64, i64 *%ptr0, i64 8

test/CodeGen/SystemZ/int-mul-10.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; Test signed high-part i64->i128 multiplications on z14.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 | FileCheck %s
 
 declare i64 @foo()
 
 ; Check sign-extended multiplication in which only the high part is used.
 define i64 @f1(i64 %dummy, i64 %a, i64 %b) {
 ; CHECK-LABEL: f1:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: mgrk %r2, %r3, %r4
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    mgrk %r0, %r3, %r4
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    br %r14
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check sign-extended multiplication in which only part of the high half
 ; is used.
 define i64 @f2(i64 %dummy, i64 %a, i64 %b) {
 ; CHECK-LABEL: f2:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: mgrk [[REG:%r[0-9]+]], %r3, %r4
-; CHECK: srlg %r2, [[REG]], 3
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    mgrk %r0, %r3, %r4
+; CHECK-NEXT:    # kill: %R0D<def> %R0D<kill> %R0Q<kill>
+; CHECK-NEXT:    srlg %r2, %r0, 3
+; CHECK-NEXT:    br %r14
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 67
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check sign-extended multiplication in which the result is split into
 ; high and low halves.
 define i64 @f3(i64 %dummy, i64 %a, i64 %b) {
 ; CHECK-LABEL: f3:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: mgrk %r2, %r3, %r4
-; CHECK: ogr %r2, %r3
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    mgrk %r0, %r3, %r4
+; CHECK-NEXT:    lgr %r3, %r1
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    ogr %r2, %r3
+; CHECK-NEXT:    br %r14
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   %low = trunc i128 %mulx to i64
   %or = or i64 %high, %low
   ret i64 %or
 }
 
 ; Check MG with no displacement.
 define i64 @f4(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f4:
-; CHECK-NOT: {{%r[234]}}
-; CHECK: mg %r2, 0(%r4)
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    mg %r0, 0(%r4)
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    br %r14
   %b = load i64 , i64 *%src
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the high end of the aligned MG range.
 define i64 @f5(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f5:
-; CHECK: mg %r2, 524280(%r4)
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    mg %r0, 524280(%r4)
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    br %r14
   %ptr = getelementptr i64, i64 *%src, i64 65535
   %b = load i64 , i64 *%ptr
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the next doubleword up, which requires separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f6(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f6:
-; CHECK: agfi %r4, 524288
-; CHECK: mg %r2, 0(%r4)
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    agfi %r4, 524288
+; CHECK-NEXT:    mg %r0, 0(%r4)
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    br %r14
   %ptr = getelementptr i64, i64 *%src, i64 65536
   %b = load i64 , i64 *%ptr
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the high end of the negative aligned MG range.
 define i64 @f7(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f7:
-; CHECK: mg %r2, -8(%r4)
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    mg %r0, -8(%r4)
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -1
   %b = load i64 , i64 *%ptr
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the low end of the MG range.
 define i64 @f8(i64 %dummy, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f8:
-; CHECK: mg %r2, -524288(%r4)
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    mg %r0, -524288(%r4)
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -65536
   %b = load i64 , i64 *%ptr
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check the next doubleword down, which needs separate address logic.
 ; Other sequences besides this one would be OK.
 define i64 @f9(i64 *%dest, i64 %a, i64 *%src) {
 ; CHECK-LABEL: f9:
-; CHECK: agfi %r4, -524296
-; CHECK: mg %r2, 0(%r4)
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    agfi %r4, -524296
+; CHECK-NEXT:    mg %r0, 0(%r4)
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    br %r14
   %ptr = getelementptr i64, i64 *%src, i64 -65537
   %b = load i64 , i64 *%ptr
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 
 ; Check that MG allows an index.
 define i64 @f10(i64 *%dest, i64 %a, i64 %src, i64 %index) {
 ; CHECK-LABEL: f10:
-; CHECK: mg %r2, 524287(%r5,%r4)
-; CHECK: br %r14
+; CHECK:       # BB#0:
+; CHECK-NEXT:    lgr %r1, %r3
+; CHECK-NEXT:    mg %r0, 524287(%r5,%r4)
+; CHECK-NEXT:    lgr %r2, %r0
+; CHECK-NEXT:    br %r14
   %add1 = add i64 %src, %index
   %add2 = add i64 %add1, 524287
   %ptr = inttoptr i64 %add2 to i64 *
   %b = load i64 , i64 *%ptr
   %ax = sext i64 %a to i128
   %bx = sext i64 %b to i128
   %mulx = mul i128 %ax, %bx
   %highx = lshr i128 %mulx, 64
   %high = trunc i128 %highx to i64
   ret i64 %high
 }
 

test/CodeGen/SystemZ/pr32372.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc %s -o - -mtriple=s390x-linux-gnu | FileCheck %s
 
 define void @pr32372(i8*) {
 ; CHECK-LABEL: pr32372:
 ; CHECK:       # BB#0: # %BB
-; CHECK-NEXT:    llc %r1, 0(%r2)
+; CHECK-NEXT:    llc %r0, 0(%r2)
 ; CHECK-NEXT:    mvhhi 0(%r1), -3825
-; CHECK-NEXT:    llill %r0, 0
-; CHECK-NEXT:    dlr %r0, %r1
+; CHECK-NEXT:    llill %r1, 0
+; CHECK-NEXT:    lgr %r2, %r1
+; CHECK-NEXT:    lgr %r3, %r0
+; CHECK-NEXT:    dlr %r2, %r0
 ; CHECK-NEXT:  .LBB0_1: # %CF251
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    j .LBB0_1
 BB:
   %L = load i8, i8* %0
   store i16 -3825, i16* undef
   %L5 = load i8, i8* %0
   %B9 = urem i8 %L5, %L

test/CodeGen/SystemZ/regalloc-GR128.ll

+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 -O3 -o /dev/null
+;
+; Test that regalloc does not run out of registers
+
+define void @test(i64 %dividend, i64 %divisor) {
+  %rem = urem i64 %dividend, %divisor
+  call void asm sideeffect "", "{r0},{r1},{r2},{r3},{r4},{r5},{r6},{r7},{r8},{r9},{r10},{r11},{r12},{r13},{r14}"(i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 0, i64 %rem)
+  ret void
+}