[SystemZ] Fix handling of CC values

[jonpa]

Summary

It was recently discovered that the handling of CC values was actually broken since overflow was not properly handled.

Add and sub instructions have now a new target specific instruction flag called SystemZII::CCIfNoSWrapOnly. This means that the CC result can be used instead of a compare with 0 only if the instruction has the 'nsw' flag set. Without 'nsw', the compare can not be eliminated.

Load complement/negative/positive can also overflow which means they should as well not be used for compare-with-0 elimination.

[jonpa]

Patch updated.

[uweigand]

lib/Target/SystemZ/SystemZInstrInfo.td

1020 ↗

(On Diff #218413)

So strictly speaking, subtraction is different than addition in that there can never be any overflow with a zero result for a subtraction. For for sub, the CompareZeroCCMask = 0x8 setting is actually correct.

Of course, the CCIfNoSWrapOnly = 1 would also be correct. Is there a way to support both (if we have nsw, support any comparison, if we don't, only support comparison against 0)?

test/CodeGen/SystemZ/int-cmp-44.ll

9 ↗

(On Diff #218413)

This comment is actually wrong now. I believe this comment should state that the CC value of an addition can be used to omit a compare if we know no signed overflow happens.

In fact, I think it would be good to have two sets of test cases: one (probably the current one) that verifies that in the absence of the nsw flag, we get the extra compares, and another one that verifies that in the presence of nsw, we never have a compare.

120 ↗

(On Diff #218413)

Same comment as above applies.

[jonpa]

Updated per the review so far.

lib/Target/SystemZ/SystemZInstrInfo.td

1020 ↗

(On Diff #218413)

As discussed this should work by just adding back the CompareZeroCCMask of 0x8 for the subs.

test/CodeGen/SystemZ/int-cmp-44.ll

9 ↗

(On Diff #218413)

Fixed comment.

As discussed, this is covered by the new int-cmp-56.ll test.

120 ↗

(On Diff #218413)

I added similar new tests for subtraction in int-cmp-45.ll.

[jonpa]

test/CodeGen/SystemZ/int-cmp-44.ll

120 ↗

(On Diff #218413)

Sorry - they were added in int-cmp-56.ll.

[uweigand]

With the current form of the patch, do we see any noticeable regression due to no additional compares after addition? If no, the patch would be OK; if yes, we'll probably have to add support for AL etc. first.

lib/Target/SystemZ/SystemZInstrFormats.td

93 ↗

(On Diff #218656)

I'd say something like "sets CC like a compare of the result against zero" instead of "according its CCValues".

test/CodeGen/SystemZ/loop-01.ll

97 ↗

(On Diff #218656)

This is because of this, right?
http://lists.llvm.org/pipermail/llvm-dev/2019-September/134876.html

That should preferably be fixed first. (Or at least marked with a FIXME here.)

[jonpa]

Patch updated:

adjustCCMasksForInstr() extended to

• recognize signed add with immediate and allow these cases by setting the overflow flag in CCMask as required. Now there are many fewer compares instead of many more on SPEC 2006.

• allow logical instructions to replace a compare with 0 for EQ/NE. This only handled just a few more cases. Next step is to find the best way to emit more logical instructions or change opcode to in cases where it will eliminate the comparison.

CCMASK_LOGICAL_NONZERO should be: CCMASK_1 | CCMASK_3, right?

[jonpa]

• Added assert when detecting an add with immediate, that this is not an immediate offset of a memory instruction but indeed an immediate operand.

• Removed check of SystemZII::getCompareZeroCCMask() when detecting add with immediate, since there are no sub with immediate with SystemZII::CCIfNoSignedWrap set.

• int-cmp-56.ll test updated to reflect the use of add logical.

• New method convertToLogical() in SystemZElimCompare that tries conversion to logical add instructions. This handles just some 130 more cases. Is opcode table ok in convertToLogical(), or should it be done next to the instruction definitions. I tried to do it in tablegen, but I couldn't find a neat way to do it - here is what I have so far:
  getLogicalOpcode.patch4 KBDownload
  .

I tried to do this conversion also during instruction selection. I made an experimental patch that would select logical adds for just those cases where the instruction does not have "nsw" while it is being compared if equal to zero. I found drawbacks with this: 1. MachineCSE is less successful now since it cannot remove an ALR even if the operands are the same as of the AR. This seemed to be ~400 cases on SPEC 2006. 2. This does not handle the cases of spilled registers that have been folded into an add with memory instruction. 3. The method I had to write for this became quite cumbersome (the might be a simpler way...), and it still was not "perfect" - there is no simple way to check if CC is clobbered between the add and the compare, by a call for instance. This is my patch where I tried this:

selectUAddForCmp0.patch5 KBDownload

In addition, I also tried to select add logical during instruction selection "always except in case of 'nsw'". This seems to give very similar results to convertToLogical(), except for all the add logical instructions everywhere. +50 tests fail with this.

• With this patch there are now ~3000 less compares (including fused ones) than on master. This is due to the improved handling of add with immediate.

• As seen in int-cmp-58.mir, the printed instruction names reflect CC as if it was set by a signed instruction: LOGICAL_ZERO becomes "locrhe", and LOGICAL_NONZERO "locrnhe", after a logical add. After a sub I see "locrh" / "locrnhe". Would it be feasible to support more reasonable mnemonics for these cases?

I am waiting for review OK for correctness before doing more extensive benchmarking.

lib/Target/SystemZ/SystemZInstrFormats.td

93 ↗

(On Diff #218656)

nice - changed it.

test/CodeGen/SystemZ/loop-01.ll

97 ↗

(On Diff #218656)

yes, that's what I saw and asked about on the list.

I marked it as FIXME for now. It should get fixed once we handle add with immediate cases in the backend, regardless of LSR/SelectionDAGBuilder.

[jonpa]

Remove the assert "A logical opcode was selected in presence of the nsw flag." This is too aggressive, as a zero extension can be folded into the instruction regardless of "nsw" (e.g. ALGFR).

[uweigand]

See inline comments. As an aside, the whole adjustCCMasksForInstr routine is now rather complicated -- this could probably use a complete rewrite, but I'm not really sure what the best approach here would be ...

lib/Target/SystemZ/SystemZElimCompare.cpp

364 ↗

(On Diff #219674)

The comment should probably elaborate a bit, along the lines of: "If adding a positive immediate overflows, the result must be less than zero. If adding a negative immediate overflow, the result must be larger than zero (except in the special case of adding the minimum value of the result range, in which case we cannot predict whether the result is larger than or equal to zero)."

Note that this logic is only correct if the operation performs an addition. Given that you only set the CCIfNoSignedWrap for add and sub instructions, and there is no sub with immediate operand, this condition happens to be true here, but that's a bit implicit ... Maybe add some assert?

366 ↗

(On Diff #219674)

This should probably use int64_t to avoid creating a dependency on the size of the host "int" type.

389 ↗

(On Diff #219674)

Probably better to move that assert now up to the only branch where it is useful.

393 ↗

(On Diff #219674)

This now looks incorrect in either the IsLogical case or in the OFImplies case. In either of those cases, we must go through the loop below rewriting CCUsers.

Again, probably best to move the setting of MIEquivalentToCmp to the branches above.

434 ↗

(On Diff #219674)

The assert seems unnecessary -- any CCMask other than CCMASK_CMP_EQ can be correctly mapped to CCMASK_LOGICAL_NONZERO here if the loop above passed the OutMask check.

[jonpa]

Updated per review.

In addition, I removed

CCValues &= ~SystemZ::CCMASK_ARITH_OVERFLOW;

from the case where both MachineInstr::NoSWrap and SystemZII::CCIfNoSignedWrap are present. This seems mostly confusing to me know, as CCValues is supposed to simply be what CC values the instruction may set, so if the instruction can set the OF bit, CCValues should already reflect this (which it also currently does).

I also added an assert for the CC operands of the CCUsers.

All the changes of this update are NFC on SPEC 2006.

As an aside, the whole adjustCCMasksForInstr routine is now rather complicated -- this could probably use a complete rewrite, but I'm not really sure what the best approach here would be ...

I agree that there are too many assumptions and complications - I might give it a try to find a more simplistic approach.

lib/Target/SystemZ/SystemZElimCompare.cpp

364 ↗

(On Diff #219674)

Comment updated.

I added an assert with a check against known opcodes for add with immediate, since I could not find any MI flag to check against.

389 ↗

(On Diff #219674)

ah, yes.

393 ↗

(On Diff #219674)

Yes... It seems that those cases were currently not passing this test anyway, but I agree that it certainly makes sense to move this up.

The first case with MachineInstr::NoSWrap also should not set this as the CCValues will now include the OF bit and this should be updated in the CCUsers CCValid operands.

434 ↗

(On Diff #219674)

ok - removed it.

[uweigand]

Thanks, this LGTM now.

I think we need to verify that benchmark results are good before committing this change, however.

[jonpa]

Here are some minor NFC changes in an attempt to make things easier to follow. Is this better?

CC_rewrite.patch32 KBDownload

[uweigand]

Yes, this does look a bit better. Maybe you can even avoid the IntCC variable by doing something like:

unsigned ReusableCCMask = CCValues;
if (CompareFlags & SystemZII::IsLogical)
  ReusableCCMask &= SystemZ::CCMASK_CMP_EQ;

before the various conditions, and then simply & new constraints into ReuableCCMask as required. For the CCIfNoSignedWrap case, this would be nothing, for the IsLogical case, this would be

ReusableCCMask &= SystemZ::CCMASK_CMP_EQ;

and for the CompareZeroCCMask case this would be

ReusableCCMask &= SystemZII::getCompareZeroCCMask(MIFlags);

[jonpa]

Maybe you can even avoid the IntCC variable...

Yes :-)

I like the AND:ing away things from ReusableCCMask after starting out with CCValues, however the logical case cannot play along here since those instructions has CCValues in the logical domain. Locigal subtracts have 0x7, which does not include the SystemZ::CCMASK_CMP_EQ bit. So this really needs to set ReusableCCMask to SystemZ::CCMASK_CMP_EQ based on the assumption that all logical instructions that has any CCValues support a comparison to zero.

I kept the empty first if clause in lack of any better ideas.

Maybe we could remove

if (ReusableCCMask == 0)

return false;

since the loop will return immediately anyway?

I strengthened the assert further for "CC operands of CCUser", to also check that CCMask is non-zero and that it is not equal to CCValid.

[uweigand]

In D66868#1710696, @jonpa wrote:

Maybe you can even avoid the IntCC variable...

Yes :-)

I like the AND:ing away things from ReusableCCMask after starting out with CCValues, however the logical case cannot play along here since those instructions has CCValues in the logical domain. Locigal subtracts have 0x7, which does not include the SystemZ::CCMASK_CMP_EQ bit. So this really needs to set ReusableCCMask to SystemZ::CCMASK_CMP_EQ based on the assumption that all logical instructions that has any CCValues support a comparison to zero.

OK, I see.

I kept the empty first if clause in lack of any better ideas.

Fine with me. Maybe just a bit more comment than "done" :-)

Maybe we could remove

if (ReusableCCMask == 0)

return false;

since the loop will return immediately anyway?

Not sure this is true. In any case, having this as early exist makes sense to me ...

I strengthened the assert further for "CC operands of CCUser", to also check that CCMask is non-zero and that it is not equal to CCValid.

I'm not sure this is completely safe either. Yes, in theory we should never see always-true or always-false conditions, but I believe it is not completely impossible that those might occur depending on optimization levels etc. In any case, the rest of the code should actually handle those cases correctly, so I'm not sure we really need to prevent them here.

Otherwise, the patch looks good to me now, assuming we get encouraging performance results.

[jonpa]

Updated per review: Comment improved, and extra assertions removed.

Not sure this is true. In any case, having this as early exist makes sense to me ...

The asserts passed with -O0/SPEC 2006, so I assumed this was guaranteed.

[jonpa]

Patch rebased, no manual changes needed...

[uweigand]

LGTM, thanks!

[jonpa]

Thanks for review!

3174683e

llvm/lib/Target/SystemZ/SystemZ.h

 const unsigned CCMASK_ARITH_EQ       = CCMASK_0;
 const unsigned CCMASK_ARITH_LT       = CCMASK_1;
 const unsigned CCMASK_ARITH_GT       = CCMASK_2;
 const unsigned CCMASK_ARITH_OVERFLOW = CCMASK_3;
 const unsigned CCMASK_ARITH          = CCMASK_ANY;
 
 // Condition-code mask assignments for logical operations.
 const unsigned CCMASK_LOGICAL_ZERO     = CCMASK_0 | CCMASK_2;
-const unsigned CCMASK_LOGICAL_NONZERO  = CCMASK_1 | CCMASK_2;
+const unsigned CCMASK_LOGICAL_NONZERO  = CCMASK_1 | CCMASK_3;
 const unsigned CCMASK_LOGICAL_CARRY    = CCMASK_2 | CCMASK_3;
 const unsigned CCMASK_LOGICAL_NOCARRY  = CCMASK_0 | CCMASK_1;
 const unsigned CCMASK_LOGICAL_BORROW   = CCMASK_LOGICAL_NOCARRY;
 const unsigned CCMASK_LOGICAL_NOBORROW = CCMASK_LOGICAL_CARRY;
 const unsigned CCMASK_LOGICAL          = CCMASK_ANY;
 
 // Condition-code mask assignments for CS.
 const unsigned CCMASK_CS_EQ = CCMASK_0;

llvm/lib/Target/SystemZ/SystemZElimCompare.cpp

 private:
   Reference getRegReferences(MachineInstr &MI, unsigned Reg);
   bool convertToBRCT(MachineInstr &MI, MachineInstr &Compare,
                      SmallVectorImpl<MachineInstr *> &CCUsers);
   bool convertToLoadAndTrap(MachineInstr &MI, MachineInstr &Compare,
                             SmallVectorImpl<MachineInstr *> &CCUsers);
   bool convertToLoadAndTest(MachineInstr &MI, MachineInstr &Compare,
                             SmallVectorImpl<MachineInstr *> &CCUsers);
+  bool convertToLogical(MachineInstr &MI, MachineInstr &Compare,
+                        SmallVectorImpl<MachineInstr *> &CCUsers);
   bool adjustCCMasksForInstr(MachineInstr &MI, MachineInstr &Compare,
                              SmallVectorImpl<MachineInstr *> &CCUsers,
                              unsigned ConvOpc = 0);
   bool optimizeCompareZero(MachineInstr &Compare,
                            SmallVectorImpl<MachineInstr *> &CCUsers);
   bool fuseCompareOperations(MachineInstr &Compare,
                              SmallVectorImpl<MachineInstr *> &CCUsers);
 
bool SystemZElimCompare::convertToLoadAndTest(
   // Mark instruction as raising an FP exception if applicable.  We already
   // verified earlier that this move is valid.
   if (Compare.mayRaiseFPException())
     MIB.setMIFlag(MachineInstr::MIFlag::FPExcept);
 
   return true;
 }
 
+// See if MI is an instruction with an equivalent "logical" opcode that can
+// be used and replace MI. This is useful for EQ/NE comparisons where the
+// "nsw" flag is missing since the "logical" opcode always sets CC to reflect
+// the result being zero or non-zero.
+bool SystemZElimCompare::convertToLogical(
+    MachineInstr &MI, MachineInstr &Compare,
+    SmallVectorImpl<MachineInstr *> &CCUsers) {
+
+  unsigned ConvOpc = 0;
+  switch (MI.getOpcode()) {
+  case SystemZ::AR:   ConvOpc = SystemZ::ALR;   break;
+  case SystemZ::ARK:  ConvOpc = SystemZ::ALRK;  break;
+  case SystemZ::AGR:  ConvOpc = SystemZ::ALGR;  break;
+  case SystemZ::AGRK: ConvOpc = SystemZ::ALGRK; break;
+  case SystemZ::A:    ConvOpc = SystemZ::AL;    break;
+  case SystemZ::AY:   ConvOpc = SystemZ::ALY;   break;
+  case SystemZ::AG:   ConvOpc = SystemZ::ALG;   break;
+  default: break;
+  }
+  if (!ConvOpc || !adjustCCMasksForInstr(MI, Compare, CCUsers, ConvOpc))
+    return false;
+
+  // Operands should be identical, so just change the opcode and remove the
+  // dead flag on CC.
+  MI.setDesc(TII->get(ConvOpc));
+  MI.clearRegisterDeads(SystemZ::CC);
+  return true;
+}
+
+#ifndef NDEBUG
+static bool isAddWithImmediate(unsigned Opcode) {
+  switch(Opcode) {
+  case SystemZ::AHI:
+  case SystemZ::AHIK:
+  case SystemZ::AGHI:
+  case SystemZ::AGHIK:
+  case SystemZ::AFI:
+  case SystemZ::AIH:
+  case SystemZ::AGFI:
+    return true;
+  default: break;
+  }
+  return false;
+}
+#endif
+
 // The CC users in CCUsers are testing the result of a comparison of some
 // value X against zero and we know that any CC value produced by MI would
 // also reflect the value of X.  ConvOpc may be used to pass the transfomed
 // opcode MI will have if this succeeds.  Try to adjust CCUsers so that they
 // test the result of MI directly, returning true on success.  Leave
 // everything unchanged on failure.
 bool SystemZElimCompare::adjustCCMasksForInstr(
     MachineInstr &MI, MachineInstr &Compare,
     SmallVectorImpl<MachineInstr *> &CCUsers,
     unsigned ConvOpc) {
+  unsigned CompareFlags = Compare.getDesc().TSFlags;
+  unsigned CompareCCValues = SystemZII::getCCValues(CompareFlags);
   int Opcode = (ConvOpc ? ConvOpc : MI.getOpcode());
   const MCInstrDesc &Desc = TII->get(Opcode);
   unsigned MIFlags = Desc.TSFlags;
 
   // If Compare may raise an FP exception, we can only eliminate it
   // if MI itself would have already raised the exception.
   if (Compare.mayRaiseFPException()) {
     // If the caller will change MI to use ConvOpc, only test whether
     // ConvOpc is suitable; it is on the caller to set the MI flag.
     if (ConvOpc && !Desc.mayRaiseFPException())
       return false;
     // If the caller will not change MI, we test the MI flag here.
     if (!ConvOpc && !MI.mayRaiseFPException())
       return false;
   }
 
   // See which compare-style condition codes are available.
-  unsigned ReusableCCMask = SystemZII::getCompareZeroCCMask(MIFlags);
-
+  unsigned CCValues = SystemZII::getCCValues(MIFlags);
+  unsigned ReusableCCMask = CCValues;
   // For unsigned comparisons with zero, only equality makes sense.
-  unsigned CompareFlags = Compare.getDesc().TSFlags;
   if (CompareFlags & SystemZII::IsLogical)
     ReusableCCMask &= SystemZ::CCMASK_CMP_EQ;
-
+  unsigned OFImplies = 0;
+  bool LogicalMI = false;
+  bool MIEquivalentToCmp = false;
+  if (MI.getFlag(MachineInstr::NoSWrap) &&
+      (MIFlags & SystemZII::CCIfNoSignedWrap)) {
+    // If MI has the NSW flag set in combination with the
+    // SystemZII::CCIfNoSignedWrap flag, all CCValues are valid.
+  }
+  else if ((MIFlags & SystemZII::CCIfNoSignedWrap) &&
+           MI.getOperand(2).isImm()) {
+    // Signed addition of immediate. If adding a positive immediate
+    // overflows, the result must be less than zero. If adding a negative
+    // immediate overflows, the result must be larger than zero (except in
+    // the special case of adding the minimum value of the result range, in
+    // which case we cannot predict whether the result is larger than or
+    // equal to zero).
+    assert(isAddWithImmediate(Opcode) && "Expected an add with immediate.");
+    assert(!MI.mayLoadOrStore() && "Expected an immediate term.");
+    int64_t RHS = MI.getOperand(2).getImm();
+    if (SystemZ::GRX32BitRegClass.contains(MI.getOperand(0).getReg()) &&
+        RHS == INT32_MIN)
+      return false;
+    OFImplies = (RHS > 0 ? SystemZ::CCMASK_CMP_LT : SystemZ::CCMASK_CMP_GT);
+  }
+  else if ((MIFlags & SystemZII::IsLogical) && CCValues) {
+    // Use CCMASK_CMP_EQ to match with CCUsers. On success CCMask:s will be
+    // converted to CCMASK_LOGICAL_ZERO or CCMASK_LOGICAL_NONZERO.
+    LogicalMI = true;
+    ReusableCCMask = SystemZ::CCMASK_CMP_EQ;
+  }
+  else {
+    ReusableCCMask &= SystemZII::getCompareZeroCCMask(MIFlags);
+    assert((ReusableCCMask & ~CCValues) == 0 && "Invalid CCValues");
+    MIEquivalentToCmp =
+      ReusableCCMask == CCValues && CCValues == CompareCCValues;
+  }
   if (ReusableCCMask == 0)
     return false;
 
-  unsigned CCValues = SystemZII::getCCValues(MIFlags);
-  assert((ReusableCCMask & ~CCValues) == 0 && "Invalid CCValues");
-
-  bool MIEquivalentToCmp =
-    (ReusableCCMask == CCValues &&
-     CCValues == SystemZII::getCCValues(CompareFlags));
-
   if (!MIEquivalentToCmp) {
     // Now check whether these flags are enough for all users.
     SmallVector<MachineOperand *, 4> AlterMasks;
     for (unsigned int I = 0, E = CCUsers.size(); I != E; ++I) {
-      MachineInstr *MI = CCUsers[I];
+      MachineInstr *CCUserMI = CCUsers[I];
 
       // Fail if this isn't a use of CC that we understand.
-      unsigned Flags = MI->getDesc().TSFlags;
+      unsigned Flags = CCUserMI->getDesc().TSFlags;
       unsigned FirstOpNum;
       if (Flags & SystemZII::CCMaskFirst)
         FirstOpNum = 0;
       else if (Flags & SystemZII::CCMaskLast)
-        FirstOpNum = MI->getNumExplicitOperands() - 2;
+        FirstOpNum = CCUserMI->getNumExplicitOperands() - 2;
       else
         return false;
 
       // Check whether the instruction predicate treats all CC values
       // outside of ReusableCCMask in the same way.  In that case it
       // doesn't matter what those CC values mean.
-      unsigned CCValid = MI->getOperand(FirstOpNum).getImm();
-      unsigned CCMask = MI->getOperand(FirstOpNum + 1).getImm();
+      unsigned CCValid = CCUserMI->getOperand(FirstOpNum).getImm();
+      unsigned CCMask = CCUserMI->getOperand(FirstOpNum + 1).getImm();
+      assert(CCValid == CompareCCValues && (CCMask & ~CCValid) == 0 &&
+             "Corrupt CC operands of CCUser.");
       unsigned OutValid = ~ReusableCCMask & CCValid;
       unsigned OutMask = ~ReusableCCMask & CCMask;
       if (OutMask != 0 && OutMask != OutValid)
         return false;
 
-      AlterMasks.push_back(&MI->getOperand(FirstOpNum));
-      AlterMasks.push_back(&MI->getOperand(FirstOpNum + 1));
+      AlterMasks.push_back(&CCUserMI->getOperand(FirstOpNum));
+      AlterMasks.push_back(&CCUserMI->getOperand(FirstOpNum + 1));
     }
 
     // All users are OK.  Adjust the masks for MI.
     for (unsigned I = 0, E = AlterMasks.size(); I != E; I += 2) {
       AlterMasks[I]->setImm(CCValues);
       unsigned CCMask = AlterMasks[I + 1]->getImm();
+      if (LogicalMI) {
+        // Translate the CCMask into its "logical" value.
+        CCMask = (CCMask == SystemZ::CCMASK_CMP_EQ ?
+                  SystemZ::CCMASK_LOGICAL_ZERO : SystemZ::CCMASK_LOGICAL_NONZERO);
+        CCMask &= CCValues; // Logical subtracts never set CC=0.
+      } else {
         if (CCMask & ~ReusableCCMask)
-        AlterMasks[I + 1]->setImm((CCMask & ReusableCCMask) |
-                                  (CCValues & ~ReusableCCMask));
+          CCMask = (CCMask & ReusableCCMask) | (CCValues & ~ReusableCCMask);
+        CCMask |= (CCMask & OFImplies) ? SystemZ::CCMASK_ARITH_OVERFLOW : 0;
+      }
+      AlterMasks[I + 1]->setImm(CCMask);
     }
   }
 
   // CC is now live after MI.
   if (!ConvOpc)
     MI.clearRegisterDeads(SystemZ::CC);
 
   // Check if MI lies before Compare.
if (resultTests(MI, SrcReg)) {
         }
         if (convertToLoadAndTrap(MI, Compare, CCUsers)) {
           LoadAndTraps += 1;
           return true;
         }
       }
       // Try to eliminate Compare by reusing a CC result from MI.
       if ((!CCRefs && convertToLoadAndTest(MI, Compare, CCUsers)) ||
-          (!CCRefs.Def && adjustCCMasksForInstr(MI, Compare, CCUsers))) {
+          (!CCRefs.Def &&
+           (adjustCCMasksForInstr(MI, Compare, CCUsers) ||
+            convertToLogical(MI, Compare, CCUsers)))) {
         EliminatedComparisons += 1;
         return true;
       }
     }
     SrcRefs |= getRegReferences(MI, SrcReg);
     if (SrcRefs.Def)
       break;
     CCRefs |= getRegReferences(MI, SystemZ::CC);

llvm/lib/Target/SystemZ/SystemZInstrFormats.td

class InstSystemZ<int size, dag outs, dag ins, string asmstr,
   // The access size of all memory operands in bytes, or 0 if not known.
   bits<5> AccessBytes = 0;
 
   // If the instruction sets CC to a useful value, this gives the mask
   // of all possible CC results.  The mask has the same form as
   // SystemZ::CCMASK_*.
   bits<4> CCValues = 0;
 
-  // The subset of CCValues that have the same meaning as they would after
-  // a comparison of the first operand against zero.
+  // The subset of CCValues that have the same meaning as they would after a
+  // comparison of the first operand against zero. "Logical" instructions
+  // leave this blank as they set CC in a different way.
   bits<4> CompareZeroCCMask = 0;
 
   // True if the instruction is conditional and if the CC mask operand
   // comes first (as for BRC, etc.).
   bit CCMaskFirst = 0;
 
   // Similar, but true if the CC mask operand comes last (as for LOC, etc.).
   bit CCMaskLast = 0;
 
   // True if the instruction is the "logical" rather than "arithmetic" form,
-  // in cases where a distinction exists.
+  // in cases where a distinction exists. Except for logical compares, if the
+  // instruction sets this flag along with a non-zero CCValues field, it is
+  // assumed to set CC to either CCMASK_LOGICAL_ZERO or
+  // CCMASK_LOGICAL_NONZERO.
   bit IsLogical = 0;
 
+  // True if the (add or sub) instruction sets CC like a compare of the
+  // result against zero, but only if the 'nsw' flag is set.
+  bit CCIfNoSignedWrap = 0;
+
   let TSFlags{0}     = SimpleBDXLoad;
   let TSFlags{1}     = SimpleBDXStore;
   let TSFlags{2}     = Has20BitOffset;
   let TSFlags{3}     = HasIndex;
   let TSFlags{4}     = Is128Bit;
   let TSFlags{9-5}   = AccessBytes;
   let TSFlags{13-10} = CCValues;
   let TSFlags{17-14} = CompareZeroCCMask;
   let TSFlags{18}    = CCMaskFirst;
   let TSFlags{19}    = CCMaskLast;
   let TSFlags{20}    = IsLogical;
+  let TSFlags{21}    = CCIfNoSignedWrap;
 }
 
 //===----------------------------------------------------------------------===//
 // Mappings between instructions
 //===----------------------------------------------------------------------===//
 
 // Return the version of an instruction that has an unsigned 12-bit
 // displacement.

llvm/lib/Target/SystemZ/SystemZInstrInfo.h

enum {
   AccessSizeMask         = (31 << 5),
   AccessSizeShift        = 5,
   CCValuesMask           = (15 << 10),
   CCValuesShift          = 10,
   CompareZeroCCMaskMask  = (15 << 14),
   CompareZeroCCMaskShift = 14,
   CCMaskFirst            = (1 << 18),
   CCMaskLast             = (1 << 19),
-  IsLogical              = (1 << 20)
+  IsLogical              = (1 << 20),
+  CCIfNoSignedWrap       = (1 << 21)
 };
 
 static inline unsigned getAccessSize(unsigned int Flags) {
   return (Flags & AccessSizeMask) >> AccessSizeShift;
 }
 
 static inline unsigned getCCValues(unsigned int Flags) {
   return (Flags & CCValuesMask) >> CCValuesShift;

llvm/lib/Target/SystemZ/SystemZInstrInfo.cpp

 static void transferDeadCC(MachineInstr *OldMI, MachineInstr *NewMI) {
   if (OldMI->registerDefIsDead(SystemZ::CC)) {
     MachineOperand *CCDef = NewMI->findRegisterDefOperand(SystemZ::CC);
     if (CCDef != nullptr)
       CCDef->setIsDead(true);
   }
 }
 
+static void transferMIFlag(MachineInstr *OldMI, MachineInstr *NewMI,
+                           MachineInstr::MIFlag Flag) {
+  if (OldMI->getFlag(Flag))
+    NewMI->setFlag(Flag);
+}
+
 MachineInstr *SystemZInstrInfo::convertToThreeAddress(
     MachineFunction::iterator &MFI, MachineInstr &MI, LiveVariables *LV) const {
   MachineBasicBlock *MBB = MI.getParent();
 
   // Try to convert an AND into an RISBG-type instruction.
   // TODO: It might be beneficial to select RISBG and shorten to AND instead.
   if (LogicOp And = interpretAndImmediate(MI.getOpcode())) {
     uint64_t Imm = MI.getOperand(2).getImm() << And.ImmLSB;
if ((Opcode == SystemZ::AHI || Opcode == SystemZ::AGHI) && OpNum == 0 &&
     // A(G)HI %reg, CONST -> A(G)SI %mem, CONST
     Opcode = (Opcode == SystemZ::AHI ? SystemZ::ASI : SystemZ::AGSI);
     MachineInstr *BuiltMI =
         BuildMI(*InsertPt->getParent(), InsertPt, MI.getDebugLoc(), get(Opcode))
             .addFrameIndex(FrameIndex)
             .addImm(0)
             .addImm(MI.getOperand(2).getImm());
     transferDeadCC(&MI, BuiltMI);
+    transferMIFlag(&MI, BuiltMI, MachineInstr::NoSWrap);
     return BuiltMI;
   }
 
   if ((Opcode == SystemZ::ALFI && OpNum == 0 &&
        isInt<8>((int32_t)MI.getOperand(2).getImm())) ||
       (Opcode == SystemZ::ALGFI && OpNum == 0 &&
        isInt<8>((int64_t)MI.getOperand(2).getImm()))) {
     // AL(G)FI %reg, CONST -> AL(G)SI %mem, CONST
if ((OpNum == NumOps - 1) || NeedsCommute) {
         MIB.add(MI.getOperand(2));
       else
         for (unsigned I = 1; I < OpNum; ++I)
           MIB.add(MI.getOperand(I));
       MIB.addFrameIndex(FrameIndex).addImm(Offset);
       if (MemDesc.TSFlags & SystemZII::HasIndex)
         MIB.addReg(0);
       transferDeadCC(&MI, MIB);
+      transferMIFlag(&MI, MIB, MachineInstr::NoSWrap);
       return MIB;
     }
   }
 
   return nullptr;
 }
 
 MachineInstr *SystemZInstrInfo::foldMemoryOperandImpl(

llvm/lib/Target/SystemZ/SystemZInstrInfo.td

def : Pat<(or (zext32 GR32:$src), imm64hf32:$imm),
           (IIHF64 (INSERT_SUBREG (i64 (IMPLICIT_DEF)), GR32:$src, subreg_l32),
                   imm64hf32:$imm)>;
 
 //===----------------------------------------------------------------------===//
 // Addition
 //===----------------------------------------------------------------------===//
 
 // Addition producing a signed overflow flag.
-let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8 in {
+let Defs = [CC], CCValues = 0xF, CCIfNoSignedWrap = 1 in {
   // Addition of a register.
   let isCommutable = 1 in {
     defm AR : BinaryRRAndK<"ar", 0x1A, 0xB9F8, z_sadd, GR32, GR32>;
     defm AGR : BinaryRREAndK<"agr", 0xB908, 0xB9E8, z_sadd, GR64, GR64>;
   }
   def AGFR : BinaryRRE<"agfr", 0xB918, null_frag, GR64, GR32>;
 
   // Addition to a high register.
let Defs = [CC], CCValues = 0xF, CCIfNoSignedWrap = 1 in {
 
   // Addition to memory.
   def ASI  : BinarySIY<"asi",  0xEB6A, add, imm32sx8>;
   def AGSI : BinarySIY<"agsi", 0xEB7A, add, imm64sx8>;
 }
 defm : SXB<z_sadd, GR64, AGFR>;
 
 // Addition producing a carry.
-let Defs = [CC] in {
+let Defs = [CC], CCValues = 0xF, IsLogical = 1 in {
   // Addition of a register.
   let isCommutable = 1 in {
     defm ALR : BinaryRRAndK<"alr", 0x1E, 0xB9FA, z_uadd, GR32, GR32>;
     defm ALGR : BinaryRREAndK<"algr", 0xB90A, 0xB9EA, z_uadd, GR64, GR64>;
   }
   def ALGFR : BinaryRRE<"algfr", 0xB91A, null_frag, GR64, GR32>;
 
   // Addition to a high register.
let Defs = [CC], CCValues = 0xF, IsLogical = 1 in {
 
   // Addition to memory.
   def ALSI  : BinarySIY<"alsi",  0xEB6E, null_frag, imm32sx8>;
   def ALGSI : BinarySIY<"algsi", 0xEB7E, null_frag, imm64sx8>;
 }
 defm : ZXB<z_uadd, GR64, ALGFR>;
 
 // Addition producing and using a carry.
-let Defs = [CC], Uses = [CC] in {
+let Defs = [CC], Uses = [CC], CCValues = 0xF, IsLogical = 1 in {
   // Addition of a register.
   def ALCR  : BinaryRRE<"alcr",  0xB998, z_addcarry, GR32, GR32>;
   def ALCGR : BinaryRRE<"alcgr", 0xB988, z_addcarry, GR64, GR64>;
 
   // Addition of memory.
   def ALC  : BinaryRXY<"alc",  0xE398, z_addcarry, GR32, load, 4>;
   def ALCG : BinaryRXY<"alcg", 0xE388, z_addcarry, GR64, load, 8>;
 }
 
 // Addition that does not modify the condition code.
 def ALSIHN : BinaryRIL<"alsihn", 0xCCB, null_frag, GRH32, simm32>,
              Requires<[FeatureHighWord]>;
 
 
 //===----------------------------------------------------------------------===//
 // Subtraction
 //===----------------------------------------------------------------------===//
 
 // Subtraction producing a signed overflow flag.
-let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8 in {
+let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8,
+    CCIfNoSignedWrap = 1 in {
   // Subtraction of a register.
   defm SR : BinaryRRAndK<"sr", 0x1B, 0xB9F9, z_ssub, GR32, GR32>;
   def SGFR : BinaryRRE<"sgfr", 0xB919, null_frag, GR64, GR32>;
   defm SGR : BinaryRREAndK<"sgr", 0xB909, 0xB9E9, z_ssub, GR64, GR64>;
 
   // Subtraction from a high register.
   def SHHHR : BinaryRRFa<"shhhr", 0xB9C9, null_frag, GRH32, GRH32, GRH32>,
               Requires<[FeatureHighWord]>;

 // constant into a register in any case, but the negated constant
 // requires fewer instructions to load.
 def : Pat<(z_saddo GR64:$src1, imm64lh16n:$src2),
           (SGR GR64:$src1, (LLILH imm64lh16n:$src2))>;
 def : Pat<(z_saddo GR64:$src1, imm64lf32n:$src2),
           (SGR GR64:$src1, (LLILF imm64lf32n:$src2))>;
 
 // Subtraction producing a carry.
-let Defs = [CC] in {
+let Defs = [CC], CCValues = 0x7, IsLogical = 1 in {
   // Subtraction of a register.
   defm SLR : BinaryRRAndK<"slr", 0x1F, 0xB9FB, z_usub, GR32, GR32>;
   def SLGFR : BinaryRRE<"slgfr", 0xB91B, null_frag, GR64, GR32>;
   defm SLGR : BinaryRREAndK<"slgr", 0xB90B, 0xB9EB, z_usub, GR64, GR64>;
 
   // Subtraction from a high register.
   def SLHHHR : BinaryRRFa<"slhhhr", 0xB9CB, null_frag, GRH32, GRH32, GRH32>,
                Requires<[FeatureHighWord]>;
def : Pat<(z_usub GR64:$src1, imm64sx16n:$src2),
         Requires<[FeatureDistinctOps]>;
 }
 
 // And vice versa in one special case (but we prefer addition).
 def : Pat<(add GR64:$src1, imm64zx32n:$src2),
           (SLGFI GR64:$src1, imm64zx32n:$src2)>;
 
 // Subtraction producing and using a carry.
-let Defs = [CC], Uses = [CC] in {
+let Defs = [CC], Uses = [CC], CCValues = 0xF, IsLogical = 1 in {
   // Subtraction of a register.
   def SLBR  : BinaryRRE<"slbr",  0xB999, z_subcarry, GR32, GR32>;
   def SLBGR : BinaryRRE<"slbgr", 0xB989, z_subcarry, GR64, GR64>;
 
   // Subtraction of memory.
   def SLB  : BinaryRXY<"slb",  0xE399, z_subcarry, GR32, load, 4>;
   def SLBG : BinaryRXY<"slbg", 0xE389, z_subcarry, GR64, load, 8>;
 }

llvm/test/CodeGen/SystemZ/int-cmp-44.ll

 ; Test that compares are omitted if CC already has the right value
 ; (z10 version).
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z10 -no-integrated-as \
 ; RUN:   -verify-machineinstrs| FileCheck %s
 
 declare void @foo()
 
-; Addition provides enough for equality comparisons with zero.  First teest
-; the EQ case.
+; Addition provides enough for comparisons with zero if we know no
+; signed overflow happens, which is when the "nsw" flag is set.
+; First test the EQ case.
 define i32 @f1(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f1:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: ber %r14
 ; CHECK: br %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
 ; ...and again with NE.
 define i32 @f2(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f2:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: bner %r14
+; CHECK-NEXT: blhr %r14
 ; CHECK: br %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp ne i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; SLT requires a comparison.
+; ...and again with SLT.
 define i32 @f3(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f3:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: cibl %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK-NEXT: blr %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp slt i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; ...SLE too.
+; ...and again with SLE.
 define i32 @f4(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f4:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: cible %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK-NEXT: bler %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp sle i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; ...SGT too.
+; ...and again with SGT.
 define i32 @f5(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f5:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: cibh %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK-NEXT: bhr %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp sgt i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; ...SGE too.
+; ...and again with SGE.
 define i32 @f6(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f6:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: cibhe %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK-NEXT: bher %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp sge i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; Subtraction also provides enough for equality comparisons with zero.
+; Subtraction provides in addition also enough for equality comparisons with
+; zero even without "nsw".
 define i32 @f7(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f7:
 ; CHECK: s %r2, 0(%r4)
 ; CHECK-NEXT: bner %r14
 ; CHECK: br %r14
 entry:
   %cur = load i32, i32 *%dest
   %res = sub i32 %a, %cur
   %cmp = icmp ne i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; ...but not for ordered comparisons.
+; ...and again with SLT.
 define i32 @f8(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f8:
 ; CHECK: s %r2, 0(%r4)
-; CHECK-NEXT: cibl %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK-NEXT: blr %r14
 entry:
   %cur = load i32, i32 *%dest
-  %res = sub i32 %a, %cur
+  %res = sub nsw i32 %a, %cur
   %cmp = icmp slt i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
exit:
   ret i32 %res
 }
 
 ; Check that stores do not interfere.
 define i32 @f23(i32 %a, i32 %b, i32 *%dest1, i32 *%dest2) {
 ; CHECK-LABEL: f23:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: st %r2, 0(%r4)
-; CHECK-NEXT: bner %r14
+; CHECK-NEXT: blhr %r14
 ; CHECK: br %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   store i32 %res, i32 *%dest1
   %cmp = icmp ne i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest2
   br label %exit
 

 
 ; Check that inline asms don't interfere if they don't clobber CC.
 define void @f25(i32 %a, i32 *%ptr) {
 ; CHECK-LABEL: f25:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: #APP
 ; CHECK-NEXT: blah
 ; CHECK-NEXT: #NO_APP
-; CHECK-NEXT: bner %r14
+; CHECK-NEXT: blhr %r14
 ; CHECK: br %r14
 entry:
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   call void asm sideeffect "blah", "r"(i32 %add)
   %cmp = icmp ne i32 %add, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %add, i32 *%ptr
   br label %exit
 

 define i32 @f27(i32 %a, i32 %b, i32 *%dest1, i32 *%dest2) {
 ; CHECK-LABEL: f27:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: sr %r3, %r2
 ; CHECK-NEXT: st %r3, 0(%r4)
 ; CHECK-NEXT: cibe %r2, 0, 0(%r14)
 ; CHECK: br %r14
 entry:
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %sub = sub i32 %b, %add
   store i32 %sub, i32 *%dest1
   %cmp = icmp eq i32 %add, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %sub, i32 *%dest2
   br label %exit

llvm/test/CodeGen/SystemZ/int-cmp-45.ll

 ; Test that compares are omitted if CC already has the right value
 ; (z196 version).
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z196 -no-integrated-as | FileCheck %s
 
-; Addition provides enough for equality comparisons with zero.  First teest
-; the EQ case with LOC.
+; Addition provides enough for comparisons with zero if we know no
+; signed overflow happens, which is when the "nsw" flag is set.
+; First test the EQ case with LOC.
 define i32 @f1(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f1:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: loce %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %c = load i32, i32 *%cptr
   %arg = select i1 %cmp, i32 %c, i32 %b
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with STOC.
 define i32 @f2(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f2:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: stoce %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %c = load i32, i32 *%cptr
   %newval = select i1 %cmp, i32 %b, i32 %c
   store i32 %newval, i32 *%cptr
   ret i32 %add
 }
 
 ; Reverse the select order and test with LOCR.
 define i32 @f3(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: f3:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: locrne %r3, %r4
+; CHECK-NEXT: locrlh %r3, %r4
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %arg = select i1 %cmp, i32 %b, i32 %c
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with LOC.
 define i32 @f4(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f4:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: locne %r3, 0(%r4)
+; CHECK-NEXT: loclh %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %c = load i32, i32 *%cptr
   %arg = select i1 %cmp, i32 %b, i32 %c
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with STOC.
 define i32 @f5(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f5:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: stocne %r3, 0(%r4)
+; CHECK-NEXT: stoclh %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %c = load i32, i32 *%cptr
   %newval = select i1 %cmp, i32 %c, i32 %b
   store i32 %newval, i32 *%cptr
   ret i32 %add
 }
 
 ; Change the EQ in f3 to NE.
 define i32 @f6(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: f6:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: locre %r3, %r4
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp ne i32 %add, 0
   %arg = select i1 %cmp, i32 %b, i32 %c
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with LOC.
 define i32 @f7(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f7:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: loce %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp ne i32 %add, 0
   %c = load i32, i32 *%cptr
   %arg = select i1 %cmp, i32 %b, i32 %c
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with STOC.
 define i32 @f8(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f8:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: stoce %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp ne i32 %add, 0
   %c = load i32, i32 *%cptr
   %newval = select i1 %cmp, i32 %c, i32 %b
   store i32 %newval, i32 *%cptr
   ret i32 %add
 }

llvm/test/CodeGen/SystemZ/int-cmp-56.ll

+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 | FileCheck %s
+;
+; Check that signed comparisons against 0 are only eliminated if the "nsw"
+; flag is present on the defining add (with register) instruction. For an
+; equality comparison, add logical can be used.
+
+define i32 @fun0(i32 %arg, i32 %arg2, i32 %arg3) {
+; CHECK-LABEL: fun0:
+; CHECK: jle .LBB0_2{{$}}
+; CHECK: je .LBB0_4{{$}}
+
+bb:
+  %tmp2 = add nsw i32 %arg, %arg2
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = add nsw i32 %arg, %arg3
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}
+
+; No "nsw" flag
+define i32 @fun1(i32 %arg, i32 %arg2, i32 %arg3) {
+; CHECK-LABEL: fun1:
+; CHECK: cijle
+; CHECK: alr
+; CHECK: jhe
+
+bb:
+  %tmp2 = add i32 %arg, %arg2
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = add i32 %arg, %arg3
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}
+
+; "nuw" flag
+define i32 @fun2(i32 %arg, i32 %arg2, i32 %arg3) {
+; CHECK-LABEL: fun2:
+; CHECK: cijle
+; CHECK: alr
+; CHECK: jhe
+
+bb:
+  %tmp2 = add nuw i32 %arg, %arg2
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = add nuw i32 %arg, %arg3
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}
+
+; Subtraction does not produce the value of zero in case of overflow, so
+; "nsw" is not needed for the equality check against zero.
+define i32 @fun3(i32 %arg, i32 %arg2, i32 %arg3) {
+; CHECK-LABEL: fun3:
+; CHECK: jle .LBB3_2{{$}}
+; CHECK: je .LBB3_4{{$}}
+
+bb:
+  %tmp2 = sub nsw i32 %arg, %arg2
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = sub nsw i32 %arg, %arg3
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}
+
+; No "nsw" flag
+define i32 @fun4(i32 %arg, i32 %arg2, i32 %arg3) {
+; CHECK-LABEL: fun4:
+; CHECK: cijle
+; CHECK: je .LBB4_4{{$}}
+
+bb:
+  %tmp2 = sub i32 %arg, %arg2
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = sub i32 %arg, %arg3
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}
+
+; "nuw" flag
+define i32 @fun5(i32 %arg, i32 %arg2, i32 %arg3) {
+; CHECK-LABEL: fun5:
+; CHECK: cijle
+; CHECK: je .LBB5_4{{$}}
+
+bb:
+  %tmp2 = sub nuw i32 %arg, %arg2
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = sub nuw i32 %arg, %arg3
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}

llvm/test/CodeGen/SystemZ/int-cmp-57.ll

+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 -disable-cgp | FileCheck %s
+;
+; Check that signed comparisons against 0 are eliminated if the defining
+; instruction is an add with immediate.
+;
+; Addition of an immediate does not depend on the "nsw" flag, since the
+; result can be predicted in case of overflow. For example, if adding a
+; positive immediate gives overflow, the result must be negative.
+
+; Addition of a negative immediate gives a positive result in case of
+; overflow (except for the case of the minimum value which may also result in
+; a zero result).
+define i32 @fun0(i32 %arg) {
+; CHECK-LABEL: fun0:
+; CHECK: ahik
+; CHECK-NEXT: locre
+bb:
+  %tmp = add i32 %arg, -1
+  %tmp1 = icmp eq i32 %tmp, 0
+  %res = select i1 %tmp1, i32 %tmp, i32 %arg
+  ret i32 %res
+}
+
+define i32 @fun1(i32 %arg) {
+; CHECK-LABEL: fun1:
+; CHECK: ahik
+; CHECK-NEXT: locrnle
+bb:
+  %tmp = add i32 %arg, -1
+  %tmp1 = icmp sgt i32 %tmp, 0
+  %res = select i1 %tmp1, i32 %tmp, i32 %arg
+  ret i32 %res
+}
+
+define i32 @fun2(i32 %arg) {
+; CHECK-LABEL: fun2:
+; CHECK: ahik
+; CHECK-NEXT: locrl
+bb:
+  %tmp = add i32 %arg, -1
+  %tmp1 = icmp slt i32 %tmp, 0
+  %res = select i1 %tmp1, i32 %tmp, i32 %arg
+  ret i32 %res
+}
+
+; Addition of a positive immediate gives a negative result in case of overflow.
+define i32 @fun3(i32 %arg) {
+; CHECK-LABEL: fun3:
+; CHECK: ahik
+; CHECK-NEXT: locre
+bb:
+  %tmp = add i32 %arg, 1
+  %tmp1 = icmp eq i32 %tmp, 0
+  %res = select i1 %tmp1, i32 %tmp, i32 %arg
+  ret i32 %res
+}
+
+define i32 @fun4(i32 %arg) {
+; CHECK-LABEL: fun4:
+; CHECK: ahik
+; CHECK-NEXT: locrh
+bb:
+  %tmp = add i32 %arg, 1
+  %tmp1 = icmp sgt i32 %tmp, 0
+  %res = select i1 %tmp1, i32 %tmp, i32 %arg
+  ret i32 %res
+}
+
+define i32 @fun5(i32 %arg) {
+; CHECK-LABEL: fun5:
+; CHECK: ahik
+; CHECK-NEXT: locrnhe
+bb:
+  %tmp = add i32 %arg, 1
+  %tmp1 = icmp slt i32 %tmp, 0
+  %res = select i1 %tmp1, i32 %tmp, i32 %arg
+  ret i32 %res
+}
+
+; Addition of the minimum value gives a positive or zero result.
+define i32 @fun6(i32 %arg) {
+; CHECK-LABEL: fun6:
+; CHECK: afi
+; CHECK-NEXT: chi
+; CHECK-NEXT: locrlh
+bb:
+  %tmp = add i32 %arg, -2147483648
+  %tmp1 = icmp eq i32 %tmp, 0
+  %res = select i1 %tmp1, i32 %tmp, i32 %arg
+  ret i32 %res
+}
+
+define i32 @fun7(i32 %arg) {
+; CHECK-LABEL: fun7:
+; CHECK: afi
+; CHECK-NEXT: chi
+; CHECK-NEXT: locrle
+bb:
+  %tmp = add i32 %arg, -2147483648
+  %tmp1 = icmp sgt i32 %tmp, 0
+  %res = select i1 %tmp1, i32 %tmp, i32 %arg
+  ret i32 %res
+}

llvm/test/CodeGen/SystemZ/int-cmp-58.mir

+# RUN: llc -mtriple=s390x-linux-gnu -mcpu=z14 -start-before=postrapseudos %s \
+# RUN:   -o - | FileCheck %s
+#
+# Test that the CC values of logical adds and subs can be used in compare
+# elimination in the cases of EQ/NE.
+
+# CHECK-LABEL:  fun0:
+# CHECK:  alr %r3, %r2
+# CHECK-NEXT: locrhe  %r2, %r3
+# CHECK-NEXT: alr %r3, %r2
+# CHECK-NEXT: locrnhe %r2, %r3
+# CHECK-NEXT: alr %r3, %r2
+# CHECK-NEXT: chi %r3, 0
+# CHECK-NEXT: locrle  %r2, %r3
+# CHECK-NEXT: alr %r3, %r2
+# CHECK-NEXT: chi %r3, 0
+# CHECK-NEXT: locrhe  %r2, %r3
+# CHECK-NEXT: slrk  %r3, %r2, %r3
+# CHECK-NEXT: locrh %r2, %r3
+# CHECK-NEXT: slrk  %r3, %r2, %r3
+# CHECK-NEXT: locrnhe %r2, %r3
+# CHECK-NEXT: slrk  %r3, %r2, %r3
+# CHECK-NEXT: chi %r3, 0
+# CHECK-NEXT: locrle  %r2, %r3
+# CHECK-NEXT: slrk  %r3, %r2, %r3
+# CHECK-NEXT: chi %r3, 0
+# CHECK-NEXT: locrhe  %r2, %r3
+
+
+--- |
+  define i32 @fun0(i32 %arg1, i32 %arg2) { bb: ret i32 0  }
+...
+---
+name:            fun0
+body:             |
+  bb.0:
+
+    renamable $r3l = ALRK renamable $r2l, killed renamable $r3l, implicit-def dead $cc
+    CHIMux renamable $r3l, 0, implicit-def $cc
+    renamable $r2l = LOCR killed renamable $r2l, killed renamable $r3l, 14, 8, implicit killed $cc
+
+    renamable $r3l = ALRK renamable $r2l, killed renamable $r3l, implicit-def dead $cc
+    CHIMux renamable $r3l, 0, implicit-def $cc
+    renamable $r2l = LOCR killed renamable $r2l, killed renamable $r3l, 14, 6, implicit killed $cc
+
+    renamable $r3l = ALRK renamable $r2l, killed renamable $r3l, implicit-def dead $cc
+    CHIMux renamable $r3l, 0, implicit-def $cc
+    renamable $r2l = LOCR killed renamable $r2l, killed renamable $r3l, 14, 12, implicit killed $cc
+
+    renamable $r3l = ALRK renamable $r2l, killed renamable $r3l, implicit-def dead $cc
+    CHIMux renamable $r3l, 0, implicit-def $cc
+    renamable $r2l = LOCR killed renamable $r2l, killed renamable $r3l, 14, 10, implicit killed $cc
+
+    renamable $r3l = SLRK renamable $r2l, killed renamable $r3l, implicit-def dead $cc
+    CHIMux renamable $r3l, 0, implicit-def $cc
+    renamable $r2l = LOCR killed renamable $r2l, killed renamable $r3l, 14, 8, implicit killed $cc
+
+    renamable $r3l = SLRK renamable $r2l, killed renamable $r3l, implicit-def dead $cc
+    CHIMux renamable $r3l, 0, implicit-def $cc
+    renamable $r2l = LOCR killed renamable $r2l, killed renamable $r3l, 14, 6, implicit killed $cc
+
+    renamable $r3l = SLRK renamable $r2l, killed renamable $r3l, implicit-def dead $cc
+    CHIMux renamable $r3l, 0, implicit-def $cc
+    renamable $r2l = LOCR killed renamable $r2l, killed renamable $r3l, 14, 12, implicit killed $cc
+
+    renamable $r3l = SLRK renamable $r2l, killed renamable $r3l, implicit-def dead $cc
+    CHIMux renamable $r3l, 0, implicit-def $cc
+    renamable $r2l = LOCR killed renamable $r2l, killed renamable $r3l, 14, 10, implicit killed $cc
+
+    Return implicit $r2l
+...