[SystemZ] Fix handling of CC values

[jonpa]

Summary

It was recently discovered that the handling of CC values was actually broken since overflow was not properly handled.

Add and sub instructions have now a new target specific instruction flag called SystemZII::CCIfNoSWrapOnly. This means that the CC result can be used instead of a compare with 0 only if the instruction has the 'nsw' flag set. Without 'nsw', the compare can not be eliminated.

Load complement/negative/positive can also overflow which means they should as well not be used for compare-with-0 elimination.

[jonpa]

Patch updated.

[uweigand]

lib/Target/SystemZ/SystemZInstrInfo.td

1020

So strictly speaking, subtraction is different than addition in that there can never be any overflow with a zero result for a subtraction. For for sub, the CompareZeroCCMask = 0x8 setting is actually correct.

Of course, the CCIfNoSWrapOnly = 1 would also be correct. Is there a way to support both (if we have nsw, support any comparison, if we don't, only support comparison against 0)?

test/CodeGen/SystemZ/int-cmp-44.ll

9

This comment is actually wrong now. I believe this comment should state that the CC value of an addition can be used to omit a compare if we know no signed overflow happens.

In fact, I think it would be good to have two sets of test cases: one (probably the current one) that verifies that in the absence of the nsw flag, we get the extra compares, and another one that verifies that in the presence of nsw, we never have a compare.

120

Same comment as above applies.

[jonpa]

Updated per the review so far.

lib/Target/SystemZ/SystemZInstrInfo.td

1020

As discussed this should work by just adding back the CompareZeroCCMask of 0x8 for the subs.

test/CodeGen/SystemZ/int-cmp-44.ll

9

Fixed comment.

As discussed, this is covered by the new int-cmp-56.ll test.

120

I added similar new tests for subtraction in int-cmp-45.ll.

[jonpa]

test/CodeGen/SystemZ/int-cmp-44.ll

120

Sorry - they were added in int-cmp-56.ll.

[uweigand]

With the current form of the patch, do we see any noticeable regression due to no additional compares after addition? If no, the patch would be OK; if yes, we'll probably have to add support for AL etc. first.

lib/Target/SystemZ/SystemZInstrFormats.td

93

I'd say something like "sets CC like a compare of the result against zero" instead of "according its CCValues".

test/CodeGen/SystemZ/loop-01.ll

97

This is because of this, right?
http://lists.llvm.org/pipermail/llvm-dev/2019-September/134876.html

That should preferably be fixed first. (Or at least marked with a FIXME here.)

[jonpa]

Patch updated:

adjustCCMasksForInstr() extended to

• recognize signed add with immediate and allow these cases by setting the overflow flag in CCMask as required. Now there are many fewer compares instead of many more on SPEC 2006.

• allow logical instructions to replace a compare with 0 for EQ/NE. This only handled just a few more cases. Next step is to find the best way to emit more logical instructions or change opcode to in cases where it will eliminate the comparison.

CCMASK_LOGICAL_NONZERO should be: CCMASK_1 | CCMASK_3, right?

[jonpa]

• Added assert when detecting an add with immediate, that this is not an immediate offset of a memory instruction but indeed an immediate operand.

• Removed check of SystemZII::getCompareZeroCCMask() when detecting add with immediate, since there are no sub with immediate with SystemZII::CCIfNoSignedWrap set.

• int-cmp-56.ll test updated to reflect the use of add logical.

• New method convertToLogical() in SystemZElimCompare that tries conversion to logical add instructions. This handles just some 130 more cases. Is opcode table ok in convertToLogical(), or should it be done next to the instruction definitions. I tried to do it in tablegen, but I couldn't find a neat way to do it - here is what I have so far:
  getLogicalOpcode.patch4 KBDownload
  .

I tried to do this conversion also during instruction selection. I made an experimental patch that would select logical adds for just those cases where the instruction does not have "nsw" while it is being compared if equal to zero. I found drawbacks with this: 1. MachineCSE is less successful now since it cannot remove an ALR even if the operands are the same as of the AR. This seemed to be ~400 cases on SPEC 2006. 2. This does not handle the cases of spilled registers that have been folded into an add with memory instruction. 3. The method I had to write for this became quite cumbersome (the might be a simpler way...), and it still was not "perfect" - there is no simple way to check if CC is clobbered between the add and the compare, by a call for instance. This is my patch where I tried this:

selectUAddForCmp0.patch5 KBDownload

In addition, I also tried to select add logical during instruction selection "always except in case of 'nsw'". This seems to give very similar results to convertToLogical(), except for all the add logical instructions everywhere. +50 tests fail with this.

• With this patch there are now ~3000 less compares (including fused ones) than on master. This is due to the improved handling of add with immediate.

• As seen in int-cmp-58.mir, the printed instruction names reflect CC as if it was set by a signed instruction: LOGICAL_ZERO becomes "locrhe", and LOGICAL_NONZERO "locrnhe", after a logical add. After a sub I see "locrh" / "locrnhe". Would it be feasible to support more reasonable mnemonics for these cases?

I am waiting for review OK for correctness before doing more extensive benchmarking.

lib/Target/SystemZ/SystemZInstrFormats.td

93

nice - changed it.

test/CodeGen/SystemZ/loop-01.ll

97

yes, that's what I saw and asked about on the list.

I marked it as FIXME for now. It should get fixed once we handle add with immediate cases in the backend, regardless of LSR/SelectionDAGBuilder.

[jonpa]

Remove the assert "A logical opcode was selected in presence of the nsw flag." This is too aggressive, as a zero extension can be folded into the instruction regardless of "nsw" (e.g. ALGFR).

[uweigand]

See inline comments. As an aside, the whole adjustCCMasksForInstr routine is now rather complicated -- this could probably use a complete rewrite, but I'm not really sure what the best approach here would be ...

lib/Target/SystemZ/SystemZElimCompare.cpp

333

The comment should probably elaborate a bit, along the lines of: "If adding a positive immediate overflows, the result must be less than zero. If adding a negative immediate overflow, the result must be larger than zero (except in the special case of adding the minimum value of the result range, in which case we cannot predict whether the result is larger than or equal to zero)."

Note that this logic is only correct if the operation performs an addition. Given that you only set the CCIfNoSignedWrap for add and sub instructions, and there is no sub with immediate operand, this condition happens to be true here, but that's a bit implicit ... Maybe add some assert?

335

This should probably use int64_t to avoid creating a dependency on the size of the host "int" type.

341

Probably better to move that assert now up to the only branch where it is useful.

344

This now looks incorrect in either the IsLogical case or in the OFImplies case. In either of those cases, we must go through the loop below rewriting CCUsers.

Again, probably best to move the setting of MIEquivalentToCmp to the branches above.

387

The assert seems unnecessary -- any CCMask other than CCMASK_CMP_EQ can be correctly mapped to CCMASK_LOGICAL_NONZERO here if the loop above passed the OutMask check.

[jonpa]

Updated per review.

In addition, I removed

CCValues &= ~SystemZ::CCMASK_ARITH_OVERFLOW;

from the case where both MachineInstr::NoSWrap and SystemZII::CCIfNoSignedWrap are present. This seems mostly confusing to me know, as CCValues is supposed to simply be what CC values the instruction may set, so if the instruction can set the OF bit, CCValues should already reflect this (which it also currently does).

I also added an assert for the CC operands of the CCUsers.

All the changes of this update are NFC on SPEC 2006.

As an aside, the whole adjustCCMasksForInstr routine is now rather complicated -- this could probably use a complete rewrite, but I'm not really sure what the best approach here would be ...

I agree that there are too many assumptions and complications - I might give it a try to find a more simplistic approach.

lib/Target/SystemZ/SystemZElimCompare.cpp

333

Comment updated.

I added an assert with a check against known opcodes for add with immediate, since I could not find any MI flag to check against.

341

ah, yes.

344

Yes... It seems that those cases were currently not passing this test anyway, but I agree that it certainly makes sense to move this up.

The first case with MachineInstr::NoSWrap also should not set this as the CCValues will now include the OF bit and this should be updated in the CCUsers CCValid operands.

387

ok - removed it.

[uweigand]

Thanks, this LGTM now.

I think we need to verify that benchmark results are good before committing this change, however.

[jonpa]

Here are some minor NFC changes in an attempt to make things easier to follow. Is this better?

CC_rewrite.patch32 KBDownload

[uweigand]

Yes, this does look a bit better. Maybe you can even avoid the IntCC variable by doing something like:

unsigned ReusableCCMask = CCValues;
if (CompareFlags & SystemZII::IsLogical)
  ReusableCCMask &= SystemZ::CCMASK_CMP_EQ;

before the various conditions, and then simply & new constraints into ReuableCCMask as required. For the CCIfNoSignedWrap case, this would be nothing, for the IsLogical case, this would be

ReusableCCMask &= SystemZ::CCMASK_CMP_EQ;

and for the CompareZeroCCMask case this would be

ReusableCCMask &= SystemZII::getCompareZeroCCMask(MIFlags);

[jonpa]

Maybe you can even avoid the IntCC variable...

Yes :-)

I like the AND:ing away things from ReusableCCMask after starting out with CCValues, however the logical case cannot play along here since those instructions has CCValues in the logical domain. Locigal subtracts have 0x7, which does not include the SystemZ::CCMASK_CMP_EQ bit. So this really needs to set ReusableCCMask to SystemZ::CCMASK_CMP_EQ based on the assumption that all logical instructions that has any CCValues support a comparison to zero.

I kept the empty first if clause in lack of any better ideas.

Maybe we could remove

if (ReusableCCMask == 0)

return false;

since the loop will return immediately anyway?

I strengthened the assert further for "CC operands of CCUser", to also check that CCMask is non-zero and that it is not equal to CCValid.

[uweigand]

In D66868#1710696, @jonpa wrote:

Maybe you can even avoid the IntCC variable...

Yes :-)

I like the AND:ing away things from ReusableCCMask after starting out with CCValues, however the logical case cannot play along here since those instructions has CCValues in the logical domain. Locigal subtracts have 0x7, which does not include the SystemZ::CCMASK_CMP_EQ bit. So this really needs to set ReusableCCMask to SystemZ::CCMASK_CMP_EQ based on the assumption that all logical instructions that has any CCValues support a comparison to zero.

OK, I see.

I kept the empty first if clause in lack of any better ideas.

Fine with me. Maybe just a bit more comment than "done" :-)

Maybe we could remove

if (ReusableCCMask == 0)

return false;

since the loop will return immediately anyway?

Not sure this is true. In any case, having this as early exist makes sense to me ...

I strengthened the assert further for "CC operands of CCUser", to also check that CCMask is non-zero and that it is not equal to CCValid.

I'm not sure this is completely safe either. Yes, in theory we should never see always-true or always-false conditions, but I believe it is not completely impossible that those might occur depending on optimization levels etc. In any case, the rest of the code should actually handle those cases correctly, so I'm not sure we really need to prevent them here.

Otherwise, the patch looks good to me now, assuming we get encouraging performance results.

[jonpa]

Updated per review: Comment improved, and extra assertions removed.

Not sure this is true. In any case, having this as early exist makes sense to me ...

The asserts passed with -O0/SPEC 2006, so I assumed this was guaranteed.

[jonpa]

Patch rebased, no manual changes needed...

[uweigand]

LGTM, thanks!

[jonpa]

Thanks for review!

3174683e

lib/Target/SystemZ/SystemZElimCompare.cpp

bool SystemZElimCompare::adjustCCMasksForInstr(
     MachineInstr &MI, MachineInstr &Compare,
     SmallVectorImpl<MachineInstr *> &CCUsers,
     unsigned ConvOpc) {
   int Opcode = (ConvOpc ? ConvOpc : MI.getOpcode());
   const MCInstrDesc &Desc = TII->get(Opcode);
   unsigned MIFlags = Desc.TSFlags;
 
   // See which compare-style condition codes are available.
-  unsigned ReusableCCMask = SystemZII::getCompareZeroCCMask(MIFlags);
+  unsigned CCValues = SystemZII::getCCValues(MIFlags);
+  unsigned ReusableCCMask;
+  if (MI.getFlag(MachineInstr::NoSWrap) &&
+      (MIFlags & SystemZII::CCIfNoSWrapOnly)) {
+    CCValues &= ~SystemZ::CCMASK_ARITH_OVERFLOW;
+    ReusableCCMask = CCValues;
+  } else
+    ReusableCCMask = SystemZII::getCompareZeroCCMask(MIFlags);
 
   // For unsigned comparisons with zero, only equality makes sense.
   unsigned CompareFlags = Compare.getDesc().TSFlags;
   if (CompareFlags & SystemZII::IsLogical)
     ReusableCCMask &= SystemZ::CCMASK_CMP_EQ;
 
   if (ReusableCCMask == 0)
     return false;
 
-  unsigned CCValues = SystemZII::getCCValues(MIFlags);
   assert((ReusableCCMask & ~CCValues) == 0 && "Invalid CCValues");
 
   bool MIEquivalentToCmp =
     (ReusableCCMask == CCValues &&
      CCValues == SystemZII::getCCValues(CompareFlags));
 
   if (!MIEquivalentToCmp) {
     // Now check whether these flags are enough for all users.
     SmallVector<MachineOperand *, 4> AlterMasks;
     for (unsigned int I = 0, E = CCUsers.size(); I != E; ++I) {
       MachineInstr *MI = CCUsers[I];
 
       // Fail if this isn't a use of CC that we understand.
for (unsigned I = 0, E = AlterMasks.size(); I != E; I += 2) {
       unsigned CCMask = AlterMasks[I + 1]->getImm();
       if (CCMask & ~ReusableCCMask)
         AlterMasks[I + 1]->setImm((CCMask & ReusableCCMask) |
                                   (CCValues & ~ReusableCCMask));
     }
   }
 
   // CC is now live after MI.
   if (!ConvOpc) {
     int CCDef = MI.findRegisterDefOperandIdx(SystemZ::CC, false, true, TRI);
     assert(CCDef >= 0 && "Couldn't find CC set");
     MI.getOperand(CCDef).setIsDead(false);
   }
 
   // Check if MI lies before Compare.
   bool BeforeCmp = false;
   MachineBasicBlock::iterator MBBI = MI, MBBE = MI.getParent()->end();

lib/Target/SystemZ/SystemZInstrFormats.td

class InstSystemZ<int size, dag outs, dag ins, string asmstr,
 
   // Similar, but true if the CC mask operand comes last (as for LOC, etc.).
   bit CCMaskLast = 0;
 
   // True if the instruction is the "logical" rather than "arithmetic" form,
   // in cases where a distinction exists.
   bit IsLogical = 0;
 
+  // True if the (add or sub) instruction sets CC according its CCValues, but
+  // only if the 'nsw' flag is set.
+  bit CCIfNoSWrapOnly = 0;
+
   let TSFlags{0}     = SimpleBDXLoad;
   let TSFlags{1}     = SimpleBDXStore;
   let TSFlags{2}     = Has20BitOffset;
   let TSFlags{3}     = HasIndex;
   let TSFlags{4}     = Is128Bit;
   let TSFlags{9-5}   = AccessBytes;
   let TSFlags{13-10} = CCValues;
   let TSFlags{17-14} = CompareZeroCCMask;
   let TSFlags{18}    = CCMaskFirst;
   let TSFlags{19}    = CCMaskLast;
   let TSFlags{20}    = IsLogical;
+  let TSFlags{21}    = CCIfNoSWrapOnly;
 }
 
 //===----------------------------------------------------------------------===//
 // Mappings between instructions
 //===----------------------------------------------------------------------===//
 
 // Return the version of an instruction that has an unsigned 12-bit
 // displacement.

lib/Target/SystemZ/SystemZInstrInfo.h

enum {
   AccessSizeMask         = (31 << 5),
   AccessSizeShift        = 5,
   CCValuesMask           = (15 << 10),
   CCValuesShift          = 10,
   CompareZeroCCMaskMask  = (15 << 14),
   CompareZeroCCMaskShift = 14,
   CCMaskFirst            = (1 << 18),
   CCMaskLast             = (1 << 19),
-  IsLogical              = (1 << 20)
+  IsLogical              = (1 << 20),
+  CCIfNoSWrapOnly        = (1 << 21)
 };
 
 static inline unsigned getAccessSize(unsigned int Flags) {
   return (Flags & AccessSizeMask) >> AccessSizeShift;
 }
 
 static inline unsigned getCCValues(unsigned int Flags) {
   return (Flags & CCValuesMask) >> CCValuesShift;

lib/Target/SystemZ/SystemZInstrInfo.cpp

 static void transferDeadCC(MachineInstr *OldMI, MachineInstr *NewMI) {
   if (OldMI->registerDefIsDead(SystemZ::CC)) {
     MachineOperand *CCDef = NewMI->findRegisterDefOperand(SystemZ::CC);
     if (CCDef != nullptr)
       CCDef->setIsDead(true);
   }
 }
 
+static void transferMIFlag(MachineInstr *OldMI, MachineInstr *NewMI,
+                           MachineInstr::MIFlag Flag) {
+  if (OldMI->getFlag(Flag))
+    NewMI->setFlag(Flag);
+}
+
 MachineInstr *SystemZInstrInfo::convertToThreeAddress(
     MachineFunction::iterator &MFI, MachineInstr &MI, LiveVariables *LV) const {
   MachineBasicBlock *MBB = MI.getParent();
 
   // Try to convert an AND into an RISBG-type instruction.
   // TODO: It might be beneficial to select RISBG and shorten to AND instead.
   if (LogicOp And = interpretAndImmediate(MI.getOpcode())) {
     uint64_t Imm = MI.getOperand(2).getImm() << And.ImmLSB;
if ((Opcode == SystemZ::AHI || Opcode == SystemZ::AGHI) && OpNum == 0 &&
     // A(G)HI %reg, CONST -> A(G)SI %mem, CONST
     Opcode = (Opcode == SystemZ::AHI ? SystemZ::ASI : SystemZ::AGSI);
     MachineInstr *BuiltMI =
         BuildMI(*InsertPt->getParent(), InsertPt, MI.getDebugLoc(), get(Opcode))
             .addFrameIndex(FrameIndex)
             .addImm(0)
             .addImm(MI.getOperand(2).getImm());
     transferDeadCC(&MI, BuiltMI);
+    transferMIFlag(&MI, BuiltMI, MachineInstr::NoSWrap);
     return BuiltMI;
   }
 
   if ((Opcode == SystemZ::ALFI && OpNum == 0 &&
        isInt<8>((int32_t)MI.getOperand(2).getImm())) ||
       (Opcode == SystemZ::ALGFI && OpNum == 0 &&
        isInt<8>((int64_t)MI.getOperand(2).getImm()))) {
     // AL(G)FI %reg, CONST -> AL(G)SI %mem, CONST
if ((OpNum == NumOps - 1) || NeedsCommute) {
         MIB.add(MI.getOperand(2));
       else
         for (unsigned I = 1; I < OpNum; ++I)
           MIB.add(MI.getOperand(I));
       MIB.addFrameIndex(FrameIndex).addImm(Offset);
       if (MemDesc.TSFlags & SystemZII::HasIndex)
         MIB.addReg(0);
       transferDeadCC(&MI, MIB);
+      transferMIFlag(&MI, MIB, MachineInstr::NoSWrap);
       return MIB;
     }
   }
 
   return nullptr;
 }
 
 MachineInstr *SystemZInstrInfo::foldMemoryOperandImpl(

lib/Target/SystemZ/SystemZInstrInfo.td

def : Pat<(or (zext32 GR32:$src), imm64hf32:$imm),
           (IIHF64 (INSERT_SUBREG (i64 (IMPLICIT_DEF)), GR32:$src, subreg_l32),
                   imm64hf32:$imm)>;
 
 //===----------------------------------------------------------------------===//
 // Addition
 //===----------------------------------------------------------------------===//
 
 // Addition producing a signed overflow flag.
-let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8 in {
+let Defs = [CC], CCValues = 0xF, CCIfNoSWrapOnly = 1 in {
   // Addition of a register.
   let isCommutable = 1 in {
     defm AR : BinaryRRAndK<"ar", 0x1A, 0xB9F8, z_sadd, GR32, GR32>;
     defm AGR : BinaryRREAndK<"agr", 0xB908, 0xB9E8, z_sadd, GR64, GR64>;
   }
   def AGFR : BinaryRRE<"agfr", 0xB918, null_frag, GR64, GR32>;
 
   // Addition to a high register.
def ALSIHN : BinaryRIL<"alsihn", 0xCCB, null_frag, GRH32, simm32>,
              Requires<[FeatureHighWord]>;
 
 
 //===----------------------------------------------------------------------===//
 // Subtraction
 //===----------------------------------------------------------------------===//
 
 // Subtraction producing a signed overflow flag.
-let Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0x8 in {
+let Defs = [CC], CCValues = 0xF, CCIfNoSWrapOnly = 1 in {
   // Subtraction of a register.
   defm SR : BinaryRRAndK<"sr", 0x1B, 0xB9F9, z_ssub, GR32, GR32>;
   def SGFR : BinaryRRE<"sgfr", 0xB919, null_frag, GR64, GR32>;
   defm SGR : BinaryRREAndK<"sgr", 0xB909, 0xB9E9, z_ssub, GR64, GR64>;
 
   // Subtraction from a high register.
   def SHHHR : BinaryRRFa<"shhhr", 0xB9C9, null_frag, GRH32, GRH32, GRH32>,
               Requires<[FeatureHighWord]>;

test/CodeGen/SystemZ/int-cmp-44.ll

 ; Test that compares are omitted if CC already has the right value
 ; (z10 version).
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z10 -no-integrated-as \
 ; RUN:   -verify-machineinstrs| FileCheck %s
 
 declare void @foo()
 
 ; Addition provides enough for equality comparisons with zero.  First teest
 ; the EQ case.
 define i32 @f1(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f1:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: ber %r14
 ; CHECK: br %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
 ; ...and again with NE.
 define i32 @f2(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f2:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: bner %r14
+; CHECK-NEXT: blhr %r14
 ; CHECK: br %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp ne i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; SLT requires a comparison.
+; ...and again with SLT.
 define i32 @f3(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f3:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: cibl %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK: blr %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp slt i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; ...SLE too.
+; ...and again with SLE.
 define i32 @f4(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f4:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: cible %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK: bler %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp sle i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; ...SGT too.
+; ...and again with SGT.
 define i32 @f5(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f5:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: cibh %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK: bhr %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp sgt i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; ...SGE too.
+; ...and again with SGE.
 define i32 @f6(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f6:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: cibhe %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK: bher %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   %cmp = icmp sge i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
 ; Subtraction also provides enough for equality comparisons with zero.
 define i32 @f7(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f7:
 ; CHECK: s %r2, 0(%r4)
-; CHECK-NEXT: bner %r14
+; CHECK-NEXT: blhr %r14
 ; CHECK: br %r14
 entry:
   %cur = load i32, i32 *%dest
-  %res = sub i32 %a, %cur
+  %res = sub nsw i32 %a, %cur
   %cmp = icmp ne i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
   ret i32 %res
 }
 
-; ...but not for ordered comparisons.
+; ...and again with SLT.
 define i32 @f8(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f8:
 ; CHECK: s %r2, 0(%r4)
-; CHECK-NEXT: cibl %r2, 0, 0(%r14)
-; CHECK: br %r14
+; CHECK: blr %r14
 entry:
   %cur = load i32, i32 *%dest
-  %res = sub i32 %a, %cur
+  %res = sub nsw i32 %a, %cur
   %cmp = icmp slt i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:
exit:
   ret i32 %res
 }
 
 ; Check that stores do not interfere.
 define i32 @f23(i32 %a, i32 %b, i32 *%dest1, i32 *%dest2) {
 ; CHECK-LABEL: f23:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: st %r2, 0(%r4)
-; CHECK-NEXT: bner %r14
+; CHECK-NEXT: blhr %r14
 ; CHECK: br %r14
 entry:
-  %res = add i32 %a, 1000000
+  %res = add nsw i32 %a, 1000000
   store i32 %res, i32 *%dest1
   %cmp = icmp ne i32 %res, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest2
   br label %exit
 

 
 ; Check that inline asms don't interfere if they don't clobber CC.
 define void @f25(i32 %a, i32 *%ptr) {
 ; CHECK-LABEL: f25:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: #APP
 ; CHECK-NEXT: blah
 ; CHECK-NEXT: #NO_APP
-; CHECK-NEXT: bner %r14
+; CHECK-NEXT: blhr %r14
 ; CHECK: br %r14
 entry:
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   call void asm sideeffect "blah", "r"(i32 %add)
   %cmp = icmp ne i32 %add, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %add, i32 *%ptr
   br label %exit
 

 define i32 @f27(i32 %a, i32 %b, i32 *%dest1, i32 *%dest2) {
 ; CHECK-LABEL: f27:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: sr %r3, %r2
 ; CHECK-NEXT: st %r3, 0(%r4)
 ; CHECK-NEXT: cibe %r2, 0, 0(%r14)
 ; CHECK: br %r14
 entry:
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %sub = sub i32 %b, %add
   store i32 %sub, i32 *%dest1
   %cmp = icmp eq i32 %add, 0
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %sub, i32 *%dest2
   br label %exit

 exit:
   ret i64 %ext
 }
 
 ; Try a form of f7 in which the subtraction operands are compared directly.
 define i32 @f41(i32 %a, i32 %b, i32 *%dest) {
 ; CHECK-LABEL: f41:
 ; CHECK: s %r2, 0(%r4)
-; CHECK-NEXT: bner %r14
+; CHECK-NEXT: blhr %r14
 ; CHECK: br %r14
 entry:
   %cur = load i32, i32 *%dest
-  %res = sub i32 %a, %cur
+  %res = sub nsw i32 %a, %cur
   %cmp = icmp ne i32 %a, %cur
   br i1 %cmp, label %exit, label %store
 
 store:
   store i32 %b, i32 *%dest
   br label %exit
 
 exit:

test/CodeGen/SystemZ/int-cmp-45.ll

 ; Test that compares are omitted if CC already has the right value
 ; (z196 version).
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z196 -no-integrated-as | FileCheck %s
 
-; Addition provides enough for equality comparisons with zero.  First teest
+; Addition provides enough for equality comparisons with zero.  First test
 ; the EQ case with LOC.
 define i32 @f1(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f1:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: loce %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %c = load i32, i32 *%cptr
   %arg = select i1 %cmp, i32 %c, i32 %b
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with STOC.
 define i32 @f2(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f2:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: stoce %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %c = load i32, i32 *%cptr
   %newval = select i1 %cmp, i32 %b, i32 %c
   store i32 %newval, i32 *%cptr
   ret i32 %add
 }
 
 ; Reverse the select order and test with LOCR.
 define i32 @f3(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: f3:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: locrne %r3, %r4
+; CHECK-NEXT: locrlh %r3, %r4
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %arg = select i1 %cmp, i32 %b, i32 %c
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with LOC.
 define i32 @f4(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f4:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: locne %r3, 0(%r4)
+; CHECK-NEXT: loclh %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %c = load i32, i32 *%cptr
   %arg = select i1 %cmp, i32 %b, i32 %c
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with STOC.
 define i32 @f5(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f5:
 ; CHECK: afi %r2, 1000000
-; CHECK-NEXT: stocne %r3, 0(%r4)
+; CHECK-NEXT: stoclh %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp eq i32 %add, 0
   %c = load i32, i32 *%cptr
   %newval = select i1 %cmp, i32 %c, i32 %b
   store i32 %newval, i32 *%cptr
   ret i32 %add
 }
 
 ; Change the EQ in f3 to NE.
 define i32 @f6(i32 %a, i32 %b, i32 %c) {
 ; CHECK-LABEL: f6:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: locre %r3, %r4
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp ne i32 %add, 0
   %arg = select i1 %cmp, i32 %b, i32 %c
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with LOC.
 define i32 @f7(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f7:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: loce %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp ne i32 %add, 0
   %c = load i32, i32 *%cptr
   %arg = select i1 %cmp, i32 %b, i32 %c
   call void asm sideeffect "blah $0", "{r3}"(i32 %arg)
   ret i32 %add
 }
 
 ; ...and again with STOC.
 define i32 @f8(i32 %a, i32 %b, i32 *%cptr) {
 ; CHECK-LABEL: f8:
 ; CHECK: afi %r2, 1000000
 ; CHECK-NEXT: stoce %r3, 0(%r4)
 ; CHECK: br %r14
-  %add = add i32 %a, 1000000
+  %add = add nsw i32 %a, 1000000
   %cmp = icmp ne i32 %add, 0
   %c = load i32, i32 *%cptr
   %newval = select i1 %cmp, i32 %c, i32 %b
   store i32 %newval, i32 *%cptr
   ret i32 %add
 }

test/CodeGen/SystemZ/int-cmp-56.ll

+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z14 | FileCheck %s
+;
+; Check that signed comparisons against 0 are only eliminated if
+; the "nsw" flag is present on the defining instruction.
+
+define i32 @fun0(i32 %arg, i32 %arg2) {
+; CHECK-LABEL: fun0:
+; CHECK: jle .LBB0_2{{$}}
+; CHECK: je .LBB0_4{{$}}
+
+bb:
+  %tmp2 = add nsw i32 %arg, -1
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = add nsw i32 %arg, %arg2
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}
+
+; No "nsw" flag
+define i32 @fun1(i32 %arg, i32 %arg2) {
+; CHECK-LABEL: fun1:
+; CHECK: cijle
+; CHECK: cije
+
+bb:
+  %tmp2 = add i32 %arg, -1
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = add i32 %arg, %arg2
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}
+
+; "nuw" flag
+define i32 @fun2(i32 %arg, i32 %arg2) {
+; CHECK-LABEL: fun2:
+; CHECK: cijle
+; CHECK: cije
+
+bb:
+  %tmp2 = add nuw i32 %arg, -1
+  %tmp3 = icmp sgt i32 %tmp2, 0
+  br i1 %tmp3, label %bb3, label %bb1
+
+bb1:
+  %tmp4 = add nuw i32 %arg, %arg2
+  %tmp5 = icmp eq i32 %tmp4, 0
+  br i1 %tmp5, label %bb4, label %bb2
+
+bb2:
+  ret i32 0
+
+bb3:
+  ret i32 1
+
+bb4:
+  ret i32 2
+}

test/CodeGen/SystemZ/loop-01.ll

 ; Test a loop with a 64-bit decremented counter in which the 32-bit
 ; low part of the counter is used after the decrement.  This is an example
 ; of a subregister use being the only thing that blocks a conversion to BRCTG.
 define void @f4(i32 *%src, i32 *%dest, i64 *%dest2, i64 %count) {
 ; CHECK-LABEL: f4:
 ; CHECK: aghi [[REG:%r[0-5]]], -1
 ; CHECK: lr [[REG2:%r[0-5]]], [[REG]]
 ; CHECK: stg [[REG2]],
-; CHECK: jne {{\..*}}
+; CHECK: cgijlh [[REG]], 0, {{\..*}}
 ; CHECK: br %r14
 entry:
   br label %loop
 
 loop:
   %left = phi i64 [ %count, %entry ], [ %next, %loop.next ]
   store volatile i64 %left, i64 *%dest2
   %val = load volatile i32, i32 *%src