[GlobalISel] Fold xor(cmp(pred, _, _), 1) -> cmp(inverse(pred), _, _)

[aemerson]

Summary

This is needed for an upcoming change to how we translate conditional branches which might generate these.

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2100–2106

Can generalize for not-i1 by checking the target boolean contents

[aemerson]

Use getBooleanContents and update AMDGPU tests.

[aemerson]

Check for multiple uses of the cmp.

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2106

Swap the order of these checks?

llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir

137

Test with vectors?

[foad]

One of the globalisel code quality regressions vs selectiondag that I'm tracking comes from code like this:

%40:_(s1) = G_ICMP intpred(uge), %35:_(s32), %37:_
%41:_(s1) = G_ICMP intpred(uge), %36:_(s32), %38:_
%42:_(s1) = G_OR %40:_, %41:_
%43:_(s1) = G_CONSTANT i1 true
%44:sreg_64_xexec(s1) = G_XOR %42:_, %43:_
%46:sreg_64_xexec(s64) = SI_IF %44:sreg_64_xexec(s1), %bb.3, implicit-def $exec, implicit-def $scc, implicit $exec

So... is there any chance of extending this patch to be able to invert a whole tree of comparisons combined with ANDs and ORs?

[foad]

In D86383#2241544, @foad wrote:

So... is there any chance of extending this patch to be able to invert a whole tree of comparisons combined with ANDs and ORs?

I've had a go at this in D86709.

[aemerson]

Add support for vectors.

[arsenm]

llvm/lib/CodeGen/GlobalISel/Utils.cpp

708

One of these parameters is isVector, which is getting lost here

[arsenm]

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2153–2165

I think this isConstTrue logic should be merged into a separate function (at least the getScalarSizeInBits() == 1 case and check TLI for the extended version should be together)

2172

I guess technically TargetBooleanContents also depends on isFloat in the fcmp case

[aemerson]

Simplify and refactor code with FP case.

[aemerson]

Ping

[paquette]

Personally I can't see anything wrong with this at this point.

@arsenm?

[rupprecht]

FYI, I reverted this (and D86413 which depends on it) in 8693ddc74371dedc742c9f3d3e4eda1da72c13ea to keep the build green, as it was causing surprising crashes when running ninja check-llvm-codegen-aarch64-globalisel in release builds.

In case it goes away, the buildbot error is:

FAIL: LLVM :: CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir (33345 of 67618)
******************** TEST 'LLVM :: CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir' FAILED ********************
Script:
--
: 'RUN: at line 2';   /home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc -mtriple aarch64-apple-ios  -run-pass=aarch64-prelegalizer-combiner --aarch64prelegalizercombinerhelper-only-enable-rule="not_cmp_fold" /home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/llvm-project/llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir -o - -verify-machineinstrs | /home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/FileCheck /home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/llvm-project/llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir
--
Exit Code: 2

Command Output (stderr):
--
LLVM ERROR: Invalid rule identifier
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc -mtriple aarch64-apple-ios -run-pass=aarch64-prelegalizer-combiner --aarch64prelegalizercombinerhelper-only-enable-rule=not_cmp_fold /home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/llvm-project/llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir -o - -verify-machineinstrs 
1.	Running pass 'Function Pass Manager' on module '/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/llvm-project/llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir'.
2.	Running pass 'AArch64PreLegalizerCombiner' on function '@icmp'
 #0 0x00005651667a183c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x25cc83c)
 #1 0x000056516679f694 llvm::sys::RunSignalHandlers() (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x25ca694)
 #2 0x000056516679f7e3 SignalHandler(int) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x25ca7e3)
 #3 0x00007f71603c58a0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x128a0)
 #4 0x00007f715f293f47 raise (/lib/x86_64-linux-gnu/libc.so.6+0x3ef47)
 #5 0x00007f715f2958b1 abort (/lib/x86_64-linux-gnu/libc.so.6+0x408b1)
 #6 0x000056516671a1d6 llvm::report_fatal_error(llvm::Twine const&, bool) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x25451d6)
 #7 0x000056516671a2f8 (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x25452f8)
 #8 0x0000565164a62751 (anonymous namespace)::AArch64PreLegalizerCombiner::runOnMachineFunction(llvm::MachineFunction&) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x88d751)
 #9 0x0000565165d08b78 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x1b33b78)
#10 0x00005651660d9af7 llvm::FPPassManager::runOnFunction(llvm::Function&) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x1f04af7)
#11 0x00005651660da211 llvm::FPPassManager::runOnModule(llvm::Module&) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x1f05211)
#12 0x00005651660d8e6f llvm::legacy::PassManagerImpl::run(llvm::Module&) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x1f03e6f)
#13 0x000056516491a6e7 compileModule(char**, llvm::LLVMContext&) (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x7456e7)
#14 0x0000565164871cf7 main (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x69ccf7)
#15 0x00007f715f276b97 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b97)
#16 0x000056516491380a _start (/home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/llc+0x73e80a)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/build/stage1/bin/FileCheck /home/buildslave/ps4-buildslave1/clang-with-thin-lto-ubuntu/llvm-project/llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir

[aemerson]

Thanks, it seems the test doesn't work without asserts enabled. I've re-committed it fixed in 520ab710fb6f9829b4e70fda1dcc91ed4f614d0a

[rupprecht]

Thanks!

Usually, tests that are marked # REQUIRES: asserts are done so because they make use of some code that is only available behind NDEBUG. However, in this case, it's outright crashing on certain inputs when not in debug mode. At a first glance, that doesn't seem like a correct fix to me -- after all, the crashing bug is still there -- but I'm not familiar with this code, so I'm probably missing something?

[aemerson]

In D86383#2250882, @rupprecht wrote:

Thanks!

Usually, tests that are marked # REQUIRES: asserts are done so because they make use of some code that is only available behind NDEBUG. However, in this case, it's outright crashing on certain inputs when not in debug mode. At a first glance, that doesn't seem like a correct fix to me -- after all, the crashing bug is still there -- but I'm not familiar with this code, so I'm probably missing something?

There's no bug in the actual patch, the crash is because the testing flag --aarch64prelegalizercombinerhelper-only-enable-rule doesn't work without assertions enabled. It's not an elegant failure mode, but nothing to worry I think.

[rupprecht]

Ok. Thanks for confirming!

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

public:
   /// \return true if \p MI is a G_AND instruction whose RHS is a mask where
   /// LHS & mask == LHS. (E.g., an all-ones value.)
   ///
   /// \param [in] MI - The G_AND instruction.
   /// \param [out] Replacement - A register the G_AND should be replaced with on
   /// success.
   bool matchAndWithTrivialMask(MachineInstr &MI, Register &Replacement);
 
+  /// Combine inverting a result of a compare into the opposite cond code.
+  bool matchNotCmp(MachineInstr &MI, Register &CmpReg);
+  bool applyNotCmp(MachineInstr &MI, Register &CmpReg);
+
   /// Try to transform \p MI by using all of the above
   /// combine functions. Returns true if changed.
   bool tryCombine(MachineInstr &MI);
 
 private:
   // Memcpy family optimization helpers.
   bool optimizeMemcpy(MachineInstr &MI, Register Dst, Register Src,
                       unsigned KnownLen, Align DstAlign, Align SrcAlign,

llvm/include/llvm/CodeGen/GlobalISel/MIPatternMatch.h

 
 template <typename LHS, typename RHS>
 inline BinaryOp_match<LHS, RHS, TargetOpcode::G_AND, true>
 m_GAnd(const LHS &L, const RHS &R) {
   return BinaryOp_match<LHS, RHS, TargetOpcode::G_AND, true>(L, R);
 }
 
 template <typename LHS, typename RHS>
+inline BinaryOp_match<LHS, RHS, TargetOpcode::G_XOR, true>
+m_GXor(const LHS &L, const RHS &R) {
+  return BinaryOp_match<LHS, RHS, TargetOpcode::G_XOR, true>(L, R);
+}
+
+template <typename LHS, typename RHS>
 inline BinaryOp_match<LHS, RHS, TargetOpcode::G_OR, true> m_GOr(const LHS &L,
                                                                 const RHS &R) {
   return BinaryOp_match<LHS, RHS, TargetOpcode::G_OR, true>(L, R);
 }
 
 template <typename LHS, typename RHS>
 inline BinaryOp_match<LHS, RHS, TargetOpcode::G_SHL, false>
 m_GShl(const LHS &L, const RHS &R) {

llvm/include/llvm/CodeGen/GlobalISel/Utils.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_GLOBALISEL_UTILS_H
 #define LLVM_CODEGEN_GLOBALISEL_UTILS_H
 
 #include "llvm/ADT/StringRef.h"
 #include "llvm/CodeGen/Register.h"
+#include "llvm/CodeGen/TargetLowering.h"
 #include "llvm/Support/Alignment.h"
 #include "llvm/Support/LowLevelTypeImpl.h"
 #include "llvm/Support/MachineValueType.h"
 
 namespace llvm {
 
 class AnalysisUsage;
 class MachineFunction;

 /// In the worst case, this returns LLT::scalar(1)
 LLVM_READNONE
 LLT getGCDType(LLT OrigTy, LLT TargetTy);
 
 /// \returns The splat index of a G_SHUFFLE_VECTOR \p MI when \p MI is a splat.
 /// If \p MI is not a splat, returns None.
 Optional<int> getSplatIndex(MachineInstr &MI);
 
+/// Returns true if the specified instruction is a G_BUILD_VECTOR or
+/// G_BUILD_VECTOR_TRUNC where all the elements are the given value
+/// \p SplatValue.
+bool isBuildVectorConstantSplat(const MachineInstr &MI,
+                                const MachineRegisterInfo &MRI,
+                                int64_t SplatValue);
+
 /// Return true if the specified instruction is a G_BUILD_VECTOR or
 /// G_BUILD_VECTOR_TRUNC where all of the elements are 0 or undef.
 bool isBuildVectorAllZeros(const MachineInstr &MI,
                            const MachineRegisterInfo &MRI);
 
 /// Return true if the specified instruction is a G_BUILD_VECTOR or
 /// G_BUILD_VECTOR_TRUNC where all of the elements are ~0 or undef.
 bool isBuildVectorAllOnes(const MachineInstr &MI,
                           const MachineRegisterInfo &MRI);
 
+/// Returns true if given the TargetLowering's boolean contents information,
+/// the value \p Val contains a true value.
+bool isConstTrueVal(const TargetLowering &TLI, int64_t Val);
+
+/// Returns an integer representing true, as defined by the
+/// TargetBooleanContents.
+int64_t getICmpTrueVal(const TargetLowering &TLI);
 } // End namespace llvm.
 #endif

llvm/include/llvm/Target/GlobalISel/Combine.td

 def and_trivial_mask_matchinfo : GIDefMatchData<"Register">;
 def and_trivial_mask: GICombineRule <
   (defs root:$root, and_trivial_mask_matchinfo:$matchinfo),
   (match (wip_match_opcode G_AND):$root,
          [{ return Helper.matchAndWithTrivialMask(*${root}, ${matchinfo}); }]),
   (apply [{ return Helper.replaceSingleDefInstWithReg(*${root}, ${matchinfo}); }])
 >;
 
+def not_cmp_fold_matchinfo : GIDefMatchData<"Register">;
+def not_cmp_fold : GICombineRule<
+  (defs root:$d, not_cmp_fold_matchinfo:$info),
+  (match (wip_match_opcode G_XOR): $d,
+  [{ return Helper.matchNotCmp(*${d}, ${info}); }]),
+  (apply [{ return Helper.applyNotCmp(*${d}, ${info}); }])
+>;
+
 // FIXME: These should use the custom predicate feature once it lands.
 def undef_combines : GICombineGroup<[undef_to_fp_zero, undef_to_int_zero,
                                      undef_to_negative_one,
                                      propagate_undef_any_op,
                                      propagate_undef_all_ops,
                                      propagate_undef_shuffle_mask,
                                      erase_undef_store]>;
 
 def identity_combines : GICombineGroup<[select_same_val, right_identity_zero,
                                         binop_same_val, binop_left_to_zero,
                                         binop_right_to_zero, p2i_to_i2p,
                                         i2p_to_p2i]>;
 
 def width_reduction_combines : GICombineGroup<[reduce_shl_of_extend]>;
 
 def trivial_combines : GICombineGroup<[copy_prop, mul_to_shl, add_p2i_to_ptradd]>;
 def all_combines : GICombineGroup<[trivial_combines, ptr_add_immed_chain,
     combines_for_extload, combine_indexed_load_store, undef_combines,
     identity_combines, simplify_add_to_sub,
     hoist_logic_op_with_same_opcode_hands,
     shl_ashr_to_sext_inreg, sext_inreg_of_load,
-    width_reduction_combines]>;
+    width_reduction_combines, not_cmp_fold]>;

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

 }
 
 bool CombinerHelper::matchAndWithTrivialMask(MachineInstr &MI,
                                              Register &Replacement) {
   // Given
   //
   // %mask:_(sN) = G_CONSTANT iN 000...0111...1
   // %x:_(sN) = G_SOMETHING
   // %y:_(sN) = G_AND %x, %mask
   //
   // Eliminate the G_AND when it is known that x & mask == x.
   //
   // Patterns like this can appear as a result of legalization. E.g.
   //
   // %cmp:_(s32) = G_ICMP intpred(pred), %x(s32), %y
   // %one:_(s32) = G_CONSTANT i32 1
   // %and:_(s32) = G_AND %cmp, %one
   //
   // In this case, G_ICMP only produces a single bit, so x & 1 == x.
   assert(MI.getOpcode() == TargetOpcode::G_AND);
   if (!KB)
     return false;
 
bool CombinerHelper::matchAndWithTrivialMask(MachineInstr &MI,
   APInt Mask(DstTy.getSizeInBits(), Cst);
   if (!Mask.isMask())
     return false;
 
   // Now, let's check that x & Mask == x. If this is true, then x & ~Mask == 0.
   return KB->maskedValueIsZero(Replacement, ~Mask);
 }
 
+bool CombinerHelper::matchNotCmp(MachineInstr &MI, Register &CmpReg) {
+  assert(MI.getOpcode() == TargetOpcode::G_XOR);
+  LLT Ty = MRI.getType(MI.getOperand(0).getReg());
+  const auto &TLI = *Builder.getMF().getSubtarget().getTargetLowering();
+  Register XorSrc;
+  Register CstReg;
+  int64_t Cst;
+  // We match xor(src, true) here.
+  if (!mi_match(MI.getOperand(0).getReg(), MRI,
+                m_GXor(m_Reg(XorSrc), m_Reg(CstReg))))
+    return false;
+
+  if (!Ty.isVector()) {
+    if (!mi_match(CstReg, MRI, m_ICst(Cst)))
+      return false;
+    // For i1, Cst will always be -1 regardless of boolean contents.
+    if (!(Ty.getSizeInBits() == 1 && Cst == -1) && !isConstTrueVal(TLI, Cst))
+      return false;
+  } else {
+    // For vectors, look for a build vector splat of true.
+    int64_t TrueCst = getICmpTrueVal(TLI);
+    // Same edge case for i1 as above.
+    if (Ty.getScalarSizeInBits() == 1)
+      TrueCst = -1;
+    MachineInstr *CstDef = MRI.getVRegDef(CstReg);
+    if (!isBuildVectorConstantSplat(*CstDef, MRI, TrueCst))
+      return false;
+  }
+
+  // Now try match src to either icmp or fcmp.
+  if (!mi_match(XorSrc, MRI, m_GICmp(m_Pred(), m_Reg(), m_Reg()))) {
+    // Try fcmp.
+    if (!mi_match(XorSrc, MRI, m_GFCmp(m_Pred(), m_Reg(), m_Reg())))
+      return false;
+  }
+  if (!MRI.hasOneNonDBGUse(XorSrc))
+    return false;
+  CmpReg = XorSrc;
+  return true;
+}
+
+bool CombinerHelper::applyNotCmp(MachineInstr &MI, Register &CmpReg) {
+  MachineInstr *CmpDef = MRI.getVRegDef(CmpReg);
+  assert(CmpDef && "Should have been given an MI reg");
+  assert(CmpDef->getOpcode() == TargetOpcode::G_ICMP ||
+         CmpDef->getOpcode() == TargetOpcode::G_FCMP);
+
+  Observer.changingInstr(*CmpDef);
+  MachineOperand &PredOp = CmpDef->getOperand(1);
+  CmpInst::Predicate NewP = CmpInst::getInversePredicate(
+      (CmpInst::Predicate)PredOp.getPredicate());
+  PredOp.setPredicate(NewP);
+  Observer.changedInstr(*CmpDef);
+
+  replaceRegWith(MRI, MI.getOperand(0).getReg(),
+                 CmpDef->getOperand(0).getReg());
+  MI.eraseFromParent();
+  return true;
+}
+
 bool CombinerHelper::tryCombine(MachineInstr &MI) {
   if (tryCombineCopy(MI))
     return true;
   if (tryCombineExtendingLoads(MI))
     return true;
   if (tryCombineIndexedLoadStore(MI))
     return true;
   return false;
 }

llvm/lib/CodeGen/GlobalISel/Utils.cpp

 }
 
 static bool isBuildVectorOp(unsigned Opcode) {
   return Opcode == TargetOpcode::G_BUILD_VECTOR ||
          Opcode == TargetOpcode::G_BUILD_VECTOR_TRUNC;
 }
 
 // TODO: Handle mixed undef elements.
-static bool isBuildVectorConstantSplat(const MachineInstr &MI,
-                                       const MachineRegisterInfo &MRI,
-                                       int64_t SplatValue) {
+bool llvm::isBuildVectorConstantSplat(const MachineInstr &MI,
   if (!isBuildVectorOp(MI.getOpcode()))
     return false;
 
   const unsigned NumOps = MI.getNumOperands();
   for (unsigned I = 1; I != NumOps; ++I) {
     Register Element = MI.getOperand(I).getReg();
     int64_t ElementValue;
     if (!mi_match(Element, MRI, m_ICst(ElementValue)) ||
         ElementValue != SplatValue)
       return false;
   }
 
   return true;
 }
 
 bool llvm::isBuildVectorAllZeros(const MachineInstr &MI,
                                  const MachineRegisterInfo &MRI) {
   return isBuildVectorConstantSplat(MI, MRI, 0);
 }
 
 bool llvm::isBuildVectorAllOnes(const MachineInstr &MI,
                                 const MachineRegisterInfo &MRI) {
   return isBuildVectorConstantSplat(MI, MRI, -1);
 }
+
+bool llvm::isConstTrueVal(const TargetLowering &TLI, int64_t Val) {
+  switch (TLI.getBooleanContents(false, false)) {
+  case TargetLowering::UndefinedBooleanContent:
+    return Val & 0x1;
+  case TargetLowering::ZeroOrOneBooleanContent:
+    return Val == 1;
+  case TargetLowering::ZeroOrNegativeOneBooleanContent:
+    return Val == -1;
+  }
+  llvm_unreachable("Invalid boolean contents");
+}
+int64_t llvm::getICmpTrueVal(const TargetLowering &TLI) {
+  switch (TLI.getBooleanContents(false, false)) {
+  case TargetLowering::UndefinedBooleanContent:
+  case TargetLowering::ZeroOrOneBooleanContent:
+    return 1;
+  case TargetLowering::ZeroOrNegativeOneBooleanContent:
+    return -1;
+  }
+  llvm_unreachable("Invalid boolean contents");
+}

llvm/test/CodeGen/AArch64/GlobalISel/prelegalizercombiner-invert-cmp.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple aarch64-apple-ios  -run-pass=aarch64-prelegalizer-combiner --aarch64prelegalizercombinerhelper-only-enable-rule="not_cmp_fold" %s -o - -verify-machineinstrs | FileCheck %s
+
+# Check that we fold an compare result inverted into just inverting the condition code.
+---
+name: icmp
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0
+
+    ; CHECK-LABEL: name: icmp
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 1
+    ; CHECK: [[ICMP:%[0-9]+]]:_(s1) = G_ICMP intpred(sle), [[COPY]](s64), [[C]]
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[ICMP]](s1)
+    ; CHECK: $w0 = COPY [[ANYEXT]](s32)
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_CONSTANT i64 1
+    %2:_(s1) = G_CONSTANT i1 1
+    %3:_(s1) = G_ICMP intpred(sgt), %0(s64), %1
+    %4:_(s1) = G_XOR %3, %2
+    %5:_(s32) = G_ANYEXT %4
+    $w0 = COPY %5(s32)
+    RET_ReallyLR implicit $w0
+...
+---
+name: fcmp
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0
+
+    ; CHECK-LABEL: name: fcmp
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 1
+    ; CHECK: [[FCMP:%[0-9]+]]:_(s1) = G_FCMP floatpred(ule), [[COPY]](s64), [[C]]
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[FCMP]](s1)
+    ; CHECK: $w0 = COPY [[ANYEXT]](s32)
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_CONSTANT i64 1
+    %2:_(s1) = G_CONSTANT i1 1
+    %3:_(s1) = G_FCMP floatpred(ogt), %0(s64), %1
+    %4:_(s1) = G_XOR %3, %2
+    %5:_(s32) = G_ANYEXT %4
+    $w0 = COPY %5(s32)
+    RET_ReallyLR implicit $w0
+...
+---
+name: icmp_not_xor_with_1
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0
+
+    ; CHECK-LABEL: name: icmp_not_xor_with_1
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 1
+    ; CHECK: [[C1:%[0-9]+]]:_(s1) = G_CONSTANT i1 false
+    ; CHECK: [[ICMP:%[0-9]+]]:_(s1) = G_ICMP intpred(sgt), [[COPY]](s64), [[C]]
+    ; CHECK: [[XOR:%[0-9]+]]:_(s1) = G_XOR [[ICMP]], [[C1]]
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[XOR]](s1)
+    ; CHECK: $w0 = COPY [[ANYEXT]](s32)
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_CONSTANT i64 1
+    %2:_(s1) = G_CONSTANT i1 0
+    %3:_(s1) = G_ICMP intpred(sgt), %0(s64), %1
+    %4:_(s1) = G_XOR %3, %2
+    %5:_(s32) = G_ANYEXT %4
+    $w0 = COPY %5(s32)
+    RET_ReallyLR implicit $w0
+...
+---
+name: icmp_not_xor_with_wrong_bool_contents
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0
+
+    ; Even though bit 0 of the constant is 1, we require zero in the upper bits
+    ; for our aarch64's zero-or-one boolean contents.
+    ; CHECK-LABEL: name: icmp_not_xor_with_wrong_bool_contents
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 1
+    ; CHECK: [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 7
+    ; CHECK: [[ICMP:%[0-9]+]]:_(s32) = G_ICMP intpred(sgt), [[COPY]](s64), [[C]]
+    ; CHECK: [[XOR:%[0-9]+]]:_(s32) = G_XOR [[ICMP]], [[C1]]
+    ; CHECK: $w0 = COPY [[XOR]](s32)
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_CONSTANT i64 1
+    %2:_(s32) = G_CONSTANT i32 7
+    %3:_(s32) = G_ICMP intpred(sgt), %0(s64), %1
+    %4:_(s32) = G_XOR %3, %2
+    $w0 = COPY %4(s32)
+    RET_ReallyLR implicit $w0
+...
+---
+name: icmp_multiple_use
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $x0
+
+    ; CHECK-LABEL: name: icmp_multiple_use
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY $x0
+    ; CHECK: [[C:%[0-9]+]]:_(s64) = G_CONSTANT i64 1
+    ; CHECK: [[C1:%[0-9]+]]:_(s1) = G_CONSTANT i1 true
+    ; CHECK: [[ICMP:%[0-9]+]]:_(s1) = G_ICMP intpred(sgt), [[COPY]](s64), [[C]]
+    ; CHECK: [[XOR:%[0-9]+]]:_(s1) = G_XOR [[ICMP]], [[C1]]
+    ; CHECK: %other_use:_(s1) = G_AND [[ICMP]], [[C1]]
+    ; CHECK: %other_use_ext:_(s32) = G_ANYEXT %other_use(s1)
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(s32) = G_ANYEXT [[XOR]](s1)
+    ; CHECK: $w0 = COPY [[ANYEXT]](s32)
+    ; CHECK: $w1 = COPY %other_use_ext(s32)
+    ; CHECK: RET_ReallyLR implicit $w0
+    %0:_(s64) = COPY $x0
+    %1:_(s64) = G_CONSTANT i64 1
+    %2:_(s1) = G_CONSTANT i1 1
+    %3:_(s1) = G_ICMP intpred(sgt), %0(s64), %1
+    %4:_(s1) = G_XOR %3, %2
+    %other_use:_(s1) = G_AND %3, %2
+    %other_use_ext:_(s32) = G_ANYEXT %other_use(s1)
+    %5:_(s32) = G_ANYEXT %4
+    $w0 = COPY %5(s32)
+    $w1 = COPY %other_use_ext
+    RET_ReallyLR implicit $w0
+...
+---
+name: icmp_vector
+tracksRegLiveness: true
+body:             |
+  bb.1:
+    liveins: $q0
+
+    ; CHECK-LABEL: name: icmp_vector
+    ; CHECK: liveins: $q0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $q0
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 5
+    ; CHECK: %splat_op2:_(<4 x s32>) = G_BUILD_VECTOR [[C]](s32), [[C]](s32), [[C]](s32), [[C]](s32)
+    ; CHECK: [[ICMP:%[0-9]+]]:_(<4 x s1>) = G_ICMP intpred(sle), [[COPY]](<4 x s32>), %splat_op2
+    ; CHECK: [[ANYEXT:%[0-9]+]]:_(<4 x s32>) = G_ANYEXT [[ICMP]](<4 x s1>)
+    ; CHECK: $q0 = COPY [[ANYEXT]](<4 x s32>)
+    ; CHECK: RET_ReallyLR implicit $q0
+    %0:_(<4 x s32>) = COPY $q0
+    %1:_(s32) = G_CONSTANT i32 5
+    %splat_op2:_(<4 x s32>) = G_BUILD_VECTOR %1, %1, %1, %1
+    %2:_(s1) = G_CONSTANT i1 1
+    %splat_true:_(<4 x s1>) = G_BUILD_VECTOR %2, %2, %2, %2
+    %3:_(<4 x s1>) = G_ICMP intpred(sgt), %0(<4 x s32>), %splat_op2
+    %4:_(<4 x s1>) = G_XOR %3, %splat_true
+    %5:_(<4 x s32>) = G_ANYEXT %4
+    $q0 = COPY %5(<4 x s32>)
+    RET_ReallyLR implicit $q0
+...

llvm/test/CodeGen/AMDGPU/GlobalISel/divergent-control-flow.ll

 ; CHECK-NEXT:    s_add_u32 s4, s4, external_constant@gotpcrel32@lo+4
 ; CHECK-NEXT:    s_addc_u32 s5, s5, external_constant@gotpcrel32@hi+4
 ; CHECK-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
 ; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
 ; CHECK-NEXT:    s_load_dword s4, s[4:5], 0x0
 ; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
 ; CHECK-NEXT:    s_cmp_lg_u32 s4, 0
 ; CHECK-NEXT:    s_cselect_b32 s4, 1, 0
-; CHECK-NEXT:    s_xor_b32 s4, s4, -1
 ; CHECK-NEXT:    s_and_b32 s4, s4, 1
 ; CHECK-NEXT:    s_cmp_lg_u32 s4, 0
-; CHECK-NEXT:    s_cbranch_scc0 BB4_6
+; CHECK-NEXT:    s_cbranch_scc1 BB4_6
 ; CHECK-NEXT:  ; %bb.1: ; %bb2
-; CHECK-NEXT:    s_getpc_b64 s[4:5]
-; CHECK-NEXT:    s_add_u32 s4, s4, const.ptr@gotpcrel32@lo+4
-; CHECK-NEXT:    s_addc_u32 s5, s5, const.ptr@gotpcrel32@hi+4
-; CHECK-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[6:7], 0, 1
+; CHECK-NEXT:    s_getpc_b64 s[6:7]
+; CHECK-NEXT:    s_add_u32 s6, s6, const.ptr@gotpcrel32@lo+4
+; CHECK-NEXT:    s_addc_u32 s7, s7, const.ptr@gotpcrel32@hi+4
+; CHECK-NEXT:    s_load_dwordx2 s[6:7], s[6:7], 0x0
+; CHECK-NEXT:    s_mov_b32 s4, -1
 ; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
-; CHECK-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; CHECK-NEXT:    s_load_dwordx2 s[6:7], s[6:7], 0x0
 ; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
-; CHECK-NEXT:    v_mov_b32_e32 v0, s4
-; CHECK-NEXT:    v_mov_b32_e32 v1, s5
+; CHECK-NEXT:    v_mov_b32_e32 v0, s6
+; CHECK-NEXT:    v_mov_b32_e32 v1, s7
 ; CHECK-NEXT:    flat_load_dword v0, v[0:1]
-; CHECK-NEXT:    s_mov_b32 s4, -1
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; CHECK-NEXT:    v_cmp_gt_f32_e32 vcc, 1.0, v0
-; CHECK-NEXT:    s_xor_b64 s[8:9], vcc, s[6:7]
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[8:9]
+; CHECK-NEXT:    v_cmp_ngt_f32_e32 vcc, 1.0, v0
+; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], vcc
 ; CHECK-NEXT:  ; %bb.2: ; %bb7
 ; CHECK-NEXT:    s_mov_b32 s4, 0
 ; CHECK-NEXT:  ; %bb.3: ; %bb8
 ; CHECK-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CHECK-NEXT:    v_cmp_eq_u32_e64 s[6:7], s4, 0
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], s[6:7]
 ; CHECK-NEXT:    s_cbranch_execz BB4_5
 ; CHECK-NEXT:  ; %bb.4: ; %bb11

 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    v_add_u32_e32 v1, 1, v1
 ; CHECK-NEXT:    v_cmp_le_i32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cmp_ne_u32_e64 s[2:3], 0, 1
 ; CHECK-NEXT:    s_cbranch_vccnz BB5_3
 ; CHECK-NEXT:  ; %bb.2: ; %bb4
 ; CHECK-NEXT:    ; in Loop: Header=BB5_1 Depth=1
 ; CHECK-NEXT:    global_load_dword v2, v[0:1], off
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[2:3], 0, 1
 ; CHECK-NEXT:    s_waitcnt vmcnt(0)
-; CHECK-NEXT:    v_cmp_lt_i32_e32 vcc, v0, v2
-; CHECK-NEXT:    s_xor_b64 s[2:3], vcc, s[2:3]
+; CHECK-NEXT:    v_cmp_ge_i32_e64 s[2:3], v0, v2
 ; CHECK-NEXT:  BB5_3: ; %Flow
 ; CHECK-NEXT:    ; in Loop: Header=BB5_1 Depth=1
 ; CHECK-NEXT:    s_and_b64 s[2:3], exec, s[2:3]
 ; CHECK-NEXT:    s_or_b64 s[0:1], s[2:3], s[0:1]
 ; CHECK-NEXT:    s_andn2_b64 exec, exec, s[0:1]
 ; CHECK-NEXT:    s_cbranch_execnz BB5_1
 ; CHECK-NEXT:  ; %bb.4: ; %bb9
 ; CHECK-NEXT:    s_endpgm

llvm/test/CodeGen/AMDGPU/GlobalISel/sdiv.i64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -amdgpu-codegenprepare-disable-idiv-expansion=1 -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -denormal-fp-math-f32=preserve-sign < %s | FileCheck -check-prefixes=CHECK,GISEL %s
 ; RUN: llc -global-isel -amdgpu-codegenprepare-disable-idiv-expansion=0 -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -denormal-fp-math-f32=preserve-sign < %s | FileCheck -check-prefixes=CHECK,CGP %s
 
 ; The same 32-bit expansion is implemented in the legalizer and in AMDGPUCodeGenPrepare.
 
 define i64 @v_sdiv_i64(i64 %num, i64 %den) {
 ; CHECK-LABEL: v_sdiv_i64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_or_b32_e32 v5, v1, v3
 ; CHECK-NEXT:    v_mov_b32_e32 v4, 0
-; CHECK-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CHECK-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CHECK-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB0_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v4, 31, v3
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v2, v4
 ; CHECK-NEXT:    v_addc_u32_e32 v3, vcc, v3, v4, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v5, v5, v4
 ; CHECK-NEXT:    v_xor_b32_e32 v3, v3, v4
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v6, v5

 
 define amdgpu_ps i64 @s_sdiv_i64(i64 inreg %num, i64 inreg %den) {
 ; CHECK-LABEL: s_sdiv_i64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_or_b64 s[6:7], s[2:3], s[4:5]
 ; CHECK-NEXT:    s_mov_b32 s0, 0
 ; CHECK-NEXT:    s_mov_b32 s1, -1
 ; CHECK-NEXT:    s_and_b64 s[6:7], s[6:7], s[0:1]
-; CHECK-NEXT:    v_cmp_eq_u64_e64 s[6:7], s[6:7], 0
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[8:9], 0, 1
-; CHECK-NEXT:    s_xor_b64 vcc, s[6:7], s[8:9]
-; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; CHECK-NEXT:    v_cmp_ne_u64_e64 vcc, s[6:7], 0
 ; CHECK-NEXT:    s_cbranch_vccz BB1_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    s_ashr_i32 s6, s3, 31
 ; CHECK-NEXT:    s_ashr_i32 s8, s5, 31
 ; CHECK-NEXT:    s_add_u32 s0, s2, s6
 ; CHECK-NEXT:    s_cselect_b32 s1, 1, 0
 ; CHECK-NEXT:    s_and_b32 s1, s1, 1
 ; CHECK-NEXT:    s_cmp_lg_u32 s1, 0

 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v4, v3, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
 ; CHECK-NEXT:    s_xor_b64 s[0:1], s[6:7], s[8:9]
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, s0, v0
 ; CHECK-NEXT:    v_subrev_i32_e32 v0, vcc, s0, v0
 ; CHECK-NEXT:    s_mov_b32 s1, 0
-; CHECK-NEXT:  BB1_2: ; %Flow
+; CHECK-NEXT:    s_branch BB1_3
+; CHECK-NEXT:  BB1_2:
+; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; CHECK-NEXT:  BB1_3: ; %Flow
 ; CHECK-NEXT:    s_and_b32 s0, s1, 1
 ; CHECK-NEXT:    s_cmp_lg_u32 s0, 0
-; CHECK-NEXT:    s_cbranch_scc0 BB1_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_cbranch_scc0 BB1_5
+; CHECK-NEXT:  ; %bb.4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, s4
 ; CHECK-NEXT:    s_sub_i32 s0, 0, s4
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, s0, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, s2, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v0, s4
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, s2, v1
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s4, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CHECK-NEXT:    v_subrev_i32_e64 v2, s[0:1], s4, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s4, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
-; CHECK-NEXT:  BB1_4:
+; CHECK-NEXT:  BB1_5:
 ; CHECK-NEXT:    v_readfirstlane_b32 s0, v0
 ; CHECK-NEXT:    s_mov_b32 s1, s0
 ; CHECK-NEXT:    ; return to shader part epilog
   %result = sdiv i64 %num, %den
   %cast = bitcast i64 %result to <2 x i32>
   %elt.0 = extractelement <2 x i32> %cast, i32 0
   %elt.1 = extractelement <2 x i32> %cast, i32 1
   %res.0 = call i32 @llvm.amdgcn.readfirstlane(i32 %elt.0)

 ;
 ; CGP-LABEL: v_sdiv_v2i64:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_mov_b32_e32 v9, v1
 ; CGP-NEXT:    v_mov_b32_e32 v8, v0
 ; CGP-NEXT:    v_or_b32_e32 v1, v9, v5
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB2_2
 ; CGP-NEXT:  ; %bb.1:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v0, 31, v5
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v0
 ; CGP-NEXT:    v_addc_u32_e32 v5, vcc, v5, v0, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v1, v1, v0
 ; CGP-NEXT:    v_xor_b32_e32 v5, v5, v0
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v10, v1

 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, 1, v0
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v4
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:  BB2_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    v_or_b32_e32 v5, v3, v7
 ; CGP-NEXT:    v_mov_b32_e32 v4, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB2_6
 ; CGP-NEXT:  ; %bb.5:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v4, 31, v7
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v4
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, v7, v4, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v5, v5, v4
 ; CGP-NEXT:    v_xor_b32_e32 v7, v7, v4
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v8, v5

 ; CHECK-LABEL: v_sdiv_i64_pow2_shl_denom:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    s_movk_i32 s4, 0x1000
 ; CHECK-NEXT:    s_mov_b32 s5, 0
 ; CHECK-NEXT:    v_lshl_b64 v[4:5], s[4:5], v2
 ; CHECK-NEXT:    v_mov_b32_e32 v2, 0
 ; CHECK-NEXT:    v_or_b32_e32 v3, v1, v5
-; CHECK-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[2:3]
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CHECK-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CHECK-NEXT:    ; implicit-def: $vgpr2_vgpr3
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB7_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v2, 31, v5
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v4, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v5, vcc, v5, v2, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v3, v3, v2
 ; CHECK-NEXT:    v_xor_b32_e32 v5, v5, v2
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v6, v3

 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    s_movk_i32 s4, 0x1000
 ; CGP-NEXT:    s_mov_b32 s5, 0
 ; CGP-NEXT:    v_lshl_b64 v[10:11], s[4:5], v4
 ; CGP-NEXT:    v_mov_b32_e32 v7, v1
 ; CGP-NEXT:    v_mov_b32_e32 v5, v0
 ; CGP-NEXT:    v_or_b32_e32 v1, v7, v11
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    v_lshl_b64 v[8:9], s[4:5], v6
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB8_2
 ; CGP-NEXT:  ; %bb.1:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v0, 31, v11
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v10, v0
 ; CGP-NEXT:    v_addc_u32_e32 v4, vcc, v11, v0, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v1, v1, v0
 ; CGP-NEXT:    v_xor_b32_e32 v4, v4, v0
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v6, v1

 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v0
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v10
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:  BB8_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    v_or_b32_e32 v5, v3, v9
 ; CGP-NEXT:    v_mov_b32_e32 v4, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB8_6
 ; CGP-NEXT:  ; %bb.5:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v4, 31, v9
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v4
 ; CGP-NEXT:    v_addc_u32_e32 v6, vcc, v9, v4, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v5, v5, v4
 ; CGP-NEXT:    v_xor_b32_e32 v6, v6, v4
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v7, v5

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -amdgpu-codegenprepare-disable-idiv-expansion=1 -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -denormal-fp-math-f32=preserve-sign < %s | FileCheck -check-prefixes=CHECK,GISEL %s
 ; RUN: llc -global-isel -amdgpu-codegenprepare-disable-idiv-expansion=0 -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -denormal-fp-math-f32=preserve-sign < %s | FileCheck -check-prefixes=CHECK,CGP %s
 
 ; The same 32-bit expansion is implemented in the legalizer and in AMDGPUCodeGenPrepare.
 
 define i64 @v_srem_i64(i64 %num, i64 %den) {
 ; CHECK-LABEL: v_srem_i64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_or_b32_e32 v5, v1, v3
 ; CHECK-NEXT:    v_mov_b32_e32 v4, 0
-; CHECK-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CHECK-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CHECK-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB0_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v4, 31, v3
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v2, v4
 ; CHECK-NEXT:    v_addc_u32_e32 v3, vcc, v3, v4, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v3, v3, v4
 ; CHECK-NEXT:    v_xor_b32_e32 v5, v5, v4
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v4, v5

 
 define amdgpu_ps i64 @s_srem_i64(i64 inreg %num, i64 inreg %den) {
 ; CHECK-LABEL: s_srem_i64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_or_b64 s[6:7], s[2:3], s[4:5]
 ; CHECK-NEXT:    s_mov_b32 s0, 0
 ; CHECK-NEXT:    s_mov_b32 s1, -1
 ; CHECK-NEXT:    s_and_b64 s[6:7], s[6:7], s[0:1]
-; CHECK-NEXT:    v_cmp_eq_u64_e64 s[6:7], s[6:7], 0
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[8:9], 0, 1
-; CHECK-NEXT:    s_xor_b64 vcc, s[6:7], s[8:9]
-; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; CHECK-NEXT:    v_cmp_ne_u64_e64 vcc, s[6:7], 0
 ; CHECK-NEXT:    s_cbranch_vccz BB1_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    s_ashr_i32 s0, s5, 31
 ; CHECK-NEXT:    s_ashr_i32 s6, s3, 31
 ; CHECK-NEXT:    s_add_u32 s8, s2, s6
 ; CHECK-NEXT:    s_cselect_b32 s7, 1, 0
 ; CHECK-NEXT:    s_and_b32 s7, s7, 1
 ; CHECK-NEXT:    s_cmp_lg_u32 s7, 0

 ; CHECK-NEXT:    v_subrev_i32_e32 v4, vcc, s10, v3
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v3, v4, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v2, v0, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, s6, v0
 ; CHECK-NEXT:    v_subrev_i32_e32 v0, vcc, s6, v0
 ; CHECK-NEXT:    s_mov_b32 s1, 0
-; CHECK-NEXT:  BB1_2: ; %Flow
+; CHECK-NEXT:    s_branch BB1_3
+; CHECK-NEXT:  BB1_2:
+; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; CHECK-NEXT:  BB1_3: ; %Flow
 ; CHECK-NEXT:    s_and_b32 s0, s1, 1
 ; CHECK-NEXT:    s_cmp_lg_u32 s0, 0
-; CHECK-NEXT:    s_cbranch_scc0 BB1_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_cbranch_scc0 BB1_5
+; CHECK-NEXT:  ; %bb.4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, s4
 ; CHECK-NEXT:    s_sub_i32 s0, 0, s4
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, s0, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, s2, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v0, v0, s4
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, s2, v0
 ; CHECK-NEXT:    v_subrev_i32_e32 v1, vcc, s4, v0
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s4, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_subrev_i32_e32 v1, vcc, s4, v0
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s4, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; CHECK-NEXT:  BB1_4:
+; CHECK-NEXT:  BB1_5:
 ; CHECK-NEXT:    v_readfirstlane_b32 s0, v0
 ; CHECK-NEXT:    s_mov_b32 s1, s0
 ; CHECK-NEXT:    ; return to shader part epilog
   %result = srem i64 %num, %den
   %cast = bitcast i64 %result to <2 x i32>
   %elt.0 = extractelement <2 x i32> %cast, i32 0
   %elt.1 = extractelement <2 x i32> %cast, i32 1
   %res.0 = call i32 @llvm.amdgcn.readfirstlane(i32 %elt.0)

 ;
 ; CGP-LABEL: v_srem_v2i64:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_mov_b32_e32 v9, v1
 ; CGP-NEXT:    v_mov_b32_e32 v8, v0
 ; CGP-NEXT:    v_or_b32_e32 v1, v9, v5
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB2_2
 ; CGP-NEXT:  ; %bb.1:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v0, 31, v5
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v4, v0
 ; CGP-NEXT:    v_addc_u32_e32 v5, vcc, v5, v0, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v1, v1, v0
 ; CGP-NEXT:    v_xor_b32_e32 v0, v5, v0
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v5, v1

 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v0, v4
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v4
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:  BB2_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    v_or_b32_e32 v5, v3, v7
 ; CGP-NEXT:    v_mov_b32_e32 v4, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB2_6
 ; CGP-NEXT:  ; %bb.5:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v4, 31, v7
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v4
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, v7, v4, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v5, v5, v4
 ; CGP-NEXT:    v_xor_b32_e32 v4, v7, v4
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v7, v5

 ; CHECK-LABEL: v_srem_i64_pow2_shl_denom:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    s_movk_i32 s4, 0x1000
 ; CHECK-NEXT:    s_mov_b32 s5, 0
 ; CHECK-NEXT:    v_lshl_b64 v[4:5], s[4:5], v2
 ; CHECK-NEXT:    v_mov_b32_e32 v2, 0
 ; CHECK-NEXT:    v_or_b32_e32 v3, v1, v5
-; CHECK-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[2:3]
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CHECK-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CHECK-NEXT:    ; implicit-def: $vgpr2_vgpr3
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB7_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_ashrrev_i32_e32 v2, 31, v5
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v4, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v5, vcc, v5, v2, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v3, v3, v2
 ; CHECK-NEXT:    v_xor_b32_e32 v2, v5, v2
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v5, v3

 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    s_movk_i32 s4, 0x1000
 ; CGP-NEXT:    s_mov_b32 s5, 0
 ; CGP-NEXT:    v_lshl_b64 v[10:11], s[4:5], v4
 ; CGP-NEXT:    v_mov_b32_e32 v7, v1
 ; CGP-NEXT:    v_mov_b32_e32 v5, v0
 ; CGP-NEXT:    v_or_b32_e32 v1, v7, v11
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    v_lshl_b64 v[8:9], s[4:5], v6
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB8_2
 ; CGP-NEXT:  ; %bb.1:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v0, 31, v11
 ; CGP-NEXT:    v_add_i32_e32 v1, vcc, v10, v0
 ; CGP-NEXT:    v_addc_u32_e32 v4, vcc, v11, v0, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v1, v1, v0
 ; CGP-NEXT:    v_xor_b32_e32 v0, v4, v0
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v1

 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v0, v10
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v10
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:  BB8_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    v_or_b32_e32 v5, v3, v9
 ; CGP-NEXT:    v_mov_b32_e32 v4, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB8_6
 ; CGP-NEXT:  ; %bb.5:
 ; CGP-NEXT:    v_ashrrev_i32_e32 v4, 31, v9
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v4
 ; CGP-NEXT:    v_addc_u32_e32 v6, vcc, v9, v4, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v5, v5, v4
 ; CGP-NEXT:    v_xor_b32_e32 v4, v6, v4
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v6, v5

llvm/test/CodeGen/AMDGPU/GlobalISel/udiv.i64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -amdgpu-codegenprepare-disable-idiv-expansion=1 -mtriple=amdgcn-amd-amdhsa -denormal-fp-math-f32=preserve-sign -mattr=+mad-mac-f32-insts < %s | FileCheck -check-prefixes=CHECK,GISEL %s
 ; RUN: llc -global-isel -amdgpu-codegenprepare-disable-idiv-expansion=0 -mtriple=amdgcn-amd-amdhsa -denormal-fp-math-f32=preserve-sign -mattr=+mad-mac-f32-insts < %s | FileCheck -check-prefixes=CHECK,CGP %s
 
 ; The same 32-bit expansion is implemented in the legalizer and in AMDGPUCodeGenPrepare.
 
 define i64 @v_udiv_i64(i64 %num, i64 %den) {
 ; CHECK-LABEL: v_udiv_i64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_or_b32_e32 v5, v1, v3
 ; CHECK-NEXT:    v_mov_b32_e32 v4, 0
-; CHECK-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CHECK-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CHECK-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB0_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v4, v2
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v5, v3
 ; CHECK-NEXT:    v_sub_i32_e32 v6, vcc, 0, v2
 ; CHECK-NEXT:    v_subb_u32_e32 v7, vcc, 0, v3, vcc
 ; CHECK-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v4, v4

 
 define amdgpu_ps i64 @s_udiv_i64(i64 inreg %num, i64 inreg %den) {
 ; CHECK-LABEL: s_udiv_i64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_or_b64 s[6:7], s[0:1], s[2:3]
 ; CHECK-NEXT:    s_mov_b32 s4, 0
 ; CHECK-NEXT:    s_mov_b32 s5, -1
 ; CHECK-NEXT:    s_and_b64 s[6:7], s[6:7], s[4:5]
-; CHECK-NEXT:    v_cmp_eq_u64_e64 s[6:7], s[6:7], 0
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[8:9], 0, 1
-; CHECK-NEXT:    s_xor_b64 vcc, s[6:7], s[8:9]
-; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; CHECK-NEXT:    v_cmp_ne_u64_e64 vcc, s[6:7], 0
 ; CHECK-NEXT:    s_cbranch_vccz BB1_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, s2
 ; CHECK-NEXT:    v_mov_b32_e32 v1, s3
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v2, s3
 ; CHECK-NEXT:    s_sub_u32 s6, 0, s2
 ; CHECK-NEXT:    s_cselect_b32 s4, 1, 0
 ; CHECK-NEXT:    v_mov_b32_e32 v3, s1

 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, s3, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v5, v3, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v8, v4, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    s_mov_b32 s5, 0
-; CHECK-NEXT:  BB1_2: ; %Flow
+; CHECK-NEXT:    s_branch BB1_3
+; CHECK-NEXT:  BB1_2:
+; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; CHECK-NEXT:  BB1_3: ; %Flow
 ; CHECK-NEXT:    s_and_b32 s1, s5, 1
 ; CHECK-NEXT:    s_cmp_lg_u32 s1, 0
-; CHECK-NEXT:    s_cbranch_scc0 BB1_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_cbranch_scc0 BB1_5
+; CHECK-NEXT:  ; %bb.4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, s2
 ; CHECK-NEXT:    s_sub_i32 s1, 0, s2
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, s1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, s0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, v0, s2
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, s0, v1
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s2, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CHECK-NEXT:    v_subrev_i32_e64 v2, s[0:1], s2, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s2, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
-; CHECK-NEXT:  BB1_4:
+; CHECK-NEXT:  BB1_5:
 ; CHECK-NEXT:    v_readfirstlane_b32 s0, v0
 ; CHECK-NEXT:    s_mov_b32 s1, s0
 ; CHECK-NEXT:    ; return to shader part epilog
   %result = udiv i64 %num, %den
   %cast = bitcast i64 %result to <2 x i32>
   %elt.0 = extractelement <2 x i32> %cast, i32 0
   %elt.1 = extractelement <2 x i32> %cast, i32 1
   %res.0 = call i32 @llvm.amdgcn.readfirstlane(i32 %elt.0)

 ;
 ; CGP-LABEL: v_udiv_v2i64:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_mov_b32_e32 v8, v0
 ; CGP-NEXT:    v_mov_b32_e32 v9, v1
 ; CGP-NEXT:    v_or_b32_e32 v1, v9, v5
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB2_2
 ; CGP-NEXT:  ; %bb.1:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v4
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v10, vcc, 0, v4
 ; CGP-NEXT:    v_subb_u32_e32 v11, vcc, 0, v5, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v0, v0

 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, 1, v0
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v4
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:  BB2_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    v_or_b32_e32 v5, v3, v7
 ; CGP-NEXT:    v_mov_b32_e32 v4, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB2_6
 ; CGP-NEXT:  ; %bb.5:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v6
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v5, v7
 ; CGP-NEXT:    v_sub_i32_e32 v8, vcc, 0, v6
 ; CGP-NEXT:    v_subb_u32_e32 v9, vcc, 0, v7, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v4, v4

 }
 
 define i64 @v_udiv_i64_pow2_shl_denom(i64 %x, i64 %y) {
 ; CHECK-LABEL: v_udiv_i64_pow2_shl_denom:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    s_movk_i32 s4, 0x1000
 ; CHECK-NEXT:    s_mov_b32 s5, 0
-; CHECK-NEXT:    v_mov_b32_e32 v6, 0
 ; CHECK-NEXT:    v_lshl_b64 v[4:5], s[4:5], v2
-; CHECK-NEXT:    v_or_b32_e32 v7, v1, v5
-; CHECK-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[6:7]
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CHECK-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CHECK-NEXT:    v_or_b32_e32 v3, v1, v5
+; CHECK-NEXT:    v_mov_b32_e32 v2, 0
+; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CHECK-NEXT:    ; implicit-def: $vgpr2_vgpr3
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB7_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v2, v4
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v3, v5
 ; CHECK-NEXT:    v_sub_i32_e32 v6, vcc, 0, v4
 ; CHECK-NEXT:    v_subb_u32_e32 v7, vcc, 0, v5, vcc
 ; CHECK-NEXT:    v_mac_f32_e32 v2, 0x4f800000, v3
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v2, v2

 ;
 ; CGP-LABEL: v_udiv_v2i64_pow2_shl_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_mov_b32_e32 v5, v0
 ; CGP-NEXT:    v_mov_b32_e32 v7, v1
 ; CGP-NEXT:    s_movk_i32 s4, 0x1000
 ; CGP-NEXT:    s_mov_b32 s5, 0
-; CGP-NEXT:    v_mov_b32_e32 v0, 0
 ; CGP-NEXT:    v_lshl_b64 v[10:11], s[4:5], v4
 ; CGP-NEXT:    v_lshl_b64 v[8:9], s[4:5], v6
 ; CGP-NEXT:    v_or_b32_e32 v1, v7, v11
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_mov_b32_e32 v0, 0
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB8_2
 ; CGP-NEXT:  ; %bb.1:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v10
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v1, v11
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, 0, v10
 ; CGP-NEXT:    v_subb_u32_e32 v6, vcc, 0, v11, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v0, v0

 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v0
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v10
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:  BB8_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[6:7]
 ; CGP-NEXT:    v_or_b32_e32 v5, v3, v9
 ; CGP-NEXT:    v_mov_b32_e32 v4, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB8_6
 ; CGP-NEXT:  ; %bb.5:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v8
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v5, v9
 ; CGP-NEXT:    v_sub_i32_e32 v6, vcc, 0, v8
 ; CGP-NEXT:    v_subb_u32_e32 v7, vcc, 0, v9, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v4, v4

llvm/test/CodeGen/AMDGPU/GlobalISel/urem.i64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -amdgpu-codegenprepare-disable-idiv-expansion=1 -mtriple=amdgcn-amd-amdhsa -denormal-fp-math-f32=preserve-sign -mattr=+mad-mac-f32-insts < %s | FileCheck -check-prefixes=CHECK,GISEL %s
 ; RUN: llc -global-isel -amdgpu-codegenprepare-disable-idiv-expansion=0 -mtriple=amdgcn-amd-amdhsa -denormal-fp-math-f32=preserve-sign -mattr=+mad-mac-f32-insts < %s | FileCheck -check-prefixes=CHECK,CGP %s
 
 ; The same 32-bit expansion is implemented in the legalizer and in AMDGPUCodeGenPrepare.
 
 define i64 @v_urem_i64(i64 %num, i64 %den) {
 ; CHECK-LABEL: v_urem_i64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    v_or_b32_e32 v5, v1, v3
 ; CHECK-NEXT:    v_mov_b32_e32 v4, 0
-; CHECK-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CHECK-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CHECK-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB0_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v4, v2
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v5, v3
 ; CHECK-NEXT:    v_sub_i32_e32 v6, vcc, 0, v2
 ; CHECK-NEXT:    v_subb_u32_e32 v7, vcc, 0, v3, vcc
 ; CHECK-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v4, v4

 
 define amdgpu_ps i64 @s_urem_i64(i64 inreg %num, i64 inreg %den) {
 ; CHECK-LABEL: s_urem_i64:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_or_b64 s[6:7], s[0:1], s[2:3]
 ; CHECK-NEXT:    s_mov_b32 s4, 0
 ; CHECK-NEXT:    s_mov_b32 s5, -1
 ; CHECK-NEXT:    s_and_b64 s[6:7], s[6:7], s[4:5]
-; CHECK-NEXT:    v_cmp_eq_u64_e64 s[6:7], s[6:7], 0
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[8:9], 0, 1
-; CHECK-NEXT:    s_xor_b64 vcc, s[6:7], s[8:9]
-; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; CHECK-NEXT:    v_cmp_ne_u64_e64 vcc, s[6:7], 0
 ; CHECK-NEXT:    s_cbranch_vccz BB1_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, s2
 ; CHECK-NEXT:    v_mov_b32_e32 v1, s3
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v2, s3
 ; CHECK-NEXT:    s_sub_u32 s6, 0, s2
 ; CHECK-NEXT:    s_cselect_b32 s4, 1, 0
 ; CHECK-NEXT:    v_mov_b32_e32 v3, s1

 ; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, s3, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v6, v4, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v3, v5, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v2, v0, vcc
 ; CHECK-NEXT:    s_mov_b32 s5, 0
-; CHECK-NEXT:  BB1_2: ; %Flow
+; CHECK-NEXT:    s_branch BB1_3
+; CHECK-NEXT:  BB1_2:
+; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
+; CHECK-NEXT:  BB1_3: ; %Flow
 ; CHECK-NEXT:    s_and_b32 s1, s5, 1
 ; CHECK-NEXT:    s_cmp_lg_u32 s1, 0
-; CHECK-NEXT:    s_cbranch_scc0 BB1_4
-; CHECK-NEXT:  ; %bb.3:
+; CHECK-NEXT:    s_cbranch_scc0 BB1_5
+; CHECK-NEXT:  ; %bb.4:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v0, s2
 ; CHECK-NEXT:    s_sub_i32 s1, 0, s2
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, s1, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v0, s0, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, s0, v0
 ; CHECK-NEXT:    v_subrev_i32_e32 v1, vcc, s2, v0
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s2, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_subrev_i32_e32 v1, vcc, s2, v0
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s2, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
-; CHECK-NEXT:  BB1_4:
+; CHECK-NEXT:  BB1_5:
 ; CHECK-NEXT:    v_readfirstlane_b32 s0, v0
 ; CHECK-NEXT:    s_mov_b32 s1, s0
 ; CHECK-NEXT:    ; return to shader part epilog
   %result = urem i64 %num, %den
   %cast = bitcast i64 %result to <2 x i32>
   %elt.0 = extractelement <2 x i32> %cast, i32 0
   %elt.1 = extractelement <2 x i32> %cast, i32 1
   %res.0 = call i32 @llvm.amdgcn.readfirstlane(i32 %elt.0)

 ;
 ; CGP-LABEL: v_urem_v2i64:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_mov_b32_e32 v8, v0
 ; CGP-NEXT:    v_mov_b32_e32 v9, v1
 ; CGP-NEXT:    v_or_b32_e32 v1, v9, v5
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB2_2
 ; CGP-NEXT:  ; %bb.1:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v4
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v10, vcc, 0, v4
 ; CGP-NEXT:    v_subb_u32_e32 v11, vcc, 0, v5, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v0, v0

 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v0, v4
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v4
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:  BB2_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    v_or_b32_e32 v5, v3, v7
 ; CGP-NEXT:    v_mov_b32_e32 v4, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB2_6
 ; CGP-NEXT:  ; %bb.5:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v6
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v5, v7
 ; CGP-NEXT:    v_sub_i32_e32 v8, vcc, 0, v6
 ; CGP-NEXT:    v_subb_u32_e32 v9, vcc, 0, v7, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v4, v4

 }
 
 define i64 @v_urem_i64_pow2_shl_denom(i64 %x, i64 %y) {
 ; CHECK-LABEL: v_urem_i64_pow2_shl_denom:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CHECK-NEXT:    s_movk_i32 s4, 0x1000
 ; CHECK-NEXT:    s_mov_b32 s5, 0
-; CHECK-NEXT:    v_mov_b32_e32 v6, 0
 ; CHECK-NEXT:    v_lshl_b64 v[4:5], s[4:5], v2
-; CHECK-NEXT:    v_or_b32_e32 v7, v1, v5
-; CHECK-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[6:7]
-; CHECK-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CHECK-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CHECK-NEXT:    v_or_b32_e32 v3, v1, v5
+; CHECK-NEXT:    v_mov_b32_e32 v2, 0
+; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CHECK-NEXT:    ; implicit-def: $vgpr2_vgpr3
-; CHECK-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB7_2
 ; CHECK-NEXT:  ; %bb.1:
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v2, v4
 ; CHECK-NEXT:    v_cvt_f32_u32_e32 v3, v5
 ; CHECK-NEXT:    v_sub_i32_e32 v6, vcc, 0, v4
 ; CHECK-NEXT:    v_subb_u32_e32 v7, vcc, 0, v5, vcc
 ; CHECK-NEXT:    v_mac_f32_e32 v2, 0x4f800000, v3
 ; CHECK-NEXT:    v_rcp_iflag_f32_e32 v2, v2

 ;
 ; CGP-LABEL: v_urem_v2i64_pow2_shl_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_mov_b32_e32 v5, v0
 ; CGP-NEXT:    v_mov_b32_e32 v7, v1
 ; CGP-NEXT:    s_movk_i32 s4, 0x1000
 ; CGP-NEXT:    s_mov_b32 s5, 0
-; CGP-NEXT:    v_mov_b32_e32 v0, 0
 ; CGP-NEXT:    v_lshl_b64 v[10:11], s[4:5], v4
 ; CGP-NEXT:    v_lshl_b64 v[8:9], s[4:5], v6
 ; CGP-NEXT:    v_or_b32_e32 v1, v7, v11
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[0:1]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_mov_b32_e32 v0, 0
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB8_2
 ; CGP-NEXT:  ; %bb.1:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, v10
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v1, v11
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, 0, v10
 ; CGP-NEXT:    v_subb_u32_e32 v6, vcc, 0, v11, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v0, v0

 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v0, v10
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v10
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CGP-NEXT:    v_mov_b32_e32 v1, 0
 ; CGP-NEXT:  BB8_4:
 ; CGP-NEXT:    s_or_b64 exec, exec, s[4:5]
 ; CGP-NEXT:    v_or_b32_e32 v5, v3, v9
 ; CGP-NEXT:    v_mov_b32_e32 v4, 0
-; CGP-NEXT:    v_cmp_eq_u64_e32 vcc, 0, v[4:5]
-; CGP-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, 1
-; CGP-NEXT:    s_xor_b64 s[4:5], vcc, s[4:5]
+; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[4:5]
 ; CGP-NEXT:    ; implicit-def: $vgpr4_vgpr5
-; CGP-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
-; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[6:7]
+; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
+; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CGP-NEXT:    s_cbranch_execz BB8_6
 ; CGP-NEXT:  ; %bb.5:
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, v8
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v5, v9
 ; CGP-NEXT:    v_sub_i32_e32 v6, vcc, 0, v8
 ; CGP-NEXT:    v_subb_u32_e32 v7, vcc, 0, v9, vcc
 ; CGP-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; CGP-NEXT:    v_rcp_iflag_f32_e32 v4, v4