[AMDGPU] Add S_MOV_B64_IMM_PSEUDO for wide constants

[rampitec]

Summary

This is to allow 64 bit constant rematerialization. If a constant
is split into two separate moves initializing sub0 and sub1 like
now RA cannot rematerizalize a 64 bit register.

This gives 10-20% uplift in a set of huge apps heavily using double
precession math.

Fixes: SWDEV-292645

[arsenm]

Do you have a MIR example that fails to rematerialize as is? We split so many 64-bit operations it might be worthwhile just teaching the coalescer how to handle it

[rampitec]

In D104874#2839691, @arsenm wrote:

Do you have a MIR example that fails to rematerialize as is? We split so many 64-bit operations it might be worthwhile just teaching the coalescer how to handle it

We fail to rematerialize this newly added test remat-fp64-constants.ll, it shall be fairly easy to stop it before RA.
I agree if we can teach RA to materialize subregs it would be the best, but honestly I have failed.

[rampitec]

In D104874#2839696, @rampitec wrote:

In D104874#2839691, @arsenm wrote:

Do you have a MIR example that fails to rematerialize as is? We split so many 64-bit operations it might be worthwhile just teaching the coalescer how to handle it

We fail to rematerialize this newly added test remat-fp64-constants.ll, it shall be fairly easy to stop it before RA.
I agree if we can teach RA to materialize subregs it would be the best, but honestly I have failed.

I have sent you the mir. If you have an idea how to dial with LiveRangeEdit::canRematerializeAt() with subregs, that would be much nicer.

[foad]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

2698

Does this ever return true? If I understand the isel changes, if these conditions were satisfied, you would have selected S_MOV_B64 instead of S_MOV_B64_IMM_PSEUDO.

[rampitec]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

2698

One can eventually end up here after a folding.

[rampitec]

Fixed some tidy warnings.

[rampitec]

Dropped dead code in the isFoldableCopy().

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

2698

Actually you are right, it is dead given checks when it is produced.

[rampitec]

I do not actually think we can fix or at least easily fix rematerialization itself when we have subregs, something like this:

undef %25.sub1:sreg_64 = S_MOV_B32 1056702873
%25.sub0:sreg_64 = S_MOV_B32 2093606576

Moving a first instruction will leave a second one with %25 use without a def. We would need to always move all defs and that hardly counts as trivially rematerializable, especially if we have multiple defs.

Although that seems to be possible to improve this and combine individual 32 bit moves into this pseudo inside the RA pipeline, after coalescer and right before the pre-RA scheduler. That way greedy will see a single instruction, but we also allow coalescer to work. This should remove the heuristic aspect of it, although will need a new pass and SReg_64 defs scan.

[rampitec]

@qcolombet do you mind to tell your opinion, if that still may be rematerializable in the RA?

undef %25.sub1:sreg_64 = S_MOV_B32 1056702873
%25.sub0:sreg_64 = S_MOV_B32 2093606576

I would really appreciate a second thought.

[rampitec]

Removed selection part, we will still select split wide constants.
Added pass to combine such assignments into the new pseudo after the coalescer and rename independent subregs.
This gives better results at a cost of compile time.

[arsenm]

Probably should rename the pass to be a bit more generic. There are other optimizations we could potentially do here (e.g. this is really where we probably should be optimizing m0 initializations instead of what we do now, and maybe some AGPR-VGPR fixups)

llvm/lib/Target/AMDGPU/GCNCombineTupleInits.cpp

87 ↗

(On Diff #355353)

uint64_t

107 ↗

(On Diff #355353)

Need to make sure this shift is done as uint64_t to avoid UB

115 ↗

(On Diff #355353)

Typo Comining

133 ↗

(On Diff #355353)

Don't need the newline

154–155 ↗

(On Diff #355353)

Initialize and define at the same time?

156 ↗

(On Diff #355353)

Why not also handle the VALU cases?

[arsenm]

llvm/lib/Target/AMDGPU/GCNCombineTupleInits.cpp

94 ↗

(On Diff #355353)

What happens if there are additional defs? I also don't see a dedicated MIR test for this?

[dfukalov]

llvm/test/CodeGen/AMDGPU/inline-constraints.ll

61–62

Can be removed?

[rampitec]

Addressed review comments.

llvm/lib/Target/AMDGPU/GCNCombineTupleInits.cpp

94 ↗

(On Diff #355353)

If a Def0 or Def1 not null and we need to assign it function exits. Added mir test.

156 ↗

(On Diff #355353)

I have tried by I see no benefits. We can add it later if we see any, I just didn't. Doing this on VALU will also need an EXEC modification scan between defs. In addition we do not always conver a v_mov_b32 into brev, so I have also seen some regressions. That would need to be addressed first too.

[rampitec]

PSDB and ePSDB passed.

[arsenm]

LGTM with missing case added

llvm/test/CodeGen/AMDGPU/combine-sreg64-inits.mir

77

Also should check the case with the implicit operand on the moves

[rampitec]

Added tests with imp-use and imp-def.

llvm/lib/Target/AMDGPU/AMDGPU.h

 FunctionPass *createAMDGPULateCodeGenPreparePass();
 FunctionPass *createAMDGPUMachineCFGStructurizerPass();
 FunctionPass *createAMDGPUPropagateAttributesEarlyPass(const TargetMachine *);
 ModulePass *createAMDGPUPropagateAttributesLatePass(const TargetMachine *);
 FunctionPass *createAMDGPURewriteOutArgumentsPass();
 ModulePass *createAMDGPUReplaceLDSUseWithPointerPass();
 ModulePass *createAMDGPULowerModuleLDSPass();
 FunctionPass *createSIModeRegisterPass();
+FunctionPass *createGCNPreRAOptimizationsPass();
 
 struct AMDGPUSimplifyLibCallsPass : PassInfoMixin<AMDGPUSimplifyLibCallsPass> {
   AMDGPUSimplifyLibCallsPass(TargetMachine &TM) : TM(TM) {}
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 
 private:
   TargetMachine &TM;
 };

 
 ModulePass *createAMDGPUOpenCLEnqueuedBlockLoweringPass();
 void initializeAMDGPUOpenCLEnqueuedBlockLoweringPass(PassRegistry &);
 extern char &AMDGPUOpenCLEnqueuedBlockLoweringID;
 
 void initializeGCNNSAReassignPass(PassRegistry &);
 extern char &GCNNSAReassignID;
 
+void initializeGCNPreRAOptimizationsPass(PassRegistry &);
+extern char &GCNPreRAOptimizationsID;
+
 namespace AMDGPU {
 enum TargetIndex {
   TI_CONSTDATA_START,
   TI_SCRATCH_RSRC_DWORD0,
   TI_SCRATCH_RSRC_DWORD1,
   TI_SCRATCH_RSRC_DWORD2,
   TI_SCRATCH_RSRC_DWORD3
 };

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

static cl::opt<bool> EnableLDSReplaceWithPointer(
     cl::desc("Enable LDS replace with pointer pass"), cl::init(false),
     cl::Hidden);
 
 static cl::opt<bool, true> EnableLowerModuleLDS(
     "amdgpu-enable-lower-module-lds", cl::desc("Enable lower module lds pass"),
     cl::location(AMDGPUTargetMachine::EnableLowerModuleLDS), cl::init(true),
     cl::Hidden);
 
+static cl::opt<bool> EnablePreRAOptimizations(
+    "amdgpu-enable-pre-ra-optimizations",
+    cl::desc("Enable Pre-RA optimizations pass"), cl::init(true),
+    cl::Hidden);
+
 extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAMDGPUTarget() {
   // Register the target
   RegisterTargetMachine<R600TargetMachine> X(getTheAMDGPUTarget());
   RegisterTargetMachine<GCNTargetMachine> Y(getTheGCNTarget());
 
   PassRegistry *PR = PassRegistry::getPassRegistry();
   initializeR600ClauseMergePassPass(*PR);
   initializeR600ControlFlowFinalizerPass(*PR);
extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAMDGPUTarget() {
   initializeSIPostRABundlerPass(*PR);
   initializeAMDGPUUnifyDivergentExitNodesPass(*PR);
   initializeAMDGPUAAWrapperPassPass(*PR);
   initializeAMDGPUExternalAAWrapperPass(*PR);
   initializeAMDGPUUseNativeCallsPass(*PR);
   initializeAMDGPUSimplifyLibCallsPass(*PR);
   initializeAMDGPUPrintfRuntimeBindingPass(*PR);
   initializeGCNNSAReassignPass(*PR);
+  initializeGCNPreRAOptimizationsPass(*PR);
 }
 
 static std::unique_ptr<TargetLoweringObjectFile> createTLOF(const Triple &TT) {
   return std::make_unique<AMDGPUTargetObjectFile>();
 }
 
 static ScheduleDAGInstrs *createR600MachineScheduler(MachineSchedContext *C) {
   return new ScheduleDAGMILive(C, std::make_unique<R600SchedStrategy>());
void GCNPassConfig::addOptimizedRegAlloc() {
   // Allow the scheduler to run before SIWholeQuadMode inserts exec manipulation
   // instructions that cause scheduling barriers.
   insertPass(&MachineSchedulerID, &SIWholeQuadModeID);
   insertPass(&MachineSchedulerID, &SIPreAllocateWWMRegsID);
 
   if (OptExecMaskPreRA)
     insertPass(&MachineSchedulerID, &SIOptimizeExecMaskingPreRAID);
 
+  if (EnablePreRAOptimizations.getNumOccurrences()
+          ? EnablePreRAOptimizations
+          : TM->getOptLevel() > CodeGenOpt::Less)
+    insertPass(&RenameIndependentSubregsID, &GCNPreRAOptimizationsID);
+
   // This is not an essential optimization and it has a noticeable impact on
   // compilation time, so we only enable it from O2.
   if (TM->getOptLevel() > CodeGenOpt::Less)
     insertPass(&MachineSchedulerID, &SIFormMemoryClausesID);
 
   // FIXME: when an instruction has a Killed operand, and the instruction is
   // inside a bundle, seems only the BUNDLE instruction appears as the Kills of
   // the register in LiveVariables, this would trigger a failure in verifier,

llvm/lib/Target/AMDGPU/CMakeLists.txt

add_llvm_target(AMDGPUCodeGen
   SIPreEmitPeephole.cpp
   SIProgramInfo.cpp
   SIRegisterInfo.cpp
   SIShrinkInstructions.cpp
   SIWholeQuadMode.cpp
   GCNILPSched.cpp
   GCNNSAReassign.cpp
   GCNDPPCombine.cpp
+  GCNPreRAOptimizations.cpp
   SIModeRegister.cpp
 
   LINK_COMPONENTS
   Analysis
   AsmPrinter
   CodeGen
   Core
   IPO

llvm/lib/Target/AMDGPU/GCNPreRAOptimizations.cpp

+//===-- GCNPreRAOptimizations.cpp -----------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// This pass combines split register tuple initialization into a single psuedo:
+///
+///   undef %0.sub1:sreg_64 = S_MOV_B32 1
+///   %0.sub0:sreg_64 = S_MOV_B32 2
+/// =>
+///   %0:sreg_64 = S_MOV_B64_IMM_PSEUDO 0x200000001
+///
+/// This is to allow rematerialization of a value instead of spilling. It is
+/// supposed to be done after register coalescer to allow it to do its job and
+/// before actual register allocation to allow rematerialization.
+///
+/// Right now the pass only handles 64 bit SGPRs with immediate initializers,
+/// although the same shall be possible with other register classes and
+/// instructions if necessary.
+///
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPU.h"
+#include "GCNSubtarget.h"
+#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
+#include "llvm/CodeGen/LiveIntervals.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/InitializePasses.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "amdgpu-pre-ra-optimizations"
+
+namespace {
+
+class GCNPreRAOptimizations : public MachineFunctionPass {
+private:
+  const SIInstrInfo *TII;
+  MachineRegisterInfo *MRI;
+  LiveIntervals *LIS;
+
+  bool processReg(Register Reg);
+
+public:
+  static char ID;
+
+  GCNPreRAOptimizations() : MachineFunctionPass(ID) {
+    initializeGCNPreRAOptimizationsPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+  StringRef getPassName() const override {
+    return "AMDGPU Pre-RA optimizations";
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.addRequired<LiveIntervals>();
+    AU.setPreservesAll();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+};
+
+} // End anonymous namespace.
+
+INITIALIZE_PASS_BEGIN(GCNPreRAOptimizations, DEBUG_TYPE,
+                      "AMDGPU Pre-RA optimizations", false, false)
+INITIALIZE_PASS_DEPENDENCY(LiveIntervals)
+INITIALIZE_PASS_END(GCNPreRAOptimizations, DEBUG_TYPE, "Pre-RA optimizations",
+                    false, false)
+
+char GCNPreRAOptimizations::ID = 0;
+
+char &llvm::GCNPreRAOptimizationsID = GCNPreRAOptimizations::ID;
+
+FunctionPass *llvm::createGCNPreRAOptimizationsPass() {
+  return new GCNPreRAOptimizations();
+}
+
+bool GCNPreRAOptimizations::processReg(Register Reg) {
+  MachineInstr *Def0 = nullptr;
+  MachineInstr *Def1 = nullptr;
+  uint64_t Init = 0;
+
+  for (MachineInstr &I : MRI->def_instructions(Reg)) {
+    if (I.getOpcode() != AMDGPU::S_MOV_B32 || I.getOperand(0).getReg() != Reg ||
+        !I.getOperand(1).isImm() || I.getNumOperands() != 2)
+      return false;
+
+    switch (I.getOperand(0).getSubReg()) {
+    default:
+      return false;
+    case AMDGPU::sub0:
+      if (Def0)
+        return false;
+      Def0 = &I;
+      Init |= I.getOperand(1).getImm() & 0xffffffff;
+      break;
+    case AMDGPU::sub1:
+      if (Def1)
+        return false;
+      Def1 = &I;
+      Init |= static_cast<uint64_t>(I.getOperand(1).getImm()) << 32;
+      break;
+    }
+  }
+
+  if (!Def0 || !Def1 || Def0->getParent() != Def1->getParent())
+    return false;
+
+  LLVM_DEBUG(dbgs() << "Combining:\n  " << *Def0 << "  " << *Def1
+                    << "    =>\n");
+
+  if (SlotIndex::isEarlierInstr(LIS->getInstructionIndex(*Def1),
+                                LIS->getInstructionIndex(*Def0)))
+    std::swap(Def0, Def1);
+
+  LIS->RemoveMachineInstrFromMaps(*Def0);
+  LIS->RemoveMachineInstrFromMaps(*Def1);
+  auto NewI = BuildMI(*Def0->getParent(), *Def0, Def0->getDebugLoc(),
+                      TII->get(AMDGPU::S_MOV_B64_IMM_PSEUDO), Reg)
+                  .addImm(Init);
+
+  Def0->eraseFromParent();
+  Def1->eraseFromParent();
+  LIS->InsertMachineInstrInMaps(*NewI);
+  LIS->removeInterval(Reg);
+  LIS->createAndComputeVirtRegInterval(Reg);
+
+  LLVM_DEBUG(dbgs() << "  " << *NewI);
+
+  return true;
+}
+
+bool GCNPreRAOptimizations::runOnMachineFunction(MachineFunction &MF) {
+  if (skipFunction(MF.getFunction()))
+    return false;
+
+  const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
+  TII = ST.getInstrInfo();
+  MRI = &MF.getRegInfo();
+  LIS = &getAnalysis<LiveIntervals>();
+  const SIRegisterInfo *TRI = ST.getRegisterInfo();
+
+  bool Changed = false;
+
+  for (unsigned I = 0, E = MRI->getNumVirtRegs(); I != E; ++I) {
+    Register Reg = Register::index2VirtReg(I);
+    if (!LIS->hasInterval(Reg))
+      continue;
+    const TargetRegisterClass *RC = MRI->getRegClass(Reg);
+    if (RC->MC->getSizeInBits() != 64 || !TRI->isSGPRClass(RC))
+      continue;
+    Changed |= processReg(Reg);
+  }
+
+  return Changed;
+}

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

case AMDGPU::V_MOV_B64_PSEUDO: {
     }
     MI.eraseFromParent();
     break;
   }
   case AMDGPU::V_MOV_B64_DPP_PSEUDO: {
     expandMovDPP64(MI);
     break;
   }
+  case AMDGPU::S_MOV_B64_IMM_PSEUDO: {
+    const MachineOperand &SrcOp = MI.getOperand(1);
+    assert(!SrcOp.isFPImm());
+    APInt Imm(64, SrcOp.getImm());
+    if (Imm.isIntN(32) || isInlineConstant(Imm)) {
+      MI.setDesc(get(AMDGPU::S_MOV_B64));
+      break;
+    }
+
+    Register Dst = MI.getOperand(0).getReg();
+    Register DstLo = RI.getSubReg(Dst, AMDGPU::sub0);
+    Register DstHi = RI.getSubReg(Dst, AMDGPU::sub1);
+
+    APInt Lo(32, Imm.getLoBits(32).getZExtValue());
+    APInt Hi(32, Imm.getHiBits(32).getZExtValue());
+    BuildMI(MBB, MI, DL, get(AMDGPU::S_MOV_B32), DstLo)
+      .addImm(Lo.getSExtValue())
+      .addReg(Dst, RegState::Implicit | RegState::Define);
+    BuildMI(MBB, MI, DL, get(AMDGPU::S_MOV_B32), DstHi)
+      .addImm(Hi.getSExtValue())
+      .addReg(Dst, RegState::Implicit | RegState::Define);
+    MI.eraseFromParent();
+    break;
+  }
   case AMDGPU::V_SET_INACTIVE_B32: {
     unsigned NotOpc = ST.isWave32() ? AMDGPU::S_NOT_B32 : AMDGPU::S_NOT_B64;
     unsigned Exec = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
     auto FirstNot = BuildMI(MBB, MI, DL, get(NotOpc), Exec).addReg(Exec);
     FirstNot->addRegisterDead(AMDGPU::SCC, TRI); // SCC is overwritten
     BuildMI(MBB, MI, DL, get(AMDGPU::V_MOV_B32_e32), MI.getOperand(0).getReg())
       .add(MI.getOperand(2));
     BuildMI(MBB, MI, DL, get(NotOpc), Exec)
bool SIInstrInfo::isFoldableCopy(const MachineInstr &MI) const {
   case AMDGPU::S_MOV_B64:
   case AMDGPU::COPY:
   case AMDGPU::V_ACCVGPR_WRITE_B32_e64:
   case AMDGPU::V_ACCVGPR_READ_B32_e64:
   case AMDGPU::V_ACCVGPR_MOV_B32:
     return true;
   default:
     return false;
   }
 }
 
 unsigned SIInstrInfo::getAddressSpaceForPseudoSourceKind(
     unsigned Kind) const {
   switch(Kind) {
   case PseudoSourceValue::Stack:
   case PseudoSourceValue::FixedStack:
     return AMDGPUAS::PRIVATE_ADDRESS;

llvm/lib/Target/AMDGPU/SIInstructions.td

 def V_MOV_B64_PSEUDO : VPseudoInstSI <(outs VReg_64:$vdst),
                                       (ins VSrc_b64:$src0)>;
 
 // 64-bit vector move with dpp. Expanded post-RA.
 def V_MOV_B64_DPP_PSEUDO : VOP_DPP_Pseudo <"v_mov_b64_dpp", VOP_I64_I64> {
   let Size = 16; // Requires two 8-byte v_mov_b32_dpp to complete.
 }
 
+// 64-bit scalar move immediate instruction. This is used to avoid subregs
+// initialization and allow rematerialization.
+def S_MOV_B64_IMM_PSEUDO : SPseudoInstSI <(outs SReg_64:$sdst),
+                                          (ins i64imm:$src0)> {
+  let isReMaterializable = 1;
+  let isAsCheapAsAMove = 1;
+  let isMoveImm = 1;
+  let SchedRW = [WriteSALU, Write64Bit];
+  let Size = 16; // Needs maximum 2 s_mov_b32 instructions 8 byte long each.
+  let Uses = [];
+}
+
 // Pseudoinstruction for @llvm.amdgcn.wqm. It is turned into a copy after the
 // WQM pass processes it.
 def WQM : PseudoInstSI <(outs unknown:$vdst), (ins unknown:$src0)>;
 
 // Pseudoinstruction for @llvm.amdgcn.softwqm. Like @llvm.amdgcn.wqm it is
 // turned into a copy by WQM pass, but does not seed WQM requirements.
 def SOFT_WQM : PseudoInstSI <(outs unknown:$vdst), (ins unknown:$src0)>;
 

llvm/test/CodeGen/AMDGPU/GlobalISel/cvt_f32_ubyte.ll

 ; SI-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; SI-NEXT:    v_cndmask_b32_e32 v2, v3, v2, vcc
 ; SI-NEXT:    v_mov_b32_e32 v3, 0xbe
 ; SI-NEXT:    v_sub_i32_e32 v4, vcc, v3, v2
 ; SI-NEXT:    v_lshl_b64 v[2:3], v[0:1], v2
 ; SI-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; SI-NEXT:    v_and_b32_e32 v1, 0x7fffffff, v3
 ; SI-NEXT:    v_cndmask_b32_e32 v0, 0, v4, vcc
+; SI-NEXT:    s_mov_b32 s4, 0
 ; SI-NEXT:    v_and_b32_e32 v3, s6, v3
+; SI-NEXT:    s_movk_i32 s5, 0x80
 ; SI-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 23, v0
-; SI-NEXT:    s_mov_b32 s4, 0
-; SI-NEXT:    s_movk_i32 s5, 0x80
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
 ; SI-NEXT:    v_cmp_eq_u64_e32 vcc, s[4:5], v[2:3]
 ; SI-NEXT:    v_and_b32_e32 v1, 1, v0
 ; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
 ; SI-NEXT:    v_cmp_lt_u64_e32 vcc, s[4:5], v[2:3]
 ; SI-NEXT:    v_cndmask_b32_e64 v1, v1, 1, vcc
 ; SI-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; SI-NEXT:    s_setpc_b64 s[30:31]

 ; VI-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; VI-NEXT:    v_cndmask_b32_e32 v2, v3, v2, vcc
 ; VI-NEXT:    v_mov_b32_e32 v3, 0xbe
 ; VI-NEXT:    v_sub_u32_e32 v4, vcc, v3, v2
 ; VI-NEXT:    v_lshlrev_b64 v[2:3], v2, v[0:1]
 ; VI-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; VI-NEXT:    v_and_b32_e32 v1, 0x7fffffff, v3
 ; VI-NEXT:    v_cndmask_b32_e32 v0, 0, v4, vcc
+; VI-NEXT:    s_mov_b32 s4, 0
 ; VI-NEXT:    v_and_b32_e32 v3, s6, v3
+; VI-NEXT:    s_movk_i32 s5, 0x80
 ; VI-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
 ; VI-NEXT:    v_lshlrev_b32_e32 v0, 23, v0
-; VI-NEXT:    s_mov_b32 s4, 0
-; VI-NEXT:    s_movk_i32 s5, 0x80
 ; VI-NEXT:    v_or_b32_e32 v0, v0, v1
 ; VI-NEXT:    v_cmp_eq_u64_e32 vcc, s[4:5], v[2:3]
 ; VI-NEXT:    v_and_b32_e32 v1, 1, v0
 ; VI-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
 ; VI-NEXT:    v_cmp_lt_u64_e32 vcc, s[4:5], v[2:3]
 ; VI-NEXT:    v_cndmask_b32_e64 v1, v1, 1, vcc
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
 ; VI-NEXT:    s_setpc_b64 s[30:31]

 ; SI-NEXT:    v_cndmask_b32_e32 v2, v3, v2, vcc
 ; SI-NEXT:    v_mov_b32_e32 v1, 0
 ; SI-NEXT:    v_sub_i32_e32 v4, vcc, 0xbe, v2
 ; SI-NEXT:    v_lshl_b64 v[2:3], v[0:1], v2
 ; SI-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; SI-NEXT:    v_and_b32_e32 v1, 0x7fffffff, v3
 ; SI-NEXT:    v_cndmask_b32_e32 v0, 0, v4, vcc
 ; SI-NEXT:    v_and_b32_e32 v3, s4, v3
-; SI-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
-; SI-NEXT:    v_lshlrev_b32_e32 v0, 23, v0
 ; SI-NEXT:    s_mov_b32 s4, 0
 ; SI-NEXT:    s_movk_i32 s5, 0x80
+; SI-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
+; SI-NEXT:    v_lshlrev_b32_e32 v0, 23, v0
 ; SI-NEXT:    v_or_b32_e32 v0, v0, v1
 ; SI-NEXT:    v_cmp_eq_u64_e32 vcc, s[4:5], v[2:3]
 ; SI-NEXT:    v_and_b32_e32 v1, 1, v0
 ; SI-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
 ; SI-NEXT:    v_cmp_lt_u64_e32 vcc, s[4:5], v[2:3]
 ; SI-NEXT:    v_cndmask_b32_e64 v1, v1, 1, vcc
 ; SI-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; SI-NEXT:    s_setpc_b64 s[30:31]

 ; VI-NEXT:    v_cndmask_b32_e32 v2, v3, v2, vcc
 ; VI-NEXT:    v_mov_b32_e32 v1, 0
 ; VI-NEXT:    v_sub_u32_e32 v4, vcc, 0xbe, v2
 ; VI-NEXT:    v_lshlrev_b64 v[2:3], v2, v[0:1]
 ; VI-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; VI-NEXT:    v_and_b32_e32 v1, 0x7fffffff, v3
 ; VI-NEXT:    v_cndmask_b32_e32 v0, 0, v4, vcc
 ; VI-NEXT:    v_and_b32_e32 v3, s4, v3
-; VI-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
-; VI-NEXT:    v_lshlrev_b32_e32 v0, 23, v0
 ; VI-NEXT:    s_mov_b32 s4, 0
 ; VI-NEXT:    s_movk_i32 s5, 0x80
+; VI-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
+; VI-NEXT:    v_lshlrev_b32_e32 v0, 23, v0
 ; VI-NEXT:    v_or_b32_e32 v0, v0, v1
 ; VI-NEXT:    v_cmp_eq_u64_e32 vcc, s[4:5], v[2:3]
 ; VI-NEXT:    v_and_b32_e32 v1, 1, v0
 ; VI-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
 ; VI-NEXT:    v_cmp_lt_u64_e32 vcc, s[4:5], v[2:3]
 ; VI-NEXT:    v_cndmask_b32_e64 v1, v1, 1, vcc
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
 ; VI-NEXT:    s_setpc_b64 s[30:31]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll

 ; GPRIDX-NEXT:     wavefront_size = 6
 ; GPRIDX-NEXT:     call_convention = -1
 ; GPRIDX-NEXT:     runtime_loader_kernel_symbol = 0
 ; GPRIDX-NEXT:    .end_amd_kernel_code_t
 ; GPRIDX-NEXT:  ; %bb.0: ; %entry
 ; GPRIDX-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x0
 ; GPRIDX-NEXT:    s_load_dword s8, s[4:5], 0x8
 ; GPRIDX-NEXT:    s_mov_b32 s0, 0
+; GPRIDX-NEXT:    s_mov_b32 s1, 0x40140000
 ; GPRIDX-NEXT:    s_mov_b32 s3, 0x40080000
 ; GPRIDX-NEXT:    s_mov_b32 s2, s0
-; GPRIDX-NEXT:    s_mov_b32 s1, 0x40140000
 ; GPRIDX-NEXT:    s_waitcnt lgkmcnt(0)
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s8, 1
 ; GPRIDX-NEXT:    s_cselect_b64 s[4:5], 2.0, 1.0
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s8, 2
 ; GPRIDX-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[4:5]
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s8, 3
 ; GPRIDX-NEXT:    s_cselect_b64 s[2:3], 4.0, s[2:3]
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s8, 4

 ; MOVREL-NEXT:     wavefront_size = 6
 ; MOVREL-NEXT:     call_convention = -1
 ; MOVREL-NEXT:     runtime_loader_kernel_symbol = 0
 ; MOVREL-NEXT:    .end_amd_kernel_code_t
 ; MOVREL-NEXT:  ; %bb.0: ; %entry
 ; MOVREL-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x0
 ; MOVREL-NEXT:    s_load_dword s8, s[4:5], 0x8
 ; MOVREL-NEXT:    s_mov_b32 s0, 0
+; MOVREL-NEXT:    s_mov_b32 s1, 0x40140000
 ; MOVREL-NEXT:    s_mov_b32 s3, 0x40080000
 ; MOVREL-NEXT:    s_mov_b32 s2, s0
-; MOVREL-NEXT:    s_mov_b32 s1, 0x40140000
 ; MOVREL-NEXT:    s_waitcnt lgkmcnt(0)
 ; MOVREL-NEXT:    s_cmp_eq_u32 s8, 1
 ; MOVREL-NEXT:    s_cselect_b64 s[4:5], 2.0, 1.0
 ; MOVREL-NEXT:    s_cmp_eq_u32 s8, 2
 ; MOVREL-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[4:5]
 ; MOVREL-NEXT:    s_cmp_eq_u32 s8, 3
 ; MOVREL-NEXT:    s_cselect_b64 s[2:3], 4.0, s[2:3]
 ; MOVREL-NEXT:    s_cmp_eq_u32 s8, 4

 ; GFX10-NEXT:     call_convention = -1
 ; GFX10-NEXT:     runtime_loader_kernel_symbol = 0
 ; GFX10-NEXT:    .end_amd_kernel_code_t
 ; GFX10-NEXT:  ; %bb.0: ; %entry
 ; GFX10-NEXT:    s_clause 0x1
 ; GFX10-NEXT:    s_load_dword s8, s[4:5], 0x8
 ; GFX10-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
 ; GFX10-NEXT:    s_mov_b32 s2, 0
+; GFX10-NEXT:    s_mov_b32 s3, 0x40140000
 ; GFX10-NEXT:    s_mov_b32 s5, 0x40080000
 ; GFX10-NEXT:    s_mov_b32 s4, s2
-; GFX10-NEXT:    s_mov_b32 s3, 0x40140000
 ; GFX10-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-NEXT:    s_cmp_eq_u32 s8, 1
 ; GFX10-NEXT:    s_cselect_b64 s[6:7], 2.0, 1.0
 ; GFX10-NEXT:    s_cmp_eq_u32 s8, 2
 ; GFX10-NEXT:    s_cselect_b64 s[4:5], s[4:5], s[6:7]
 ; GFX10-NEXT:    s_cmp_eq_u32 s8, 3
 ; GFX10-NEXT:    s_cselect_b64 s[4:5], 4.0, s[4:5]

 ; GPRIDX-NEXT:     kernarg_segment_alignment = 4
 ; GPRIDX-NEXT:     group_segment_alignment = 4
 ; GPRIDX-NEXT:     private_segment_alignment = 4
 ; GPRIDX-NEXT:     wavefront_size = 6
 ; GPRIDX-NEXT:     call_convention = -1
 ; GPRIDX-NEXT:     runtime_loader_kernel_symbol = 0
 ; GPRIDX-NEXT:    .end_amd_kernel_code_t
 ; GPRIDX-NEXT:  ; %bb.0: ; %entry
-; GPRIDX-NEXT:    s_load_dwordx2 s[2:3], s[4:5], 0x0
+; GPRIDX-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
 ; GPRIDX-NEXT:    s_load_dword s6, s[4:5], 0x8
-; GPRIDX-NEXT:    s_mov_b32 s0, 0
-; GPRIDX-NEXT:    s_mov_b32 s1, 0x40080000
+; GPRIDX-NEXT:    s_mov_b32 s2, 0
+; GPRIDX-NEXT:    s_mov_b32 s3, 0x40080000
 ; GPRIDX-NEXT:    v_mov_b32_e32 v2, 0
 ; GPRIDX-NEXT:    s_waitcnt lgkmcnt(0)
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s6, 1
 ; GPRIDX-NEXT:    s_cselect_b64 s[4:5], 2.0, 1.0
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s6, 2
-; GPRIDX-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[4:5]
+; GPRIDX-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[4:5]
 ; GPRIDX-NEXT:    s_cmp_eq_u32 s6, 3
-; GPRIDX-NEXT:    s_cselect_b64 s[0:1], 4.0, s[0:1]
-; GPRIDX-NEXT:    v_mov_b32_e32 v0, s0
-; GPRIDX-NEXT:    v_mov_b32_e32 v1, s1
-; GPRIDX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GPRIDX-NEXT:    s_cselect_b64 s[2:3], 4.0, s[2:3]
+; GPRIDX-NEXT:    v_mov_b32_e32 v0, s2
+; GPRIDX-NEXT:    v_mov_b32_e32 v1, s3
+; GPRIDX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
 ; GPRIDX-NEXT:    s_endpgm
 ;
 ; MOVREL-LABEL: dyn_extract_v4f64_s_s_s:
 ; MOVREL:         .amd_kernel_code_t
 ; MOVREL-NEXT:     amd_code_version_major = 1
 ; MOVREL-NEXT:     amd_code_version_minor = 2
 ; MOVREL-NEXT:     amd_machine_kind = 1
 ; MOVREL-NEXT:     amd_machine_version_major = 8

 ; MOVREL-NEXT:     kernarg_segment_alignment = 4
 ; MOVREL-NEXT:     group_segment_alignment = 4
 ; MOVREL-NEXT:     private_segment_alignment = 4
 ; MOVREL-NEXT:     wavefront_size = 6
 ; MOVREL-NEXT:     call_convention = -1
 ; MOVREL-NEXT:     runtime_loader_kernel_symbol = 0
 ; MOVREL-NEXT:    .end_amd_kernel_code_t
 ; MOVREL-NEXT:  ; %bb.0: ; %entry
-; MOVREL-NEXT:    s_load_dwordx2 s[2:3], s[4:5], 0x0
+; MOVREL-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
 ; MOVREL-NEXT:    s_load_dword s6, s[4:5], 0x8
-; MOVREL-NEXT:    s_mov_b32 s0, 0
-; MOVREL-NEXT:    s_mov_b32 s1, 0x40080000
+; MOVREL-NEXT:    s_mov_b32 s2, 0
+; MOVREL-NEXT:    s_mov_b32 s3, 0x40080000
 ; MOVREL-NEXT:    s_waitcnt lgkmcnt(0)
-; MOVREL-NEXT:    v_mov_b32_e32 v2, s2
+; MOVREL-NEXT:    v_mov_b32_e32 v3, s1
 ; MOVREL-NEXT:    s_cmp_eq_u32 s6, 1
 ; MOVREL-NEXT:    s_cselect_b64 s[4:5], 2.0, 1.0
 ; MOVREL-NEXT:    s_cmp_eq_u32 s6, 2
-; MOVREL-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[4:5]
+; MOVREL-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[4:5]
 ; MOVREL-NEXT:    s_cmp_eq_u32 s6, 3
-; MOVREL-NEXT:    s_cselect_b64 s[0:1], 4.0, s[0:1]
-; MOVREL-NEXT:    v_mov_b32_e32 v0, s0
-; MOVREL-NEXT:    v_mov_b32_e32 v1, s1
-; MOVREL-NEXT:    v_mov_b32_e32 v3, s3
+; MOVREL-NEXT:    s_cselect_b64 s[2:3], 4.0, s[2:3]
+; MOVREL-NEXT:    v_mov_b32_e32 v0, s2
+; MOVREL-NEXT:    v_mov_b32_e32 v1, s3
+; MOVREL-NEXT:    v_mov_b32_e32 v2, s0
 ; MOVREL-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
 ; MOVREL-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: dyn_extract_v4f64_s_s_s:
 ; GFX10:         .amd_kernel_code_t
 ; GFX10-NEXT:     amd_code_version_major = 1
 ; GFX10-NEXT:     amd_code_version_minor = 2
 ; GFX10-NEXT:     amd_machine_kind = 1

 ; GPRIDX-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GPRIDX-NEXT:    global_load_dwordx4 v[0:3], v[0:1], off offset:128
 ; GPRIDX-NEXT:    s_waitcnt vmcnt(0)
 ; GPRIDX-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; MOVREL-LABEL: v_extract_v64i32_32:
 ; MOVREL:       ; %bb.0:
 ; MOVREL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; MOVREL-NEXT:    s_movk_i32 s4, 0x80
-; MOVREL-NEXT:    s_mov_b32 s5, 0
+; MOVREL-NEXT:    s_mov_b64 s[4:5], 0x80
 ; MOVREL-NEXT:    v_mov_b32_e32 v2, s4
 ; MOVREL-NEXT:    v_mov_b32_e32 v3, s5
 ; MOVREL-NEXT:    v_add_u32_e32 v0, vcc, v0, v2
 ; MOVREL-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; MOVREL-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
 ; MOVREL-NEXT:    s_waitcnt vmcnt(0)
 ; MOVREL-NEXT:    s_setpc_b64 s[30:31]
 ;

 ; GPRIDX-NEXT:    global_load_dwordx4 v[0:3], v[0:1], off offset:128
 ; GPRIDX-NEXT:    s_waitcnt vmcnt(0)
 ; GPRIDX-NEXT:    v_mov_b32_e32 v0, v1
 ; GPRIDX-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; MOVREL-LABEL: v_extract_v64i32_33:
 ; MOVREL:       ; %bb.0:
 ; MOVREL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; MOVREL-NEXT:    s_movk_i32 s4, 0x80
-; MOVREL-NEXT:    s_mov_b32 s5, 0
+; MOVREL-NEXT:    s_mov_b64 s[4:5], 0x80
 ; MOVREL-NEXT:    v_mov_b32_e32 v2, s4
 ; MOVREL-NEXT:    v_mov_b32_e32 v3, s5
 ; MOVREL-NEXT:    v_add_u32_e32 v0, vcc, v0, v2
 ; MOVREL-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; MOVREL-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
 ; MOVREL-NEXT:    s_waitcnt vmcnt(0)
 ; MOVREL-NEXT:    v_mov_b32_e32 v0, v1
 ; MOVREL-NEXT:    s_setpc_b64 s[30:31]
; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %elt = extractelement <64 x i32> %vec, i32 33
   ret i32 %elt
 }
 
 define i32 @v_extract_v64i32_37(<64 x i32> addrspace(1)* %ptr) {
 ; GPRIDX-LABEL: v_extract_v64i32_37:
 ; GPRIDX:       ; %bb.0:
 ; GPRIDX-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GPRIDX-NEXT:    s_movk_i32 s4, 0x80
-; GPRIDX-NEXT:    s_mov_b32 s5, 0
+; GPRIDX-NEXT:    s_mov_b64 s[4:5], 0x80
 ; GPRIDX-NEXT:    v_mov_b32_e32 v2, s4
 ; GPRIDX-NEXT:    v_mov_b32_e32 v3, s5
 ; GPRIDX-NEXT:    v_add_co_u32_e32 v0, vcc, v0, v2
 ; GPRIDX-NEXT:    v_addc_co_u32_e32 v1, vcc, v1, v3, vcc
 ; GPRIDX-NEXT:    global_load_dwordx4 v[4:7], v[0:1], off offset:16
 ; GPRIDX-NEXT:    s_waitcnt vmcnt(0)
 ; GPRIDX-NEXT:    v_mov_b32_e32 v0, v5
 ; GPRIDX-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; MOVREL-LABEL: v_extract_v64i32_37:
 ; MOVREL:       ; %bb.0:
 ; MOVREL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; MOVREL-NEXT:    s_movk_i32 s4, 0x80
-; MOVREL-NEXT:    s_mov_b32 s5, 0
+; MOVREL-NEXT:    s_mov_b64 s[4:5], 0x80
 ; MOVREL-NEXT:    v_mov_b32_e32 v2, s4
 ; MOVREL-NEXT:    v_mov_b32_e32 v3, s5
 ; MOVREL-NEXT:    v_add_u32_e32 v0, vcc, v0, v2
 ; MOVREL-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; MOVREL-NEXT:    v_add_u32_e32 v0, vcc, 16, v0
 ; MOVREL-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; MOVREL-NEXT:    flat_load_dwordx4 v[4:7], v[0:1]
 ; MOVREL-NEXT:    s_waitcnt vmcnt(0)
 ; MOVREL-NEXT:    v_mov_b32_e32 v0, v5
 ; MOVREL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX10-LABEL: v_extract_v64i32_37:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
-; GFX10-NEXT:    s_movk_i32 s4, 0x80
-; GFX10-NEXT:    s_mov_b32 s5, 0
+; GFX10-NEXT:    s_mov_b64 s[4:5], 0x80
 ; GFX10-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX10-NEXT:    v_mov_b32_e32 v3, s5
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v2
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v3, vcc_lo
 ; GFX10-NEXT:    global_load_dwordx4 v[4:7], v[0:1], off offset:16
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_mov_b32_e32 v0, v5
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %vec = load <64 x i32>, <64 x i32> addrspace(1)* %ptr
   %elt = extractelement <64 x i32> %vec, i32 37
   ret i32 %elt
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/floor.f64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=tahiti < %s | FileCheck -check-prefix=GFX6 %s
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=bonaire < %s | FileCheck -check-prefix=GFX78 %s
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=fiji < %s | FileCheck -check-prefix=GFX78 %s
 
 define double @v_floor_f64_ieee(double %x) {
 ; GFX6-LABEL: v_floor_f64_ieee:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX6-NEXT:    v_fract_f64_e32 v[2:3], v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_fract_f64_e32 v[2:3], v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s5, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[2:3], v[2:3], s[4:5]
 ; GFX6-NEXT:    v_cmp_o_f64_e32 vcc, v[0:1], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, v2, v0, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v1, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], v[0:1], -v[2:3]
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX78-LABEL: v_floor_f64_ieee:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX78-NEXT:    v_floor_f64_e32 v[0:1], v[0:1]
 ; GFX78-NEXT:    s_setpc_b64 s[30:31]
   %result = call double @llvm.floor.f64(double %x)
   ret double %result
 }
 
 define double @v_floor_f64_ieee_nnan(double %x) {
 ; GFX6-LABEL: v_floor_f64_ieee_nnan:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX6-NEXT:    v_fract_f64_e32 v[2:3], v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_fract_f64_e32 v[2:3], v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s5, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[2:3], v[2:3], s[4:5]
 ; GFX6-NEXT:    v_add_f64 v[0:1], v[0:1], -v[2:3]
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX78-LABEL: v_floor_f64_ieee_nnan:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX78-NEXT:    v_floor_f64_e32 v[0:1], v[0:1]
 ; GFX78-NEXT:    s_setpc_b64 s[30:31]
   %result = call nnan double @llvm.floor.f64(double %x)
   ret double %result
 }
 
 define double @v_floor_f64_ieee_fneg(double %x) {
 ; GFX6-LABEL: v_floor_f64_ieee_fneg:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX6-NEXT:    v_fract_f64_e64 v[2:3], -v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_fract_f64_e64 v[2:3], -v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s5, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[2:3], v[2:3], s[4:5]
 ; GFX6-NEXT:    v_cmp_o_f64_e32 vcc, v[0:1], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, v2, v0, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v1, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], -v[0:1], -v[2:3]
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX78-LABEL: v_floor_f64_ieee_fneg:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX78-NEXT:    v_floor_f64_e64 v[0:1], -v[0:1]
 ; GFX78-NEXT:    s_setpc_b64 s[30:31]
   %neg.x = fneg double %x
   %result = call double @llvm.floor.f64(double %neg.x)
   ret double %result
 }
 
 define double @v_floor_f64_nonieee(double %x) #1 {
 ; GFX6-LABEL: v_floor_f64_nonieee:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX6-NEXT:    v_fract_f64_e32 v[2:3], v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_fract_f64_e32 v[2:3], v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s5, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[2:3], v[2:3], s[4:5]
 ; GFX6-NEXT:    v_cmp_o_f64_e32 vcc, v[0:1], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, v2, v0, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v1, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], v[0:1], -v[2:3]
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX78-LABEL: v_floor_f64_nonieee:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX78-NEXT:    v_floor_f64_e32 v[0:1], v[0:1]
 ; GFX78-NEXT:    s_setpc_b64 s[30:31]
   %result = call double @llvm.floor.f64(double %x)
   ret double %result
 }
 
 define double @v_floor_f64_nonieee_nnan(double %x) #1 {
 ; GFX6-LABEL: v_floor_f64_nonieee_nnan:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX6-NEXT:    v_fract_f64_e32 v[2:3], v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_fract_f64_e32 v[2:3], v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s5, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[2:3], v[2:3], s[4:5]
 ; GFX6-NEXT:    v_add_f64 v[0:1], v[0:1], -v[2:3]
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX78-LABEL: v_floor_f64_nonieee_nnan:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX78-NEXT:    v_floor_f64_e32 v[0:1], v[0:1]
 ; GFX78-NEXT:    s_setpc_b64 s[30:31]
   %result = call nnan double @llvm.floor.f64(double %x)
   ret double %result
 }
 
 define double @v_floor_f64_non_ieee_fneg(double %x) #1 {
 ; GFX6-LABEL: v_floor_f64_non_ieee_fneg:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX6-NEXT:    v_fract_f64_e64 v[2:3], -v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_fract_f64_e64 v[2:3], -v[0:1]
 ; GFX6-NEXT:    s_mov_b32 s5, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[2:3], v[2:3], s[4:5]
 ; GFX6-NEXT:    v_cmp_o_f64_e32 vcc, v[0:1], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, v2, v0, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v1, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], -v[0:1], -v[2:3]
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX78-LABEL: v_floor_f64_non_ieee_fneg:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX78-NEXT:    v_floor_f64_e64 v[0:1], -v[0:1]
 ; GFX78-NEXT:    s_setpc_b64 s[30:31]
   %neg.x = fneg double %x
   %result = call double @llvm.floor.f64(double %neg.x)
   ret double %result
 }
 
 define double @v_floor_f64_fabs(double %x) {
 ; GFX6-LABEL: v_floor_f64_fabs:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX6-NEXT:    v_fract_f64_e64 v[2:3], |v[0:1]|
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_fract_f64_e64 v[2:3], |v[0:1]|
 ; GFX6-NEXT:    s_mov_b32 s5, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[2:3], v[2:3], s[4:5]
 ; GFX6-NEXT:    v_cmp_o_f64_e32 vcc, v[0:1], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, v2, v0, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v1, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], |v[0:1]|, -v[2:3]
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
; GFX7-NEXT:    s_setpc_b64 s[30:31]
   %result = call double @llvm.floor.f64(double %abs.x)
   ret double %result
 }
 
 define double @v_floor_f64_fneg_fabs(double %x) {
 ; GFX6-LABEL: v_floor_f64_fneg_fabs:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX6-NEXT:    v_fract_f64_e64 v[2:3], -|v[0:1]|
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_fract_f64_e64 v[2:3], -|v[0:1]|
 ; GFX6-NEXT:    s_mov_b32 s5, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[2:3], v[2:3], s[4:5]
 ; GFX6-NEXT:    v_cmp_o_f64_e32 vcc, v[0:1], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, v2, v0, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v1, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], -|v[0:1]|, -v[2:3]
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX78-LABEL: v_floor_f64_fneg_fabs:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX78-NEXT:    v_floor_f64_e64 v[0:1], -|v[0:1]|
 ; GFX78-NEXT:    s_setpc_b64 s[30:31]
   %abs.x = call double @llvm.fabs.f64(double %x)
   %neg.abs.x = fneg double %abs.x
   %result = call double @llvm.floor.f64(double %neg.abs.x)
   ret double %result
 }
 
 define amdgpu_ps <2 x float> @s_floor_f64(double inreg %x) {
 ; GFX6-LABEL: s_floor_f64:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    v_fract_f64_e32 v[0:1], s[2:3]
 ; GFX6-NEXT:    s_mov_b32 s0, -1
+; GFX6-NEXT:    v_fract_f64_e32 v[0:1], s[2:3]
 ; GFX6-NEXT:    s_mov_b32 s1, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[0:1], v[0:1], s[0:1]
 ; GFX6-NEXT:    v_cmp_o_f64_e64 vcc, s[2:3], s[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], s[2:3], -v[0:1]
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX78-LABEL: s_floor_f64:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    v_floor_f64_e32 v[0:1], s[2:3]
 ; GFX78-NEXT:    ; return to shader part epilog
   %result = call double @llvm.floor.f64(double %x)
   %cast = bitcast double %result to <2 x float>
   ret <2 x float> %cast
 }
 
 define amdgpu_ps <2 x float> @s_floor_f64_fneg(double inreg %x) {
 ; GFX6-LABEL: s_floor_f64_fneg:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    v_fract_f64_e64 v[0:1], -s[2:3]
 ; GFX6-NEXT:    s_mov_b32 s0, -1
+; GFX6-NEXT:    v_fract_f64_e64 v[0:1], -s[2:3]
 ; GFX6-NEXT:    s_mov_b32 s1, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[0:1], v[0:1], s[0:1]
 ; GFX6-NEXT:    v_cmp_o_f64_e64 vcc, s[2:3], s[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], -s[2:3], -v[0:1]
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX78-LABEL: s_floor_f64_fneg:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    v_floor_f64_e64 v[0:1], -s[2:3]
 ; GFX78-NEXT:    ; return to shader part epilog
   %neg.x = fneg double %x
   %result = call double @llvm.floor.f64(double %neg.x)
   %cast = bitcast double %result to <2 x float>
   ret <2 x float> %cast
 }
 
 define amdgpu_ps <2 x float> @s_floor_f64_fabs(double inreg %x) {
 ; GFX6-LABEL: s_floor_f64_fabs:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    v_fract_f64_e64 v[0:1], |s[2:3]|
 ; GFX6-NEXT:    s_mov_b32 s0, -1
+; GFX6-NEXT:    v_fract_f64_e64 v[0:1], |s[2:3]|
 ; GFX6-NEXT:    s_mov_b32 s1, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[0:1], v[0:1], s[0:1]
 ; GFX6-NEXT:    v_cmp_o_f64_e64 vcc, s[2:3], s[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], |s[2:3]|, -v[0:1]
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX78-LABEL: s_floor_f64_fabs:
 ; GFX78:       ; %bb.0:
 ; GFX78-NEXT:    v_floor_f64_e64 v[0:1], |s[2:3]|
 ; GFX78-NEXT:    ; return to shader part epilog
   %abs.x = call double @llvm.fabs.f64(double %x)
   %result = call double @llvm.floor.f64(double %abs.x)
   %cast = bitcast double %result to <2 x float>
   ret <2 x float> %cast
 }
 
 define amdgpu_ps <2 x float> @s_floor_f64_fneg_fabs(double inreg %x) {
 ; GFX6-LABEL: s_floor_f64_fneg_fabs:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    v_fract_f64_e64 v[0:1], -|s[2:3]|
 ; GFX6-NEXT:    s_mov_b32 s0, -1
+; GFX6-NEXT:    v_fract_f64_e64 v[0:1], -|s[2:3]|
 ; GFX6-NEXT:    s_mov_b32 s1, 0x3fefffff
 ; GFX6-NEXT:    v_min_f64 v[0:1], v[0:1], s[0:1]
 ; GFX6-NEXT:    v_cmp_o_f64_e64 vcc, s[2:3], s[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; GFX6-NEXT:    v_add_f64 v[0:1], -|s[2:3]|, -v[0:1]

llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = call <2 x i64> @llvm.fshl.v2i64(<2 x i64> %lhs, <2 x i64> %rhs, <2 x i64> %amt)
   ret <2 x i64> %result
 }
 
 define amdgpu_ps i128 @s_fshl_i128(i128 inreg %lhs, i128 inreg %rhs, i128 inreg %amt) {
 ; GFX6-LABEL: s_fshl_i128:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s10, 0x7f
-; GFX6-NEXT:    s_mov_b32 s11, 0
+; GFX6-NEXT:    s_mov_b64 s[10:11], 0x7f
 ; GFX6-NEXT:    s_and_b64 s[12:13], s[8:9], s[10:11]
 ; GFX6-NEXT:    s_andn2_b64 s[8:9], s[10:11], s[8:9]
 ; GFX6-NEXT:    s_sub_i32 s9, s12, 64
 ; GFX6-NEXT:    s_sub_i32 s13, 64, s12
 ; GFX6-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX6-NEXT:    s_cselect_b32 s16, 1, 0
 ; GFX6-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX6-NEXT:    s_cselect_b32 s17, 1, 0

 ; GFX6-NEXT:    s_cmp_lg_u32 s15, 0
 ; GFX6-NEXT:    s_cselect_b64 s[4:5], s[6:7], 0
 ; GFX6-NEXT:    s_or_b64 s[0:1], s[10:11], s[0:1]
 ; GFX6-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_fshl_i128:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s10, 0x7f
-; GFX8-NEXT:    s_mov_b32 s11, 0
+; GFX8-NEXT:    s_mov_b64 s[10:11], 0x7f
 ; GFX8-NEXT:    s_and_b64 s[12:13], s[8:9], s[10:11]
 ; GFX8-NEXT:    s_andn2_b64 s[8:9], s[10:11], s[8:9]
 ; GFX8-NEXT:    s_sub_i32 s9, s12, 64
 ; GFX8-NEXT:    s_sub_i32 s13, 64, s12
 ; GFX8-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX8-NEXT:    s_cselect_b32 s16, 1, 0
 ; GFX8-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX8-NEXT:    s_cselect_b32 s17, 1, 0

 ; GFX8-NEXT:    s_cmp_lg_u32 s15, 0
 ; GFX8-NEXT:    s_cselect_b64 s[4:5], s[6:7], 0
 ; GFX8-NEXT:    s_or_b64 s[0:1], s[10:11], s[0:1]
 ; GFX8-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_fshl_i128:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s10, 0x7f
-; GFX9-NEXT:    s_mov_b32 s11, 0
+; GFX9-NEXT:    s_mov_b64 s[10:11], 0x7f
 ; GFX9-NEXT:    s_and_b64 s[12:13], s[8:9], s[10:11]
 ; GFX9-NEXT:    s_andn2_b64 s[8:9], s[10:11], s[8:9]
 ; GFX9-NEXT:    s_sub_i32 s9, s12, 64
 ; GFX9-NEXT:    s_sub_i32 s13, 64, s12
 ; GFX9-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX9-NEXT:    s_cselect_b32 s16, 1, 0
 ; GFX9-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX9-NEXT:    s_cselect_b32 s17, 1, 0

 ; GFX9-NEXT:    s_cmp_lg_u32 s15, 0
 ; GFX9-NEXT:    s_cselect_b64 s[4:5], s[6:7], 0
 ; GFX9-NEXT:    s_or_b64 s[0:1], s[10:11], s[0:1]
 ; GFX9-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_fshl_i128:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_movk_i32 s10, 0x7f
-; GFX10-NEXT:    s_mov_b32 s11, 0
+; GFX10-NEXT:    s_mov_b64 s[10:11], 0x7f
 ; GFX10-NEXT:    s_and_b64 s[12:13], s[8:9], s[10:11]
 ; GFX10-NEXT:    s_andn2_b64 s[8:9], s[10:11], s[8:9]
 ; GFX10-NEXT:    s_sub_i32 s9, s12, 64
 ; GFX10-NEXT:    s_sub_i32 s10, 64, s12
 ; GFX10-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX10-NEXT:    s_cselect_b32 s16, 1, 0
 ; GFX10-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX10-NEXT:    s_cselect_b32 s17, 1, 0
; GFX10-NEXT:    ; return to shader part epilog
   %result = call i128 @llvm.fshl.i128(i128 %lhs, i128 %rhs, i128 %amt)
   %cast.result = bitcast i128 %result to <4 x float>
   ret <4 x float> %cast.result
 }
 
 define amdgpu_ps <4 x float> @v_fshl_i128_svs(i128 inreg %lhs, i128 %rhs, i128 inreg %amt) {
 ; GFX6-LABEL: v_fshl_i128_svs:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s6, 0x7f
-; GFX6-NEXT:    s_mov_b32 s7, 0
+; GFX6-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX6-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX6-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX6-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX6-NEXT:    s_sub_i32 s9, 64, s8
 ; GFX6-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX6-NEXT:    s_cselect_b32 s12, 1, 0
 ; GFX6-NEXT:    s_cmp_eq_u32 s8, 0
 ; GFX6-NEXT:    s_cselect_b32 s13, 1, 0

 ; GFX6-NEXT:    v_or_b32_e32 v0, s6, v0
 ; GFX6-NEXT:    v_or_b32_e32 v1, s7, v1
 ; GFX6-NEXT:    v_or_b32_e32 v2, s0, v2
 ; GFX6-NEXT:    v_or_b32_e32 v3, s1, v3
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: v_fshl_i128_svs:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s6, 0x7f
-; GFX8-NEXT:    s_mov_b32 s7, 0
+; GFX8-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX8-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX8-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX8-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX8-NEXT:    s_sub_i32 s9, 64, s8
 ; GFX8-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX8-NEXT:    s_cselect_b32 s12, 1, 0
 ; GFX8-NEXT:    s_cmp_eq_u32 s8, 0
 ; GFX8-NEXT:    s_cselect_b32 s13, 1, 0

 ; GFX8-NEXT:    v_or_b32_e32 v0, s6, v0
 ; GFX8-NEXT:    v_or_b32_e32 v1, s7, v1
 ; GFX8-NEXT:    v_or_b32_e32 v2, s0, v2
 ; GFX8-NEXT:    v_or_b32_e32 v3, s1, v3
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshl_i128_svs:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s6, 0x7f
-; GFX9-NEXT:    s_mov_b32 s7, 0
+; GFX9-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX9-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX9-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX9-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX9-NEXT:    s_sub_i32 s9, 64, s8
 ; GFX9-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX9-NEXT:    s_cselect_b32 s12, 1, 0
 ; GFX9-NEXT:    s_cmp_eq_u32 s8, 0
 ; GFX9-NEXT:    s_cselect_b32 s13, 1, 0

 ; GFX9-NEXT:    v_or_b32_e32 v0, s6, v0
 ; GFX9-NEXT:    v_or_b32_e32 v1, s7, v1
 ; GFX9-NEXT:    v_or_b32_e32 v2, s0, v2
 ; GFX9-NEXT:    v_or_b32_e32 v3, s1, v3
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshl_i128_svs:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_movk_i32 s6, 0x7f
-; GFX10-NEXT:    s_mov_b32 s7, 0
+; GFX10-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX10-NEXT:    v_lshrrev_b64 v[0:1], 1, v[0:1]
 ; GFX10-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX10-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX10-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX10-NEXT:    s_sub_i32 s6, 64, s8
 ; GFX10-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX10-NEXT:    s_cselect_b32 s12, 1, 0
 ; GFX10-NEXT:    s_cmp_eq_u32 s8, 0
; GFX10-NEXT:    ; return to shader part epilog
   %result = call i128 @llvm.fshl.i128(i128 %lhs, i128 %rhs, i128 %amt)
   %cast.result = bitcast i128 %result to <4 x float>
   ret <4 x float> %cast.result
 }
 
 define amdgpu_ps <4 x float> @v_fshl_i128_vss(i128 %lhs, i128 inreg %rhs, i128 inreg %amt) {
 ; GFX6-LABEL: v_fshl_i128_vss:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s6, 0x7f
-; GFX6-NEXT:    s_mov_b32 s7, 0
+; GFX6-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX6-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX6-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX6-NEXT:    s_sub_i32 s6, 64, s8
 ; GFX6-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX6-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX6-NEXT:    s_cselect_b32 s7, 1, 0
 ; GFX6-NEXT:    s_cmp_eq_u32 s8, 0
 ; GFX6-NEXT:    v_lshr_b64 v[4:5], v[0:1], s6

 ; GFX6-NEXT:    v_or_b32_e32 v0, s0, v6
 ; GFX6-NEXT:    v_or_b32_e32 v1, s1, v7
 ; GFX6-NEXT:    v_or_b32_e32 v2, s2, v2
 ; GFX6-NEXT:    v_or_b32_e32 v3, s3, v3
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: v_fshl_i128_vss:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s6, 0x7f
-; GFX8-NEXT:    s_mov_b32 s7, 0
+; GFX8-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX8-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX8-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX8-NEXT:    s_sub_i32 s6, 64, s8
 ; GFX8-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX8-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX8-NEXT:    s_cselect_b32 s7, 1, 0
 ; GFX8-NEXT:    s_cmp_eq_u32 s8, 0
 ; GFX8-NEXT:    v_lshrrev_b64 v[4:5], s6, v[0:1]

 ; GFX8-NEXT:    v_or_b32_e32 v0, s0, v6
 ; GFX8-NEXT:    v_or_b32_e32 v1, s1, v7
 ; GFX8-NEXT:    v_or_b32_e32 v2, s2, v2
 ; GFX8-NEXT:    v_or_b32_e32 v3, s3, v3
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshl_i128_vss:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s6, 0x7f
-; GFX9-NEXT:    s_mov_b32 s7, 0
+; GFX9-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX9-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX9-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX9-NEXT:    s_sub_i32 s6, 64, s8
 ; GFX9-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX9-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX9-NEXT:    s_cselect_b32 s7, 1, 0
 ; GFX9-NEXT:    s_cmp_eq_u32 s8, 0
 ; GFX9-NEXT:    v_lshrrev_b64 v[4:5], s6, v[0:1]

 ; GFX9-NEXT:    v_or_b32_e32 v0, s0, v6
 ; GFX9-NEXT:    v_or_b32_e32 v1, s1, v7
 ; GFX9-NEXT:    v_or_b32_e32 v2, s2, v2
 ; GFX9-NEXT:    v_or_b32_e32 v3, s3, v3
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshl_i128_vss:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_movk_i32 s6, 0x7f
-; GFX10-NEXT:    s_mov_b32 s7, 0
+; GFX10-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX10-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX10-NEXT:    s_andn2_b64 s[6:7], s[6:7], s[4:5]
 ; GFX10-NEXT:    s_sub_i32 s4, 64, s8
 ; GFX10-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX10-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX10-NEXT:    v_lshrrev_b64 v[4:5], s4, v[0:1]
 ; GFX10-NEXT:    v_lshlrev_b64 v[6:7], s8, v[2:3]
 ; GFX10-NEXT:    s_cselect_b32 vcc_lo, 1, 0

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = call i128 @llvm.fshl.i128(i128 %lhs, i128 %rhs, i128 65)
   ret i128 %result
 }
 
 define amdgpu_ps <2 x i128> @s_fshl_v2i128(<2 x i128> inreg %lhs, <2 x i128> inreg %rhs, <2 x i128> inreg %amt) {
 ; GFX6-LABEL: s_fshl_v2i128:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s18, 0x7f
-; GFX6-NEXT:    s_mov_b32 s19, 0
+; GFX6-NEXT:    s_mov_b64 s[18:19], 0x7f
 ; GFX6-NEXT:    s_and_b64 s[22:23], s[16:17], s[18:19]
 ; GFX6-NEXT:    s_andn2_b64 s[16:17], s[18:19], s[16:17]
 ; GFX6-NEXT:    s_sub_i32 s17, s22, 64
 ; GFX6-NEXT:    s_sub_i32 s23, 64, s22
 ; GFX6-NEXT:    s_cmp_lt_u32 s22, 64
 ; GFX6-NEXT:    s_cselect_b32 s28, 1, 0
 ; GFX6-NEXT:    s_cmp_eq_u32 s22, 0
 ; GFX6-NEXT:    s_cselect_b32 s29, 1, 0

 ; GFX6-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX6-NEXT:    s_cselect_b64 s[8:9], s[12:13], 0
 ; GFX6-NEXT:    s_or_b64 s[4:5], s[16:17], s[4:5]
 ; GFX6-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_fshl_v2i128:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s18, 0x7f
-; GFX8-NEXT:    s_mov_b32 s19, 0
+; GFX8-NEXT:    s_mov_b64 s[18:19], 0x7f
 ; GFX8-NEXT:    s_and_b64 s[22:23], s[16:17], s[18:19]
 ; GFX8-NEXT:    s_andn2_b64 s[16:17], s[18:19], s[16:17]
 ; GFX8-NEXT:    s_sub_i32 s17, s22, 64
 ; GFX8-NEXT:    s_sub_i32 s23, 64, s22
 ; GFX8-NEXT:    s_cmp_lt_u32 s22, 64
 ; GFX8-NEXT:    s_cselect_b32 s28, 1, 0
 ; GFX8-NEXT:    s_cmp_eq_u32 s22, 0
 ; GFX8-NEXT:    s_cselect_b32 s29, 1, 0

 ; GFX8-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX8-NEXT:    s_cselect_b64 s[8:9], s[12:13], 0
 ; GFX8-NEXT:    s_or_b64 s[4:5], s[16:17], s[4:5]
 ; GFX8-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_fshl_v2i128:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s18, 0x7f
-; GFX9-NEXT:    s_mov_b32 s19, 0
+; GFX9-NEXT:    s_mov_b64 s[18:19], 0x7f
 ; GFX9-NEXT:    s_and_b64 s[22:23], s[16:17], s[18:19]
 ; GFX9-NEXT:    s_andn2_b64 s[16:17], s[18:19], s[16:17]
 ; GFX9-NEXT:    s_sub_i32 s17, s22, 64
 ; GFX9-NEXT:    s_sub_i32 s23, 64, s22
 ; GFX9-NEXT:    s_cmp_lt_u32 s22, 64
 ; GFX9-NEXT:    s_cselect_b32 s28, 1, 0
 ; GFX9-NEXT:    s_cmp_eq_u32 s22, 0
 ; GFX9-NEXT:    s_cselect_b32 s29, 1, 0

 ; GFX9-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX9-NEXT:    s_cselect_b64 s[8:9], s[12:13], 0
 ; GFX9-NEXT:    s_or_b64 s[4:5], s[16:17], s[4:5]
 ; GFX9-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_fshl_v2i128:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_movk_i32 s18, 0x7f
-; GFX10-NEXT:    s_mov_b32 s19, 0
+; GFX10-NEXT:    s_mov_b64 s[18:19], 0x7f
 ; GFX10-NEXT:    s_and_b64 s[22:23], s[16:17], s[18:19]
 ; GFX10-NEXT:    s_andn2_b64 s[16:17], s[18:19], s[16:17]
 ; GFX10-NEXT:    s_sub_i32 s17, s22, 64
 ; GFX10-NEXT:    s_sub_i32 s23, 64, s22
 ; GFX10-NEXT:    s_cmp_lt_u32 s22, 64
 ; GFX10-NEXT:    s_cselect_b32 s28, 1, 0
 ; GFX10-NEXT:    s_cmp_eq_u32 s22, 0
 ; GFX10-NEXT:    s_cselect_b32 s29, 1, 0

llvm/test/CodeGen/AMDGPU/GlobalISel/fshr.ll

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = call <2 x i64> @llvm.fshr.v2i64(<2 x i64> %lhs, <2 x i64> %rhs, <2 x i64> %amt)
   ret <2 x i64> %result
 }
 
 define amdgpu_ps i128 @s_fshr_i128(i128 inreg %lhs, i128 inreg %rhs, i128 inreg %amt) {
 ; GFX6-LABEL: s_fshr_i128:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s10, 0x7f
-; GFX6-NEXT:    s_mov_b32 s11, 0
+; GFX6-NEXT:    s_mov_b64 s[10:11], 0x7f
 ; GFX6-NEXT:    s_and_b64 s[12:13], s[8:9], s[10:11]
 ; GFX6-NEXT:    s_andn2_b64 s[8:9], s[10:11], s[8:9]
 ; GFX6-NEXT:    s_sub_i32 s9, 64, 1
 ; GFX6-NEXT:    s_lshl_b64 s[10:11], s[0:1], 1
 ; GFX6-NEXT:    s_lshr_b64 s[0:1], s[0:1], s9
 ; GFX6-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX6-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX6-NEXT:    s_sub_i32 s13, s8, 64

 ; GFX6-NEXT:    s_cmp_lg_u32 s15, 0
 ; GFX6-NEXT:    s_cselect_b64 s[6:7], s[0:1], 0
 ; GFX6-NEXT:    s_or_b64 s[0:1], s[2:3], s[4:5]
 ; GFX6-NEXT:    s_or_b64 s[2:3], s[8:9], s[6:7]
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_fshr_i128:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s10, 0x7f
-; GFX8-NEXT:    s_mov_b32 s11, 0
+; GFX8-NEXT:    s_mov_b64 s[10:11], 0x7f
 ; GFX8-NEXT:    s_and_b64 s[12:13], s[8:9], s[10:11]
 ; GFX8-NEXT:    s_andn2_b64 s[8:9], s[10:11], s[8:9]
 ; GFX8-NEXT:    s_sub_i32 s9, 64, 1
 ; GFX8-NEXT:    s_lshl_b64 s[10:11], s[0:1], 1
 ; GFX8-NEXT:    s_lshr_b64 s[0:1], s[0:1], s9
 ; GFX8-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX8-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX8-NEXT:    s_sub_i32 s13, s8, 64

 ; GFX8-NEXT:    s_cmp_lg_u32 s15, 0
 ; GFX8-NEXT:    s_cselect_b64 s[6:7], s[0:1], 0
 ; GFX8-NEXT:    s_or_b64 s[0:1], s[2:3], s[4:5]
 ; GFX8-NEXT:    s_or_b64 s[2:3], s[8:9], s[6:7]
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_fshr_i128:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s10, 0x7f
-; GFX9-NEXT:    s_mov_b32 s11, 0
+; GFX9-NEXT:    s_mov_b64 s[10:11], 0x7f
 ; GFX9-NEXT:    s_and_b64 s[12:13], s[8:9], s[10:11]
 ; GFX9-NEXT:    s_andn2_b64 s[8:9], s[10:11], s[8:9]
 ; GFX9-NEXT:    s_sub_i32 s9, 64, 1
 ; GFX9-NEXT:    s_lshl_b64 s[10:11], s[0:1], 1
 ; GFX9-NEXT:    s_lshr_b64 s[0:1], s[0:1], s9
 ; GFX9-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX9-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX9-NEXT:    s_sub_i32 s13, s8, 64

 ; GFX9-NEXT:    s_cmp_lg_u32 s15, 0
 ; GFX9-NEXT:    s_cselect_b64 s[6:7], s[0:1], 0
 ; GFX9-NEXT:    s_or_b64 s[0:1], s[2:3], s[4:5]
 ; GFX9-NEXT:    s_or_b64 s[2:3], s[8:9], s[6:7]
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_fshr_i128:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_movk_i32 s10, 0x7f
-; GFX10-NEXT:    s_mov_b32 s11, 0
+; GFX10-NEXT:    s_mov_b64 s[10:11], 0x7f
 ; GFX10-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX10-NEXT:    s_and_b64 s[12:13], s[8:9], s[10:11]
 ; GFX10-NEXT:    s_sub_i32 s13, 64, 1
 ; GFX10-NEXT:    s_andn2_b64 s[8:9], s[10:11], s[8:9]
 ; GFX10-NEXT:    s_lshr_b64 s[10:11], s[0:1], s13
 ; GFX10-NEXT:    s_lshl_b64 s[0:1], s[0:1], 1
 ; GFX10-NEXT:    s_or_b64 s[2:3], s[10:11], s[2:3]
 ; GFX10-NEXT:    s_sub_i32 s13, s8, 64
; GFX10-NEXT:    ; return to shader part epilog
   %result = call i128 @llvm.fshr.i128(i128 %lhs, i128 %rhs, i128 %amt)
   %cast.result = bitcast i128 %result to <4 x float>
   ret <4 x float> %cast.result
 }
 
 define amdgpu_ps <4 x float> @v_fshr_i128_svs(i128 inreg %lhs, i128 %rhs, i128 inreg %amt) {
 ; GFX6-LABEL: v_fshr_i128_svs:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s6, 0x7f
-; GFX6-NEXT:    s_mov_b32 s7, 0
+; GFX6-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX6-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX6-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX6-NEXT:    s_sub_i32 s5, 64, 1
 ; GFX6-NEXT:    s_lshl_b64 s[6:7], s[0:1], 1
 ; GFX6-NEXT:    s_lshr_b64 s[0:1], s[0:1], s5
 ; GFX6-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX6-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX6-NEXT:    s_sub_i32 s9, s4, 64

 ; GFX6-NEXT:    v_or_b32_e32 v0, s2, v0
 ; GFX6-NEXT:    v_or_b32_e32 v1, s3, v1
 ; GFX6-NEXT:    v_or_b32_e32 v2, s0, v2
 ; GFX6-NEXT:    v_or_b32_e32 v3, s1, v3
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: v_fshr_i128_svs:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s6, 0x7f
-; GFX8-NEXT:    s_mov_b32 s7, 0
+; GFX8-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX8-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX8-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX8-NEXT:    s_sub_i32 s5, 64, 1
 ; GFX8-NEXT:    s_lshl_b64 s[6:7], s[0:1], 1
 ; GFX8-NEXT:    s_lshr_b64 s[0:1], s[0:1], s5
 ; GFX8-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX8-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX8-NEXT:    s_sub_i32 s9, s4, 64

 ; GFX8-NEXT:    v_or_b32_e32 v0, s2, v0
 ; GFX8-NEXT:    v_or_b32_e32 v1, s3, v1
 ; GFX8-NEXT:    v_or_b32_e32 v2, s0, v2
 ; GFX8-NEXT:    v_or_b32_e32 v3, s1, v3
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshr_i128_svs:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s6, 0x7f
-; GFX9-NEXT:    s_mov_b32 s7, 0
+; GFX9-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX9-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX9-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX9-NEXT:    s_sub_i32 s5, 64, 1
 ; GFX9-NEXT:    s_lshl_b64 s[6:7], s[0:1], 1
 ; GFX9-NEXT:    s_lshr_b64 s[0:1], s[0:1], s5
 ; GFX9-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX9-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX9-NEXT:    s_sub_i32 s9, s4, 64

 ; GFX9-NEXT:    v_or_b32_e32 v0, s2, v0
 ; GFX9-NEXT:    v_or_b32_e32 v1, s3, v1
 ; GFX9-NEXT:    v_or_b32_e32 v2, s0, v2
 ; GFX9-NEXT:    v_or_b32_e32 v3, s1, v3
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshr_i128_svs:
 ; GFX10:       ; %bb.0:
-; GFX10-NEXT:    s_movk_i32 s6, 0x7f
-; GFX10-NEXT:    s_mov_b32 s7, 0
+; GFX10-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX10-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX10-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX10-NEXT:    s_sub_i32 s9, 64, 1
 ; GFX10-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX10-NEXT:    s_lshr_b64 s[6:7], s[0:1], s9
 ; GFX10-NEXT:    s_lshl_b64 s[0:1], s[0:1], 1
 ; GFX10-NEXT:    s_or_b64 s[2:3], s[6:7], s[2:3]
 ; GFX10-NEXT:    s_sub_i32 s9, s4, 64
; GFX10-NEXT:    ; return to shader part epilog
   %result = call i128 @llvm.fshr.i128(i128 %lhs, i128 %rhs, i128 %amt)
   %cast.result = bitcast i128 %result to <4 x float>
   ret <4 x float> %cast.result
 }
 
 define amdgpu_ps <4 x float> @v_fshr_i128_vss(i128 %lhs, i128 inreg %rhs, i128 inreg %amt) {
 ; GFX6-LABEL: v_fshr_i128_vss:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s6, 0x7f
-; GFX6-NEXT:    s_mov_b32 s7, 0
+; GFX6-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX6-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX6-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX6-NEXT:    s_sub_i32 s5, 64, 1
 ; GFX6-NEXT:    v_lshr_b64 v[4:5], v[0:1], s5
 ; GFX6-NEXT:    v_lshl_b64 v[2:3], v[2:3], 1
 ; GFX6-NEXT:    s_sub_i32 s5, s4, 64
 ; GFX6-NEXT:    s_sub_i32 s6, 64, s4
 ; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 1

 ; GFX6-NEXT:    v_or_b32_e32 v0, s0, v6
 ; GFX6-NEXT:    v_or_b32_e32 v1, s1, v7
 ; GFX6-NEXT:    v_or_b32_e32 v2, s2, v2
 ; GFX6-NEXT:    v_or_b32_e32 v3, s3, v3
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: v_fshr_i128_vss:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s6, 0x7f
-; GFX8-NEXT:    s_mov_b32 s7, 0
+; GFX8-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX8-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX8-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX8-NEXT:    s_sub_i32 s5, 64, 1
 ; GFX8-NEXT:    v_lshrrev_b64 v[4:5], s5, v[0:1]
 ; GFX8-NEXT:    v_lshlrev_b64 v[2:3], 1, v[2:3]
 ; GFX8-NEXT:    s_sub_i32 s5, s4, 64
 ; GFX8-NEXT:    s_sub_i32 s6, 64, s4
 ; GFX8-NEXT:    v_lshlrev_b64 v[0:1], 1, v[0:1]

 ; GFX8-NEXT:    v_or_b32_e32 v0, s0, v6
 ; GFX8-NEXT:    v_or_b32_e32 v1, s1, v7
 ; GFX8-NEXT:    v_or_b32_e32 v2, s2, v2
 ; GFX8-NEXT:    v_or_b32_e32 v3, s3, v3
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: v_fshr_i128_vss:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s6, 0x7f
-; GFX9-NEXT:    s_mov_b32 s7, 0
+; GFX9-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX9-NEXT:    s_and_b64 s[8:9], s[4:5], s[6:7]
 ; GFX9-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; GFX9-NEXT:    s_sub_i32 s5, 64, 1
 ; GFX9-NEXT:    v_lshrrev_b64 v[4:5], s5, v[0:1]
 ; GFX9-NEXT:    v_lshlrev_b64 v[2:3], 1, v[2:3]
 ; GFX9-NEXT:    s_sub_i32 s5, s4, 64
 ; GFX9-NEXT:    s_sub_i32 s6, 64, s4
 ; GFX9-NEXT:    v_lshlrev_b64 v[0:1], 1, v[0:1]

 ; GFX9-NEXT:    v_or_b32_e32 v3, s3, v3
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: v_fshr_i128_vss:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_sub_i32 s6, 64, 1
 ; GFX10-NEXT:    v_lshlrev_b64 v[2:3], 1, v[2:3]
 ; GFX10-NEXT:    v_lshrrev_b64 v[4:5], s6, v[0:1]
-; GFX10-NEXT:    s_movk_i32 s6, 0x7f
-; GFX10-NEXT:    s_mov_b32 s7, 0
+; GFX10-NEXT:    s_mov_b64 s[6:7], 0x7f
 ; GFX10-NEXT:    v_lshlrev_b64 v[0:1], 1, v[0:1]
 ; GFX10-NEXT:    s_andn2_b64 s[8:9], s[6:7], s[4:5]
 ; GFX10-NEXT:    s_and_b64 s[6:7], s[4:5], s[6:7]
+; GFX10-NEXT:    s_sub_i32 s4, 64, s8
 ; GFX10-NEXT:    v_or_b32_e32 v2, v4, v2
 ; GFX10-NEXT:    v_or_b32_e32 v3, v5, v3
-; GFX10-NEXT:    s_sub_i32 s4, 64, s8
 ; GFX10-NEXT:    s_sub_i32 s5, s8, 64
 ; GFX10-NEXT:    s_cmp_lt_u32 s8, 64
 ; GFX10-NEXT:    v_lshrrev_b64 v[4:5], s4, v[0:1]
-; GFX10-NEXT:    v_lshlrev_b64 v[6:7], s8, v[2:3]
 ; GFX10-NEXT:    s_cselect_b32 vcc_lo, 1, 0
+; GFX10-NEXT:    v_lshlrev_b64 v[6:7], s8, v[2:3]
 ; GFX10-NEXT:    s_cmp_eq_u32 s8, 0
 ; GFX10-NEXT:    v_lshlrev_b64 v[8:9], s8, v[0:1]
 ; GFX10-NEXT:    s_cselect_b32 s7, 1, 0
 ; GFX10-NEXT:    s_and_b32 s4, 1, vcc_lo
 ; GFX10-NEXT:    v_lshlrev_b64 v[0:1], s5, v[0:1]
 ; GFX10-NEXT:    v_or_b32_e32 v4, v4, v6
 ; GFX10-NEXT:    v_or_b32_e32 v5, v5, v7
 ; GFX10-NEXT:    v_cmp_ne_u32_e64 s4, 0, s4

 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
   %result = call i128 @llvm.fshr.i128(i128 %lhs, i128 %rhs, i128 65)
   ret i128 %result
 }
 
 define amdgpu_ps <2 x i128> @s_fshr_v2i128(<2 x i128> inreg %lhs, <2 x i128> inreg %rhs, <2 x i128> inreg %amt) {
 ; GFX6-LABEL: s_fshr_v2i128:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s18, 0x7f
-; GFX6-NEXT:    s_mov_b32 s19, 0
+; GFX6-NEXT:    s_mov_b64 s[18:19], 0x7f
 ; GFX6-NEXT:    s_sub_i32 s28, 64, 1
 ; GFX6-NEXT:    s_and_b64 s[22:23], s[16:17], s[18:19]
 ; GFX6-NEXT:    s_andn2_b64 s[16:17], s[18:19], s[16:17]
 ; GFX6-NEXT:    s_lshl_b64 s[24:25], s[0:1], 1
 ; GFX6-NEXT:    s_lshr_b64 s[0:1], s[0:1], s28
 ; GFX6-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX6-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX6-NEXT:    s_sub_i32 s23, s16, 64

 ; GFX6-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX6-NEXT:    s_cselect_b64 s[12:13], s[4:5], 0
 ; GFX6-NEXT:    s_or_b64 s[4:5], s[6:7], s[8:9]
 ; GFX6-NEXT:    s_or_b64 s[6:7], s[10:11], s[12:13]
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: s_fshr_v2i128:
 ; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_movk_i32 s18, 0x7f
-; GFX8-NEXT:    s_mov_b32 s19, 0
+; GFX8-NEXT:    s_mov_b64 s[18:19], 0x7f
 ; GFX8-NEXT:    s_sub_i32 s28, 64, 1
 ; GFX8-NEXT:    s_and_b64 s[22:23], s[16:17], s[18:19]
 ; GFX8-NEXT:    s_andn2_b64 s[16:17], s[18:19], s[16:17]
 ; GFX8-NEXT:    s_lshl_b64 s[24:25], s[0:1], 1
 ; GFX8-NEXT:    s_lshr_b64 s[0:1], s[0:1], s28
 ; GFX8-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX8-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX8-NEXT:    s_sub_i32 s23, s16, 64

 ; GFX8-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX8-NEXT:    s_cselect_b64 s[12:13], s[4:5], 0
 ; GFX8-NEXT:    s_or_b64 s[4:5], s[6:7], s[8:9]
 ; GFX8-NEXT:    s_or_b64 s[6:7], s[10:11], s[12:13]
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_fshr_v2i128:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s18, 0x7f
-; GFX9-NEXT:    s_mov_b32 s19, 0
+; GFX9-NEXT:    s_mov_b64 s[18:19], 0x7f
 ; GFX9-NEXT:    s_sub_i32 s28, 64, 1
 ; GFX9-NEXT:    s_and_b64 s[22:23], s[16:17], s[18:19]
 ; GFX9-NEXT:    s_andn2_b64 s[16:17], s[18:19], s[16:17]
 ; GFX9-NEXT:    s_lshl_b64 s[24:25], s[0:1], 1
 ; GFX9-NEXT:    s_lshr_b64 s[0:1], s[0:1], s28
 ; GFX9-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX9-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX9-NEXT:    s_sub_i32 s23, s16, 64

 ; GFX9-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX9-NEXT:    s_cselect_b64 s[12:13], s[4:5], 0
 ; GFX9-NEXT:    s_or_b64 s[4:5], s[6:7], s[8:9]
 ; GFX9-NEXT:    s_or_b64 s[6:7], s[10:11], s[12:13]
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: s_fshr_v2i128:
 ; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_mov_b64 s[18:19], 0x7f
 ; GFX10-NEXT:    s_sub_i32 s28, 64, 1
-; GFX10-NEXT:    s_movk_i32 s18, 0x7f
-; GFX10-NEXT:    s_mov_b32 s19, 0
-; GFX10-NEXT:    s_lshr_b64 s[24:25], s[0:1], s28
-; GFX10-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX10-NEXT:    s_and_b64 s[22:23], s[16:17], s[18:19]
 ; GFX10-NEXT:    s_andn2_b64 s[16:17], s[18:19], s[16:17]
+; GFX10-NEXT:    s_lshr_b64 s[24:25], s[0:1], s28
+; GFX10-NEXT:    s_lshl_b64 s[2:3], s[2:3], 1
 ; GFX10-NEXT:    s_lshl_b64 s[0:1], s[0:1], 1
 ; GFX10-NEXT:    s_or_b64 s[2:3], s[24:25], s[2:3]
 ; GFX10-NEXT:    s_sub_i32 s23, s16, 64
 ; GFX10-NEXT:    s_sub_i32 s17, 64, s16
 ; GFX10-NEXT:    s_cmp_lt_u32 s16, 64
 ; GFX10-NEXT:    s_cselect_b32 s29, 1, 0
 ; GFX10-NEXT:    s_cmp_eq_u32 s16, 0
 ; GFX10-NEXT:    s_cselect_b32 s30, 1, 0

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.global.atomic.csub.ll

; GCN-NEXT:    s_setpc_b64 s[30:31]
   ret i32 %ret
 }
 
 define i32 @global_atomic_csub_offset(i32 addrspace(1)* %ptr, i32 %data) {
 ; GCN-LABEL: global_atomic_csub_offset:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    s_waitcnt_vscnt null, 0x0
-; GCN-NEXT:    s_movk_i32 s4, 0x1000
-; GCN-NEXT:    s_mov_b32 s5, 0
+; GCN-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; GCN-NEXT:    v_mov_b32_e32 v3, s4
 ; GCN-NEXT:    v_mov_b32_e32 v4, s5
 ; GCN-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v3
 ; GCN-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v4, vcc_lo
 ; GCN-NEXT:    global_atomic_csub v0, v[0:1], v2, off glc
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %gep = getelementptr i32, i32 addrspace(1)* %ptr, i64 1024
; GCN-NEXT:    s_setpc_b64 s[30:31]
   ret void
 }
 
 define void @global_atomic_csub_offset_nortn(i32 addrspace(1)* %ptr, i32 %data) {
 ; GCN-LABEL: global_atomic_csub_offset_nortn:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    s_waitcnt_vscnt null, 0x0
-; GCN-NEXT:    s_movk_i32 s4, 0x1000
-; GCN-NEXT:    s_mov_b32 s5, 0
+; GCN-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; GCN-NEXT:    v_mov_b32_e32 v3, s4
 ; GCN-NEXT:    v_mov_b32_e32 v4, s5
 ; GCN-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v3
 ; GCN-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, v1, v4, vcc_lo
 ; GCN-NEXT:    global_atomic_csub v0, v[0:1], v2, off glc
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %gep = getelementptr i32, i32 addrspace(1)* %ptr, i64 1024

llvm/test/CodeGen/AMDGPU/GlobalISel/mubuf-global.ll

; GFX7-NEXT:    s_endpgm
   %gep1 = getelementptr i32, i32 addrspace(1)* %gep0, i32 256
   store i32 0, i32 addrspace(1)* %gep1
   ret void
 }
 
 define amdgpu_ps void @mubuf_store_vgpr_ptr_sgpr_offset256_offset(i32 addrspace(1)* %ptr, i32 inreg %soffset) {
 ; GFX6-LABEL: mubuf_store_vgpr_ptr_sgpr_offset256_offset:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s4, 0x400
-; GFX6-NEXT:    s_mov_b32 s5, 0
+; GFX6-NEXT:    s_mov_b64 s[4:5], 0x400
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX6-NEXT:    s_bfe_i64 s[0:1], s[2:3], 0x200000
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s5
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    s_lshl_b64 s[0:1], s[0:1], 2
 ; GFX6-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, s5
 ; GFX6-NEXT:    buffer_store_dword v2, v[0:1], s[0:3], 0 addr64
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX7-LABEL: mubuf_store_vgpr_ptr_sgpr_offset256_offset:
 ; GFX7:       ; %bb.0:
-; GFX7-NEXT:    s_movk_i32 s4, 0x400
-; GFX7-NEXT:    s_mov_b32 s5, 0
+; GFX7-NEXT:    s_mov_b64 s[4:5], 0x400
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX7-NEXT:    s_bfe_i64 s[0:1], s[2:3], 0x200000
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s5
 ; GFX7-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX7-NEXT:    s_lshl_b64 s[0:1], s[0:1], 2
 ; GFX7-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX7-NEXT:    s_mov_b32 s3, 0xf000
; GFX7-NEXT:    ; return to shader part epilog
   %gep1 = getelementptr float, float addrspace(1)* %gep0, i32 256
   %val = load volatile float, float addrspace(1)* %gep1
   ret float %val
 }
 
 define amdgpu_ps float @mubuf_load_vgpr_ptr_sgpr_offset256_offset(float addrspace(1)* %ptr, i32 inreg %soffset) {
 ; GFX6-LABEL: mubuf_load_vgpr_ptr_sgpr_offset256_offset:
 ; GFX6:       ; %bb.0:
-; GFX6-NEXT:    s_movk_i32 s4, 0x400
-; GFX6-NEXT:    s_mov_b32 s5, 0
+; GFX6-NEXT:    s_mov_b64 s[4:5], 0x400
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX6-NEXT:    s_bfe_i64 s[0:1], s[2:3], 0x200000
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s5
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    s_lshl_b64 s[0:1], s[0:1], 2
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, s5
 ; GFX6-NEXT:    buffer_load_dword v0, v[0:1], s[0:3], 0 addr64 glc
 ; GFX6-NEXT:    s_waitcnt vmcnt(0)
 ; GFX6-NEXT:    ; return to shader part epilog
 ;
 ; GFX7-LABEL: mubuf_load_vgpr_ptr_sgpr_offset256_offset:
 ; GFX7:       ; %bb.0:
-; GFX7-NEXT:    s_movk_i32 s4, 0x400
-; GFX7-NEXT:    s_mov_b32 s5, 0
+; GFX7-NEXT:    s_mov_b64 s[4:5], 0x400
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX7-NEXT:    s_bfe_i64 s[0:1], s[2:3], 0x200000
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s5
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX7-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX7-NEXT:    s_lshl_b64 s[0:1], s[0:1], 2
 ; GFX7-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7-NEXT:    s_mov_b32 s2, s5

llvm/test/CodeGen/AMDGPU/GlobalISel/roundeven.ll

 ; GFX6-LABEL: v_roundeven_f64_fneg:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX6-NEXT:    v_xor_b32_e32 v2, 0x80000000, v1
 ; GFX6-NEXT:    v_and_b32_e32 v4, 0x80000000, v2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, 0
 ; GFX6-NEXT:    v_or_b32_e32 v4, 0x43300000, v4
 ; GFX6-NEXT:    v_add_f64 v[5:6], -v[0:1], v[3:4]
-; GFX6-NEXT:    v_mov_b32_e32 v1, v0
 ; GFX6-NEXT:    s_mov_b32 s4, -1
+; GFX6-NEXT:    v_mov_b32_e32 v1, v0
 ; GFX6-NEXT:    s_mov_b32 s5, 0x432fffff
 ; GFX6-NEXT:    v_add_f64 v[3:4], v[5:6], -v[3:4]
 ; GFX6-NEXT:    v_cmp_gt_f64_e64 vcc, |v[1:2]|, s[4:5]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v3, v0, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v4, v2, vcc
 ; GFX6-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX7-LABEL: v_roundeven_f64_fneg:

llvm/test/CodeGen/AMDGPU/GlobalISel/sdiv.i64.ll

; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv <2 x i64> %num, <i64 1235195, i64 1235195>
   ret <2 x i64> %result
 }
 
 define i64 @v_sdiv_i64_pow2_shl_denom(i64 %x, i64 %y) {
 ; CHECK-LABEL: v_sdiv_i64_pow2_shl_denom:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CHECK-NEXT:    s_movk_i32 s4, 0x1000
-; CHECK-NEXT:    s_mov_b32 s5, 0
+; CHECK-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CHECK-NEXT:    v_lshl_b64 v[4:5], s[4:5], v2
 ; CHECK-NEXT:    v_mov_b32_e32 v2, 0
 ; CHECK-NEXT:    v_or_b32_e32 v3, v1, v5
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CHECK-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB7_2
; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %r = sdiv i64 %x, %shl.y
   ret i64 %r
 }
 
 define <2 x i64> @v_sdiv_v2i64_pow2_shl_denom(<2 x i64> %x, <2 x i64> %y) {
 ; GISEL-LABEL: v_sdiv_v2i64_pow2_shl_denom:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GISEL-NEXT:    s_movk_i32 s6, 0x1000
-; GISEL-NEXT:    s_mov_b32 s7, 0
+; GISEL-NEXT:    s_mov_b64 s[6:7], 0x1000
 ; GISEL-NEXT:    v_lshl_b64 v[4:5], s[6:7], v4
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v10, 31, v1
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v7, 31, v5
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v7, vcc
 ; GISEL-NEXT:    v_xor_b32_e32 v4, v4, v7
 ; GISEL-NEXT:    v_xor_b32_e32 v5, v5, v7
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v8, v4

 ; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v4
 ; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v4
 ; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v4, vcc
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_sdiv_v2i64_pow2_shl_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CGP-NEXT:    s_movk_i32 s4, 0x1000
-; CGP-NEXT:    s_mov_b32 s5, 0
+; CGP-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CGP-NEXT:    v_lshl_b64 v[10:11], s[4:5], v4
 ; CGP-NEXT:    v_mov_b32_e32 v7, v1
 ; CGP-NEXT:    v_mov_b32_e32 v5, v0
 ; CGP-NEXT:    v_or_b32_e32 v1, v7, v11
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    v_lshl_b64 v[8:9], s[4:5], v6
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i64.ll

; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = srem <2 x i64> %num, <i64 1235195, i64 1235195>
   ret <2 x i64> %result
 }
 
 define i64 @v_srem_i64_pow2_shl_denom(i64 %x, i64 %y) {
 ; CHECK-LABEL: v_srem_i64_pow2_shl_denom:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CHECK-NEXT:    s_movk_i32 s4, 0x1000
-; CHECK-NEXT:    s_mov_b32 s5, 0
+; CHECK-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CHECK-NEXT:    v_lshl_b64 v[4:5], s[4:5], v2
 ; CHECK-NEXT:    v_mov_b32_e32 v2, 0
 ; CHECK-NEXT:    v_or_b32_e32 v3, v1, v5
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CHECK-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB7_2
; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %r = srem i64 %x, %shl.y
   ret i64 %r
 }
 
 define <2 x i64> @v_srem_v2i64_pow2_shl_denom(<2 x i64> %x, <2 x i64> %y) {
 ; GISEL-LABEL: v_srem_v2i64_pow2_shl_denom:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GISEL-NEXT:    s_movk_i32 s6, 0x1000
-; GISEL-NEXT:    s_mov_b32 s7, 0
+; GISEL-NEXT:    s_mov_b64 s[6:7], 0x1000
 ; GISEL-NEXT:    v_lshl_b64 v[4:5], s[6:7], v4
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v9, 31, v1
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v7, 31, v5
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v7, vcc
 ; GISEL-NEXT:    v_xor_b32_e32 v5, v5, v7
 ; GISEL-NEXT:    v_xor_b32_e32 v4, v4, v7
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v7, v4

 ; GISEL-NEXT:    v_xor_b32_e32 v4, v2, v8
 ; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v3, v8
 ; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v4, v8, vcc
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_srem_v2i64_pow2_shl_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CGP-NEXT:    s_movk_i32 s4, 0x1000
-; CGP-NEXT:    s_mov_b32 s5, 0
+; CGP-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CGP-NEXT:    v_lshl_b64 v[10:11], s[4:5], v4
 ; CGP-NEXT:    v_mov_b32_e32 v7, v1
 ; CGP-NEXT:    v_mov_b32_e32 v5, v0
 ; CGP-NEXT:    v_or_b32_e32 v1, v7, v11
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    v_lshl_b64 v[8:9], s[4:5], v6
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1

llvm/test/CodeGen/AMDGPU/GlobalISel/udiv.i64.ll

; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = udiv <2 x i64> %num, <i64 1235195, i64 1235195>
   ret <2 x i64> %result
 }
 
 define i64 @v_udiv_i64_pow2_shl_denom(i64 %x, i64 %y) {
 ; CHECK-LABEL: v_udiv_i64_pow2_shl_denom:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CHECK-NEXT:    s_movk_i32 s4, 0x1000
-; CHECK-NEXT:    s_mov_b32 s5, 0
+; CHECK-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CHECK-NEXT:    v_lshl_b64 v[4:5], s[4:5], v2
 ; CHECK-NEXT:    v_or_b32_e32 v3, v1, v5
 ; CHECK-NEXT:    v_mov_b32_e32 v2, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CHECK-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB7_2
; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %r = udiv i64 %x, %shl.y
   ret i64 %r
 }
 
 define <2 x i64> @v_udiv_v2i64_pow2_shl_denom(<2 x i64> %x, <2 x i64> %y) {
 ; GISEL-LABEL: v_udiv_v2i64_pow2_shl_denom:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GISEL-NEXT:    s_movk_i32 s4, 0x1000
-; GISEL-NEXT:    s_mov_b32 s5, 0
+; GISEL-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; GISEL-NEXT:    v_lshl_b64 v[4:5], s[4:5], v4
 ; GISEL-NEXT:    v_lshl_b64 v[6:7], s[4:5], v6
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v8, v4
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v9, v5
 ; GISEL-NEXT:    v_mac_f32_e32 v8, 0x4f800000, v9
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v8, v8
 ; GISEL-NEXT:    v_mul_f32_e32 v8, 0x5f7ffffc, v8
 ; GISEL-NEXT:    v_mul_f32_e32 v9, 0x2f800000, v8

 ; GISEL-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_udiv_v2i64_pow2_shl_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_mov_b32_e32 v5, v0
 ; CGP-NEXT:    v_mov_b32_e32 v7, v1
-; CGP-NEXT:    s_movk_i32 s4, 0x1000
-; CGP-NEXT:    s_mov_b32 s5, 0
+; CGP-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CGP-NEXT:    v_lshl_b64 v[10:11], s[4:5], v4
 ; CGP-NEXT:    v_lshl_b64 v[8:9], s[4:5], v6
 ; CGP-NEXT:    v_or_b32_e32 v1, v7, v11
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]

llvm/test/CodeGen/AMDGPU/GlobalISel/urem.i64.ll

; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = urem <2 x i64> %num, <i64 1235195, i64 1235195>
   ret <2 x i64> %result
 }
 
 define i64 @v_urem_i64_pow2_shl_denom(i64 %x, i64 %y) {
 ; CHECK-LABEL: v_urem_i64_pow2_shl_denom:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CHECK-NEXT:    s_movk_i32 s4, 0x1000
-; CHECK-NEXT:    s_mov_b32 s5, 0
+; CHECK-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CHECK-NEXT:    v_lshl_b64 v[4:5], s[4:5], v2
 ; CHECK-NEXT:    v_or_b32_e32 v3, v1, v5
 ; CHECK-NEXT:    v_mov_b32_e32 v2, 0
 ; CHECK-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[2:3]
 ; CHECK-NEXT:    ; implicit-def: $vgpr2_vgpr3
 ; CHECK-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CHECK-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]
 ; CHECK-NEXT:    s_cbranch_execz BB7_2
; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %r = urem i64 %x, %shl.y
   ret i64 %r
 }
 
 define <2 x i64> @v_urem_v2i64_pow2_shl_denom(<2 x i64> %x, <2 x i64> %y) {
 ; GISEL-LABEL: v_urem_v2i64_pow2_shl_denom:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GISEL-NEXT:    s_movk_i32 s4, 0x1000
-; GISEL-NEXT:    s_mov_b32 s5, 0
+; GISEL-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; GISEL-NEXT:    v_lshl_b64 v[4:5], s[4:5], v4
 ; GISEL-NEXT:    v_lshl_b64 v[6:7], s[4:5], v6
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v8, v4
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v9, v5
 ; GISEL-NEXT:    v_mac_f32_e32 v8, 0x4f800000, v9
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v8, v8
 ; GISEL-NEXT:    v_mul_f32_e32 v8, 0x5f7ffffc, v8
 ; GISEL-NEXT:    v_mul_f32_e32 v9, 0x2f800000, v8

 ; GISEL-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_urem_v2i64_pow2_shl_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_mov_b32_e32 v5, v0
 ; CGP-NEXT:    v_mov_b32_e32 v7, v1
-; CGP-NEXT:    s_movk_i32 s4, 0x1000
-; CGP-NEXT:    s_mov_b32 s5, 0
+; CGP-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; CGP-NEXT:    v_lshl_b64 v[10:11], s[4:5], v4
 ; CGP-NEXT:    v_lshl_b64 v[8:9], s[4:5], v6
 ; CGP-NEXT:    v_or_b32_e32 v1, v7, v11
 ; CGP-NEXT:    v_mov_b32_e32 v0, 0
 ; CGP-NEXT:    v_cmp_ne_u64_e32 vcc, 0, v[0:1]
 ; CGP-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CGP-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; CGP-NEXT:    s_xor_b64 s[6:7], exec, s[4:5]

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll

 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s3, v1
 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, v1, v2, s[0:1]
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s3, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s3
 ; GFX9-NEXT:    s_sub_i32 s4, 0, s3

 ; GFX6-NEXT:    v_subrev_i32_e32 v1, vcc, s5, v0
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s5, v0
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v1, vcc, s5, v0
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s5, v0
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s3
 ; GFX9-NEXT:    s_sub_i32 s4, 0, s3

 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, v1, v2, s[0:1]
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s9, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s2, v0
 ; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, s2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s4, s3, 31
 ; GFX9-NEXT:    s_add_i32 s3, s3, s4

 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v1, vcc, s6, v0
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s6, v0
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s4, s3, 31
 ; GFX9-NEXT:    s_add_i32 s3, s3, s4

 ; GFX6-NEXT:    v_mul_f32_e32 v2, v1, v2
 ; GFX6-NEXT:    v_trunc_f32_e32 v2, v2
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v3, v2
 ; GFX6-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, 0, v3, vcc
 ; GFX6-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v3, 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshr_b32 s3, s2, 16
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s3

 ; GFX6-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, 0, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshr_b32 s3, s2, 16
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s3

 ; GFX6-NEXT:    v_trunc_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX6-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, |v0|
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, 0, v3, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    buffer_store_short v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s0, s4, 16
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s0

 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, 0, v3, vcc
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s5, s4, 16
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s5

 ; GFX6-NEXT:    v_mul_f32_e32 v1, v2, v1
 ; GFX6-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v3, v1
 ; GFX6-NEXT:    v_mad_f32 v1, -v1, v0, v2
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, 0, v3, vcc
 ; GFX6-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i8:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_cvt_f32_ubyte1_e32 v0, s2
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v0

 ; GFX6-NEXT:    v_mad_f32 v1, -v1, v0, v2
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, 0, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i8:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_cvt_f32_ubyte1_e32 v0, s2
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v0

 ; GFX6-NEXT:    v_trunc_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX6-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, |v0|
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, 0, v3, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i8:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_i32 s0, s4, 0x80008
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s0

 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, |v0|
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, 0, v3, vcc
 ; GFX6-NEXT:    s_lshr_b32 s2, s0, 8
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s0, v0
 ; GFX6-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i8:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_i32 s2, s4, 0x80008
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s2

 ; GFX6-NEXT:    v_cndmask_b32_e64 v4, v4, v5, s[0:1]
 ; GFX6-NEXT:    v_subrev_i32_e32 v5, vcc, s11, v3
 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, v3, v5, s[0:1]
 ; GFX6-NEXT:    v_add_i32_e32 v5, vcc, 1, v4
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s11, v3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v4, v5, vcc
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[12:15], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v4i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x34
 ; GFX9-NEXT:    s_mov_b32 s12, 0x4f7ffffe
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s8

 ; GFX6-NEXT:    v_subrev_i32_e32 v4, vcc, s11, v3
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s11, v3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v4, vcc, s11, v3
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s11, v3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_v4i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x34
 ; GFX9-NEXT:    s_mov_b32 s12, 0x4f7ffffe
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s8

 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, v3, v5, s[0:1]
 ; GFX6-NEXT:    v_add_i32_e32 v5, vcc, 1, v4
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s8, v3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v4, v5, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v3, s2, v3
 ; GFX6-NEXT:    v_subrev_i32_e32 v3, vcc, s2, v3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_v4i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x34
 ; GFX9-NEXT:    s_mov_b32 s13, 0x4f7ffffe
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s2, s8, 31

 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v4, vcc, s8, v3
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s8, v3
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v3, s5, v3
 ; GFX6-NEXT:    v_subrev_i32_e32 v3, vcc, s5, v3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_v4i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx8 s[4:11], s[0:1], 0x34
 ; GFX9-NEXT:    s_mov_b32 s13, 0x4f7ffffe
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s2, s8, 31

 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v5, vcc
 ; GFX6-NEXT:    v_and_b32_e32 v0, s8, v0
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
 ; GFX6-NEXT:    v_and_b32_e32 v1, s8, v1
 ; GFX6-NEXT:    v_or_b32_e32 v1, v1, v3
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v2
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v4i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x2c
 ; GFX9-NEXT:    s_mov_b32 s8, 0xffff
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s1, s6, s8

 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, s10, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
 ; GFX6-NEXT:    v_and_b32_e32 v1, s8, v1
 ; GFX6-NEXT:    v_or_b32_e32 v1, v1, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v5
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v2
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_v4i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x2c
 ; GFX9-NEXT:    s_mov_b32 s8, 0xffff
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s1, s6, s8

 ; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
 ; GFX6-NEXT:    v_and_b32_e32 v1, s0, v1
 ; GFX6-NEXT:    v_or_b32_e32 v1, v1, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v3
 ; GFX6-NEXT:    v_and_b32_e32 v0, s0, v0
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v2
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_v4i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_sext_i32_i16 s0, s6
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s0

 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, s2, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
 ; GFX6-NEXT:    v_or_b32_e32 v1, v1, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 16, v3
 ; GFX6-NEXT:    v_and_b32_e32 v0, s0, v0
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v2
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_v4i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_sext_i32_i16 s0, s6
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s0

 ; GFX6-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v3, v1
 ; GFX6-NEXT:    v_mad_f32 v1, -v1, v0, v2
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, 0, v3, vcc
 ; GFX6-NEXT:    v_and_b32_e32 v0, 7, v0
 ; GFX6-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i3:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_u32 s0, s4, 0x30008
 ; GFX9-NEXT:    v_cvt_f32_ubyte0_e32 v0, s0

 ; GFX6-NEXT:    v_mad_f32 v1, -v1, v0, v2
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, 0, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s1
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s0, v0
 ; GFX6-NEXT:    v_and_b32_e32 v0, 7, v0
 ; GFX6-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i3:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_u32 s3, s2, 0x30008
 ; GFX9-NEXT:    v_cvt_f32_ubyte0_e32 v0, s3

 ; GFX6-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX6-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, |v0|
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, 0, v3, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_and_b32_e32 v0, 7, v0
 ; GFX6-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i3:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_i32 s0, s4, 0x30008
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s0

 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, 0, v3, vcc
 ; GFX6-NEXT:    s_lshr_b32 s2, s0, 8
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s0, v0
 ; GFX6-NEXT:    v_and_b32_e32 v0, 7, v0
 ; GFX6-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i3:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_i32 s2, s4, 0x30008
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s2

 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v2|, v4
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, s8, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v2, vcc, 0, v3, vcc
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    buffer_store_short v2, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v3i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_mov_b32 s8, 0xffff
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, s8, v0
 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, s3, v3
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    buffer_store_short v2, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_v3i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_mov_b32 s8, 0xffff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s4, s8

 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, 0, v5, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    buffer_store_short v2, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_v3i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_sext_i32_i16 s1, s4

 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, v1, v2
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, s3, v3
 ; GFX6-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX6-NEXT:    buffer_store_short v2, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_v3i16:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x2c
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_sext_i32_i16 s8, s2
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s8

 ; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
 ; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX6-NEXT:    buffer_store_short v0, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v3i15:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0x7fff
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
 ; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX6-NEXT:    buffer_store_short v0, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_v3i15:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0x7fff
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX6-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
 ; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX6-NEXT:    buffer_store_short v0, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_v3i15:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_i32 s1, s4, 0xf0000

 ; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
 ; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3
 ; GFX6-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_waitcnt expcnt(0)
 ; GFX6-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX6-NEXT:    buffer_store_short v0, off, s[4:7], 0 offset:4
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_v3i15:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0x7fff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s4, s8

 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s0, v0
 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, s0, v0
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 1, v1
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v0, 20, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i32_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mul_hi_u32 s0, s4, 0xb2a50881
 ; GFX9-NEXT:    s_sub_i32 s1, s4, s0

 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_lshr_b32 s0, s0, 12
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i32_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshr_b32 s0, s4, 12
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s0

 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_add_i32 s1, s1, 12
 ; GFX6-NEXT:    s_lshr_b32 s0, s0, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i32_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_add_i32 s0, s5, 12
 ; GFX9-NEXT:    s_lshr_b32 s0, s4, s0

 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_lshr_b32 s0, s0, 12
 ; GFX6-NEXT:    s_lshr_b32 s1, s1, 12
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v2i32_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshr_b32 s0, s4, 12
 ; GFX9-NEXT:    s_lshr_b32 s1, s5, 12

 ; GFX6-NEXT:    s_lshr_b32 s0, s0, 12
 ; GFX6-NEXT:    v_sub_i32_e32 v1, vcc, s1, v0
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 1, v1
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 11, v0
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v2i32_mixed_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mul_hi_u32 s1, s5, 0x100101
 ; GFX9-NEXT:    s_lshr_b32 s0, s4, 12

 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, v1, v3, s[0:1]
 ; GFX6-NEXT:    v_subrev_i32_e32 v3, vcc, s9, v2
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, v2, v3, s[0:1]
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, 1, v1
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s9, v2
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v2i32_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s4, 0x1000
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b32 s7, s4, s2
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s7
 ; GFX9-NEXT:    s_lshl_b32 s6, s4, s3

 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 1, v1
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v0, 20, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i32_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mul_hi_u32 s0, s4, 0xb2a50881
 ; GFX9-NEXT:    s_sub_i32 s1, s4, s0

 ; GFX6-NEXT:    s_load_dword s0, s[0:1], 0xb
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_and_b32 s0, s0, 0xfff
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i32_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s4, 0xfff
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s0

 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_lshl_b32 s1, 0x1000, s1
 ; GFX6-NEXT:    s_add_i32 s1, s1, -1
 ; GFX6-NEXT:    s_and_b32 s0, s0, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i32_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b32 s0, 0x1000, s5
 ; GFX9-NEXT:    s_add_i32 s0, s0, -1

 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_and_b32 s0, s0, s2
 ; GFX6-NEXT:    s_and_b32 s1, s1, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_v2i32_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_movk_i32 s0, 0xfff
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s1, s4, s0

 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s3, v1
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s3, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s3, v1
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s3, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_v2i32_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s4, 0x1000
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b32 s5, s4, s3
 ; GFX9-NEXT:    s_lshl_b32 s4, s4, s2
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s4

 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    v_mul_hi_i32 v0, s0, v0
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, s0, v0
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 31, v0
 ; GFX6-NEXT:    v_ashrrev_i32_e32 v0, 20, v0
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i32_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mul_hi_i32 s0, s4, 0xd9528441
 ; GFX9-NEXT:    s_add_i32 s0, s0, s4

 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_ashr_i32 s1, s0, 31
 ; GFX6-NEXT:    s_lshr_b32 s1, s1, 20
 ; GFX6-NEXT:    s_add_i32 s0, s0, s1
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, 12
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i32_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s0, s4, 31
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, 20

 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s7, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i32_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b32 s3, 0x1000, s3
 ; GFX9-NEXT:    s_ashr_i32 s4, s3, 31

 ; GFX6-NEXT:    s_lshr_b32 s2, s3, 20
 ; GFX6-NEXT:    s_add_i32 s1, s1, s2
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, 12
 ; GFX6-NEXT:    s_ashr_i32 s1, s1, 12
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_v2i32_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s0, s4, 31
 ; GFX9-NEXT:    s_ashr_i32 s1, s5, 31

 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, s1, v0
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 31, v0
 ; GFX6-NEXT:    v_ashrrev_i32_e32 v0, 11, v0
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, 12
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v0
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: ssdiv_v2i32_mixed_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s0, s4, 31
 ; GFX9-NEXT:    s_mul_hi_i32 s1, s5, 0x80080081

 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, v2, v3, s[0:1]
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, 1, v1
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s10, v2
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s2, v1
 ; GFX6-NEXT:    v_subrev_i32_e32 v1, vcc, s2, v1
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_v2i32_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0x1000
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x2c
 ; GFX9-NEXT:    s_mov_b32 s11, 0x4f7ffffe
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0

 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 31, v0
 ; GFX6-NEXT:    v_ashrrev_i32_e32 v0, 20, v0
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i32_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_mul_hi_i32 s0, s4, 0xd9528441
 ; GFX9-NEXT:    s_add_i32 s0, s0, s4

 ; GFX6-NEXT:    s_ashr_i32 s1, s0, 31
 ; GFX6-NEXT:    s_lshr_b32 s1, s1, 20
 ; GFX6-NEXT:    s_add_i32 s1, s0, s1
 ; GFX6-NEXT:    s_and_b32 s1, s1, 0xfffff000
 ; GFX6-NEXT:    s_sub_i32 s0, s0, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i32_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s0, s4, 31
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, 20

 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v1, vcc, s6, v0
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s6, v0
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s4, v0
 ; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i32_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x2c
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b32 s3, 0x1000, s3
 ; GFX9-NEXT:    s_ashr_i32 s4, s3, 31

 ; GFX6-NEXT:    s_lshr_b32 s3, s3, 20
 ; GFX6-NEXT:    s_add_i32 s3, s1, s3
 ; GFX6-NEXT:    s_and_b32 s2, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s1, s1, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_v2i32_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_movk_i32 s6, 0xf000
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s0, s4, 31

 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s8, v1
 ; GFX6-NEXT:    v_cmp_le_u32_e32 vcc, s8, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s0, v1
 ; GFX6-NEXT:    v_subrev_i32_e32 v1, vcc, s0, v1
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_v2i32_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0x1000
 ; GFX9-NEXT:    s_mov_b32 s9, 0x4f7ffffe
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX6-NEXT:    v_cmp_eq_u32_e32 vcc, s3, v2
 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, v6, v3, vcc
 ; GFX6-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, v7, v5, s[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v4, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[8:11], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i64_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0x4f176a73
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0x4f800000
 ; GFX9-NEXT:    v_madmk_f32 v0, v1, 0x438f8000, v0
 ; GFX9-NEXT:    v_rcp_f32_e32 v0, v0
 ; GFX9-NEXT:    s_movk_i32 s4, 0xfee0
 ; GFX9-NEXT:    s_mov_b32 s5, 0x68958c89

 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_mov_b32 s4, s0
 ; GFX6-NEXT:    s_mov_b32 s5, s1
 ; GFX6-NEXT:    s_lshr_b64 s[0:1], s[2:3], 12
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i64_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshr_b64 s[2:3], s[2:3], 12
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s3

 ; GFX6-NEXT:    s_mov_b32 s0, s4
 ; GFX6-NEXT:    s_add_i32 s8, s8, 12
 ; GFX6-NEXT:    s_mov_b32 s1, s5
 ; GFX6-NEXT:    s_lshr_b64 s[4:5], s[6:7], s8
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_i64_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_add_i32 s2, s2, 12
 ; GFX9-NEXT:    s_lshr_b64 s[0:1], s[6:7], s2

 ; GFX6-NEXT:    s_lshr_b64 s[0:1], s[0:1], 12
 ; GFX6-NEXT:    s_lshr_b64 s[2:3], s[2:3], 12
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v2i64_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshr_b64 s[0:1], s[4:5], 12
 ; GFX9-NEXT:    s_lshr_b64 s[4:5], s[6:7], 12

 ; GFX6-NEXT:    v_cmp_ne_u32_e64 s[0:1], 0, v2
 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, v1, v3, s[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v7, v5, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, v0, v1, s[0:1]
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v2i64_mixed_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0x4f800000
 ; GFX9-NEXT:    v_madak_f32 v0, 0, v0, 0x457ff000
 ; GFX9-NEXT:    v_rcp_f32_e32 v0, v0
 ; GFX9-NEXT:    s_movk_i32 s4, 0xf001
 ; GFX9-NEXT:    v_mov_b32_e32 v7, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0

 ; GFX6-NEXT:    s_lshr_b64 s[0:1], s[8:9], s0
 ; GFX6-NEXT:    s_lshr_b64 s[2:3], s[10:11], s2
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: udiv_v2i64_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x44
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_add_i32 s0, s8, 12

 ; GFX6-NEXT:    v_cmp_eq_u32_e32 vcc, s12, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v5, v5, v6, vcc
 ; GFX6-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v5
 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, v4, v3, s[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[8:11], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i64_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0x4f1761f8
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0x4f800000
 ; GFX9-NEXT:    v_madmk_f32 v0, v1, 0x438f8000, v0
 ; GFX9-NEXT:    v_rcp_f32_e32 v0, v0
 ; GFX9-NEXT:    s_movk_i32 s4, 0xfee0
 ; GFX9-NEXT:    s_mov_b32 s5, 0x689e0837

 ; GFX6-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_mov_b32 s0, s4
 ; GFX6-NEXT:    s_and_b32 s4, s6, 0xfff
 ; GFX6-NEXT:    s_mov_b32 s1, s5
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i64_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s2, s2, 0xfff
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX9-NEXT:    global_store_dwordx2 v1, v[0:1], s[0:1]

 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dword s8, s[0:1], 0xd
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_mov_b32 s0, s4
 ; GFX6-NEXT:    s_mov_b32 s1, s5
-; GFX6-NEXT:    s_mov_b32 s5, 0
-; GFX6-NEXT:    s_movk_i32 s4, 0x1000
+; GFX6-NEXT:    s_mov_b64 s[4:5], 0x1000
 ; GFX6-NEXT:    s_lshl_b64 s[4:5], s[4:5], s8
 ; GFX6-NEXT:    s_add_u32 s4, s4, -1
 ; GFX6-NEXT:    s_addc_u32 s5, s5, -1
 ; GFX6-NEXT:    s_and_b64 s[4:5], s[6:7], s[4:5]
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_i64_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x34
-; GFX9-NEXT:    s_mov_b32 s1, 0
-; GFX9-NEXT:    s_movk_i32 s0, 0x1000
+; GFX9-NEXT:    s_mov_b64 s[0:1], 0x1000
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; GFX9-NEXT:    s_add_u32 s0, s0, -1
 ; GFX9-NEXT:    s_addc_u32 s1, s1, -1
 ; GFX9-NEXT:    s_and_b64 s[0:1], s[6:7], s[0:1]
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1

 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_and_b32 s0, s0, s8
 ; GFX6-NEXT:    s_and_b32 s1, s2, s8
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v3, v1
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_v2i64_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s0, 0xfff
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v3, v1
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; CHECK-NEXT:    store <2 x i64> [[TMP8]], <2 x i64> addrspace(1)* [[OUT:%.*]], align 16
 ; CHECK-NEXT:    ret void
 ;
 ; GFX6-LABEL: urem_v2i64_pow2_shl_denom:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0xd
 ; GFX6-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x11
-; GFX6-NEXT:    s_mov_b32 s13, 0
-; GFX6-NEXT:    s_movk_i32 s12, 0x1000
+; GFX6-NEXT:    s_mov_b64 s[12:13], 0x1000
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_lshl_b64 s[2:3], s[12:13], s2
 ; GFX6-NEXT:    s_lshl_b64 s[0:1], s[12:13], s0
 ; GFX6-NEXT:    s_add_u32 s0, s0, -1
 ; GFX6-NEXT:    s_addc_u32 s1, s1, -1
 ; GFX6-NEXT:    s_and_b64 s[0:1], s[8:9], s[0:1]
 ; GFX6-NEXT:    s_add_u32 s2, s2, -1
 ; GFX6-NEXT:    s_addc_u32 s3, s3, -1
 ; GFX6-NEXT:    s_and_b64 s[2:3], s[10:11], s[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: urem_v2i64_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x44
-; GFX9-NEXT:    s_mov_b32 s1, 0
-; GFX9-NEXT:    s_movk_i32 s0, 0x1000
+; GFX9-NEXT:    s_mov_b64 s[0:1], 0x1000
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b64 s[10:11], s[0:1], s10
 ; GFX9-NEXT:    s_lshl_b64 s[0:1], s[0:1], s8
 ; GFX9-NEXT:    s_add_u32 s0, s0, -1
 ; GFX9-NEXT:    s_addc_u32 s1, s1, -1
 ; GFX9-NEXT:    s_and_b64 s[0:1], s[4:5], s[0:1]
 ; GFX9-NEXT:    s_add_u32 s4, s10, -1

 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, v1, v3, s[0:1]
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s2, v0
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s2, v1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, s2, v0
 ; GFX6-NEXT:    v_subb_u32_e32 v1, vcc, v1, v2, vcc
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i64_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0x4f800000
 ; GFX9-NEXT:    v_madak_f32 v0, 0, v0, 0x4996c7d8
 ; GFX9-NEXT:    v_rcp_f32_e32 v0, v0
 ; GFX9-NEXT:    s_mov_b32 s8, 0xffed2705
 ; GFX9-NEXT:    v_mov_b32_e32 v7, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0

 ; GFX6-NEXT:    s_add_u32 s0, s2, s0
 ; GFX6-NEXT:    s_mov_b32 s5, s1
 ; GFX6-NEXT:    s_addc_u32 s1, s3, 0
 ; GFX6-NEXT:    s_ashr_i64 s[0:1], s[0:1], 12
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i64_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s4, s3, 31
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 20
 ; GFX9-NEXT:    s_add_u32 s2, s2, s4

 ; CHECK-NEXT:    [[SHL_Y:%.*]] = shl i64 4096, [[Y:%.*]]
 ; CHECK-NEXT:    [[R:%.*]] = sdiv i64 [[X:%.*]], [[SHL_Y]]
 ; CHECK-NEXT:    store i64 [[R]], i64 addrspace(1)* [[OUT:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 ; GFX6-LABEL: sdiv_i64_pow2_shl_denom:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dword s4, s[0:1], 0xd
-; GFX6-NEXT:    s_mov_b32 s3, 0
-; GFX6-NEXT:    s_movk_i32 s2, 0x1000
+; GFX6-NEXT:    s_mov_b64 s[2:3], 0x1000
 ; GFX6-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x9
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
+; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX6-NEXT:    s_ashr_i32 s12, s3, 31
 ; GFX6-NEXT:    s_add_u32 s2, s2, s12
 ; GFX6-NEXT:    s_mov_b32 s13, s12
 ; GFX6-NEXT:    s_addc_u32 s3, s3, s12
 ; GFX6-NEXT:    s_xor_b64 s[2:3], s[2:3], s[12:13]
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v0, s2
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v1, s3
 ; GFX6-NEXT:    s_sub_u32 s4, 0, s2
 ; GFX6-NEXT:    s_subb_u32 s5, 0, s3
 ; GFX6-NEXT:    s_ashr_i32 s14, s11, 31
 ; GFX6-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; GFX6-NEXT:    v_rcp_f32_e32 v0, v0
 ; GFX6-NEXT:    s_mov_b32 s15, s14
-; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0
 ; GFX6-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX6-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s4, v1

 ; GFX6-NEXT:    v_cndmask_b32_e32 v1, v1, v4, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s0, v0
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s1, v1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s1
 ; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, s0, v0
 ; GFX6-NEXT:    v_subb_u32_e32 v1, vcc, v1, v2, vcc
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_i64_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x34
-; GFX9-NEXT:    s_mov_b32 s3, 0
-; GFX9-NEXT:    s_movk_i32 s2, 0x1000
+; GFX9-NEXT:    s_mov_b64 s[2:3], 0x1000
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX9-NEXT:    s_ashr_i32 s8, s3, 31
 ; GFX9-NEXT:    s_add_u32 s2, s2, s8
 ; GFX9-NEXT:    s_mov_b32 s9, s8
 ; GFX9-NEXT:    s_addc_u32 s3, s3, s8
 ; GFX9-NEXT:    s_xor_b64 s[10:11], s[2:3], s[8:9]

 ; GFX6-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX6-NEXT:    s_ashr_i64 s[2:3], s[2:3], 12
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_v2i64_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s0, s5, 31
 ; GFX9-NEXT:    s_lshr_b32 s0, s0, 20

 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s8, v0
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s8, v1
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s8
 ; GFX6-NEXT:    v_subb_u32_e32 v3, vcc, v1, v3, vcc
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: ssdiv_v2i64_mixed_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0x457ff000
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0x4f800000
 ; GFX9-NEXT:    v_mac_f32_e32 v0, 0, v1
 ; GFX9-NEXT:    v_rcp_f32_e32 v0, v0
 ; GFX9-NEXT:    s_movk_i32 s8, 0xf001
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34

 ; CHECK-NEXT:    [[TMP7:%.*]] = sdiv i64 [[TMP5]], [[TMP6]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x i64> [[TMP4]], i64 [[TMP7]], i64 1
 ; CHECK-NEXT:    store <2 x i64> [[TMP8]], <2 x i64> addrspace(1)* [[OUT:%.*]], align 16
 ; CHECK-NEXT:    ret void
 ;
 ; GFX6-LABEL: sdiv_v2i64_pow2_shl_denom:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x11
-; GFX6-NEXT:    s_mov_b32 s3, 0
-; GFX6-NEXT:    s_movk_i32 s2, 0x1000
+; GFX6-NEXT:    s_mov_b64 s[2:3], 0x1000
 ; GFX6-NEXT:    s_mov_b32 s18, 0x4f800000
 ; GFX6-NEXT:    s_mov_b32 s19, 0x5f7ffffc
+; GFX6-NEXT:    s_mov_b32 s20, 0x2f800000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_lshl_b64 s[12:13], s[2:3], s6
 ; GFX6-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX6-NEXT:    s_ashr_i32 s16, s3, 31
 ; GFX6-NEXT:    s_add_u32 s2, s2, s16
 ; GFX6-NEXT:    s_mov_b32 s17, s16
 ; GFX6-NEXT:    s_addc_u32 s3, s3, s16
 ; GFX6-NEXT:    s_xor_b64 s[14:15], s[2:3], s[16:17]
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v0, s14
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v1, s15
-; GFX6-NEXT:    s_mov_b32 s20, 0x2f800000
 ; GFX6-NEXT:    s_mov_b32 s21, 0xcf800000
 ; GFX6-NEXT:    s_sub_u32 s6, 0, s14
+; GFX6-NEXT:    s_subb_u32 s7, 0, s15
 ; GFX6-NEXT:    v_mac_f32_e32 v0, s18, v1
 ; GFX6-NEXT:    v_rcp_f32_e32 v0, v0
-; GFX6-NEXT:    s_subb_u32 s7, 0, s15
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0xd
 ; GFX6-NEXT:    v_mul_f32_e32 v0, s19, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, s20, v0
 ; GFX6-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX6-NEXT:    v_mac_f32_e32 v0, s21, v1
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1

 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v2, s0, v2
 ; GFX6-NEXT:    v_xor_b32_e32 v3, s1, v3
 ; GFX6-NEXT:    v_mov_b32_e32 v4, s1
 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s0, v2
 ; GFX6-NEXT:    v_subb_u32_e32 v3, vcc, v3, v4, vcc
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: sdiv_v2i64_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x44
-; GFX9-NEXT:    s_mov_b32 s3, 0
-; GFX9-NEXT:    s_movk_i32 s2, 0x1000
+; GFX9-NEXT:    s_mov_b64 s[2:3], 0x1000
 ; GFX9-NEXT:    s_mov_b32 s18, 0x4f800000
 ; GFX9-NEXT:    s_mov_b32 s19, 0x5f7ffffc
+; GFX9-NEXT:    s_mov_b32 s20, 0x2f800000
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b64 s[8:9], s[2:3], s6
 ; GFX9-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX9-NEXT:    s_ashr_i32 s12, s3, 31
 ; GFX9-NEXT:    s_add_u32 s2, s2, s12
 ; GFX9-NEXT:    s_mov_b32 s13, s12
 ; GFX9-NEXT:    s_addc_u32 s3, s3, s12
 ; GFX9-NEXT:    s_xor_b64 s[10:11], s[2:3], s[12:13]
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s10
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, s11
-; GFX9-NEXT:    s_mov_b32 s20, 0x2f800000
 ; GFX9-NEXT:    s_mov_b32 s21, 0xcf800000
 ; GFX9-NEXT:    s_sub_u32 s14, 0, s10
+; GFX9-NEXT:    s_subb_u32 s4, 0, s11
 ; GFX9-NEXT:    v_mac_f32_e32 v0, s18, v1
 ; GFX9-NEXT:    v_rcp_f32_e32 v0, v0
-; GFX9-NEXT:    s_subb_u32 s4, 0, s11
 ; GFX9-NEXT:    v_mov_b32_e32 v6, 0
 ; GFX9-NEXT:    v_mul_f32_e32 v0, s19, v0
 ; GFX9-NEXT:    v_mul_f32_e32 v1, s20, v0
 ; GFX9-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mac_f32_e32 v0, s21, v1
 ; GFX9-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX9-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mul_hi_u32 v3, s14, v0

 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, v1, v3, s[0:1]
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s2, v0
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s2, v1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, s2, v0
 ; GFX6-NEXT:    v_subb_u32_e32 v1, vcc, v1, v2, vcc
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i64_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0x4f800000
 ; GFX9-NEXT:    v_madak_f32 v0, 0, v0, 0x4996c7d8
 ; GFX9-NEXT:    v_rcp_f32_e32 v0, v0
 ; GFX9-NEXT:    s_mov_b32 s8, 0xffed2705
 ; GFX9-NEXT:    v_mov_b32_e32 v7, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0

 ; GFX6-NEXT:    s_addc_u32 s5, s7, 0
 ; GFX6-NEXT:    s_and_b32 s4, s4, 0xfffff000
 ; GFX6-NEXT:    s_sub_u32 s4, s6, s4
 ; GFX6-NEXT:    s_subb_u32 s5, s7, s5
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i64_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s4, s3, 31
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 20
 ; GFX9-NEXT:    s_add_u32 s4, s2, s4

 ; CHECK-NEXT:    [[SHL_Y:%.*]] = shl i64 4096, [[Y:%.*]]
 ; CHECK-NEXT:    [[R:%.*]] = srem i64 [[X:%.*]], [[SHL_Y]]
 ; CHECK-NEXT:    store i64 [[R]], i64 addrspace(1)* [[OUT:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 ; GFX6-LABEL: srem_i64_pow2_shl_denom:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dword s4, s[0:1], 0xd
-; GFX6-NEXT:    s_mov_b32 s3, 0
-; GFX6-NEXT:    s_movk_i32 s2, 0x1000
+; GFX6-NEXT:    s_mov_b64 s[2:3], 0x1000
 ; GFX6-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x9
 ; GFX6-NEXT:    s_mov_b32 s7, 0xf000
+; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX6-NEXT:    s_ashr_i32 s4, s3, 31
 ; GFX6-NEXT:    s_add_u32 s2, s2, s4
 ; GFX6-NEXT:    s_mov_b32 s5, s4
 ; GFX6-NEXT:    s_addc_u32 s3, s3, s4
 ; GFX6-NEXT:    s_xor_b64 s[12:13], s[2:3], s[4:5]
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v0, s12
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v1, s13
 ; GFX6-NEXT:    s_sub_u32 s2, 0, s12
 ; GFX6-NEXT:    s_subb_u32 s3, 0, s13
 ; GFX6-NEXT:    s_ashr_i32 s14, s11, 31
 ; GFX6-NEXT:    v_mac_f32_e32 v0, 0x4f800000, v1
 ; GFX6-NEXT:    v_rcp_f32_e32 v0, v0
 ; GFX6-NEXT:    s_mov_b32 s15, s14
-; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    s_mov_b32 s4, s8
+; GFX6-NEXT:    s_mov_b32 s5, s9
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0
 ; GFX6-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX6-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1
-; GFX6-NEXT:    s_mov_b32 s5, s9
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s2, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s2, v1
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s3, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s2, v0
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v0, v2

 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s14, v0
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s14, v1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s14
 ; GFX6-NEXT:    v_subrev_i32_e32 v0, vcc, s14, v0
 ; GFX6-NEXT:    v_subb_u32_e32 v1, vcc, v1, v2, vcc
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_i64_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x34
-; GFX9-NEXT:    s_mov_b32 s3, 0
-; GFX9-NEXT:    s_movk_i32 s2, 0x1000
+; GFX9-NEXT:    s_mov_b64 s[2:3], 0x1000
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX9-NEXT:    s_ashr_i32 s4, s3, 31
 ; GFX9-NEXT:    s_add_u32 s2, s2, s4
 ; GFX9-NEXT:    s_mov_b32 s5, s4
 ; GFX9-NEXT:    s_addc_u32 s3, s3, s4
 ; GFX9-NEXT:    s_xor_b64 s[8:9], s[2:3], s[4:5]

 ; GFX6-NEXT:    s_sub_u32 s2, s2, s8
 ; GFX6-NEXT:    s_subb_u32 s3, s3, s10
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_v2i64_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0xf000
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_ashr_i32 s0, s5, 31

 ; CHECK-NEXT:    [[TMP7:%.*]] = srem i64 [[TMP5]], [[TMP6]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x i64> [[TMP4]], i64 [[TMP7]], i64 1
 ; CHECK-NEXT:    store <2 x i64> [[TMP8]], <2 x i64> addrspace(1)* [[OUT:%.*]], align 16
 ; CHECK-NEXT:    ret void
 ;
 ; GFX6-LABEL: srem_v2i64_pow2_shl_denom:
 ; GFX6:       ; %bb.0:
 ; GFX6-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x11
-; GFX6-NEXT:    s_mov_b32 s3, 0
-; GFX6-NEXT:    s_movk_i32 s2, 0x1000
+; GFX6-NEXT:    s_mov_b64 s[2:3], 0x1000
 ; GFX6-NEXT:    s_mov_b32 s18, 0x4f800000
 ; GFX6-NEXT:    s_mov_b32 s19, 0x5f7ffffc
+; GFX6-NEXT:    s_mov_b32 s20, 0x2f800000
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    s_lshl_b64 s[14:15], s[2:3], s6
 ; GFX6-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX6-NEXT:    s_ashr_i32 s4, s3, 31
 ; GFX6-NEXT:    s_add_u32 s2, s2, s4
 ; GFX6-NEXT:    s_mov_b32 s5, s4
 ; GFX6-NEXT:    s_addc_u32 s3, s3, s4
 ; GFX6-NEXT:    s_xor_b64 s[16:17], s[2:3], s[4:5]
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v0, s16
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v1, s17
-; GFX6-NEXT:    s_mov_b32 s20, 0x2f800000
 ; GFX6-NEXT:    s_mov_b32 s21, 0xcf800000
 ; GFX6-NEXT:    s_sub_u32 s6, 0, s16
+; GFX6-NEXT:    s_subb_u32 s7, 0, s17
 ; GFX6-NEXT:    v_mac_f32_e32 v0, s18, v1
 ; GFX6-NEXT:    v_rcp_f32_e32 v0, v0
-; GFX6-NEXT:    s_subb_u32 s7, 0, s17
 ; GFX6-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; GFX6-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0xd
 ; GFX6-NEXT:    v_mul_f32_e32 v0, s19, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, s20, v0
 ; GFX6-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX6-NEXT:    v_mac_f32_e32 v0, s21, v1
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1

 ; GFX6-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; GFX6-NEXT:    v_xor_b32_e32 v2, s14, v2
 ; GFX6-NEXT:    v_xor_b32_e32 v3, s14, v3
 ; GFX6-NEXT:    v_mov_b32_e32 v4, s14
 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s14, v2
 ; GFX6-NEXT:    v_subb_u32_e32 v3, vcc, v3, v4, vcc
 ; GFX6-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
+;
 ; GFX9-LABEL: srem_v2i64_pow2_shl_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x44
-; GFX9-NEXT:    s_mov_b32 s3, 0
-; GFX9-NEXT:    s_movk_i32 s2, 0x1000
+; GFX9-NEXT:    s_mov_b64 s[2:3], 0x1000
 ; GFX9-NEXT:    s_mov_b32 s16, 0x4f800000
 ; GFX9-NEXT:    s_mov_b32 s17, 0x5f7ffffc
+; GFX9-NEXT:    s_mov_b32 s18, 0x2f800000
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshl_b64 s[12:13], s[2:3], s6
 ; GFX9-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX9-NEXT:    s_ashr_i32 s4, s3, 31
 ; GFX9-NEXT:    s_add_u32 s2, s2, s4
 ; GFX9-NEXT:    s_mov_b32 s5, s4
 ; GFX9-NEXT:    s_addc_u32 s3, s3, s4
 ; GFX9-NEXT:    s_xor_b64 s[14:15], s[2:3], s[4:5]
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s14
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, s15
-; GFX9-NEXT:    s_mov_b32 s18, 0x2f800000
 ; GFX9-NEXT:    s_mov_b32 s19, 0xcf800000
 ; GFX9-NEXT:    s_sub_u32 s4, 0, s14
+; GFX9-NEXT:    s_subb_u32 s5, 0, s15
 ; GFX9-NEXT:    v_mac_f32_e32 v0, s16, v1
 ; GFX9-NEXT:    v_rcp_f32_e32 v0, v0
-; GFX9-NEXT:    s_subb_u32 s5, 0, s15
 ; GFX9-NEXT:    v_mov_b32_e32 v6, 0
 ; GFX9-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x34
 ; GFX9-NEXT:    v_mul_f32_e32 v0, s17, v0
 ; GFX9-NEXT:    v_mul_f32_e32 v1, s18, v0
 ; GFX9-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX9-NEXT:    v_mac_f32_e32 v0, s19, v1
 ; GFX9-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX9-NEXT:    v_cvt_u32_f32_e32 v1, v1

llvm/test/CodeGen/AMDGPU/combine-sreg64-inits.mir

+# RUN: llc -march=amdgcn -verify-machineinstrs -run-pass=liveintervals,amdgpu-pre-ra-optimizations %s -o - | FileCheck -check-prefix=GCN %s
+
+---
+# GCN-LABEL: name: combine_sreg64_inits
+# GCN: %0:sgpr_64 = S_MOV_B64_IMM_PSEUDO 8589934593
+# GCN: S_NOP 0
+name:            combine_sreg64_inits
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    undef %0.sub0:sgpr_64 = S_MOV_B32 1
+    S_NOP 0
+    %0.sub1:sgpr_64 = S_MOV_B32 2
+...
+---
+# GCN-LABEL: name: combine_sreg64_inits_swap
+# GCN: %0:sgpr_64 = S_MOV_B64_IMM_PSEUDO 8589934593
+# GCN: S_NOP 0
+name:            combine_sreg64_inits_swap
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    undef %0.sub1:sgpr_64 = S_MOV_B32 2
+    S_NOP 0
+    %0.sub0:sgpr_64 = S_MOV_B32 1
+...
+---
+# GCN-LABEL: name: sreg64_inits_different_blocks
+# GCN: undef %0.sub0:sgpr_64 = S_MOV_B32 1
+# GCN: %0.sub1:sgpr_64 = S_MOV_B32 2
+name:            sreg64_inits_different_blocks
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    undef %0.sub0:sgpr_64 = S_MOV_B32 1
+
+  bb.1:
+    %0.sub1:sgpr_64 = S_MOV_B32 2
+...
+---
+# GCN-LABEL: name: sreg64_inits_two_defs_sub1
+# GCN: undef %0.sub0:sgpr_64 = S_MOV_B32 1
+# GCN: %0.sub1:sgpr_64 = S_MOV_B32 2
+# GCN: %0.sub1:sgpr_64 = S_MOV_B32 3
+name:            sreg64_inits_two_defs_sub1
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    undef %0.sub0:sgpr_64 = S_MOV_B32 1
+    %0.sub1:sgpr_64 = S_MOV_B32 2
+    %0.sub1:sgpr_64 = S_MOV_B32 3
+...
+---
+# GCN-LABEL: name: sreg64_inits_two_defs_sub0
+# GCN: undef %0.sub0:sgpr_64 = S_MOV_B32 1
+# GCN: %0.sub1:sgpr_64 = S_MOV_B32 2
+# GCN: %0.sub0:sgpr_64 = S_MOV_B32 3
+name:            sreg64_inits_two_defs_sub0
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    undef %0.sub0:sgpr_64 = S_MOV_B32 1
+    %0.sub1:sgpr_64 = S_MOV_B32 2
+    %0.sub0:sgpr_64 = S_MOV_B32 3
+...
+---
+# GCN-LABEL: name: sreg64_inits_full_def
+# GCN: undef %1.sub0:sgpr_64 = S_MOV_B32 1
+# GCN: %0:sgpr_64 = S_MOV_B64 3
+name:            sreg64_inits_full_def
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    undef %0.sub0:sgpr_64 = S_MOV_B32 1
+    %0:sgpr_64 = S_MOV_B64 3
+...
+---
+# GCN-LABEL: name: sreg64_inits_imp_use
+# GCN: %0.sub0:sgpr_64 = S_MOV_B32 1, implicit $m0
+# GCN: %0.sub1:sgpr_64 = S_MOV_B32 2
+name:            sreg64_inits_imp_use
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    undef %0.sub0:sgpr_64 = S_MOV_B32 1, implicit $m0
+    %0.sub1:sgpr_64 = S_MOV_B32 2
+...
+---
+# GCN-LABEL: name: sreg64_inits_imp_def
+# GCN: %0.sub0:sgpr_64 = S_MOV_B32 1, implicit-def $scc
+# GCN: %0.sub1:sgpr_64 = S_MOV_B32 2
+name:            sreg64_inits_imp_def
+tracksRegLiveness: true
+body:             |
+  bb.0:
+    undef %0.sub0:sgpr_64 = S_MOV_B32 1, implicit-def $scc
+    %0.sub1:sgpr_64 = S_MOV_B32 2
+...

llvm/test/CodeGen/AMDGPU/ds_write2.ll

; GFX9-NEXT:    s_endpgm
   ret void
 }
 
 @bar.large = addrspace(3) global [4096 x i64] undef, align 4
 
 define amdgpu_kernel void @store_misaligned64_constant_large_offsets() {
 ; CI-LABEL: store_misaligned64_constant_large_offsets:
 ; CI:       ; %bb.0:
-; CI-NEXT:    s_movk_i32 s0, 0x7b
-; CI-NEXT:    s_mov_b32 s1, 0
+; CI-NEXT:    s_mov_b64 s[0:1], 0x7b
 ; CI-NEXT:    v_mov_b32_e32 v0, s0
 ; CI-NEXT:    v_mov_b32_e32 v2, 0
 ; CI-NEXT:    v_mov_b32_e32 v1, s1
 ; CI-NEXT:    s_mov_b32 m0, -1
 ; CI-NEXT:    ds_write_b64 v2, v[0:1] offset:16384
 ; CI-NEXT:    ds_write_b64 v2, v[0:1] offset:32760
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: store_misaligned64_constant_large_offsets:
 ; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_movk_i32 s0, 0x7b
-; GFX9-NEXT:    s_mov_b32 s1, 0
+; GFX9-NEXT:    s_mov_b64 s[0:1], 0x7b
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    ds_write_b64 v2, v[0:1] offset:16384
 ; GFX9-NEXT:    ds_write_b64 v2, v[0:1] offset:32760
 ; GFX9-NEXT:    s_endpgm
   store i64 123, i64 addrspace(3)* getelementptr inbounds ([4096 x i64], [4096 x i64] addrspace(3)* @bar.large, i32 0, i32 2048), align 4
   store i64 123, i64 addrspace(3)* getelementptr inbounds ([4096 x i64], [4096 x i64] addrspace(3)* @bar.large, i32 0, i32 4095), align 4

llvm/test/CodeGen/AMDGPU/inline-asm.ll

; separate comment
     ; trailing comment
     ; extra comment
   ", ""()
   ret void
 }
 
 ; FIXME: Should not have intermediate sgprs
 ; CHECK-LABEL: {{^}}i64_imm_input_phys_vgpr:
-; CHECK-DAG: s_mov_b32 s1, 0
-; CHECK-DAG: s_mov_b32 s0, 0x1e240
+; CHECK: s_mov_b64 s[0:1], 0x1e240
 ; CHECK: v_mov_b32_e32 v0, s0
 ; CHECK: v_mov_b32_e32 v1, s1
 ; CHECK: use v[0:1]
 define amdgpu_kernel void @i64_imm_input_phys_vgpr() {
 entry:
   call void asm sideeffect "; use $0 ", "{v[0:1]}"(i64 123456)
   ret void
 }

llvm/test/CodeGen/AMDGPU/inline-constraints.ll

 
 ; GCN-LABEL: {{^}}inline_sreg_constraint_imm_f32:
 ; GCN: s_mov_b32 [[REG:s[0-9]+]], 1.0
 ; GCN: ; use [[REG]]
 define amdgpu_kernel void @inline_sreg_constraint_imm_f32() {
   tail call void asm sideeffect "; use $0", "s"(float 1.0)
   ret void
 }
 
-; FIXME: Should be able to use s_mov_b64
 ; GCN-LABEL: {{^}}inline_sreg_constraint_imm_i64:
-; GCN-DAG: s_mov_b32 s[[REG_LO:[0-9]+]], -4{{$}}
-; GCN-DAG: s_mov_b32 s[[REG_HI:[0-9]+]], -1{{$}}
-; GCN: ; use s{{\[}}[[REG_LO]]:[[REG_HI]]{{\]}}
+; GCN: s_mov_b64 [[REG:s\[[0-9:]+\]]], -4{{$}}
+; GCN: ; use [[REG]]
 define amdgpu_kernel void @inline_sreg_constraint_imm_i64() {
   tail call void asm sideeffect "; use $0", "s"(i64 -4)
   ret void
 }
 
 ; GCN-LABEL: {{^}}inline_sreg_constraint_imm_f64:
-; GCN-DAG: s_mov_b32 s[[REG_LO:[0-9]+]], 0{{$}}
-; GCN-DAG: s_mov_b32 s[[REG_HI:[0-9]+]], 0x3ff00000{{$}}
-; GCN: ; use s{{\[}}[[REG_LO]]:[[REG_HI]]{{\]}}
+; GCN: s_mov_b64 [[REG:s\[[0-9:]+\]]], 1.0{{$}}
+; GCN: ; use [[REG]]
 define amdgpu_kernel void @inline_sreg_constraint_imm_f64() {
   tail call void asm sideeffect "; use $0", "s"(double 1.0)
   ret void
 }
 
 ;==============================================================================
 ; 'A' constraint, 16-bit operand
 ;==============================================================================

llvm/test/CodeGen/AMDGPU/insert_vector_elt.ll

 }
 
 define amdgpu_kernel void @dynamic_insertelement_v3i16(<3 x i16> addrspace(1)* %out, <3 x i16> %a, i32 %b) nounwind {
 ; SI-LABEL: dynamic_insertelement_v3i16:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
 ; SI-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x2
 ; SI-NEXT:    s_load_dword s4, s[4:5], 0x4
-; SI-NEXT:    s_mov_b32 s5, 0
 ; SI-NEXT:    s_mov_b32 s3, 0x100f000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_lshl_b32 s8, s4, 4
-; SI-NEXT:    s_mov_b32 s4, 0xffff
+; SI-NEXT:    s_mov_b64 s[4:5], 0xffff
 ; SI-NEXT:    s_lshl_b64 s[4:5], s[4:5], s8
 ; SI-NEXT:    s_mov_b32 s8, 0x50005
 ; SI-NEXT:    s_and_b32 s9, s5, s8
 ; SI-NEXT:    s_and_b32 s8, s4, s8
 ; SI-NEXT:    s_andn2_b64 s[4:5], s[6:7], s[4:5]
 ; SI-NEXT:    s_or_b64 s[4:5], s[8:9], s[4:5]
 ; SI-NEXT:    v_mov_b32_e32 v0, s5
 ; SI-NEXT:    buffer_store_short v0, off, s[0:3], 0 offset:4
 ; SI-NEXT:    v_mov_b32_e32 v0, s4
 ; SI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: dynamic_insertelement_v3i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x0
 ; VI-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x8
 ; VI-NEXT:    s_load_dword s4, s[4:5], 0x10
-; VI-NEXT:    s_mov_b32 s5, 0
 ; VI-NEXT:    s_mov_b32 s3, 0x1100f000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_lshl_b32 s8, s4, 4
-; VI-NEXT:    s_mov_b32 s4, 0xffff
+; VI-NEXT:    s_mov_b64 s[4:5], 0xffff
 ; VI-NEXT:    s_lshl_b64 s[4:5], s[4:5], s8
 ; VI-NEXT:    s_mov_b32 s8, 0x50005
 ; VI-NEXT:    s_mov_b32 s9, s8
 ; VI-NEXT:    s_andn2_b64 s[6:7], s[6:7], s[4:5]
 ; VI-NEXT:    s_and_b64 s[4:5], s[4:5], s[8:9]
 ; VI-NEXT:    s_or_b64 s[4:5], s[4:5], s[6:7]
 ; VI-NEXT:    v_mov_b32_e32 v0, s5
 ; VI-NEXT:    buffer_store_short v0, off, s[0:3], 0 offset:4
; VI-NEXT:    s_endpgm
   ret void
 }
 
 define amdgpu_kernel void @s_dynamic_insertelement_v8i8(<8 x i8> addrspace(1)* %out, <8 x i8> addrspace(4)* %a.ptr, i32 %b) nounwind {
 ; SI-LABEL: s_dynamic_insertelement_v8i8:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx4 s[8:11], s[4:5], 0x0
 ; SI-NEXT:    s_load_dword s6, s[4:5], 0x4
-; SI-NEXT:    s_mov_b32 s7, 0
 ; SI-NEXT:    s_mov_b32 s3, 0x100f000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_load_dwordx2 s[4:5], s[10:11], 0x0
 ; SI-NEXT:    s_mov_b32 s0, s8
 ; SI-NEXT:    s_lshl_b32 s8, s6, 3
-; SI-NEXT:    s_mov_b32 s6, 0xffff
+; SI-NEXT:    s_mov_b64 s[6:7], 0xffff
 ; SI-NEXT:    s_lshl_b64 s[6:7], s[6:7], s8
 ; SI-NEXT:    s_mov_b32 s8, 0x5050505
 ; SI-NEXT:    s_mov_b32 s1, s9
 ; SI-NEXT:    s_and_b32 s9, s7, s8
 ; SI-NEXT:    s_and_b32 s8, s6, s8
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_andn2_b64 s[4:5], s[4:5], s[6:7]
 ; SI-NEXT:    s_or_b64 s[4:5], s[8:9], s[4:5]
 ; SI-NEXT:    v_mov_b32_e32 v0, s4
 ; SI-NEXT:    v_mov_b32_e32 v1, s5
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_dynamic_insertelement_v8i8:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[8:11], s[4:5], 0x0
 ; VI-NEXT:    s_load_dword s6, s[4:5], 0x10
-; VI-NEXT:    s_mov_b32 s7, 0
 ; VI-NEXT:    s_mov_b32 s3, 0x1100f000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[10:11], 0x0
 ; VI-NEXT:    s_mov_b32 s0, s8
 ; VI-NEXT:    s_lshl_b32 s8, s6, 3
-; VI-NEXT:    s_mov_b32 s6, 0xffff
+; VI-NEXT:    s_mov_b64 s[6:7], 0xffff
 ; VI-NEXT:    s_lshl_b64 s[6:7], s[6:7], s8
 ; VI-NEXT:    s_mov_b32 s8, 0x5050505
 ; VI-NEXT:    s_mov_b32 s1, s9
 ; VI-NEXT:    s_and_b32 s9, s7, s8
 ; VI-NEXT:    s_and_b32 s8, s6, s8
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_andn2_b64 s[4:5], s[4:5], s[6:7]
 ; VI-NEXT:    s_or_b64 s[4:5], s[8:9], s[4:5]

llvm/test/CodeGen/AMDGPU/insert_vector_elt.v2i16.ll

 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    global_load_dword v2, v[0:1], off glc
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s6, s[4:5], 0x10
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dwordx2 v[0:1], v4, s[2:3]
-; GFX9-NEXT:    s_mov_b32 s3, 0
-; GFX9-NEXT:    s_mov_b32 s2, 0xffff
+; GFX9-NEXT:    s_mov_b64 s[2:3], 0xffff
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 4, v2
 ; GFX9-NEXT:    v_lshlrev_b64 v[2:3], v2, s[2:3]
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s2, s6, s6
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_bfi_b32 v1, v3, s2, v1
 ; GFX9-NEXT:    v_bfi_b32 v0, v2, s2, v0
 ; GFX9-NEXT:    global_store_dwordx2 v4, v[0:1], s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; VI-LABEL: v_insertelement_v4i16_dynamic_vgpr:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x0
 ; VI-NEXT:    s_load_dword s4, s[4:5], 0x10
 ; VI-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v1, s3
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, s2, v2
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; VI-NEXT:    flat_load_dword v4, v[0:1] glc
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
-; VI-NEXT:    s_mov_b32 s2, 0xffff
+; VI-NEXT:    s_mov_b64 s[2:3], 0xffff
 ; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    s_mov_b32 s3, 0
 ; VI-NEXT:    s_and_b32 s1, s4, s2
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
 ; VI-NEXT:    s_lshl_b32 s0, s1, 16
 ; VI-NEXT:    s_or_b32 s0, s1, s0
 ; VI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; VI-NEXT:    v_lshlrev_b32_e32 v4, 4, v4
 ; VI-NEXT:    v_lshlrev_b64 v[4:5], v4, s[2:3]
 ; VI-NEXT:    s_waitcnt vmcnt(0)

 ; CI-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    v_mov_b32_e32 v1, s3
 ; CI-NEXT:    v_add_i32_e32 v0, vcc, s2, v2
 ; CI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; CI-NEXT:    flat_load_dword v4, v[0:1] glc
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
-; CI-NEXT:    s_mov_b32 s3, 0
-; CI-NEXT:    s_mov_b32 s2, 0xffff
+; CI-NEXT:    s_mov_b64 s[2:3], 0xffff
 ; CI-NEXT:    v_mov_b32_e32 v3, s1
 ; CI-NEXT:    s_lshl_b32 s1, s4, 16
 ; CI-NEXT:    s_and_b32 s4, s4, s2
 ; CI-NEXT:    v_add_i32_e32 v2, vcc, s0, v2
 ; CI-NEXT:    s_or_b32 s0, s4, s1
 ; CI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; CI-NEXT:    v_lshlrev_b32_e32 v4, 4, v4
 ; CI-NEXT:    v_lshl_b64 v[4:5], s[2:3], v4

 ; GFX9-LABEL: v_insertelement_v4f16_dynamic_sgpr:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x10
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dwordx2 v[0:1], v2, s[2:3]
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s5, s6, s6
-; GFX9-NEXT:    s_mov_b32 s3, 0
-; GFX9-NEXT:    s_mov_b32 s2, 0xffff
+; GFX9-NEXT:    s_mov_b64 s[2:3], 0xffff
 ; GFX9-NEXT:    s_lshl_b32 s4, s7, 4
 ; GFX9-NEXT:    s_lshl_b64 s[2:3], s[2:3], s4
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s5
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_bfi_b32 v1, s3, v3, v1
 ; GFX9-NEXT:    v_bfi_b32 v0, s2, v4, v0
 ; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; VI-LABEL: v_insertelement_v4f16_dynamic_sgpr:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x0
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x10
 ; VI-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    v_mov_b32_e32 v1, s3
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, s2, v2
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; VI-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
-; VI-NEXT:    s_mov_b32 s2, 0xffff
+; VI-NEXT:    s_mov_b64 s[2:3], 0xffff
 ; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    s_mov_b32 s3, 0
 ; VI-NEXT:    s_lshl_b32 s1, s5, 4
 ; VI-NEXT:    s_and_b32 s4, s4, s2
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
 ; VI-NEXT:    s_lshl_b64 s[0:1], s[2:3], s1
 ; VI-NEXT:    s_lshl_b32 s2, s4, 16
 ; VI-NEXT:    s_or_b32 s2, s4, s2
 ; VI-NEXT:    v_mov_b32_e32 v4, s2
 ; VI-NEXT:    v_mov_b32_e32 v5, s2

 ; CI-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x0
 ; CI-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x4
 ; CI-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    v_mov_b32_e32 v1, s3
 ; CI-NEXT:    v_add_i32_e32 v0, vcc, s2, v2
 ; CI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; CI-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
-; CI-NEXT:    s_mov_b32 s2, 0xffff
+; CI-NEXT:    s_mov_b64 s[2:3], 0xffff
 ; CI-NEXT:    v_mov_b32_e32 v3, s1
 ; CI-NEXT:    s_and_b32 s6, s4, s2
-; CI-NEXT:    s_mov_b32 s3, 0
 ; CI-NEXT:    s_lshl_b32 s1, s5, 4
 ; CI-NEXT:    s_lshl_b32 s4, s4, 16
 ; CI-NEXT:    v_add_i32_e32 v2, vcc, s0, v2
 ; CI-NEXT:    s_lshl_b64 s[0:1], s[2:3], s1
 ; CI-NEXT:    s_or_b32 s2, s6, s4
 ; CI-NEXT:    v_mov_b32_e32 v4, s2
 ; CI-NEXT:    v_mov_b32_e32 v5, s2
 ; CI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc

llvm/test/CodeGen/AMDGPU/llc-pipeline.ll

 ; When EXPENSIVE_CHECKS are enabled, the machine verifier appears between each
 ; pass. Ignore it with 'grep -v'.
 ; fixme: the following line is added to cleanup bots, will be removed in weeks.
 ; RUN: rm -f %S/llc-pipeline.s
 ; RUN: llc -O0 -mtriple=amdgcn--amdhsa -disable-verify -debug-pass=Structure < %s 2>&1 \
 ; RUN:   | grep -v 'Verify generated machine code' | FileCheck -check-prefix=GCN-O0 %s
 ; RUN: llc -O1 -mtriple=amdgcn--amdhsa -disable-verify -debug-pass=Structure < %s 2>&1 \
 ; RUN:   | grep -v 'Verify generated machine code' | FileCheck -check-prefix=GCN-O1 %s
 ; RUN: llc -O1 -mtriple=amdgcn--amdhsa -disable-verify -amdgpu-scalar-ir-passes -amdgpu-sdwa-peephole \
-; RUN:   -amdgpu-load-store-vectorizer -debug-pass=Structure < %s 2>&1 \
+; RUN:   -amdgpu-load-store-vectorizer -amdgpu-enable-pre-ra-optimizations -debug-pass=Structure < %s 2>&1 \
 ; RUN:   | grep -v 'Verify generated machine code' | FileCheck -check-prefix=GCN-O1-OPTS %s
 ; RUN: llc -O2 -mtriple=amdgcn--amdhsa -disable-verify -debug-pass=Structure < %s 2>&1 \
 ; RUN:   | grep -v 'Verify generated machine code' | FileCheck -check-prefix=GCN-O2 %s
 ; RUN: llc -O3 -mtriple=amdgcn--amdhsa -disable-verify -debug-pass=Structure < %s 2>&1 \
 ; RUN:   | grep -v 'Verify generated machine code' | FileCheck -check-prefix=GCN-O3 %s
 
 ; REQUIRES: asserts
 

 ; GCN-O1-OPTS-NEXT:         SI Lower control flow pseudo instructions
 ; GCN-O1-OPTS-NEXT:         Two-Address instruction pass
 ; GCN-O1-OPTS-NEXT:         MachineDominator Tree Construction
 ; GCN-O1-OPTS-NEXT:         Slot index numbering
 ; GCN-O1-OPTS-NEXT:         Live Interval Analysis
 ; GCN-O1-OPTS-NEXT:         Machine Natural Loop Construction
 ; GCN-O1-OPTS-NEXT:         Simple Register Coalescing
 ; GCN-O1-OPTS-NEXT:         Rename Disconnected Subregister Components
+; GCN-O1-OPTS-NEXT:         AMDGPU Pre-RA optimizations
 ; GCN-O1-OPTS-NEXT:         Machine Instruction Scheduler
 ; GCN-O1-OPTS-NEXT:         MachinePostDominator Tree Construction
 ; GCN-O1-OPTS-NEXT:         SI Whole Quad Mode
 ; GCN-O1-OPTS-NEXT:         Virtual Register Map
 ; GCN-O1-OPTS-NEXT:         Live Register Matrix
 ; GCN-O1-OPTS-NEXT:         SI Pre-allocate WWM Registers
 ; GCN-O1-OPTS-NEXT:         SI optimize exec mask operations pre-RA
 ; GCN-O1-OPTS-NEXT:         Machine Natural Loop Construction

 ; GCN-O2-NEXT:         SI Lower control flow pseudo instructions
 ; GCN-O2-NEXT:         Two-Address instruction pass
 ; GCN-O2-NEXT:         MachineDominator Tree Construction
 ; GCN-O2-NEXT:         Slot index numbering
 ; GCN-O2-NEXT:         Live Interval Analysis
 ; GCN-O2-NEXT:         Machine Natural Loop Construction
 ; GCN-O2-NEXT:         Simple Register Coalescing
 ; GCN-O2-NEXT:         Rename Disconnected Subregister Components
+; GCN-O2-NEXT:         AMDGPU Pre-RA optimizations
 ; GCN-O2-NEXT:         Machine Instruction Scheduler
 ; GCN-O2-NEXT:         MachinePostDominator Tree Construction
 ; GCN-O2-NEXT:         SI Whole Quad Mode
 ; GCN-O2-NEXT:         Virtual Register Map
 ; GCN-O2-NEXT:         Live Register Matrix
 ; GCN-O2-NEXT:         SI Pre-allocate WWM Registers
 ; GCN-O2-NEXT:         SI optimize exec mask operations pre-RA
 ; GCN-O2-NEXT:         SI Form memory clauses

 ; GCN-O3-NEXT:         SI Lower control flow pseudo instructions
 ; GCN-O3-NEXT:         Two-Address instruction pass
 ; GCN-O3-NEXT:         MachineDominator Tree Construction
 ; GCN-O3-NEXT:         Slot index numbering
 ; GCN-O3-NEXT:         Live Interval Analysis
 ; GCN-O3-NEXT:         Machine Natural Loop Construction
 ; GCN-O3-NEXT:         Simple Register Coalescing
 ; GCN-O3-NEXT:         Rename Disconnected Subregister Components
+; GCN-O3-NEXT:         AMDGPU Pre-RA optimizations
 ; GCN-O3-NEXT:         Machine Instruction Scheduler
 ; GCN-O3-NEXT:         MachinePostDominator Tree Construction
 ; GCN-O3-NEXT:         SI Whole Quad Mode
 ; GCN-O3-NEXT:         Virtual Register Map
 ; GCN-O3-NEXT:         Live Register Matrix
 ; GCN-O3-NEXT:         SI Pre-allocate WWM Registers
 ; GCN-O3-NEXT:         SI optimize exec mask operations pre-RA
 ; GCN-O3-NEXT:         SI Form memory clauses

llvm/test/CodeGen/AMDGPU/llvm.round.f64.ll

 ; SI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, 0
 ; SI-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
 ; SI-NEXT:    v_mov_b32_e32 v1, 0
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b64 s[0:1], s[6:7]
 ; SI-NEXT:    buffer_load_dwordx2 v[2:3], v[0:1], s[0:3], 0 addr64
+; SI-NEXT:    s_mov_b32 s0, -1
 ; SI-NEXT:    s_movk_i32 s7, 0xfc01
 ; SI-NEXT:    s_mov_b32 s1, 0xfffff
-; SI-NEXT:    s_mov_b32 s0, -1
 ; SI-NEXT:    s_brev_b32 s6, -2
 ; SI-NEXT:    v_mov_b32_e32 v8, 0x3ff00000
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_bfe_u32 v4, v3, 20, 11
 ; SI-NEXT:    v_add_i32_e32 v6, vcc, s7, v4
 ; SI-NEXT:    v_lshr_b64 v[4:5], s[0:1], v6
 ; SI-NEXT:    v_and_b32_e32 v7, 0x80000000, v3
 ; SI-NEXT:    v_not_b32_e32 v4, v4

llvm/test/CodeGen/AMDGPU/packed-fp32.ll

define amdgpu_kernel void @fadd_v2_v_lit_splat(<2 x float> addrspace(1)* %a) {
   %add = fadd <2 x float> %load, <float 1.0, float 1.0>
   store <2 x float> %add, <2 x float> addrspace(1)* %gep, align 8
   ret void
 }
 
 ; GCN-LABEL: {{^}}fadd_v2_v_lit_hi0:
 ; GFX900-DAG: v_add_f32_e32 v{{[0-9]+}}, 0, v{{[0-9]+}}
 ; GFX900-DAG: v_add_f32_e32 v{{[0-9]+}}, 1.0, v{{[0-9]+}}
-; GFX90A-DAG: s_mov_b32 s[[HI:[0-9]+]], 0
-; GFX90A-DAG: s_mov_b32 s[[LO:[0-9]+]], 1.0
-; GFX90A:     v_pk_add_f32 v[{{[0-9:]+}}], v[{{[0-9:]+}}], s{{\[}}[[LO]]:[[HI]]]{{$}}
+; GFX90A-DAG: s_mov_b64 [[K:s\[[0-9:]+\]]], 0x3f800000
+; GFX90A:     v_pk_add_f32 v[{{[0-9:]+}}], v[{{[0-9:]+}}], [[K]]
 define amdgpu_kernel void @fadd_v2_v_lit_hi0(<2 x float> addrspace(1)* %a) {
   %id = tail call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr inbounds <2 x float>, <2 x float> addrspace(1)* %a, i32 %id
   %load = load <2 x float>, <2 x float> addrspace(1)* %gep, align 8
   %add = fadd <2 x float> %load, <float 1.0, float 0.0>
   store <2 x float> %add, <2 x float> addrspace(1)* %gep, align 8
   ret void
 }

llvm/test/CodeGen/AMDGPU/promote-alloca-vector-to-vector.ll

entry:
   store half %load, half addrspace(1)* %out, align 2
   ret void
 }
 
 ; GCN-LABEL: {{^}}half4_alloca_load4:
 ; OPT-LABEL: define amdgpu_kernel void @half4_alloca_load4
 
 ; GCN-NOT: buffer_
-; GCN-DAG: s_mov_b32 s[[SH:[0-9]+]], 0
-; GCN-DAG: s_mov_b32 s[[SL:[0-9]+]], 0xffff
+; GCN:     s_mov_b64 s[{{[0-9:]+}}], 0xffff
 
 ; OPT: %gep = getelementptr inbounds <4 x half>, <4 x half> addrspace(5)* %alloca, i32 0, i32 %sel2
 ; OPT: %0 = load <4 x half>, <4 x half> addrspace(5)* %alloca
 ; OPT: %1 = insertelement <4 x half> %0, half 0xH3C00, i32 %sel2
 ; OPT: store <4 x half> %1, <4 x half> addrspace(5)* %alloca
 ; OPT: %load = load <4 x half>, <4 x half> addrspace(5)* %alloca, align 2
 ; OPT: store <4 x half> %load, <4 x half> addrspace(1)* %out, align 2
 
entry:
   store i16 %load, i16 addrspace(1)* %out, align 2
   ret void
 }
 
 ; GCN-LABEL: {{^}}short4_alloca_load4:
 ; OPT-LABEL: define amdgpu_kernel void @short4_alloca_load4
 
 ; GCN-NOT: buffer_
-; GCN-DAG: s_mov_b32 s[[SH:[0-9]+]], 0
-; GCN-DAG: s_mov_b32 s[[SL:[0-9]+]], 0xffff
+; GCN:     s_mov_b64 s[{{[0-9:]+}}], 0xffff
 
 ; OPT: %gep = getelementptr inbounds <4 x i16>, <4 x i16> addrspace(5)* %alloca, i32 0, i32 %sel2
 ; OPT: %0 = load <4 x i16>, <4 x i16> addrspace(5)* %alloca
 ; OPT: %1 = insertelement <4 x i16> %0, i16 1, i32 %sel2
 ; OPT: store <4 x i16> %1, <4 x i16> addrspace(5)* %alloca
 ; OPT: %load = load <4 x i16>, <4 x i16> addrspace(5)* %alloca, align 2
 ; OPT: store <4 x i16> %load, <4 x i16> addrspace(1)* %out, align 2
 

llvm/test/CodeGen/AMDGPU/remat-fp64-constants.ll

+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -verify-machineinstrs --stress-regalloc=10 < %s | FileCheck -check-prefix=GCN %s
+; RUN: llc -global-isel -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -verify-machineinstrs --stress-regalloc=10 < %s | FileCheck -check-prefix=GCN %s
+
+; GCN-LABEL: {{^}}test_remat_sgpr:
+; GCN-NOT:     v_writelane_b32
+; GCN:         {{^}}[[LOOP:BB[0-9_]+]]:
+; GCN-COUNT-6: s_mov_b32 s{{[0-9]+}}, 0x
+; GCN-NOT:     v_writelane_b32
+; GCN:         s_cbranch_{{[^ ]+}} [[LOOP]]
+; GCN: .sgpr_spill_count: 0
+define amdgpu_kernel void @test_remat_sgpr(double addrspace(1)* %arg, double addrspace(1)* %arg1) {
+bb:
+  %i = tail call i32 @llvm.amdgcn.workitem.id.x()
+  br label %bb3
+
+bb2:                                              ; preds = %bb3
+  ret void
+
+bb3:                                              ; preds = %bb3, %bb
+  %i4 = phi i32 [ 0, %bb ], [ %i22, %bb3 ]
+  %i5 = add nuw nsw i32 %i4, %i
+  %i6 = zext i32 %i5 to i64
+  %i7 = getelementptr inbounds double, double addrspace(1)* %arg, i64 %i6
+  %i8 = load double, double addrspace(1)* %i7, align 8
+  %i9 = fadd double %i8, 0x3EFC01997CC9E6B0
+  %i10 = tail call double @llvm.fma.f64(double %i8, double %i9, double 0x3FBE25E43ABE935A)
+  %i11 = tail call double @llvm.fma.f64(double %i10, double %i9, double 0x3FC110EF47E6C9C2)
+  %i12 = tail call double @llvm.fma.f64(double %i11, double %i9, double 0x3FC3B13BCFA74449)
+  %i13 = tail call double @llvm.fma.f64(double %i12, double %i9, double 0x3FC745D171BF3C30)
+  %i14 = tail call double @llvm.fma.f64(double %i13, double %i9, double 0x3FCC71C71C7792CE)
+  %i15 = tail call double @llvm.fma.f64(double %i14, double %i9, double 0x3FD24924924920DA)
+  %i16 = tail call double @llvm.fma.f64(double %i15, double %i9, double 0x3FD999999999999C)
+  %i17 = tail call double @llvm.fma.f64(double %i16, double %i9, double 0x3FD899999999899C)
+  %i18 = tail call double @llvm.fma.f64(double %i17, double %i9, double 0x3FD799999999799C)
+  %i19 = tail call double @llvm.fma.f64(double %i18, double %i9, double 0x3FD699999999699C)
+  %i20 = tail call double @llvm.fma.f64(double %i19, double %i9, double 0x3FD599999999599C)
+  %i21 = getelementptr inbounds double, double addrspace(1)* %arg1, i64 %i6
+  store double %i19, double addrspace(1)* %i21, align 8
+  %i22 = add nuw nsw i32 %i4, 1
+  %i23 = icmp eq i32 %i22, 1024
+  br i1 %i23, label %bb2, label %bb3
+}
+
+declare double @llvm.fma.f64(double, double, double)
+declare i32 @llvm.amdgcn.workitem.id.x()

llvm/test/CodeGen/AMDGPU/sdiv64.ll

 ; GCN-IR-NEXT:    v_ffbh_u32_e32 v5, v1
 ; GCN-IR-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; GCN-IR-NEXT:    v_cndmask_b32_e32 v10, v5, v4, vcc
 ; GCN-IR-NEXT:    s_movk_i32 s6, 0xffd0
 ; GCN-IR-NEXT:    v_add_i32_e32 v4, vcc, s6, v10
 ; GCN-IR-NEXT:    v_addc_u32_e64 v5, s[6:7], 0, -1, vcc
 ; GCN-IR-NEXT:    v_cmp_eq_u64_e64 s[4:5], 0, v[0:1]
 ; GCN-IR-NEXT:    v_cmp_lt_u64_e32 vcc, 63, v[4:5]
-; GCN-IR-NEXT:    s_mov_b32 s8, 0x8000
+; GCN-IR-NEXT:    s_mov_b64 s[8:9], 0x8000
 ; GCN-IR-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
 ; GCN-IR-NEXT:    v_mov_b32_e32 v6, s8
 ; GCN-IR-NEXT:    v_cmp_ne_u64_e32 vcc, 63, v[4:5]
 ; GCN-IR-NEXT:    v_mov_b32_e32 v11, 0
 ; GCN-IR-NEXT:    v_cndmask_b32_e64 v6, v6, 0, s[4:5]
 ; GCN-IR-NEXT:    s_xor_b64 s[4:5], s[4:5], -1
 ; GCN-IR-NEXT:    v_mov_b32_e32 v3, v2
-; GCN-IR-NEXT:    s_mov_b32 s9, 0
 ; GCN-IR-NEXT:    v_mov_b32_e32 v7, v11
 ; GCN-IR-NEXT:    s_and_b64 s[4:5], s[4:5], vcc
 ; GCN-IR-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
 ; GCN-IR-NEXT:    s_cbranch_execz BB12_6
 ; GCN-IR-NEXT:  ; %bb.1: ; %udiv-bb1
 ; GCN-IR-NEXT:    v_add_i32_e32 v8, vcc, 1, v4
 ; GCN-IR-NEXT:    v_addc_u32_e32 v9, vcc, 0, v5, vcc
 ; GCN-IR-NEXT:    v_cmp_ge_u64_e32 vcc, v[8:9], v[4:5]
 ; GCN-IR-NEXT:    v_sub_i32_e64 v4, s[4:5], 63, v4
 ; GCN-IR-NEXT:    v_lshl_b64 v[4:5], s[8:9], v4
 ; GCN-IR-NEXT:    v_mov_b32_e32 v6, 0
 ; GCN-IR-NEXT:    s_mov_b64 s[8:9], 0
 ; GCN-IR-NEXT:    v_mov_b32_e32 v7, 0
 ; GCN-IR-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; GCN-IR-NEXT:    s_xor_b64 s[10:11], exec, s[4:5]
 ; GCN-IR-NEXT:    s_cbranch_execz BB12_5
 ; GCN-IR-NEXT:  ; %bb.2: ; %udiv-preheader
-; GCN-IR-NEXT:    s_mov_b32 s5, 0
-; GCN-IR-NEXT:    s_mov_b32 s4, 0x8000
+; GCN-IR-NEXT:    s_mov_b64 s[4:5], 0x8000
 ; GCN-IR-NEXT:    v_lshr_b64 v[12:13], s[4:5], v8
 ; GCN-IR-NEXT:    v_add_i32_e32 v8, vcc, -1, v0
 ; GCN-IR-NEXT:    v_addc_u32_e32 v9, vcc, -1, v1, vcc
 ; GCN-IR-NEXT:    v_sub_i32_e32 v10, vcc, 47, v10
 ; GCN-IR-NEXT:    v_mov_b32_e32 v14, 0
 ; GCN-IR-NEXT:    v_subb_u32_e32 v11, vcc, 0, v11, vcc
 ; GCN-IR-NEXT:    v_mov_b32_e32 v15, 0
 ; GCN-IR-NEXT:  BB12_3: ; %udiv-do-while

llvm/test/CodeGen/AMDGPU/shift-i128.ll

; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = shl i128 17, %rhs
   ret i128 %shl
 }
 
 define i128 @v_lshr_i128_kv(i128 %rhs) {
 ; GCN-LABEL: v_lshr_i128_kv:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GCN-NEXT:    s_movk_i32 s4, 0x41
-; GCN-NEXT:    s_mov_b32 s5, 0
+; GCN-NEXT:    s_mov_b64 s[4:5], 0x41
 ; GCN-NEXT:    v_lshr_b64 v[1:2], s[4:5], v0
 ; GCN-NEXT:    v_cmp_gt_u32_e32 vcc, 64, v0
 ; GCN-NEXT:    v_mov_b32_e32 v3, s4
 ; GCN-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, v0
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
 ; GCN-NEXT:    s_and_b64 vcc, s[4:5], vcc
 ; GCN-NEXT:    v_cndmask_b32_e64 v0, v3, v1, s[4:5]
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, 0, v2, vcc

llvm/test/CodeGen/AMDGPU/shl.ll

 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_mov_b32 s10, s2
 ; SI-NEXT:    s_mov_b32 s11, s3
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s8, s6
 ; SI-NEXT:    s_mov_b32 s9, s7
 ; SI-NEXT:    buffer_load_dword v0, off, s[8:11], 0
-; SI-NEXT:    s_movk_i32 s7, 0x11e
 ; SI-NEXT:    s_mov_b32 s6, 0xab19b207
+; SI-NEXT:    s_movk_i32 s7, 0x11e
 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshl_b64 v[0:1], s[6:7], v0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: v_shl_constant_i64:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; VI-NEXT:    s_mov_b32 s7, 0xf000
 ; VI-NEXT:    s_mov_b32 s6, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dword s2, s[2:3], 0x0
 ; VI-NEXT:    s_mov_b32 s4, s0
 ; VI-NEXT:    s_mov_b32 s5, s1
-; VI-NEXT:    s_movk_i32 s1, 0x11e
 ; VI-NEXT:    s_mov_b32 s0, 0xab19b207
+; VI-NEXT:    s_movk_i32 s1, 0x11e
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
 ;
 ; EG-LABEL: v_shl_constant_i64:

 ; SI-NEXT:    s_mov_b32 s3, 0xf000
 ; SI-NEXT:    s_mov_b32 s2, -1
 ; SI-NEXT:    s_mov_b32 s10, s2
 ; SI-NEXT:    s_mov_b32 s11, s3
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s8, s6
 ; SI-NEXT:    s_mov_b32 s9, s7
 ; SI-NEXT:    buffer_load_dword v0, off, s[8:11], 0
-; SI-NEXT:    s_mov_b32 s7, 0
-; SI-NEXT:    s_mov_b32 s6, 0x12d687
+; SI-NEXT:    s_mov_b64 s[6:7], 0x12d687
 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshl_b64 v[0:1], s[6:7], v0
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: v_shl_i64_32_bit_constant:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; VI-NEXT:    s_mov_b32 s7, 0xf000
 ; VI-NEXT:    s_mov_b32 s6, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dword s2, s[2:3], 0x0
 ; VI-NEXT:    s_mov_b32 s4, s0
 ; VI-NEXT:    s_mov_b32 s5, s1
-; VI-NEXT:    s_mov_b32 s1, 0
-; VI-NEXT:    s_mov_b32 s0, 0x12d687
+; VI-NEXT:    s_mov_b64 s[0:1], 0x12d687
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
 ;
 ; EG-LABEL: v_shl_i64_32_bit_constant:

 
 ; Test with the 64-bit integer bitpattern for a 32-bit float in the
 ; low 32-bits, which is not a valid 64-bit inline immmediate.
 define amdgpu_kernel void @s_shl_inline_imm_f32_4_0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
 ; SI-LABEL: s_shl_inline_imm_f32_4_0_i64:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; SI-NEXT:    s_load_dword s2, s[0:1], 0xd
-; SI-NEXT:    s_mov_b32 s1, 0
-; SI-NEXT:    s_mov_b32 s0, 4.0
+; SI-NEXT:    s_mov_b64 s[0:1], 0x40800000
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; SI-NEXT:    v_mov_b32_e32 v0, s0
 ; SI-NEXT:    v_mov_b32_e32 v1, s1
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_shl_inline_imm_f32_4_0_i64:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; VI-NEXT:    s_load_dword s2, s[0:1], 0x34
-; VI-NEXT:    s_mov_b32 s1, 0
-; VI-NEXT:    s_mov_b32 s0, 4.0
+; VI-NEXT:    s_mov_b64 s[0:1], 0x40800000
 ; VI-NEXT:    s_mov_b32 s7, 0xf000
 ; VI-NEXT:    s_mov_b32 s6, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
   ret void
 }
 
 define amdgpu_kernel void @s_shl_inline_high_imm_f32_4_0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
 ; SI-LABEL: s_shl_inline_high_imm_f32_4_0_i64:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; SI-NEXT:    s_load_dword s2, s[0:1], 0xd
-; SI-NEXT:    s_mov_b32 s1, 4.0
 ; SI-NEXT:    s_mov_b32 s0, 0
+; SI-NEXT:    s_mov_b32 s1, 4.0
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; SI-NEXT:    v_mov_b32_e32 v0, s0
 ; SI-NEXT:    v_mov_b32_e32 v1, s1
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_shl_inline_high_imm_f32_4_0_i64:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; VI-NEXT:    s_load_dword s2, s[0:1], 0x34
-; VI-NEXT:    s_mov_b32 s1, 4.0
 ; VI-NEXT:    s_mov_b32 s0, 0
+; VI-NEXT:    s_mov_b32 s1, 4.0
 ; VI-NEXT:    s_mov_b32 s7, 0xf000
 ; VI-NEXT:    s_mov_b32 s6, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm
; EG-NEXT:    2(2.802597e-45), 0(0.000000e+00)
   ret void
 }
 
 define amdgpu_kernel void @s_shl_inline_high_imm_f32_neg_4_0_i64(i64 addrspace(1)* %out, i64 addrspace(1)* %aptr, i64 %a) {
 ; SI-LABEL: s_shl_inline_high_imm_f32_neg_4_0_i64:
 ; SI:       ; %bb.0:
 ; SI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
 ; SI-NEXT:    s_load_dword s2, s[0:1], 0xd
-; SI-NEXT:    s_mov_b32 s1, -4.0
 ; SI-NEXT:    s_mov_b32 s0, 0
+; SI-NEXT:    s_mov_b32 s1, -4.0
 ; SI-NEXT:    s_mov_b32 s7, 0xf000
 ; SI-NEXT:    s_mov_b32 s6, -1
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; SI-NEXT:    v_mov_b32_e32 v0, s0
 ; SI-NEXT:    v_mov_b32_e32 v1, s1
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: s_shl_inline_high_imm_f32_neg_4_0_i64:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; VI-NEXT:    s_load_dword s2, s[0:1], 0x34
-; VI-NEXT:    s_mov_b32 s1, -4.0
 ; VI-NEXT:    s_mov_b32 s0, 0
+; VI-NEXT:    s_mov_b32 s1, -4.0
 ; VI-NEXT:    s_mov_b32 s7, 0xf000
 ; VI-NEXT:    s_mov_b32 s6, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_lshl_b64 s[0:1], s[0:1], s2
 ; VI-NEXT:    v_mov_b32_e32 v0, s0
 ; VI-NEXT:    v_mov_b32_e32 v1, s1
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; VI-NEXT:    s_endpgm

llvm/test/CodeGen/AMDGPU/sopk-compares.ll

 
 endif:
   store volatile i32 1, i32 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}br_scc_eq_i64_simm16:
 ; VI-DAG: s_movk_i32 s[[K_LO:[0-9]+]], 0x4d2
-; VI-DAG: s_mov_b32 s[[K_HI:[0-9]+]], 0
+; VI-DAG: s_mov_b32 s[[K_HI:[0-9]+]], 1
 ; VI: s_cmp_eq_u64 s{{\[[0-9]+:[0-9]+\]}}, s{{\[}}[[K_LO]]:[[K_HI]]{{\]}}
 
 ; SI: v_cmp_eq_u64_e32
 define amdgpu_kernel void @br_scc_eq_i64_simm16(i64 %cond, i32 addrspace(1)* %out) #0 {
 entry:
-  %cmp0 = icmp eq i64 %cond, 1234
+  %cmp0 = icmp eq i64 %cond, 4294968530
   br i1 %cmp0, label %endif, label %if
 
 if:
   call void asm sideeffect "", ""()
   br label %endif
 
 endif:
   store volatile i32 1, i32 addrspace(1)* %out

 
 endif:
   store volatile i32 1, i32 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}br_scc_ne_i64_simm16:
 ; VI-DAG: s_movk_i32 s[[K_LO:[0-9]+]], 0x4d2
-; VI-DAG: s_mov_b32 s[[K_HI:[0-9]+]], 0
+; VI-DAG: s_mov_b32 s[[K_HI:[0-9]+]], 1
 ; VI: s_cmp_lg_u64 s{{\[[0-9]+:[0-9]+\]}}, s{{\[}}[[K_LO]]:[[K_HI]]{{\]}}
 
 ; SI: v_cmp_ne_u64_e32
 define amdgpu_kernel void @br_scc_ne_i64_simm16(i64 %cond, i32 addrspace(1)* %out) #0 {
 entry:
-  %cmp0 = icmp ne i64 %cond, 1234
+  %cmp0 = icmp ne i64 %cond, 4294968530
   br i1 %cmp0, label %endif, label %if
 
 if:
   call void asm sideeffect "", ""()
   br label %endif
 
 endif:
   store volatile i32 1, i32 addrspace(1)* %out
   ret void
 }
 
 attributes #0 = { nounwind }
 attributes #1 = { nounwind readnone }

llvm/test/CodeGen/AMDGPU/srem64.ll

 ; GCN-IR-NEXT:    v_ffbh_u32_e32 v3, v1
 ; GCN-IR-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; GCN-IR-NEXT:    v_cndmask_b32_e32 v8, v3, v2, vcc
 ; GCN-IR-NEXT:    s_movk_i32 s6, 0xffd0
 ; GCN-IR-NEXT:    v_add_i32_e32 v2, vcc, s6, v8
 ; GCN-IR-NEXT:    v_addc_u32_e64 v3, s[6:7], 0, -1, vcc
 ; GCN-IR-NEXT:    v_cmp_eq_u64_e64 s[4:5], 0, v[0:1]
 ; GCN-IR-NEXT:    v_cmp_lt_u64_e32 vcc, 63, v[2:3]
-; GCN-IR-NEXT:    s_mov_b32 s8, 0x8000
+; GCN-IR-NEXT:    s_mov_b64 s[8:9], 0x8000
 ; GCN-IR-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
 ; GCN-IR-NEXT:    v_mov_b32_e32 v4, s8
 ; GCN-IR-NEXT:    v_cmp_ne_u64_e32 vcc, 63, v[2:3]
 ; GCN-IR-NEXT:    v_mov_b32_e32 v9, 0
 ; GCN-IR-NEXT:    v_cndmask_b32_e64 v4, v4, 0, s[4:5]
 ; GCN-IR-NEXT:    s_xor_b64 s[4:5], s[4:5], -1
-; GCN-IR-NEXT:    s_mov_b32 s9, 0
 ; GCN-IR-NEXT:    v_mov_b32_e32 v5, v9
 ; GCN-IR-NEXT:    s_and_b64 s[4:5], s[4:5], vcc
 ; GCN-IR-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
 ; GCN-IR-NEXT:    s_cbranch_execz BB12_6
 ; GCN-IR-NEXT:  ; %bb.1: ; %udiv-bb1
 ; GCN-IR-NEXT:    v_add_i32_e32 v6, vcc, 1, v2
 ; GCN-IR-NEXT:    v_addc_u32_e32 v7, vcc, 0, v3, vcc
 ; GCN-IR-NEXT:    v_cmp_ge_u64_e32 vcc, v[6:7], v[2:3]
 ; GCN-IR-NEXT:    v_sub_i32_e64 v2, s[4:5], 63, v2
 ; GCN-IR-NEXT:    v_lshl_b64 v[2:3], s[8:9], v2
 ; GCN-IR-NEXT:    v_mov_b32_e32 v4, 0
 ; GCN-IR-NEXT:    s_mov_b64 s[8:9], 0
 ; GCN-IR-NEXT:    v_mov_b32_e32 v5, 0
 ; GCN-IR-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; GCN-IR-NEXT:    s_xor_b64 s[10:11], exec, s[4:5]
 ; GCN-IR-NEXT:    s_cbranch_execz BB12_5
 ; GCN-IR-NEXT:  ; %bb.2: ; %udiv-preheader
-; GCN-IR-NEXT:    s_mov_b32 s5, 0
-; GCN-IR-NEXT:    s_mov_b32 s4, 0x8000
+; GCN-IR-NEXT:    s_mov_b64 s[4:5], 0x8000
 ; GCN-IR-NEXT:    v_lshr_b64 v[10:11], s[4:5], v6
 ; GCN-IR-NEXT:    v_add_i32_e32 v6, vcc, -1, v0
 ; GCN-IR-NEXT:    v_addc_u32_e32 v7, vcc, -1, v1, vcc
 ; GCN-IR-NEXT:    v_sub_i32_e32 v8, vcc, 47, v8
 ; GCN-IR-NEXT:    v_mov_b32_e32 v12, 0
 ; GCN-IR-NEXT:    v_subb_u32_e32 v9, vcc, 0, v9, vcc
 ; GCN-IR-NEXT:    v_mov_b32_e32 v13, 0
 ; GCN-IR-NEXT:  BB12_3: ; %udiv-do-while

llvm/test/CodeGen/AMDGPU/udiv64.ll

 ; GCN-IR-NEXT:    v_add_i32_e32 v2, vcc, 32, v2
 ; GCN-IR-NEXT:    v_ffbh_u32_e32 v3, v1
 ; GCN-IR-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; GCN-IR-NEXT:    v_cndmask_b32_e32 v8, v3, v2, vcc
 ; GCN-IR-NEXT:    v_add_i32_e32 v4, vcc, 0xffffffd0, v8
 ; GCN-IR-NEXT:    v_addc_u32_e64 v5, s[6:7], 0, -1, vcc
 ; GCN-IR-NEXT:    v_cmp_eq_u64_e64 s[4:5], 0, v[0:1]
 ; GCN-IR-NEXT:    v_cmp_lt_u64_e32 vcc, 63, v[4:5]
-; GCN-IR-NEXT:    s_mov_b32 s8, 0x8000
+; GCN-IR-NEXT:    s_mov_b64 s[8:9], 0x8000
 ; GCN-IR-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
 ; GCN-IR-NEXT:    v_mov_b32_e32 v2, s8
 ; GCN-IR-NEXT:    v_cmp_ne_u64_e32 vcc, 63, v[4:5]
 ; GCN-IR-NEXT:    v_mov_b32_e32 v9, 0
 ; GCN-IR-NEXT:    v_cndmask_b32_e64 v2, v2, 0, s[4:5]
 ; GCN-IR-NEXT:    s_xor_b64 s[4:5], s[4:5], -1
-; GCN-IR-NEXT:    s_mov_b32 s9, 0
 ; GCN-IR-NEXT:    v_mov_b32_e32 v3, v9
 ; GCN-IR-NEXT:    s_and_b64 s[4:5], s[4:5], vcc
 ; GCN-IR-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
 ; GCN-IR-NEXT:    s_cbranch_execz BB9_6
 ; GCN-IR-NEXT:  ; %bb.1: ; %udiv-bb1
 ; GCN-IR-NEXT:    v_add_i32_e32 v6, vcc, 1, v4
 ; GCN-IR-NEXT:    v_addc_u32_e32 v7, vcc, 0, v5, vcc
 ; GCN-IR-NEXT:    v_sub_i32_e64 v2, s[4:5], 63, v4
 ; GCN-IR-NEXT:    v_cmp_ge_u64_e32 vcc, v[6:7], v[4:5]
 ; GCN-IR-NEXT:    v_lshl_b64 v[2:3], s[8:9], v2
 ; GCN-IR-NEXT:    v_mov_b32_e32 v4, 0
 ; GCN-IR-NEXT:    s_mov_b64 s[8:9], 0
 ; GCN-IR-NEXT:    v_mov_b32_e32 v5, 0
 ; GCN-IR-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; GCN-IR-NEXT:    s_xor_b64 s[10:11], exec, s[4:5]
 ; GCN-IR-NEXT:    s_cbranch_execz BB9_5
 ; GCN-IR-NEXT:  ; %bb.2: ; %udiv-preheader
-; GCN-IR-NEXT:    s_mov_b32 s5, 0
-; GCN-IR-NEXT:    s_mov_b32 s4, 0x8000
+; GCN-IR-NEXT:    s_mov_b64 s[4:5], 0x8000
 ; GCN-IR-NEXT:    v_lshr_b64 v[10:11], s[4:5], v6
 ; GCN-IR-NEXT:    v_add_i32_e32 v6, vcc, -1, v0
 ; GCN-IR-NEXT:    v_addc_u32_e32 v7, vcc, -1, v1, vcc
 ; GCN-IR-NEXT:    v_sub_i32_e32 v8, vcc, 47, v8
 ; GCN-IR-NEXT:    v_mov_b32_e32 v12, 0
 ; GCN-IR-NEXT:    v_subb_u32_e32 v9, vcc, 0, v9, vcc
 ; GCN-IR-NEXT:    v_mov_b32_e32 v13, 0
 ; GCN-IR-NEXT:  BB9_3: ; %udiv-do-while

llvm/test/CodeGen/AMDGPU/urem64.ll

 ; GCN-IR-NEXT:    v_ffbh_u32_e32 v3, v1
 ; GCN-IR-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; GCN-IR-NEXT:    v_cndmask_b32_e32 v8, v3, v2, vcc
 ; GCN-IR-NEXT:    s_movk_i32 s6, 0xffd0
 ; GCN-IR-NEXT:    v_add_i32_e32 v2, vcc, s6, v8
 ; GCN-IR-NEXT:    v_addc_u32_e64 v3, s[6:7], 0, -1, vcc
 ; GCN-IR-NEXT:    v_cmp_eq_u64_e64 s[4:5], 0, v[0:1]
 ; GCN-IR-NEXT:    v_cmp_lt_u64_e32 vcc, 63, v[2:3]
-; GCN-IR-NEXT:    s_mov_b32 s8, 0x8000
+; GCN-IR-NEXT:    s_mov_b64 s[8:9], 0x8000
 ; GCN-IR-NEXT:    s_or_b64 s[4:5], s[4:5], vcc
 ; GCN-IR-NEXT:    v_mov_b32_e32 v4, s8
 ; GCN-IR-NEXT:    v_cmp_ne_u64_e32 vcc, 63, v[2:3]
 ; GCN-IR-NEXT:    v_mov_b32_e32 v9, 0
 ; GCN-IR-NEXT:    v_cndmask_b32_e64 v4, v4, 0, s[4:5]
 ; GCN-IR-NEXT:    s_xor_b64 s[4:5], s[4:5], -1
-; GCN-IR-NEXT:    s_mov_b32 s9, 0
 ; GCN-IR-NEXT:    v_mov_b32_e32 v5, v9
 ; GCN-IR-NEXT:    s_and_b64 s[4:5], s[4:5], vcc
 ; GCN-IR-NEXT:    s_and_saveexec_b64 s[6:7], s[4:5]
 ; GCN-IR-NEXT:    s_cbranch_execz BB8_6
 ; GCN-IR-NEXT:  ; %bb.1: ; %udiv-bb1
 ; GCN-IR-NEXT:    v_add_i32_e32 v6, vcc, 1, v2
 ; GCN-IR-NEXT:    v_addc_u32_e32 v7, vcc, 0, v3, vcc
 ; GCN-IR-NEXT:    v_cmp_ge_u64_e32 vcc, v[6:7], v[2:3]
 ; GCN-IR-NEXT:    v_sub_i32_e64 v2, s[4:5], 63, v2
 ; GCN-IR-NEXT:    v_lshl_b64 v[2:3], s[8:9], v2
 ; GCN-IR-NEXT:    v_mov_b32_e32 v4, 0
 ; GCN-IR-NEXT:    s_mov_b64 s[8:9], 0
 ; GCN-IR-NEXT:    v_mov_b32_e32 v5, 0
 ; GCN-IR-NEXT:    s_and_saveexec_b64 s[4:5], vcc
 ; GCN-IR-NEXT:    s_xor_b64 s[10:11], exec, s[4:5]
 ; GCN-IR-NEXT:    s_cbranch_execz BB8_5
 ; GCN-IR-NEXT:  ; %bb.2: ; %udiv-preheader
-; GCN-IR-NEXT:    s_mov_b32 s5, 0
-; GCN-IR-NEXT:    s_mov_b32 s4, 0x8000
+; GCN-IR-NEXT:    s_mov_b64 s[4:5], 0x8000
 ; GCN-IR-NEXT:    v_lshr_b64 v[10:11], s[4:5], v6
 ; GCN-IR-NEXT:    v_add_i32_e32 v6, vcc, -1, v0
 ; GCN-IR-NEXT:    v_addc_u32_e32 v7, vcc, -1, v1, vcc
 ; GCN-IR-NEXT:    v_sub_i32_e32 v8, vcc, 47, v8
 ; GCN-IR-NEXT:    v_mov_b32_e32 v12, 0
 ; GCN-IR-NEXT:    v_subb_u32_e32 v9, vcc, 0, v9, vcc
 ; GCN-IR-NEXT:    v_mov_b32_e32 v13, 0
 ; GCN-IR-NEXT:  BB8_3: ; %udiv-do-while

llvm/test/CodeGen/AMDGPU/wwm-reserved-spill.ll

 ; GFX9-O3-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:12 ; 4-byte Folded Spill
 ; GFX9-O3-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:16 ; 4-byte Folded Spill
 ; GFX9-O3-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-O3-NEXT:    buffer_store_dword v6, off, s[0:3], s32 offset:20 ; 4-byte Folded Spill
 ; GFX9-O3-NEXT:    s_mov_b64 exec, s[8:9]
 ; GFX9-O3-NEXT:    v_lshlrev_b32_e32 v0, 5, v0
 ; GFX9-O3-NEXT:    buffer_load_dwordx4 v[1:4], v0, s[4:7], 0 offen
 ; GFX9-O3-NEXT:    buffer_load_dwordx2 v[5:6], v0, s[4:7], 0 offen offset:16
-; GFX9-O3-NEXT:    s_brev_b32 s9, -2
 ; GFX9-O3-NEXT:    s_mov_b32 s8, -1
+; GFX9-O3-NEXT:    s_brev_b32 s9, -2
 ; GFX9-O3-NEXT:    s_not_b64 exec, exec
 ; GFX9-O3-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-O3-NEXT:    v_mov_b32_e32 v1, s8
 ; GFX9-O3-NEXT:    v_mov_b32_e32 v2, s9
 ; GFX9-O3-NEXT:    s_not_b64 exec, exec
 ; GFX9-O3-NEXT:    s_not_b64 exec, exec
 ; GFX9-O3-NEXT:    v_mov_b32_e32 v3, s8
 ; GFX9-O3-NEXT:    v_mov_b32_e32 v4, s9