[AMDGPU] Improve abs modifier usage

[tsymalla]

Summary

If a call to the llvm.fabs intrinsic has users in another reachable
BB, SelectionDAG will not apply the abs modifier to these users and
instead generate a v_and ..., 0x7fffffff instruction.
For fneg instructions, the issue is similar.
This patch implements AMDGPUIselLowering::shouldSinkOperands,
which allows CodegenPrepare to call tryToSinkFreeOperands.

[foad]

Whatever we end up doing for fabs, we should do the same for fneg and fneg-fabs. (But that doesn't have to be part of this patch.)

llvm/lib/Target/AMDGPU/AMDGPULateCodeGenPrepare.cpp

223–225

Why do you need this special case at all? And why only for intrinsics?

232–234

All users of Instructions are Instructions, so this cast will always succeed. (And anyway if you want a cast that can fail, you need to use dyn_cast.)

247

Surely the "dominates" test can never fail, since IR uses SSA form?

253

I don't think you need a special case for this. Any fabs(fabs(x)) - even in different basic blocks - should already have been simplified by generic IR optimizations.

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPULateCodeGenPrepare.cpp

223–225

We don't really need to handle it, but it is useless to clone (for instance) an fabs call with an undef operand.

232–234

Thanks. I have overlooked that one.

253

Makes sense. Thanks.

[foad]

llvm/lib/Target/AMDGPU/AMDGPULateCodeGenPrepare.cpp

223–225

Any useless case should have been optimized away already, so it is much cleaner if you do not add code to handle it specially here.

[tsymalla]

Simplify checks a bit.

[arsenm]

Same should apply for fneg. This is also a candidate for generic CodeGenPrepare checked with isFAbsFree/isFNegFree

[arsenm]

Generic CGP already has tryToSinkFreeOperands, don't know why this wouldn't already work for fabs/fneg

[tsymalla]

In D150347#4341606, @arsenm wrote:

Generic CGP already has tryToSinkFreeOperands, don't know why this wouldn't already work for fabs/fneg

When I tested the implementation in AMD CGP, it did not work because the CSE pass did eliminate some of them again. I will double-check if I can make tryToSinkFreeOperands work similarly.

[tsymalla]

In D150347#4341666, @tsymalla wrote:

In D150347#4341606, @arsenm wrote:

Generic CGP already has tryToSinkFreeOperands, don't know why this wouldn't already work for fabs/fneg

When I tested the implementation in AMD CGP, it did not work because the CSE pass did eliminate some of them again. I will double-check if I can make tryToSinkFreeOperands work similarly.

Probably doesn't work because we don't override shouldSinkOperands

[tsymalla]

Implement shouldSinkOperands, remove old implementation from AMDGPULateCodegenPrepare.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

5148 ↗

(On Diff #522203)

What does this do? Did you mean to check basic blocks here? In any case I am not sure this is required.

[tsymalla]

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

5148 ↗

(On Diff #522203)

It checks whether one of the operands is registered twice, which appears like common practice amongst other implementations as well

[foad]

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

5148 ↗

(On Diff #522203)

Oh I see. Thanks.

[foad]

Looks good overall.

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

5148 ↗

(On Diff #522203)

Nit: I find this confusing because U and Op are both Uses but you only call get on one of them. I guess it relies on an implicit conversion for the other one. I think U->get() == Op.get() would be clearer.

llvm/test/CodeGen/AMDGPU/andorbitset.ll

56 ↗

(On Diff #522203)

I guess we no longer form s_bitset because the s_and uses two different registers? That means that this is no longer testing the original bug from D85307. Maybe change the test to use and i32 undef, 0xBFFFFFFF (clear bit 30) instead of fabs, so that your new optimization won't spoil it?

[tsymalla]

llvm/test/CodeGen/AMDGPU/andorbitset.ll

56 ↗

(On Diff #522203)

Yes, this is, because two different SGPRs are being used. Thanks for the suggestion. I think it makes sense, but unfortunately, it doesn't work. In the original test, the fabs calls are kept until we reach isel - which is a bit strange, because the fabs calls are trivially redundant semantically.

%i = call float @llvm.fabs.f32(float undef) #6
%i1 = bitcast float %i to i32
store i32 %i1, ptr addrspace(1) @gv, align 4
%0 = call float @llvm.fabs.f32(float undef) #6

However, when using and instead, the expressions are eliminated early.

[tsymalla]

Fix nit
Change test order a bit to prevent BB renaming change

[tsymalla]

Regarding the lit test from andorbitset.ll, I don't think there's a simple way to preserve the bitset instruction with this change.

[foad]

LGTM, thanks, but please try to simplify the code - I've added some ideas inline.

llvm/lib/CodeGen/CodeGenPrepare.cpp

7122 ↗

(On Diff #522594)

Please commit this separately (no review required!)

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

5151 ↗

(On Diff #522594)

Nit: if you're using PatternMatch already, you can use m_FAbs here.

5166 ↗

(On Diff #522594)

Nit: might be simpler to return !Ops.empty()?

[tsymalla]

Simplified code

[foad]

Thanks!

llvm/lib/Target/AMDGPU/AMDGPULateCodeGenPrepare.cpp

 //===----------------------------------------------------------------------===//
 
 #include "AMDGPU.h"
 #include "llvm/Analysis/AssumptionCache.h"
 #include "llvm/Analysis/UniformityAnalysis.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstVisitor.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/PatternMatch.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/KnownBits.h"
 #include "llvm/Transforms/Utils/Local.h"
 
 #define DEBUG_TYPE "amdgpu-late-codegenprepare"
 
 using namespace llvm;

 class AMDGPULateCodeGenPrepare
     : public FunctionPass,
       public InstVisitor<AMDGPULateCodeGenPrepare, bool> {
   Module *Mod = nullptr;
   const DataLayout *DL = nullptr;
 
   AssumptionCache *AC = nullptr;
   UniformityInfo *UA = nullptr;
+  DominatorTree *DT = nullptr;
 
 public:
   static char ID;
 
   AMDGPULateCodeGenPrepare() : FunctionPass(ID) {}
 
   StringRef getPassName() const override {
     return "AMDGPU IR late optimizations";
public:
   // Check if the specified value is at least DWORD aligned.
   bool isDWORDAligned(const Value *V) const {
     KnownBits Known = computeKnownBits(V, *DL, 0, AC);
     return Known.countMinTrailingZeros() >= 2;
   }
 
   bool canWidenScalarExtLoad(LoadInst &LI) const;
   bool visitLoadInst(LoadInst &LI);
+  bool visitIntrinsicInst(IntrinsicInst &I);
+  bool cloneInstructionToUsers(Instruction *I);
+  bool hasUndefOrPoisonOperand(IntrinsicInst &I, unsigned Operand = 0) const;
 };
 
 } // end anonymous namespace
 
 bool AMDGPULateCodeGenPrepare::doInitialization(Module &M) {
   Mod = &M;
   DL = &Mod->getDataLayout();
   return false;
 }
 
 bool AMDGPULateCodeGenPrepare::runOnFunction(Function &F) {
   if (skipFunction(F))
     return false;
 
   AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
   UA = &getAnalysis<UniformityInfoWrapperPass>().getUniformityInfo();
 
+  auto *DTWP = getAnalysisIfAvailable<DominatorTreeWrapperPass>();
+  DT = DTWP ? &DTWP->getDomTree() : nullptr;
+
   bool Changed = false;
   for (auto &BB : F)
     for (Instruction &I : llvm::make_early_inc_range(BB))
       Changed |= visit(I);
 
   return Changed;
 }
 
bool AMDGPULateCodeGenPrepare::visitLoadInst(LoadInst &LI) {
   auto *NewVal = IRB.CreateBitCast(
       IRB.CreateTrunc(IRB.CreateLShr(NewLd, ShAmt), IntNTy), LI.getType());
   LI.replaceAllUsesWith(NewVal);
   RecursivelyDeleteTriviallyDeadInstructions(&LI);
 
   return true;
 }
 
+bool AMDGPULateCodeGenPrepare::visitIntrinsicInst(IntrinsicInst &I) {
+  switch (I.getIntrinsicID()) {
+  case Intrinsic::fabs:
+    return cloneInstructionToUsers(&I);
+  default:
+    return false;
+  }
+}
+
+/**
+ * This function enables the SelectionDAG to use some modifiers more often.
+ * In some cases, like after inlining a function which introduces a call to
+ * the llvm.fabs intrinsic, the IR ends up with users which reside outside
+ * of the basic block. The generated code then will be suboptimal because
+ * instruction selection cannot apply the abs modifier.
+ * In this function, all users of a given instruction are recorded if
+ * they don't reside in the same BB, and are dominated by the intrinsic call.
+ * A unique clone of the intrinsic call will be inserted at the earliest point
+ * in the BB and then all recorded users are adjusted to use the clone instead.
+ * For now, this is only implemented for the fabs intrinsic.
+ * It appears that this already works for fneg instructions.
+ * NOTE: This could be adjusted to move intrinsic calls to their use if there is
+ * only a single user outside of the caller BB.
+ *
+ * @param I The instruction to clone
+ * @return Whether something has changed or not.
+ */
+bool AMDGPULateCodeGenPrepare::cloneInstructionToUsers(Instruction *I) {
+  using namespace PatternMatch;
+
+  bool Changed = false;
+  IntrinsicInst *Intrinsic = dyn_cast<IntrinsicInst>(I);
+
+  // Early opt-out as we don't want to operate on undef or poison values.
+  if (Intrinsic && hasUndefOrPoisonOperand(*Intrinsic))
+    return false;
+
+  DenseMap<BasicBlock *, SmallVector<Instruction *, 1>> UsersPerBB;
+  bool CanErase = true;
+
+  // Filter all users which we don't want to operate on, keep the rest.
+  for (User *U : I->users()) {
+    Instruction *UI = cast<Instruction>(U);
+    if (!UI)
+      continue;
+
+    // Don't do anything if there is a user in the same BB.
+    if (UI->getParent() == I->getParent()) {
+      CanErase = false;
+      continue;
+    }
+
+    // Ignore complex control flow for now.
+    const bool IsPhi = isa<PHINode>(UI);
+
+    // Removing the original instruction will cause badref for
+    // phi nodes, so ignore these cases as well.
+    if (IsPhi || !DT || !DT->dominates(I, UI)) {
+      CanErase = false;
+      continue;
+    }
+
+    if (Intrinsic) {
+      // Don't generate fabs(fabs(x)) calls.
+      if (auto *UII = dyn_cast<IntrinsicInst>(UI);
+          UII && (UII->getIntrinsicID() == Intrinsic->getIntrinsicID()))
+        continue;
+    }
+
+    // Record the user so we can process on it later.
+    UsersPerBB[UI->getParent()].push_back(UI);
+  }
+
+  DenseMap<BasicBlock *, Instruction *> ClonesPerBB;
+
+  // Generate one clone for each BB.
+  for (auto &[BB, Users] : UsersPerBB) {
+    Instruction *Clone = nullptr;
+    if (ClonesPerBB.contains(BB)) {
+      Clone = ClonesPerBB[BB];
+    } else {
+      Clone = I->clone();
+      Clone->setName(I->getName() + Twine(".clone"));
+      Clone->insertBefore(&*BB->getFirstInsertionPt());
+      ClonesPerBB[BB] = Clone;
+    }
+
+    // Adjust the users so they use the BB-local clone.
+    for (Instruction *UI : Users) {
+      UI->replaceUsesOfWith(I, Clone);
+      Changed = true;
+    }
+  }
+
+  if (Changed && CanErase)
+    I->eraseFromParent();
+
+  return Changed;
+}
+
+bool AMDGPULateCodeGenPrepare::hasUndefOrPoisonOperand(IntrinsicInst &I,
+                                                       unsigned Operand) const {
+  using PatternMatch::m_Poison;
+  using PatternMatch::m_Undef;
+
+  return match(I.getOperand(Operand), m_Undef()) ||
+         match(I.getOperand(Operand), m_Poison());
+}
+
 INITIALIZE_PASS_BEGIN(AMDGPULateCodeGenPrepare, DEBUG_TYPE,
                       "AMDGPU IR late optimizations", false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
 INITIALIZE_PASS_DEPENDENCY(UniformityInfoWrapperPass)
 INITIALIZE_PASS_END(AMDGPULateCodeGenPrepare, DEBUG_TYPE,
                     "AMDGPU IR late optimizations", false, false)
 
 char AMDGPULateCodeGenPrepare::ID = 0;
 
 FunctionPass *llvm::createAMDGPULateCodeGenPreparePass() {
   return new AMDGPULateCodeGenPrepare();
 }

llvm/test/CodeGen/AMDGPU/bug-sdag-emitcopyfromreg.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 < %s | FileCheck %s -check-prefix=ISA
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 -stop-before=si-fix-sgpr-copies < %s | FileCheck %s -check-prefix=MIR
 
 define void @f(i32 %arg, ptr %ptr) {
 ; ISA-LABEL: f:
 ; ISA:       ; %bb.0: ; %bb
 ; ISA-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; ISA-NEXT:    s_waitcnt_vscnt null, 0x0
 ; ISA-NEXT:    s_mov_b64 s[4:5], 0
 ; ISA-NEXT:    v_cmp_gt_i32_e32 vcc_lo, 1, v0
 ; ISA-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
 ; ISA-NEXT:    v_mov_b32_e32 v7, 0
 ; ISA-NEXT:    s_waitcnt lgkmcnt(0)
-; ISA-NEXT:    s_cmp_lg_u32 s4, 0
-; ISA-NEXT:    s_cselect_b32 s6, -1, 0
-; ISA-NEXT:    s_and_b32 s6, s6, exec_lo
-; ISA-NEXT:    s_cselect_b32 s6, s5, 0
+; ISA-NEXT:    s_lshr_b32 s6, s5, 1
 ; ISA-NEXT:    s_lshr_b32 s7, 1, s4
 ; ISA-NEXT:    s_cmp_lg_u32 s4, 0
-; ISA-NEXT:    v_cvt_f32_i32_e32 v0, s6
-; ISA-NEXT:    s_cselect_b32 s8, -1, 0
-; ISA-NEXT:    s_and_b32 s8, s8, exec_lo
-; ISA-NEXT:    s_cselect_b32 s7, s7, 0
-; ISA-NEXT:    s_lshr_b32 s5, s5, 1
-; ISA-NEXT:    s_cmp_lg_u32 s4, 0
-; ISA-NEXT:    v_cvt_f32_ubyte0_e32 v4, s7
 ; ISA-NEXT:    s_cselect_b32 s4, -1, 0
-; ISA-NEXT:    v_cndmask_b32_e64 v3, 0, 1.0, s4
+; ISA-NEXT:    v_cndmask_b32_e64 v0, 0, 1.0, s4
 ; ISA-NEXT:    s_and_b32 s4, s4, exec_lo
-; ISA-NEXT:    s_cselect_b32 s4, s5, 0
-; ISA-NEXT:    v_cvt_f32_i32_e32 v5, s4
+; ISA-NEXT:    s_cselect_b32 s4, s6, 0
+; ISA-NEXT:    s_cselect_b32 s6, s7, 0
+; ISA-NEXT:    s_cselect_b32 s5, s5, 0
+; ISA-NEXT:    v_cvt_f32_i32_e32 v3, s4
+; ISA-NEXT:    v_cvt_f32_ubyte0_e32 v4, s6
+; ISA-NEXT:    v_cvt_f32_i32_e32 v5, s5
 ; ISA-NEXT:    s_mov_b32 s4, 0
-; ISA-NEXT:    v_and_b32_e32 v5, 0x7fffffff, v5
 ; ISA-NEXT:  .LBB0_1: ; %bb14
 ; ISA-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; ISA-NEXT:    v_mov_b32_e32 v6, v7
 ; ISA-NEXT:    s_and_b32 s5, exec_lo, vcc_lo
 ; ISA-NEXT:    s_or_b32 s4, s5, s4
-; ISA-NEXT:    v_add_f32_e32 v7, v6, v3
-; ISA-NEXT:    v_add_f32_e32 v7, v7, v5
+; ISA-NEXT:    v_add_f32_e32 v7, v6, v0
+; ISA-NEXT:    v_add_f32_e64 v7, v7, |v3|
 ; ISA-NEXT:    v_add_f32_e32 v7, v7, v4
-; ISA-NEXT:    v_add_f32_e32 v7, v7, v0
+; ISA-NEXT:    v_add_f32_e32 v7, v7, v5
 ; ISA-NEXT:    s_andn2_b32 exec_lo, exec_lo, s4
 ; ISA-NEXT:    s_cbranch_execnz .LBB0_1
 ; ISA-NEXT:  ; %bb.2: ; %bb21
 ; ISA-NEXT:    s_or_b32 exec_lo, exec_lo, s4
 ; ISA-NEXT:    flat_store_dword v[1:2], v6
 ; ISA-NEXT:    s_waitcnt lgkmcnt(0)
 ; ISA-NEXT:    s_waitcnt_vscnt null, 0x0
 ; ISA-NEXT:    s_setpc_b64 s[30:31]
   ; MIR-LABEL: name: f
   ; MIR: bb.0.bb:
   ; MIR-NEXT:   successors: %bb.1(0x80000000)
   ; MIR-NEXT:   liveins: $vgpr0, $vgpr1, $vgpr2
   ; MIR-NEXT: {{  $}}
   ; MIR-NEXT:   [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr2
   ; MIR-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
   ; MIR-NEXT:   [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr0
+  ; MIR-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[COPY1]], %subreg.sub0, [[COPY]], %subreg.sub1
+  ; MIR-NEXT:   [[COPY3:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]]
   ; MIR-NEXT:   [[S_MOV_B64_:%[0-9]+]]:sreg_64 = S_MOV_B64 0
   ; MIR-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM killed [[S_MOV_B64_]], 0, 0 :: (invariant load (s64) from `ptr addrspace(4) null`, align 4294967296, addrspace 4)
-  ; MIR-NEXT:   [[COPY3:%[0-9]+]]:sreg_32 = COPY [[S_LOAD_DWORDX2_IMM]].sub1
-  ; MIR-NEXT:   [[COPY4:%[0-9]+]]:sreg_32 = COPY [[S_LOAD_DWORDX2_IMM]].sub0
-  ; MIR-NEXT:   [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 0
-  ; MIR-NEXT:   S_CMP_LG_U32 [[COPY4]], [[S_MOV_B32_]], implicit-def $scc
-  ; MIR-NEXT:   [[COPY5:%[0-9]+]]:sreg_32_xm0_xexec = COPY $scc
-  ; MIR-NEXT:   $scc = COPY [[COPY5]]
-  ; MIR-NEXT:   [[S_CSELECT_B32_:%[0-9]+]]:sreg_32 = S_CSELECT_B32 [[COPY3]], [[S_MOV_B32_]], implicit $scc
-  ; MIR-NEXT:   [[S_MOV_B32_1:%[0-9]+]]:sreg_32 = S_MOV_B32 1
-  ; MIR-NEXT:   [[S_LSHR_B32_:%[0-9]+]]:sreg_32 = S_LSHR_B32 [[S_MOV_B32_1]], [[COPY4]], implicit-def dead $scc
-  ; MIR-NEXT:   S_CMP_LG_U32 [[COPY4]], [[S_MOV_B32_]], implicit-def $scc
+  ; MIR-NEXT:   [[COPY4:%[0-9]+]]:sreg_32 = COPY [[S_LOAD_DWORDX2_IMM]].sub1
+  ; MIR-NEXT:   [[COPY5:%[0-9]+]]:sreg_32 = COPY [[S_LOAD_DWORDX2_IMM]].sub0
+  ; MIR-NEXT:   [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 1
+  ; MIR-NEXT:   [[S_LSHR_B32_:%[0-9]+]]:sreg_32 = S_LSHR_B32 [[COPY4]], [[S_MOV_B32_]], implicit-def dead $scc
+  ; MIR-NEXT:   [[S_LSHR_B32_1:%[0-9]+]]:sreg_32 = S_LSHR_B32 [[S_MOV_B32_]], [[COPY5]], implicit-def dead $scc
+  ; MIR-NEXT:   [[S_MOV_B32_1:%[0-9]+]]:sreg_32 = S_MOV_B32 0
+  ; MIR-NEXT:   S_CMP_LG_U32 [[COPY5]], [[S_MOV_B32_1]], implicit-def $scc
   ; MIR-NEXT:   [[COPY6:%[0-9]+]]:sreg_32_xm0_xexec = COPY $scc
   ; MIR-NEXT:   $scc = COPY [[COPY6]]
-  ; MIR-NEXT:   [[S_CSELECT_B32_1:%[0-9]+]]:sreg_32 = S_CSELECT_B32 killed [[S_LSHR_B32_]], [[S_MOV_B32_]], implicit $scc
-  ; MIR-NEXT:   [[S_LSHR_B32_1:%[0-9]+]]:sreg_32 = S_LSHR_B32 [[COPY3]], [[S_MOV_B32_1]], implicit-def dead $scc
-  ; MIR-NEXT:   S_CMP_LG_U32 [[COPY4]], [[S_MOV_B32_]], implicit-def $scc
-  ; MIR-NEXT:   [[COPY7:%[0-9]+]]:sreg_32_xm0_xexec = COPY $scc
-  ; MIR-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[COPY1]], %subreg.sub0, [[COPY]], %subreg.sub1
-  ; MIR-NEXT:   [[COPY8:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]]
-  ; MIR-NEXT:   $scc = COPY [[COPY7]]
-  ; MIR-NEXT:   [[S_CSELECT_B32_2:%[0-9]+]]:sreg_32 = S_CSELECT_B32 killed [[S_LSHR_B32_1]], [[S_MOV_B32_]], implicit $scc
-  ; MIR-NEXT:   [[V_CVT_F32_I32_e64_:%[0-9]+]]:vgpr_32 = V_CVT_F32_I32_e64 killed [[S_CSELECT_B32_2]], 0, 0, implicit $mode, implicit $exec
-  ; MIR-NEXT:   [[S_MOV_B32_2:%[0-9]+]]:sreg_32 = S_MOV_B32 2147483647
-  ; MIR-NEXT:   [[COPY9:%[0-9]+]]:sreg_32 = COPY [[V_CVT_F32_I32_e64_]]
-  ; MIR-NEXT:   [[S_AND_B32_:%[0-9]+]]:sgpr_32 = S_AND_B32 killed [[COPY9]], killed [[S_MOV_B32_2]], implicit-def dead $scc
-  ; MIR-NEXT:   [[S_MOV_B32_3:%[0-9]+]]:sgpr_32 = S_MOV_B32 1065353216
-  ; MIR-NEXT:   [[S_MOV_B32_4:%[0-9]+]]:sgpr_32 = S_MOV_B32 0
-  ; MIR-NEXT:   [[COPY10:%[0-9]+]]:vgpr_32 = COPY killed [[S_MOV_B32_3]]
-  ; MIR-NEXT:   [[V_CNDMASK_B32_e64_:%[0-9]+]]:vgpr_32 = V_CNDMASK_B32_e64 0, [[S_MOV_B32_4]], 0, [[COPY10]], [[COPY7]], implicit $exec
-  ; MIR-NEXT:   [[COPY11:%[0-9]+]]:sgpr_32 = COPY [[V_CNDMASK_B32_e64_]]
+  ; MIR-NEXT:   [[S_CSELECT_B32_:%[0-9]+]]:sreg_32 = S_CSELECT_B32 killed [[S_LSHR_B32_]], [[S_MOV_B32_1]], implicit $scc
+  ; MIR-NEXT:   [[V_CVT_F32_I32_e64_:%[0-9]+]]:vgpr_32 = V_CVT_F32_I32_e64 killed [[S_CSELECT_B32_]], 0, 0, implicit $mode, implicit $exec
+  ; MIR-NEXT:   [[COPY7:%[0-9]+]]:sgpr_32 = COPY [[V_CVT_F32_I32_e64_]]
+  ; MIR-NEXT:   [[S_MOV_B32_2:%[0-9]+]]:sgpr_32 = S_MOV_B32 1065353216
+  ; MIR-NEXT:   [[S_MOV_B32_3:%[0-9]+]]:sgpr_32 = S_MOV_B32 0
+  ; MIR-NEXT:   [[COPY8:%[0-9]+]]:vgpr_32 = COPY killed [[S_MOV_B32_2]]
+  ; MIR-NEXT:   [[V_CNDMASK_B32_e64_:%[0-9]+]]:vgpr_32 = V_CNDMASK_B32_e64 0, [[S_MOV_B32_3]], 0, [[COPY8]], [[COPY6]], implicit $exec
+  ; MIR-NEXT:   [[COPY9:%[0-9]+]]:sgpr_32 = COPY [[V_CNDMASK_B32_e64_]]
+  ; MIR-NEXT:   $scc = COPY [[COPY6]]
+  ; MIR-NEXT:   [[S_CSELECT_B32_1:%[0-9]+]]:sreg_32 = S_CSELECT_B32 killed [[S_LSHR_B32_1]], [[S_MOV_B32_1]], implicit $scc
   ; MIR-NEXT:   [[V_CVT_F32_UBYTE0_e64_:%[0-9]+]]:vgpr_32 = V_CVT_F32_UBYTE0_e64 killed [[S_CSELECT_B32_1]], 0, 0, implicit $exec
-  ; MIR-NEXT:   [[COPY12:%[0-9]+]]:sgpr_32 = COPY [[V_CVT_F32_UBYTE0_e64_]]
-  ; MIR-NEXT:   [[V_CVT_F32_I32_e64_1:%[0-9]+]]:vgpr_32 = V_CVT_F32_I32_e64 killed [[S_CSELECT_B32_]], 0, 0, implicit $mode, implicit $exec
-  ; MIR-NEXT:   [[COPY13:%[0-9]+]]:sgpr_32 = COPY [[V_CVT_F32_I32_e64_1]]
-  ; MIR-NEXT:   [[V_CMP_LT_I32_e64_:%[0-9]+]]:sreg_32 = V_CMP_LT_I32_e64 [[COPY2]], [[S_MOV_B32_1]], implicit $exec
-  ; MIR-NEXT:   [[COPY14:%[0-9]+]]:vreg_1 = COPY [[V_CMP_LT_I32_e64_]]
+  ; MIR-NEXT:   [[COPY10:%[0-9]+]]:sgpr_32 = COPY [[V_CVT_F32_UBYTE0_e64_]]
+  ; MIR-NEXT:   $scc = COPY [[COPY6]]
+  ; MIR-NEXT:   [[S_CSELECT_B32_2:%[0-9]+]]:sreg_32 = S_CSELECT_B32 [[COPY4]], [[S_MOV_B32_1]], implicit $scc
+  ; MIR-NEXT:   [[V_CVT_F32_I32_e64_1:%[0-9]+]]:vgpr_32 = V_CVT_F32_I32_e64 killed [[S_CSELECT_B32_2]], 0, 0, implicit $mode, implicit $exec
+  ; MIR-NEXT:   [[COPY11:%[0-9]+]]:sgpr_32 = COPY [[V_CVT_F32_I32_e64_1]]
+  ; MIR-NEXT:   [[V_CMP_LT_I32_e64_:%[0-9]+]]:sreg_32 = V_CMP_LT_I32_e64 [[COPY2]], [[S_MOV_B32_]], implicit $exec
+  ; MIR-NEXT:   [[COPY12:%[0-9]+]]:vreg_1 = COPY [[V_CMP_LT_I32_e64_]]
   ; MIR-NEXT: {{  $}}
   ; MIR-NEXT: bb.1.bb14:
   ; MIR-NEXT:   successors: %bb.2(0x04000000), %bb.1(0x7c000000)
   ; MIR-NEXT: {{  $}}
-  ; MIR-NEXT:   [[PHI:%[0-9]+]]:sreg_32 = PHI [[S_MOV_B32_]], %bb.0, %7, %bb.1
-  ; MIR-NEXT:   [[PHI1:%[0-9]+]]:sgpr_32 = PHI [[S_MOV_B32_4]], %bb.0, %8, %bb.1
-  ; MIR-NEXT:   [[COPY15:%[0-9]+]]:sreg_32 = COPY [[COPY14]]
-  ; MIR-NEXT:   [[SI_IF_BREAK:%[0-9]+]]:sreg_32 = SI_IF_BREAK [[COPY15]], [[PHI]], implicit-def dead $scc
-  ; MIR-NEXT:   [[V_ADD_F32_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, [[PHI1]], 0, [[COPY11]], 0, 0, implicit $mode, implicit $exec
-  ; MIR-NEXT:   [[V_ADD_F32_e64_1:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, killed [[V_ADD_F32_e64_]], 0, [[S_AND_B32_]], 0, 0, implicit $mode, implicit $exec
-  ; MIR-NEXT:   [[V_ADD_F32_e64_2:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, killed [[V_ADD_F32_e64_1]], 0, [[COPY12]], 0, 0, implicit $mode, implicit $exec
-  ; MIR-NEXT:   [[V_ADD_F32_e64_3:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, killed [[V_ADD_F32_e64_2]], 0, [[COPY13]], 0, 0, implicit $mode, implicit $exec
-  ; MIR-NEXT:   [[COPY16:%[0-9]+]]:sgpr_32 = COPY [[V_ADD_F32_e64_3]]
+  ; MIR-NEXT:   [[PHI:%[0-9]+]]:sreg_32 = PHI [[S_MOV_B32_1]], %bb.0, %7, %bb.1
+  ; MIR-NEXT:   [[PHI1:%[0-9]+]]:sgpr_32 = PHI [[S_MOV_B32_3]], %bb.0, %8, %bb.1
+  ; MIR-NEXT:   [[COPY13:%[0-9]+]]:sreg_32 = COPY [[COPY12]]
+  ; MIR-NEXT:   [[SI_IF_BREAK:%[0-9]+]]:sreg_32 = SI_IF_BREAK [[COPY13]], [[PHI]], implicit-def dead $scc
+  ; MIR-NEXT:   [[V_ADD_F32_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, [[PHI1]], 0, [[COPY9]], 0, 0, implicit $mode, implicit $exec
+  ; MIR-NEXT:   [[V_ADD_F32_e64_1:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, killed [[V_ADD_F32_e64_]], 2, [[COPY7]], 0, 0, implicit $mode, implicit $exec
+  ; MIR-NEXT:   [[V_ADD_F32_e64_2:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, killed [[V_ADD_F32_e64_1]], 0, [[COPY10]], 0, 0, implicit $mode, implicit $exec
+  ; MIR-NEXT:   [[V_ADD_F32_e64_3:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, killed [[V_ADD_F32_e64_2]], 0, [[COPY11]], 0, 0, implicit $mode, implicit $exec
+  ; MIR-NEXT:   [[COPY14:%[0-9]+]]:sgpr_32 = COPY [[V_ADD_F32_e64_3]]
   ; MIR-NEXT:   SI_LOOP [[SI_IF_BREAK]], %bb.1, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; MIR-NEXT:   S_BRANCH %bb.2
   ; MIR-NEXT: {{  $}}
   ; MIR-NEXT: bb.2.bb21:
   ; MIR-NEXT:   [[PHI2:%[0-9]+]]:vgpr_32 = PHI [[PHI1]], %bb.1
   ; MIR-NEXT:   [[PHI3:%[0-9]+]]:sreg_32 = PHI [[SI_IF_BREAK]], %bb.1
   ; MIR-NEXT:   SI_END_CF [[PHI3]], implicit-def dead $exec, implicit-def dead $scc, implicit $exec
-  ; MIR-NEXT:   FLAT_STORE_DWORD [[COPY8]], [[PHI2]], 0, 0, implicit $exec, implicit $flat_scr :: (store (s32) into %ir.ptr)
+  ; MIR-NEXT:   FLAT_STORE_DWORD [[COPY3]], [[PHI2]], 0, 0, implicit $exec, implicit $flat_scr :: (store (s32) into %ir.ptr)
   ; MIR-NEXT:   SI_RETURN
 bb:
   %i = load <2 x i32>, ptr addrspace(4) null, align 4294967296
   %i1 = extractelement <2 x i32> %i, i64 1
   %i2 = extractelement <2 x i32> %i, i64 0
   %i3 = lshr i32 %i1, 1
   %i4 = icmp ne i32 %i2, 0
   %i5 = select i1 %i4, i32 %i3, i32 0

llvm/test/CodeGen/AMDGPU/fold-fabs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc -march=amdgcn -mcpu=gfx1030 -verify-machineinstrs -disable-machine-sink=1 -start-before=amdgpu-late-codegenprepare < %s | FileCheck -check-prefix=GFX10 %s
 
 define float @fold_abs_in_branch(float %arg1, float %arg2) {
 ; GFX10-LABEL: fold_abs_in_branch:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    v_add_f32_e32 v0, v0, v1
 ; GFX10-NEXT:    s_mov_b32 s4, exec_lo
 ; GFX10-NEXT:    v_add_f32_e32 v1, v0, v1
 ; GFX10-NEXT:    v_add_f32_e64 v0, |v1|, |v1|
-; GFX10-NEXT:    v_and_b32_e32 v1, 0x7fffffff, v1
 ; GFX10-NEXT:    v_cmpx_nlt_f32_e32 1.0, v0
 ; GFX10-NEXT:  ; %bb.1: ; %if
-; GFX10-NEXT:    v_mul_f32_e32 v0, 0x3e4ccccd, v1
+; GFX10-NEXT:    v_mul_f32_e64 v0, 0x3e4ccccd, |v1|
 ; GFX10-NEXT:  ; %bb.2: ; %exit
 ; GFX10-NEXT:    s_or_b32 exec_lo, exec_lo, s4
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 entry:
   %0 = fadd reassoc nnan nsz arcp contract afn float %arg1, %arg2
   %1 = fadd reassoc nnan nsz arcp contract afn float %0, %arg2
   %2 = call reassoc nnan nsz arcp contract afn float @llvm.fabs.f32(float %1)
   %3 = fmul reassoc nnan nsz arcp contract afn float %2, 2.000000e+00

 
 define float @fold_abs_in_branch_multiple_users(float %arg1, float %arg2) {
 ; GFX10-LABEL: fold_abs_in_branch_multiple_users:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX10-NEXT:    v_add_f32_e32 v0, v0, v1
 ; GFX10-NEXT:    s_mov_b32 s4, exec_lo
-; GFX10-NEXT:    v_add_f32_e32 v1, v0, v1
-; GFX10-NEXT:    v_add_f32_e64 v0, |v1|, |v1|
-; GFX10-NEXT:    v_and_b32_e32 v1, 0x7fffffff, v1
-; GFX10-NEXT:    v_cmpx_nlt_f32_e32 1.0, v0
+; GFX10-NEXT:    v_add_f32_e32 v0, v0, v1
+; GFX10-NEXT:    v_add_f32_e64 v1, |v0|, |v0|
+; GFX10-NEXT:    v_cmpx_nlt_f32_e32 1.0, v1
 ; GFX10-NEXT:  ; %bb.1: ; %if
-; GFX10-NEXT:    v_mul_f32_e32 v0, 0x3e4ccccd, v1
+; GFX10-NEXT:    v_mul_f32_e64 v1, 0x3e4ccccd, |v0|
 ; GFX10-NEXT:  ; %bb.2: ; %exit
 ; GFX10-NEXT:    s_or_b32 exec_lo, exec_lo, s4
-; GFX10-NEXT:    v_add_f32_e32 v1, 2.0, v1
-; GFX10-NEXT:    v_mul_f32_e32 v0, v1, v0
+; GFX10-NEXT:    v_add_f32_e64 v0, |v0|, 2.0
+; GFX10-NEXT:    v_mul_f32_e32 v0, v0, v1
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 entry:
   %0 = fadd reassoc nnan nsz arcp contract afn float %arg1, %arg2
   %1 = fadd reassoc nnan nsz arcp contract afn float %0, %arg2
   %2 = call reassoc nnan nsz arcp contract afn float @llvm.fabs.f32(float %1)
   %3 = fmul reassoc nnan nsz arcp contract afn float %2, 2.000000e+00
   %4 = fcmp ule float %3, 1.000000e+00
   br i1 %4, label %if, label %exit