[DAGCombine] Add node in the worklist in topological order in CombineTo

[deadalnix]

Summary

This is part of an ongoing effort toward making DAGCombine process the nodes in topological order.

This is able to discover a couple of new optimizations, but also causes a couple of regression. I nevertheless chose to submit this patch for review as to start the discussion with people working on the backend so we can find a good way forward.

[deadalnix]

llvm/test/CodeGen/X86/movmsk-cmp.ll

4566

I investigated this regression.

before this patch, the DAG after type legalization and optimization looks like:

SelectionDAG has 25 nodes:
  t0: ch = EntryToken
            t62: i8 = xor t52, Constant:i8<-1>
          t55: i8 = and t62, Constant:i8<1>
          t53: i8 = and t52, Constant:i8<2>
        t56: i8 = or t55, t53
      t57: i8 = setcc t56, Constant:i8<2>, seteq:ch
    t40: i32 = select t57, Constant:i32<42>, Constant:i32<99>
  t19: ch,glue = CopyToReg t0, Register:i32 $eax, t40
        t2: v2f64,ch = CopyFromReg t0, Register:v2f64 %0
        t4: v2f64,ch = CopyFromReg t0, Register:v2f64 %1
      t41: v2i64 = setcc t2, t4, setogt:ch
    t24: i32 = X86ISD::MOVMSK t41
  t52: i8 = truncate t24
  t20: ch = X86ISD::RET_FLAG t19, TargetConstant:i32<0>, Register:i32 $eax, t19:1

After, it looks like:

SelectionDAG has 20 nodes:
  t0: ch = EntryToken
      t55: i8 = and t46, Constant:i8<1>
        t56: i8 = srl t46, Constant:i8<1>
      t21: i32 = select t56, Constant:i32<42>, Constant:i32<99>
    t22: i32 = select t55, t21, Constant:i32<99>
  t19: ch,glue = CopyToReg t0, Register:i32 $eax, t22
        t2: v2f64,ch = CopyFromReg t0, Register:v2f64 %0
        t4: v2f64,ch = CopyFromReg t0, Register:v2f64 %1
      t40: v2i64 = setcc t2, t4, setogt:ch
    t24: i32 = X86ISD::MOVMSK t40
  t46: i8 = truncate t24
  t20: ch = X86ISD::RET_FLAG t19, TargetConstant:i32<0>, Register:i32 $eax, t19:1

Arguably, the DAG after this patch looks simpler/better, so it is hard to argue this does the wrong thing here. However, the rest of the backend doesn't seems to be able to do as good of a job on this DAG. Is there something that can be done here? It seems like there is a legitimate blind spot in the X86 backend rather than a problem with this patch. @RKSimon maybe?

[deadalnix]

llvm/test/CodeGen/AArch64/arm64-abi-varargs.ll

23

This seems to throw a wrench in D70450 . @fhahn , @hans , @paquette , what do you think?

[RKSimon]

llvm/test/CodeGen/X86/movmsk-cmp.ll

4566

Adding a select(c2, select(c1, x, y), y) fold is possible.

Another option is to just alter the test - InstCombine will now optimize this code to a reduction pattern: https://simd.godbolt.org/z/hfv3YW4Y6

define i32 @PR39665_c_ray(<2 x double> %x, <2 x double> %y) {
  %cmp = fcmp ogt <2 x double> %x, %y
  %shift = shufflevector <2 x i1> %cmp, <2 x i1> poison, <2 x i32> <i32 1, i32 undef>
  %1 = and <2 x i1> %cmp, %shift
  %u = extractelement <2 x i1> %1, i64 0
  %r = select i1 %u, i32 42, i32 99
  ret i32 %r
}

[deadalnix]

llvm/test/CodeGen/X86/movmsk-cmp.ll

4566

There is already a fold for select(c2, select(c1, x, y), y) , but the X86 backend tell DAGCombine that it prefers the double select. I tried to force the transform, and I got a mixed bag out of it. Sometime it improve things, sometime it makes them worse. Overall, the result was unclear.

I'll put up a patch with the updated IR then.

[RKSimon]

llvm/test/CodeGen/X86/movmsk-cmp.ll

4566

Great - can you keep the current test as well (with a TODO comment)? We don't care much about the regression but it'd be useful to keep track of any fix that comes along.

[deadalnix]

llvm/test/CodeGen/X86/movmsk-cmp.ll

4566

Just saw that after submitting the patch. I'll update it to keep both.

[deadalnix]

llvm/test/CodeGen/X86/movmsk-cmp.ll

4566

Done, see D124756

[deadalnix]

Rebase on top of D124756 .

This ensures there is no regression for optimizd IR, so this is workable - even though there is probably a couple of improvements that the backend could benefit from.

[deadalnix]

llvm/test/CodeGen/X86/movmsk-cmp.ll

4566

I rebased and was able to confirm that the regression doesn't show up on optimized IR, so I think we are good here.

[RKSimon]

llvm/test/CodeGen/ARM/swifterror.ll

687

This looks like another lost multiple load/store ?

[deadalnix]

llvm/test/CodeGen/ARM/swifterror.ll

687

Yes, this looks similar to what is happening to arm64-abi-varargs.ll .

[deadalnix]

llvm/test/CodeGen/ARM/swifterror.ll

687

Investigating this a bit more, it looks like we get some crazy combine happening when CombineTo work in topological order that are missed otherwise.

The dag looks like this with topological CombineTo:

SelectionDAG has 58 nodes:
  t0: ch = EntryToken
  t5: i32,ch = CopyFromReg t0, Register:i32 %1
  t9: i32 = add FrameIndex:i32<-1>, Constant:i32<4>
  t11: i32,ch = CopyFromReg t0, Register:i32 %2
  t15: i32,ch = CopyFromReg t0, Register:i32 %3
  t19: i32,ch = CopyFromReg t0, Register:i32 %4
          t73: i32 = add FrameIndex:i32<-1>, Constant:i32<12>
        t68: ch = store<(store (s32) into unknown-address + 12)> t0, t19, t73, undef:i32
          t77: i32 = add FrameIndex:i32<-1>, Constant:i32<8>
        t74: ch = store<(store (s32) into unknown-address + 8)> t0, t15, t77, undef:i32
        t78: ch = store<(store (s32) into %fixed-stack.0 + 4)> t0, t11, t9, undef:i32
        t81: ch = store<(store (s32) into %fixed-stack.0)> t0, t5, FrameIndex:i32<-1>, undef:i32
      t83: ch = TokenFactor t19:1, t68, t15:1, t74, t11:1, t78, t5:1, t81
    t29: ch,glue = callseq_start t83, TargetConstant:i32<0>, TargetConstant:i32<0>
  t33: ch,glue = CopyToReg t29, Register:i32 $r0, Constant:i32<16>
  t35: ch,glue = CopyToReg t33, Register:i32 $r1, Constant:i32<0>, t33:1
  t38: ch,glue = ARMISD::CALL t35, TargetGlobalAddress:i32<i8* (i64)* @malloc> 0, Register:i32 $r0, Register:i32 $r1, RegisterMask:Untyped, t35:1
  t40: ch,glue = callseq_end t38, TargetConstant:i32<0>, TargetConstant:i32<-1>, t38:1
  t41: i32,ch,glue = CopyFromReg t40, Register:i32 $r0, t40:1
  t43: ch = CopyToReg t41:1, Register:i32 %5, t41
    t45: i32 = add nuw t41, Constant:i32<8>
  t85: ch = store<(store (s8) into %ir.tmp), trunc to i8> t43, Constant:i32<1>, t45, undef:i32
      t137: ch = store<(store (s32) into %ir.a12)> t43, t135, FrameIndex:i32<3>, undef:i32
      t128: ch = store<(store (s32) into %ir.args, align 8)> t43, t135:1, FrameIndex:i32<0>, undef:i32
      t140: ch = store<(store (s32) into %ir.a11)> t43, t122, FrameIndex:i32<2>, undef:i32
      t144: ch = store<(store (s32) into %ir.a10)> t43, t109, FrameIndex:i32<1>, undef:i32
    t147: ch = TokenFactor t137, t135:2, t128, t140, t122:2, t144, t109:1
  t64: ch,glue = CopyToReg t147, Register:i32 $r0, Constant:i32<1065353216>
  t66: ch,glue = CopyToReg t64, Register:i32 $r8, Register:i32 %5, t64:1
  t109: i32,ch = load<(load (s32))> t85, FrameIndex:i32<-1>, undef:i32
  t122: i32,i32,ch = load<(load (s32) from %fixed-stack.0 + 8), <post-inc>> t85, t9, Constant:i32<4>
  t135: i32,i32,ch = load<(load (s32)), <post-inc>> t85, t122:1, Constant:i32<4>
  t67: ch = ARMISD::RET_FLAG t66, Register:i32 $r0, Register:i32 $r8, t66:1

And without:

SelectionDAG has 58 nodes:
  t0: ch = EntryToken
  t5: i32,ch = CopyFromReg t0, Register:i32 %1
  t9: i32 = add FrameIndex:i32<-1>, Constant:i32<4>
  t11: i32,ch = CopyFromReg t0, Register:i32 %2
  t15: i32,ch = CopyFromReg t0, Register:i32 %3
  t19: i32,ch = CopyFromReg t0, Register:i32 %4
  t77: i32 = add FrameIndex:i32<-1>, Constant:i32<8>
  t73: i32 = add FrameIndex:i32<-1>, Constant:i32<12>
        t68: ch = store<(store (s32) into unknown-address + 12)> t0, t19, t73, undef:i32
        t74: ch = store<(store (s32) into unknown-address + 8)> t0, t15, t77, undef:i32
        t78: ch = store<(store (s32) into %fixed-stack.0 + 4)> t0, t11, t9, undef:i32
        t81: ch = store<(store (s32) into %fixed-stack.0)> t0, t5, FrameIndex:i32<-1>, undef:i32
      t83: ch = TokenFactor t19:1, t68, t15:1, t74, t11:1, t78, t5:1, t81
    t29: ch,glue = callseq_start t83, TargetConstant:i32<0>, TargetConstant:i32<0>
  t33: ch,glue = CopyToReg t29, Register:i32 $r0, Constant:i32<16>
  t35: ch,glue = CopyToReg t33, Register:i32 $r1, Constant:i32<0>, t33:1
  t38: ch,glue = ARMISD::CALL t35, TargetGlobalAddress:i32<i8* (i64)* @malloc> 0, Register:i32 $r0, Register:i32 $r1, RegisterMask:Untyped, t35:1
  t40: ch,glue = callseq_end t38, TargetConstant:i32<0>, TargetConstant:i32<-1>, t38:1
  t41: i32,ch,glue = CopyFromReg t40, Register:i32 $r0, t40:1
  t43: ch = CopyToReg t41:1, Register:i32 %5, t41
    t45: i32 = add nuw t41, Constant:i32<8>
  t85: ch = store<(store (s8) into %ir.tmp), trunc to i8> t43, Constant:i32<1>, t45, undef:i32
      t135: ch = store<(store (s32) into %ir.a12)> t43, t132, FrameIndex:i32<3>, undef:i32
      t127: ch = store<(store (s32) into %ir.args, align 8)> t43, t73, FrameIndex:i32<0>, undef:i32
      t138: ch = store<(store (s32) into %ir.a11)> t43, t120, FrameIndex:i32<2>, undef:i32
      t142: ch = store<(store (s32) into %ir.a10)> t43, t109, FrameIndex:i32<1>, undef:i32
    t145: ch = TokenFactor t135, t132:1, t127, t138, t120:1, t142, t109:1
  t64: ch,glue = CopyToReg t145, Register:i32 $r0, Constant:i32<1065353216>
  t66: ch,glue = CopyToReg t64, Register:i32 $r8, Register:i32 %5, t64:1
  t109: i32,ch = load<(load (s32))> t85, FrameIndex:i32<-1>, undef:i32
  t120: i32,ch = load<(load (s32))> t85, t9, undef:i32
  t132: i32,ch = load<(load (s32))> t85, t77, undef:i32
  t67: ch = ARMISD::RET_FLAG t66, Register:i32 $r0, Register:i32 $r8, t66:1

The meaningful change seems to be these crazy loads:

t122: i32,i32,ch = load<(load (s32) from %fixed-stack.0 + 8), <post-inc>> t85, t9, Constant:i32<4>
t135: i32,i32,ch = load<(load (s32)), <post-inc>> t85, t122:1, Constant:i32<4>

It'd be really helpful to have some ARM expert here to help us knowing what the right path forward is.

[efriedma]

llvm/test/CodeGen/ARM/swifterror.ll

687

We fail to form a post-inc load, which changes the register allocation, which blocks use of "ldm".

Post-inc load formation failing to trigger is a bit concerning; makes me think we somehow aren't running the relevant combine late enough. DAGCombiner::CombineToPreIndexedLoadStore is a target-independent combine with a bit of target-specific code; see getPreIndexedAddressParts().

"ldm" formation is unreliable on ARM because it runs after register allocation; I wouldn't worry about that part.

[deadalnix]

llvm/test/CodeGen/AArch64/arm64-abi-varargs.ll

23

Pre DAG:

SelectionDAG has 55 nodes:
  t0: ch = EntryToken
  t19: i32,ch = load<(load (s32) from %fixed-stack.1, align 16)> t0, FrameIndex:i64<-1>, undef:i64
    t2: i64,ch = CopyFromReg t0, Register:i64 %0
  t51: ch = store<(store (s32) into %ir.a1)> t19:1, t19, t2, undef:i64
  t98: i32,ch = load<(load (s32) from %fixed-stack.0, align 8)> t51, FrameIndex:i64<-2>, undef:i64
    t84: i64 = add FrameIndex:i64<-2>, Constant:i64<8>
  t109: i32,ch = load<(load (s32) from %fixed-stack.0 + 8, align 8)> t51, t84, undef:i64
    t108: i64 = add FrameIndex:i64<-2>, Constant:i64<16>
  t121: i32,ch = load<(load (s32) from %fixed-stack.0 + 16, align 8)> t51, t108, undef:i64
      t128: ch = store<(store (s32) into %ir.a12)> t0, t121, FrameIndex:i64<12>, undef:i64
        t120: i64 = add FrameIndex:i64<-2>, Constant:i64<24>
      t116: ch = store<(store (s64) into %ir.args)> t0, t120, FrameIndex:i64<9>, undef:i64
      t124: ch = store<(store (s32) into %ir.a11)> t0, t109, FrameIndex:i64<11>, undef:i64
      t131: ch = store<(store (s32) into %ir.a10)> t0, t98, FrameIndex:i64<10>, undef:i64
        t16: i32,ch = CopyFromReg t0, Register:i32 %7
      t53: ch = store<(store (s32) into %ir.8)> t0, t16, FrameIndex:i64<7>, undef:i64
        t14: i32,ch = CopyFromReg t0, Register:i32 %6
      t56: ch = store<(store (s32) into %ir.7)> t0, t14, FrameIndex:i64<6>, undef:i64
        t12: i32,ch = CopyFromReg t0, Register:i32 %5
      t59: ch = store<(store (s32) into %ir.6)> t0, t12, FrameIndex:i64<5>, undef:i64
        t10: i32,ch = CopyFromReg t0, Register:i32 %4
      t62: ch = store<(store (s32) into %ir.5)> t0, t10, FrameIndex:i64<4>, undef:i64
        t8: i32,ch = CopyFromReg t0, Register:i32 %3
      t65: ch = store<(store (s32) into %ir.4)> t0, t8, FrameIndex:i64<3>, undef:i64
        t6: i32,ch = CopyFromReg t0, Register:i32 %2
      t68: ch = store<(store (s32) into %ir.3)> t0, t6, FrameIndex:i64<2>, undef:i64
        t4: i32,ch = CopyFromReg t0, Register:i32 %1
      t71: ch = store<(store (s32) into %ir.2)> t0, t4, FrameIndex:i64<1>, undef:i64
    t134: ch = TokenFactor t128, t121:1, t116, t124, t109:1, t131, t98:1, t53, t56, t59, t62, t65, t68, t71
  t50: ch = AArch64ISD::RET_FLAG t134

Post DAG:

SelectionDAG has 51 nodes:
  t0: ch = EntryToken
  t19: i32,ch = load<(load (s32) from %fixed-stack.1, align 16)> t0, FrameIndex:i64<-1>, undef:i64
    t2: i64,ch = CopyFromReg t0, Register:i64 %0
  t51: ch = store<(store (s32) into %ir.a1)> t19:1, t19, t2, undef:i64
  t98: i32,ch = load<(load (s32) from %fixed-stack.0, align 8)> t51, FrameIndex:i64<-2>, undef:i64
    t84: i64 = add FrameIndex:i64<-2>, Constant:i64<8>
  t109: i32,i64,ch = load<(load (s32) from %fixed-stack.0 + 8, align 8), <post-inc>> t51, t84, Constant:i64<8>
  t120: i32,i64,ch = load<(load (s32)), <post-inc>> t51, t109:1, Constant:i64<8>
      t126: ch = store<(store (s32) into %ir.a12)> t0, t120, FrameIndex:i64<12>, undef:i64
      t115: ch = store<(store (s64) into %ir.args)> t0, t120:1, FrameIndex:i64<9>, undef:i64
      t122: ch = store<(store (s32) into %ir.a11)> t0, t109, FrameIndex:i64<11>, undef:i64
      t129: ch = store<(store (s32) into %ir.a10)> t0, t98, FrameIndex:i64<10>, undef:i64
        t16: i32,ch = CopyFromReg t0, Register:i32 %7
      t53: ch = store<(store (s32) into %ir.8)> t0, t16, FrameIndex:i64<7>, undef:i64
        t14: i32,ch = CopyFromReg t0, Register:i32 %6
      t56: ch = store<(store (s32) into %ir.7)> t0, t14, FrameIndex:i64<6>, undef:i64
        t12: i32,ch = CopyFromReg t0, Register:i32 %5
      t59: ch = store<(store (s32) into %ir.6)> t0, t12, FrameIndex:i64<5>, undef:i64
        t10: i32,ch = CopyFromReg t0, Register:i32 %4
      t62: ch = store<(store (s32) into %ir.5)> t0, t10, FrameIndex:i64<4>, undef:i64
        t8: i32,ch = CopyFromReg t0, Register:i32 %3
      t65: ch = store<(store (s32) into %ir.4)> t0, t8, FrameIndex:i64<3>, undef:i64
        t6: i32,ch = CopyFromReg t0, Register:i32 %2
      t68: ch = store<(store (s32) into %ir.3)> t0, t6, FrameIndex:i64<2>, undef:i64
        t4: i32,ch = CopyFromReg t0, Register:i32 %1
      t71: ch = store<(store (s32) into %ir.2)> t0, t4, FrameIndex:i64<1>, undef:i64
    t132: ch = TokenFactor t126, t120:2, t115, t122, t109:2, t129, t98:1, t53, t56, t59, t62, t65, t68, t71
  t50: ch = AArch64ISD::RET_FLAG t132

It is hard to argue with this one that the DAG after this change to the combiner is worse, it definitively seems better to me.

llvm/test/CodeGen/AArch64/swifterror.ll

946

Pre DAG:

SelectionDAG has 41 nodes:
      t0: ch = EntryToken
    t6: ch,glue = callseq_start t0, TargetConstant:i64<0>, TargetConstant:i64<0>
  t10: ch,glue = CopyToReg t6, Register:i64 $x0, Constant:i64<16>
  t13: ch,glue = AArch64ISD::CALL t10, TargetGlobalAddress:i64<i8* (i64)* @malloc> 0, Register:i64 $x0, RegisterMask:Untyped, t10:1
  t14: ch,glue = callseq_end t13, TargetConstant:i64<0>, TargetConstant:i64<0>, t13:1
  t15: i64,ch,glue = CopyFromReg t14, Register:i64 $x0, t14:1
  t17: ch = CopyToReg t15:1, Register:i64 %1, t15
    t19: i64 = add nuw t15, Constant:i64<8>
  t45: ch = store<(store (s8) into %ir.tmp), trunc to i8> t17, Constant:i32<1>, t19, undef:i64
      t100: ch = store<(store (s32) into %ir.a12)> t17, t97, FrameIndex:i64<3>, undef:i64
        t95: i64 = add FrameIndex:i64<-1>, Constant:i64<24>
      t90: ch = store<(store (s64) into %ir.args)> t17, t95, FrameIndex:i64<0>, undef:i64
      t103: ch = store<(store (s32) into %ir.a11)> t17, t83, FrameIndex:i64<2>, undef:i64
      t107: ch = store<(store (s32) into %ir.a10)> t17, t71, FrameIndex:i64<1>, undef:i64
    t110: ch = TokenFactor t100, t97:1, t90, t103, t83:1, t107, t71:1
  t40: ch,glue = CopyToReg t110, Register:f32 $s0, ConstantFP:f32<1.000000e+00>
  t42: ch,glue = CopyToReg t40, Register:i64 $x21, Register:i64 %1, t40:1
  t71: i32,ch = load<(load (s32) from %fixed-stack.0, align 16)> t45, FrameIndex:i64<-1>, undef:i64
    t69: i64 = or FrameIndex:i64<-1>, Constant:i64<8>
  t83: i32,ch = load<(load (s32), align 8)> t45, t69, undef:i64
    t81: i64 = add FrameIndex:i64<-1>, Constant:i64<16>
  t97: i32,ch = load<(load (s32) from %fixed-stack.0 + 16, align 16)> t45, t81, undef:i64
  t43: ch = AArch64ISD::RET_FLAG t42, Register:f32 $s0, Register:i64 $x21, t42:1

Post DAG:

SelectionDAG has 38 nodes:
      t0: ch = EntryToken
    t6: ch,glue = callseq_start t0, TargetConstant:i64<0>, TargetConstant:i64<0>
  t10: ch,glue = CopyToReg t6, Register:i64 $x0, Constant:i64<16>
  t13: ch,glue = AArch64ISD::CALL t10, TargetGlobalAddress:i64<i8* (i64)* @malloc> 0, Register:i64 $x0, RegisterMask:Untyped, t10:1
  t14: ch,glue = callseq_end t13, TargetConstant:i64<0>, TargetConstant:i64<0>, t13:1
  t15: i64,ch,glue = CopyFromReg t14, Register:i64 $x0, t14:1
  t17: ch = CopyToReg t15:1, Register:i64 %1, t15
    t19: i64 = add nuw t15, Constant:i64<8>
  t45: ch = store<(store (s8) into %ir.tmp), trunc to i8> t17, Constant:i32<1>, t19, undef:i64
      t99: ch = store<(store (s32) into %ir.a12)> t17, t97, FrameIndex:i64<3>, undef:i64
      t90: ch = store<(store (s64) into %ir.args)> t17, t97:1, FrameIndex:i64<0>, undef:i64
      t102: ch = store<(store (s32) into %ir.a11)> t17, t84, FrameIndex:i64<2>, undef:i64
      t106: ch = store<(store (s32) into %ir.a10)> t17, t70, FrameIndex:i64<1>, undef:i64
    t109: ch = TokenFactor t99, t97:2, t90, t102, t84:2, t106, t70:1
  t40: ch,glue = CopyToReg t109, Register:f32 $s0, ConstantFP:f32<1.000000e+00>
  t42: ch,glue = CopyToReg t40, Register:i64 $x21, Register:i64 %1, t40:1
  t70: i32,ch = load<(load (s32) from %fixed-stack.0, align 16)> t45, FrameIndex:i64<-1>, undef:i64
    t73: i64 = or FrameIndex:i64<-1>, Constant:i64<8>
  t84: i32,i64,ch = load<(load (s32), align 8), <post-inc>> t45, t73, Constant:i64<8>
  t97: i32,i64,ch = load<(load (s32)), <post-inc>> t45, t84:1, Constant:i64<8>
  t43: ch = AArch64ISD::RET_FLAG t42, Register:f32 $s0, Register:i64 $x21, t42:1

Once again, this looks like it is improving things at the DAGCombine level, but trips up later stages in the backend.

llvm/test/CodeGen/ARM/swifterror.ll

687

I have to apologize because I now notice a major error in my previous message. I swapped the two DAGs. The post-inc loads are in this diff's version of the DAG, and they do not exists on main. This has typically be the pattern with these changes: they allow DAGCombine to do more, but sometime, later stages in the pipeline gets confused. Nevertheless, if ldm is combine post register allocation, then it is indeed going to be fragile.

When you say that you wouldn't worry about this, do you mean that, because ldm is generally unreliable (as it happen post register allocation), then we should proceed regardless of this specific change, assuming there is no other outstanding open question?

[efriedma]

llvm/test/CodeGen/ARM/swifterror.ll

687

Oh, we're forming more post-inc loads? Then sure, this test change is fine as-is.

If it causes issues in practice, we can mess with the post-inc formation heuristics.

[efriedma]

llvm/test/CodeGen/AArch64/arm64-abi-varargs.ll

23

It looks like post-inc formation is triggering more on AArch64 as well?

I think it's just something we're going to have to watch out for; if it causes practical issues, we might need to make post-inc formation less aggressive.

[deadalnix]

llvm/test/CodeGen/ARM/swifterror.ll

687

Yes, we are forming more of them. In a way, this is the whole point of this patch: traverse the node in a way that exposes more combining opportunities, but in this case, it looks like we either don't want to do the combine as aggressively, or we want to handle it better in later stages. Regressions in the ARM and AArch64 backends are variation around that same theme.

Now the question is what do we do from there? Do we land this and try to mess with the heuristics?

[efriedma]

llvm/test/CodeGen/ARM/swifterror.ll

687

I don't think the specific cases here really represent a general trend... but it's going to be hard to tell for sure without actually landing it, and seeing the fallout.

If the cases here do represent a trend, probably the first thing I'd look at is restricting the formation of post-inc memory ops that refer to an offset from a frame index.

[dmgreen]

I ran some quick benchmarks and didn't notice anything alarming. Almost nothing changed across the AArch64/Thumb2/Thumb1 tests I tried (at least in terms of codesize or performance). There can always be different code that might behave differently, but no objections from me.

[deadalnix]

Fix aix32-cc-abi-vaarg.ll

[deadalnix]

Ok, this looks like it is good to go then. @RKSimon , if this is good for you, then we shoudl land this.

[RKSimon]

LGTM cheers

[fhahn]

llvm/test/CodeGen/AArch64/swifterror.ll

946

It would be good to investigate what's blocking the load/store optimizer here.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

assert((!To[i].getNode() ||
             N->getValueType(i) == To[i].getValueType()) &&
            "Cannot combine value to value of different type!");
 
   WorklistRemover DeadNodes(*this);
   DAG.ReplaceAllUsesWith(N, To);
   if (AddTo) {
     // Push the new nodes and any users onto the worklist
     for (unsigned i = 0, e = NumTo; i != e; ++i) {
-      if (To[i].getNode()) {
-        AddToWorklist(To[i].getNode());
-        AddUsersToWorklist(To[i].getNode());
-      }
+      if (To[i].getNode())
+        AddToWorklistWithUsers(To[i].getNode());
     }
   }
 
   // Finally, if the node is now dead, remove it from the graph.  The node
   // may not be dead if the replacement process recursively simplified to
   // something else needing this node.
   if (N->use_empty())
     deleteAndRecombine(N);

llvm/test/CodeGen/AArch64/arm64-abi-varargs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-load-store-renaming=true < %s -mtriple=arm64-apple-ios7.0.0 -mcpu=cyclone -enable-misched=false | FileCheck %s
 
 ; rdar://13625505
 ; Here we have 9 fixed integer arguments the 9th argument in on stack, the
 ; varargs start right after at 8-byte alignment.
 define void @fn9(i32* %a1, i32 %a2, i32 %a3, i32 %a4, i32 %a5, i32 %a6, i32 %a7, i32 %a8, i32 %a9, ...) nounwind noinline ssp {
 ; CHECK-LABEL: fn9:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    sub sp, sp, #64
 ; CHECK-NEXT:    ldr w8, [sp, #64]
 ; CHECK-NEXT:    stp w2, w1, [sp, #52]
 ; CHECK-NEXT:    stp w4, w3, [sp, #44]
 ; CHECK-NEXT:    stp w6, w5, [sp, #36]
 ; CHECK-NEXT:    str w7, [sp, #32]
 ; CHECK-NEXT:    str w8, [x0]
-; CHECK-NEXT:    ldr w9, [sp, #72]
-; CHECK-NEXT:    ldr w8, [sp, #80]
-; CHECK-NEXT:    stp w8, w9, [sp, #16]
 ; CHECK-NEXT:    add x8, sp, #72
-; CHECK-NEXT:    add x8, x8, #24
+; CHECK-NEXT:    add x8, x8, #8
+; CHECK-NEXT:    ldr w9, [sp, #72]
+; CHECK-NEXT:    str w9, [sp, #20]
+; CHECK-NEXT:    ldr w9, [x8], #8
+; CHECK-NEXT:    str w9, [sp, #16]
+; CHECK-NEXT:    ldr w9, [x8], #8
 ; CHECK-NEXT:    str x8, [sp, #24]
-; CHECK-NEXT:    ldr w8, [sp, #88]
-; CHECK-NEXT:    str w8, [sp, #12]
+; CHECK-NEXT:    str w9, [sp, #12]
 ; CHECK-NEXT:    add sp, sp, #64
 ; CHECK-NEXT:    ret
   %1 = alloca i32, align 4
   %2 = alloca i32, align 4
   %3 = alloca i32, align 4
   %4 = alloca i32, align 4
   %5 = alloca i32, align 4
   %6 = alloca i32, align 4

llvm/test/CodeGen/AArch64/pr51476.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-unknown-linux-gnu < %s | FileCheck %s
 
 define void @test(i8 %arg) nounwind {
 ; CHECK-LABEL: test:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    str x30, [sp, #-16]! // 8-byte Folded Spill
-; CHECK-NEXT:    and w8, w0, #0xff
-; CHECK-NEXT:    cmp w8, #1
+; CHECK-NEXT:    and w9, w0, #0xff
+; CHECK-NEXT:    mov w8, #1
+; CHECK-NEXT:    cmp w9, #1
 ; CHECK-NEXT:    cset w0, ne
-; CHECK-NEXT:    cmp w0, #3
 ; CHECK-NEXT:    strb w0, [sp, #12]
-; CHECK-NEXT:    b.eq .LBB0_2
+; CHECK-NEXT:    cbz w8, .LBB0_2
 ; CHECK-NEXT:  // %bb.1: // %do_call
 ; CHECK-NEXT:    bl unknown
 ; CHECK-NEXT:  .LBB0_2: // %common.ret
 ; CHECK-NEXT:    ldr x30, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
   %tmp = alloca i8
   %cmp1 = icmp ne i8 %arg, 1
   %zext = zext i1 %cmp1 to i8

llvm/test/CodeGen/AArch64/swifterror.ll

 ; CHECK-APPLE-AARCH64-NEXT:    stp x29, x30, [sp, #32] ; 16-byte Folded Spill
 ; CHECK-APPLE-AARCH64-NEXT:    add x29, sp, #32
 ; CHECK-APPLE-AARCH64-NEXT:    .cfi_def_cfa w29, 16
 ; CHECK-APPLE-AARCH64-NEXT:    .cfi_offset w30, -8
 ; CHECK-APPLE-AARCH64-NEXT:    .cfi_offset w29, -16
 ; CHECK-APPLE-AARCH64-NEXT:    mov w0, #16
 ; CHECK-APPLE-AARCH64-NEXT:    bl _malloc
 ; CHECK-APPLE-AARCH64-NEXT:    mov w8, #1
-; CHECK-APPLE-AARCH64-NEXT:    ldr w9, [x29, #16]
-; CHECK-APPLE-AARCH64-NEXT:    add x10, x29, #16
-; CHECK-APPLE-AARCH64-NEXT:    ldr w11, [x29, #32]
+; CHECK-APPLE-AARCH64-NEXT:    add x9, x29, #16
+; CHECK-APPLE-AARCH64-NEXT:    ldr w10, [x29, #16]
+; CHECK-APPLE-AARCH64-NEXT:    orr x9, x9, #0x8
 ; CHECK-APPLE-AARCH64-NEXT:    strb w8, [x0, #8]
-; CHECK-APPLE-AARCH64-NEXT:    add x8, x10, #24
-; CHECK-APPLE-AARCH64-NEXT:    stur w9, [x29, #-12]
-; CHECK-APPLE-AARCH64-NEXT:    ldr w9, [x29, #24]
+; CHECK-APPLE-AARCH64-NEXT:    stur w10, [x29, #-12]
+; CHECK-APPLE-AARCH64-NEXT:    ldr w8, [x9], #8
+; CHECK-APPLE-AARCH64-NEXT:    str w8, [sp, #16]
+; CHECK-APPLE-AARCH64-NEXT:    ldr w8, [x9], #8
 ; CHECK-APPLE-AARCH64-NEXT:    fmov s0, #1.00000000
 ; CHECK-APPLE-AARCH64-NEXT:    mov x21, x0
-; CHECK-APPLE-AARCH64-NEXT:    stur x8, [x29, #-8]
-; CHECK-APPLE-AARCH64-NEXT:    stp w11, w9, [sp, #12]
+; CHECK-APPLE-AARCH64-NEXT:    stur x9, [x29, #-8]
+; CHECK-APPLE-AARCH64-NEXT:    str w8, [sp, #12]
 ; CHECK-APPLE-AARCH64-NEXT:    ldp x29, x30, [sp, #32] ; 16-byte Folded Reload
 ; CHECK-APPLE-AARCH64-NEXT:    add sp, sp, #48
 ; CHECK-APPLE-AARCH64-NEXT:    ret
 ;
 ; CHECK-O0-AARCH64-LABEL: foo_vararg:
 ; CHECK-O0-AARCH64:       ; %bb.0: ; %entry
 ; CHECK-O0-AARCH64-NEXT:    sub sp, sp, #48
 ; CHECK-O0-AARCH64-NEXT:    .cfi_def_cfa_offset 48

llvm/test/CodeGen/ARM/and-load-combine.ll

entry:
   %2 = and i16 %and3, %1
   %cmp = icmp eq i16 %2, 0
   ret i1 %cmp
 }
 
 define arm_aapcscc zeroext i1 @cmp_and8_short_int(i16* nocapture readonly %a, i32* nocapture readonly %b) {
 ; ARM-LABEL: cmp_and8_short_int:
 ; ARM:       @ %bb.0: @ %entry
-; ARM-NEXT:    ldrb r1, [r1]
 ; ARM-NEXT:    ldrb r0, [r0]
-; ARM-NEXT:    and r0, r0, r1
+; ARM-NEXT:    ldr r1, [r1]
+; ARM-NEXT:    and r0, r1, r0
 ; ARM-NEXT:    clz r0, r0
 ; ARM-NEXT:    lsr r0, r0, #5
 ; ARM-NEXT:    bx lr
 ;
 ; ARMEB-LABEL: cmp_and8_short_int:
 ; ARMEB:       @ %bb.0: @ %entry
-; ARMEB-NEXT:    ldrb r1, [r1, #3]
 ; ARMEB-NEXT:    ldrb r0, [r0, #1]
-; ARMEB-NEXT:    and r0, r0, r1
+; ARMEB-NEXT:    ldr r1, [r1]
+; ARMEB-NEXT:    and r0, r1, r0
 ; ARMEB-NEXT:    clz r0, r0
 ; ARMEB-NEXT:    lsr r0, r0, #5
 ; ARMEB-NEXT:    bx lr
 ;
 ; THUMB1-LABEL: cmp_and8_short_int:
 ; THUMB1:       @ %bb.0: @ %entry
-; THUMB1-NEXT:    ldrb r1, [r1]
-; THUMB1-NEXT:    ldrb r2, [r0]
-; THUMB1-NEXT:    ands r2, r1
-; THUMB1-NEXT:    rsbs r0, r2, #0
-; THUMB1-NEXT:    adcs r0, r2
+; THUMB1-NEXT:    ldrb r0, [r0]
+; THUMB1-NEXT:    ldr r1, [r1]
+; THUMB1-NEXT:    ands r1, r0
+; THUMB1-NEXT:    rsbs r0, r1, #0
+; THUMB1-NEXT:    adcs r0, r1
 ; THUMB1-NEXT:    bx lr
 ;
 ; THUMB2-LABEL: cmp_and8_short_int:
 ; THUMB2:       @ %bb.0: @ %entry
-; THUMB2-NEXT:    ldrb r1, [r1]
 ; THUMB2-NEXT:    ldrb r0, [r0]
+; THUMB2-NEXT:    ldr r1, [r1]
 ; THUMB2-NEXT:    ands r0, r1
 ; THUMB2-NEXT:    clz r0, r0
 ; THUMB2-NEXT:    lsrs r0, r0, #5
 ; THUMB2-NEXT:    bx lr
 entry:
   %0 = load i16, i16* %a, align 2
   %1 = load i32, i32* %b, align 4
   %2 = and i16 %0, 255

llvm/test/CodeGen/ARM/swifterror.ll

 ; CHECK-APPLE-NEXT:    bfc sp, #0, #3
 ; CHECK-APPLE-NEXT:    add r8, r7, #8
 ; CHECK-APPLE-NEXT:    stm r8, {r0, r1, r2, r3}
 ; CHECK-APPLE-NEXT:    mov r0, #16
 ; CHECK-APPLE-NEXT:    mov r1, #0
 ; CHECK-APPLE-NEXT:    bl _malloc
 ; CHECK-APPLE-NEXT:    mov r8, r0
 ; CHECK-APPLE-NEXT:    mov r0, #1
-; CHECK-APPLE-NEXT:    add r3, r7, #8
 ; CHECK-APPLE-NEXT:    strb r0, [r8, #8]
 ; CHECK-APPLE-NEXT:    add r0, r7, #8
-; CHECK-APPLE-NEXT:    ldm r3, {r1, r2, r3}
-; CHECK-APPLE-NEXT:    add r0, r0, #12
+; CHECK-APPLE-NEXT:    add r0, r0, #4
+; CHECK-APPLE-NEXT:    ldr r2, [r7, #8]
+; CHECK-APPLE-NEXT:    ldr r1, [r0], #4
+; CHECK-APPLE-NEXT:    ldr r3, [r0], #4
 ; CHECK-APPLE-NEXT:    str r0, [sp, #16]
 ; CHECK-APPLE-NEXT:    mov r0, #1065353216
-; CHECK-APPLE-NEXT:    str r1, [sp, #12]
-; CHECK-APPLE-NEXT:    str r2, [sp, #8]
+; CHECK-APPLE-NEXT:    str r2, [sp, #12]
+; CHECK-APPLE-NEXT:    str r1, [sp, #8]
 ; CHECK-APPLE-NEXT:    str r3, [sp, #4]
 ; CHECK-APPLE-NEXT:    mov sp, r7
 ; CHECK-APPLE-NEXT:    pop {r7, lr}
 ; CHECK-APPLE-NEXT:    add sp, sp, #16
 ; CHECK-APPLE-NEXT:    bx lr
 ;
 ; CHECK-O0-LABEL: foo_vararg:
 ; CHECK-O0:       @ %bb.0: @ %entry

 ; CHECK-ANDROID-NEXT:    sub sp, sp, #24
 ; CHECK-ANDROID-NEXT:    add r8, sp, #32
 ; CHECK-ANDROID-NEXT:    stm r8, {r0, r1, r2, r3}
 ; CHECK-ANDROID-NEXT:    mov r0, #16
 ; CHECK-ANDROID-NEXT:    mov r1, #0
 ; CHECK-ANDROID-NEXT:    bl malloc
 ; CHECK-ANDROID-NEXT:    mov r8, r0
 ; CHECK-ANDROID-NEXT:    mov r0, #1
-; CHECK-ANDROID-NEXT:    add r3, sp, #32
 ; CHECK-ANDROID-NEXT:    strb r0, [r8, #8]
 ; CHECK-ANDROID-NEXT:    add r0, sp, #32
-; CHECK-ANDROID-NEXT:    ldm r3, {r1, r2, r3}
-; CHECK-ANDROID-NEXT:    add r0, r0, #12
+; CHECK-ANDROID-NEXT:    orr r0, r0, #4
+; CHECK-ANDROID-NEXT:    ldr r2, [sp, #32]
+; CHECK-ANDROID-NEXT:    ldr r1, [r0], #4
+; CHECK-ANDROID-NEXT:    ldr r3, [r0], #4
 ; CHECK-ANDROID-NEXT:    str r0, [sp, #16]
 ; CHECK-ANDROID-NEXT:    mov r0, #1065353216
-; CHECK-ANDROID-NEXT:    str r1, [sp, #12]
-; CHECK-ANDROID-NEXT:    str r2, [sp, #8]
+; CHECK-ANDROID-NEXT:    str r2, [sp, #12]
+; CHECK-ANDROID-NEXT:    str r1, [sp, #8]
 ; CHECK-ANDROID-NEXT:    str r3, [sp, #4]
 ; CHECK-ANDROID-NEXT:    add sp, sp, #24
 ; CHECK-ANDROID-NEXT:    pop {r11, lr}
 ; CHECK-ANDROID-NEXT:    add sp, sp, #16
 ; CHECK-ANDROID-NEXT:    bx lr
 
 entry:
   %call = call i8* @malloc(i64 16)

llvm/test/CodeGen/PowerPC/aix32-cc-abi-vaarg.ll

 ; 32BIT-DAG:     renamable $r3 = nsw ADD4 killed renamable $r3, renamable $r4
 ; 32BIT-DAG:     renamable $r4 = LWZ 0, %fixed-stack.0 :: (load (s32) from %ir.4, align 8)
 ; 32BIT-DAG:     renamable $r11 = LI 4
 ; 32BIT-DAG:     BLR implicit $lr, implicit $rm, implicit $r3
 
 define double @double_va_arg(double %a, ...) local_unnamed_addr  {
 ; ASM32-LABEL: double_va_arg:
 ; ASM32:       # %bb.0: # %entry
-; ASM32-NEXT:    stw 6, -12(1)
-; ASM32-NEXT:    addi 3, 1, 32
 ; ASM32-NEXT:    stw 5, -16(1)
+; ASM32-NEXT:    addi 3, 1, 32
+; ASM32-NEXT:    stw 6, -12(1)
 ; ASM32-NEXT:    lfd 0, -16(1)
-; ASM32-NEXT:    stw 6, -20(1)
-; ASM32-NEXT:    fadd 0, 0, 1
 ; ASM32-NEXT:    stw 5, -24(1)
+; ASM32-NEXT:    fadd 0, 0, 1
+; ASM32-NEXT:    stw 6, -20(1)
 ; ASM32-NEXT:    lfd 1, -24(1)
 ; ASM32-NEXT:    fadd 1, 1, 1
 ; ASM32-NEXT:    stw 7, 40(1)
 ; ASM32-NEXT:    fadd 1, 0, 1
 ; ASM32-NEXT:    stw 5, 32(1)
 ; ASM32-NEXT:    stw 6, 36(1)
 ; ASM32-NEXT:    stw 8, 44(1)
 ; ASM32-NEXT:    stw 9, 48(1)

 ; 32BIT-DAG:     STW renamable $r3, 0, %stack.0.arg1 :: (store (s32) into %ir.0)
 ; 32BIT-DAG:     STW killed renamable $r3, 0, %stack.1.arg2 :: (store (s32) into %ir.1)
 ; 32BIT-DAG:     BLR implicit $lr, implicit $rm, implicit $f1
 
 define double @double_stack_va_arg(double %one, double %two, double %three, double %four, double %five, double %six, double %seven, double %eight, double %nine, double %ten, double %eleven, double %twelve, double %thirteen, ...) local_unnamed_addr  {
 ; ASM32-LABEL: double_stack_va_arg:
 ; ASM32:       # %bb.0: # %entry
 ; ASM32-NEXT:    fadd 0, 1, 2
-; ASM32-NEXT:    addi 3, 1, 128
-; ASM32-NEXT:    lwz 4, 132(1)
+; ASM32-NEXT:    addi 4, 1, 128
+; ASM32-NEXT:    lwz 3, 132(1)
 ; ASM32-NEXT:    fadd 0, 0, 3
-; ASM32-NEXT:    stw 3, -4(1)
+; ASM32-NEXT:    stw 4, -4(1)
 ; ASM32-NEXT:    fadd 0, 0, 4
-; ASM32-NEXT:    lwz 3, 128(1)
+; ASM32-NEXT:    lwz 4, 128(1)
 ; ASM32-NEXT:    fadd 0, 0, 5
-; ASM32-NEXT:    stw 3, -16(1)
+; ASM32-NEXT:    stw 3, -12(1)
 ; ASM32-NEXT:    fadd 0, 0, 6
-; ASM32-NEXT:    stw 4, -12(1)
+; ASM32-NEXT:    stw 4, -16(1)
 ; ASM32-NEXT:    fadd 0, 0, 7
 ; ASM32-NEXT:    lfd 1, -16(1)
 ; ASM32-NEXT:    fadd 0, 0, 8
-; ASM32-NEXT:    stw 3, -24(1)
+; ASM32-NEXT:    stw 3, -20(1)
 ; ASM32-NEXT:    fadd 0, 0, 9
-; ASM32-NEXT:    stw 4, -20(1)
+; ASM32-NEXT:    stw 4, -24(1)
 ; ASM32-NEXT:    fadd 0, 0, 10
 ; ASM32-NEXT:    fadd 0, 0, 11
 ; ASM32-NEXT:    fadd 0, 0, 12
 ; ASM32-NEXT:    fadd 0, 0, 13
 ; ASM32-NEXT:    fadd 0, 0, 1
 ; ASM32-NEXT:    lfd 1, -24(1)
 ; ASM32-NEXT:    fadd 1, 1, 1
 ; ASM32-NEXT:    fadd 1, 0, 1

 ; 32BIT-LABEL:   stack:
 ; 32BIT-DAG:     - { id: 0, name: arg1, type: default, offset: 0, size: 4, alignment: 4,
 ; 32BIT-DAG:     - { id: 1, name: arg2, type: default, offset: 0, size: 4, alignment: 4,
 ; 32BIT-DAG:     - { id: 2, name: '', type: default, offset: 0, size: 8, alignment: 8,
 ; 32BIT-DAG:     - { id: 3, name: '', type: default, offset: 0, size: 8, alignment: 8,
 
 ; 32BIT-LABEL:   body:             |
 ; 32BIT-DAG:     liveins: $f1, $f2, $f3, $f4, $f5, $f6, $f7, $f8, $f9, $f10, $f11, $f12, $f13
-; 32BIT-DAG:     renamable $r3 = ADDI %fixed-stack.0, 0
-; 32BIT-DAG:     STW killed renamable $r3, 0, %stack.0.arg1 :: (store (s32) into %ir.0)
-; 32BIT-DAG:     renamable $r3 = LWZ 0, %fixed-stack.0 :: (load (s32) from %ir.argp.cur142, align 16)
+; 32BIT-DAG:     renamable $r4 = ADDI %fixed-stack.0, 0
+; 32BIT-DAG:     STW killed renamable $r4, 0, %stack.0.arg1 :: (store (s32) into %ir.0)
+; 32BIT-DAG:     renamable $r4 = LWZ 0, %fixed-stack.0 :: (load (s32) from %ir.argp.cur142, align 16)
 ; 32BIT-DAG:     renamable $f1 = nofpexcept FADD killed renamable $f0, killed renamable $f1, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f1, killed renamable $f2, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f3, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f4, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f5, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f6, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f7, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f8, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f9, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f10, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f11, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f12, implicit $rm
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f13, implicit $rm
-; 32BIT-DAG:     renamable $r4 = LWZ 4, %fixed-stack.0 :: (load (s32) from %ir.argp.cur1 + 4)
-; 32BIT-DAG:     STW renamable $r4, 4, %stack.2 :: (store (s32) into %stack.2 + 4)
+; 32BIT-DAG:     renamable $r3 = LWZ 4, %fixed-stack.0 :: (load (s32) from %ir.argp.cur142 + 4)
+; 32BIT-DAG:     STW renamable $r3, 4, %stack.2 :: (store (s32) into %stack.2 + 4)
 ; 32BIT-DAG:     renamable $f1 = LFD 0, %stack.2 :: (load (s64) from %stack.2)
-; 32BIT-DAG:     STW killed renamable $r3, 0, %stack.3 :: (store (s32) into %stack.3, align 8)
-; 32BIT-DAG:     STW killed renamable $r4, 4, %stack.3 :: (store (s32) into %stack.3 + 4)
+; 32BIT-DAG:     STW killed renamable $r4, 0, %stack.3 :: (store (s32) into %stack.3, align 8)
+; 32BIT-DAG:     STW killed renamable $r3, 4, %stack.3 :: (store (s32) into %stack.3 + 4)
 ; 32BIT-DAG:     renamable $f2 = LFD 0, %stack.3 :: (load (s64) from %stack.3)
 ; 32BIT-DAG:     renamable $f0 = nofpexcept FADD killed renamable $f0, killed renamable $f1, implicit $rm
-; 32BIT-DAG:     STW renamable $r3, 0, %stack.2 :: (store (s32) into %stack.2, align 8)
+; 32BIT-DAG:     STW renamable $r4, 0, %stack.2 :: (store (s32) into %stack.2, align 8)
 ; 32BIT-DAG:     renamable $f1 = nofpexcept FADD killed renamable $f2, renamable $f2, implicit $rm
 ; 32BIT-DAG:     BLR implicit $lr, implicit $rm, implicit $f1
-

llvm/test/CodeGen/X86/and-with-overflow.ll

; X64-NEXT:    retq
   %4 = icmp eq i8 %3, 0
   %5 = select i1 %4, i8 %0, i8 %3
   ret i8 %5
 }
 
 define i16 @and_i16_ri(i16 zeroext %0, i16 zeroext %1) {
 ; X86-LABEL: and_i16_ri:
 ; X86:       # %bb.0:
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %eax, %ecx
 ; X86-NEXT:    andl $-17, %ecx
-; X86-NEXT:    testw %cx, %cx
 ; X86-NEXT:    je .LBB2_2
 ; X86-NEXT:  # %bb.1:
 ; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:  .LBB2_2:
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: and_i16_ri:

llvm/test/CodeGen/X86/movmsk-cmp.ll

 }
 
 ; TODO: We expect similar result as for PR39665_c_ray_opt,
 ; but this is not the case in practice.
 define i32 @PR39665_c_ray(<2 x double> %x, <2 x double> %y) {
 ; SSE-LABEL: PR39665_c_ray:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    cmpltpd %xmm0, %xmm1
-; SSE-NEXT:    movmskpd %xmm1, %eax
-; SSE-NEXT:    cmpb $3, %al
-; SSE-NEXT:    movl $42, %ecx
-; SSE-NEXT:    movl $99, %eax
-; SSE-NEXT:    cmovel %ecx, %eax
+; SSE-NEXT:    movmskpd %xmm1, %ecx
+; SSE-NEXT:    testb $2, %cl
+; SSE-NEXT:    movl $42, %eax
+; SSE-NEXT:    movl $99, %edx
+; SSE-NEXT:    cmovel %edx, %eax
+; SSE-NEXT:    testb $1, %cl
+; SSE-NEXT:    cmovel %edx, %eax
 ; SSE-NEXT:    retq
 ;
 ; AVX1OR2-LABEL: PR39665_c_ray:
 ; AVX1OR2:       # %bb.0:
 ; AVX1OR2-NEXT:    vcmpltpd %xmm0, %xmm1, %xmm0
-; AVX1OR2-NEXT:    vmovmskpd %xmm0, %eax
-; AVX1OR2-NEXT:    cmpb $3, %al
-; AVX1OR2-NEXT:    movl $42, %ecx
-; AVX1OR2-NEXT:    movl $99, %eax
-; AVX1OR2-NEXT:    cmovel %ecx, %eax
+; AVX1OR2-NEXT:    vmovmskpd %xmm0, %ecx
+; AVX1OR2-NEXT:    testb $2, %cl
+; AVX1OR2-NEXT:    movl $42, %eax
+; AVX1OR2-NEXT:    movl $99, %edx
+; AVX1OR2-NEXT:    cmovel %edx, %eax
+; AVX1OR2-NEXT:    testb $1, %cl
+; AVX1OR2-NEXT:    cmovel %edx, %eax
 ; AVX1OR2-NEXT:    retq
 ;
 ; KNL-LABEL: PR39665_c_ray:
 ; KNL:       # %bb.0:
 ; KNL-NEXT:    # kill: def $xmm1 killed $xmm1 def $zmm1
 ; KNL-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
 ; KNL-NEXT:    vcmpltpd %zmm0, %zmm1, %k0
 ; KNL-NEXT:    kshiftrw $1, %k0, %k1

llvm/test/CodeGen/X86/or-with-overflow.ll

; X64-NEXT:    retq
   ret i8 %5
 }
 
 define i16 @or_i16_ri(i16 zeroext %0, i16 zeroext %1) {
 ; X86-LABEL: or_i16_ri:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    orl $-17, %ecx
+; X86-NEXT:    orl $65519, %ecx # imm = 0xFFEF
 ; X86-NEXT:    testw %cx, %cx
 ; X86-NEXT:    je .LBB2_2
 ; X86-NEXT:  # %bb.1:
 ; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:  .LBB2_2:
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: or_i16_ri:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %edi, %eax
-; X64-NEXT:    orl $-17, %eax
-; X64-NEXT:    testw %ax, %ax
+; X64-NEXT:    orl $65519, %eax # imm = 0xFFEF
 ; X64-NEXT:    cmovel %edi, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
   %3 = or i16 %0, -17
   %4 = icmp eq i16 %3, 0
   %5 = select i1 %4, i16 %0, i16 %3
   ret i16 %5
 }

llvm/test/CodeGen/X86/pr51175.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
 
 ; The and with -9 has multiple users which prevents
 ; SimplifyDemandedBits from touching it. The truncate only
 ; demands the lower 8 bits. X86ISelDAGToDAG.cpp wants form
 ; a test instruction from the icmp+trunc+and, but needs to
 ; mask the -9 to 8 bits since SimplifyDemandedBits didn't.
 
 define i32 @foo(i16 signext %0, i32 %1, i32* nocapture %2) {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    incl %edi
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    andl $65527, %eax # imm = 0xFFF7
-; CHECK-NEXT:    movl %eax, (%rdx)
+; CHECK-NEXT:    andl $65527, %edi # imm = 0xFFF7
+; CHECK-NEXT:    movl %edi, (%rdx)
 ; CHECK-NEXT:    xorl %eax, %eax
-; CHECK-NEXT:    testb $-9, %dil
+; CHECK-NEXT:    testb %dil, %dil
 ; CHECK-NEXT:    cmovel %esi, %eax
 ; CHECK-NEXT:    retq
   %4 = add i16 %0, 1
   %5 = and i16 %4, -9
   %6 = zext i16 %5 to i32
   store i32 %6, i32* %2, align 4
   %7 = trunc i16 %5 to i8
   %8 = icmp eq i8 %7, 0
   %9 = select i1 %8, i32 %1, i32 0
   ret i32 %9
 }

llvm/test/CodeGen/X86/vector-mulfix-legalize.ll

 ; CHECK-NEXT:    cmpl $16384, %ecx # imm = 0x4000
 ; CHECK-NEXT:    movl $32767, %r8d # imm = 0x7FFF
 ; CHECK-NEXT:    cmovgel %r8d, %edx
 ; CHECK-NEXT:    cmpl $-16384, %ecx # imm = 0xC000
 ; CHECK-NEXT:    movl $32768, %ecx # imm = 0x8000
 ; CHECK-NEXT:    cmovll %ecx, %edx
 ; CHECK-NEXT:    pextrw $1, %xmm0, %esi
 ; CHECK-NEXT:    movswl %si, %edi
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    shrl $16, %eax
-; CHECK-NEXT:    leal (%rdi,%rdi), %esi
-; CHECK-NEXT:    shrdw $15, %ax, %si
+; CHECK-NEXT:    leal (%rdi,%rdi), %eax
+; CHECK-NEXT:    movl %eax, %esi
+; CHECK-NEXT:    shrl $16, %esi
+; CHECK-NEXT:    shldw $1, %ax, %si
 ; CHECK-NEXT:    sarl $15, %edi
 ; CHECK-NEXT:    cmpl $16384, %edi # imm = 0x4000
 ; CHECK-NEXT:    cmovgel %r8d, %esi
 ; CHECK-NEXT:    cmpl $-16384, %edi # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %esi
 ; CHECK-NEXT:    movd %xmm0, %eax
 ; CHECK-NEXT:    cwtl
 ; CHECK-NEXT:    movl %eax, %edi
 ; CHECK-NEXT:    shrl $16, %edi
 ; CHECK-NEXT:    shldw $1, %ax, %di
 ; CHECK-NEXT:    sarl $16, %eax
 ; CHECK-NEXT:    cmpl $16384, %eax # imm = 0x4000
 ; CHECK-NEXT:    cmovgel %r8d, %edi
 ; CHECK-NEXT:    cmpl $-16384, %eax # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %edi
 ; CHECK-NEXT:    movzwl %di, %eax
 ; CHECK-NEXT:    movd %eax, %xmm1
 ; CHECK-NEXT:    pinsrw $1, %esi, %xmm1
 ; CHECK-NEXT:    pinsrw $2, %edx, %xmm1
 ; CHECK-NEXT:    pextrw $3, %xmm0, %eax
 ; CHECK-NEXT:    cwtl
-; CHECK-NEXT:    movl %eax, %edx
-; CHECK-NEXT:    shrl $14, %edx
-; CHECK-NEXT:    leal (,%rax,4), %esi
-; CHECK-NEXT:    shrdw $15, %dx, %si
+; CHECK-NEXT:    leal (,%rax,4), %edx
+; CHECK-NEXT:    movl %edx, %esi
+; CHECK-NEXT:    shrl $16, %esi
+; CHECK-NEXT:    shldw $1, %dx, %si
 ; CHECK-NEXT:    sarl $14, %eax
 ; CHECK-NEXT:    cmpl $16384, %eax # imm = 0x4000
 ; CHECK-NEXT:    cmovgel %r8d, %esi
 ; CHECK-NEXT:    cmpl $-16384, %eax # imm = 0xC000
 ; CHECK-NEXT:    cmovll %ecx, %esi
 ; CHECK-NEXT:    pinsrw $3, %esi, %xmm1
 ; CHECK-NEXT:    movdqa %xmm1, %xmm0
 ; CHECK-NEXT:    retq

llvm/test/CodeGen/X86/xor-with-overflow.ll

; X64-NEXT:    retq
   ret i8 %5
 }
 
 define i16 @xor_i16_ri(i16 zeroext %0, i16 zeroext %1) {
 ; X86-LABEL: xor_i16_ri:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    xorl $-17, %ecx
+; X86-NEXT:    xorl $65519, %ecx # imm = 0xFFEF
 ; X86-NEXT:    testw %cx, %cx
 ; X86-NEXT:    je .LBB2_2
 ; X86-NEXT:  # %bb.1:
 ; X86-NEXT:    movl %ecx, %eax
 ; X86-NEXT:  .LBB2_2:
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: xor_i16_ri:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %edi, %eax
-; X64-NEXT:    xorl $-17, %eax
-; X64-NEXT:    testw %ax, %ax
+; X64-NEXT:    xorl $65519, %eax # imm = 0xFFEF
 ; X64-NEXT:    cmovel %edi, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
   %3 = xor i16 %0, -17
   %4 = icmp eq i16 %3, 0
   %5 = select i1 %4, i16 %0, i16 %3
   ret i16 %5
 }