[SDAG] Allow scalable vectors in ComputeKnownBits

[reames]

Summary

This is the SelectionDAG equivalent of D136470, and is thus an alternate patch to D128159.

The basic idea here is that we track a single lane for scalable vectors which corresponds to an unknown number of lanes at runtime. This is enough for us to perform lane wise reasoning on many arithmetic operations.

This patch also includes an implementation for SPLAT_VECTOR as without it, the lane wise reasoning has no base case. The original patch which inspired this (D128159), also included STEP_VECTOR. I plan to do that as a separate patch.

[frasercrmck]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

2973

Is this possible? A BUILD_VECTOR with scalable-vector value type?

3676

Can EXTRACT_VECTOR_ELT produce a scalable vector? I'd have thought the existing check was sufficient.

[reames]

Address review comments

[reames]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

2973

I vaguely think I've seen this in debug output before, but converting it into an assert didn't catch any failures, and I don't remember the case. Let's make it an assert and see if anything falls out.

3676

Good catch

[reames]

ping

[paulwalker-arm]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

2999–3001

I don't believe this is possible because ISD::VECTOR_SHUFFLE only supports fixed length vectors. The zeroinitializer pattern you mention should be covered by ISD::SPLAT_VECTOR?

So I guess thus can be an assert like with the ISD::BUILD_VECTOR case. Although both those ISD nodes are baked by a SDNode classes that should have asserts within their constructors and so we should have failed long before we get to this code.

3869–3871

I agree. It seems like if removing this causes issues then that indicates a target specific bug. I did a quick parse of the AArch64TargetLowering version and nothing jumped out. In the past we did share AArch64ISD::DUP across both vector types but I broke this a while back and forced all scalable vectors to use ISD::SPLAT_VECTOR instead. That said, I think it's best to remove this as a separate patch so that it doesn't gate the rest of the code.

[reames]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

2999–3001

I'm happy to turn this into an assert if desired. I see value in making it locally obvious that a code path is dead. This was not obvious to me.

You mention an assert in the constructor. Glancing at ShuffleVectorSDNode constructor, I do not see such a thing. Am I missing something obvious?

[paulwalker-arm]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

2999–3001

Now you mention it, I'm looking at our downstream version that contains some asserts we really should get upstream :)

[reames]

Convert unreachable case to assert per reviewer comment.

[reames]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

2999–3001

Lol, yes, please! :)

[MaskRay]

Early heads-up: this change causes a timeout (likely an infinite loop) for a Halide Hexagon test, but no reduced example is available yet.

[reames]

In D137140#3938320, @MaskRay wrote:

Early heads-up: this change causes a timeout (likely an infinite loop) for a Halide Hexagon test, but no reduced example is available yet.

Given it's a Friday going into a holiday week, I reverted for now. Please share your findings when you have them.

[reames]

In D137140#3938320, @MaskRay wrote:

Early heads-up: this change causes a timeout (likely an infinite loop) for a Halide Hexagon test, but no reduced example is available yet.

@MaskRay Any luck on confirming or reducing this?

[reames]

@MaskRay ping?

[MaskRay]

In D137140#3964366, @reames wrote:

@MaskRay ping?

Sorry for my belated response. I am travelling and don't read email much.
I am still waiting for internal response but forwarded the ping and CCed more folks who may help get a reproduce......

[rupprecht]

; ModuleID = 'repro.ll'
source_filename = "repro.ll"
target datalayout = "e-m:e-p:32:32:32-a:0-n16:32-i64:64:64-i32:32:32-i16:16:16-i1:8:8-f32:32:32-f64:64:64-v32:32:32-v64:64:64-v512:512:512-v1024:1024:1024-v2048:2048:2048"
target triple = "hexagon-unknown--elf"

; Function Attrs: nocallback nofree nosync nounwind willreturn memory(none)
declare <32 x i32> @llvm.hexagon.V6.lo.128B(<64 x i32>) #0

define i32 @widget(<128 x i16> %arg, ptr %arg1) #1 {
bb:
  br label %bb2

bb2:                                              ; preds = %bb2, %bb
  %tmp = add <128 x i16> %arg, <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>
  %tmp3 = bitcast <128 x i16> %tmp to <64 x i32>
  %tmp4 = tail call <32 x i32> @llvm.hexagon.V6.lo.128B(<64 x i32> %tmp3)
  store <32 x i32> %tmp4, ptr %arg1, align 1024
  br label %bb2
}

attributes #0 = { nocallback nofree nosync nounwind willreturn memory(none) }
attributes #1 = { "target-features"="+hvx-length128b,+long-calls,+hvxv62" }

Repro steps: llc repro.ll -o /dev/null. That command is ~instant near trunk (122.6 ms ± 5.9 ms). It seems to be going forever with this patch.

Here's a couple stack traces where it's stuck:

(lldb) bt
* thread #1, name = 'llc', stop reason = signal SIGSTOP
  * frame #0: 0x00005555590bd7b4 llc`AddNodeIDNode(llvm::FoldingSetNodeID&, unsigned int, llvm::SDVTList, llvm::ArrayRef<llvm::SDValue>) [inlined] llvm::SmallVectorTemplateBase<unsigned int, true>::push_back(this=0x00007fffffff8970, Elt=1679) at SmallVector.h:568:33
    frame #1: 0x00005555590bd7a7 llc`AddNodeIDNode(llvm::FoldingSetNodeID&, unsigned int, llvm::SDVTList, llvm::ArrayRef<llvm::SDValue>) [inlined] llvm::FoldingSetNodeID::AddInteger(this=0x00007fffffff8970, I=1679) at FoldingSet.h:340:38
    frame #2: 0x00005555590bd7a7 llc`AddNodeIDNode(llvm::FoldingSetNodeID&, unsigned int, llvm::SDVTList, llvm::ArrayRef<llvm::SDValue>) [inlined] llvm::FoldingSetNodeID::AddInteger(this=0x00007fffffff8970, I=<unavailable>) at FoldingSet.h:354:5
    frame #3: 0x00005555590bd77c llc`AddNodeIDNode(llvm::FoldingSetNodeID&, unsigned int, llvm::SDVTList, llvm::ArrayRef<llvm::SDValue>) [inlined] llvm::FoldingSetNodeID::AddInteger(this=0x00007fffffff8970, I=<unavailable>) at FoldingSet.h:346:7
    frame #4: 0x00005555590bd77c llc`AddNodeIDNode(llvm::FoldingSetNodeID&, unsigned int, llvm::SDVTList, llvm::ArrayRef<llvm::SDValue>) [inlined] llvm::FoldingSetNodeID::AddPointer(this=0x00007fffffff8970, Ptr=<unavailable>) at FoldingSet.h:337:5
    frame #5: 0x00005555590bd77c llc`AddNodeIDNode(llvm::FoldingSetNodeID&, unsigned int, llvm::SDVTList, llvm::ArrayRef<llvm::SDValue>) [inlined] AddNodeIDValueTypes(ID=0x00007fffffff8970, VTList=<unavailable>) at SelectionDAG.cpp:643:6
    frame #6: 0x00005555590bd77c llc`AddNodeIDNode(ID=0x00007fffffff8970, OpC=11, VTList=<unavailable>, OpList=ArrayRef<llvm::SDValue> @ 0x0000246d579f2b40) at SelectionDAG.cpp:667:3
    frame #7: 0x00005555590c140e llc`llvm::SelectionDAG::getConstant(this=0x0000068f3fc1ae00, Val=<unavailable>, DL=0x00007fffffffa5e0, VT=EVT @ 0x00007fffffff8a90, isT=<unavailable>, isO=<unavailable>) at SelectionDAG.cpp:1612:3
    frame #8: 0x00005555590c0716 llc`llvm::SelectionDAG::getConstant(unsigned long, llvm::SDLoc const&, llvm::EVT, bool, bool) [inlined] llvm::SelectionDAG::getConstant(this=0x0000068f3fc1ae00, Val=0x00007fffffff8b18, DL=0x00007fffffffa5e0, VT=EVT @ 0x0000246d4b6ffc50, isT=false, isO=false) at SelectionDAG.cpp:1525:10
    frame #9: 0x00005555590c06e5 llc`llvm::SelectionDAG::getConstant(this=0x0000068f3fc1ae00, Val=<unavailable>, DL=0x00007fffffffa5e0, VT=EVT @ 0x00007fffffff8b28, isT=false, isO=false) at SelectionDAG.cpp:1520:10
    frame #10: 0x00005555584d1172 llc`llvm::HexagonTargetLowering::buildVector32(this=0x0000068f3fd442e0, Elem=ArrayRef<llvm::SDValue> @ 0x0000246d579f2b40, dl=<unavailable>, VecTy=(SimpleTy = v2i16), DAG=0x0000068f3fc1ae00) const at HexagonISelLowering.cpp:2513:40
    frame #11: 0x00005555584dac39 llc`llvm::HexagonTargetLowering::buildHvxVectorReg(this=0x0000068f3fd442e0, Values=ArrayRef<llvm::SDValue> @ 0x0000246d4b6ffc50, dl=0x00007fffffffa5e0, VecTy=(SimpleTy = v64i16), DAG=0x0000068f3fc1ae00) const at HexagonISelLoweringHVX.cpp:799:19
    frame #12: 0x00005555584dfff2 llc`llvm::HexagonTargetLowering::LowerHvxBuildVector(this=0x0000068f3fd442e0, Op=<unavailable>, DAG=0x0000068f3fc1ae00) const at HexagonISelLoweringHVX.cpp:1621:18
    frame #13: 0x00005555584e8dc5 llc`llvm::HexagonTargetLowering::LowerHvxOperation(this=0x0000068f3fd442e0, Op=SDValue @ 0x0000246d55f19080, DAG=0x0000068f3fc1ae00) const at HexagonISelLoweringHVX.cpp:3202:47
    frame #14: 0x00005555584d5be0 llc`llvm::HexagonTargetLowering::LowerOperation(this=0x0000068f3fd442e0, Op=SDValue @ 0x0000246d5d77ac20, DAG=0x0000068f3fc1ae00) const at HexagonISelLowering.cpp:3316:21
    frame #15: 0x0000555559085280 llc`(anonymous namespace)::SelectionDAGLegalize::LegalizeOp(this=0x00007fffffffaa98, Node=0x0000068f3fd9f690) at LegalizeDAG.cpp:1294:29
    frame #16: 0x000055555908869a llc`llvm::SelectionDAG::LegalizeOp(this=<unavailable>, N=<unavailable>, UpdatedNodes=<unavailable>) at LegalizeDAG.cpp:5125:13
    frame #17: 0x0000555558feeeeb llc`llvm::SelectionDAG::Combine(llvm::CombineLevel, llvm::AAResults*, llvm::CodeGenOpt::Level) at DAGCombiner.cpp:1608:27
    frame #18: 0x0000555558feeac6 llc`llvm::SelectionDAG::Combine(this=<unavailable>, Level=<unavailable>, AA=<unavailable>, OptLevel=<unavailable>) at DAGCombiner.cpp:25513:36
    frame #19: 0x000055555914cb17 llc`llvm::SelectionDAGISel::CodeGenAndEmitDAG(this=0x0000068f3fc4e600) at SelectionDAGISel.cpp:913:13
    frame #20: 0x000055555914c67f llc`llvm::SelectionDAGISel::SelectBasicBlock(this=<unavailable>, Begin=<unavailable>, End=<unavailable>, HadTailCall=<unavailable>) at SelectionDAGISel.cpp:689:3 [artificial]
    frame #21: 0x000055555914bfd9 llc`llvm::SelectionDAGISel::SelectAllBasicBlocks(this=0x0000068f3fc4e600, Fn=0x0000068f3fc92088) at SelectionDAGISel.cpp:1637:7
    frame #22: 0x0000555559149941 llc`llvm::SelectionDAGISel::runOnMachineFunction(this=0x0000068f3fc4e600, mf=0x0000068f3fc7bb00) at SelectionDAGISel.cpp:469:3
    frame #23: 0x00005555585a1b67 llc`llvm::HexagonDAGToDAGISel::runOnMachineFunction(this=0x0000068f3fc4e600, MF=0x0000068f3fc7bb00) at HexagonISelDAGToDAG.h:44:23
    frame #24: 0x0000555558e57ec9 llc`llvm::MachineFunctionPass::runOnFunction(this=0x0000068f3fc4e600, F=0x0000068f3fc92088) at MachineFunctionPass.cpp:91:13
    frame #25: 0x0000555559b656b5 llc`llvm::FPPassManager::runOnFunction(this=0x0000068f3fc90000, F=0x0000068f3fc92088) at LegacyPassManager.cpp:1430:27
    frame #26: 0x0000555559b6c4a4 llc`llvm::FPPassManager::runOnModule(this=0x0000068f3fc90000, M=<unavailable>) at LegacyPassManager.cpp:1476:16
    frame #27: 0x0000555559b65d75 llc`llvm::legacy::PassManagerImpl::run(llvm::Module&) at LegacyPassManager.cpp:1545:27
    frame #28: 0x0000555559b65a31 llc`llvm::legacy::PassManagerImpl::run(this=0x0000068f3fc7a900, M=0x0000068f3fc4e300) at LegacyPassManager.cpp:535:44
    frame #29: 0x0000555557cc937c llc`main at llc.cpp:733:8
    frame #30: 0x0000555557cc887c llc`main(argc=<unavailable>, argv=<unavailable>) at llc.cpp:417:22
    frame #31: 0x00007ffff7d0d633 libc.so.6`__libc_start_main + 243
    frame #32: 0x0000555557cc442a llc`_start at start.S:120
(lldb) c
Process 2800685 resuming
Process 2800685 stopped
* thread #1, name = 'llc', stop reason = signal SIGSTOP
    frame #0: 0x00005555590cbf6f llc`llvm::SelectionDAG::computeKnownBits(this=0x0000068f3fc1ae00, Op=SDValue @ 0x00007fffffff8a90, DemandedElts=0x00007fffffff9300, Depth=2) const at SelectionDAG.cpp:3154:37
(lldb) bt
* thread #1, name = 'llc', stop reason = signal SIGSTOP
  * frame #0: 0x00005555590cbf6f llc`llvm::SelectionDAG::computeKnownBits(this=0x0000068f3fc1ae00, Op=SDValue @ 0x00007fffffff8a90, DemandedElts=0x00007fffffff9300, Depth=2) const at SelectionDAG.cpp:3154:37
    frame #1: 0x00005555591e2ebd llc`llvm::TargetLowering::SimplifyDemandedBits(this=0x0000068f3fd442e0, Op=SDValue @ 0x00007fffffff9388, OriginalDemandedBits=0x00007fffffff9b38, OriginalDemandedElts=0x00007fffffff9a60, Known=0x00007fffffff99d0, TLO=0x00007fffffffa3f0, Depth=2, AssumeSingleUse=<unavailable>) const at TargetLowering.cpp:0
    frame #2: 0x00005555591dc4c1 llc`llvm::TargetLowering::SimplifyDemandedBits(this=0x0000068f3fd442e0, Op=SDValue @ 0x00007fffffff9b48, OriginalDemandedBits=0x00007fffffffa2a0, OriginalDemandedElts=0x00007fffffffa280, Known=0x00007fffffffa190, TLO=0x00007fffffffa3f0, Depth=1, AssumeSingleUse=<unavailable>) const at TargetLowering.cpp:1282:11
    frame #3: 0x00005555591de0ad llc`llvm::TargetLowering::SimplifyDemandedBits(this=0x0000068f3fd442e0, Op=SDValue @ 0x00007fffffffa308, OriginalDemandedBits=0x00007fffffffa480, OriginalDemandedElts=0x00007fffffffa388, Known=0x00007fffffffa420, TLO=0x00007fffffffa3f0, Depth=0, AssumeSingleUse=<unavailable>) const at TargetLowering.cpp:2585:9
    frame #4: 0x00005555591db272 llc`llvm::TargetLowering::SimplifyDemandedBits(this=0x0000068f3fd442e0, Op=SDValue @ 0x0000246d57d150e0, DemandedBits=0x00007fffffffa480, Known=0x00007fffffffa420, TLO=0x00007fffffffa3f0, Depth=0, AssumeSingleUse=<unavailable>) const at TargetLowering.cpp:649:10
    frame #5: 0x000055555903ce7c llc`(anonymous namespace)::DAGCombiner::SimplifyDemandedBits(this=0x00007fffffffaaf8, Op=<unavailable>, DemandedBits=<unavailable>) at DAGCombiner.cpp:333:16
    frame #6: 0x000055555903b866 llc`(anonymous namespace)::DAGCombiner::SimplifyDemandedBits(this=0x00007fffffffaaf8, Op=SDValue @ 0x0000246d71aca5c0) at DAGCombiner.cpp:327:14
    frame #7: 0x0000555559038bb4 llc`(anonymous namespace)::DAGCombiner::visitADDLike(this=0x00007fffffffaaf8, N=0x0000068f3fd9f1c0) at DAGCombiner.cpp:2590:7
    frame #8: 0x0000555558ff37eb llc`(anonymous namespace)::DAGCombiner::visitADD(this=0x00007fffffffaaf8, N=0x0000068f3fd9f1c0) at DAGCombiner.cpp:2649:26
    frame #9: 0x0000555558ff1f40 llc`(anonymous namespace)::DAGCombiner::combine(llvm::SDNode*) [inlined] (anonymous namespace)::DAGCombiner::visit(this=0x00007fffffffaaf8, N=0x0000068f3fd9f1c0) at DAGCombiner.cpp:1682:40
    frame #10: 0x0000555558ff1eda llc`(anonymous namespace)::DAGCombiner::combine(this=0x00007fffffffaaf8, N=0x0000068f3fd9f1c0) at DAGCombiner.cpp:1829:10
    frame #11: 0x0000555558fef0f2 llc`llvm::SelectionDAG::Combine(llvm::CombineLevel, llvm::AAResults*, llvm::CodeGenOpt::Level) at DAGCombiner.cpp:1627:18
    frame #12: 0x0000555558feeac6 llc`llvm::SelectionDAG::Combine(this=<unavailable>, Level=<unavailable>, AA=<unavailable>, OptLevel=<unavailable>) at DAGCombiner.cpp:25513:36
    frame #13: 0x000055555914cb17 llc`llvm::SelectionDAGISel::CodeGenAndEmitDAG(this=0x0000068f3fc4e600) at SelectionDAGISel.cpp:913:13
    frame #14: 0x000055555914c67f llc`llvm::SelectionDAGISel::SelectBasicBlock(this=<unavailable>, Begin=<unavailable>, End=<unavailable>, HadTailCall=<unavailable>) at SelectionDAGISel.cpp:689:3 [artificial]
    frame #15: 0x000055555914bfd9 llc`llvm::SelectionDAGISel::SelectAllBasicBlocks(this=0x0000068f3fc4e600, Fn=0x0000068f3fc92088) at SelectionDAGISel.cpp:1637:7
    frame #16: 0x0000555559149941 llc`llvm::SelectionDAGISel::runOnMachineFunction(this=0x0000068f3fc4e600, mf=0x0000068f3fc7bb00) at SelectionDAGISel.cpp:469:3
    frame #17: 0x00005555585a1b67 llc`llvm::HexagonDAGToDAGISel::runOnMachineFunction(this=0x0000068f3fc4e600, MF=0x0000068f3fc7bb00) at HexagonISelDAGToDAG.h:44:23
    frame #18: 0x0000555558e57ec9 llc`llvm::MachineFunctionPass::runOnFunction(this=0x0000068f3fc4e600, F=0x0000068f3fc92088) at MachineFunctionPass.cpp:91:13
    frame #19: 0x0000555559b656b5 llc`llvm::FPPassManager::runOnFunction(this=0x0000068f3fc90000, F=0x0000068f3fc92088) at LegacyPassManager.cpp:1430:27
    frame #20: 0x0000555559b6c4a4 llc`llvm::FPPassManager::runOnModule(this=0x0000068f3fc90000, M=<unavailable>) at LegacyPassManager.cpp:1476:16
    frame #21: 0x0000555559b65d75 llc`llvm::legacy::PassManagerImpl::run(llvm::Module&) at LegacyPassManager.cpp:1545:27
    frame #22: 0x0000555559b65a31 llc`llvm::legacy::PassManagerImpl::run(this=0x0000068f3fc7a900, M=0x0000068f3fc4e300) at LegacyPassManager.cpp:535:44
    frame #23: 0x0000555557cc937c llc`main at llc.cpp:733:8
    frame #24: 0x0000555557cc887c llc`main(argc=<unavailable>, argv=<unavailable>) at llc.cpp:417:22
    frame #25: 0x00007ffff7d0d633 libc.so.6`__libc_start_main + 243
    frame #26: 0x0000555557cc442a llc`_start at start.S:120

[reames]

@rupprecht, thanks! Will take a look today.

Your example is quite interesting as it doesn't even use scalable vectors. So, something definitely odd happened here.

[reames]

Ok, I think this is a Hexagon target bug. What's going on here is that Hexagon appears to be using SPLAT_VECTOR for fixed length vector types, and are legalizing build_vector splats of narrow types to SPLAT_VECTOR of i32.

The problem is that this doesn't work. Generic DAG combine canonicalizes the other way given known bits information. Before this change, that wasn't *visible* and didn't kick in because generic DAG couldn't see the SPLAT_VALUE known bits, but that's it.

I'm going to remove the implicit truncation support in SPLAT_VECTOR from this patch (that hides the above bug), and then check this in again. Someone on the Hexagon side should definite figure out how to fix this properly and add back the implicit truncation.

[hokein]

We also see a timeout for a Halide test after the re-landed revision, no reduced testcase yet (trying to get one).

[dmgreen]

Can you check if it helps to add an implementation of isTargetCanonicalConstantNode for hexagon? It looks like CONCAT is needed, and maybe some other nodes like bitcasts.

bool HexagonTargetLowering::isTargetCanonicalConstantNode(SDValue Op) const {
  return Op.getOpcode() == ISD::CONCAT_VECTORS ||
         TargetLowering::isTargetCanonicalConstantNode(Op);
}

[hokein]

Here is a reduced testcase:

target datalayout = "e-m:e-p:32:32:32-a:0-n16:32-i64:64:64-i32:32:32-i16:16:16-i1:8:8-f32:32:32-f64:64:64-v32:32:32-v64:64:64-v512:512:512-v1024:1024:1024-v2048:2048:2048"
target triple = "hexagon-unknown--elf"

; Function Attrs: nocallback nofree nosync nounwind willreturn memory(none)
declare <32 x i32> @llvm.hexagon.V6.lo.128B(<64 x i32>) #0

define i32 @offload_rpc_output_s0___outermost_par_for_output_s0_y_line_chunk_chunk(<64 x i32> %0, ptr %linearized) #1 {
entry:
  br label %"for linearized.s0.x.x"

"for linearized.s0.x.x":                          ; preds = %"for linearized.s0.x.x", %entry
  %1 = add <64 x i32> %0, <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
  %2 = call <32 x i32> @llvm.hexagon.V6.lo.128B(<64 x i32> %1)
  store <32 x i32> %2, ptr %linearized, align 128
  br label %"for linearized.s0.x.x"
}

attributes #0 = { nocallback nofree nosync nounwind willreturn memory(none) }
attributes #1 = { "target-features"="+hvx-length128b,+long-calls,+hvxv62" }

In D137140#3977513, @dmgreen wrote:

Can you check if it helps to add an implementation of isTargetCanonicalConstantNode for hexagon? It looks like CONCAT is needed, and maybe some other nodes like bitcasts.

bool HexagonTargetLowering::isTargetCanonicalConstantNode(SDValue Op) const {
  return Op.getOpcode() == ISD::CONCAT_VECTORS ||
         TargetLowering::isTargetCanonicalConstantNode(Op);
}

Thanks, it seems to work (for the reduced testcase).

[hokein]

I have sent out https://reviews.llvm.org/D139525 (verified with the reduced testcase).

[reames]

In D137140#3977775, @hokein wrote:

I have sent out https://reviews.llvm.org/D139525 (verified with the reduced testcase).

Sorry for the breakage, and thank you for fixing the target. If you need help isolating further problems, please let me know.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

 }
 
 /// Determine which bits of Op are known to be either zero or one and return
 /// them in Known. For vectors, the known bits are those that are shared by
 /// every vector element.
 KnownBits SelectionDAG::computeKnownBits(SDValue Op, unsigned Depth) const {
   EVT VT = Op.getValueType();
 
-  // TOOD: Until we have a plan for how to represent demanded elements for
-  // scalable vectors, we can just bail out for now.
-  if (Op.getValueType().isScalableVector()) {
-    unsigned BitWidth = Op.getScalarValueSizeInBits();
-    return KnownBits(BitWidth);
-  }
-
-  APInt DemandedElts = VT.isVector()
+  // Since the number of lanes in a scalable vector is unknown at compile time,
+  // we track one bit which is implicitly broadcast to all lanes.  This means
+  // that all lanes in a scalable vector are considered demanded.
+  APInt DemandedElts = VT.isFixedLengthVector()
                            ? APInt::getAllOnes(VT.getVectorNumElements())
                            : APInt(1, 1);
   return computeKnownBits(Op, DemandedElts, Depth);
 }
 
 /// Determine which bits of Op are known to be either zero or one and return
 /// them in Known. The DemandedElts argument allows us to only collect the known
 /// bits that are shared by the requested vector elements.
 KnownBits SelectionDAG::computeKnownBits(SDValue Op, const APInt &DemandedElts,
                                          unsigned Depth) const {
   unsigned BitWidth = Op.getScalarValueSizeInBits();
 
   KnownBits Known(BitWidth);   // Don't know anything.
 
-  // TOOD: Until we have a plan for how to represent demanded elements for
-  // scalable vectors, we can just bail out for now.
-  if (Op.getValueType().isScalableVector())
-    return Known;
-
   if (auto *C = dyn_cast<ConstantSDNode>(Op)) {
     // We know all of the bits for a constant!
     return KnownBits::makeConstant(C->getAPIntValue());
   }
   if (auto *C = dyn_cast<ConstantFPSDNode>(Op)) {
     // We know all of the bits for a constant fp!
     return KnownBits::makeConstant(C->getValueAPF().bitcastToAPInt());
   }
 
   if (Depth >= MaxRecursionDepth)
     return Known;  // Limit search depth.
 
   KnownBits Known2;
   unsigned NumElts = DemandedElts.getBitWidth();
-  assert((!Op.getValueType().isVector() ||
+  assert((!Op.getValueType().isFixedLengthVector() ||
           NumElts == Op.getValueType().getVectorNumElements()) &&
          "Unexpected vector size");
 
   if (!DemandedElts)
     return Known;  // No demanded elts, better to assume we don't know anything.
 
   unsigned Opcode = Op.getOpcode();
   switch (Opcode) {
   case ISD::MERGE_VALUES:
     return computeKnownBits(Op.getOperand(Op.getResNo()), DemandedElts,
                             Depth + 1);
+  case ISD::SPLAT_VECTOR: {
+    SDValue SrcOp = Op.getOperand(0);
+    if (SrcOp.getValueSizeInBits() != BitWidth) {
+      assert(SrcOp.getValueSizeInBits() > BitWidth &&
+             "Expected SPLAT_VECTOR implicit truncation");
+      // FIXME: We should be able to truncate the known bits here to match
+      // the official semantics of SPLAT_VECTOR, but doing so exposes a
+      // Hexagon target bug which results in an infinite loop during
+      // DAGCombine.  (See D137140 for repo).  Once that's fixed, we can
+      // strengthen this.
+      break;
+    }
+    Known = computeKnownBits(SrcOp, Depth + 1);
+    break;
+  }
   case ISD::BUILD_VECTOR:
+    assert(!Op.getValueType().isScalableVector());
     // Collect the known bits that are shared by every demanded vector element.
     Known.Zero.setAllBits(); Known.One.setAllBits();
     for (unsigned i = 0, e = Op.getNumOperands(); i != e; ++i) {
       if (!DemandedElts[i])
         continue;
 
       SDValue SrcOp = Op.getOperand(i);
       Known2 = computeKnownBits(SrcOp, Depth + 1);
for (unsigned i = 0, e = Op.getNumOperands(); i != e; ++i) {
       Known = KnownBits::commonBits(Known, Known2);
 
       // If we don't know any bits, early out.
       if (Known.isUnknown())
         break;
     }
     break;
   case ISD::VECTOR_SHUFFLE: {
+    assert(!Op.getValueType().isScalableVector());
     // Collect the known bits that are shared by every vector element referenced
     // by the shuffle.
     APInt DemandedLHS, DemandedRHS;
     const ShuffleVectorSDNode *SVN = cast<ShuffleVectorSDNode>(Op);
     assert(NumElts == SVN->getMask().size() && "Unexpected vector size");
     if (!getShuffleDemandedElts(NumElts, SVN->getMask(), DemandedElts,
                                 DemandedLHS, DemandedRHS))
       break;
 
     // Known bits are the values that are shared by every demanded element.
case ISD::VECTOR_SHUFFLE: {
     if (!!DemandedRHS) {
       SDValue RHS = Op.getOperand(1);
       Known2 = computeKnownBits(RHS, DemandedRHS, Depth + 1);
       Known = KnownBits::commonBits(Known, Known2);
     }
     break;
   }
   case ISD::CONCAT_VECTORS: {
+    if (Op.getValueType().isScalableVector())
+      break;
     // Split DemandedElts and test each of the demanded subvectors.
     Known.Zero.setAllBits(); Known.One.setAllBits();
     EVT SubVectorVT = Op.getOperand(0).getValueType();
     unsigned NumSubVectorElts = SubVectorVT.getVectorNumElements();
     unsigned NumSubVectors = Op.getNumOperands();
     for (unsigned i = 0; i != NumSubVectors; ++i) {
       APInt DemandedSub =
           DemandedElts.extractBits(NumSubVectorElts, i * NumSubVectorElts);
       if (!!DemandedSub) {
         SDValue Sub = Op.getOperand(i);
         Known2 = computeKnownBits(Sub, DemandedSub, Depth + 1);
         Known = KnownBits::commonBits(Known, Known2);
       }
       // If we don't know any bits, early out.
       if (Known.isUnknown())
         break;
     }
     break;
   }
   case ISD::INSERT_SUBVECTOR: {
+    if (Op.getValueType().isScalableVector())
+      break;
     // Demand any elements from the subvector and the remainder from the src its
     // inserted into.
     SDValue Src = Op.getOperand(0);
     SDValue Sub = Op.getOperand(1);
     uint64_t Idx = Op.getConstantOperandVal(2);
     unsigned NumSubElts = Sub.getValueType().getVectorNumElements();
     APInt DemandedSubElts = DemandedElts.extractBits(NumSubElts, Idx);
     APInt DemandedSrcElts = DemandedElts;
if (!!DemandedSrcElts) {
       Known = KnownBits::commonBits(Known, Known2);
     }
     break;
   }
   case ISD::EXTRACT_SUBVECTOR: {
     // Offset the demanded elts by the subvector index.
     SDValue Src = Op.getOperand(0);
     // Bail until we can represent demanded elements for scalable vectors.
-    if (Src.getValueType().isScalableVector())
+    if (Op.getValueType().isScalableVector() || Src.getValueType().isScalableVector())
       break;
     uint64_t Idx = Op.getConstantOperandVal(1);
     unsigned NumSrcElts = Src.getValueType().getVectorNumElements();
     APInt DemandedSrcElts = DemandedElts.zext(NumSrcElts).shl(Idx);
     Known = computeKnownBits(Src, DemandedSrcElts, Depth + 1);
     break;
   }
   case ISD::SCALAR_TO_VECTOR: {
+    if (Op.getValueType().isScalableVector())
+      break;
     // We know about scalar_to_vector as much as we know about it source,
     // which becomes the first element of otherwise unknown vector.
     if (DemandedElts != 1)
       break;
 
     SDValue N0 = Op.getOperand(0);
     Known = computeKnownBits(N0, Depth + 1);
     if (N0.getValueSizeInBits() != BitWidth)
       Known = Known.trunc(BitWidth);
 
     break;
   }
   case ISD::BITCAST: {
+    if (Op.getValueType().isScalableVector())
+      break;
+
     SDValue N0 = Op.getOperand(0);
     EVT SubVT = N0.getValueType();
     unsigned SubBitWidth = SubVT.getScalarSizeInBits();
 
     // Ignore bitcasts from unsupported types.
     if (!(SubVT.isInteger() || SubVT.isFloatingPoint()))
       break;
 
case ISD::PARITY: {
     break;
   }
   case ISD::LOAD: {
     LoadSDNode *LD = cast<LoadSDNode>(Op);
     const Constant *Cst = TLI->getTargetConstantFromLoad(LD);
     if (ISD::isNON_EXTLoad(LD) && Cst) {
       // Determine any common known bits from the loaded constant pool value.
       Type *CstTy = Cst->getType();
-      if ((NumElts * BitWidth) == CstTy->getPrimitiveSizeInBits()) {
+      if ((NumElts * BitWidth) == CstTy->getPrimitiveSizeInBits() &&
+          !Op.getValueType().isScalableVector()) {
         // If its a vector splat, then we can (quickly) reuse the scalar path.
         // NOTE: We assume all elements match and none are UNDEF.
         if (CstTy->isVectorTy()) {
           if (const Constant *Splat = Cst->getSplatValue()) {
             Cst = Splat;
             CstTy = Cst->getType();
           }
         }
if (ISD::isNON_EXTLoad(LD) && Cst) {
           Known = KnownFull.trunc(BitWidth);
         } else
           computeKnownBitsFromRangeMetadata(*Ranges, Known);
       }
     }
     break;
   }
   case ISD::ZERO_EXTEND_VECTOR_INREG: {
+    if (Op.getValueType().isScalableVector())
+      break;
     EVT InVT = Op.getOperand(0).getValueType();
     APInt InDemandedElts = DemandedElts.zext(InVT.getVectorNumElements());
     Known = computeKnownBits(Op.getOperand(0), InDemandedElts, Depth + 1);
     Known = Known.zext(BitWidth);
     break;
   }
   case ISD::ZERO_EXTEND: {
     Known = computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
     Known = Known.zext(BitWidth);
     break;
   }
   case ISD::SIGN_EXTEND_VECTOR_INREG: {
+    if (Op.getValueType().isScalableVector())
+      break;
     EVT InVT = Op.getOperand(0).getValueType();
     APInt InDemandedElts = DemandedElts.zext(InVT.getVectorNumElements());
     Known = computeKnownBits(Op.getOperand(0), InDemandedElts, Depth + 1);
     // If the sign bit is known to be zero or one, then sext will extend
     // it to the top bits, else it will just zext.
     Known = Known.sext(BitWidth);
     break;
   }
   case ISD::SIGN_EXTEND: {
     Known = computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
     // If the sign bit is known to be zero or one, then sext will extend
     // it to the top bits, else it will just zext.
     Known = Known.sext(BitWidth);
     break;
   }
   case ISD::ANY_EXTEND_VECTOR_INREG: {
+    if (Op.getValueType().isScalableVector())
+      break;
     EVT InVT = Op.getOperand(0).getValueType();
     APInt InDemandedElts = DemandedElts.zext(InVT.getVectorNumElements());
     Known = computeKnownBits(Op.getOperand(0), InDemandedElts, Depth + 1);
     Known = Known.anyext(BitWidth);
     break;
   }
   case ISD::ANY_EXTEND: {
     Known = computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
case ISD::EXTRACT_ELEMENT: {
     Known = Known.trunc(EltBitWidth);
     break;
   }
   case ISD::EXTRACT_VECTOR_ELT: {
     SDValue InVec = Op.getOperand(0);
     SDValue EltNo = Op.getOperand(1);
     EVT VecVT = InVec.getValueType();
     // computeKnownBits not yet implemented for scalable vectors.
     if (VecVT.isScalableVector())
       break;
     const unsigned EltBitWidth = VecVT.getScalarSizeInBits();
     const unsigned NumSrcElts = VecVT.getVectorNumElements();
 
     // If BitWidth > EltBitWidth the value is anyext:ed. So we do not know
     // anything about the extended bits.
     if (BitWidth > EltBitWidth)
       Known = Known.trunc(EltBitWidth);
 
     // If we know the element index, just demand that vector element, else for
     // an unknown element index, ignore DemandedElts and demand them all.
     APInt DemandedSrcElts = APInt::getAllOnes(NumSrcElts);
     auto *ConstEltNo = dyn_cast<ConstantSDNode>(EltNo);
     if (ConstEltNo && ConstEltNo->getAPIntValue().ult(NumSrcElts))
       DemandedSrcElts =
           APInt::getOneBitSet(NumSrcElts, ConstEltNo->getZExtValue());
 
     Known = computeKnownBits(InVec, DemandedSrcElts, Depth + 1);
     if (BitWidth > EltBitWidth)
       Known = Known.anyext(BitWidth);
     break;
   }
   case ISD::INSERT_VECTOR_ELT: {
+    if (Op.getValueType().isScalableVector())
+      break;
+
     // If we know the element index, split the demand between the
     // source vector and the inserted element, otherwise assume we need
     // the original demanded vector elements and the value.
     SDValue InVec = Op.getOperand(0);
     SDValue InVal = Op.getOperand(1);
     SDValue EltNo = Op.getOperand(2);
     bool DemandedVal = true;
     APInt DemandedVecElts = DemandedElts;
KnownBits SelectionDAG::computeKnownBits(SDValue Op, const APInt &DemandedElts,
 
   default:
     if (Opcode < ISD::BUILTIN_OP_END)
       break;
     [[fallthrough]];
   case ISD::INTRINSIC_WO_CHAIN:
   case ISD::INTRINSIC_W_CHAIN:
   case ISD::INTRINSIC_VOID:
+    // TODO: Probably okay to remove after audit; here to reduce change size
+    // in initial enablement patch for scalable vectors
+    if (Op.getValueType().isScalableVector())
+      break;
+
     // Allow the target to implement this method for its nodes.
     TLI->computeKnownBitsForTargetNode(Op, Known, DemandedElts, *this, Depth);
     break;
   }
 
   assert(!Known.hasConflict() && "Bits known to be one AND zero?");
   return Known;
 }

llvm/test/CodeGen/AArch64/sve-intrinsics-index.ll

 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i64> @llvm.aarch64.sve.index.nxv2i64(i64 -17, i64 16)
   ret <vscale x 2 x i64> %out
 }
 
 define <vscale x 8 x i16> @index_ii_range_combine(i16 %a) {
 ; CHECK-LABEL: index_ii_range_combine:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    index z0.h, #2, #8
+; CHECK-NEXT:    index z0.h, #0, #8
+; CHECK-NEXT:    orr z0.h, z0.h, #0x2
 ; CHECK-NEXT:    ret
   %val = insertelement <vscale x 8 x i16> poison, i16 2, i32 0
   %val1 = shufflevector <vscale x 8 x i16> %val, <vscale x 8 x i16> poison, <vscale x 8 x i32> zeroinitializer
   %val2 = call <vscale x 8 x i16> @llvm.aarch64.sve.index.nxv8i16(i16 0, i16 2)
   %val3 = shl <vscale x 8 x i16> %val2, %val1
   %out = add <vscale x 8 x i16> %val3, %val1
   ret <vscale x 8 x i16> %out
 }

llvm/test/CodeGen/AArch64/sve-intrinsics-perm-select.ll

 }
 
 ; NOTE: Index out of range (0-3)
 define <vscale x 2 x i64> @dupq_i64_range(<vscale x 2 x i64> %a) {
 ; CHECK-LABEL: dupq_i64_range:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    index z1.d, #0, #1
 ; CHECK-NEXT:    and z1.d, z1.d, #0x1
-; CHECK-NEXT:    add z1.d, z1.d, #8 // =0x8
+; CHECK-NEXT:    orr z1.d, z1.d, #0x8
 ; CHECK-NEXT:    tbl z0.d, { z0.d }, z1.d
 ; CHECK-NEXT:    ret
   %out = call <vscale x 2 x i64> @llvm.aarch64.sve.dupq.lane.nxv2i64(<vscale x 2 x i64> %a, i64 4)
   ret <vscale x 2 x i64> %out
 }
 ;
 ; EXT
 ;

llvm/test/CodeGen/AArch64/sve-umulo-sdnode.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve -verify-machineinstrs < %s | FileCheck %s
 
 declare { <vscale x 2 x i8>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i8(<vscale x 2 x i8>, <vscale x 2 x i8>)
 
 define <vscale x 2 x i8> @umulo_nxv2i8(<vscale x 2 x i8> %x, <vscale x 2 x i8> %y) {
 ; CHECK-LABEL: umulo_nxv2i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.d, p0/m, z2.d, z1.d
-; CHECK-NEXT:    umulh z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    lsr z1.d, z2.d, #8
-; CHECK-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    lsr z1.d, z0.d, #8
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z1.d, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.d, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.d, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 2 x i8>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i8(<vscale x 2 x i8> %x, <vscale x 2 x i8> %y)
   %b = extractvalue { <vscale x 2 x i8>, <vscale x 2 x i1> } %a, 0
   %c = extractvalue { <vscale x 2 x i8>, <vscale x 2 x i1> } %a, 1
   %d = select <vscale x 2 x i1> %c, <vscale x 2 x i8> zeroinitializer, <vscale x 2 x i8> %b
   ret <vscale x 2 x i8> %d
 }
 
 declare { <vscale x 4 x i8>, <vscale x 4 x i1> } @llvm.umul.with.overflow.nxv4i8(<vscale x 4 x i8>, <vscale x 4 x i8>)
 
 define <vscale x 4 x i8> @umulo_nxv4i8(<vscale x 4 x i8> %x, <vscale x 4 x i8> %y) {
 ; CHECK-LABEL: umulo_nxv4i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    and z1.s, z1.s, #0xff
 ; CHECK-NEXT:    and z0.s, z0.s, #0xff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.s, p0/m, z2.s, z1.s
-; CHECK-NEXT:    umulh z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    lsr z1.s, z2.s, #8
-; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    mul z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    lsr z1.s, z0.s, #8
 ; CHECK-NEXT:    cmpne p0.s, p0/z, z1.s, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.s, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.s, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 4 x i8>, <vscale x 4 x i1> } @llvm.umul.with.overflow.nxv4i8(<vscale x 4 x i8> %x, <vscale x 4 x i8> %y)
   %b = extractvalue { <vscale x 4 x i8>, <vscale x 4 x i1> } %a, 0
   %c = extractvalue { <vscale x 4 x i8>, <vscale x 4 x i1> } %a, 1
   %d = select <vscale x 4 x i1> %c, <vscale x 4 x i8> zeroinitializer, <vscale x 4 x i8> %b
   ret <vscale x 4 x i8> %d
 }
 

 declare { <vscale x 2 x i16>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i16(<vscale x 2 x i16>, <vscale x 2 x i16>)
 
 define <vscale x 2 x i16> @umulo_nxv2i16(<vscale x 2 x i16> %x, <vscale x 2 x i16> %y) {
 ; CHECK-LABEL: umulo_nxv2i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xffff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xffff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.d, p0/m, z2.d, z1.d
-; CHECK-NEXT:    umulh z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    lsr z1.d, z2.d, #16
-; CHECK-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    lsr z1.d, z0.d, #16
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z1.d, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.d, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.d, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 2 x i16>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i16(<vscale x 2 x i16> %x, <vscale x 2 x i16> %y)
   %b = extractvalue { <vscale x 2 x i16>, <vscale x 2 x i1> } %a, 0
   %c = extractvalue { <vscale x 2 x i16>, <vscale x 2 x i1> } %a, 1
   %d = select <vscale x 2 x i1> %c, <vscale x 2 x i16> zeroinitializer, <vscale x 2 x i16> %b
   ret <vscale x 2 x i16> %d
 }
 
 declare { <vscale x 4 x i16>, <vscale x 4 x i1> } @llvm.umul.with.overflow.nxv4i16(<vscale x 4 x i16>, <vscale x 4 x i16>)
 
 define <vscale x 4 x i16> @umulo_nxv4i16(<vscale x 4 x i16> %x, <vscale x 4 x i16> %y) {
 ; CHECK-LABEL: umulo_nxv4i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s
 ; CHECK-NEXT:    and z1.s, z1.s, #0xffff
 ; CHECK-NEXT:    and z0.s, z0.s, #0xffff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.s, p0/m, z2.s, z1.s
-; CHECK-NEXT:    umulh z0.s, p0/m, z0.s, z1.s
-; CHECK-NEXT:    lsr z1.s, z2.s, #16
-; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    mul z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    lsr z1.s, z0.s, #16
 ; CHECK-NEXT:    cmpne p0.s, p0/z, z1.s, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.s, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.s, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 4 x i16>, <vscale x 4 x i1> } @llvm.umul.with.overflow.nxv4i16(<vscale x 4 x i16> %x, <vscale x 4 x i16> %y)
   %b = extractvalue { <vscale x 4 x i16>, <vscale x 4 x i1> } %a, 0
   %c = extractvalue { <vscale x 4 x i16>, <vscale x 4 x i1> } %a, 1
   %d = select <vscale x 4 x i1> %c, <vscale x 4 x i16> zeroinitializer, <vscale x 4 x i16> %b
   ret <vscale x 4 x i16> %d
 }
 

 declare { <vscale x 2 x i32>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i32(<vscale x 2 x i32>, <vscale x 2 x i32>)
 
 define <vscale x 2 x i32> @umulo_nxv2i32(<vscale x 2 x i32> %x, <vscale x 2 x i32> %y) {
 ; CHECK-LABEL: umulo_nxv2i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xffffffff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff
-; CHECK-NEXT:    movprfx z2, z0
-; CHECK-NEXT:    mul z2.d, p0/m, z2.d, z1.d
-; CHECK-NEXT:    umulh z0.d, p0/m, z0.d, z1.d
-; CHECK-NEXT:    lsr z1.d, z2.d, #32
-; CHECK-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    lsr z1.d, z0.d, #32
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z1.d, #0
-; CHECK-NEXT:    sel p0.b, p0, p0.b, p1.b
-; CHECK-NEXT:    mov z2.d, p0/m, #0 // =0x0
-; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    mov z0.d, p0/m, #0 // =0x0
 ; CHECK-NEXT:    ret
   %a = call { <vscale x 2 x i32>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i32(<vscale x 2 x i32> %x, <vscale x 2 x i32> %y)
   %b = extractvalue { <vscale x 2 x i32>, <vscale x 2 x i1> } %a, 0
   %c = extractvalue { <vscale x 2 x i32>, <vscale x 2 x i1> } %a, 1
   %d = select <vscale x 2 x i1> %c, <vscale x 2 x i32> zeroinitializer, <vscale x 2 x i32> %b
   ret <vscale x 2 x i32> %d
 }