[SelectionDAG] Fixed the scalable vectors issue on WidenVecRes_OverflowOp&WidenVecRes_SELECT

[tangxingxin1008]

Summary

This change make WidenVecRes_OverflowOp&WidenVecRes_SELECT work
for scalable vectors.

This patch base on https://reviews.llvm.org/D110058

Signed-off-by: Eric Tang <[e-mail]>

[frasercrmck]

I know they're both small changes but I think it would be good to split this patch up into separate parts: overflow and select. They're conceptually distinct changes. Then we could have more targeted tests for each operation and each change.

Also RISC-V would benefit from the select change, e.g., adding this sort of test to test/CodeGen/RISCV/rvv/vselect-int-rv32.ll and vselect-int-rv64.ll

define <vscale x 3 x i8> @vmerge_vv_nxv3i8(<vscale x 3 x i8> %va, <vscale x 3 x i8> %vb, <vscale x 3 x i1> %cond) {
  %vc = select <vscale x 3 x i1> %cond, <vscale x 3 x i8> %va, <vscale x 3 x i8> %vb
  ret <vscale x 3 x i8> %vc
}

llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

854

This looks suspicious to me. What's it for?

[tangxingxin1008]

In D110319#3017678, @frasercrmck wrote:

I know they're both small changes but I think it would be good to split this patch up into separate parts: overflow and select. They're conceptually distinct changes. Then we could have more targeted tests for each operation and each change.

Also RISC-V would benefit from the select change, e.g., adding this sort of test to test/CodeGen/RISCV/rvv/vselect-int-rv32.ll and vselect-int-rv64.ll

define <vscale x 3 x i8> @vmerge_vv_nxv3i8(<vscale x 3 x i8> %va, <vscale x 3 x i8> %vb, <vscale x 3 x i1> %cond) {
  %vc = select <vscale x 3 x i1> %cond, <vscale x 3 x i8> %va, <vscale x 3 x i8> %vb
  ret <vscale x 3 x i8> %vc
}

thanks, I will be changed according to your advice.

llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

854

In AArch sve, only [nxv2i1, nxv4i1, nxv8i1, nxv16i1] [nxv16i8, nxv8i16, nxv4i32, nxv2i64] are legal, so <nxv1i8, nxv1i1> would be widened to <nxv16i8, nxv16i1> by WidenVecRes_OverflowOp.
In below code:

unsigned OtherNo = 1 - ResNo;
EVT OtherVT = N->getValueType(OtherNo);
if (getTypeAction(OtherVT) == TargetLowering::TypeWidenVector) {
  SetWidenedVector(SDValue(N, OtherNo), SDValue(WideNode, OtherNo));
}

getTypeAction(OtherVT) would convert <nxv1i1> to <nxv2i1>.

So the SDValue(N, OtherNo) value type is <nxv2i1>, but SDValue(WideNode, OtherNo) is <nxv16i1>. In this case, SetWidenedVector will be failed in

assert(Result.getValueType() ==
           TLI.getTypeToTransformTo(*DAG.getContext(), Op.getValueType()) &&
           "Invalid type for widened vector");

In my understand, I do this change like as SetSplitVector does. is that right?

[tangxingxin1008]

rebase

[tangxingxin1008]

For SetWidenedVector, this change reduced the restrictions. In AArch64 sve, normally, SetWidenedVector only support nxv1i1 -> nxv2i1, after this change, the nxv1i1 -> nxv4i1, nxv1i1 -> nxv8i1, nxv1i1 -> nxv16i1 can also work.

[david-arm]

Nothing seems to have happened on this patch for a year.

llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

 //===-- LegalizeTypes.cpp - Common code for DAG type legalizer ------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
void DAGTypeLegalizer::SetSplitVector(SDValue Op, SDValue Lo,
   // Remember that this is the result of the node.
   std::pair<TableId, TableId> &Entry = SplitVectors[getTableId(Op)];
   assert((Entry.first == 0) && "Node already split");
   Entry.first = getTableId(Lo);
   Entry.second = getTableId(Hi);
 }
 
 void DAGTypeLegalizer::SetWidenedVector(SDValue Op, SDValue Result) {
-  assert(Result.getValueType() ==
-         TLI.getTypeToTransformTo(*DAG.getContext(), Op.getValueType()) &&
+  assert(Result.getValueType().getVectorElementType() ==
+             TLI.getTypeToTransformTo(*DAG.getContext(), Op.getValueType())
+                 .getVectorElementType() &&
          "Invalid type for widened vector");
   AnalyzeNewValue(Result);
 
   auto &OpIdEntry = WidenedVectors[getTableId(Op)];
   assert((OpIdEntry == 0) && "Node already widened!");
   OpIdEntry = getTableId(Result);
 }
 

llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

 //===------- LegalizeVectorTypes.cpp - Legalization of vector types -------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
SDValue DAGTypeLegalizer::WidenVecRes_OverflowOp(SDNode *N, unsigned ResNo) {
   EVT WideResVT, WideOvVT;
   SDValue WideLHS, WideRHS;
 
   // TODO: This might result in a widen/split loop.
   if (ResNo == 0) {
     WideResVT = TLI.getTypeToTransformTo(*DAG.getContext(), ResVT);
     WideOvVT = EVT::getVectorVT(
         *DAG.getContext(), OvVT.getVectorElementType(),
-        WideResVT.getVectorNumElements());
+        WideResVT.getVectorElementCount());
 
     WideLHS = GetWidenedVector(N->getOperand(0));
     WideRHS = GetWidenedVector(N->getOperand(1));
   } else {
     WideOvVT = TLI.getTypeToTransformTo(*DAG.getContext(), OvVT);
     WideResVT = EVT::getVectorVT(
         *DAG.getContext(), ResVT.getVectorElementType(),
-        WideOvVT.getVectorNumElements());
+        WideOvVT.getVectorElementCount());
 
     SDValue Zero = DAG.getVectorIdxConstant(0, DL);
     WideLHS = DAG.getNode(
         ISD::INSERT_SUBVECTOR, DL, WideResVT, DAG.getUNDEF(WideResVT),
         N->getOperand(0), Zero);
     WideRHS = DAG.getNode(
         ISD::INSERT_SUBVECTOR, DL, WideResVT, DAG.getUNDEF(WideResVT),
         N->getOperand(1), Zero);
SDValue DAGTypeLegalizer::WidenVSELECTMask(SDNode *N) {
   } else
     return SDValue();
 
   return Mask;
 }
 
 SDValue DAGTypeLegalizer::WidenVecRes_SELECT(SDNode *N) {
   EVT WidenVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
-  unsigned WidenNumElts = WidenVT.getVectorNumElements();
+  ElementCount WidenNumElts = WidenVT.getVectorElementCount();
 
   SDValue Cond1 = N->getOperand(0);
   EVT CondVT = Cond1.getValueType();
   if (CondVT.isVector()) {
     if (SDValue WideCond = WidenVSELECTMask(N)) {
       SDValue InOp1 = GetWidenedVector(N->getOperand(1));
       SDValue InOp2 = GetWidenedVector(N->getOperand(2));
       assert(InOp1.getValueType() == WidenVT && InOp2.getValueType() == WidenVT);

llvm/test/CodeGen/AArch64/sve-smulo-sdnode.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve -verify-machineinstrs < %s | FileCheck %s
 
+declare { <vscale x 1 x i8>, <vscale x 1 x i1> } @llvm.smul.with.overflow.nxv1i8(<vscale x 1 x i8>, <vscale x 1 x i8>)
+
+define <vscale x 1 x i8> @smulo_nxv1i8(<vscale x 1 x i8> %x, <vscale x 1 x i8> %y) {
+; CHECK-LABEL: smulo_nxv1i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    movprfx z2, z0
+; CHECK-NEXT:    smulh z2.b, p0/m, z2.b, z1.b
+; CHECK-NEXT:    mul z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    asr z1.b, z0.b, #7
+; CHECK-NEXT:    cmpne p0.b, p0/z, z2.b, z1.b
+; CHECK-NEXT:    mov z0.b, p0/m, #0 // =0x0
+; CHECK-NEXT:    ret
+  %a = call { <vscale x 1 x i8>, <vscale x 1 x i1> } @llvm.smul.with.overflow.nxv1i8(<vscale x 1 x i8> %x, <vscale x 1 x i8> %y)
+  %b = extractvalue { <vscale x 1 x i8>, <vscale x 1 x i1> } %a, 0
+  %c = extractvalue { <vscale x 1 x i8>, <vscale x 1 x i1> } %a, 1
+  %d = select <vscale x 1 x i1> %c, <vscale x 1 x i8> zeroinitializer, <vscale x 1 x i8> %b
+  ret <vscale x 1 x i8> %d
+}
+
 declare { <vscale x 2 x i8>, <vscale x 2 x i1> } @llvm.smul.with.overflow.nxv2i8(<vscale x 2 x i8>, <vscale x 2 x i8>)
 
 define <vscale x 2 x i8> @smulo_nxv2i8(<vscale x 2 x i8> %x, <vscale x 2 x i8> %y) {
 ; CHECK-LABEL: smulo_nxv2i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    sxtb z1.d, p0/m, z1.d
 ; CHECK-NEXT:    sxtb z0.d, p0/m, z0.d

 ; CHECK-NEXT:    ret
   %a = call { <vscale x 64 x i8>, <vscale x 64 x i1> } @llvm.smul.with.overflow.nxv64i8(<vscale x 64 x i8> %x, <vscale x 64 x i8> %y)
   %b = extractvalue { <vscale x 64 x i8>, <vscale x 64 x i1> } %a, 0
   %c = extractvalue { <vscale x 64 x i8>, <vscale x 64 x i1> } %a, 1
   %d = select <vscale x 64 x i1> %c, <vscale x 64 x i8> zeroinitializer, <vscale x 64 x i8> %b
   ret <vscale x 64 x i8> %d
 }
 
+declare { <vscale x 1 x i16>, <vscale x 1 x i1> } @llvm.smul.with.overflow.nxv1i16(<vscale x 1 x i16>, <vscale x 1 x i16>)
+
+define <vscale x 1 x i16> @smulo_nxv1i16(<vscale x 1 x i16> %x, <vscale x 1 x i16> %y) {
+; CHECK-LABEL: smulo_nxv1i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    movprfx z2, z0
+; CHECK-NEXT:    smulh z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT:    mul z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    asr z1.h, z0.h, #15
+; CHECK-NEXT:    cmpne p0.h, p0/z, z2.h, z1.h
+; CHECK-NEXT:    mov z0.h, p0/m, #0 // =0x0
+; CHECK-NEXT:    ret
+  %a = call { <vscale x 1 x i16>, <vscale x 1 x i1> } @llvm.smul.with.overflow.nxv1i16(<vscale x 1 x i16> %x, <vscale x 1 x i16> %y)
+  %b = extractvalue { <vscale x 1 x i16>, <vscale x 1 x i1> } %a, 0
+  %c = extractvalue { <vscale x 1 x i16>, <vscale x 1 x i1> } %a, 1
+  %d = select <vscale x 1 x i1> %c, <vscale x 1 x i16> zeroinitializer, <vscale x 1 x i16> %b
+  ret <vscale x 1 x i16> %d
+}
+
 declare { <vscale x 2 x i16>, <vscale x 2 x i1> } @llvm.smul.with.overflow.nxv2i16(<vscale x 2 x i16>, <vscale x 2 x i16>)
 
 define <vscale x 2 x i16> @smulo_nxv2i16(<vscale x 2 x i16> %x, <vscale x 2 x i16> %y) {
 ; CHECK-LABEL: smulo_nxv2i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    sxth z1.d, p0/m, z1.d
 ; CHECK-NEXT:    sxth z0.d, p0/m, z0.d

 ; CHECK-NEXT:    ret
   %a = call { <vscale x 32 x i16>, <vscale x 32 x i1> } @llvm.smul.with.overflow.nxv32i16(<vscale x 32 x i16> %x, <vscale x 32 x i16> %y)
   %b = extractvalue { <vscale x 32 x i16>, <vscale x 32 x i1> } %a, 0
   %c = extractvalue { <vscale x 32 x i16>, <vscale x 32 x i1> } %a, 1
   %d = select <vscale x 32 x i1> %c, <vscale x 32 x i16> zeroinitializer, <vscale x 32 x i16> %b
   ret <vscale x 32 x i16> %d
 }
 
+declare { <vscale x 1 x i32>, <vscale x 1 x i1> } @llvm.smul.with.overflow.nxv1i32(<vscale x 1 x i32>, <vscale x 1 x i32>)
+
+define <vscale x 1 x i32> @smulo_nxv1i32(<vscale x 1 x i32> %x, <vscale x 1 x i32> %y) {
+; CHECK-LABEL: smulo_nxv1i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    movprfx z2, z0
+; CHECK-NEXT:    smulh z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT:    mul z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    asr z1.s, z0.s, #31
+; CHECK-NEXT:    cmpne p0.s, p0/z, z2.s, z1.s
+; CHECK-NEXT:    mov z0.s, p0/m, #0 // =0x0
+; CHECK-NEXT:    ret
+  %a = call { <vscale x 1 x i32>, <vscale x 1 x i1> } @llvm.smul.with.overflow.nxv1i32(<vscale x 1 x i32> %x, <vscale x 1 x i32> %y)
+  %b = extractvalue { <vscale x 1 x i32>, <vscale x 1 x i1> } %a, 0
+  %c = extractvalue { <vscale x 1 x i32>, <vscale x 1 x i1> } %a, 1
+  %d = select <vscale x 1 x i1> %c, <vscale x 1 x i32> zeroinitializer, <vscale x 1 x i32> %b
+  ret <vscale x 1 x i32> %d
+}
+
 declare { <vscale x 2 x i32>, <vscale x 2 x i1> } @llvm.smul.with.overflow.nxv2i32(<vscale x 2 x i32>, <vscale x 2 x i32>)
 
 define <vscale x 2 x i32> @smulo_nxv2i32(<vscale x 2 x i32> %x, <vscale x 2 x i32> %y) {
 ; CHECK-LABEL: smulo_nxv2i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    sxtw z1.d, p0/m, z1.d
 ; CHECK-NEXT:    sxtw z0.d, p0/m, z0.d

 ; CHECK-NEXT:    ret
   %a = call { <vscale x 16 x i32>, <vscale x 16 x i1> } @llvm.smul.with.overflow.nxv16i32(<vscale x 16 x i32> %x, <vscale x 16 x i32> %y)
   %b = extractvalue { <vscale x 16 x i32>, <vscale x 16 x i1> } %a, 0
   %c = extractvalue { <vscale x 16 x i32>, <vscale x 16 x i1> } %a, 1
   %d = select <vscale x 16 x i1> %c, <vscale x 16 x i32> zeroinitializer, <vscale x 16 x i32> %b
   ret <vscale x 16 x i32> %d
 }
 
+declare { <vscale x 1 x i64>, <vscale x 1 x i1> } @llvm.smul.with.overflow.nxv1i64(<vscale x 1 x i64>, <vscale x 1 x i64>)
+
+define <vscale x 1 x i64> @smulo_nxv1i64(<vscale x 1 x i64> %x, <vscale x 1 x i64> %y) {
+; CHECK-LABEL: smulo_nxv1i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    movprfx z2, z0
+; CHECK-NEXT:    smulh z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT:    mul z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    asr z1.d, z0.d, #63
+; CHECK-NEXT:    cmpne p0.d, p0/z, z2.d, z1.d
+; CHECK-NEXT:    mov z0.d, p0/m, #0 // =0x0
+; CHECK-NEXT:    ret
+  %a = call { <vscale x 1 x i64>, <vscale x 1 x i1> } @llvm.smul.with.overflow.nxv1i64(<vscale x 1 x i64> %x, <vscale x 1 x i64> %y)
+  %b = extractvalue { <vscale x 1 x i64>, <vscale x 1 x i1> } %a, 0
+  %c = extractvalue { <vscale x 1 x i64>, <vscale x 1 x i1> } %a, 1
+  %d = select <vscale x 1 x i1> %c, <vscale x 1 x i64> zeroinitializer, <vscale x 1 x i64> %b
+  ret <vscale x 1 x i64> %d
+}
+
 declare { <vscale x 2 x i64>, <vscale x 2 x i1> } @llvm.smul.with.overflow.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i64>)
 
 define <vscale x 2 x i64> @smulo_nxv2i64(<vscale x 2 x i64> %x, <vscale x 2 x i64> %y) {
 ; CHECK-LABEL: smulo_nxv2i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    movprfx z2, z0
 ; CHECK-NEXT:    smulh z2.d, p0/m, z2.d, z1.d

llvm/test/CodeGen/AArch64/sve-umulo-sdnode.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve -verify-machineinstrs < %s | FileCheck %s
 
+declare { <vscale x 1 x i8>, <vscale x 1 x i1> } @llvm.umul.with.overflow.nxv1i8(<vscale x 1 x i8>, <vscale x 1 x i8>)
+
+define <vscale x 1 x i8> @umulo_nxv1i8(<vscale x 1 x i8> %x, <vscale x 1 x i8> %y) {
+; CHECK-LABEL: umulo_nxv1i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    movprfx z2, z0
+; CHECK-NEXT:    mul z2.b, p0/m, z2.b, z1.b
+; CHECK-NEXT:    umulh z0.b, p0/m, z0.b, z1.b
+; CHECK-NEXT:    cmpne p0.b, p0/z, z0.b, #0
+; CHECK-NEXT:    mov z2.b, p0/m, #0 // =0x0
+; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    ret
+  %a = call { <vscale x 1 x i8>, <vscale x 1 x i1> } @llvm.umul.with.overflow.nxv1i8(<vscale x 1 x i8> %x, <vscale x 1 x i8> %y)
+  %b = extractvalue { <vscale x 1 x i8>, <vscale x 1 x i1> } %a, 0
+  %c = extractvalue { <vscale x 1 x i8>, <vscale x 1 x i1> } %a, 1
+  %d = select <vscale x 1 x i1> %c, <vscale x 1 x i8> zeroinitializer, <vscale x 1 x i8> %b
+  ret <vscale x 1 x i8> %d
+}
+
 declare { <vscale x 2 x i8>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i8(<vscale x 2 x i8>, <vscale x 2 x i8>)
 
 define <vscale x 2 x i8> @umulo_nxv2i8(<vscale x 2 x i8> %x, <vscale x 2 x i8> %y) {
 ; CHECK-LABEL: umulo_nxv2i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xff

 ; CHECK-NEXT:    ret
   %a = call { <vscale x 64 x i8>, <vscale x 64 x i1> } @llvm.umul.with.overflow.nxv64i8(<vscale x 64 x i8> %x, <vscale x 64 x i8> %y)
   %b = extractvalue { <vscale x 64 x i8>, <vscale x 64 x i1> } %a, 0
   %c = extractvalue { <vscale x 64 x i8>, <vscale x 64 x i1> } %a, 1
   %d = select <vscale x 64 x i1> %c, <vscale x 64 x i8> zeroinitializer, <vscale x 64 x i8> %b
   ret <vscale x 64 x i8> %d
 }
 
+declare { <vscale x 1 x i16>, <vscale x 1 x i1> } @llvm.umul.with.overflow.nxv1i16(<vscale x 1 x i16>, <vscale x 1 x i16>)
+
+define <vscale x 1 x i16> @umulo_nxv1i16(<vscale x 1 x i16> %x, <vscale x 1 x i16> %y) {
+; CHECK-LABEL: umulo_nxv1i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    movprfx z2, z0
+; CHECK-NEXT:    mul z2.h, p0/m, z2.h, z1.h
+; CHECK-NEXT:    umulh z0.h, p0/m, z0.h, z1.h
+; CHECK-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; CHECK-NEXT:    mov z2.h, p0/m, #0 // =0x0
+; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    ret
+  %a = call { <vscale x 1 x i16>, <vscale x 1 x i1> } @llvm.umul.with.overflow.nxv1i16(<vscale x 1 x i16> %x, <vscale x 1 x i16> %y)
+  %b = extractvalue { <vscale x 1 x i16>, <vscale x 1 x i1> } %a, 0
+  %c = extractvalue { <vscale x 1 x i16>, <vscale x 1 x i1> } %a, 1
+  %d = select <vscale x 1 x i1> %c, <vscale x 1 x i16> zeroinitializer, <vscale x 1 x i16> %b
+  ret <vscale x 1 x i16> %d
+}
+
 declare { <vscale x 2 x i16>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i16(<vscale x 2 x i16>, <vscale x 2 x i16>)
 
 define <vscale x 2 x i16> @umulo_nxv2i16(<vscale x 2 x i16> %x, <vscale x 2 x i16> %y) {
 ; CHECK-LABEL: umulo_nxv2i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xffff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xffff

 ; CHECK-NEXT:    ret
   %a = call { <vscale x 32 x i16>, <vscale x 32 x i1> } @llvm.umul.with.overflow.nxv32i16(<vscale x 32 x i16> %x, <vscale x 32 x i16> %y)
   %b = extractvalue { <vscale x 32 x i16>, <vscale x 32 x i1> } %a, 0
   %c = extractvalue { <vscale x 32 x i16>, <vscale x 32 x i1> } %a, 1
   %d = select <vscale x 32 x i1> %c, <vscale x 32 x i16> zeroinitializer, <vscale x 32 x i16> %b
   ret <vscale x 32 x i16> %d
 }
 
+declare { <vscale x 1 x i32>, <vscale x 1 x i1> } @llvm.umul.with.overflow.nxv1i32(<vscale x 1 x i32>, <vscale x 1 x i32>)
+
+define <vscale x 1 x i32> @umulo_nxv1i32(<vscale x 1 x i32> %x, <vscale x 1 x i32> %y) {
+; CHECK-LABEL: umulo_nxv1i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    movprfx z2, z0
+; CHECK-NEXT:    mul z2.s, p0/m, z2.s, z1.s
+; CHECK-NEXT:    umulh z0.s, p0/m, z0.s, z1.s
+; CHECK-NEXT:    cmpne p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    mov z2.s, p0/m, #0 // =0x0
+; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    ret
+  %a = call { <vscale x 1 x i32>, <vscale x 1 x i1> } @llvm.umul.with.overflow.nxv1i32(<vscale x 1 x i32> %x, <vscale x 1 x i32> %y)
+  %b = extractvalue { <vscale x 1 x i32>, <vscale x 1 x i1> } %a, 0
+  %c = extractvalue { <vscale x 1 x i32>, <vscale x 1 x i1> } %a, 1
+  %d = select <vscale x 1 x i1> %c, <vscale x 1 x i32> zeroinitializer, <vscale x 1 x i32> %b
+  ret <vscale x 1 x i32> %d
+}
+
 declare { <vscale x 2 x i32>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i32(<vscale x 2 x i32>, <vscale x 2 x i32>)
 
 define <vscale x 2 x i32> @umulo_nxv2i32(<vscale x 2 x i32> %x, <vscale x 2 x i32> %y) {
 ; CHECK-LABEL: umulo_nxv2i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    and z1.d, z1.d, #0xffffffff
 ; CHECK-NEXT:    and z0.d, z0.d, #0xffffffff

 ; CHECK-NEXT:    ret
   %a = call { <vscale x 16 x i32>, <vscale x 16 x i1> } @llvm.umul.with.overflow.nxv16i32(<vscale x 16 x i32> %x, <vscale x 16 x i32> %y)
   %b = extractvalue { <vscale x 16 x i32>, <vscale x 16 x i1> } %a, 0
   %c = extractvalue { <vscale x 16 x i32>, <vscale x 16 x i1> } %a, 1
   %d = select <vscale x 16 x i1> %c, <vscale x 16 x i32> zeroinitializer, <vscale x 16 x i32> %b
   ret <vscale x 16 x i32> %d
 }
 
+declare { <vscale x 1 x i64>, <vscale x 1 x i1> } @llvm.umul.with.overflow.nxv1i64(<vscale x 1 x i64>, <vscale x 1 x i64>)
+
+define <vscale x 1 x i64> @umulo_nxv1i64(<vscale x 1 x i64> %x, <vscale x 1 x i64> %y) {
+; CHECK-LABEL: umulo_nxv1i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    movprfx z2, z0
+; CHECK-NEXT:    mul z2.d, p0/m, z2.d, z1.d
+; CHECK-NEXT:    umulh z0.d, p0/m, z0.d, z1.d
+; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; CHECK-NEXT:    mov z2.d, p0/m, #0 // =0x0
+; CHECK-NEXT:    mov z0.d, z2.d
+; CHECK-NEXT:    ret
+  %a = call { <vscale x 1 x i64>, <vscale x 1 x i1> } @llvm.umul.with.overflow.nxv1i64(<vscale x 1 x i64> %x, <vscale x 1 x i64> %y)
+  %b = extractvalue { <vscale x 1 x i64>, <vscale x 1 x i1> } %a, 0
+  %c = extractvalue { <vscale x 1 x i64>, <vscale x 1 x i1> } %a, 1
+  %d = select <vscale x 1 x i1> %c, <vscale x 1 x i64> zeroinitializer, <vscale x 1 x i64> %b
+  ret <vscale x 1 x i64> %d
+}
+
 declare { <vscale x 2 x i64>, <vscale x 2 x i1> } @llvm.umul.with.overflow.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i64>)
 
 define <vscale x 2 x i64> @umulo_nxv2i64(<vscale x 2 x i64> %x, <vscale x 2 x i64> %y) {
 ; CHECK-LABEL: umulo_nxv2i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d
 ; CHECK-NEXT:    movprfx z2, z0
 ; CHECK-NEXT:    mul z2.d, p0/m, z2.d, z1.d