[SystemZ] Wait with VGBM selection until after DAGCombine2.

[jonpa]

Summary

I was looking back at the shouldScalarizeBinop() hook, and found that if I used the X86 implementation of it, nothing changed on SPEC.

I then also remembered your discussion that perhaps it would be better to keep BUILD_VECTOR nodes during combine2, instead of replacing with SYSTEMZ::BYTE_MASK during legalization. I decided to try this and this is the patch I have.

It seems to be not too complicated to do this, since apart from the handling in Select() it is enough to redefine the z_vzero and z_vones nodes to recognize BUILD_VECTORs instead, and the pattern matching will work as before.

I am not quite sure if this is the best solution, but as it is now tryBuildVectorByteMask() is used first during legalization to build a new BUILD_VECTOR with the right constants, and then again in Select() to get the same mask back again. I first thought it would be possible to just leave the BUILD_VECTORS during legalization, but then I found a case where this doesn't
work which involved ConstantFP<nan>, which ended up in the constant pool.

First observations on benchmarks is that just one file (462.libquantum/build/qec.s) changes like (inside a vectorized loop):

vno            :                  203                  193      -10
vnc            :                  149                  151       +2
vn             :                  557                  555       -2

This is a surprisingly small improvement. Perhaps some piece is missing to unlock more improvements?

With this patch in place, I tried the shouldScalarizeBinop() hook again (copied from X86), and now one additional file changed (454.calculix/build/InpMtx_init.s) like

xilf           :                 4922                 4942      +20
vno            :                  193                  190       -3
tmll           :                14386                14385       -1
jne            :                15139                15138       -1
la             :               199056               199055       -1
vlgvf          :                 1266                 1265       -1

It seems this is a loop with many extracts and test-under-mask:s, that now do a scalar xilf before each tmll...

The handling of replication of constant in lowerBUILD_VECTOR() is perhaps the next step after this to rework in a similar way...

[uweigand]

I am not quite sure if this is the best solution, but as it is now tryBuildVectorByteMask() is used first during legalization to build a new BUILD_VECTOR with the right constants, and then again in Select() to get the same mask back again. I first thought it would be possible to just leave the BUILD_VECTORS during legalization, but then I found a case where this doesn't work which involved ConstantFP<nan>, which ended up in the constant pool.

Can you explain this further? If you leave the BUILD_VECTOR as-is, where is it pushed into the constant pool?

The handling of replication of constant in lowerBUILD_VECTOR() is perhaps the next step after this to rework in a similar way...

That would probably make sense.

It seems this is a loop with many extracts and test-under-mask:s, that now do a scalar xilf before each tmll...

Well, that's certainly not good.

[jonpa]

Can you explain this further? If you leave the BUILD_VECTOR as-is, where is it pushed into the constant pool?

If Op is returned unchanged from lowerBUILD_VECTOR(), SelectionDAG::legalize() will later call Legalizer.LegalizeOp(N), with the ConstantFP operand of the BUILD_VECTOR. Since it is not a TargetConstant, this will go on to find the Action to be 'Expand'. The ExpandNode() case for ConstantFP will call TLI.isFPImmLegal() which return false (in most cases), and therefore ExpandFPConstant() is called, which returns a load from the constant pool.

It is in fact not just NaN ConstantFPs that is in question here, but rather any ConstantFP that is a candidate for VGBM. In order to select into VGBM later, it seems we need to detect this case here and produce a new BUILD_VECTOR, either with i8 operands, or perhaps with TargetConstantFP operands, which makes a bit more sense in a way.

I also discovered that it is not so good to replace undef operands with 0, as the patch currently does by using the Mask produced by tryBuildVectorByteMask(). In cases where some elements are all-ones and the rest undef, it is best to keep it that way so that SD::isBuildVectorAllOnes() can return true.

[jonpa]

Patch updated per previous comment so that it returns all integer BVNs unmodified, and for FP a new BVN is build with TargetConstantFP operands, to avoid expansion into the constant pool.

I had to adjust two vllez patterns to make all tests pass.

Due to keeping the undef operands instead of making them explicitly 0, one more file on benchmarks is slightly improved, which gives 2 in total :-)

Not sure if it would simplify the code a bit to build integer (16xi8) BVNs instead of insisting on keeping the FP operands as TargetConstantFP:s.

I am still using tryBuildVectorByteMask() in two places which I think saves some duplicated code even though the Mask isn't used during legalization.

[uweigand]

In D57152#1371369, @jonpa wrote:

The ExpandNode() case for ConstantFP will call TLI.isFPImmLegal() which return false (in most cases), and therefore ExpandFPConstant() is called, which returns a load from the constant pool.

If we actually can load the FP constant using an immediate vector instruction, shouldn't we then return true from TLI.isFPImmLegal? Maybe that's the real underlying problem ...

[jonpa]

If we actually can load the FP constant using an immediate vector instruction, shouldn't we then return true from TLI.isFPImmLegal? Maybe that's the real underlying problem ...

I actually tried that earlier, and found that it worked as you say in the case where BUILD_VECTOR can be selected into VGBM. However, an FP-immediate (outside any BUILD_VECTOR) then crashes isel, since there is no handling for it other than loading it from the constant pool.

I was also a bit skeptical about using isFPImmLegal since there is no way to tell if this is a scalar or vector element constant. However, I tried this now again, and found that this actually seems to work out fairly well in lower_BUILD_VECTOR:

• If all fp-elements are legal, we can return it unmodified just as with an int vector and emit VGBM in Select().
• If not all fp-elements are legal, we return SDValue(), so the entire vector constant is loaded from the constant pool.

This is just what we want, so the question now is what to do with the new legal scalar FP immediates. We could either decide to generate them with VGBM, or we could keep them in the constant pool where they are now. There are plenty of reg-mem FP instructions, and I suspect we would like to keep using them? In that case I think we could either extend isFPImmLegal with an argument that informs us to make the right decision depending on scalar/vector context, or we could possibly say that the scalar constant is legal, but we decide to load it from the constant pool in Select().

[jonpa]

Handle the FP case by building a v16i8 vector instead of using TargetFPConstants (or isFPImmLegal).

NFC to using TargetFPConstants.

[jonpa]

The support of VGBM for non-zero FP vectors has been removed, which greatly simplifies the patch. This is NFC on SPEC compared to previous version of patch, except for in one file where the floating point VGBM mask of 0xf0f0 is selected as VGMG insteadof VGBM (1 file, 8 places).

test functions in vec-const-05.ll and vec-const-06.ll for VGBM masks of non-zero FP vectors removed.

[uweigand]

Yes, I think this version makes most sense. LGTM.

[jonpa]

Thanks for review. r353325.

lib/Target/SystemZ/SystemZISelDAGToDAG.cpp

 //===-- SystemZISelDAGToDAG.cpp - A dag to dag inst selector for SystemZ --===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines an instruction selector for the SystemZ target.
 //
 //===----------------------------------------------------------------------===//
 
 #include "SystemZTargetMachine.h"
+#include "SystemZISelLowering.h"
 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/CodeGen/SelectionDAGISel.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/KnownBits.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace llvm;
 
if (ElemBitSize == 32) {
         return;
     } else if (ElemBitSize == 64) {
       if (tryGather(Node, SystemZ::VGEG))
         return;
     }
     break;
   }
 
+  case ISD::BUILD_VECTOR: {
+    auto *BVN = cast<BuildVectorSDNode>(Node);
+    SDLoc DL(Node);
+    EVT VT = Node->getValueType(0);
+    uint64_t Mask = 0;
+    if (SystemZTargetLowering::tryBuildVectorByteMask(BVN, Mask)) {
+      SDNode *Res = CurDAG->getMachineNode(SystemZ::VGBM, DL, VT,
+                                CurDAG->getTargetConstant(Mask, DL, MVT::i32));
+      ReplaceNode(Node, Res);
+      return;
+    }
+    break;
+  }
+
   case ISD::STORE: {
     if (tryFoldLoadStoreIntoMemOperand(Node))
       return;
     auto *Store = cast<StoreSDNode>(Node);
     unsigned ElemBitSize = Store->getValue().getValueSizeInBits();
     if (ElemBitSize == 32) {
       if (tryScatter(Store, SystemZ::VSCEF))
         return;

lib/Target/SystemZ/SystemZISelLowering.h

enum NodeType : unsigned {
   // the TDB pointer, and the third the immediate control field.
   // Returns CC value and chain.
   TBEGIN,
   TBEGIN_NOFLOAT,
 
   // Transaction end.  Just the chain operand.  Returns CC value and chain.
   TEND,
 
-  // Create a vector constant by filling byte N of the result with bit
-  // 15-N of the single operand.
-  BYTE_MASK,
-
   // Create a vector constant by replicating an element-sized RISBG-style mask.
   // The first operand specifies the starting set bit and the second operand
   // specifies the ending set bit.  Both operands count from the MSB of the
   // element.
   ROTATE_MASK,
 
   // Replicate a GPR scalar value into all elements of a vector.
   REPLICATE,
public:
   ISD::NodeType getExtendForAtomicOps() const override {
     return ISD::ANY_EXTEND;
   }
 
   bool supportSwiftError() const override {
     return true;
   }
 
+  static bool tryBuildVectorByteMask(BuildVectorSDNode *BVN, uint64_t &Mask);
+
 private:
   const SystemZSubtarget &Subtarget;
 
   // Implement LowerOperation for individual opcodes.
   SDValue getVectorCmp(SelectionDAG &DAG, unsigned Opcode,
                        const SDLoc &DL, EVT VT,
                        SDValue CmpOp0, SDValue CmpOp1) const;
   SDValue lowerVectorSETCC(SelectionDAG &DAG, const SDLoc &DL,

lib/Target/SystemZ/SystemZISelLowering.cpp

else {
       if (unsigned Opcode = getVectorComparisonOrInvert(CC, IsFP, Invert))
         Cmp = getVectorCmp(DAG, Opcode, DL, VT, CmpOp1, CmpOp0);
       else
         llvm_unreachable("Unhandled comparison");
     }
     break;
   }
   if (Invert) {
-    SDValue Mask = DAG.getNode(SystemZISD::BYTE_MASK, DL, MVT::v16i8,
-                               DAG.getConstant(65535, DL, MVT::i32));
-    Mask = DAG.getNode(ISD::BITCAST, DL, VT, Mask);
+    SDValue Mask =
+      DAG.getSplatBuildVector(VT, DL, DAG.getConstant(-1, DL, MVT::i64));
     Cmp = DAG.getNode(ISD::XOR, DL, VT, Cmp, Mask);
   }
   return Cmp;
 }
 
 SDValue SystemZTargetLowering::lowerSETCC(SDValue Op,
                                           SelectionDAG &DAG) const {
   SDValue CmpOp0   = Op.getOperand(0);
case 16: {
       Op = DAG.getNode(ISD::BITCAST, DL, VT, Op);
       SDValue Shift = DAG.getConstant(8, DL, MVT::i32);
       SDValue Tmp = DAG.getNode(SystemZISD::VSHL_BY_SCALAR, DL, VT, Op, Shift);
       Op = DAG.getNode(ISD::ADD, DL, VT, Op, Tmp);
       Op = DAG.getNode(SystemZISD::VSRL_BY_SCALAR, DL, VT, Op, Shift);
       break;
     }
     case 32: {
-      SDValue Tmp = DAG.getNode(SystemZISD::BYTE_MASK, DL, MVT::v16i8,
+      SDValue Tmp = DAG.getSplatBuildVector(MVT::v16i8, DL,
                                             DAG.getConstant(0, DL, MVT::i32));
       Op = DAG.getNode(SystemZISD::VSUM, DL, VT, Op, Tmp);
       break;
     }
     case 64: {
-      SDValue Tmp = DAG.getNode(SystemZISD::BYTE_MASK, DL, MVT::v16i8,
+      SDValue Tmp = DAG.getSplatBuildVector(MVT::v16i8, DL,
                                             DAG.getConstant(0, DL, MVT::i32));
       Op = DAG.getNode(SystemZISD::VSUM, DL, MVT::v4i32, Op, Tmp);
       Op = DAG.getNode(SystemZISD::VSUM, DL, VT, Op, Tmp);
       break;
     }
     default:
       llvm_unreachable("Unexpected type");
     }
     return Op;
else if (Op1.isUndef())
     Op0 = Op1 = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, Op0);
   else {
     Op0 = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, Op0);
     Op1 = DAG.getNode(ISD::ANY_EXTEND, DL, MVT::i64, Op1);
   }
   return DAG.getNode(SystemZISD::JOIN_DWORDS, DL, MVT::v2i64, Op0, Op1);
 }
 
-// Try to represent constant BUILD_VECTOR node BVN using a
-// SystemZISD::BYTE_MASK-style mask.  Store the mask value in Mask
-// on success.
-static bool tryBuildVectorByteMask(BuildVectorSDNode *BVN, uint64_t &Mask) {
+// Try to represent constant BUILD_VECTOR node BVN using a BYTE MASK style
+// mask.  Store the mask value in Mask on success.
+bool SystemZTargetLowering::
+tryBuildVectorByteMask(BuildVectorSDNode *BVN, uint64_t &Mask) {
   EVT ElemVT = BVN->getValueType(0).getVectorElementType();
   unsigned BytesPerElement = ElemVT.getStoreSize();
   for (unsigned I = 0, E = BVN->getNumOperands(); I != E; ++I) {
     SDValue Op = BVN->getOperand(I);
     if (!Op.isUndef()) {
       uint64_t Value;
       if (Op.getOpcode() == ISD::Constant)
         Value = cast<ConstantSDNode>(Op)->getZExtValue();
SDValue SystemZTargetLowering::lowerBUILD_VECTOR(SDValue Op,
   auto *BVN = cast<BuildVectorSDNode>(Op.getNode());
   SDLoc DL(Op);
   EVT VT = Op.getValueType();
 
   if (BVN->isConstant()) {
     // Try using VECTOR GENERATE BYTE MASK.  This is the architecturally-
     // preferred way of creating all-zero and all-one vectors so give it
     // priority over other methods below.
-    uint64_t Mask = 0;
-    if (tryBuildVectorByteMask(BVN, Mask)) {
-      SDValue Op = DAG.getNode(
-          SystemZISD::BYTE_MASK, DL, MVT::v16i8,
-          DAG.getConstant(Mask, DL, MVT::i32, false, true /*isOpaque*/));
-      return DAG.getNode(ISD::BITCAST, DL, VT, Op);
-    }
+    uint64_t Mask;
+    if (ISD::isBuildVectorAllZeros(Op.getNode()) ||
+        ISD::isBuildVectorAllOnes(Op.getNode()) ||
+        (VT.isInteger() && tryBuildVectorByteMask(BVN, Mask)))
+      return Op;
 
     // Try using some form of replication.
     APInt SplatBits, SplatUndef;
     unsigned SplatBitSize;
     bool HasAnyUndefs;
     if (BVN->isConstantSplat(SplatBits, SplatUndef, SplatBitSize, HasAnyUndefs,
                              8, true) &&
         SplatBitSize <= 64) {
switch ((SystemZISD::NodeType)Opcode) {
     OPCODE(STPCPY);
     OPCODE(STRCMP);
     OPCODE(SEARCH_STRING);
     OPCODE(IPM);
     OPCODE(MEMBARRIER);
     OPCODE(TBEGIN);
     OPCODE(TBEGIN_NOFLOAT);
     OPCODE(TEND);
-    OPCODE(BYTE_MASK);
     OPCODE(ROTATE_MASK);
     OPCODE(REPLICATE);
     OPCODE(JOIN_DWORDS);
     OPCODE(SPLAT);
     OPCODE(MERGE_HIGH);
     OPCODE(MERGE_LOW);
     OPCODE(SHL_DOUBLE);
     OPCODE(PERMUTE_DWORDS);

 SDValue SystemZTargetLowering::combineMERGE(
     SDNode *N, DAGCombinerInfo &DCI) const {
   SelectionDAG &DAG = DCI.DAG;
   unsigned Opcode = N->getOpcode();
   SDValue Op0 = N->getOperand(0);
   SDValue Op1 = N->getOperand(1);
   if (Op0.getOpcode() == ISD::BITCAST)
     Op0 = Op0.getOperand(0);
-  if (Op0.getOpcode() == SystemZISD::BYTE_MASK &&
-      cast<ConstantSDNode>(Op0.getOperand(0))->getZExtValue() == 0) {
+  if (ISD::isBuildVectorAllZeros(Op0.getNode())) {
     // (z_merge_* 0, 0) -> 0.  This is mostly useful for using VLLEZF
     // for v4f32.
     if (Op1 == N->getOperand(0))
       return Op1;
     // (z_merge_? 0, X) -> (z_unpackl_? 0, X).
     EVT VT = Op1.getValueType();
     unsigned ElemBytes = VT.getVectorElementType().getStoreSize();
     if (ElemBytes <= 4) {

lib/Target/SystemZ/SystemZInstrVector.td

 //===----------------------------------------------------------------------===//
 
 let Predicates = [FeatureVector] in {
   let isAsCheapAsAMove = 1, isMoveImm = 1, isReMaterializable = 1 in {
 
     // Generate byte mask.
     def VZERO : InherentVRIa<"vzero", 0xE744, 0>;
     def VONE  : InherentVRIa<"vone", 0xE744, 0xffff>;
-    def VGBM  : UnaryVRIa<"vgbm", 0xE744, z_byte_mask, v128b, imm32zx16>;
+    def VGBM  : UnaryVRIa<"vgbm", 0xE744, null_frag, v128b, imm32zx16>;
 
     // Generate mask.
     def VGM  : BinaryVRIbGeneric<"vgm", 0xE746>;
     def VGMB : BinaryVRIb<"vgmb", 0xE746, z_rotate_mask, v128b, 0>;
     def VGMH : BinaryVRIb<"vgmh", 0xE746, z_rotate_mask, v128h, 1>;
     def VGMF : BinaryVRIb<"vgmf", 0xE746, z_rotate_mask, v128f, 2>;
     def VGMG : BinaryVRIb<"vgmg", 0xE746, z_rotate_mask, v128g, 3>;
 

lib/Target/SystemZ/SystemZOperators.td

 
 def z_tdc               : SDNode<"SystemZISD::TDC", SDT_ZTest>;
 
 // Defined because the index is an i32 rather than a pointer.
 def z_vector_insert     : SDNode<"ISD::INSERT_VECTOR_ELT",
                                  SDT_ZInsertVectorElt>;
 def z_vector_extract    : SDNode<"ISD::EXTRACT_VECTOR_ELT",
                                  SDT_ZExtractVectorElt>;
-def z_byte_mask         : SDNode<"SystemZISD::BYTE_MASK", SDT_ZReplicate>;
 def z_rotate_mask       : SDNode<"SystemZISD::ROTATE_MASK", SDT_ZRotateMask>;
 def z_replicate         : SDNode<"SystemZISD::REPLICATE", SDT_ZReplicate>;
 def z_join_dwords       : SDNode<"SystemZISD::JOIN_DWORDS", SDT_ZJoinDwords>;
 def z_splat             : SDNode<"SystemZISD::SPLAT", SDT_ZVecBinaryInt>;
 def z_merge_high        : SDNode<"SystemZISD::MERGE_HIGH", SDT_ZVecBinary>;
 def z_merge_low         : SDNode<"SystemZISD::MERGE_LOW", SDT_ZVecBinary>;
 def z_shl_double        : SDNode<"SystemZISD::SHL_DOUBLE", SDT_ZVecTernaryInt>;
 def z_permute_dwords    : SDNode<"SystemZISD::PERMUTE_DWORDS",

 def imm32bottom6set : PatLeaf<(i32 imm), [{
   return (N->getZExtValue() & 0x3f) == 0x3f;
 }]>;
 class shiftop<SDPatternOperator operator>
   : PatFrags<(ops node:$val, node:$count),
              [(operator node:$val, node:$count),
               (operator node:$val, (and node:$count, imm32bottom6set))]>;
 
-// Vector representation of all-zeros and all-ones.
-def z_vzero : PatFrag<(ops), (bitconvert (v16i8 (z_byte_mask (i32 0))))>;
-def z_vones : PatFrag<(ops), (bitconvert (v16i8 (z_byte_mask (i32 65535))))>;
-
 // Load a scalar and replicate it in all elements of a vector.
 class z_replicate_load<ValueType scalartype, SDPatternOperator load>
   : PatFrag<(ops node:$addr),
             (z_replicate (scalartype (load node:$addr)))>;
 def z_replicate_loadi8  : z_replicate_load<i32, anyextloadi8>;
 def z_replicate_loadi16 : z_replicate_load<i32, anyextloadi16>;
 def z_replicate_loadi32 : z_replicate_load<i32, load>;
 def z_replicate_loadi64 : z_replicate_load<i64, load>;

 def z_vlei64 : z_vle<i64, load>;
 def z_vlef32 : z_vle<f32, load>;
 def z_vlef64 : z_vle<f64, load>;
 
 // Load a scalar and insert it into the low element of the high i64 of a
 // zeroed vector.
 class z_vllez<ValueType scalartype, SDPatternOperator load, int index>
   : PatFrag<(ops node:$addr),
-            (z_vector_insert (z_vzero),
+            (z_vector_insert (immAllZerosV),
                              (scalartype (load node:$addr)), (i32 index))>;
 def z_vllezi8  : z_vllez<i32, anyextloadi8, 7>;
 def z_vllezi16 : z_vllez<i32, anyextloadi16, 3>;
 def z_vllezi32 : z_vllez<i32, load, 1>;
 def z_vllezi64 : PatFrags<(ops node:$addr),
-                          [(z_vector_insert (z_vzero),
+                          [(z_vector_insert (immAllZerosV),
                                             (i64 (load node:$addr)), (i32 0)),
                            (z_join_dwords (i64 (load node:$addr)), (i64 0))]>;
 // We use high merges to form a v4f32 from four f32s.  Propagating zero
 // into all elements but index 1 gives this expression.
 def z_vllezf32 : PatFrag<(ops node:$addr),
                          (z_merge_high
                           (v2i64
                            (z_unpackl_high
                             (v4i32
                              (bitconvert
                               (v4f32 (scalar_to_vector
                                       (f32 (load node:$addr)))))))),
-                          (v2i64 (z_vzero)))>;
+                          (v2i64
+                           (bitconvert (v4f32 (immAllZerosV)))))>;
 def z_vllezf64 : PatFrag<(ops node:$addr),
                          (z_merge_high
                           (v2f64 (scalar_to_vector (f64 (load node:$addr)))),
-                          (z_vzero))>;
+                          (immAllZerosV))>;
 
 // Similarly for the high element of a zeroed vector.
 def z_vllezli32 : z_vllez<i32, load, 0>;
 def z_vllezlf32 : PatFrag<(ops node:$addr),
                           (z_merge_high
                            (v2i64
                             (bitconvert
                              (z_merge_high
                               (v4f32 (scalar_to_vector
                                       (f32 (load node:$addr)))),
-                              (v4f32 (z_vzero))))),
-                           (v2i64 (z_vzero)))>;
+                              (v4f32 (immAllZerosV))))),
+                           (v2i64
+                            (bitconvert (v4f32 (immAllZerosV)))))>;
 
 // Store one element of a vector.
 class z_vste<ValueType scalartype, SDPatternOperator store>
   : PatFrag<(ops node:$vec, node:$addr, node:$index),
             (store (scalartype (z_vector_extract node:$vec, node:$index)),
                    node:$addr)>;
 def z_vstei8  : z_vste<i32, truncstorei8>;
 def z_vstei16 : z_vste<i32, truncstorei16>;
 def z_vstei32 : z_vste<i32, store>;
 def z_vstei64 : z_vste<i64, store>;
 def z_vstef32 : z_vste<f32, store>;
 def z_vstef64 : z_vste<f64, store>;
 
 // Arithmetic negation on vectors.
-def z_vneg : PatFrag<(ops node:$x), (sub (z_vzero), node:$x)>;
+def z_vneg : PatFrag<(ops node:$x), (sub (immAllZerosV), node:$x)>;
 
 // Bitwise negation on vectors.
-def z_vnot : PatFrag<(ops node:$x), (xor node:$x, (z_vones))>;
+def z_vnot : PatFrag<(ops node:$x), (xor node:$x, (immAllOnesV))>;
 
 // Signed "integer greater than zero" on vectors.
-def z_vicmph_zero : PatFrag<(ops node:$x), (z_vicmph node:$x, (z_vzero))>;
+def z_vicmph_zero : PatFrag<(ops node:$x), (z_vicmph node:$x, (immAllZerosV))>;
 
 // Signed "integer less than zero" on vectors.
-def z_vicmpl_zero : PatFrag<(ops node:$x), (z_vicmph (z_vzero), node:$x)>;
+def z_vicmpl_zero : PatFrag<(ops node:$x), (z_vicmph (immAllZerosV), node:$x)>;
 
 // Integer absolute on vectors.
 class z_viabs<int shift>
   : PatFrag<(ops node:$src),
             (xor (add node:$src, (z_vsra_by_scalar node:$src, (i32 shift))),
                  (z_vsra_by_scalar node:$src, (i32 shift)))>;
 def z_viabs8  : z_viabs<7>;
 def z_viabs16 : z_viabs<15>;

test/CodeGen/SystemZ/buildvector-00.ll

+; Test that the dag combiner can understand that some vector operands are
+; all-zeros and then optimize the logical operations.
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 | FileCheck %s
+
+define void @f1() {
+; CHECK-LABEL: f1:
+; CHECK: vno
+; CHECK-NOT: vno
+
+bb:
+  %tmp = shufflevector <2 x i64> undef, <2 x i64> undef, <2 x i32> zeroinitializer
+  br label %bb1
+
+bb1:                                              ; preds = %bb
+  %tmp2 = load i64, i64* undef, align 8
+  %tmp3 = insertelement <2 x i64> undef, i64 %tmp2, i32 1
+  %tmp4 = icmp ne <2 x i64> undef, zeroinitializer
+  %tmp5 = xor <2 x i1> %tmp4, zeroinitializer
+  %tmp6 = xor <2 x i1> zeroinitializer, %tmp5
+  %tmp7 = and <2 x i64> %tmp3, %tmp
+  %tmp8 = icmp ne <2 x i64> %tmp7, zeroinitializer
+  %tmp9 = xor <2 x i1> zeroinitializer, %tmp8
+  %tmp10 = icmp ne <2 x i64> undef, zeroinitializer
+  %tmp11 = xor <2 x i1> %tmp10, %tmp9
+  %tmp12 = and <2 x i1> %tmp6, %tmp11
+  %tmp13 = extractelement <2 x i1> %tmp12, i32 0
+  br i1 %tmp13, label %bb14, label %bb15
+
+bb14:                                             ; preds = %bb1
+  store i64 undef, i64* undef, align 8
+  br label %bb15
+
+bb15:                                             ; preds = %bb14, %bb1
+  unreachable
+}

test/CodeGen/SystemZ/vec-const-05.ll

-; Test vector byte masks, v4f32 version.
+; Test vector byte masks, v4f32 version. Only all-zero vectors are handled.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 | FileCheck %s
 
 ; Test an all-zeros vector.
-define <4 x float> @f1() {
-; CHECK-LABEL: f1:
+define <4 x float> @f0() {
+; CHECK-LABEL: f0:
 ; CHECK: vgbm %v24, 0
 ; CHECK: br %r14
   ret <4 x float> zeroinitializer
 }
 
-; Test an all-ones vector.
-define <4 x float> @f2() {
-; CHECK-LABEL: f2:
-; CHECK: vgbm %v24, 65535
-; CHECK: br %r14
-  ret <4 x float> <float 0xffffffffe0000000, float 0xffffffffe0000000,
-                   float 0xffffffffe0000000, float 0xffffffffe0000000>
-}
-
-; Test a mixed vector (mask 0xc731).
-define <4 x float> @f3() {
-; CHECK-LABEL: f3:
-; CHECK: vgbm %v24, 50993
-; CHECK: br %r14
-  ret <4 x float> <float 0xffffe00000000000, float 0x381fffffe0000000,
-                   float 0x379fffe000000000, float 0x371fe00000000000>
-}
-
-; Test that undefs are treated as zero (mask 0xc031).
-define <4 x float> @f4() {
-; CHECK-LABEL: f4:
-; CHECK: vgbm %v24, 49201
-; CHECK: br %r14
-  ret <4 x float> <float 0xffffe00000000000, float undef,
-                   float 0x379fffe000000000, float 0x371fe00000000000>
-}
-
-; Test that we don't use VGBM if one of the bytes is not 0 or 0xff.
-define <4 x float> @f5() {
-; CHECK-LABEL: f5:
-; CHECK-NOT: vgbm
+; Test that undefs are treated as zero.
+define <4 x float> @f1() {
+; CHECK-LABEL: f1:
+; CHECK: vgbm %v24, 0
 ; CHECK: br %r14
-  ret <4 x float> <float 0xffffe00000000000, float 0x381fffffc0000000,
-                   float 0x379fffe000000000, float 0x371fe00000000000>
+  ret <4 x float> <float zeroinitializer, float undef,
+                   float zeroinitializer, float undef>
 }
 
 ; Test an all-zeros v2f32 that gets promoted to v4f32.
-define <2 x float> @f6() {
-; CHECK-LABEL: f6:
+define <2 x float> @f2() {
+; CHECK-LABEL: f2:
 ; CHECK: vgbm %v24, 0
 ; CHECK: br %r14
   ret <2 x float> zeroinitializer
 }
-
-; Test a mixed v2f32 that gets promoted to v4f32 (mask 0xc700).
-define <2 x float> @f7() {
-; CHECK-LABEL: f7:
-; CHECK: vgbm %v24, 50944
-; CHECK: br %r14
-  ret <2 x float> <float 0xffffe00000000000, float 0x381fffffe0000000>
-}

test/CodeGen/SystemZ/vec-const-06.ll

-; Test vector byte masks, v2f64 version.
+; Test vector byte masks, v2f64 version. Only all-zero vectors are handled.
 ;
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 | FileCheck %s
 
 ; Test an all-zeros vector.
-define <2 x double> @f1() {
-; CHECK-LABEL: f1:
+define <2 x double> @f0() {
+; CHECK-LABEL: f0:
 ; CHECK: vgbm %v24, 0
 ; CHECK: br %r14
   ret <2 x double> zeroinitializer
 }
 
-; Test an all-ones vector.
-define <2 x double> @f2() {
-; CHECK-LABEL: f2:
-; CHECK: vgbm %v24, 65535
-; CHECK: br %r14
-  ret <2 x double> <double 0xffffffffffffffff, double 0xffffffffffffffff>
-}
-
-; Test a mixed vector (mask 0x8c76).
-define <2 x double> @f3() {
-; CHECK-LABEL: f3:
-; CHECK: vgbm %v24, 35958
-; CHECK: br %r14
-  ret <2 x double> <double 0xff000000ffff0000, double 0x00ffffff00ffff00>
-}
-
-; Test that undefs are treated as zero (mask 0x8c00).
-define <2 x double> @f4() {
-; CHECK-LABEL: f4:
-; CHECK: vgbm %v24, 35840
-; CHECK: br %r14
-  ret <2 x double> <double 0xff000000ffff0000, double undef>
-}
-
-; Test that we don't use VGBM if one of the bytes is not 0 or 0xff.
-define <2 x double> @f5() {
-; CHECK-LABEL: f5:
-; CHECK-NOT: vgbm
+; Test that undefs are treated as zero.
+define <2 x double> @f1() {
+; CHECK-LABEL: f1:
+; CHECK: vgbm %v24, 0
 ; CHECK: br %r14
-  ret <2 x double> <double 0xfe000000ffff0000, double 0x00ffffff00ffff00>
+  ret <2 x double> <double zeroinitializer, double undef>
 }