[SystemZ] Use VREP for storing replicated regs/immediates.

[jonpa]

Summary

When a replicated register / immediate is immediately stored, it is better to use a Vector Replicate rather than a scalar multipliation (by e.g. 0x0101), or two immediate loads into a GPR.

This patch transforms such stores in SystemZ::combineSTORE() before type legalization.

I also tried doing this after legalization, but that seems to be more work without any benefit (actually the i128 case is better handled after splitting). If the types have to be legal, the right vector type have to be produced, with an extracted element. If that element is i16 (not uncommon), an i32 first needs to be extracted and the the store must truncate itself. This is all taken care of by the type legalizer if it is transformed before it.

What's more, the zero-extend node which the patch depends on in order to be sure that the multiply produces a replicated word is easy to detect on the initial DAG, but it is removed later by DAGCombine. And computeKnownBits() do not necessarily work (at least not on the i16 it seems).

I also don't think the DAGCombiner / legalizer will produce these patterns so if there is no other argument, I think it is probably simplest to do this with the inital DAG..?

• Maybe detect the immediate splat with SystemZVectorConstantInfo, and perhaps also see if there are other immediates to be built/stored instead of via GPRs?

• I tried avoiding the extra LAY:s but it did not seem to be better on benchmarks - the scalar multiply is slower than the LAY so it should always be better anyway, right?

• OK to do before legalize only, or continue work for Combine2?

SPEC:

vsteg          :                 5918                 6372     +454
stg            :               370142               369696     -446
vrepif         :                 7729                 8077     +348
llihl          :                 7265                 6940     -325
oill           :                18574                18254     -320
vsteh          :                 2557                 2875     +318
vstef          :                 5796                 6105     +309
vlrepb         :                  207                  509     +302
llc            :                39072                38771     -301
sth            :                25741                25463     -278
mhi            :                 6070                 5802     -268
lay            :                55017                55271     +254
st             :               127451               127280     -171
msfi           :                 7106                 6974     -132
sty            :                 3627                 3514     -113
iilf           :                 6305                 6212      -93
vrepih         :                 1133                 1211      +78
vreph          :                  191                  259      +68
vlvgp          :                 8511                 8576      +65
...
OPCDIFFS: -251
...
Spill|Reload   :               607848               607780      -68
Copies         :               995492               995481      -11

Some improvements on benchmarks on z14, but more neutral on z15.

[uweigand]

I agree that this looks like the right thing to do. However, this really should use the SystemZVectorConstantInfo mechanism. (You'll probably have to add an APInt constructor there in addition to the APFloat one - currently this is only used for floating-point and vector constants, but now we also need it for integer constants. But that should be straightforward.)

[jonpa]

Patch updated to use SystemZVectorConstantInfo to detect the constant splats in combineSTORE().

If we were to only handle replicated constants / registers, it seems quite straightforward to simply build a splatVector in combineSTORE() and be done.

There are however also other constants that SystemZVectorConstantInfo can handle (with VGBM / VGM), which I have experimented with. The handling of stores of constants is in this version done instead in SystemZISelDAGToDAG.cpp, where Select() detects the case and loadVectorConstant() builds the vector with a vector element extraction. The experimental option '-replicate-only' skips generation of the non-REPLICATE opcodes, which should then show the value of being able to generate any vector constant.

trunk <> patch "replicate only":

This is the important part of the patch I think, as it eliminates the scalar multiplies (common case seems to be load-replicate-store):

vlrepb         :                  187                  489     +302
llc            :                39073                38772     -301
mhi            :                 6119                 5851     -268
msfi           :                 7066                 6934     -132
msgrkc         :                 7116                 7071      -45
msgc           :                  407                  403       -4

Generally it looks like a good change:

stef           :                 5750                 6386     +636
vsteg          :                 5912                 6475     +563
stg            :               373052               372504     -548
st             :               123399               122875     -524
vrepif         :                 7708                 8127     +419
llihl          :                 7344                 7022     -322
oill           :                18487                18169     -318
vsteh          :                 2557                 2875     +318
iilf           :                 5388                 5108     -280
sth            :                25810                25532     -278
lay            :                54928                55179     +251
vrepih         :                 1131                 1374     +243
sty            :                 3642                 3527     -115
oilf           :                 9138                 9048      -90
llihf          :                11328                11238      -90
OPCDIFFS: -534

Spill|Reload   :               613545               613472      -73
Copies         :              1005714              1005692      -22

patch "replicate only" <> patch "with vgbm/vgm":

There seems to be many cases where this could be enabled:

vsteg          :                 6475                 8912    +2437
stg            :               372504               370085    -2419
vstef          :                 6386                 8551    +2165
st             :               122875               121122    -1753
vgmf           :                13118                14454    +1336
vgmg           :                 8171                 9496    +1325
llilh          :                 6887                 5763    -1124
llihh          :                 2516                 1662     -854
llihl          :                 7022                 6564     -458
lg             :               986019               986363     +344
l              :               179157               179407     +250
stfh           :                21086                20837     -249
vgbm           :                10985                11189     +204
oilh           :                 1533                 1341     -192
llihf          :                11238                11063     -175
oilf           :                 9048                 8909     -139
iilf           :                 5108                 4974     -134
iihf           :                 4940                 4837     -103
cg             :                32046                32139      +93
OPCDIFFS: 554

Spill|Reload   :               613472               614178     +706
Copies         :              1005692              1005667      -25

The increase in spill/reload instructions seems to be due to one file with some +816 of them. I checked and have so far learned that it was not more vector register spills due to the constants being loaded into them, and it also seems that there are many more reloads, but not spills. This file is quite big, so it may be that the change is not that much relatively greater than in other smaller files.

One advantage with handling this in Select() (or rather after legalize-types), is that i128 constants now actually get handled - see @fun_128i. Not sure if that is of any real interest...

I agree that this looks like the right thing to do. However, this really should use the SystemZVectorConstantInfo mechanism. (You'll probably have to add an APInt constructor there in addition to the APFloat one - currently this is only used for floating-point and vector constants, but now we also need it for integer constants. But that should be straightforward.)

Per above we could either just handle the replication of regs/constants in combineSTORE(), or if we would like to be able to use vgbm/vgm also for storing integer constants we can extend also Select(), but perhaps also try again to figure out why one file gets more reloading (f510.parest_r/build/vectors.s)...

[jonpa]

This patch includes some experimental parts, which have been removed here: https://reviews.llvm.org/D122105.

Please look there during further review, instead of at this post...

[jonpa]

Part of eaa7803.

llvm/lib/Target/SystemZ/SystemZISelDAGToDAG.cpp

class SystemZDAGToDAGISel : public SelectionDAGISel {
   //   (Opcode UpperVal LowerVal)
   //
   // If Op0 is nonnull, then Node can be implemented using:
   //
   //   (Opcode (Opcode Op0 UpperVal) LowerVal)
   void splitLargeImmediate(unsigned Opcode, SDNode *Node, SDValue Op0,
                            uint64_t UpperVal, uint64_t LowerVal);
 
   void loadVectorConstant(const SystemZVectorConstantInfo &VCI,
-                          SDNode *Node);
+                          SDNode *Node, EVT VT);
 
   // Try to use gather instruction Opcode to implement vector insertion N.
   bool tryGather(SDNode *N, unsigned Opcode);
 
   // Try to use scatter instruction Opcode to implement store Store.
   bool tryScatter(StoreSDNode *Store, unsigned Opcode);
 
   // Change a chain of {load; op; store} of the same value into a simple op
void SystemZDAGToDAGISel::splitLargeImmediate(unsigned Opcode, SDNode *Node,
   SDValue Or = CurDAG->getNode(Opcode, DL, VT, Upper, Lower);
 
   ReplaceNode(Node, Or.getNode());
 
   SelectCode(Or.getNode());
 }
 
 void SystemZDAGToDAGISel::loadVectorConstant(
-    const SystemZVectorConstantInfo &VCI, SDNode *Node) {
+    const SystemZVectorConstantInfo &VCI, SDNode *Node, EVT VT) {
   assert((VCI.Opcode == SystemZISD::BYTE_MASK ||
           VCI.Opcode == SystemZISD::REPLICATE ||
           VCI.Opcode == SystemZISD::ROTATE_MASK) &&
          "Bad opcode!");
   assert(VCI.VecVT.getSizeInBits() == 128 && "Expected a vector type");
-  EVT VT = Node->getValueType(0);
   SDLoc DL(Node);
   SmallVector<SDValue, 2> Ops;
   for (unsigned OpVal : VCI.OpVals)
     Ops.push_back(CurDAG->getTargetConstant(OpVal, DL, MVT::i32));
   SDValue Op = CurDAG->getNode(VCI.Opcode, DL, VCI.VecVT, Ops);
 
   if (VCI.VecVT == VT.getSimpleVT())
     ReplaceNode(Node, Op.getNode());
   else if (VT.getSizeInBits() == 128) {
     SDValue BitCast = CurDAG->getNode(ISD::BITCAST, DL, VT, Op);
     ReplaceNode(Node, BitCast.getNode());
     SelectCode(BitCast.getNode());
-  } else { // float or double
+  } else if (VT.isFloatingPoint()) {
     unsigned SubRegIdx =
         (VT.getSizeInBits() == 32 ? SystemZ::subreg_h32 : SystemZ::subreg_h64);
     ReplaceNode(
         Node, CurDAG->getTargetExtractSubreg(SubRegIdx, DL, VT, Op).getNode());
+  } else {
+    unsigned NumBytes = VT.getStoreSize();
+    assert((NumBytes == 4 || NumBytes == 8) && "Unexpected vector element size");
+    EVT VecVT = EVT::getVectorVT(*CurDAG->getContext(), VT,
+                                 SystemZ::VectorBytes / NumBytes);
+    SDValue BitCast = CurDAG->getBitcast(VecVT, Op);
+    SDValue ValueToUse = CurDAG->getNode(ISD::EXTRACT_VECTOR_ELT, DL, VT,
+                                 BitCast, CurDAG->getVectorIdxConstant(0, DL));
+    ReplaceNode(Node, ValueToUse.getNode());
   }
   SelectCode(Op.getNode());
 }
 
 bool SystemZDAGToDAGISel::tryGather(SDNode *N, unsigned Opcode) {
   SDValue ElemV = N->getOperand(2);
   auto *ElemN = dyn_cast<ConstantSDNode>(ElemV);
   if (!ElemN)
if (GlobalAddressSDNode *GA =
       const DataLayout &DL = GV->getParent()->getDataLayout();
       if (GV->getPointerAlignment(DL).value() < StoreSize)
         return false;
     }
 
   return true;
 }
 
+// EXPERIMENTAL
+static cl::opt<bool> REPLICATE_ONLY("replicate-only", cl::init(false), cl::Hidden);
+
 void SystemZDAGToDAGISel::Select(SDNode *Node) {
   // If we have a custom node, we already have selected!
   if (Node->isMachineOpcode()) {
     LLVM_DEBUG(errs() << "== "; Node->dump(CurDAG); errs() << "\n");
     Node->setNodeId(-1);
     return;
   }
 
case ISD::INSERT_VECTOR_ELT: {
     }
     break;
   }
 
   case ISD::BUILD_VECTOR: {
     auto *BVN = cast<BuildVectorSDNode>(Node);
     SystemZVectorConstantInfo VCI(BVN);
     if (VCI.isVectorConstantLegal(*Subtarget)) {
-      loadVectorConstant(VCI, Node);
+      loadVectorConstant(VCI, Node, Node->getValueType(0));
       return;
     }
     break;
   }
 
   case ISD::ConstantFP: {
     APFloat Imm = cast<ConstantFPSDNode>(Node)->getValueAPF();
     if (Imm.isZero() || Imm.isNegZero())
       break;
     SystemZVectorConstantInfo VCI(Imm);
     bool Success = VCI.isVectorConstantLegal(*Subtarget); (void)Success;
     assert(Success && "Expected legal FP immediate");
-    loadVectorConstant(VCI, Node);
+    loadVectorConstant(VCI, Node, Node->getValueType(0));
     return;
   }
 
   case ISD::STORE: {
     if (tryFoldLoadStoreIntoMemOperand(Node))
       return;
     auto *Store = cast<StoreSDNode>(Node);
+    auto &Op1 = Node->getOperand(1);
     unsigned ElemBitSize = Store->getValue().getValueSizeInBits();
     if (ElemBitSize == 32) {
       if (tryScatter(Store, SystemZ::VSCEF))
         return;
     } else if (ElemBitSize == 64) {
       if (tryScatter(Store, SystemZ::VSCEG))
         return;
     }
+    if (auto *C = dyn_cast<ConstantSDNode>(Op1)) {
+      EVT MemVT = Store->getMemoryVT();
+      unsigned NumMemBytes = MemVT.getStoreSize();
+      if (C->getAPIntValue().getBitWidth() <= 64 &&
+          !isInt<16>(C->getSExtValue()) && !C->isAllOnes() && NumMemBytes > 2) {
+        SmallVector<SDNode *, 4> Stores;
+        for (auto *U : C->uses())
+          if (StoreSDNode *ST = dyn_cast<StoreSDNode>(U))
+            Stores.push_back(ST);
+        if (Stores.size() == C->use_size()) {
+          SystemZVectorConstantInfo VCI(C->getZExtValue(), NumMemBytes * 8);
+          if (VCI.isVectorConstantLegal(*Subtarget) &&
+              (VCI.Opcode == SystemZISD::REPLICATE || !REPLICATE_ONLY)) {
+            loadVectorConstant(VCI, Op1.getNode(), MemVT);
+            // Need to select all stores into VSTE before the bitcast is removed.
+            for (auto *STNode : Stores)
+              SelectCode(STNode);
+            auto &StoredVal = Node->getOperand(0);
+            if (StoredVal->getOpcode() == ISD::BITCAST)
+              SelectCode(StoredVal.getNode());
+            return;
+          }
+        }
+      }
+    }
     break;
   }
   }
 
   SelectCode(Node);
 }
 
 bool SystemZDAGToDAGISel::

llvm/lib/Target/SystemZ/SystemZISelLowering.h

 
 struct SystemZVectorConstantInfo {
 private:
   APInt IntBits;             // The 128 bits as an integer.
   APInt SplatBits;           // Smallest splat value.
   APInt SplatUndef;          // Bits correspoding to undef operands of the BVN.
   unsigned SplatBitSize = 0;
   bool isFP128 = false;
-
+  void findSplat();
 public:
   unsigned Opcode = 0;
   SmallVector<unsigned, 2> OpVals;
   MVT VecVT;
   SystemZVectorConstantInfo(APFloat FPImm);
+  SystemZVectorConstantInfo(uint64_t Imm, unsigned WordBits);
   SystemZVectorConstantInfo(BuildVectorSDNode *BVN);
   bool isVectorConstantLegal(const SystemZSubtarget &Subtarget);
 };
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/SystemZ/SystemZISelLowering.cpp

bool SystemZVectorConstantInfo::isVectorConstantLegal(
 
   // Now try assuming that any undefined bits between the first and
   // last defined set bits are set.  This increases the chances of
   // using a non-wraparound mask.
   uint64_t Middle = SplatUndefZ & ~Upper & ~Lower;
   return tryValue(SplatBitsZ | Middle);
 }
 
-SystemZVectorConstantInfo::SystemZVectorConstantInfo(APFloat FPImm) {
-  IntBits = FPImm.bitcastToAPInt().zextOrSelf(128);
-  isFP128 = (&FPImm.getSemantics() == &APFloat::IEEEquad());
-  SplatBits = FPImm.bitcastToAPInt();
-  unsigned Width = SplatBits.getBitWidth();
-  IntBits <<= (SystemZ::VectorBits - Width);
-
+void SystemZVectorConstantInfo::findSplat()  {
   // Find the smallest splat.
+  unsigned Width = SplatBits.getBitWidth();
   while (Width > 8) {
     unsigned HalfSize = Width / 2;
     APInt HighValue = SplatBits.lshr(HalfSize).trunc(HalfSize);
     APInt LowValue = SplatBits.trunc(HalfSize);
 
     // If the two halves do not match, stop here.
     if (HighValue != LowValue || 8 > HalfSize)
       break;
 
     SplatBits = HighValue;
     Width = HalfSize;
   }
   SplatUndef = 0;
   SplatBitSize = Width;
 }
 
+SystemZVectorConstantInfo::SystemZVectorConstantInfo(APFloat FPImm) {
+  IntBits = FPImm.bitcastToAPInt().zextOrSelf(128);
+  isFP128 = (&FPImm.getSemantics() == &APFloat::IEEEquad());
+  SplatBits = FPImm.bitcastToAPInt();
+  IntBits <<= (SystemZ::VectorBits - SplatBits.getBitWidth());
+  findSplat();
+}
+
+SystemZVectorConstantInfo::SystemZVectorConstantInfo(uint64_t Imm,
+                                                     unsigned WordBits) {
+  IntBits = APInt(128, Imm);
+  IntBits <<= (SystemZ::VectorBits - WordBits);
+  SplatBits = APInt(WordBits, Imm);
+  findSplat();
+}
+
 SystemZVectorConstantInfo::SystemZVectorConstantInfo(BuildVectorSDNode *BVN) {
   assert(BVN->isConstant() && "Expected a constant BUILD_VECTOR");
   bool HasAnyUndefs;
 
   // Get IntBits by finding the 128 bit splat.
   BVN->isConstantSplat(IntBits, SplatUndef, SplatBitSize, HasAnyUndefs, 128,
                        true);
 
if (isVectorElementSwap(ShuffleMask, Op1.getValueType())) {
       };
 
       return DAG.getMemIntrinsicNode(SystemZISD::VSTER, SDLoc(N),
                                      DAG.getVTList(MVT::Other),
                                      Ops, MemVT, SN->getMemOperand());
     }
   }
 
+  bool OnlyUsedByStores = true;
+  for (auto *U : Op1->uses()) {
+    if (StoreSDNode *ST = dyn_cast<StoreSDNode>(U)) {
+      EVT CurrMemVT = ST->getMemoryVT().getScalarType();
+      if (CurrMemVT.isRound() && CurrMemVT.getStoreSize() <= 16)
+        continue;
+    }
+    OnlyUsedByStores = false;
+    break;
+  }
+
+  // Replicate a reg or immediate with VREP instead of scalar mul / immediate
+  // load. It seems best to do this during the first DAGCombine as it is
+  // straight-forward to handle the zero-extend node in the initial DAG, and
+  // also not worry about the keeping the new MemVT legal (e.g. extracting an
+  // i16 element from a v16i8 vector).
+  if (Subtarget.hasVector() && OnlyUsedByStores &&
+      DCI.Level == BeforeLegalizeTypes) {
+    SDValue Word = SDValue();
+    EVT WordVT;
+
+    // Return a replicated word produced by MulOp. If found, return the value
+    // in Word and its type in WordVT.
+    auto FindReplicatedReg = [&](SDValue MulOp) {
+      EVT MulVT = MulOp.getValueType();
+      if (MulOp->getOpcode() == ISD::MUL &&
+          (MulVT == MVT::i16 || MulVT == MVT::i32 || MulVT == MVT::i64)) {
+        SDValue LHS = MulOp->getOperand(0);
+        if (LHS->getOpcode() == ISD::ZERO_EXTEND)
+          WordVT = LHS->getOperand(0).getValueType();
+        else if (LHS->getOpcode() == ISD::AssertZext)
+          WordVT = cast<VTSDNode>(LHS->getOperand(1))->getVT();
+        else
+          return;
+        if (auto *C = dyn_cast<ConstantSDNode>(MulOp->getOperand(1))) {
+          SystemZVectorConstantInfo VCI(C->getZExtValue(),
+                                        MulVT.getSizeInBits());
+          if (VCI.isVectorConstantLegal(Subtarget) &&
+              VCI.Opcode == SystemZISD::REPLICATE &&
+              VCI.OpVals[0] == 1 && WordVT == VCI.VecVT.getScalarType())
+            Word = DAG.getZExtOrTrunc(LHS->getOperand(0), SDLoc(SN), WordVT);
+        }
+      }
+    };
+
+    if (isa<BuildVectorSDNode>(Op1) &&
+        DAG.isSplatValue(Op1, true/*AllowUndefs*/)) {
+      if (auto *C = dyn_cast<ConstantSDNode>(Op1->getOperand(0))) {
+        if (C->getAPIntValue().getBitWidth() <= 64 &&
+            !isInt<16>(C->getSExtValue()) && !C->isAllOnes() &&
+            MemVT.getStoreSize() > 2) {
+          SystemZVectorConstantInfo VCI(C->getZExtValue(),
+                                        C->getValueType(0).getSizeInBits());
+          if (VCI.isVectorConstantLegal(Subtarget) &&
+              VCI.Opcode == SystemZISD::REPLICATE) {
+            Word = DAG.getConstant(VCI.OpVals[0], SDLoc(SN), MVT::i32);
+            WordVT = VCI.VecVT.getScalarType();
+          }
+        }
+      }
+      else
+        FindReplicatedReg(Op1->getOperand(0));
+    }
+    else
+        FindReplicatedReg(Op1);
+
+    if (Word != SDValue()) {
+      assert(MemVT.getSizeInBits() % WordVT.getSizeInBits() == 0 &&
+             "Bad type handling");
+      unsigned NumElts = MemVT.getSizeInBits() / WordVT.getSizeInBits();
+      EVT SplatVT = EVT::getVectorVT(*DAG.getContext(), WordVT, NumElts);
+      SDValue SplatVal = DAG.getSplatVector(SplatVT, SDLoc(SN), Word);
+      return DAG.getStore(SN->getChain(), SDLoc(SN), SplatVal,
+                          SN->getBasePtr(), SN->getMemOperand());
+    }
+  }
+
   return SDValue();
 }
 
 SDValue SystemZTargetLowering::combineVECTOR_SHUFFLE(
     SDNode *N, DAGCombinerInfo &DCI) const {
   SelectionDAG &DAG = DCI.DAG;
   // Combine element-swap (LOAD) into VLER
   if (ISD::isNON_EXTLoad(N->getOperand(0).getNode()) &&

llvm/test/CodeGen/SystemZ/store-replicated-vals.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+;
+; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z15 | FileCheck %s
+
+define void @fun_2x1b(i8* %Src, i16* %Dst) {
+; CHECK-LABEL: fun_2x1b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlrepb %v0, 0(%r2)
+; CHECK-NEXT:    vsteh %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ %i = load i8, i8* %Src
+ %ZE = zext i8 %i to i16
+ %Val = mul i16 %ZE, 257
+ store i16 %Val, i16* %Dst
+ ret void
+}
+
+; multiple stores of same value
+define void @fun_4x1b(i8* %Src, i32* %Dst, i32* %Dst2) {
+; CHECK-LABEL: fun_4x1b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlrepb %v0, 0(%r2)
+; CHECK-NEXT:    vstef %v0, 0(%r3), 0
+; CHECK-NEXT:    vstef %v0, 0(%r4), 0
+; CHECK-NEXT:    br %r14
+ %i = load i8, i8* %Src
+ %ZE = zext i8 %i to i32
+ %Val = mul i32 %ZE, 16843009
+ store i32 %Val, i32* %Dst
+ store i32 %Val, i32* %Dst2
+ ret void
+}
+
+define void @fun_8x1b(i8* %Src, i64* %Dst) {
+; CHECK-LABEL: fun_8x1b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlrepb %v0, 0(%r2)
+; CHECK-NEXT:    vsteg %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ %i = load i8, i8* %Src
+ %ZE = zext i8 %i to i64
+ %Val = mul i64 %ZE, 72340172838076673
+ store i64 %Val, i64* %Dst
+ ret void
+}
+
+; A second truncated store of same value.
+define void @fun_8x1b_4x1b(i8* %Src, i64* %Dst, i32* %Dst2) {
+; CHECK-LABEL: fun_8x1b_4x1b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlrepb %v0, 0(%r2)
+; CHECK-NEXT:    vsteg %v0, 0(%r3), 0
+; CHECK-NEXT:    vstef %v0, 0(%r4), 0
+; CHECK-NEXT:    br %r14
+ %i = load i8, i8* %Src
+ %ZE = zext i8 %i to i64
+ %Val = mul i64 %ZE, 72340172838076673
+ store i64 %Val, i64* %Dst
+ %TrVal = trunc i64 %Val to i32
+ store i32 %TrVal, i32* %Dst2
+ ret void
+}
+
+define void @fun_2x2b(i16* %Src, i32* %Dst) {
+; CHECK-LABEL: fun_2x2b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlreph %v0, 0(%r2)
+; CHECK-NEXT:    vstef %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ %i = load i16, i16* %Src
+ %ZE = zext i16 %i to i32
+ %Val = mul i32 %ZE, 65537
+ store i32 %Val, i32* %Dst
+ ret void
+}
+
+define void @fun_4x2b(i16* %Src, i64* %Dst) {
+; CHECK-LABEL: fun_4x2b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlreph %v0, 0(%r2)
+; CHECK-NEXT:    vsteg %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ %i = load i16, i16* %Src
+ %ZE = zext i16 %i to i64
+ %Val = mul i64 %ZE, 281479271743489
+ store i64 %Val, i64* %Dst
+ ret void
+}
+
+define void @fun_2x4b(i32* %Src, i64* %Dst) {
+; CHECK-LABEL: fun_2x4b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlrepf %v0, 0(%r2)
+; CHECK-NEXT:    vsteg %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ %i = load i32, i32* %Src
+ %ZE = zext i32 %i to i64
+ %Val = mul i64 %ZE, 4294967297
+ store i64 %Val, i64* %Dst
+ ret void
+}
+
+; Multiple stores of a replicated byte
+define void @fun_2x8x1b(i8* %Src, <2 x i64>* %Dst, <2 x i64>* %Dst2) {
+; CHECK-LABEL: fun_2x8x1b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlrepb %v0, 0(%r2)
+; CHECK-NEXT:    vst %v0, 0(%r3), 3
+; CHECK-NEXT:    vst %v0, 0(%r4), 3
+; CHECK-NEXT:    br %r14
+ %i = load i8, i8* %Src
+ %ZE = zext i8 %i to i64
+ %Mul = mul i64 %ZE, 72340172838076673
+ %tmp = insertelement <2 x i64> undef, i64 %Mul, i32 0
+ %Val = shufflevector <2 x i64> %tmp, <2 x i64> undef, <2 x i32> zeroinitializer
+ store <2 x i64> %Val, <2 x i64>* %Dst
+ store <2 x i64> %Val, <2 x i64>* %Dst2
+ ret void
+}
+
+define void @fun_4x2x2b(i16* %Src, <4 x i32>* %Dst) {
+; CHECK-LABEL: fun_4x2x2b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlreph %v0, 0(%r2)
+; CHECK-NEXT:    vst %v0, 0(%r3), 3
+; CHECK-NEXT:    br %r14
+ %i = load i16, i16* %Src
+ %ZE = zext i16 %i to i32
+ %Mul = mul i32 %ZE, 65537
+ %tmp = insertelement <4 x i32> undef, i32 %Mul, i32 0
+ %Val = shufflevector <4 x i32> %tmp, <4 x i32> undef, <4 x i32> zeroinitializer
+ store <4 x i32> %Val, <4 x i32>* %Dst
+ ret void
+}
+
+define void @fun_6x2x2b(i16* %Src, <6 x i32>* %Dst) {
+; CHECK-LABEL: fun_6x2x2b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlreph %v0, 0(%r2)
+; CHECK-NEXT:    vsteg %v0, 16(%r3), 0
+; CHECK-NEXT:    vst %v0, 0(%r3), 4
+; CHECK-NEXT:    br %r14
+ %i = load i16, i16* %Src
+ %ZE = zext i16 %i to i32
+ %Mul = mul i32 %ZE, 65537
+ %tmp = insertelement <6 x i32> undef, i32 %Mul, i32 0
+ %Val = shufflevector <6 x i32> %tmp, <6 x i32> undef, <6 x i32> zeroinitializer
+ store <6 x i32> %Val, <6 x i32>* %Dst
+ ret void
+}
+
+define void @fun_2x2x4b(i32* %Src, <2 x i64>* %Dst) {
+; CHECK-LABEL: fun_2x2x4b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlrepf %v0, 0(%r2)
+; CHECK-NEXT:    vst %v0, 0(%r3), 3
+; CHECK-NEXT:    br %r14
+ %i = load i32, i32* %Src
+ %ZE = zext i32 %i to i64
+ %Mul = mul i64 %ZE, 4294967297
+ %tmp = insertelement <2 x i64> undef, i64 %Mul, i32 0
+ %Val = shufflevector <2 x i64> %tmp, <2 x i64> undef, <2 x i32> zeroinitializer
+ store <2 x i64> %Val, <2 x i64>* %Dst
+ ret void
+}
+
+define void @fun_5x2x4b(i32* %Src, <5 x i64>* %Dst) {
+; CHECK-LABEL: fun_5x2x4b:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlrepf %v0, 0(%r2)
+; CHECK-NEXT:    vsteg %v0, 32(%r3), 0
+; CHECK-NEXT:    vst %v0, 16(%r3), 4
+; CHECK-NEXT:    vst %v0, 0(%r3), 4
+; CHECK-NEXT:    br %r14
+ %i = load i32, i32* %Src
+ %ZE = zext i32 %i to i64
+ %Mul = mul i64 %ZE, 4294967297
+ %tmp = insertelement <5 x i64> undef, i64 %Mul, i32 0
+ %Val = shufflevector <5 x i64> %tmp, <5 x i64> undef, <5 x i32> zeroinitializer
+ store <5 x i64> %Val, <5 x i64>* %Dst
+ ret void
+}
+
+define void @fun_8x2b_arg(i8 %Arg, i64* %Dst) {
+; CHECK-LABEL: fun_8x2b_arg:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vlvgp %v0, %r2, %r2
+; CHECK-NEXT:    vrepb %v0, %v0, 7
+; CHECK-NEXT:    vsteg %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ %ZE = zext i8 %Arg to i64
+ %Val = mul i64 %ZE, 72340172838076673
+ store i64 %Val, i64* %Dst
+ ret void
+}
+
+; A replication of a non-local value (ISD::AssertZext case).
+define void @fun_nonlocalval() {
+; CHECK-LABEL: fun_nonlocalval:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    lhi %r0, 0
+; CHECK-NEXT:    ciblh %r0, 0, 0(%r14)
+; CHECK-NEXT:  .LBB13_1: # %bb2
+; CHECK-NEXT:    llgf %r0, 0(%r1)
+; CHECK-NEXT:    vlvgp %v0, %r0, %r0
+; CHECK-NEXT:    vrepf %v0, %v0, 1
+; CHECK-NEXT:    vst %v0, 0(%r1), 3
+; CHECK-NEXT:    br %r14
+  %i = load i32, i32* undef, align 4
+  br i1 undef, label %bb2, label %bb7
+
+bb2:                                              ; preds = %bb1
+  %i3 = zext i32 %i to i64
+  %i4 = mul nuw i64 %i3, 4294967297
+  %i5 = insertelement <2 x i64> poison, i64 %i4, i64 0
+  %i6 = shufflevector <2 x i64> %i5, <2 x i64> poison, <2 x i32> zeroinitializer
+  store <2 x i64> %i6, <2 x i64>* undef, align 8
+  ret void
+
+bb7:
+  ret void
+}
+
+;; Replicated immediates
+
+; Some cases where scalar instruction is better
+define void @fun_8_1i0(i64* %Dst) {
+; CHECK-LABEL: fun_8_1i0:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    mvghi 0(%r2), 0
+; CHECK-NEXT:    br %r14
+ store i64 0, i64* %Dst
+ ret void
+}
+
+define void @fun_4_1iM1(i32* %Dst) {
+; CHECK-LABEL: fun_4_1iM1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    mvhi 0(%r2), -1
+; CHECK-NEXT:    br %r14
+ store i32 -1, i32* %Dst
+ ret void
+}
+
+define void @fun_4_1iAllOnes(i32* %Dst) {
+; CHECK-LABEL: fun_4_1iAllOnes:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    mvhi 0(%r2), -1
+; CHECK-NEXT:    br %r14
+ store i32 4294967295, i32* %Dst
+ ret void
+}
+
+define void @fun_2i(i16* %Dst) {
+; CHECK-LABEL: fun_2i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    mvhhi 0(%r2), 1
+; CHECK-NEXT:    br %r14
+ store i16 1, i16* %Dst
+ ret void
+}
+
+define void @fun_2_2i(i32* %Dst) {
+; CHECK-LABEL: fun_2_2i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepih %v0, 1
+; CHECK-NEXT:    vstef %v0, 0(%r2), 0
+; CHECK-NEXT:    br %r14
+ store i32 65537, i32* %Dst
+ ret void
+}
+
+define void @fun_4_2i(i64* %Dst) {
+; CHECK-LABEL: fun_4_2i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepih %v0, 1
+; CHECK-NEXT:    vsteg %v0, 0(%r2), 0
+; CHECK-NEXT:    br %r14
+ store i64 281479271743489, i64* %Dst
+ ret void
+}
+
+define void @fun_2_4i(i64* %Dst) {
+; CHECK-LABEL: fun_2_4i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepif %v0, 1
+; CHECK-NEXT:    vsteg %v0, 0(%r2), 0
+; CHECK-NEXT:    br %r14
+ store i64 4294967297, i64* %Dst
+ ret void
+}
+
+; Store replicated immediate twice using the same vector.
+define void @fun_4_1i(i32* %Dst, i32* %Dst2) {
+; CHECK-LABEL: fun_4_1i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepib %v0, 3
+; CHECK-NEXT:    vstef %v0, 0(%r2), 0
+; CHECK-NEXT:    vstef %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ store i32 50529027, i32* %Dst
+ store i32 50529027, i32* %Dst2
+ ret void
+}
+
+define void @fun_8_1i(i64* %Dst, i64* %Dst2) {
+; CHECK-LABEL: fun_8_1i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepib %v0, 1
+; CHECK-NEXT:    vsteg %v0, 0(%r2), 0
+; CHECK-NEXT:    vsteg %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ store i64 72340172838076673, i64* %Dst
+ store i64 72340172838076673, i64* %Dst2
+ ret void
+}
+
+; Similar, but with vectors.
+define void @fun_4_4_1i_2_4_1i(<4 x i32>* %Dst, <2 x i32>* %Dst2) {
+; CHECK-LABEL: fun_4_4_1i_2_4_1i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepib %v0, 3
+; CHECK-NEXT:    vst %v0, 0(%r2), 3
+; CHECK-NEXT:    vsteg %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ %tmp = insertelement <4 x i32> undef, i32 50529027, i32 0
+ %Val = shufflevector <4 x i32> %tmp, <4 x i32> undef, <4 x i32> zeroinitializer
+ store <4 x i32> %Val, <4 x i32>* %Dst
+ %tmp2 = insertelement <2 x i32> undef, i32 50529027, i32 0
+ %Val2 = shufflevector <2 x i32> %tmp2, <2 x i32> undef, <2 x i32> zeroinitializer
+ store <2 x i32> %Val2, <2 x i32>* %Dst2
+ ret void
+}
+
+; Same, but 64-bit store is scalar.
+define void @fun_4_4_1i_2_4_1i_scalar(<4 x i32>* %Dst, i64* %Dst2) {
+; CHECK-LABEL: fun_4_4_1i_2_4_1i_scalar:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepib %v0, 3
+; CHECK-NEXT:    vst %v0, 0(%r2), 3
+; CHECK-NEXT:    vsteg %v0, 0(%r3), 0
+; CHECK-NEXT:    br %r14
+ %tmp = insertelement <4 x i32> undef, i32 50529027, i32 0
+ %Val = shufflevector <4 x i32> %tmp, <4 x i32> undef, <4 x i32> zeroinitializer
+ store <4 x i32> %Val, <4 x i32>* %Dst
+ store i64 217020518514230019, i64* %Dst2
+ ret void
+}
+
+define void @fun_3_2_4i(<3 x i64>* %Dst) {
+; CHECK-LABEL: fun_3_2_4i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepif %v0, 1
+; CHECK-NEXT:    vsteg %v0, 16(%r2), 0
+; CHECK-NEXT:    vst %v0, 0(%r2), 4
+; CHECK-NEXT:    br %r14
+ %tmp = insertelement <3 x i64> undef, i64 4294967297, i32 0
+ %Val = shufflevector <3 x i64> %tmp, <3 x i64> undef, <3 x i32> zeroinitializer
+ store <3 x i64> %Val, <3 x i64>* %Dst
+ ret void
+}
+
+; i128 replicated '1'.
+define void @fun_128i(i128* %Dst) {
+; CHECK-LABEL: fun_128i:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vrepib %v0, 1
+; CHECK-NEXT:    vsteg %v0, 8(%r2), 0
+; CHECK-NEXT:    vsteg %v0, 0(%r2), 0
+; CHECK-NEXT:    br %r14
+ store i128 1334440654591915542993625911497130241, i128* %Dst
+ ret void
+}