Type legalizer for masked gather/scatter intrinsics

[delena]

Summary

Full type legalizer that works with all vectors length - from 2 to 16, (i32, i64, float, double).

This intrinsic, for example
void @llvm.masked.scatter.v2f32(<2 x float>%data , <2 x float*>%ptrs , i32 align , <2 x i1>%mask )
requires type widening for data and type promotion for mask.

[delena]

Ping.

[spatel]

Sorry, but I haven't looked at scatter/gather at all, so I don't think I can provide any help with this patch.

[mbodart]

../lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

503

minor typos: "ther" => "the"

And a little below, and in several other places in this file/change set: "Legalized the chain result ..." => "Legalize the chain result ..."

1235

This is the first time I've looked at vector type legalization, so there's something I don't quite understand. A masked store (PromoteIntOp_MSTORE) is in some sense just a degenerative case of a masked scatter. So intuitively it would seem PromoteIntOp_MSTORE would have a more simple implementation than PromoteIntOp_MSCATTER. But here it seems the reverse is true. Why does PromoteIntOp_MSTORE try to legalize both the mask and data simultaneously, while for scatter we just do the one operand? Could PromoteIntOp_MSTORE be simplified? If not, what is the essential difference?

../lib/CodeGen/SelectionDAG/LegalizeTypes.h

756

Can you please change the parameter name from WidenVT to NVT, so that it matches this comment and the code in LegalizeVectorTypes.cpp?

Also a typo: defalut => default

764–765

The implementation of UseExistingVal gives it an ambiguous meaning. If a widening operation is an even multiple of the original size, then the full original operand is replicated to fill out the new value. Otherwise, only its first element is replicated. Was that the intent? If so, it seems like a difficult interface to use robustly.

An enum with possible values of Zext, Splat or Undef would make the widening interface
more clear, assuming the Splat ambiguity is resolved to consistently replicate all original values.

But as a further question, when would a larger vector length not be a multiple of a smaller one?
Aren't they all powers of 2?

../lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

2704–2715

Now that ModifyToType supports zero fill, why aren't we using it here (and in WidenVecOp_MSTORE)?

3663–3664

I would think we want this moved below the check for "if (InVT == NVT)".

../lib/Target/X86/X86ISelLowering.cpp

1574–1575

Why would 512-bit gathers and scatters have different operation actions here?

19260

It's unfortunate that we have to duplicate much of the functionality of the generic type modifier in CodeGen's ModifyToType.

Is there a way to unify them?

If not, please add a source comment describing how this implementation differs from the one in CodeGen.

../test/CodeGen/X86/masked_gather_scatter.ll

2

Have you tried your changes with gathers/scatters targetting 32-bit X86?

Poking around in the X86 test area, it seems all triples use x86_64 for gather/scatter tests.

[delena]

../lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

1235

I wanted to simplify MLOAD / MSTORE but it requires additional changes in X86ISelLowering.cpp. I can do this in a separate patch.

../lib/CodeGen/SelectionDAG/LegalizeTypes.h

764–765

The implementation of UseExistingVal gives it an ambiguous meaning. If a widening operation is an even multiple of the original size, then the full original operand is replicated to fill out the new value. Otherwise, only its first element is replicated. Was that the intent? If so, it seems like a difficult interface to use robustly.

When I extend vector of indices, I want to add existing values (replicated small vector or replicated first element - does not matter). The gather/scatter will be faster, as far as I know (I'll check it again). But now I think that may be this is too X86 specific?
May be I should fill indices with "undef" and then replace these "undefs" in target specific part?

An enum with possible values of Zext, Splat or Undef would make the widening interface more clear

I thought about this, but I don't see too many enums in LLVM. All enums have global senсe, not per function.

But as a further question, when would a larger vector length not be a multiple of a smaller one?

The original type may not be power of 2. It is type legalizer, it should be able to deal with any type.
< 3 x i64 > -> < 4 x i64 >

../lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

2704–2715

I'll simplify MSTORE in a separate patch.

../lib/Target/X86/X86ISelLowering.cpp

1574–1575

The SCATTER is more problematic than GATHER. MSCATTER node returns only the Chain.
The VPSCATTER instruction in X86 zeroes mask operand and I should specify it as "return value".

19260

I simplified the X86 version. I call it ExtendToType. It works with legal types only and optimized for X86.

../test/CodeGen/X86/masked_gather_scatter.ll

2

I tried before, saw that it works. I added tests for 32-bit.

[delena]

Updated the patch according to Mitch's comments.

[mbodart]

../include/llvm/CodeGen/SelectionDAGNodes.h

2140–2142

This new assertion is fine.

But why are we removing the assertion that the mask's element type is i1?

../lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

1235

OK

../lib/CodeGen/SelectionDAG/LegalizeTypes.h

764–765

I agree that this optimization is too X86 specific, and CodeGen should simply extend with undef values.

If you want to proceed with those X86-specific extensions, please do so in a separate change set.

That will simplify this change set, and render the other comments here moot for now.
If it turns out that replication is needed, we will want to define its behavior more crisply.

../lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

2704–2715

That's fine. But then can you please add a "FIX ME" comment here indicating
such a change is desired?

2739

A simple source comment here to the effect "// Zero extend the mask" would help readers remember the meaning of "true".

2754

Another instance of "Legalized" => "Legalize".

../lib/Target/X86/X86ISelLowering.cpp

11681

Unrelated to your change, I'm finding the optimizations of MVT::i1 here confusing as there
are no checks for SubVecVT.getVectorNumElements(). The code appears to be making the
assumption that when IdxVal is 0, or half the full vector length, then SubVec is exactly half the
size of Vec. Why is that not being checked?

It's also unclear to me how VSHLI/VSHRI operate on a vector of MVT::i1 elements.
When operating on a vector of i32 or i64 elements, I would have thought these
instructions perform a bit shift of each individual element (not cross element).
But it seems the code here is trying to shift the whole i1 vector left and right,
as if it is one big integer. Maybe that's how VSHLI/VSHRI are defined to behave,
I don't know. Can you clarify?

19265

Another case where this new Undef creation should probably be moved below the InVT == NVT early return.

../test/CodeGen/X86/masked_gather_scatter.ll

140

I don't understand how SKX can use 32-bit indices here (i.e., vpgatherdd instead of vpgatherqd), when targetting x86_64. How does lowering of the @llvm.masked.gather.v8i32 calls know that only the low 32-bits of the <8 x i32*> pointer values are needed?
Is there some analysis of the insertelement;shufflevector/getelementptr instructions which feed to < 8 x i32*> pointers?

[delena]

Extending indices does not require replication. I simplified the functions that modify values to a new type.

[delena]

../lib/CodeGen/SelectionDAG/LegalizeTypes.h

764–765

I removed "Slpat". I don't need it.
"Zext" extension is not target specific, at least for the mask.

../lib/Target/X86/X86ISelLowering.cpp

11681

There are the special SHIFT instructions for mask vector KSHIFTR, KSHIFTL (in AVX-512)

When we insert v8i1 into v16i1, the index should be 8 (I'll add an assertion) or 0.
If v16i1 is allzero, it's enough to shift the input vector left-right.

../test/CodeGen/X86/masked_gather_scatter.ll

140

The base address in %rdi. (In %edi for 32-bit). In %ymm0 we have only indices. The real address of each element is "base +index*scale" -> %rdi + ymm[i] * 4

[delena]

../include/llvm/CodeGen/SelectionDAGNodes.h

2140–2142

Mask types are not always legal. When <2xi1> is illegal, the type legalizer promotes it to <2 x i64>.
I create the node with <2 x i64> mask and then handle it in X86 code.

[mbodart]

../include/llvm/CodeGen/SelectionDAGNodes.h

2140–2142

Allowing non-i1 element types, even temporarily, seems like it breaks the definition of MGATHER/MSCATTER. Do we need to update the LLVM documentation to allow simd masks here? Or is it the intent to require i1 masks throughout LLVM IR, and only allow simd masks while we perform type legalization during during DAG selection? If the latter, that still seems a bit risky as type legalization can use utilities in the non-target-specific parts of CodeGen, and they would not like to see a non-i1 mask element. That's just paranoid speculation on my part. I don't know whether DAG selection is in general allowed to take such liberties.

A related question: Why isn't <2xi1> legalized to a full mask register size (<64xi1>, <32xi1>, <16xi1> or <8xi1>) instead of <2xi64>?
Is this just inherent in type legalization of Nxi1 vectors, or is it needed for an AVX2 simd mask?

../test/CodeGen/X86/masked_gather_scatter.ll

140

OK, thanks.

[delena]

I added a function WidenTargetBoolean() for mask legalization in all masked operations load/store/gather/scatter.

[mbodart]

Hi Elena,

I added just a few more comments in line.

regards,

• mitch

../include/llvm/CodeGen/SelectionDAGNodes.h

2136

Not your change, but minor typo: PathThru => PassThru

../lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

506

Is there any particular reason that the mask operand legalization is handled in a different place for MLOAD and MGATHER? For MLOAD it is processed during PromoteIntRes_MLOAD, while for MGATHER it is processed when legalizing its operands. The MGATHER method seems cleaner, but I'm curious as to why MLOAD does it differently. Also note that for MLOAD, the call to PromoteTargetBoolean is conditioned on whether the type mismatches, while in PromoteIntOp_MGATHER, the call is unconditional. I don't know if these inconsistencies are innocuous from a functionality standpoint, but they certainly make it difficult to understand the requirements.

1235

There appears to be an extraneous blank line here.

../lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

1115 ↗

(On Diff #41020)

It seems important here that the value returned by GetPromotedInteger matches the promotion operation (e.g.. sign extension) performed by PromoteTargetBoolean.
How is that guaranteed?

../lib/CodeGen/SelectionDAG/LegalizeTypes.h

177

Should that be "of ValVT", not "if"?

../lib/Target/X86/X86ISelLowering.cpp

19380

It seems odd to me to allow a masked store/scatter where the value element size differs from the memory element size. Is this an unavoidable result of type legalization? What are the semantics of such an operation? If the value is i64 and the memory is i32, apparently we just store the low 32 bits of each i64 element. Is that the expected behavior? Do we ever have to worry about a size difference in the opposite direction, e.g. storing an i32 to an i64?

../test/CodeGen/X86/masked_gather_scatter.ll

522–755

Please keep the last <3 x i32> all on the same line.

[delena]

Fixed WidenTargetBoolean() and added more tests to check that the mask is sign-extended.
Thanks to Mitch for catching this.

Addressed all other issues.

Sorry for adding a line to X86InstrAVX512.td, just one of the new tests triggered the failure.
I'm trying to reproduce the same failure on a standalone test to make a separate commit.

[mbodart]

../lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

1115 ↗

(On Diff #41020)

Using SExt unconditionally seems incorrect.
Don't we want to make the choice of ZExt vs SExt dependent on getBooleanContents?
Perhaps we should add getExtendForVectorContent, similar to the existing scalar getExtendForContent.

[delena]

../lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

506

No reason. Just the old code. I removed the mask promotion and all tests still pass.

1235

I did not see any blank line. I'll run dos2unix on all files again.

../lib/CodeGen/SelectionDAG/LegalizeTypes.cpp

1115 ↗

(On Diff #41020)

Fixed! Thank you.

../lib/Target/X86/X86ISelLowering.cpp

19380

This is the case of promotion of v2i32 to v2i64. Only MemVT keeps the original VT. I'm actually "redo" the TypeLegalizer's work.
The type legalizer promoted v2i32 to v2i64 and I retrieve v2i32 with shuffle {0, 2, -1, -1} and then widen the result to v4i32.
I'll add comments.

[delena]

Updated revision according to Mitch's comments.

[mbodart]

Hi Elena,

I added few more minor comments and a question, but in general it LGTM.

regards,

• mitch

../lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

3164

Redundant assert, already checked a few lines above.

../lib/Target/X86/X86ISelLowering.cpp

19435

Minor mechanics question: should we first check if Mask's element type is i1, and if so, avoid inserting the TRUNCATE?

19476

For a 32-bit target, we can gather 16 elements with a single gather.
Though the comment says 8 is the minimum number of elements in
a gather, which is true, the code also seems to be treating this as
the maximum number. How are 16-element gathers supported?

26146

It is hard to interpret this comment, and thus the correctness of this routine,
without any additional context.
Why is a mask always truncated at this point?
How do you know that it is truncated down to
the element size of the SIGN_EXTEND_INREG's operand?
Please add some supporting comments answering these kinds of questions.

../test/CodeGen/X86/masked_gather_scatter.ll

2–4

I would think we also want a test for i386 and avx512vl.

[delena]

../lib/Target/X86/X86ISelLowering.cpp

19435

getNode(ISD::TRUNCATE..) does this optimization.

19476

We always can sign-extend the indices. VGATHERQPS is supported in 32-bit mode.

26146

I added more words here:

// Gather and Scatter instructions use k-registers for masks. The type of
// the masks is v*i1. So the mask will be truncated anyway.
// The SIGN_EXTEND_INREG my be dropped.

../test/CodeGen/X86/masked_gather_scatter.ll

2–4

I added.

../include/llvm/CodeGen/SelectionDAGNodes.h

 class MaskedGatherSDNode : public MaskedGatherScatterSDNode {
 public:
   friend class SelectionDAG;
   MaskedGatherSDNode(unsigned Order, DebugLoc dl, ArrayRef<SDValue> Operands,
                      SDVTList VTs, EVT MemVT, MachineMemOperand *MMO)
     : MaskedGatherScatterSDNode(ISD::MGATHER, Order, dl, Operands, VTs, MemVT,
                                 MMO) {
     assert(getValue().getValueType() == getValueType(0) &&
            "Incompatible type of the PathThru value in MaskedGatherSDNode");
     assert(getMask().getValueType().getVectorNumElements() ==
-               getValueType(0).getVectorNumElements() &&
            "Vector width mismatch between mask and data");
-    assert(getMask().getValueType().getScalarType() == MVT::i1 &&
-           "Vector width mismatch between mask and data");
+    assert(getIndex().getValueType().getVectorNumElements() ==
+           getValueType(0).getVectorNumElements() &&
+           "Vector width mismatch between index and data");
   }
 
   static bool classof(const SDNode *N) {
     return N->getOpcode() == ISD::MGATHER;
   }
 };
 
 /// This class is used to represent an MSCATTER node
 ///
 class MaskedScatterSDNode : public MaskedGatherScatterSDNode {
 
 public:
   friend class SelectionDAG;
   MaskedScatterSDNode(unsigned Order, DebugLoc dl,ArrayRef<SDValue> Operands,
                       SDVTList VTs, EVT MemVT, MachineMemOperand *MMO)
-      : MaskedGatherScatterSDNode(ISD::MSCATTER, Order, dl, Operands, VTs,
-                                  MemVT, MMO) {
+    : MaskedGatherScatterSDNode(ISD::MSCATTER, Order, dl, Operands, VTs, MemVT,
+                                MMO) {
     assert(getMask().getValueType().getVectorNumElements() ==
-               getValue().getValueType().getVectorNumElements() &&
            "Vector width mismatch between mask and data");
-    assert(getMask().getValueType().getScalarType() == MVT::i1 &&
-           "Vector width mismatch between mask and data");
+    assert(getIndex().getValueType().getVectorNumElements() ==
+           getValue().getValueType().getVectorNumElements() &&
+           "Vector width mismatch between index and data");
   }
 
   static bool classof(const SDNode *N) {
     return N->getOpcode() == ISD::MSCATTER;
   }
 };
 
 /// An SDNode that represents everything that will be needed

../lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

case ISD::CONVERT_RNDSAT:
                          Res = PromoteIntRes_CONVERT_RNDSAT(N); break;
   case ISD::CTLZ_ZERO_UNDEF:
   case ISD::CTLZ:        Res = PromoteIntRes_CTLZ(N); break;
   case ISD::CTPOP:       Res = PromoteIntRes_CTPOP(N); break;
   case ISD::CTTZ_ZERO_UNDEF:
   case ISD::CTTZ:        Res = PromoteIntRes_CTTZ(N); break;
   case ISD::EXTRACT_VECTOR_ELT:
                          Res = PromoteIntRes_EXTRACT_VECTOR_ELT(N); break;
-  case ISD::LOAD:        Res = PromoteIntRes_LOAD(cast<LoadSDNode>(N));break;
-  case ISD::MLOAD:       Res = PromoteIntRes_MLOAD(cast<MaskedLoadSDNode>(N));break;
+  case ISD::LOAD:        Res = PromoteIntRes_LOAD(cast<LoadSDNode>(N)); break;
+  case ISD::MLOAD:       Res = PromoteIntRes_MLOAD(cast<MaskedLoadSDNode>(N));
+    break;
+  case ISD::MGATHER:     Res = PromoteIntRes_MGATHER(cast<MaskedGatherSDNode>(N)); break;
   case ISD::SELECT:      Res = PromoteIntRes_SELECT(N); break;
   case ISD::VSELECT:     Res = PromoteIntRes_VSELECT(N); break;
   case ISD::SELECT_CC:   Res = PromoteIntRes_SELECT_CC(N); break;
   case ISD::SETCC:       Res = PromoteIntRes_SETCC(N); break;
   case ISD::SMIN:
   case ISD::SMAX:
   case ISD::UMIN:
   case ISD::UMAX:        Res = PromoteIntRes_SimpleIntBinOp(N); break;
SDValue DAGTypeLegalizer::PromoteIntRes_MLOAD(MaskedLoadSDNode *N) {
   SDValue Res = DAG.getMaskedLoad(NVT, dl, N->getChain(), N->getBasePtr(),
                                   Mask, ExtSrc0, N->getMemoryVT(),
                                   N->getMemOperand(), ISD::SEXTLOAD);
   // Legalized the chain result - switch anything that used the old chain to
   // use the new one.
   ReplaceValueWith(SDValue(N, 1), Res.getValue(1));
   return Res;
 }
+
+SDValue DAGTypeLegalizer::PromoteIntRes_MGATHER(MaskedGatherSDNode *N) {
+  EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+  SDValue ExtSrc0 = GetPromotedInteger(N->getValue());
+  assert(NVT == ExtSrc0.getValueType() &&
+      "Gather result type and the passThru agrument type should be ther same");
+
+  SDLoc dl(N);
+  SDValue Ops[] = {N->getChain(), ExtSrc0, N->getMask(), N->getBasePtr(),
+                   N->getIndex()};
+  SDValue Res = DAG.getMaskedGather(DAG.getVTList(NVT, MVT::Other),
+                                    N->getMemoryVT(), dl, Ops,
+                                    N->getMemOperand()); 
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Res.getValue(1));
+  return Res;
+}
+
 /// Promote the overflow flag of an overflowing arithmetic node.
 SDValue DAGTypeLegalizer::PromoteIntRes_Overflow(SDNode *N) {
   // Simply change the return type of the boolean result.
   EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(1));
   EVT ValueVTs[] = { N->getValueType(0), NVT };
   SDValue Ops[] = { N->getOperand(0), N->getOperand(1) };
   SDValue Res = DAG.getNode(N->getOpcode(), SDLoc(N),
                             DAG.getVTList(ValueVTs), Ops);
bool DAGTypeLegalizer::PromoteIntegerOperand(SDNode *N, unsigned OpNo) {
   case ISD::SIGN_EXTEND:  Res = PromoteIntOp_SIGN_EXTEND(N); break;
   case ISD::SINT_TO_FP:   Res = PromoteIntOp_SINT_TO_FP(N); break;
   case ISD::STORE:        Res = PromoteIntOp_STORE(cast<StoreSDNode>(N),
                                                    OpNo); break;
   case ISD::MSTORE:       Res = PromoteIntOp_MSTORE(cast<MaskedStoreSDNode>(N),
                                                     OpNo); break;
   case ISD::MLOAD:        Res = PromoteIntOp_MLOAD(cast<MaskedLoadSDNode>(N),
                                                     OpNo); break;
+  case ISD::MGATHER:  Res = PromoteIntOp_MGATHER(cast<MaskedGatherSDNode>(N),
+                                                 OpNo); break;
+  case ISD::MSCATTER: Res = PromoteIntOp_MSCATTER(cast<MaskedScatterSDNode>(N),
+                                                  OpNo); break;
   case ISD::TRUNCATE:     Res = PromoteIntOp_TRUNCATE(N); break;
   case ISD::FP16_TO_FP:
   case ISD::UINT_TO_FP:   Res = PromoteIntOp_UINT_TO_FP(N); break;
   case ISD::ZERO_EXTEND:  Res = PromoteIntOp_ZERO_EXTEND(N); break;
   case ISD::EXTRACT_SUBVECTOR: Res = PromoteIntOp_EXTRACT_SUBVECTOR(N); break;
 
   case ISD::SHL:
   case ISD::SRA:
if (getTypeAction(DataVT) == TargetLowering::TypePromoteInteger) {
         Mask = PromoteTargetBoolean(Mask, DataOp.getValueType());
       TruncateStore = true;
     }
     else {
       assert(getTypeAction(DataVT) == TargetLowering::TypeWidenVector &&
              "Unexpected data legalization in MSTORE");
       DataOp = GetWidenedVector(DataOp);
 
-      if (getTypeAction(MaskVT) == TargetLowering::TypeWidenVector)
-        Mask = GetWidenedVector(Mask);
-      else {
-        EVT BoolVT = getSetCCResultType(DataOp.getValueType());
-
-        // We can't use ModifyToType() because we should fill the mask with
-        // zeroes
-        unsigned WidenNumElts = BoolVT.getVectorNumElements();
-        unsigned MaskNumElts = MaskVT.getVectorNumElements();
 
-        unsigned NumConcat = WidenNumElts / MaskNumElts;
-        SmallVector<SDValue, 16> Ops(NumConcat);
-        SDValue ZeroVal = DAG.getConstant(0, dl, MaskVT);
-        Ops[0] = Mask;
-        for (unsigned i = 1; i != NumConcat; ++i)
-          Ops[i] = ZeroVal;
 
-        Mask = DAG.getNode(ISD::CONCAT_VECTORS, dl, BoolVT, Ops);
-      }
-    }
-  }
   else
-    Mask = PromoteTargetBoolean(N->getMask(), DataOp.getValueType());
+    Mask = PromoteTargetBoolean(Mask, DataOp.getValueType());
   return DAG.getMaskedStore(N->getChain(), dl, DataOp, N->getBasePtr(), Mask,
                             N->getMemoryVT(), N->getMemOperand(),
                             TruncateStore);
 }
 
-SDValue DAGTypeLegalizer::PromoteIntOp_MLOAD(MaskedLoadSDNode *N, unsigned OpNo){
+SDValue DAGTypeLegalizer::PromoteIntOp_MLOAD(MaskedLoadSDNode *N,
+                                             unsigned OpNo) {
   assert(OpNo == 2 && "Only know how to promote the mask!");
   EVT DataVT = N->getValueType(0);
   SDValue Mask = PromoteTargetBoolean(N->getOperand(OpNo), DataVT);
   SmallVector<SDValue, 4> NewOps(N->op_begin(), N->op_end());
   NewOps[OpNo] = Mask;
   return SDValue(DAG.UpdateNodeOperands(N, NewOps), 0);
 }
 
+SDValue DAGTypeLegalizer::PromoteIntOp_MGATHER(MaskedGatherSDNode *N,
+                                               unsigned OpNo) {
+
+  SmallVector<SDValue, 5> NewOps(N->op_begin(), N->op_end());
+  NewOps[OpNo] = GetPromotedInteger(N->getOperand(OpNo));
+  return SDValue(DAG.UpdateNodeOperands(N, NewOps), 0);
+}
+
+SDValue DAGTypeLegalizer::PromoteIntOp_MSCATTER(MaskedScatterSDNode *N,
+                                                unsigned OpNo) {
+  SmallVector<SDValue, 5> NewOps(N->op_begin(), N->op_end());
+  NewOps[OpNo] = GetPromotedInteger(N->getOperand(OpNo));
+  return SDValue(DAG.UpdateNodeOperands(N, NewOps), 0);
+}
+
 SDValue DAGTypeLegalizer::PromoteIntOp_TRUNCATE(SDNode *N) {
   SDValue Op = GetPromotedInteger(N->getOperand(0));
   return DAG.getNode(ISD::TRUNCATE, SDLoc(N), N->getValueType(0), Op);
 }
 
 SDValue DAGTypeLegalizer::PromoteIntOp_UINT_TO_FP(SDNode *N) {
   return SDValue(DAG.UpdateNodeOperands(N,
                                 ZExtPromotedInteger(N->getOperand(0))), 0);

../lib/CodeGen/SelectionDAG/LegalizeTypes.h

private:
   SDValue LibCallify(RTLIB::Libcall LC, SDNode *N, bool isSigned);
 
   std::pair<SDValue, SDValue> ExpandChainLibCall(RTLIB::Libcall LC,
                                                  SDNode *Node, bool isSigned);
   std::pair<SDValue, SDValue> ExpandAtomic(SDNode *Node);
 
   SDValue PromoteTargetBoolean(SDValue Bool, EVT ValVT);
   void ReplaceValueWith(SDValue From, SDValue To);
   void SplitInteger(SDValue Op, SDValue &Lo, SDValue &Hi);
   void SplitInteger(SDValue Op, EVT LoVT, EVT HiVT,
                     SDValue &Lo, SDValue &Hi);
 
   //===--------------------------------------------------------------------===//
   // Integer Promotion Support: LegalizeIntegerTypes.cpp
   //===--------------------------------------------------------------------===//
 
   /// GetPromotedInteger - Given a processed operand Op which was promoted to a
private:
   SDValue PromoteIntRes_CTPOP(SDNode *N);
   SDValue PromoteIntRes_CTTZ(SDNode *N);
   SDValue PromoteIntRes_EXTRACT_VECTOR_ELT(SDNode *N);
   SDValue PromoteIntRes_FP_TO_XINT(SDNode *N);
   SDValue PromoteIntRes_FP_TO_FP16(SDNode *N);
   SDValue PromoteIntRes_INT_EXTEND(SDNode *N);
   SDValue PromoteIntRes_LOAD(LoadSDNode *N);
   SDValue PromoteIntRes_MLOAD(MaskedLoadSDNode *N);
+  SDValue PromoteIntRes_MGATHER(MaskedGatherSDNode *N);
   SDValue PromoteIntRes_Overflow(SDNode *N);
   SDValue PromoteIntRes_SADDSUBO(SDNode *N, unsigned ResNo);
   SDValue PromoteIntRes_SDIV(SDNode *N);
   SDValue PromoteIntRes_SELECT(SDNode *N);
   SDValue PromoteIntRes_VSELECT(SDNode *N);
   SDValue PromoteIntRes_SELECT_CC(SDNode *N);
   SDValue PromoteIntRes_SETCC(SDNode *N);
   SDValue PromoteIntRes_SHL(SDNode *N);
private:
   SDValue PromoteIntOp_SIGN_EXTEND(SDNode *N);
   SDValue PromoteIntOp_SINT_TO_FP(SDNode *N);
   SDValue PromoteIntOp_STORE(StoreSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_TRUNCATE(SDNode *N);
   SDValue PromoteIntOp_UINT_TO_FP(SDNode *N);
   SDValue PromoteIntOp_ZERO_EXTEND(SDNode *N);
   SDValue PromoteIntOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_MLOAD(MaskedLoadSDNode *N, unsigned OpNo);
+  SDValue PromoteIntOp_MSCATTER(MaskedScatterSDNode *N, unsigned OpNo);
+  SDValue PromoteIntOp_MGATHER(MaskedGatherSDNode *N, unsigned OpNo);
 
   void PromoteSetCCOperands(SDValue &LHS,SDValue &RHS, ISD::CondCode Code);
 
   //===--------------------------------------------------------------------===//
   // Integer Expansion Support: LegalizeIntegerTypes.cpp
   //===--------------------------------------------------------------------===//
 
   /// GetExpandedInteger - Given a processed operand Op which was expanded into
private:
   SDValue WidenVecRes_BITCAST(SDNode* N);
   SDValue WidenVecRes_BUILD_VECTOR(SDNode* N);
   SDValue WidenVecRes_CONCAT_VECTORS(SDNode* N);
   SDValue WidenVecRes_CONVERT_RNDSAT(SDNode* N);
   SDValue WidenVecRes_EXTRACT_SUBVECTOR(SDNode* N);
   SDValue WidenVecRes_INSERT_VECTOR_ELT(SDNode* N);
   SDValue WidenVecRes_LOAD(SDNode* N);
   SDValue WidenVecRes_MLOAD(MaskedLoadSDNode* N);
+  SDValue WidenVecRes_MGATHER(MaskedGatherSDNode* N);
   SDValue WidenVecRes_SCALAR_TO_VECTOR(SDNode* N);
   SDValue WidenVecRes_SIGN_EXTEND_INREG(SDNode* N);
   SDValue WidenVecRes_SELECT(SDNode* N);
   SDValue WidenVecRes_SELECT_CC(SDNode* N);
   SDValue WidenVecRes_SETCC(SDNode* N);
   SDValue WidenVecRes_UNDEF(SDNode *N);
   SDValue WidenVecRes_VECTOR_SHUFFLE(ShuffleVectorSDNode *N);
   SDValue WidenVecRes_VSETCC(SDNode* N);
private:
   bool WidenVectorOperand(SDNode *N, unsigned OpNo);
   SDValue WidenVecOp_BITCAST(SDNode *N);
   SDValue WidenVecOp_CONCAT_VECTORS(SDNode *N);
   SDValue WidenVecOp_EXTEND(SDNode *N);
   SDValue WidenVecOp_EXTRACT_VECTOR_ELT(SDNode *N);
   SDValue WidenVecOp_EXTRACT_SUBVECTOR(SDNode *N);
   SDValue WidenVecOp_STORE(SDNode* N);
   SDValue WidenVecOp_MSTORE(SDNode* N, unsigned OpNo);
+  SDValue WidenVecOp_MSCATTER(SDNode* N, unsigned OpNo);
   SDValue WidenVecOp_SETCC(SDNode* N);
 
   SDValue WidenVecOp_Convert(SDNode *N);
   SDValue WidenVecOp_FCOPYSIGN(SDNode *N);
 
   //===--------------------------------------------------------------------===//
   // Vector Widening Utilities Support: LegalizeVectorTypes.cpp
   //===--------------------------------------------------------------------===//
private:
 
   /// Helper genWidenVectorTruncStores - Helper function to generate a set of
   /// stores to store a truncate widen vector into non-widen memory
   ///   StChain: list of chains for the stores we have generated
   ///   ST:      store of a widen value
   void GenWidenVectorTruncStores(SmallVectorImpl<SDValue> &StChain,
                                  StoreSDNode *ST);
 
   /// Modifies a vector input (widen or narrows) to a vector of NVT.  The
   /// input vector must have the same element type as NVT.
-  SDValue ModifyToType(SDValue InOp, EVT WidenVT);
+  /// When FillWithZeroes is "on" the vector will be widened with
+  /// zeroes. UseExistingVal means that the existing elements from InOp
+  /// should be used for the widening.
+  /// By defalut, the vector will be widened with undefined values.
+  SDValue ModifyToType(SDValue InOp, EVT WidenVT, bool FillWithZeroes = false,
+                       bool UseExistingVal = false);
 
 
   //===--------------------------------------------------------------------===//
   // Generic Splitting: LegalizeTypesGeneric.cpp
   //===--------------------------------------------------------------------===//
 
   // Legalization methods which only use that the illegal type is split into two
   // not necessarily identical types.  As such they can be used for splitting
   // vectors and expanding integers and floats.
 

../lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

#endif
   case ISD::SETCC:             Res = WidenVecRes_SETCC(N); break;
   case ISD::UNDEF:             Res = WidenVecRes_UNDEF(N); break;
   case ISD::VECTOR_SHUFFLE:
     Res = WidenVecRes_VECTOR_SHUFFLE(cast<ShuffleVectorSDNode>(N));
     break;
   case ISD::MLOAD:
     Res = WidenVecRes_MLOAD(cast<MaskedLoadSDNode>(N));
     break;
+  case ISD::MGATHER:
+    Res = WidenVecRes_MGATHER(cast<MaskedGatherSDNode>(N));
+    break;
 
   case ISD::ADD:
   case ISD::AND:
   case ISD::MUL:
   case ISD::MULHS:
   case ISD::MULHU:
   case ISD::OR:
   case ISD::SUB:
SDValue DAGTypeLegalizer::WidenVecRes_MLOAD(MaskedLoadSDNode *N) {
   
   EVT WidenVT = TLI.getTypeToTransformTo(*DAG.getContext(),N->getValueType(0));
   SDValue Mask = N->getMask();
   EVT MaskVT = Mask.getValueType();
   SDValue Src0 = GetWidenedVector(N->getSrc0());
   ISD::LoadExtType ExtType = N->getExtensionType();
   SDLoc dl(N);
 
-  if (getTypeAction(MaskVT) == TargetLowering::TypeWidenVector)
-    Mask = GetWidenedVector(Mask);
-  else {
-    EVT BoolVT = getSetCCResultType(WidenVT);
 
-    // We can't use ModifyToType() because we should fill the mask with
-    // zeroes
-    unsigned WidenNumElts = BoolVT.getVectorNumElements();
-    unsigned MaskNumElts = MaskVT.getVectorNumElements();
 
-    unsigned NumConcat = WidenNumElts / MaskNumElts;
-    SmallVector<SDValue, 16> Ops(NumConcat);
-    SDValue ZeroVal = DAG.getConstant(0, dl, MaskVT);
-    Ops[0] = Mask;
-    for (unsigned i = 1; i != NumConcat; ++i)
-      Ops[i] = ZeroVal;
 
-    Mask = DAG.getNode(ISD::CONCAT_VECTORS, dl, BoolVT, Ops);
-  }
 
   SDValue Res = DAG.getMaskedLoad(WidenVT, dl, N->getChain(), N->getBasePtr(),
                                   Mask, Src0, N->getMemoryVT(),
                                   N->getMemOperand(), ExtType);
   // Legalized the chain result - switch anything that used the old chain to
   // use the new one.
   ReplaceValueWith(SDValue(N, 1), Res.getValue(1));
   return Res;
 }
 
+SDValue DAGTypeLegalizer::WidenVecRes_MGATHER(MaskedGatherSDNode *N) {
+
+  EVT WidenVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+  SDValue Mask = N->getMask();
+  MVT MaskVT = Mask.getSimpleValueType();
+  SDValue Src0 = GetWidenedVector(N->getValue());
+  SDLoc dl(N);
+
+  if (getTypeAction(MaskVT) == TargetLowering::TypePromoteInteger) {
+    Mask = GetPromotedInteger(Mask);
+    MaskVT = Mask.getSimpleValueType();
+  }
+  MVT WideMaskVT = MVT::getVectorVT(MaskVT.getScalarType(),
+                                    WidenVT.getVectorNumElements());
+  Mask = ModifyToType(Mask, WideMaskVT, true);
+
+  SDValue Index = N->getIndex();
+  if (getTypeAction(Index.getValueType()) == TargetLowering::TypePromoteInteger)
+    Index = GetPromotedInteger(Index);
+ 
+  MVT WideIndexVT =
+    MVT::getVectorVT(Index.getSimpleValueType().getScalarType(),
+                     WidenVT.getVectorNumElements());
+  Index = ModifyToType(Index, WideIndexVT, false, true);
+  SDValue Ops[] = { N->getChain(), Src0, Mask, N->getBasePtr(), Index };
+  SDValue Res = DAG.getMaskedGather(DAG.getVTList(WidenVT, MVT::Other),
+                                    N->getMemoryVT(), dl, Ops,
+                                    N->getMemOperand());
+
+  // Legalized the chain result - switch anything that used the old chain to
+  // use the new one.
+  ReplaceValueWith(SDValue(N, 1), Res.getValue(1));
+  return Res;
+}
+
 SDValue DAGTypeLegalizer::WidenVecRes_SCALAR_TO_VECTOR(SDNode *N) {
   EVT WidenVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
   return DAG.getNode(ISD::SCALAR_TO_VECTOR, SDLoc(N),
                      WidenVT, N->getOperand(0));
 }
 
 SDValue DAGTypeLegalizer::WidenVecRes_SELECT(SDNode *N) {
   EVT WidenVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
#endif
     llvm_unreachable("Do not know how to widen this operator's operand!");
 
   case ISD::BITCAST:            Res = WidenVecOp_BITCAST(N); break;
   case ISD::CONCAT_VECTORS:     Res = WidenVecOp_CONCAT_VECTORS(N); break;
   case ISD::EXTRACT_SUBVECTOR:  Res = WidenVecOp_EXTRACT_SUBVECTOR(N); break;
   case ISD::EXTRACT_VECTOR_ELT: Res = WidenVecOp_EXTRACT_VECTOR_ELT(N); break;
   case ISD::STORE:              Res = WidenVecOp_STORE(N); break;
   case ISD::MSTORE:             Res = WidenVecOp_MSTORE(N, OpNo); break;
+  case ISD::MSCATTER:           Res = WidenVecOp_MSCATTER(N, OpNo); break;
   case ISD::SETCC:              Res = WidenVecOp_SETCC(N); break;
   case ISD::FCOPYSIGN:          Res = WidenVecOp_FCOPYSIGN(N); break;
 
   case ISD::ANY_EXTEND:
   case ISD::SIGN_EXTEND:
   case ISD::ZERO_EXTEND:
     Res = WidenVecOp_EXTEND(N);
     break;
else
     return DAG.getNode(ISD::TokenFactor, SDLoc(ST), MVT::Other, StChain);
 }
 
 SDValue DAGTypeLegalizer::WidenVecOp_MSTORE(SDNode *N, unsigned OpNo) {
   MaskedStoreSDNode *MST = cast<MaskedStoreSDNode>(N);
   SDValue Mask = MST->getMask();
   EVT MaskVT = Mask.getValueType();
   SDValue StVal = MST->getValue();
+
+  assert(OpNo == 3 && "Unexpected operand number");
   // Widen the value
   SDValue WideVal = GetWidenedVector(StVal);
   SDLoc dl(N);
 
-  if (OpNo == 2 || getTypeAction(MaskVT) == TargetLowering::TypeWidenVector)
-    Mask = GetWidenedVector(Mask);
-  else {
-    // The mask should be widened as well
-    EVT BoolVT = getSetCCResultType(WideVal.getValueType());
-    // We can't use ModifyToType() because we should fill the mask with
-    // zeroes
-    unsigned WidenNumElts = BoolVT.getVectorNumElements();
-    unsigned MaskNumElts = MaskVT.getVectorNumElements();
 
-    unsigned NumConcat = WidenNumElts / MaskNumElts;
-    SmallVector<SDValue, 16> Ops(NumConcat);
-    SDValue ZeroVal = DAG.getConstant(0, dl, MaskVT);
-    Ops[0] = Mask;
-    for (unsigned i = 1; i != NumConcat; ++i)
-      Ops[i] = ZeroVal;
 
-    Mask = DAG.getNode(ISD::CONCAT_VECTORS, dl, BoolVT, Ops);
-  }
-  assert(Mask.getValueType().getVectorNumElements() ==
-         WideVal.getValueType().getVectorNumElements() &&
-         "Mask and data vectors should have the same number of elements");
+
   return DAG.getMaskedStore(MST->getChain(), dl, WideVal, MST->getBasePtr(),
                             Mask, MST->getMemoryVT(), MST->getMemOperand(),
                             false);
 }
 
+SDValue DAGTypeLegalizer::WidenVecOp_MSCATTER(SDNode *N, unsigned OpNo) {
+  assert(OpNo == 1 && "Can widen only data operand of mscatter");
+  MaskedScatterSDNode *MSC = cast<MaskedScatterSDNode>(N);
+  SDValue DataOp = MSC->getValue();
+  SDValue Mask = MSC->getMask();
+  MVT MaskVT = Mask.getSimpleValueType();
+
+  // Widen the value
+  SDValue WideVal = GetWidenedVector(DataOp);
+  unsigned NumElts = WideVal.getValueType().getVectorNumElements();
+  SDLoc dl(N);
+
+  // The mask should be widened as well
+  if (getTypeAction(MaskVT) == TargetLowering::TypePromoteInteger) {
+    Mask = GetPromotedInteger(Mask);
+    MaskVT = Mask.getSimpleValueType();
+  }
+
+  MVT WideMaskVT = MVT::getVectorVT(MaskVT.getScalarType(), NumElts);
+  Mask = ModifyToType(Mask, WideMaskVT, true);
+
+  // Widen index
+  SDValue Index = MSC->getIndex();
+  MVT WideIndexVT =
+    MVT::getVectorVT(Index.getSimpleValueType().getScalarType(), NumElts);
+  Index = ModifyToType(Index, WideIndexVT, false, true);
+
+  SDValue Ops[] = {MSC->getChain(), WideVal, Mask, MSC->getBasePtr(), Index};
+  return DAG.getMaskedScatter(DAG.getVTList(MVT::Other),
+                              MSC->getMemoryVT(), dl, Ops,
+                              MSC->getMemOperand());
+}
+
 SDValue DAGTypeLegalizer::WidenVecOp_SETCC(SDNode *N) {
   SDValue InOp0 = GetWidenedVector(N->getOperand(0));
   SDValue InOp1 = GetWidenedVector(N->getOperand(1));
   SDLoc dl(N);
 
   // WARNING: In this code we widen the compare instruction with garbage.
   // This garbage may contain denormal floats which may be slow. Is this a real
   // concern ? Should we zero the unused lanes if this is a float compare ?
StChain.push_back(DAG.getTruncStore(Chain, dl, EOp, NewBasePtr,
                                       ST->getPointerInfo().getWithOffset(Offset),
                                         StEltVT, isVolatile, isNonTemporal,
                                         MinAlign(Align, Offset), AAInfo));
   }
 }
 
 /// Modifies a vector input (widen or narrows) to a vector of NVT.  The
 /// input vector must have the same element type as NVT.
-SDValue DAGTypeLegalizer::ModifyToType(SDValue InOp, EVT NVT) {
+SDValue DAGTypeLegalizer::ModifyToType(SDValue InOp, EVT NVT,
+                                       bool FillWithZeroes,
+                                       bool UseExistingVal) {
   // Note that InOp might have been widened so it might already have
   // the right width or it might need be narrowed.
   EVT InVT = InOp.getValueType();
+  EVT EltVT = NVT.getVectorElementType();
   assert(InVT.getVectorElementType() == NVT.getVectorElementType() &&
          "input and widen element type must match");
+  if (InOp.isUndef())
+    return DAG.getUNDEF(NVT);
+
   SDLoc dl(InOp);
 
   // Check if InOp already has the right width.
   if (InVT == NVT)
     return InOp;
 
   unsigned InNumElts = InVT.getVectorNumElements();
   unsigned WidenNumElts = NVT.getVectorNumElements();
+  SDValue FillVal;
   if (WidenNumElts > InNumElts && WidenNumElts % InNumElts == 0) {
     unsigned NumConcat = WidenNumElts / InNumElts;
     SmallVector<SDValue, 16> Ops(NumConcat);
-    SDValue UndefVal = DAG.getUNDEF(InVT);
+    if (FillWithZeroes)
+      FillVal = DAG.getConstant(0, dl, InVT);
+    else if (UseExistingVal)
+      FillVal = InOp;
+    else
+      FillVal = DAG.getUNDEF(InVT);
     Ops[0] = InOp;
     for (unsigned i = 1; i != NumConcat; ++i)
-      Ops[i] = UndefVal;
+      Ops[i] = FillVal;
 
     return DAG.getNode(ISD::CONCAT_VECTORS, dl, NVT, Ops);
   }
 
   if (WidenNumElts < InNumElts && InNumElts % WidenNumElts)
     return DAG.getNode(
         ISD::EXTRACT_SUBVECTOR, dl, NVT, InOp,
         DAG.getConstant(0, dl, TLI.getVectorIdxTy(DAG.getDataLayout())));
 
   // Fall back to extract and build.
   SmallVector<SDValue, 16> Ops(WidenNumElts);
-  EVT EltVT = NVT.getVectorElementType();
   unsigned MinNumElts = std::min(WidenNumElts, InNumElts);
   unsigned Idx;
   for (Idx = 0; Idx < MinNumElts; ++Idx)
     Ops[Idx] = DAG.getNode(
         ISD::EXTRACT_VECTOR_ELT, dl, EltVT, InOp,
         DAG.getConstant(Idx, dl, TLI.getVectorIdxTy(DAG.getDataLayout())));
 
-  SDValue UndefVal = DAG.getUNDEF(EltVT);
+  if (FillWithZeroes)
+    FillVal = DAG.getConstant(0, dl, EltVT);
+  else if (UseExistingVal)
+    FillVal = Ops[0];
+  else
+    FillVal = DAG.getUNDEF(EltVT);
   for ( ; Idx < WidenNumElts; ++Idx)
-    Ops[Idx] = UndefVal;
+    Ops[Idx] = FillVal;
   return DAG.getNode(ISD::BUILD_VECTOR, dl, NVT, Ops);
 }

../lib/Target/X86/X86ISelLowering.cpp

if (Subtarget->hasVLX()){
       setTruncStoreAction(MVT::v8i32, MVT::v8i16, Legal);
 
       setTruncStoreAction(MVT::v2i64, MVT::v2i8,  Legal);
       setTruncStoreAction(MVT::v2i64, MVT::v2i16, Legal);
       setTruncStoreAction(MVT::v2i64, MVT::v2i32, Legal);
       setTruncStoreAction(MVT::v4i32, MVT::v4i8,  Legal);
       setTruncStoreAction(MVT::v4i32, MVT::v4i16, Legal);
     }
-    setOperationAction(ISD::TRUNCATE,           MVT::i1, Custom);
+    setOperationAction(ISD::TRUNCATE,           MVT::i1,    Custom);
     setOperationAction(ISD::TRUNCATE,           MVT::v16i8, Custom);
     setOperationAction(ISD::TRUNCATE,           MVT::v8i32, Custom);
     setOperationAction(ISD::VECTOR_SHUFFLE,     MVT::v8i1,  Custom);
     setOperationAction(ISD::VECTOR_SHUFFLE,     MVT::v16i1, Custom);
     if (Subtarget->hasDQI()) {
       setOperationAction(ISD::TRUNCATE,         MVT::v2i1, Custom);
       setOperationAction(ISD::TRUNCATE,         MVT::v4i1, Custom);
 
if (!Subtarget->useSoftFloat() && Subtarget->hasAVX512()) {
     // Custom lower several nodes.
     for (MVT VT : MVT::vector_valuetypes()) {
       unsigned EltSize = VT.getVectorElementType().getSizeInBits();
       if (EltSize == 1) {
         setOperationAction(ISD::AND, VT, Legal);
         setOperationAction(ISD::OR,  VT, Legal);
         setOperationAction(ISD::XOR,  VT, Legal);
       }
-      if (EltSize >= 32 && VT.getSizeInBits() <= 512) {
+      if ((VT.is128BitVector() || VT.is256BitVector()) && EltSize >= 32) {
         setOperationAction(ISD::MGATHER,  VT, Custom);
         setOperationAction(ISD::MSCATTER, VT, Custom);
       }
       // Extract subvector is special because the value type
       // (result) is 256/128-bit but the source is 512-bit wide.
       if (VT.is128BitVector() || VT.is256BitVector()) {
         setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);
       }
for (MVT VT : MVT::vector_valuetypes()) {
         setOperationAction(ISD::INSERT_VECTOR_ELT,   VT, Custom);
         setOperationAction(ISD::BUILD_VECTOR,        VT, Custom);
         setOperationAction(ISD::VSELECT,             VT, Legal);
         setOperationAction(ISD::EXTRACT_VECTOR_ELT,  VT, Custom);
         setOperationAction(ISD::SCALAR_TO_VECTOR,    VT, Custom);
         setOperationAction(ISD::INSERT_SUBVECTOR,    VT, Custom);
         setOperationAction(ISD::MLOAD,               VT, Legal);
         setOperationAction(ISD::MSTORE,              VT, Legal);
+        setOperationAction(ISD::MGATHER,             VT, Legal);
+        setOperationAction(ISD::MSCATTER,            VT, Custom);
       }
     }
     for (int i = MVT::v32i8; i != MVT::v8i64; ++i) {
       MVT VT = (MVT::SimpleValueType)i;
 
       // Do not attempt to promote non-512-bit vectors.
       if (!VT.is512BitVector())
         continue;
if (OpVT.getVectorElementType() == MVT::i1) {
     SDValue Undef = DAG.getUNDEF(OpVT);
     unsigned NumElems = OpVT.getVectorNumElements();
     SDValue ShiftBits = DAG.getConstant(NumElems/2, dl, MVT::i8);
 
     if (IdxVal == OpVT.getVectorNumElements() / 2) {
       // Zero upper bits of the Vec
       Vec = DAG.getNode(X86ISD::VSHLI, dl, OpVT, Vec, ShiftBits);
       Vec = DAG.getNode(X86ISD::VSRLI, dl, OpVT, Vec, ShiftBits);
 
       SDValue Vec2 = DAG.getNode(ISD::INSERT_SUBVECTOR, dl, OpVT, Undef,
                                  SubVec, ZeroIdx);
       Vec2 = DAG.getNode(X86ISD::VSHLI, dl, OpVT, Vec2, ShiftBits);
       return DAG.getNode(ISD::OR, dl, OpVT, Vec, Vec2);
     }
     if (IdxVal == 0) {
       SDValue Vec2 = DAG.getNode(ISD::INSERT_SUBVECTOR, dl, OpVT, Undef,
                                  SubVec, ZeroIdx);
static SDValue LowerFSINCOS(SDValue Op, const X86Subtarget *Subtarget,
   SDValue SinVal = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, ArgVT,
                                CallResult.first, DAG.getIntPtrConstant(0, dl));
   SDValue CosVal = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, ArgVT,
                                CallResult.first, DAG.getIntPtrConstant(1, dl));
   SDVTList Tys = DAG.getVTList(ArgVT, ArgVT);
   return DAG.getNode(ISD::MERGE_VALUES, dl, Tys, SinVal, CosVal);
 }
 
+/// Modifies a vector input (widen or narrows) to a vector of NVT.  The
+/// input vector must have the same element type as NVT.
+static SDValue ModifyToType(SDValue InOp, EVT NVT, SelectionDAG &DAG,
+                            bool FillWithZeroes = false,
+                            bool UseExistingVal = false) {
+  if (InOp.isUndef())
+    return DAG.getUNDEF(NVT);
+  // Check if InOp already has the right width.
+  EVT InVT = InOp.getValueType();
+  if (InVT == NVT)
+    return InOp;
+
+  assert(InVT.getVectorElementType() == NVT.getVectorElementType() &&
+         "input and widen element type must match");
+
+  unsigned InNumElts = InVT.getVectorNumElements();
+  unsigned WidenNumElts = NVT.getVectorNumElements();
+  EVT EltVT = NVT.getVectorElementType();
+
+  SDLoc dl(InOp);
+  SDValue FillVal;
+  if (WidenNumElts > InNumElts && WidenNumElts % InNumElts == 0) {
+    unsigned NumConcat = WidenNumElts / InNumElts;
+    if (ISD::isBuildVectorOfConstantSDNodes(InOp.getNode()) && NumConcat > 2) {
+      // Special case, because CONCAT_VECTORS with many operands is not
+      // converted to the BUILD_VECTOR 
+      SmallVector<SDValue, 16> Ops;
+      for (unsigned i = 0; i < InNumElts; ++i)
+        Ops.push_back(InOp.getOperand(i));
+      for (unsigned i = 0; i < WidenNumElts - InNumElts; ++i)
+        Ops.push_back(
+         UseExistingVal ? InOp.getOperand(0) :
+         FillWithZeroes ? DAG.getConstant(0, dl, EltVT) :
+         DAG.getUNDEF(EltVT));
+      return DAG.getNode(ISD::BUILD_VECTOR, dl, NVT, Ops);
+    }
+    if (InOp.getOpcode() == ISD::CONCAT_VECTORS && InOp.getNumOperands() == 2) {
+      SDValue N0 = InOp.getOperand(0);
+      SDValue N1 = InOp.getOperand(1);
+      if ((ISD::isBuildVectorAllZeros(N1.getNode()) && FillWithZeroes) ||
+          N1.isUndef() || (N0 == N1 && UseExistingVal)) {
+        InOp = InOp.getOperand(0);
+        InVT = InOp.getValueType();
+        InNumElts = InVT.getVectorNumElements();
+        NumConcat = WidenNumElts / InNumElts;
+      }
+    }
+    SDValue ZeroIndex = DAG.getIntPtrConstant(0, dl);
+    if (UseExistingVal && NumConcat > 2) {
+      SDValue ExtInOp = DAG.getNode(ISD::INSERT_SUBVECTOR, dl, NVT,
+                                    DAG.getUNDEF(NVT), InOp, ZeroIndex);
+      SmallVector<int, 16> ShuffleMask;
+      for (unsigned i = 0; i < NumConcat; ++i)
+        for (unsigned j = 0; j < InNumElts; ++j)
+          ShuffleMask.push_back(j);
+      return DAG.getVectorShuffle(NVT, dl, ExtInOp, DAG.getUNDEF(NVT),
+                                  ShuffleMask);
+    }
+    if (NumConcat > 2) {
+      FillVal = FillWithZeroes ? DAG.getConstant(0, dl, NVT) : DAG.getUNDEF(NVT);
+      return DAG.getNode(ISD::INSERT_SUBVECTOR, dl, NVT, FillVal,
+                         InOp, ZeroIndex);
+    }
+    FillVal = FillWithZeroes ? DAG.getConstant(0, dl, InVT) :
+      UseExistingVal ? InOp : DAG.getUNDEF(InVT);
+
+    return DAG.getNode(ISD::CONCAT_VECTORS, dl, NVT, InOp, FillVal);
+  }
+
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  if (WidenNumElts < InNumElts && InNumElts % WidenNumElts)
+    return DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, NVT, InOp,
+      DAG.getConstant(0, dl, TLI.getVectorIdxTy(DAG.getDataLayout())));
+
+  // Fall back to extract and build.
+  SmallVector<SDValue, 16> Ops(WidenNumElts);
+  unsigned MinNumElts = std::min(WidenNumElts, InNumElts);
+  unsigned Idx;
+  for (Idx = 0; Idx < MinNumElts; ++Idx)
+    Ops[Idx] = DAG.getNode(
+    ISD::EXTRACT_VECTOR_ELT, dl, EltVT, InOp,
+    DAG.getConstant(Idx, dl, TLI.getVectorIdxTy(DAG.getDataLayout())));
+
+  if (FillWithZeroes)
+    FillVal = DAG.getConstant(0, dl, EltVT);
+  else if (UseExistingVal)
+    FillVal = Ops[0];
+  else
+    FillVal = DAG.getUNDEF(EltVT);
+  for (; Idx < WidenNumElts; ++Idx)
+    Ops[Idx] = FillVal;
+  return DAG.getNode(ISD::BUILD_VECTOR, dl, NVT, Ops);
+}
+
 static SDValue LowerMSCATTER(SDValue Op, const X86Subtarget *Subtarget,
                              SelectionDAG &DAG) {
   assert(Subtarget->hasAVX512() &&
          "MGATHER/MSCATTER are supported on AVX-512 arch only");
 
+  // X86 scatter kills mask register, so its type should be added to
+  // the list of return values.
+  // If the "scatter" has 2 return values, it is already handled.
+  if (Op.getNode()->getNumValues() == 2)
+    return Op;
+
   MaskedScatterSDNode *N = cast<MaskedScatterSDNode>(Op.getNode());
-  EVT VT = N->getValue().getValueType();
+  SDValue Src = N->getValue();
+  MVT VT = Src.getSimpleValueType();
   assert(VT.getScalarSizeInBits() >= 32 && "Unsupported scatter op");
   SDLoc dl(Op);
 
-  // X86 scatter kills mask register, so its type should be added to
-  // the list of return values
-  if (N->getNumValues() == 1) {
-    SDValue Index = N->getIndex();
-    if (!Subtarget->hasVLX() && !VT.is512BitVector() &&
-        !Index.getValueType().is512BitVector())
+  SDValue NewScatter;
+  SDValue Mask = N->getMask();
+  SDValue Chain = N->getChain();
+  SDValue BasePtr = N->getBasePtr();
+  EVT MemVT = N->getMemoryVT();
+  MVT IndexVT = Index.getSimpleValueType();
+  MVT MaskVT = Mask.getSimpleValueType();
+
+  if (MemVT.getScalarSizeInBits() < VT.getScalarSizeInBits()) {
+    // Promoted data type
+    assert((MemVT == MVT::v2i32 && VT == MVT::v2i64) &&
+           "Unexpected memory type");
+    int ShuffleMask[] = {0, 2, -1, -1};
+    Src = DAG.getVectorShuffle(MVT::v4i32, dl, 
+                               DAG.getNode(ISD::BITCAST, dl, MVT::v4i32, Src),
+                               DAG.getUNDEF(MVT::v4i32), ShuffleMask);
+    // Now we have 4 elements instead of 2.
+    // Expand the index.
+    MVT NewIndexVT = MVT::getVectorVT(IndexVT.getScalarType(), 4);
+    Index = ModifyToType(Index, NewIndexVT, DAG, false, true);
+
+    // Expand the mask with zeroes
+    // Mask may be <2 x i64> or <2 x i1> at this moment
+    assert((MaskVT == MVT::v2i1 || MaskVT == MVT::v2i64) &&
+           "Unexpected mask type");
+    MVT ExtMaskVT = MVT::getVectorVT(MaskVT.getScalarType(), 4);
+    Mask = ModifyToType(Mask, ExtMaskVT, DAG, true);
+    VT = MVT::v4i32;
+  }
+
+  unsigned NumElts = VT.getVectorNumElements();
+      !Index.getValueType().is512BitVector()) {
+    // AVX512F supports only 512-bit vectors. Or data or index should
+    // be 512 bit wide. If now the both index and data are 256-bit, but
+    // the vector contains 8 elements, we just sign-extend the index
+    if (IndexVT == MVT::v8i32)
+      // Just extend index
+      Index = DAG.getNode(ISD::SIGN_EXTEND, dl, MVT::v8i64, Index);
+    else {
+      // The minimal number of elts in scatter is 8
+      NumElts = 8;
+      // Index
+      MVT NewIndexVT = MVT::getVectorVT(IndexVT.getScalarType(), NumElts);
+      // Use original index here, do not modify the index twice
+      Index = ModifyToType(N->getIndex(), NewIndexVT, DAG, false, true);
+      if (IndexVT.getScalarType() == MVT::i32)
         Index = DAG.getNode(ISD::SIGN_EXTEND, dl, MVT::v8i64, Index);
       
-    SDVTList VTs = DAG.getVTList(N->getMask().getValueType(), MVT::Other);
-    SDValue Ops[] = { N->getOperand(0), N->getOperand(1),  N->getOperand(2),
-                      N->getOperand(3), Index };
-
-    SDValue NewScatter = DAG.getMaskedScatter(VTs, VT, dl, Ops, N->getMemOperand());
-    DAG.ReplaceAllUsesWith(Op, SDValue(NewScatter.getNode(), 1));
-    return SDValue(NewScatter.getNode(), 0);
-  }
-  return Op;
-}
+      // Mask
+     // At this point we have promoted mask operand
+      assert(MaskVT.getScalarSizeInBits() >= 32 && "unexpected mask type");
+      MVT ExtMaskVT = MVT::getVectorVT(MaskVT.getScalarType(), NumElts);
+      // Use the original mask here, do not modify the mask twice
+      Mask = ModifyToType(N->getMask(), ExtMaskVT, DAG, true); 
+
+      // The value that should be stored
+      MVT NewVT = MVT::getVectorVT(VT.getScalarType(), NumElts);
+      Src = ModifyToType(Src, NewVT, DAG);
+    }
+  }
+  // If the mask is "wide" at this point - truncate it to i1 vector
+  MVT BitMaskVT = MVT::getVectorVT(MVT::i1, NumElts);
+  Mask = DAG.getNode(ISD::TRUNCATE, dl, BitMaskVT, Mask);
+
+  // The mask is killed by scatter, add it to the values
+  SDVTList VTs = DAG.getVTList(BitMaskVT, MVT::Other);
+  SDValue Ops[] = {Chain, Src, Mask, BasePtr, Index};
+  NewScatter = DAG.getMaskedScatter(VTs, N->getMemoryVT(), dl, Ops,
+                                    N->getMemOperand());
 
 static SDValue LowerMGATHER(SDValue Op, const X86Subtarget *Subtarget,
                             SelectionDAG &DAG) {
   assert(Subtarget->hasAVX512() &&
          "MGATHER/MSCATTER are supported on AVX-512 arch only");
 
   MaskedGatherSDNode *N = cast<MaskedGatherSDNode>(Op.getNode());
-  EVT VT = Op.getValueType();
-  assert(VT.getScalarSizeInBits() >= 32 && "Unsupported gather op");
   SDLoc dl(Op);
-
+  MVT VT = Op.getSimpleValueType();
   SDValue Index = N->getIndex();
+  SDValue Mask = N->getMask();
+  SDValue Src0 = N->getValue();
+  MVT IndexVT = Index.getSimpleValueType();
+  MVT MaskVT = Mask.getSimpleValueType();
+
+  unsigned NumElts = VT.getVectorNumElements();
+  assert(VT.getScalarSizeInBits() >= 32 && "Unsupported gather op");
+
   if (!Subtarget->hasVLX() && !VT.is512BitVector() &&
       !Index.getValueType().is512BitVector()) {
+    // AVX512F supports only 512-bit vectors. Or data or index should
+    // be 512 bit wide. If now the both index and data are 256-bit, but
+    // the vector contains 8 elements, we just sign-extend the index
+    if (NumElts == 8) {
       Index = DAG.getNode(ISD::SIGN_EXTEND, dl, MVT::v8i64, Index);
       SDValue Ops[] = { N->getOperand(0), N->getOperand(1),  N->getOperand(2),
                         N->getOperand(3), Index };
       DAG.UpdateNodeOperands(N, Ops);
+      return Op;
+    }
+
+    // Minimal number of elements in Gather
+    NumElts = 8;
+    // Index
+    MVT NewIndexVT = MVT::getVectorVT(IndexVT.getScalarType(), NumElts);
+    Index = ModifyToType(Index, NewIndexVT, DAG, false, true);
+    if (IndexVT.getScalarType() == MVT::i32)
+      Index = DAG.getNode(ISD::SIGN_EXTEND, dl, MVT::v8i64, Index);
+
+    // Mask
+    MVT MaskBitVT = MVT::getVectorVT(MVT::i1, NumElts);
+    // At this point we have promoted mask operand
+    assert(MaskVT.getScalarSizeInBits() >= 32 && "unexpected mask type");
+    MVT ExtMaskVT = MVT::getVectorVT(MaskVT.getScalarType(), NumElts);
+    Mask = ModifyToType(Mask, ExtMaskVT, DAG, true);
+    Mask = DAG.getNode(ISD::TRUNCATE, dl, MaskBitVT, Mask);
+
+    // The pass-thru value
+    MVT NewVT = MVT::getVectorVT(VT.getScalarType(), NumElts);
+    Src0 = ModifyToType(Src0, NewVT, DAG);
+
+    SDValue Ops[] = { N->getChain(), Src0, Mask, N->getBasePtr(), Index };
+    SDValue NewGather = DAG.getMaskedGather(DAG.getVTList(NewVT, MVT::Other),
+                                            N->getMemoryVT(), dl, Ops,
+                                            N->getMemOperand());
+    SDValue Exract = DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, VT,
+                                 NewGather.getValue(0),
+                                 DAG.getIntPtrConstant(0, dl));
+    SDValue RetOps[] = {Exract, NewGather.getValue(1)};
+    return DAG.getMergeValues(RetOps, dl);
   }
   return Op;
 }
 
 SDValue X86TargetLowering::LowerGC_TRANSITION_START(SDValue Op,
                                                     SelectionDAG &DAG) const {
   // TODO: Eventually, the lowering of these nodes should be informed by or
   // deferred to the GC strategy for the function in which they appear. For
if (auto *Mask = dyn_cast<ConstantSDNode>(N->getOperand(2)))
         return DAG.getNode(X86ISD::BLENDI, DL, VT, V1, V0, NewMask);
       }
 
   return SDValue();
 }
 
 // Helper function of PerformSETCCCombine. It is to materialize "setb reg"
 // as "sbb reg,reg", since it can be extended without zext and produces
 // an all-ones bit which is more useful than 0/1 in some cases.
 static SDValue MaterializeSETB(SDLoc DL, SDValue EFLAGS, SelectionDAG &DAG,
                                MVT VT) {
   if (VT == MVT::i8)
     return DAG.getNode(ISD::AND, DL, VT,
                        DAG.getNode(X86ISD::SETCC_CARRY, DL, MVT::i8,
                                    DAG.getConstant(X86::COND_B, DL, MVT::i8),
                                    EFLAGS),
                        DAG.getConstant(1, DL, VT));

../test/CodeGen/X86/masked_gather_scatter.ll

-; RUN: llc -mtriple=x86_64-apple-darwin  -mcpu=knl < %s | FileCheck %s -check-prefix=KNL
+; RUN: llc -mtriple=x86_64-apple-darwin  -mcpu=x86-64 -mattr=+avx512f < %s | FileCheck %s --check-prefix=ALL --check-prefix=KNL
+; RUN: llc -mtriple=x86_64-apple-darwin  -mcpu=x86-64 -mattr=+avx512vl -mattr=+avx512dq < %s | FileCheck %s --check-prefix=ALL --check-prefix=SKX
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; KNL-LABEL: test1
-; KNL: kxnorw  %k1, %k1, %k1
-; KNL: vgatherdps      (%rdi,%zmm0,4), %zmm1 {%k1}
 define <16 x float> @test1(float* %base, <16 x i32> %ind) {
+; ALL-LABEL: test1:
+; ALL:       ## BB#0:
+; ALL-NEXT:    kxnorw %k1, %k1, %k1
+; ALL-NEXT:    vgatherdps (%rdi,%zmm0,4), %zmm1 {%k1}
+; ALL-NEXT:    vmovaps %zmm1, %zmm0
+; ALL-NEXT:    retq
 
   %broadcast.splatinsert = insertelement <16 x float*> undef, float* %base, i32 0
   %broadcast.splat = shufflevector <16 x float*> %broadcast.splatinsert, <16 x float*> undef, <16 x i32> zeroinitializer
 
   %sext_ind = sext <16 x i32> %ind to <16 x i64>
   %gep.random = getelementptr float, <16 x float*> %broadcast.splat, <16 x i64> %sext_ind
-  
   %res = call <16 x float> @llvm.masked.gather.v16f32(<16 x float*> %gep.random, i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <16 x float> undef)
   ret <16 x float>%res
 }
 
 declare <16 x i32> @llvm.masked.gather.v16i32(<16 x i32*>, i32, <16 x i1>, <16 x i32>)
 declare <16 x float> @llvm.masked.gather.v16f32(<16 x float*>, i32, <16 x i1>, <16 x float>)
 declare <8 x i32> @llvm.masked.gather.v8i32(<8 x i32*> , i32, <8 x i1> , <8 x i32> )
-  
-; KNL-LABEL: test2
-; KNL: kmovw %esi, %k1
-; KNL: vgatherdps      (%rdi,%zmm0,4), %zmm1 {%k1}
 define <16 x float> @test2(float* %base, <16 x i32> %ind, i16 %mask) {
+; ALL-LABEL: test2:
+; ALL:       ## BB#0:
+; ALL-NEXT:    kmovw %esi, %k1
+; ALL-NEXT:    vgatherdps (%rdi,%zmm0,4), %zmm1 {%k1}
+; ALL-NEXT:    vmovaps %zmm1, %zmm0
+; ALL-NEXT:    retq
 
   %broadcast.splatinsert = insertelement <16 x float*> undef, float* %base, i32 0
   %broadcast.splat = shufflevector <16 x float*> %broadcast.splatinsert, <16 x float*> undef, <16 x i32> zeroinitializer
 
   %sext_ind = sext <16 x i32> %ind to <16 x i64>
   %gep.random = getelementptr float, <16 x float*> %broadcast.splat, <16 x i64> %sext_ind
   %imask = bitcast i16 %mask to <16 x i1>
   %res = call <16 x float> @llvm.masked.gather.v16f32(<16 x float*> %gep.random, i32 4, <16 x i1> %imask, <16 x float>undef)
   ret <16 x float> %res
 }
 
-; KNL-LABEL: test3
-; KNL: kmovw %esi, %k1
-; KNL: vpgatherdd      (%rdi,%zmm0,4), %zmm1 {%k1}
 define <16 x i32> @test3(i32* %base, <16 x i32> %ind, i16 %mask) {
+; ALL-LABEL: test3:
+; ALL:       ## BB#0:
+; ALL-NEXT:    kmovw %esi, %k1
+; ALL-NEXT:    vpgatherdd (%rdi,%zmm0,4), %zmm1 {%k1}
+; ALL-NEXT:    vmovaps %zmm1, %zmm0
+; ALL-NEXT:    retq
 
   %broadcast.splatinsert = insertelement <16 x i32*> undef, i32* %base, i32 0
   %broadcast.splat = shufflevector <16 x i32*> %broadcast.splatinsert, <16 x i32*> undef, <16 x i32> zeroinitializer
 
   %sext_ind = sext <16 x i32> %ind to <16 x i64>
   %gep.random = getelementptr i32, <16 x i32*> %broadcast.splat, <16 x i64> %sext_ind
   %imask = bitcast i16 %mask to <16 x i1>
   %res = call <16 x i32> @llvm.masked.gather.v16i32(<16 x i32*> %gep.random, i32 4, <16 x i1> %imask, <16 x i32>undef)
   ret <16 x i32> %res
 }
 
-; KNL-LABEL: test4
-; KNL: kmovw %esi, %k1
-; KNL: kmovw
-; KNL: vpgatherdd
-; KNL: vpgatherdd
-
 define <16 x i32> @test4(i32* %base, <16 x i32> %ind, i16 %mask) {
+; ALL-LABEL: test4:
+; ALL:       ## BB#0:
+; ALL-NEXT:    kmovw %esi, %k1
+; ALL-NEXT:    kmovw %k1, %k2
+; ALL-NEXT:    vpgatherdd (%rdi,%zmm0,4), %zmm1 {%k2}
+; ALL-NEXT:    vmovaps %zmm1, %zmm2
+; ALL-NEXT:    vpgatherdd (%rdi,%zmm0,4), %zmm2 {%k1}
+; ALL-NEXT:    vpaddd %zmm2, %zmm1, %zmm0
+; ALL-NEXT:    retq
 
   %broadcast.splatinsert = insertelement <16 x i32*> undef, i32* %base, i32 0
   %broadcast.splat = shufflevector <16 x i32*> %broadcast.splatinsert, <16 x i32*> undef, <16 x i32> zeroinitializer
 
   %gep.random = getelementptr i32, <16 x i32*> %broadcast.splat, <16 x i32> %ind
   %imask = bitcast i16 %mask to <16 x i1>
   %gt1 = call <16 x i32> @llvm.masked.gather.v16i32(<16 x i32*> %gep.random, i32 4, <16 x i1> %imask, <16 x i32>undef)
   %gt2 = call <16 x i32> @llvm.masked.gather.v16i32(<16 x i32*> %gep.random, i32 4, <16 x i1> %imask, <16 x i32>%gt1)
   %res = add <16 x i32> %gt1, %gt2
   ret <16 x i32> %res
 }
 
-; KNL-LABEL: test5
-; KNL: kmovw %k1, %k2
-; KNL: vpscatterdd {{.*}}%k2
-; KNL: vpscatterdd {{.*}}%k1
-
 define void @test5(i32* %base, <16 x i32> %ind, i16 %mask, <16 x i32>%val) {
+; ALL-LABEL: test5:
+; ALL:       ## BB#0:
+; ALL-NEXT:    kmovw %esi, %k1
+; ALL-NEXT:    kmovw %k1, %k2
+; ALL-NEXT:    vpscatterdd %zmm1, (%rdi,%zmm0,4) {%k2}
+; ALL-NEXT:    vpscatterdd %zmm1, (%rdi,%zmm0,4) {%k1}
+; ALL-NEXT:    retq
 
   %broadcast.splatinsert = insertelement <16 x i32*> undef, i32* %base, i32 0
   %broadcast.splat = shufflevector <16 x i32*> %broadcast.splatinsert, <16 x i32*> undef, <16 x i32> zeroinitializer
 
   %gep.random = getelementptr i32, <16 x i32*> %broadcast.splat, <16 x i32> %ind
   %imask = bitcast i16 %mask to <16 x i1>
   call void @llvm.masked.scatter.v16i32(<16 x i32>%val, <16 x i32*> %gep.random, i32 4, <16 x i1> %imask)
   call void @llvm.masked.scatter.v16i32(<16 x i32>%val, <16 x i32*> %gep.random, i32 4, <16 x i1> %imask)
   ret void
 }
 
 declare void @llvm.masked.scatter.v8i32(<8 x i32> , <8 x i32*> , i32 , <8 x i1> )
 declare void @llvm.masked.scatter.v16i32(<16 x i32> , <16 x i32*> , i32 , <16 x i1> )
 
-; KNL-LABEL: test6
-; KNL: kxnorw  %k1, %k1, %k1
-; KNL: kxnorw  %k2, %k2, %k2
-; KNL: vpgatherqd      (,%zmm{{.*}}), %ymm{{.*}} {%k2}
-; KNL: vpscatterqd     %ymm{{.*}}, (,%zmm{{.*}}) {%k1}
 define <8 x i32> @test6(<8 x i32>%a1, <8 x i32*> %ptr) {
+; ALL-LABEL: test6:
+; ALL:       ## BB#0:
+; ALL-NEXT:    kxnorw %k1, %k1, %k1
+; ALL-NEXT:    kxnorw %k2, %k2, %k2
+; ALL-NEXT:    vpgatherqd (,%zmm1), %ymm2 {%k2}
+; ALL-NEXT:    vpscatterqd %ymm0, (,%zmm1) {%k1}
+; ALL-NEXT:    vmovaps %zmm2, %zmm0
+; ALL-NEXT:    retq
 
   %a = call <8 x i32> @llvm.masked.gather.v8i32(<8 x i32*> %ptr, i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> undef)
 
   call void @llvm.masked.scatter.v8i32(<8 x i32> %a1, <8 x i32*> %ptr, i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>)
   ret <8 x i32>%a
 }
 
 ; In this case the index should be promoted to <8 x i64> for KNL
-; KNL-LABEL: test7
-; KNL: vpmovsxdq %ymm0, %zmm0
-; KNL: kmovw   %k1, %k2
-; KNL: vpgatherqd {{.*}} {%k2}
-; KNL: vpgatherqd {{.*}} {%k1}
 define <8 x i32> @test7(i32* %base, <8 x i32> %ind, i8 %mask) {
+; KNL-LABEL: test7:
+; KNL:       ## BB#0:
+; KNL-NEXT:    movzbl %sil, %eax
+; KNL-NEXT:    kmovw %eax, %k1
+; KNL-NEXT:    vpmovsxdq %ymm0, %zmm0
+; KNL-NEXT:    kmovw %k1, %k2
+; KNL-NEXT:    vpgatherqd (%rdi,%zmm0,4), %ymm1 {%k2}
+; KNL-NEXT:    vmovaps %zmm1, %zmm2
+; KNL-NEXT:    vpgatherqd (%rdi,%zmm0,4), %ymm2 {%k1}
+; KNL-NEXT:    vpaddd %ymm2, %ymm1, %ymm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test7:
+; SKX:       ## BB#0:
+; SKX-NEXT:    kmovb %esi, %k1
+; SKX-NEXT:    kmovw %k1, %k2
+; SKX-NEXT:    vpgatherdd (%rdi,%ymm0,4), %ymm1 {%k2}
+; SKX-NEXT:    vmovaps %zmm1, %zmm2
+; SKX-NEXT:    vpgatherdd (%rdi,%ymm0,4), %ymm2 {%k1}
+; SKX-NEXT:    vpaddd %ymm2, %ymm1, %ymm0
+; SKX-NEXT:    retq
 
   %broadcast.splatinsert = insertelement <8 x i32*> undef, i32* %base, i32 0
   %broadcast.splat = shufflevector <8 x i32*> %broadcast.splatinsert, <8 x i32*> undef, <8 x i32> zeroinitializer
 
   %gep.random = getelementptr i32, <8 x i32*> %broadcast.splat, <8 x i32> %ind
   %imask = bitcast i8 %mask to <8 x i1>
   %gt1 = call <8 x i32> @llvm.masked.gather.v8i32(<8 x i32*> %gep.random, i32 4, <8 x i1> %imask, <8 x i32>undef)
   %gt2 = call <8 x i32> @llvm.masked.gather.v8i32(<8 x i32*> %gep.random, i32 4, <8 x i1> %imask, <8 x i32>%gt1)
   %res = add <8 x i32> %gt1, %gt2
   ret <8 x i32> %res
 }
 
 ; No uniform base in this case, index <8 x i64> contains addresses,
 ; each gather call will be split into two
-; KNL-LABEL: test8
-; KNL: kshiftrw        $8, %k1, %k2
-; KNL: vpgatherqd
-; KNL: vpgatherqd
-; KNL: vinserti64x4
-; KNL: vpgatherqd
-; KNL: vpgatherqd
-; KNL: vinserti64x4
 define <16 x i32> @test8(<16 x i32*> %ptr.random, <16 x i32> %ind, i16 %mask) {
+; KNL-LABEL: test8:
+; KNL:       ## BB#0:
+; KNL-NEXT:    kmovw %edi, %k1
+; KNL-NEXT:    kshiftrw $8, %k1, %k2
+; KNL-NEXT:    kmovw %k2, %k3
+; KNL-NEXT:    vpgatherqd (,%zmm1), %ymm2 {%k3}
+; KNL-NEXT:    kmovw %k1, %k3
+; KNL-NEXT:    vpgatherqd (,%zmm0), %ymm3 {%k3}
+; KNL-NEXT:    vinserti64x4 $1, %ymm2, %zmm3, %zmm4
+; KNL-NEXT:    vpgatherqd (,%zmm1), %ymm2 {%k2}
+; KNL-NEXT:    vpgatherqd (,%zmm0), %ymm3 {%k1}
+; KNL-NEXT:    vinserti64x4 $1, %ymm2, %zmm3, %zmm0
+; KNL-NEXT:    vpaddd %zmm0, %zmm4, %zmm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test8:
+; SKX:       ## BB#0:
+; SKX-NEXT:    kmovw %edi, %k1
+; SKX-NEXT:    kshiftrw $8, %k1, %k2
+; SKX-NEXT:    kmovw %k2, %k3
+; SKX-NEXT:    vpgatherqd (,%zmm1), %ymm2 {%k3}
+; SKX-NEXT:    kmovw %k1, %k3
+; SKX-NEXT:    vpgatherqd (,%zmm0), %ymm3 {%k3}
+; SKX-NEXT:    vinserti32x8 $1, %ymm2, %zmm3, %zmm4
+; SKX-NEXT:    vpgatherqd (,%zmm1), %ymm2 {%k2}
+; SKX-NEXT:    vpgatherqd (,%zmm0), %ymm3 {%k1}
+; SKX-NEXT:    vinserti32x8 $1, %ymm2, %zmm3, %zmm0
+; SKX-NEXT:    vpaddd %zmm0, %zmm4, %zmm0
+; SKX-NEXT:    retq
   %imask = bitcast i16 %mask to <16 x i1>
   %gt1 = call <16 x i32> @llvm.masked.gather.v16i32(<16 x i32*> %ptr.random, i32 4, <16 x i1> %imask, <16 x i32>undef)
   %gt2 = call <16 x i32> @llvm.masked.gather.v16i32(<16 x i32*> %ptr.random, i32 4, <16 x i1> %imask, <16 x i32>%gt1)
   %res = add <16 x i32> %gt1, %gt2
   ret <16 x i32> %res
 }
 
 %struct.RT = type { i8, [10 x [20 x i32]], i8 }
 %struct.ST = type { i32, double, %struct.RT }
 
 ; Masked gather for agregate types
 ; Test9 and Test10 should give the same result (scalar and vector indices in GEP)
 
-; KNL-LABEL: test9
-; KNL: vpbroadcastq    %rdi, %zmm
-; KNL: vpmovsxdq
-; KNL: vpbroadcastq
-; KNL: vpmuludq
-; KNL: vpaddq
-; KNL: vpaddq
-; KNL: vpaddq
-; KNL: vpaddq
-; KNL: vpgatherqd      (,%zmm
-
 define <8 x i32> @test9(%struct.ST* %base, <8 x i64> %ind1, <8 x i32>%ind5) {
+; KNL-LABEL: test9:
+; KNL:       ## BB#0: ## %entry
+; KNL-NEXT:    vpbroadcastq %rdi, %zmm2
+; KNL-NEXT:    vpmovsxdq %ymm1, %zmm1
+; KNL-NEXT:    vpbroadcastq {{.*}}(%rip), %zmm3
+; KNL-NEXT:    vpmuludq %zmm3, %zmm1, %zmm4
+; KNL-NEXT:    vpsrlq $32, %zmm1, %zmm1
+; KNL-NEXT:    vpmuludq %zmm3, %zmm1, %zmm1
+; KNL-NEXT:    vpsllq $32, %zmm1, %zmm1
+; KNL-NEXT:    vpaddq %zmm1, %zmm4, %zmm1
+; KNL-NEXT:    vpbroadcastq {{.*}}(%rip), %zmm3
+; KNL-NEXT:    vpmuludq %zmm3, %zmm0, %zmm4
+; KNL-NEXT:    vpsrlq $32, %zmm0, %zmm0
+; KNL-NEXT:    vpmuludq %zmm3, %zmm0, %zmm0
+; KNL-NEXT:    vpsllq $32, %zmm0, %zmm0
+; KNL-NEXT:    vpaddq %zmm0, %zmm4, %zmm0
+; KNL-NEXT:    vpaddq %zmm0, %zmm2, %zmm0
+; KNL-NEXT:    vpaddq %zmm1, %zmm0, %zmm0
+; KNL-NEXT:    vpaddq {{.*}}(%rip){1to8}, %zmm0, %zmm1
+; KNL-NEXT:    kxnorw %k1, %k1, %k1
+; KNL-NEXT:    vpgatherqd (,%zmm1), %ymm0 {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test9:
+; SKX:       ## BB#0: ## %entry
+; SKX-NEXT:    vpbroadcastq %rdi, %zmm2
+; SKX-NEXT:    vpmullq {{.*}}(%rip){1to8}, %zmm0, %zmm0
+; SKX-NEXT:    vpaddq %zmm0, %zmm2, %zmm0
+; SKX-NEXT:    vpmovsxdq %ymm1, %zmm1
+; SKX-NEXT:    vpmullq {{.*}}(%rip){1to8}, %zmm1, %zmm1
+; SKX-NEXT:    vpaddq %zmm1, %zmm0, %zmm0
+; SKX-NEXT:    vpaddq {{.*}}(%rip){1to8}, %zmm0, %zmm1
+; SKX-NEXT:    kxnorw %k1, %k1, %k1
+; SKX-NEXT:    vpgatherqd (,%zmm1), %ymm0 {%k1}
+; SKX-NEXT:    retq
 entry:
   %broadcast.splatinsert = insertelement <8 x %struct.ST*> undef, %struct.ST* %base, i32 0
   %broadcast.splat = shufflevector <8 x %struct.ST*> %broadcast.splatinsert, <8 x %struct.ST*> undef, <8 x i32> zeroinitializer
 
   %arrayidx = getelementptr  %struct.ST, <8 x %struct.ST*> %broadcast.splat, <8 x i64> %ind1, <8 x i32> <i32 2, i32 2, i32 2, i32 2, i32 2, i32 2, i32 2, i32 2>, <8 x i32><i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>, <8 x i32> %ind5, <8 x i64> <i64 13, i64 13, i64 13, i64 13, i64 13, i64 13, i64 13, i64 13>
   %res = call <8 x i32 >  @llvm.masked.gather.v8i32(<8 x i32*>%arrayidx, i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> undef)
   ret <8 x i32> %res
 }
 
-; KNL-LABEL: test10
-; KNL: vpbroadcastq    %rdi, %zmm
-; KNL: vpmovsxdq
-; KNL: vpbroadcastq
-; KNL: vpmuludq
-; KNL: vpaddq
-; KNL: vpaddq
-; KNL: vpaddq
-; KNL: vpaddq
-; KNL: vpgatherqd      (,%zmm
 define <8 x i32> @test10(%struct.ST* %base, <8 x i64> %i1, <8 x i32>%ind5) {
+; KNL-LABEL: test10:
+; KNL:       ## BB#0: ## %entry
+; KNL-NEXT:    vpbroadcastq %rdi, %zmm2
+; KNL-NEXT:    vpmovsxdq %ymm1, %zmm1
+; KNL-NEXT:    vpbroadcastq {{.*}}(%rip), %zmm3
+; KNL-NEXT:    vpmuludq %zmm3, %zmm1, %zmm4
+; KNL-NEXT:    vpsrlq $32, %zmm1, %zmm1
+; KNL-NEXT:    vpmuludq %zmm3, %zmm1, %zmm1
+; KNL-NEXT:    vpsllq $32, %zmm1, %zmm1
+; KNL-NEXT:    vpaddq %zmm1, %zmm4, %zmm1
+; KNL-NEXT:    vpbroadcastq {{.*}}(%rip), %zmm3
+; KNL-NEXT:    vpmuludq %zmm3, %zmm0, %zmm4
+; KNL-NEXT:    vpsrlq $32, %zmm0, %zmm0
+; KNL-NEXT:    vpmuludq %zmm3, %zmm0, %zmm0
+; KNL-NEXT:    vpsllq $32, %zmm0, %zmm0
+; KNL-NEXT:    vpaddq %zmm0, %zmm4, %zmm0
+; KNL-NEXT:    vpaddq %zmm0, %zmm2, %zmm0
+; KNL-NEXT:    vpaddq %zmm1, %zmm0, %zmm0
+; KNL-NEXT:    vpaddq {{.*}}(%rip){1to8}, %zmm0, %zmm1
+; KNL-NEXT:    kxnorw %k1, %k1, %k1
+; KNL-NEXT:    vpgatherqd (,%zmm1), %ymm0 {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test10:
+; SKX:       ## BB#0: ## %entry
+; SKX-NEXT:    vpbroadcastq %rdi, %zmm2
+; SKX-NEXT:    vpmullq {{.*}}(%rip){1to8}, %zmm0, %zmm0
+; SKX-NEXT:    vpaddq %zmm0, %zmm2, %zmm0
+; SKX-NEXT:    vpmovsxdq %ymm1, %zmm1
+; SKX-NEXT:    vpmullq {{.*}}(%rip){1to8}, %zmm1, %zmm1
+; SKX-NEXT:    vpaddq %zmm1, %zmm0, %zmm0
+; SKX-NEXT:    vpaddq {{.*}}(%rip){1to8}, %zmm0, %zmm1
+; SKX-NEXT:    kxnorw %k1, %k1, %k1
+; SKX-NEXT:    vpgatherqd (,%zmm1), %ymm0 {%k1}
+; SKX-NEXT:    retq
 entry:
   %broadcast.splatinsert = insertelement <8 x %struct.ST*> undef, %struct.ST* %base, i32 0
   %broadcast.splat = shufflevector <8 x %struct.ST*> %broadcast.splatinsert, <8 x %struct.ST*> undef, <8 x i32> zeroinitializer
 
   %arrayidx = getelementptr  %struct.ST, <8 x %struct.ST*> %broadcast.splat, <8 x i64> %i1, i32 2, i32 1, <8 x i32> %ind5, i64 13
   %res = call <8 x i32 >  @llvm.masked.gather.v8i32(<8 x i32*>%arrayidx, i32 4, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i32> undef)
   ret <8 x i32> %res
 }
 
 ; Splat index in GEP, requires broadcast
-; KNL-LABEL: test11
-; KNL: vpbroadcastd    %esi, %zmm
-; KNL: vgatherdps      (%rdi,%zmm
 define <16 x float> @test11(float* %base, i32 %ind) {
+; ALL-LABEL: test11:
+; ALL:       ## BB#0:
+; ALL-NEXT:    vpbroadcastd %esi, %zmm1
+; ALL-NEXT:    kxnorw %k1, %k1, %k1
+; ALL-NEXT:    vgatherdps (%rdi,%zmm1,4), %zmm0 {%k1}
+; ALL-NEXT:    retq
 
   %broadcast.splatinsert = insertelement <16 x float*> undef, float* %base, i32 0
   %broadcast.splat = shufflevector <16 x float*> %broadcast.splatinsert, <16 x float*> undef, <16 x i32> zeroinitializer
 
   %gep.random = getelementptr float, <16 x float*> %broadcast.splat, i32 %ind
 
   %res = call <16 x float> @llvm.masked.gather.v16f32(<16 x float*> %gep.random, i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <16 x float> undef)
   ret <16 x float>%res
 }
 
 ; We are checking the uniform base here. It is taken directly from input to vgatherdps
-; KNL-LABEL: test12
-; KNL: kxnorw  %k1, %k1, %k1
-; KNL: vgatherdps      (%rdi,%zmm
 define <16 x float> @test12(float* %base, <16 x i32> %ind) {
+; ALL-LABEL: test12:
+; ALL:       ## BB#0:
+; ALL-NEXT:    kxnorw %k1, %k1, %k1
+; ALL-NEXT:    vgatherdps (%rdi,%zmm0,4), %zmm1 {%k1}
+; ALL-NEXT:    vmovaps %zmm1, %zmm0
+; ALL-NEXT:    retq
 
   %sext_ind = sext <16 x i32> %ind to <16 x i64>
   %gep.random = getelementptr float, float *%base, <16 x i64> %sext_ind
 
   %res = call <16 x float> @llvm.masked.gather.v16f32(<16 x float*> %gep.random, i32 4, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <16 x float> undef)
   ret <16 x float>%res
 }
 
 ; The same as the previous, but the mask is undefined
-; KNL-LABEL: test13
-; KNL-NOT: kxnorw
-; KNL: vgatherdps      (%rdi,%zmm
 define <16 x float> @test13(float* %base, <16 x i32> %ind) {
+; ALL-LABEL: test13:
+; ALL:       ## BB#0:
+; ALL-NEXT:    vgatherdps (%rdi,%zmm0,4), %zmm1 {%k1}
+; ALL-NEXT:    vmovaps %zmm1, %zmm0
+; ALL-NEXT:    retq
 
   %sext_ind = sext <16 x i32> %ind to <16 x i64>
   %gep.random = getelementptr float, float *%base, <16 x i64> %sext_ind
 
   %res = call <16 x float> @llvm.masked.gather.v16f32(<16 x float*> %gep.random, i32 4, <16 x i1> undef, <16 x float> undef)
   ret <16 x float>%res
 }
 
 ; The base pointer is not splat, can't find unform base
-; KNL-LABEL: test14
-; KNL: vgatherqps      (,%zmm0)
-; KNL: vgatherqps      (,%zmm0)
 define <16 x float> @test14(float* %base, i32 %ind, <16 x float*> %vec) {
+; KNL-LABEL: test14:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vpinsrq $1, %rdi, %xmm0, %xmm1
+; KNL-NEXT:    vinserti32x4 $0, %xmm1, %zmm0, %zmm0
+; KNL-NEXT:    vpbroadcastq %xmm0, %zmm0
+; KNL-NEXT:    vmovd %esi, %xmm1
+; KNL-NEXT:    vpbroadcastd %xmm1, %ymm1
+; KNL-NEXT:    vpmovsxdq %ymm1, %zmm1
+; KNL-NEXT:    vpsllq $2, %zmm1, %zmm1
+; KNL-NEXT:    vpaddq %zmm1, %zmm0, %zmm0
+; KNL-NEXT:    kshiftrw $8, %k0, %k1
+; KNL-NEXT:    vgatherqps (,%zmm0), %ymm1 {%k1}
+; KNL-NEXT:    vgatherqps (,%zmm0), %ymm2 {%k1}
+; KNL-NEXT:    vinsertf64x4 $1, %ymm1, %zmm2, %zmm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test14:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vpinsrq $1, %rdi, %xmm0, %xmm1
+; SKX-NEXT:    vinserti64x2 $0, %xmm1, %zmm0, %zmm0
+; SKX-NEXT:    vpbroadcastq %xmm0, %zmm0
+; SKX-NEXT:    vmovd %esi, %xmm1
+; SKX-NEXT:    vpbroadcastd %xmm1, %ymm1
+; SKX-NEXT:    vpmovsxdq %ymm1, %zmm1
+; SKX-NEXT:    vpsllq $2, %zmm1, %zmm1
+; SKX-NEXT:    vpaddq %zmm1, %zmm0, %zmm0
+; SKX-NEXT:    kshiftrw $8, %k0, %k1
+; SKX-NEXT:    vgatherqps (,%zmm0), %ymm1 {%k1}
+; SKX-NEXT:    vgatherqps (,%zmm0), %ymm2 {%k1}
+; SKX-NEXT:    vinsertf32x8 $1, %ymm1, %zmm2, %zmm0
+; SKX-NEXT:    retq
 
   %broadcast.splatinsert = insertelement <16 x float*> %vec, float* %base, i32 1
   %broadcast.splat = shufflevector <16 x float*> %broadcast.splatinsert, <16 x float*> undef, <16 x i32> zeroinitializer
 
   %gep.random = getelementptr float, <16 x float*> %broadcast.splat, i32 %ind
 
   %res = call <16 x float> @llvm.masked.gather.v16f32(<16 x float*> %gep.random, i32 4, <16 x i1> undef, <16 x float> undef)
   ret <16 x float>%res
 }
 
+declare <4 x float> @llvm.masked.gather.v4f32(<4 x float*>, i32, <4 x i1>, <4 x float>)
+declare <4 x double> @llvm.masked.gather.v4f64(<4 x double*>, i32, <4 x i1>, <4 x double>)
+declare <2 x double> @llvm.masked.gather.v2f64(<2 x double*>, i32, <2 x i1>, <2 x double>)
+
+; Mask that requires type legalization on KNL
+; Gather smaller than existing instruction
+define <4 x float> @test15(float* %base, <4 x i32> %ind, <4 x i1> %mask) {
+; KNL-LABEL: test15:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vxorps %xmm2, %xmm2, %xmm2
+; KNL-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
+; KNL-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; KNL-NEXT:    vpmovsxdq %ymm0, %zmm2
+; KNL-NEXT:    vpmovsxdq %ymm1, %zmm0
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm0, %zmm0
+; KNL-NEXT:    vptestmq %zmm0, %zmm0, %k1
+; KNL-NEXT:    vgatherqps (%rdi,%zmm2,4), %ymm0 {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test15:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vpmovd2m %xmm1, %k1
+; SKX-NEXT:    vgatherdps (%rdi,%xmm0,4), %xmm1 {%k1}
+; SKX-NEXT:    vmovaps %zmm1, %zmm0
+; SKX-NEXT:    retq
+
+  %sext_ind = sext <4 x i32> %ind to <4 x i64>
+  %gep.random = getelementptr float, float* %base, <4 x i64> %sext_ind
+  %res = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %gep.random, i32 4, <4 x i1> %mask, <4 x float> undef)
+  ret <4 x float>%res
+}
+
+; Mask that requires type legalization on KNL
+; Gather smaller than existing instruction
+define <4 x double> @test16(double* %base, <4 x i32> %ind, <4 x i1> %mask, <4 x double> %src0) {
+; KNL-LABEL: test16:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vxorps %xmm3, %xmm3, %xmm3
+; KNL-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
+; KNL-NEXT:    vinsertf128 $1, %xmm0, %ymm0, %ymm0
+; KNL-NEXT:    vpmovsxdq %ymm0, %zmm0
+; KNL-NEXT:    vpmovsxdq %ymm1, %zmm1
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm1, %zmm1
+; KNL-NEXT:    vptestmq %zmm1, %zmm1, %k1
+; KNL-NEXT:    vgatherqpd (%rdi,%zmm0,8), %zmm2 {%k1}
+; KNL-NEXT:    vmovaps %zmm2, %zmm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test16:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vpmovd2m %xmm1, %k1
+; SKX-NEXT:    vgatherdpd (%rdi,%xmm0,8), %ymm2 {%k1}
+; SKX-NEXT:    vmovaps %zmm2, %zmm0
+; SKX-NEXT:    retq
+
+  %sext_ind = sext <4 x i32> %ind to <4 x i64>
+  %gep.random = getelementptr double, double* %base, <4 x i64> %sext_ind
+  %res = call <4 x double> @llvm.masked.gather.v4f64(<4 x double*> %gep.random, i32 4, <4 x i1> %mask, <4 x double> %src0)
+  ret <4 x double>%res
+}
+
+define <2 x double> @test17(double* %base, <2 x i32> %ind, <2 x i1> %mask, <2 x double> %src0) {
+; KNL-LABEL: test17:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm3 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm0, %zmm3, %zmm0
+; KNL-NEXT:    vpxord %zmm3, %zmm3, %zmm3
+; KNL-NEXT:    vinserti32x4 $0, %xmm1, %zmm3, %zmm1
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm1, %zmm1
+; KNL-NEXT:    vptestmq %zmm1, %zmm1, %k1
+; KNL-NEXT:    vgatherqpd (%rdi,%zmm0,8), %zmm2 {%k1}
+; KNL-NEXT:    vmovaps %zmm2, %zmm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test17:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vpmovq2m %xmm1, %k1
+; SKX-NEXT:    vgatherqpd (%rdi,%xmm0,8), %xmm2 {%k1}
+; SKX-NEXT:    vmovaps %zmm2, %zmm0
+; SKX-NEXT:    retq
+
+  %sext_ind = sext <2 x i32> %ind to <2 x i64>
+  %gep.random = getelementptr double, double* %base, <2 x i64> %sext_ind
+  %res = call <2 x double> @llvm.masked.gather.v2f64(<2 x double*> %gep.random, i32 4, <2 x i1> %mask, <2 x double> %src0)
+  ret <2 x double>%res
+}
+
+declare void @llvm.masked.scatter.v4i32(<4 x i32> , <4 x i32*> , i32 , <4 x i1> )
+declare void @llvm.masked.scatter.v4f64(<4 x double> , <4 x double*> , i32 , <4 x i1> )
+declare void @llvm.masked.scatter.v2i64(<2 x i64> , <2 x i64*> , i32 , <2 x i1> )
+declare void @llvm.masked.scatter.v2i32(<2 x i32> , <2 x i32*> , i32 , <2 x i1> )
+declare void @llvm.masked.scatter.v2f32(<2 x float> , <2 x float*> , i32 , <2 x i1> )
+
+define void @test18(<4 x i32>%a1, <4 x i32*> %ptr, <4 x i1>%mask) {
+; KNL-LABEL: test18:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vinserti64x4 $1, %ymm1, %zmm1, %zmm1
+; KNL-NEXT:    vxorps %xmm3, %xmm3, %xmm3
+; KNL-NEXT:    vinsertf128 $1, %xmm3, %ymm2, %ymm2
+; KNL-NEXT:    vpmovsxdq %ymm2, %zmm2
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm2, %zmm2
+; KNL-NEXT:    vptestmq %zmm2, %zmm2, %k1
+; KNL-NEXT:    vpscatterqd %ymm0, (,%zmm1) {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test18:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vpmovd2m %xmm2, %k1
+; SKX-NEXT:    vpscatterqd %xmm0, (,%ymm1) {%k1}
+; SKX-NEXT:    retq
+  call void @llvm.masked.scatter.v4i32(<4 x i32> %a1, <4 x i32*> %ptr, i32 4, <4 x i1> %mask)
+  ret void
+}
 
+define void @test19(<4 x double>%a1, double* %ptr, <4 x i1>%mask, <4 x i64> %ind) {
+; KNL-LABEL: test19:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vinserti64x4 $1, %ymm2, %zmm2, %zmm2
+; KNL-NEXT:    vxorps %xmm3, %xmm3, %xmm3
+; KNL-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
+; KNL-NEXT:    vpmovsxdq %ymm1, %zmm1
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm1, %zmm1
+; KNL-NEXT:    vptestmq %zmm1, %zmm1, %k1
+; KNL-NEXT:    vscatterqpd %zmm0, (%rdi,%zmm2,8) {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test19:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vpmovd2m %xmm1, %k1
+; SKX-NEXT:    vscatterqpd %ymm0, (%rdi,%ymm2,8) {%k1}
+; SKX-NEXT:    retq
+  %gep = getelementptr double, double* %ptr, <4 x i64> %ind
+  call void @llvm.masked.scatter.v4f64(<4 x double> %a1, <4 x double*> %gep, i32 8, <4 x i1> %mask)
+  ret void
+}
+
+; Data type requires widening
+define void @test20(<2 x float>%a1, <2 x float*> %ptr, <2 x i1>%mask) {
+; KNL-LABEL: test20:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm3 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm1, %zmm3, %zmm1
+; KNL-NEXT:    vpxord %zmm3, %zmm3, %zmm3
+; KNL-NEXT:    vinserti32x4 $0, %xmm2, %zmm3, %zmm2
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm2, %zmm2
+; KNL-NEXT:    vptestmq %zmm2, %zmm2, %k1
+; KNL-NEXT:    vscatterqps %ymm0, (,%zmm1) {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test20:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vinserti64x2 $1, %xmm1, %ymm1, %ymm1
+; SKX-NEXT:    vpmovq2m %xmm2, %k0
+; SKX-NEXT:    kshiftlw $2, %k0, %k0
+; SKX-NEXT:    kshiftrw $2, %k0, %k1
+; SKX-NEXT:    vscatterqps %xmm0, (,%ymm1) {%k1}
+; SKX-NEXT:    retq
+  call void @llvm.masked.scatter.v2f32(<2 x float> %a1, <2 x float*> %ptr, i32 4, <2 x i1> %mask)
+  ret void
+}
+
+; Data type requires promotion
+define void @test21(<2 x i32>%a1, <2 x i32*> %ptr, <2 x i1>%mask) {
+; KNL-LABEL: test21:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm3 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm1, %zmm3, %zmm1
+; KNL-NEXT:    vpxord %zmm3, %zmm3, %zmm3
+; KNL-NEXT:    vinserti32x4 $0, %xmm2, %zmm3, %zmm2
+; KNL-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm2, %zmm2
+; KNL-NEXT:    vptestmq %zmm2, %zmm2, %k1
+; KNL-NEXT:    vpscatterqd %ymm0, (,%zmm1) {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test21:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vinserti64x2 $1, %xmm1, %ymm1, %ymm1
+; SKX-NEXT:    vpmovq2m %xmm2, %k0
+; SKX-NEXT:    kshiftlw $2, %k0, %k0
+; SKX-NEXT:    kshiftrw $2, %k0, %k1
+; SKX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; SKX-NEXT:    vpscatterqd %xmm0, (,%ymm1) {%k1}
+; SKX-NEXT:    retq
+  call void @llvm.masked.scatter.v2i32(<2 x i32> %a1, <2 x i32*> %ptr, i32 4, <2 x i1> %mask)
+  ret void
+}
+
+; The result type requires widening
+declare <2 x float> @llvm.masked.gather.v2f32(<2 x float*>, i32, <2 x i1>, <2 x float>)
+
+define <2 x float> @test22(float* %base, <2 x i32> %ind, <2 x i1> %mask, <2 x float> %src0) {
+; KNL-LABEL: test22:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm3 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm0, %zmm3, %zmm0
+; KNL-NEXT:    vpxord %zmm3, %zmm3, %zmm3
+; KNL-NEXT:    vinserti32x4 $0, %xmm1, %zmm3, %zmm1
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm1, %zmm1
+; KNL-NEXT:    vptestmq %zmm1, %zmm1, %k1
+; KNL-NEXT:    vgatherqps (%rdi,%zmm0,4), %ymm2 {%k1}
+; KNL-NEXT:    vmovaps %zmm2, %zmm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test22:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vinserti64x2 $1, %xmm0, %ymm0, %ymm0
+; SKX-NEXT:    vpmovq2m %xmm1, %k0
+; SKX-NEXT:    kshiftlw $2, %k0, %k0
+; SKX-NEXT:    kshiftrw $2, %k0, %k1
+; SKX-NEXT:    vgatherqps (%rdi,%ymm0,4), %xmm2 {%k1}
+; SKX-NEXT:    vmovaps %zmm2, %zmm0
+; SKX-NEXT:    retq
+  %sext_ind = sext <2 x i32> %ind to <2 x i64>
+  %gep.random = getelementptr float, float* %base, <2 x i64> %sext_ind
+  %res = call <2 x float> @llvm.masked.gather.v2f32(<2 x float*> %gep.random, i32 4, <2 x i1> %mask, <2 x float> %src0)
+  ret <2 x float>%res
+}
+
+declare <2 x i32> @llvm.masked.gather.v2i32(<2 x i32*>, i32, <2 x i1>, <2 x i32>)
+declare <2 x i64> @llvm.masked.gather.v2i64(<2 x i64*>, i32, <2 x i1>, <2 x i64>)
+
+define <2 x i32> @test23(i32* %base, <2 x i32> %ind, <2 x i1> %mask, <2 x i32> %src0) {
+; KNL-LABEL: test23:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm3 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm0, %zmm3, %zmm0
+; KNL-NEXT:    vpxord %zmm3, %zmm3, %zmm3
+; KNL-NEXT:    vinserti32x4 $0, %xmm1, %zmm3, %zmm1
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm1, %zmm1
+; KNL-NEXT:    vptestmq %zmm1, %zmm1, %k1
+; KNL-NEXT:    vpgatherqq (%rdi,%zmm0,8), %zmm2 {%k1}
+; KNL-NEXT:    vmovaps %zmm2, %zmm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test23:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vpmovq2m %xmm1, %k1
+; SKX-NEXT:    vpgatherqq (%rdi,%xmm0,8), %xmm2 {%k1}
+; SKX-NEXT:    vmovaps %zmm2, %zmm0
+; SKX-NEXT:    retq
+  %sext_ind = sext <2 x i32> %ind to <2 x i64>
+  %gep.random = getelementptr i32, i32* %base, <2 x i64> %sext_ind
+  %res = call <2 x i32> @llvm.masked.gather.v2i32(<2 x i32*> %gep.random, i32 4, <2 x i1> %mask, <2 x i32> %src0)
+  ret <2 x i32>%res
+}
+
+define <2 x i32> @test24(i32* %base, <2 x i32> %ind) {
+; KNL-LABEL: test24:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm1 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm0, %zmm1, %zmm1
+; KNL-NEXT:    movb $3, %al
+; KNL-NEXT:    movzbl %al, %eax
+; KNL-NEXT:    kmovw %eax, %k1
+; KNL-NEXT:    vpgatherqq (%rdi,%zmm1,8), %zmm0 {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test24:
+; SKX:       ## BB#0:
+; SKX-NEXT:    kxnorw %k1, %k1, %k1
+; SKX-NEXT:    vpgatherqq (%rdi,%xmm0,8), %xmm1 {%k1}
+; SKX-NEXT:    vmovaps %zmm1, %zmm0
+; SKX-NEXT:    retq
+  %sext_ind = sext <2 x i32> %ind to <2 x i64>
+  %gep.random = getelementptr i32, i32* %base, <2 x i64> %sext_ind
+  %res = call <2 x i32> @llvm.masked.gather.v2i32(<2 x i32*> %gep.random, i32 4, <2 x i1> <i1 true, i1 true>, <2 x i32> undef)
+  ret <2 x i32>%res
+}
+
+define <2 x i64> @test25(i64* %base, <2 x i32> %ind, <2 x i1> %mask, <2 x i64> %src0) {
+; KNL-LABEL: test25:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm3 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm0, %zmm3, %zmm0
+; KNL-NEXT:    vpxord %zmm3, %zmm3, %zmm3
+; KNL-NEXT:    vinserti32x4 $0, %xmm1, %zmm3, %zmm1
+; KNL-NEXT:    vpandq {{.*}}(%rip){1to8}, %zmm1, %zmm1
+; KNL-NEXT:    vptestmq %zmm1, %zmm1, %k1
+; KNL-NEXT:    vpgatherqq (%rdi,%zmm0,8), %zmm2 {%k1}
+; KNL-NEXT:    vmovaps %zmm2, %zmm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test25:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vpmovq2m %xmm1, %k1
+; SKX-NEXT:    vpgatherqq (%rdi,%xmm0,8), %xmm2 {%k1}
+; SKX-NEXT:    vmovaps %zmm2, %zmm0
+; SKX-NEXT:    retq
+  %sext_ind = sext <2 x i32> %ind to <2 x i64>
+  %gep.random = getelementptr i64, i64* %base, <2 x i64> %sext_ind
+  %res = call <2 x i64> @llvm.masked.gather.v2i64(<2 x i64*> %gep.random, i32 8, <2 x i1> %mask, <2 x i64> %src0)
+  ret <2 x i64>%res
+}
+
+; Legal on SKX; all-ones mask
+define <2 x i64> @test26(i64* %base, <2 x i32> %ind, <2 x i64> %src0) {
+; KNL-LABEL: test26:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm2 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm0, %zmm2, %zmm0
+; KNL-NEXT:    movb $3, %al
+; KNL-NEXT:    movzbl %al, %eax
+; KNL-NEXT:    kmovw %eax, %k1
+; KNL-NEXT:    vpgatherqq (%rdi,%zmm0,8), %zmm1 {%k1}
+; KNL-NEXT:    vmovaps %zmm1, %zmm0
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test26:
+; SKX:       ## BB#0:
+; SKX-NEXT:    kxnorw %k1, %k1, %k1
+; SKX-NEXT:    vpgatherqq (%rdi,%xmm0,8), %xmm1 {%k1}
+; SKX-NEXT:    vmovaps %zmm1, %zmm0
+; SKX-NEXT:    retq
+  %sext_ind = sext <2 x i32> %ind to <2 x i64>
+  %gep.random = getelementptr i64, i64* %base, <2 x i64> %sext_ind
+  %res = call <2 x i64> @llvm.masked.gather.v2i64(<2 x i64*> %gep.random, i32 8, <2 x i1> <i1 true, i1 true>, <2 x i64> %src0)
+  ret <2 x i64>%res
+}
+
+; Result type requires widening; all-ones mask
+define <2 x float> @test27(float* %base, <2 x i32> %ind) {
+; KNL-LABEL: test27:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm1 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm0, %zmm1, %zmm1
+; KNL-NEXT:    movb $3, %al
+; KNL-NEXT:    movzbl %al, %eax
+; KNL-NEXT:    kmovw %eax, %k1
+; KNL-NEXT:    vgatherqps (%rdi,%zmm1,4), %ymm0 {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test27:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vinserti64x2 $1, %xmm0, %ymm0, %ymm1
+; SKX-NEXT:    movb $3, %al
+; SKX-NEXT:    kmovb %eax, %k1
+; SKX-NEXT:    vgatherqps (%rdi,%ymm1,4), %xmm0 {%k1}
+; SKX-NEXT:    retq
+  %sext_ind = sext <2 x i32> %ind to <2 x i64>
+  %gep.random = getelementptr float, float* %base, <2 x i64> %sext_ind
+  %res = call <2 x float> @llvm.masked.gather.v2f32(<2 x float*> %gep.random, i32 4, <2 x i1> <i1 true, i1 true>, <2 x float> undef)
+  ret <2 x float>%res
+}
+
+; Data type requires promotion, mask is all-ones
+define void @test28(<2 x i32>%a1, <2 x i32*> %ptr) {
+; KNL-LABEL: test28:
+; KNL:       ## BB#0:
+; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm2 = [0,1,0,1,0,1,0,1]
+; KNL-NEXT:    vpermq %zmm1, %zmm2, %zmm1
+; KNL-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; KNL-NEXT:    movb $3, %al
+; KNL-NEXT:    movzbl %al, %eax
+; KNL-NEXT:    kmovw %eax, %k1
+; KNL-NEXT:    vpscatterqd %ymm0, (,%zmm1) {%k1}
+; KNL-NEXT:    retq
+;
+; SKX-LABEL: test28:
+; SKX:       ## BB#0:
+; SKX-NEXT:    vinserti64x2 $1, %xmm1, %ymm1, %ymm1
+; SKX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; SKX-NEXT:    movb $3, %al
+; SKX-NEXT:    kmovb %eax, %k1
+; SKX-NEXT:    vpscatterqd %xmm0, (,%ymm1) {%k1}
+; SKX-NEXT:    retq
+  call void @llvm.masked.scatter.v2i32(<2 x i32> %a1, <2 x i32*> %ptr, i32 4, <2 x i1> <i1 true, i1 true>)
+  ret void
+}