RegCall - Handling v64i1 in 32/64 bit target

[oren_ben_simhon]

Summary

Register Calling Convention defines a new behavior for v64i1 types.
This type should be saved in GPR.
However for 32 bit machine we need to split the value into 2 GPRs (because each is 32 bit).

[oren_ben_simhon]

A kindly reminder. I will appreciate your inputs on the review.

[igorb]

Hi, please change
Variable names should be camel case, and start with an upper case letter (e.g. Leader or Boats).

[delena]

lib/Target/X86/X86CallingConv.h

27

This function is too big to be inline. Can you move it to .c** file?

34

dot at the end of the sentence.

35

What is this list for?

39

std::vector is not "popular" in the LLVM code. SmallVector can be used instead.

50

Why 2 ? Please add comments here.

53

Does this loop allays have 2 iterations?

[oren_ben_simhon]

lib/Target/X86/X86CallingConv.h

35

This is the list of registers that can be used to pass the value, according to regcall calling convention.
Only if two registers will be available the value will be passed.

[oren_ben_simhon]

Implementing comments posted until 11/12

[delena]

lib/Target/X86/X86CallingConv.cpp

26 ↗

(On Diff #77745)

#defines like this are not allowed in LLVM, I think.

43 ↗

(On Diff #77745)

How the case is handled when there is no available registers? I did not see assign-to stack for this type.

49 ↗

(On Diff #77745)

All variables names should start with capital letter

lib/Target/X86/X86CallingConv.td

19

do you implement CC_X86_RegCall_Error for the types that will come in the next commit?

lib/Target/X86/X86ISelLowering.cpp

194–1

I don't understand this change

194–1

Why do you drop SIGN_EXTEND?

195

wrong line alignment

196

Please add 2 words about the Root, VA and NextVA arguments in doxygen format. And InFlag role is not clear.

196

Why do we need this additional condition? Please add comments.

199

wrong line alignment, function name lower case

200

BMI implies BWI

201

the original condition for this operation was VA.getValVT().getScalarType() == MVT::i1

234

you can combine this "if" with the previous one, right?

255

do not pass calling convention here

262

compilation will fail for unreferenced variable in the buildbot.

269

unreachable or assert - ?

273

The function may return SDValue. But you do many asserts and one bitcast. Do you really need a function?

[oren_ben_simhon]

lib/Target/X86/X86CallingConv.cpp

43 ↗

(On Diff #77745)

Since v64i1 is promoted to i64, the following rule will assign to stack (in case we don't have available registers):

// In 64 bit, assign 64/32 bit values to 8 byte stack
CCIfSubtarget<"is64Bit()", CCIfType<[i32, i64, f32, f64], 
  CCAssignToStack<8, 8>>>,

lib/Target/X86/X86CallingConv.td

19

Yes

lib/Target/X86/X86ISelLowering.cpp

194–1

LLVM Coding Standards prefer all variables to be capitalized including iterator.
They share the following example:

for (unsigned I = 0, E = List.size(); I != E; ++I)
    if (List[I]->isFoo())
      return true;

194–1

It resides inside the function

195

Clang-format did it

199

The alignment was created by clang-format.
I will move to lower case.

234

I can do so but it will require duplication of the following lines:

// Convert the i32 type into v32i1 type
Lo = DAG.getBitcast(MVT::v32i1, ArgValue);

...

// Convert the i32 type into v32i1 type
Hi = DAG.getBitcast(MVT::v32i1, ArgValue);

So i prefered to seprate it into two different if statements.

262

It is referenced in the next line. It will be referenced in the future as well.

269

This decision was made due to future design.

[delena]

lib/Target/X86/X86CallingConv.td

19

I think we can stay with TODO comment till the next commit and remove CC_X86_RegCall_Error

[delena]

lib/Target/X86/X86ISelLowering.cpp

234

Use ArgValueLo and ArgValueHi inside if-else blocks

[oren_ben_simhon]

Implemented comments posted until 11/16.

[oren_ben_simhon]

lib/Target/X86/X86CallingConv.td

19

The custom function was introduced in the previous commit and was requested by other reviewers.
The community wants a user (of this version) to have an error message if it uses something that is not yet supported.
I agree with this approach and prefer to leave this as is.

[oren_ben_simhon]

Implemented additional 2 comments posted by delena.

[delena]

lib/Target/X86/X86ISelLowering.cpp

2411

Please refer to my comment here

lib/Target/X86/X86CallingConv.h

 #define LLVM_LIB_TARGET_X86_X86CALLINGCONV_H
 
 #include "MCTargetDesc/X86MCTargetDesc.h"
 #include "llvm/CodeGen/CallingConvLower.h"
 #include "llvm/IR/CallingConv.h"
 
 namespace llvm {
 
+/// When regcall calling convention compiled to 32 bit arch, special treatment
+/// is required for 64 bit masks.
+/// The value should be assigned to two GPRs.
+inline bool CC_X86_32_RegCall_Assign2Regs(unsigned &ValNo, MVT &ValVT,
+                                          MVT &LocVT,
+                                          CCValAssign::LocInfo &LocInfo,
+                                          ISD::ArgFlagsTy &ArgFlags,
+                                          CCState &State) {
+
+  // See also the regcall calling convention ABI for the full list of possible
+  // registers for mask types
+  static const std::vector<MCPhysReg> RegList{X86::EAX, X86::ECX, X86::EDX,
+                                              X86::EDI, X86::ESI};
+
+  // The vector will save all the available registers for allocation
+  std::vector<unsigned> availableRegs;
+
+  // searching for the available registers
+  for (unsigned idx = 0, end = RegList.size(); idx < end; ++idx) {
+    if (!State.isAllocated(RegList[idx]))
+      availableRegs.push_back(RegList[idx]);
+  }
+
+  if (availableRegs.size() < 2)
+    return false; // Not enough free registers - continue the search
+
+  unsigned remainingRegs = 2;
+
+  // Allocating the available registers
+  for (std::vector<unsigned>::iterator it = availableRegs.begin(),
+                                       end = availableRegs.end();
+       (it != end) && (remainingRegs > 0); ++it, --remainingRegs) {
+
+    // Marking the register as located
+    unsigned reg = State.AllocateReg(*it);
+
+    // Since we previously made sure that 2 registers are available
+    // we expect that a real register number will be returned
+    assert(reg && "Expecting a register will be available");
+
+    // Assign the value to the allocated register
+    State.addLoc(CCValAssign::getCustomReg(ValNo, ValVT, *it, LocVT, LocInfo));
+  }
+
+  // Make sure that all required registers were allocated
+  assert(0 == remainingRegs && "Not all registers were allocated");
+
+  // Successful in allocating regsiters - stop scanning next rules
+  return true;
+}
+
 inline bool CC_X86_32_VectorCallIndirect(unsigned &ValNo, MVT &ValVT,
                                          MVT &LocVT,
                                          CCValAssign::LocInfo &LocInfo,
                                          ISD::ArgFlagsTy &ArgFlags,
                                          CCState &State) {
   // Similar to CCPassIndirect, with the addition of inreg.
   LocVT = MVT::i32;
   LocInfo = CCValAssign::Indirect;

lib/Target/X86/X86CallingConv.td

 // architectures.
 //
 //===----------------------------------------------------------------------===//
 
 /// CCIfSubtarget - Match if the current subtarget has a feature F.
 class CCIfSubtarget<string F, CCAction A>
     : CCIf<!strconcat("static_cast<const X86Subtarget&>"
                        "(State.getMachineFunction().getSubtarget()).", F),
            A>;
 
 // Register classes for RegCall
 class RC_X86_RegCall {
   list<Register> GPR_8 = [];
   list<Register> GPR_16 = [];
   list<Register> GPR_32 = [];
   list<Register> GPR_64 = [];
   list<Register> XMM = [];
// Handles byval parameters.
     CCIfByVal<CCPassByVal<4, 4>>,
 
     // Promote i1/i8/i16 arguments to i32.
     CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
 
     // bool, char, int, enum, long, pointer --> GPR
     CCIfType<[i32], CCAssignToReg<RC.GPR_32>>,
 
-    // TODO: Handle the case of mask types (v*i1)
-    // TODO: Handle the case of 32 bit machine with v64i1 argument 
-    //       (split to 2 registers)
-    CCIfType<[v8i1, v16i1, v32i1, v64i1], CCCustom<"CC_X86_RegCall_Error">>,
-
     // long long, __int64 --> GPR
     CCIfType<[i64], CCAssignToReg<RC.GPR_64>>,
 
+    // __mmask64 (v64i1) --> GPR64 (for x64) or 2 x GPR32 (for IA32)
+    CCIfType<[v64i1], CCPromoteToType<i64>>,
+    CCIfSubtarget<"is64Bit()", CCIfType<[i64], 
+      CCAssignToReg<RC.GPR_64>>>,
+    CCIfSubtarget<"is32Bit()", CCIfType<[i64], 
+      CCCustom<"CC_X86_32_RegCall_Assign2Regs">>>,
+
+    // TODO: Handle the case of mask types (v*i1)
+    CCIfType<[v8i1, v16i1, v32i1], CCCustom<"CC_X86_RegCall_Error">>,
+
     // TODO: Handle the case of long double (f80)
     CCIfType<[f80], CCCustom<"CC_X86_RegCall_Error">>,
 
     // float, double, float128 --> XMM
     // In the case of SSE disabled --> save to stack
     CCIfType<[f32, f64, f128], 
       CCIfSubtarget<"hasSSE1()", CCAssignToReg<RC.XMM>>>,
 
// Handles byval parameters.
     // If no register was found -> assign to stack
 
     // In 64 bit, assign 64/32 bit values to 8 byte stack
     CCIfSubtarget<"is64Bit()", CCIfType<[i32, i64, f32, f64], 
       CCAssignToStack<8, 8>>>,
 
     // In 32 bit, assign 64/32 bit values to 8/4 byte stack
     CCIfType<[i32, f32], CCAssignToStack<4, 4>>,
-    CCIfType<[f64], CCAssignToStack<8, 4>>,
+    CCIfType<[i64, f64], CCAssignToStack<8, 4>>,
 
     // MMX type gets 8 byte slot in stack , while alignment depends on target
     CCIfSubtarget<"is64Bit()", CCIfType<[x86mmx], CCAssignToStack<8, 8>>>,
     CCIfType<[x86mmx], CCAssignToStack<8, 4>>,
 
     // float 128 get stack slots whose size and alignment depends 
     // on the subtarget.
     CCIfType<[f128], CCAssignToStack<0, 0>>,
def RetCC_#NAME : CallingConv<[
     // Promote i1 arguments to i8.
     CCIfType<[i1], CCPromoteToType<i8>>,
 
     // bool, char, int, enum, long, pointer --> GPR
     CCIfType<[i8], CCAssignToReg<RC.GPR_8>>,
     CCIfType<[i16], CCAssignToReg<RC.GPR_16>>,
     CCIfType<[i32], CCAssignToReg<RC.GPR_32>>,
 
-    // TODO: Handle the case of mask types (v*i1)
-    // TODO: Handle the case of 32 bit machine with v64i1 argument 
-    //  (split to 2 registers)
-    CCIfType<[v8i1, v16i1, v32i1, v64i1], CCCustom<"CC_X86_RegCall_Error">>,
-
     // long long, __int64 --> GPR
     CCIfType<[i64], CCAssignToReg<RC.GPR_64>>,
 
+    // __mmask64 (v64i1) --> GPR64 (for x64) or 2 x GPR32 (for IA32)
+    CCIfType<[v64i1], CCPromoteToType<i64>>,
+    CCIfSubtarget<"is64Bit()", CCIfType<[i64], 
+      CCAssignToReg<RC.GPR_64>>>,
+    CCIfSubtarget<"is32Bit()", CCIfType<[i64], 
+      CCCustom<"CC_X86_32_RegCall_Assign2Regs">>>,
+
+    // TODO: Handle the case of mask types (v*i1)
+    CCIfType<[v8i1, v16i1, v32i1], CCCustom<"CC_X86_RegCall_Error">>,
+
     // long double --> FP
     CCIfType<[f80], CCAssignToReg<[FP0]>>,
 
     // float, double, float128 --> XMM
     CCIfType<[f32, f64, f128], 
       CCIfSubtarget<"hasSSE1()", CCAssignToReg<RC.XMM>>>,
 
     // __m128, __m128i, __m128d --> XMM

lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
   setOperationAction(ISD::SINT_TO_FP       , MVT::i1   , Promote);
   setOperationAction(ISD::SINT_TO_FP       , MVT::i8   , Promote);
 
   if (!Subtarget.useSoftFloat()) {
     // SSE has no i16 to fp conversion, only i32.
     if (X86ScalarSSEf32) {
       setOperationAction(ISD::SINT_TO_FP     , MVT::i16  , Promote);
       // f32 and f64 cases are Legal, f80 case is not
       setOperationAction(ISD::SINT_TO_FP     , MVT::i32  , Custom);
     } else {
       setOperationAction(ISD::SINT_TO_FP     , MVT::i16  , Custom);
       setOperationAction(ISD::SINT_TO_FP     , MVT::i32  , Custom);
     }
   } else {
     setOperationAction(ISD::SINT_TO_FP     , MVT::i16  , Promote);
     setOperationAction(ISD::SINT_TO_FP     , MVT::i32  , Promote);
   }
 
   // Promote i1/i8 FP_TO_SINT to larger FP_TO_SINTS's, as X86 doesn't have
   // this operation.
   setOperationAction(ISD::FP_TO_SINT       , MVT::i1   , Promote);
   setOperationAction(ISD::FP_TO_SINT       , MVT::i8   , Promote);
 
if (!Subtarget.useSoftFloat()) {
     setOperationAction(ISD::FP_TO_SINT     , MVT::i32  , Expand);
     setOperationAction(ISD::FP_TO_SINT     , MVT::i64  , Expand);
   }
 
   // Handle FP_TO_UINT by promoting the destination to a larger signed
   // conversion.
   setOperationAction(ISD::FP_TO_UINT       , MVT::i1   , Promote);
   setOperationAction(ISD::FP_TO_UINT       , MVT::i8   , Promote);
   setOperationAction(ISD::FP_TO_UINT       , MVT::i16  , Promote);
 
   if (Subtarget.is64Bit()) {
     if (!Subtarget.useSoftFloat() && Subtarget.hasAVX512()) {
       // FP_TO_UINT-i32/i64 is legal for f32/f64, but custom for f80.
       setOperationAction(ISD::FP_TO_UINT   , MVT::i32  , Custom);
       setOperationAction(ISD::FP_TO_UINT   , MVT::i64  , Custom);
     } else {
       setOperationAction(ISD::FP_TO_UINT   , MVT::i32  , Promote);
       setOperationAction(ISD::FP_TO_UINT   , MVT::i64  , Expand);
     }
   } else if (!Subtarget.useSoftFloat()) {
     // Since AVX is a superset of SSE3, only check for SSE here.
     if (Subtarget.hasSSE1() && !Subtarget.hasSSE3())
       // Expand FP_TO_UINT into a select.
       // FIXME: We would like to use a Custom expander here eventually to do
       // the optimal thing for SSE vs. the default expansion in the legalizer.
       setOperationAction(ISD::FP_TO_UINT   , MVT::i32  , Expand);
     else
       // With AVX512 we can use vcvts[ds]2usi for f32/f64->i32, f80 is custom.
       // With SSE3 we can use fisttpll to convert to a signed i64; without
       // SSE, we're stuck with a fistpll.
       setOperationAction(ISD::FP_TO_UINT   , MVT::i32  , Custom);
 
     setOperationAction(ISD::FP_TO_UINT     , MVT::i64  , Custom);
   }
 
   // TODO: when we have SSE, these could be more efficient, by using movd/movq.
   if (!X86ScalarSSEf64) {
     setOperationAction(ISD::BITCAST        , MVT::f32  , Expand);
     setOperationAction(ISD::BITCAST        , MVT::i32  , Expand);
     if (Subtarget.is64Bit()) {
       setOperationAction(ISD::BITCAST      , MVT::f64  , Expand);
       // Without SSE, i64->f64 goes through memory.
       setOperationAction(ISD::BITCAST      , MVT::i64  , Expand);
     }
   } else if (!Subtarget.is64Bit())
     setOperationAction(ISD::BITCAST      , MVT::i64  , Custom);
 
   // Scalar integer divide and remainder are lowered to use operations that
   // produce two results, to match the available instructions. This exposes
   // the two-result form to trivial CSE, which is able to combine x/y and x%y
   // into a single instruction.
   //
   // Scalar integer multiply-high is also lowered to use two-result
   // operations, to match the available instructions. However, plain multiply
   // (low) operations are left as Legal, as there are single-result
   // instructions for this in x86. Using the two-result multiply instructions
bool X86TargetLowering::CanLowerReturn(
   return CCInfo.CheckReturn(Outs, RetCC_X86);
 }
 
 const MCPhysReg *X86TargetLowering::getScratchRegisters(CallingConv::ID) const {
   static const MCPhysReg ScratchRegs[] = { X86::R11, 0 };
   return ScratchRegs;
 }
 
+/// Lowers masks values (v*i1) to the local register values
+static void LowerMasksToReg(SDValue &ValToCopy, const EVT &ValLoc,
+                            CallingConv::ID CallConv, const SDLoc &dl,
+                            SelectionDAG &DAG) {
+  EVT ValVT = ValToCopy.getValueType();
+
+  if (ValVT == MVT::v64i1 && ValLoc == MVT::i64) {
+    // One stage lowering is required
+    // bitcast:   v64i1 -> i64
+    assert(CallingConv::X86_RegCall == CallConv &&
+           "expecting regcall calling convention");
+    ValToCopy = DAG.getBitcast(MVT::i64, ValToCopy);
+  } else
+    ValToCopy = DAG.getNode(ISD::SIGN_EXTEND, dl, ValLoc, ValToCopy);
+}
+
+/// Breaks v64i1 value into two registers and adds the new node to the DAG
+static void Passv64i1ArgInRegs(
+    const SDLoc &dl, SelectionDAG &DAG, SDValue Chain, SDValue &Arg,
+    SmallVector<std::pair<unsigned, SDValue>, 8> &RegsToPass, CCValAssign &VA,
+    CCValAssign &NextVA, const X86Subtarget &Subtarget) {
+  assert((Subtarget.hasBWI() || Subtarget.hasBMI()) &&
+         "Expected AVX512BW or AVX512BMI target!");
+  assert(Subtarget.is32Bit() && "Expecting 32 bit target");
+  assert(Arg.getValueType() == MVT::i64 && "Expecting 64 bit value");
+  assert(VA.isRegLoc() && NextVA.isRegLoc() &&
+         "The value should reside in two registers");
+
+  // Before splitting the value we cast it to i64
+  Arg = DAG.getBitcast(MVT::i64, Arg);
+
+  // Splitting the value into two i32 types
+  SDValue Lo, Hi;
+  Lo = DAG.getNode(ISD::EXTRACT_ELEMENT, dl, MVT::i32, Arg,
+                   DAG.getConstant(0, dl, MVT::i32));
+  Hi = DAG.getNode(ISD::EXTRACT_ELEMENT, dl, MVT::i32, Arg,
+                   DAG.getConstant(1, dl, MVT::i32));
+
+  // Attach the two i32 types into corresponding registers
+  RegsToPass.push_back(std::make_pair(VA.getLocReg(), Lo));
+  RegsToPass.push_back(std::make_pair(NextVA.getLocReg(), Hi));
+}
+
 SDValue
 X86TargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
                                bool isVarArg,
                                const SmallVectorImpl<ISD::OutputArg> &Outs,
                                const SmallVectorImpl<SDValue> &OutVals,
                                const SDLoc &dl, SelectionDAG &DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
   X86MachineFunctionInfo *FuncInfo = MF.getInfo<X86MachineFunctionInfo>();
 
   if (CallConv == CallingConv::X86_INTR && !Outs.empty())
     report_fatal_error("X86 interrupts may not return any value");
 
   SmallVector<CCValAssign, 16> RVLocs;
   CCState CCInfo(CallConv, isVarArg, MF, RVLocs, *DAG.getContext());
   CCInfo.AnalyzeReturn(Outs, RetCC_X86);
 
   SDValue Flag;
   SmallVector<SDValue, 6> RetOps;
   RetOps.push_back(Chain); // Operand #0 = Chain (updated below)
   // Operand #1 = Bytes To Pop
   RetOps.push_back(DAG.getTargetConstant(FuncInfo->getBytesToPopOnReturn(), dl,
                    MVT::i32));
 
   // Copy the result values into the output registers.
-  for (unsigned i = 0, e = RVLocs.size(); i != e; ++i) {
+  for (unsigned i = 0, outsIndex = 0, e = RVLocs.size(); i != e;
+       ++i, ++outsIndex) {
     CCValAssign &VA = RVLocs[i];
     assert(VA.isRegLoc() && "Can only return in registers!");
-    SDValue ValToCopy = OutVals[i];
+    SDValue ValToCopy = OutVals[outsIndex];
     EVT ValVT = ValToCopy.getValueType();
 
     // Promote values to the appropriate types.
     if (VA.getLocInfo() == CCValAssign::SExt)
       ValToCopy = DAG.getNode(ISD::SIGN_EXTEND, dl, VA.getLocVT(), ValToCopy);
     else if (VA.getLocInfo() == CCValAssign::ZExt)
       ValToCopy = DAG.getNode(ISD::ZERO_EXTEND, dl, VA.getLocVT(), ValToCopy);
     else if (VA.getLocInfo() == CCValAssign::AExt) {
       if (ValVT.isVector() && ValVT.getVectorElementType() == MVT::i1)
-        ValToCopy = DAG.getNode(ISD::SIGN_EXTEND, dl, VA.getLocVT(), ValToCopy);
+        LowerMasksToReg(ValToCopy, VA.getLocVT(), CallConv, dl, DAG);
       else
         ValToCopy = DAG.getNode(ISD::ANY_EXTEND, dl, VA.getLocVT(), ValToCopy);
     }
     else if (VA.getLocInfo() == CCValAssign::BCvt)
       ValToCopy = DAG.getBitcast(VA.getLocVT(), ValToCopy);
 
     assert(VA.getLocInfo() != CCValAssign::FPExt &&
            "Unexpected FP-extend for return value.");
if (Subtarget.is64Bit()) {
           // If we don't have SSE2 available, convert to v4f32 so the generated
           // register is legal.
           if (!Subtarget.hasSSE2())
             ValToCopy = DAG.getBitcast(MVT::v4f32, ValToCopy);
         }
       }
     }
 
-    Chain = DAG.getCopyToReg(Chain, dl, VA.getLocReg(), ValToCopy, Flag);
+    SmallVector<std::pair<unsigned, SDValue>, 8> RegsToPass;
+
+    if (VA.needsCustom()) {
+      assert(VA.getValVT() == MVT::v64i1 &&
+             "Currently the only custom case is when we split v64i1 to 2 regs");
+
+      Passv64i1ArgInRegs(dl, DAG, Chain, ValToCopy, RegsToPass, VA, RVLocs[++i],
+                         Subtarget);
+
+      assert(2 == RegsToPass.size() &&
+             "Expecting two registers after Pass64BitArgInRegs");
+    } else {
+      RegsToPass.push_back(std::make_pair(VA.getLocReg(), ValToCopy));
+    }
+
+    // Add nodes to the DAG and add the values into the RetOps list
+    for (auto &Reg : RegsToPass) {
+      Chain = DAG.getCopyToReg(Chain, dl, Reg.first, Reg.second, Flag);
       Flag = Chain.getValue(1);
-    RetOps.push_back(DAG.getRegister(VA.getLocReg(), VA.getLocVT()));
+      RetOps.push_back(DAG.getRegister(Reg.first, Reg.second.getValueType()));
+    }
   }
 
   // Swift calling convention does not require we copy the sret argument
   // into %rax/%eax for the return, and SRetReturnReg is not set for Swift.
 
   // All x86 ABIs require that for returning structs by value we copy
   // the sret argument into %rax/%eax (depending on ABI) for the return.
   // We saved the argument into a virtual register in the entry block,
if (VT == MVT::i1 || (!Darwin && (VT == MVT::i8 || VT == MVT::i16))) {
     // (PR26665).
     ReturnMVT = MVT::i8;
   }
 
   EVT MinVT = getRegisterType(Context, ReturnMVT);
   return VT.bitsLT(MinVT) ? MinVT : VT;
 }
 
+/// Reads two 32 bit registers and creates a 64 bit mask value
+static SDValue Getv64i1Argument(CCValAssign &VA, CCValAssign &NextVA,
+                                SDValue &Root, SelectionDAG &DAG,
+                                const SDLoc &dl, const X86Subtarget &Subtarget,
+                                SDValue *InFlag = nullptr) {
+  assert((Subtarget.hasBWI() || Subtarget.hasBMI()) &&
+         "Expected AVX512BW or AVX512BMI target!");
+  assert(Subtarget.is32Bit() && "Expecting 32 bit target");
+  assert(VA.getValVT() == MVT::v64i1 &&
+         "Expecting first location of 64 bit width type");
+  assert(NextVA.getValVT() == VA.getValVT() &&
+         "The locations should have the same type");
+  assert(VA.isRegLoc() && NextVA.isRegLoc() &&
+         "The values should reside in two registers");
+
+  SDValue Lo, Hi;
+  unsigned Reg;
+  SDValue ArgValue;
+
+  MachineFunction &MF = DAG.getMachineFunction();
+  const TargetRegisterClass *RC = &X86::GR32RegClass;
+
+  // Read a 32 bit value from the register
+  if (nullptr == InFlag) {
+    // When no physical register is present,
+    // create an intermediate virtual register
+    Reg = MF.addLiveIn(VA.getLocReg(), RC);
+    ArgValue = DAG.getCopyFromReg(Root, dl, Reg, MVT::i32);
+  } else {
+    // When a physical register is available read the value from it and glue
+    // the reads together.
+    ArgValue = DAG.getCopyFromReg(Root, dl, VA.getLocReg(), MVT::i32, *InFlag);
+    *InFlag = ArgValue.getValue(2);
+  }
+
+  // Convert the i32 type into v32i1 type
+  Lo = DAG.getBitcast(MVT::v32i1, ArgValue);
+
+  // Read a 32 bit value from the register
+  if (nullptr == InFlag) {
+    // When no physical register is present,
+    // create an intermediate virtual register
+    Reg = MF.addLiveIn(NextVA.getLocReg(), RC);
+    ArgValue = DAG.getCopyFromReg(Root, dl, Reg, MVT::i32);
+  } else {
+    // When a physical register is available read the value from it and glue
+    // the reads together.
+    ArgValue =
+        DAG.getCopyFromReg(Root, dl, NextVA.getLocReg(), MVT::i32, *InFlag);
+    *InFlag = ArgValue.getValue(2);
+  }
+
+  // Convert the i32 type into v32i1 type
+  Hi = DAG.getBitcast(MVT::v32i1, ArgValue);
+
+  // Concantenate the two values together
+  return DAG.getNode(ISD::CONCAT_VECTORS, dl, MVT::v64i1, Lo, Hi);
+}
+
+static void LowerRegToMasks(SDValue &ValReturned, const EVT &ValVT,
+                            const EVT &ValLoc, CallingConv::ID CallConv,
+                            const SDLoc &dl, SelectionDAG &DAG) {
+  assert((ValLoc == MVT::i64 || ValLoc == MVT::i32) &&
+         "Expecting register location of size 32/64 bit");
+  assert(CallingConv::X86_RegCall == CallConv &&
+         "Expecting mask vector to i32/i64 registers in regcall only");
+
+  // Currently not referenced - will be used in other mask lowering
+  (void)dl;
+
+  // In the case of v64i1 no special handling is required due to two reasons:
+  // In 32 bit machine, this case is handled by Get64BitFormalArgument
+  // In 64 bit machine, There is no need to truncate the value only bitcast
+  if (ValVT == MVT::v64i1 && ValLoc == MVT::i32) {
+    llvm_unreachable("Expecting only i64 locations");
+    return;
+  }
+
+  ValReturned = DAG.getBitcast(ValVT, ValReturned);
+}
+
 /// Lower the result values of a call into the
 /// appropriate copies out of appropriate physical registers.
 ///
 SDValue X86TargetLowering::LowerCallResult(
     SDValue Chain, SDValue InFlag, CallingConv::ID CallConv, bool isVarArg,
     const SmallVectorImpl<ISD::InputArg> &Ins, const SDLoc &dl,
     SelectionDAG &DAG, SmallVectorImpl<SDValue> &InVals) const {
 
   // Assign locations to each value returned by this call.
   SmallVector<CCValAssign, 16> RVLocs;
   bool Is64Bit = Subtarget.is64Bit();
   CCState CCInfo(CallConv, isVarArg, DAG.getMachineFunction(), RVLocs,
                  *DAG.getContext());
   CCInfo.AnalyzeCallResult(Ins, RetCC_X86);
 
   // Copy all of the result registers out of their specified physreg.
-  for (unsigned i = 0, e = RVLocs.size(); i != e; ++i) {
+  for (unsigned i = 0, insIndex = 0, e = RVLocs.size(); i != e;
+       ++i, ++insIndex) {
     CCValAssign &VA = RVLocs[i];
     EVT CopyVT = VA.getLocVT();
 
     // If this is x86-64, and we disabled SSE, we can't return FP values
     if ((CopyVT == MVT::f32 || CopyVT == MVT::f64 || CopyVT == MVT::f128) &&
-        ((Is64Bit || Ins[i].Flags.isInReg()) && !Subtarget.hasSSE1())) {
+        ((Is64Bit || Ins[insIndex].Flags.isInReg()) && !Subtarget.hasSSE1())) {
       report_fatal_error("SSE register return with SSE disabled");
     }
 
     // If we prefer to use the value in xmm registers, copy it out as f80 and
     // use a truncate to move it from fp stack reg to xmm reg.
     bool RoundAfterCopy = false;
     if ((VA.getLocReg() == X86::FP0 || VA.getLocReg() == X86::FP1) &&
         isScalarFPTypeInSSEReg(VA.getValVT())) {
       if (!Subtarget.hasX87())
         report_fatal_error("X87 register return with X87 disabled");
       CopyVT = MVT::f80;
       RoundAfterCopy = (CopyVT != VA.getLocVT());
     }
 
-    Chain = DAG.getCopyFromReg(Chain, dl, VA.getLocReg(),
-                               CopyVT, InFlag).getValue(1);
-    SDValue Val = Chain.getValue(0);
+    SDValue Val;
+    if (VA.needsCustom()) {
+      assert(VA.getValVT() == MVT::v64i1 &&
+             "Currently the only custom case is when we split v64i1 to 2 regs");
+      Val =
+          Getv64i1Argument(VA, RVLocs[++i], Chain, DAG, dl, Subtarget, &InFlag);
+    } else {
+      Chain = DAG.getCopyFromReg(Chain, dl, VA.getLocReg(), CopyVT, InFlag)
+                  .getValue(1);
+      Val = Chain.getValue(0);
+      InFlag = Chain.getValue(2);
+    }
 
     if (RoundAfterCopy)
       Val = DAG.getNode(ISD::FP_ROUND, dl, VA.getValVT(), Val,
                         // This truncation won't change the value.
                         DAG.getIntPtrConstant(1, dl));
 
-    if (VA.isExtInLoc() && VA.getValVT().getScalarType() == MVT::i1)
+    if (VA.isExtInLoc()) {
+      if (VA.getValVT().isVector() &&
+          VA.getValVT().getScalarType() == MVT::i1 &&
+          (VA.getLocVT() == MVT::i32 || VA.getLocVT() == MVT::i64)) {
+        // promoting a mask type (v*i1) into a register of type i64/i32
+        LowerRegToMasks(Val, VA.getValVT(), VA.getLocVT(), CallConv, dl, DAG);
+      } else
         Val = DAG.getNode(ISD::TRUNCATE, dl, VA.getValVT(), Val);
+    }
 
-    InFlag = Chain.getValue(2);
     InVals.push_back(Val);
   }
 
   return Chain;
 }
 
 //===----------------------------------------------------------------------===//
 //                C & StdCall & Fast Calling Convention implementation
return DAG.getMemcpy(Chain, dl, Dst, Src, SizeNode, Flags.getByValAlign(),
                        /*isVolatile*/false, /*AlwaysInline=*/true,
                        /*isTailCall*/false,
                        MachinePointerInfo(), MachinePointerInfo());
 }
 
 /// Return true if the calling convention is one that we can guarantee TCO for.
 static bool canGuaranteeTCO(CallingConv::ID CC) {
   return (CC == CallingConv::Fast || CC == CallingConv::GHC ||
-          CC == CallingConv::HiPE || CC == CallingConv::HHVM);
+          CC == CallingConv::X86_RegCall || CC == CallingConv::HiPE ||
+          CC == CallingConv::HHVM);
 }
 
 /// Return true if we might ever do TCO for calls with this calling convention.
 static bool mayTailCallThisCC(CallingConv::ID CC) {
   switch (CC) {
   // C calling conventions:
   case CallingConv::C:
   case CallingConv::X86_64_Win64:
X86TargetLowering::LowerMemArgument(SDValue Chain, CallingConv::ID CallConv,
   ISD::ArgFlagsTy Flags = Ins[i].Flags;
   bool AlwaysUseMutable = shouldGuaranteeTCO(
       CallConv, DAG.getTarget().Options.GuaranteedTailCallOpt);
   bool isImmutable = !AlwaysUseMutable && !Flags.isByVal();
   EVT ValVT;
 
   // If value is passed by pointer we have address passed instead of the value
   // itself.
-  bool ExtendedInMem = VA.isExtInLoc() &&
-    VA.getValVT().getScalarType() == MVT::i1;
+  bool ExtendedInMem =
+      VA.isExtInLoc() && VA.getValVT().getScalarType() == MVT::i1 &&
+      VA.getValVT().getSizeInBits() != VA.getLocVT().getSizeInBits();
 
   if (VA.getLocInfo() == CCValAssign::Indirect || ExtendedInMem)
     ValVT = VA.getLocVT();
   else
     ValVT = VA.getValVT();
 
   // Calculate SP offset of interrupt parameter, re-arrange the slot normally
   // taken by a return address.
if (Fn->hasExternalLinkage() &&
       Subtarget.isTargetCygMing() &&
       Fn->getName() == "main")
     FuncInfo->setForceFramePointer(true);
 
   MachineFrameInfo &MFI = MF.getFrameInfo();
   bool Is64Bit = Subtarget.is64Bit();
   bool IsWin64 = Subtarget.isCallingConvWin64(CallConv);
 
-  assert(!(isVarArg && canGuaranteeTCO(CallConv)) &&
-         "Var args not supported with calling convention fastcc, ghc or hipe");
+  assert(
+      !(isVarArg && canGuaranteeTCO(CallConv)) &&
+      "Var args not supported with calling conv' regcall, fastcc, ghc or hipe");
 
   if (CallConv == CallingConv::X86_INTR) {
     bool isLegal = Ins.size() == 1 ||
                    (Ins.size() == 2 && ((Is64Bit && Ins[1].VT == MVT::i64) ||
                                         (!Is64Bit && Ins[1].VT == MVT::i32)));
     if (!isLegal)
       report_fatal_error("X86 interrupts may take one or two arguments");
   }
 
   // Assign locations to all of the incoming arguments.
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(CallConv, isVarArg, MF, ArgLocs, *DAG.getContext());
 
   // Allocate shadow area for Win64
   if (IsWin64)
     CCInfo.AllocateStack(32, 8);
 
   CCInfo.AnalyzeFormalArguments(Ins, CC_X86);
 
-  unsigned LastVal = ~0U;
   SDValue ArgValue;
-  for (unsigned i = 0, e = ArgLocs.size(); i != e; ++i) {
+  for (unsigned i = 0, insIndex = 0, e = ArgLocs.size(); i != e;
+       ++i, ++insIndex) {
+    assert(insIndex < Ins.size() && "Invalid Ins index");
     CCValAssign &VA = ArgLocs[i];
-    // TODO: If an arg is passed in two places (e.g. reg and stack), skip later
-    // places.
-    assert(VA.getValNo() != LastVal &&
-           "Don't support value assigned to multiple locs yet");
-    (void)LastVal;
-    LastVal = VA.getValNo();
 
     if (VA.isRegLoc()) {
       EVT RegVT = VA.getLocVT();
+      if (VA.needsCustom()) {
+        assert(
+            VA.getValVT() == MVT::v64i1 &&
+            "Currently the only custom case is when we split v64i1 to 2 regs");
+
+        // v64i1 values, in regcall calling convention, that are
+        // compiled to 32 bit arch, are splited up into two registers.
+        ArgValue =
+            Getv64i1Argument(VA, ArgLocs[++i], Chain, DAG, dl, Subtarget);
+      } else {
         const TargetRegisterClass *RC;
         if (RegVT == MVT::i32)
           RC = &X86::GR32RegClass;
         else if (Is64Bit && RegVT == MVT::i64)
           RC = &X86::GR64RegClass;
         else if (RegVT == MVT::f32)
           RC = &X86::FR32RegClass;
         else if (RegVT == MVT::f64)
           RC = &X86::FR64RegClass;
         else if (RegVT == MVT::f128)
           RC = &X86::FR128RegClass;
         else if (RegVT.is512BitVector())
           RC = &X86::VR512RegClass;
         else if (RegVT.is256BitVector())
           RC = &X86::VR256RegClass;
         else if (RegVT.is128BitVector())
           RC = &X86::VR128RegClass;
         else if (RegVT == MVT::x86mmx)
           RC = &X86::VR64RegClass;
         else if (RegVT == MVT::i1)
           RC = &X86::VK1RegClass;
         else if (RegVT == MVT::v8i1)
           RC = &X86::VK8RegClass;
         else if (RegVT == MVT::v16i1)
           RC = &X86::VK16RegClass;
         else if (RegVT == MVT::v32i1)
           RC = &X86::VK32RegClass;
         else if (RegVT == MVT::v64i1)
           RC = &X86::VK64RegClass;
         else
           llvm_unreachable("Unknown argument type!");
 
         unsigned Reg = MF.addLiveIn(VA.getLocReg(), RC);
         ArgValue = DAG.getCopyFromReg(Chain, dl, Reg, RegVT);
+      }
 
       // If this is an 8 or 16-bit value, it is really passed promoted to 32
       // bits.  Insert an assert[sz]ext to capture this, then truncate to the
       // right size.
       if (VA.getLocInfo() == CCValAssign::SExt)
         ArgValue = DAG.getNode(ISD::AssertSext, dl, RegVT, ArgValue,
                                DAG.getValueType(VA.getValVT()));
       else if (VA.getLocInfo() == CCValAssign::ZExt)
         ArgValue = DAG.getNode(ISD::AssertZext, dl, RegVT, ArgValue,
                                DAG.getValueType(VA.getValVT()));
       else if (VA.getLocInfo() == CCValAssign::BCvt)
         ArgValue = DAG.getBitcast(VA.getValVT(), ArgValue);
 
       if (VA.isExtInLoc()) {
         // Handle MMX values passed in XMM regs.
         if (RegVT.isVector() && VA.getValVT().getScalarType() != MVT::i1)
           ArgValue = DAG.getNode(X86ISD::MOVDQ2Q, dl, VA.getValVT(), ArgValue);
-        else
+        else if (VA.getValVT().isVector() &&
+                 VA.getValVT().getScalarType() == MVT::i1 &&
+                 ((RegVT == MVT::i32) || (RegVT == MVT::i64))) {
+          // Promoting a mask type (v*i1) into a register of type i64/i32
+          LowerRegToMasks(ArgValue, VA.getValVT(), RegVT, CallConv, dl, DAG);
+        } else
           ArgValue = DAG.getNode(ISD::TRUNCATE, dl, VA.getValVT(), ArgValue);
       }
     } else {
       assert(VA.isMemLoc());
-      ArgValue = LowerMemArgument(Chain, CallConv, Ins, dl, DAG, VA, MFI, i);
+      ArgValue =
+          LowerMemArgument(Chain, CallConv, Ins, dl, DAG, VA, MFI, insIndex);
     }
 
     // If value is passed via pointer - do a load.
     if (VA.getLocInfo() == CCValAssign::Indirect)
       ArgValue =
           DAG.getLoad(VA.getValVT(), dl, Chain, ArgValue, MachinePointerInfo());
 
     InVals.push_back(ArgValue);
   }
 
-  for (unsigned i = 0, e = ArgLocs.size(); i != e; ++i) {
+  for (unsigned i = 0, e = Ins.size(); i != e; ++i) {
     // Swift calling convention does not require we copy the sret argument
     // into %rax/%eax for the return. We don't set SRetReturnReg for Swift.
     if (CallConv == CallingConv::Swift)
       continue;
 
     // All x86 ABIs require that for returning structs by value we copy the
     // sret argument into %rax/%eax (depending on ABI) for the return. Save
     // the argument into a virtual register so that we can access it from the
X86TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
 
   SmallVector<std::pair<unsigned, SDValue>, 8> RegsToPass;
   SmallVector<SDValue, 8> MemOpChains;
   SDValue StackPtr;
 
   // Walk the register/memloc assignments, inserting copies/loads.  In the case
   // of tail call optimization arguments are handle later.
   const X86RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
-  for (unsigned i = 0, e = ArgLocs.size(); i != e; ++i) {
+  for (unsigned i = 0, outIndex = 0, e = ArgLocs.size(); i != e;
+       ++i, ++outIndex) {
+    assert(outIndex < Outs.size() && "Invalid Out index");
     // Skip inalloca arguments, they have already been written.
-    ISD::ArgFlagsTy Flags = Outs[i].Flags;
+    ISD::ArgFlagsTy Flags = Outs[outIndex].Flags;
     if (Flags.isInAlloca())
       continue;
 
     CCValAssign &VA = ArgLocs[i];
     EVT RegVT = VA.getLocVT();
-    SDValue Arg = OutVals[i];
+    SDValue Arg = OutVals[outIndex];
     bool isByVal = Flags.isByVal();
 
     // Promote the value if needed.
     switch (VA.getLocInfo()) {
     default: llvm_unreachable("Unknown loc info!");
     case CCValAssign::Full: break;
     case CCValAssign::SExt:
       Arg = DAG.getNode(ISD::SIGN_EXTEND, dl, RegVT, Arg);
       break;
     case CCValAssign::ZExt:
       Arg = DAG.getNode(ISD::ZERO_EXTEND, dl, RegVT, Arg);
       break;
     case CCValAssign::AExt:
       if (Arg.getValueType().isVector() &&
           Arg.getValueType().getVectorElementType() == MVT::i1)
-        Arg = DAG.getNode(ISD::SIGN_EXTEND, dl, RegVT, Arg);
+        LowerMasksToReg(Arg, RegVT, CallConv, dl, DAG);
       else if (RegVT.is128BitVector()) {
         // Special case: passing MMX values in XMM registers.
         Arg = DAG.getBitcast(MVT::i64, Arg);
         Arg = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, MVT::v2i64, Arg);
         Arg = getMOVL(DAG, dl, MVT::v2i64, DAG.getUNDEF(MVT::v2i64), Arg);
       } else
         Arg = DAG.getNode(ISD::ANY_EXTEND, dl, RegVT, Arg);
       break;
     case CCValAssign::BCvt:
       Arg = DAG.getBitcast(RegVT, Arg);
       break;
     case CCValAssign::Indirect: {
       // Store the argument.
       SDValue SpillSlot = DAG.CreateStackTemporary(VA.getValVT());
       int FI = cast<FrameIndexSDNode>(SpillSlot)->getIndex();
       Chain = DAG.getStore(
           Chain, dl, Arg, SpillSlot,
           MachinePointerInfo::getFixedStack(DAG.getMachineFunction(), FI));
       Arg = SpillSlot;
       break;
     }
     }
 
-    if (VA.isRegLoc()) {
+    if (VA.needsCustom()) {
+      assert(VA.getValVT() == MVT::v64i1 &&
+             "Currently the only custom case is when we split v64i1 to 2 regs");
+      // Split v64i1 value into two registers
+      Passv64i1ArgInRegs(dl, DAG, Chain, Arg, RegsToPass, VA, ArgLocs[++i],
+                         Subtarget);
+    } else if (VA.isRegLoc()) {
       RegsToPass.push_back(std::make_pair(VA.getLocReg(), Arg));
       if (isVarArg && IsWin64) {
         // Win64 ABI requires argument XMM reg to be copied to the corresponding
         // shadow reg if callee is a varargs function.
         unsigned ShadowReg = 0;
         switch (VA.getLocReg()) {
         case X86::XMM0: ShadowReg = X86::RCX; break;
         case X86::XMM1: ShadowReg = X86::RDX; break;
if (!IsSibcall && isTailCall) {
     // the alias isn't otherwise explicit. This is slightly more conservative
     // than necessary, because it means that each store effectively depends
     // on every argument instead of just those arguments it would clobber.
     SDValue ArgChain = DAG.getStackArgumentTokenFactor(Chain);
 
     SmallVector<SDValue, 8> MemOpChains2;
     SDValue FIN;
     int FI = 0;
-    for (unsigned i = 0, e = ArgLocs.size(); i != e; ++i) {
+    for (unsigned i = 0, outsIndex = 0, e = ArgLocs.size(); i != e;
+         ++i, ++outsIndex) {
       CCValAssign &VA = ArgLocs[i];
-      if (VA.isRegLoc())
+
+      if (VA.isRegLoc()) {
+        if (VA.needsCustom()) {
+          assert((CallConv == CallingConv::X86_RegCall) &&
+                 "Expecting custome case only in regcall calling convention");
+          // This means that we are in special case where one argument was
+          // passed through two register locations - Skip the next location
+          ++i;
+        }
+
         continue;
+      }
+
       assert(VA.isMemLoc());
-      SDValue Arg = OutVals[i];
-      ISD::ArgFlagsTy Flags = Outs[i].Flags;
+      SDValue Arg = OutVals[outsIndex];
+      ISD::ArgFlagsTy Flags = Outs[outsIndex].Flags;
       // Skip inalloca arguments.  They don't require any work.
       if (Flags.isInAlloca())
         continue;
       // Create frame index.
       int32_t Offset = VA.getLocMemOffset()+FPDiff;
       uint32_t OpSize = (VA.getLocVT().getSizeInBits()+7)/8;
       FI = MF.getFrameInfo().CreateFixedObject(OpSize, Offset, true);
       FIN = DAG.getFrameIndex(FI, getPointerTy(DAG.getDataLayout()));

test/CodeGen/X86/avx512-regcall-Mask.ll

+; RUN: llc < %s -mtriple=i386-pc-win32       -mattr=+avx512bw  | FileCheck --check-prefix=X32 %s
+; RUN: llc < %s -mtriple=x86_64-win32        -mattr=+avx512bw  | FileCheck --check-prefix=WIN64 %s
+; RUN: llc < %s -mtriple=x86_64-linux-gnu    -mattr=+avx512bw  | FileCheck --check-prefix=LINUXOSX64 %s
+
+; X32-LABEL:  test_argv64i1:
+; X32:        kmovd   %edx, %k0
+; X32:        kmovd   %edi, %k1
+; X32:        kmovd   %eax, %k1
+; X32:        kmovd   %ecx, %k2
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        ad{{d|c}}l  {{([0-9])*}}(%ebp), %e{{a|c}}x
+; X32:        retl
+
+; WIN64-LABEL: test_argv64i1:
+; WIN64:       addq    %rcx, %rax
+; WIN64:       addq    %rdx, %rax
+; WIN64:       addq    %rdi, %rax
+; WIN64:       addq    %rsi, %rax
+; WIN64:       addq    %r8, %rax
+; WIN64:       addq    %r9, %rax
+; WIN64:       addq    %r10, %rax
+; WIN64:       addq    %r11, %rax
+; WIN64:       addq    %r12, %rax
+; WIN64:       addq    %r14, %rax
+; WIN64:       addq    %r15, %rax
+; WIN64:       addq  {{([0-9])*}}(%rsp), %rax
+; WIN64:       retq
+
+; LINUXOSX64-LABEL: test_argv64i1:
+; LINUXOSX64:       addq    %rcx, %rax
+; LINUXOSX64:       addq    %rdx, %rax
+; LINUXOSX64:       addq    %rdi, %rax
+; LINUXOSX64:       addq    %rsi, %rax
+; LINUXOSX64:       addq    %r8, %rax
+; LINUXOSX64:       addq    %r9, %rax
+; LINUXOSX64:       addq    %r12, %rax
+; LINUXOSX64:       addq    %r13, %rax
+; LINUXOSX64:       addq    %r14, %rax
+; LINUXOSX64:       addq    %r15, %rax
+; LINUXOSX64:       addq    {{([0-9])*}}(%rsp), %rax
+; LINUXOSX64:       addq    {{([0-9])*}}(%rsp), %rax
+; LINUXOSX64:       retq
+
+; Test regcall when receiving arguments of v64i1 type
+define x86_regcallcc i64 @test_argv64i1(<64 x i1> %x0, <64 x i1> %x1, <64 x i1> %x2,
+                                        <64 x i1> %x3, <64 x i1> %x4, <64 x i1> %x5,
+                                        <64 x i1> %x6, <64 x i1> %x7, <64 x i1> %x8,
+                                        <64 x i1> %x9, <64 x i1> %x10, <64 x i1> %x11,
+                                        <64 x i1> %x12)  {
+  %y0 = bitcast <64 x i1> %x0 to i64
+  %y1 = bitcast <64 x i1> %x1 to i64
+  %y2 = bitcast <64 x i1> %x2 to i64
+  %y3 = bitcast <64 x i1> %x3 to i64
+  %y4 = bitcast <64 x i1> %x4 to i64
+  %y5 = bitcast <64 x i1> %x5 to i64
+  %y6 = bitcast <64 x i1> %x6 to i64
+  %y7 = bitcast <64 x i1> %x7 to i64
+  %y8 = bitcast <64 x i1> %x8 to i64
+  %y9 = bitcast <64 x i1> %x9 to i64
+  %y10 = bitcast <64 x i1> %x10 to i64
+  %y11 = bitcast <64 x i1> %x11 to i64
+  %y12 = bitcast <64 x i1> %x12 to i64
+  %add1 = add i64 %y0, %y1
+  %add2 = add i64 %add1, %y2
+  %add3 = add i64 %add2, %y3
+  %add4 = add i64 %add3, %y4
+  %add5 = add i64 %add4, %y5
+  %add6 = add i64 %add5, %y6
+  %add7 = add i64 %add6, %y7
+  %add8 = add i64 %add7, %y8
+  %add9 = add i64 %add8, %y9
+  %add10 = add i64 %add9, %y10
+  %add11 = add i64 %add10, %y11
+  %add12 = add i64 %add11, %y12
+  ret i64 %add12
+}
+
+; X32-LABEL:  caller_argv64i1:
+; X32:        movl    $2, %eax
+; X32:        movl    $1, %ecx
+; X32:        movl    $2, %edx
+; X32:        movl    $1, %edi
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        pushl    ${{1|2}}
+; X32:        call{{.*}}   _test_argv64i1
+        
+; WIN64-LABEL: caller_argv64i1:
+; WIN64:       movabsq    $4294967298, %rax
+; WIN64:       movq   %rax, (%rsp)
+; WIN64:       movq   %rax, %rcx
+; WIN64:       movq   %rax, %rdx
+; WIN64:       movq   %rax, %rdi
+; WIN64:       movq   %rax, %rsi
+; WIN64:       movq   %rax, %r8
+; WIN64:       movq   %rax, %r9
+; WIN64:       movq   %rax, %r10
+; WIN64:       movq   %rax, %r11
+; WIN64:       movq   %rax, %r12
+; WIN64:       movq   %rax, %r14
+; WIN64:       movq   %rax, %r15
+; WIN64:       callq   test_argv64i1
+
+; LINUXOSX64-LABEL: caller_argv64i1:
+; LINUXOSX64:       movabsq    $4294967298, %rax
+; LINUXOSX64:       movq   %rax, %rcx
+; LINUXOSX64:       movq   %rax, %rdx
+; LINUXOSX64:       movq   %rax, %rdi
+; LINUXOSX64:       movq   %rax, %rsi
+; LINUXOSX64:       movq   %rax, %r8
+; LINUXOSX64:       movq   %rax, %r9
+; LINUXOSX64:       movq   %rax, %r12
+; LINUXOSX64:       movq   %rax, %r13
+; LINUXOSX64:       movq   %rax, %r14
+; LINUXOSX64:       movq   %rax, %r15
+; LINUXOSX64:       call{{.*}}   test_argv64i1
+
+; Test regcall when passing arguments of v64i1 type
+define x86_regcallcc i64 @caller_argv64i1() #0 {
+entry:
+  %v0 = bitcast i64 4294967298 to <64 x i1>
+  %call = call x86_regcallcc i64 @test_argv64i1(<64 x i1> %v0, <64 x i1> %v0, <64 x i1> %v0,
+                                                <64 x i1> %v0, <64 x i1> %v0, <64 x i1> %v0,
+                                                <64 x i1> %v0, <64 x i1> %v0, <64 x i1> %v0,
+                                                <64 x i1> %v0, <64 x i1> %v0, <64 x i1> %v0,
+                                                <64 x i1> %v0)
+  ret i64 %call
+}
+
+; X32-LABEL: test_retv64i1:
+; X32:       mov{{.*}}    $2, %eax
+; X32:       mov{{.*}}    $1, %ecx
+; X32:       ret{{.*}}
+
+; WIN64-LABEL: test_retv64i1:
+; WIN64:       mov{{.*}} $4294967298, %rax
+; WIN64:       ret{{.*}}
+
+; Test regcall when returning v64i1 type
+define x86_regcallcc <64 x i1> @test_retv64i1()  {
+  %a = bitcast i64 4294967298 to <64 x i1>
+ ret <64 x i1> %a
+}
+
+; X32-LABEL: caller_retv64i1:
+; X32:       call{{.*}}   _test_retv64i1
+; X32:       kmov{{.*}}   %eax, %k0
+; X32:       kmov{{.*}}   %ecx, %k1
+; X32:       kunpckdq     %k0, %k1, %k0
+
+; Test regcall when processing result of v64i1 type
+define x86_regcallcc <64 x i1> @caller_retv64i1() #0 {
+entry:
+  %call = call x86_regcallcc <64 x i1> @test_retv64i1()
+  ret <64 x i1> %call
+}