CGCall: Factor out the logic mapping call arguments to LLVM IR arguments.

[samsonov]

Summary

This refactoring introduces ClangToLLVMArgMapping class, which
encapsulates the information about the order in which function arguments listed
in CGFunctionInfo should be passed to actual LLVM IR function, such as:

1. positions of sret, if there is any
2. position of inalloca argument, if there is any
3. position of helper padding argument for each call argument
4. positions of regular argument (there can be many if it's expanded).

Simplify several related methods (ConstructAttributeList, EmitFunctionProlog
and EmitCall): now they don't have to maintain iterators over the list
of LLVM IR function arguments, dealing with all the sret/inalloca/this complexities,
and just use expected positions of LLVM IR arguments stored in ClangToLLVMArgMapping.

This may increase the running time of EmitFunctionProlog, as we have to traverse
expandable arguments twice, but in further refactoring we will be able
to speed up EmitCall by passing already calculated CallArgsToIRArgsMapping to
ConstructAttributeList, thus avoiding traversing expandable argument there.

No functionality change.

[rnk]

Nice! I've contemplated doing a cleanup like this, but it looks like you have succeeded where I have failed.

lib/CodeGen/CGCall.cpp

1044

I think we can sink all of this into CGFunctionInfo once we remove the AAPCS issue. I pinged James Molloy about this.

1136

Maybe we can do this once after computing CGFunctionInfo and then store it in CGFunctionInfo?

[samsonov]

lib/CodeGen/CGCall.cpp

1044

I thought that CGFunctionInfo is designed to be as small as possible (all of them are memoized in CodeGenTypes, for instance), and Args->IRArgs mapping in fact describes the "algorithm", not a function definition and would hardly be useful outside if CGCall routines.

1136

See comment above. We might, though, have a different cache that would tell the number of IR arguments each QualType will extend to.

[rnk]

+David Majnemer, since he had opinions about doing sret/this the Right Way.

lib/CodeGen/CGCall.cpp

1044

Well, previously Mark Lacey was trying to teach LLDB about these kinds of things, so I thought it might be useful to hoist out. However, it looks like that effort was abandoned, so let's keep this here until we hear otherwise.

Can we name this better? Maybe ClangToLLVMArgMapping or ASTToIRArgMapping? Or just IRArgMapping?

1050

99% of the time the inner SmallVector will have size 1. This basically what TinyPtrVector is for, but I guess we can't use that to store unsigned ints. Hm.

1088

'construct' is pretty big, I'd pull it out of line to reduce indentation.

1144–1146

Looks like all argument types consume either zero, one, or sequential IR arguments. This only needs to store one number per AST-level argument. If there are zero IR args, use InvalidIndex. If there is one IR arg, store the IR arg number. If there are more than one, store the first IR arg number. The consumers can increment that during argument expansion.

Does that sound reasonable?

[samsonov]

lib/CodeGen/CGCall.cpp

1144–1146

Yes, I think the assumption that all IR arguments are sequential is reasonable. Currently, we would still need to do argument expansion both here, and in EmitPrologue/EmitCall methods. I think it makes to store the number of expanded arguments to assert that the numbers match (at least until we refactor terribly similar GetExpandedTypes / ExpandTypeFromArgs / ExpandTypeToArgs into a single routine and/or introduce a map "QualType -> number of arguments it expands to").

What we can do is replace PaddingIRArgIndex and IRArgs vectors with a single vector of triples <PaddingIRArgIndex, FirstIRArgIndex, NumberOfIRArgs>. WDYT?

[rnk]

lib/CodeGen/CGCall.cpp

1144–1146

Yeah, sounds good.

[samsonov]

lib/CodeGen/CGCall.cpp

1044

Done

1050

Got rid of this in favor of vector of triples.

1088

Done

1144–1146

Done

[samsonov]

Address reviewer's comments.

[rnk]

lgtm

[samsonov]

Rebase.

[samsonov]

Thanks!

lib/CodeGen/CGCall.cpp

void CodeGenTypes::GetExpandedTypes(QualType type,
   } else if (const ComplexType *CT = type->getAs<ComplexType>()) {
     llvm::Type *EltTy = ConvertType(CT->getElementType());
     expandedTypes.push_back(EltTy);
     expandedTypes.push_back(EltTy);
   } else
     expandedTypes.push_back(ConvertType(type));
 }
 
-llvm::Function::arg_iterator
-CodeGenFunction::ExpandTypeFromArgs(QualType Ty, LValue LV,
-                                    llvm::Function::arg_iterator AI) {
+void CodeGenFunction::ExpandTypeFromArgs(
+    QualType Ty, LValue LV, SmallVectorImpl<llvm::Argument *>::iterator &AI) {
   assert(LV.isSimple() &&
          "Unexpected non-simple lvalue during struct expansion.");
 
   if (const ConstantArrayType *AT = getContext().getAsConstantArrayType(Ty)) {
     unsigned NumElts = AT->getSize().getZExtValue();
     QualType EltTy = AT->getElementType();
     for (unsigned Elt = 0; Elt < NumElts; ++Elt) {
       llvm::Value *EltAddr = Builder.CreateConstGEP2_32(LV.getAddress(), 0, Elt);
       LValue LV = MakeAddrLValue(EltAddr, EltTy);
-      AI = ExpandTypeFromArgs(EltTy, LV, AI);
+      ExpandTypeFromArgs(EltTy, LV, AI);
     }
-  } else if (const RecordType *RT = Ty->getAs<RecordType>()) {
+    return;
+  }
+  if (const RecordType *RT = Ty->getAs<RecordType>()) {
     RecordDecl *RD = RT->getDecl();
     if (RD->isUnion()) {
       // Unions can be here only in degenerative cases - all the fields are same
       // after flattening. Thus we have to use the "largest" field.
       const FieldDecl *LargestFD = nullptr;
       CharUnits UnionSize = CharUnits::Zero();
 
       for (const auto *FD : RD->fields()) {
         assert(!FD->isBitField() &&
                "Cannot expand structure with bit-field members.");
         CharUnits FieldSize = getContext().getTypeSizeInChars(FD->getType());
         if (UnionSize < FieldSize) {
           UnionSize = FieldSize;
           LargestFD = FD;
         }
       }
       if (LargestFD) {
         // FIXME: What are the right qualifiers here?
         LValue SubLV = EmitLValueForField(LV, LargestFD);
-        AI = ExpandTypeFromArgs(LargestFD->getType(), SubLV, AI);
+        ExpandTypeFromArgs(LargestFD->getType(), SubLV, AI);
       }
     } else {
       for (const auto *FD : RD->fields()) {
         QualType FT = FD->getType();
-
         // FIXME: What are the right qualifiers here?
         LValue SubLV = EmitLValueForField(LV, FD);
-        AI = ExpandTypeFromArgs(FT, SubLV, AI);
-      }
-    }
-  } else if (const ComplexType *CT = Ty->getAs<ComplexType>()) {
+        ExpandTypeFromArgs(FT, SubLV, AI);
+      }
+    return;
+  if (const ComplexType *CT = Ty->getAs<ComplexType>()) {
     QualType EltTy = CT->getElementType();
     llvm::Value *RealAddr = Builder.CreateStructGEP(LV.getAddress(), 0, "real");
-    EmitStoreThroughLValue(RValue::get(AI++), MakeAddrLValue(RealAddr, EltTy));
+    EmitStoreThroughLValue(RValue::get(*AI++), MakeAddrLValue(RealAddr, EltTy));
     llvm::Value *ImagAddr = Builder.CreateStructGEP(LV.getAddress(), 1, "imag");
-    EmitStoreThroughLValue(RValue::get(AI++), MakeAddrLValue(ImagAddr, EltTy));
-  } else {
-    EmitStoreThroughLValue(RValue::get(AI), LV);
-    ++AI;
+    EmitStoreThroughLValue(RValue::get(*AI++), MakeAddrLValue(ImagAddr, EltTy));
+    return;
   }
-
-  return AI;
+  EmitStoreThroughLValue(RValue::get(*AI++), LV);
 }
 
 /// EnterStructPointerForCoercedAccess - Given a struct pointer that we are
 /// accessing some number of bytes out of it, try to gep into the struct to get
 /// at its inner goodness.  Dive as deep as possible without entering an element
 /// with an in-memory size smaller than DstSize.
 static llvm::Value *
 EnterStructPointerForCoercedAccess(llvm::Value *SrcPtr,
llvm::Type *CodeGenTypes::GetFunctionTypeForVTable(GlobalDecl GD) {
   const CGFunctionInfo *Info;
   if (isa<CXXDestructorDecl>(MD))
     Info = &arrangeCXXDestructor(cast<CXXDestructorDecl>(MD), GD.getDtorType());
   else
     Info = &arrangeCXXMethodDeclaration(MD);
   return GetFunctionType(*Info);
 }
 
+namespace {
+
+/// Encapsulates information about the way function arguments from
+/// CGFunctionInfo should be passed to actual LLVM IR function.
+class ClangToLLVMArgMapping {
+  static const unsigned InvalidIndex = ~0U;
+  unsigned InallocaArgNo;
+  unsigned SRetArgNo;
+  unsigned TotalIRArgs;
+
+  /// Arguments of LLVM IR function corresponding to single Clang argument.
+  struct IRArgs {
+    unsigned PaddingArgIndex;
+    // Argument is expanded to IR arguments at positions
+    // [FirstArgIndex, FirstArgIndex + NumberOfArgs).
+    unsigned FirstArgIndex;
+    unsigned NumberOfArgs;
+
+    IRArgs()
+        : PaddingArgIndex(InvalidIndex), FirstArgIndex(InvalidIndex),
+          NumberOfArgs(0) {}
+  };
+
+  SmallVector<IRArgs, 8> ArgInfo;
+
+public:
+  ClangToLLVMArgMapping(CodeGenModule &CGM, const CGFunctionInfo &FI)
+      : InallocaArgNo(InvalidIndex), SRetArgNo(InvalidIndex), TotalIRArgs(0),
+        ArgInfo(FI.arg_size()) {
+    construct(CGM, FI);
+  }
+
+  bool hasInallocaArg() const { return InallocaArgNo != InvalidIndex; }
+  unsigned getInallocaArgNo() const {
+    assert(hasInallocaArg());
+    return InallocaArgNo;
+  }
+
+  bool hasSRetArg() const { return SRetArgNo != InvalidIndex; }
+  unsigned getSRetArgNo() const {
+    assert(hasSRetArg());
+    return SRetArgNo;
+  }
+
+  unsigned totalIRArgs() const { return TotalIRArgs; }
+
+  bool hasPaddingArg(unsigned ArgNo) const {
+    assert(ArgNo < ArgInfo.size());
+    return ArgInfo[ArgNo].PaddingArgIndex != InvalidIndex;
+  }
+  unsigned getPaddingArgNo(unsigned ArgNo) const {
+    assert(hasPaddingArg(ArgNo));
+    return ArgInfo[ArgNo].PaddingArgIndex;
+  }
+
+  /// Returns index of first IR argument corresponding to ArgNo, and their
+  /// quantity.
+  std::pair<unsigned, unsigned> getIRArgs(unsigned ArgNo) const {
+    assert(ArgNo < ArgInfo.size());
+    return std::make_pair(ArgInfo[ArgNo].FirstArgIndex,
+                          ArgInfo[ArgNo].NumberOfArgs);
+  }
+
+private:
+  void construct(CodeGenModule &CGM, const CGFunctionInfo &FI);
+};
+
+void ClangToLLVMArgMapping::construct(CodeGenModule &CGM,
+                                      const CGFunctionInfo &FI) {
+  unsigned IRArgNo = 0;
+  bool SwapThisWithSRet = false;
+  const ABIArgInfo &RetAI = FI.getReturnInfo();
+
+  if (RetAI.getKind() == ABIArgInfo::Indirect) {
+    SwapThisWithSRet = RetAI.isSRetAfterThis();
+    SRetArgNo = SwapThisWithSRet ? 1 : IRArgNo++;
+  }
+
+  unsigned ArgNo = 0;
+  for (CGFunctionInfo::const_arg_iterator I = FI.arg_begin(),
+                                          E = FI.arg_end();
+       I != E; ++I, ++ArgNo) {
+    QualType ArgType = I->type;
+    const ABIArgInfo &AI = I->info;
+    // Collect data about IR arguments corresponding to Clang argument ArgNo.
+    auto &IRArgs = ArgInfo[ArgNo];
+
+    if (AI.getPaddingType())
+      IRArgs.PaddingArgIndex = IRArgNo++;
+
+    switch (AI.getKind()) {
+    case ABIArgInfo::Extend:
+    case ABIArgInfo::Direct: {
+      // FIXME: handle sseregparm someday...
+      llvm::StructType *STy = dyn_cast<llvm::StructType>(AI.getCoerceToType());
+      if (!isAAPCSVFP(FI, CGM.getTarget()) && STy) {
+        IRArgs.NumberOfArgs = STy->getNumElements();
+      } else {
+        IRArgs.NumberOfArgs = 1;
+      }
+      break;
+    }
+    case ABIArgInfo::Indirect:
+      IRArgs.NumberOfArgs = 1;
+      break;
+    case ABIArgInfo::Ignore:
+    case ABIArgInfo::InAlloca:
+      // ignore and inalloca doesn't have matching LLVM parameters.
+      IRArgs.NumberOfArgs = 0;
+      break;
+    case ABIArgInfo::Expand: {
+      SmallVector<llvm::Type*, 8> Types;
+      // FIXME: This is rather inefficient. Do we ever actually need to do
+      // anything here? The result should be just reconstructed on the other
+      // side, so extension should be a non-issue.
+      CGM.getTypes().GetExpandedTypes(ArgType, Types);
+      IRArgs.NumberOfArgs = Types.size();
+      break;
+    }
+    }
+
+    if (IRArgs.NumberOfArgs > 0) {
+      IRArgs.FirstArgIndex = IRArgNo;
+      IRArgNo += IRArgs.NumberOfArgs;
+    }
+
+    // Skip over the sret parameter when it comes second.  We already handled it
+    // above.
+    if (IRArgNo == 1 && SwapThisWithSRet)
+      IRArgNo++;
+  }
+  assert(ArgNo == FI.arg_size());
+
+  if (FI.usesInAlloca())
+    InallocaArgNo = IRArgNo++;
+
+  TotalIRArgs = IRArgNo;
+}
+}  // namespace
+
 void CodeGenModule::ConstructAttributeList(const CGFunctionInfo &FI,
                                            const Decl *TargetDecl,
                                            AttributeListType &PAL,
                                            unsigned &CallingConv,
                                            bool AttrOnCallSite) {
   llvm::AttrBuilder FuncAttrs;
   llvm::AttrBuilder RetAttrs;
 
FuncAttrs.addAttribute("use-soft-float",
                            llvm::toStringRef(CodeGenOpts.SoftFloat));
     FuncAttrs.addAttribute("stack-protector-buffer-size",
                            llvm::utostr(CodeGenOpts.SSPBufferSize));
 
     if (!CodeGenOpts.StackRealignment)
       FuncAttrs.addAttribute("no-realign-stack");
   }
 
+  ClangToLLVMArgMapping IRFunctionArgs(*this, FI);
+
   QualType RetTy = FI.getReturnType();
-  unsigned Index = 1;
-  bool SwapThisWithSRet = false;
   const ABIArgInfo &RetAI = FI.getReturnInfo();
   switch (RetAI.getKind()) {
   case ABIArgInfo::Extend:
     if (RetTy->hasSignedIntegerRepresentation())
       RetAttrs.addAttribute(llvm::Attribute::SExt);
     else if (RetTy->hasUnsignedIntegerRepresentation())
       RetAttrs.addAttribute(llvm::Attribute::ZExt);
     // FALL THROUGH
   case ABIArgInfo::Direct:
     if (RetAI.getInReg())
       RetAttrs.addAttribute(llvm::Attribute::InReg);
     break;
   case ABIArgInfo::Ignore:
     break;
 
-  case ABIArgInfo::InAlloca: {
-    // inalloca disables readnone and readonly
-    FuncAttrs.removeAttribute(llvm::Attribute::ReadOnly)
-      .removeAttribute(llvm::Attribute::ReadNone);
-    break;
-  }
-
+  case ABIArgInfo::InAlloca:
   case ABIArgInfo::Indirect: {
-    llvm::AttrBuilder SRETAttrs;
-    SRETAttrs.addAttribute(llvm::Attribute::StructRet);
-    if (RetAI.getInReg())
-      SRETAttrs.addAttribute(llvm::Attribute::InReg);
-    SwapThisWithSRet = RetAI.isSRetAfterThis();
-    PAL.push_back(llvm::AttributeSet::get(
-        getLLVMContext(), SwapThisWithSRet ? 2 : Index, SRETAttrs));
-
-    if (!SwapThisWithSRet)
-      ++Index;
-    // sret disables readnone and readonly
+    // inalloca and sret disable readnone and readonly
     FuncAttrs.removeAttribute(llvm::Attribute::ReadOnly)
       .removeAttribute(llvm::Attribute::ReadNone);
     break;
   }
 
   case ABIArgInfo::Expand:
     llvm_unreachable("Invalid ABI kind for return argument");
   }
 
   if (const auto *RefTy = RetTy->getAs<ReferenceType>()) {
     QualType PTy = RefTy->getPointeeType();
     if (!PTy->isIncompleteType() && PTy->isConstantSizeType())
       RetAttrs.addDereferenceableAttr(getContext().getTypeSizeInChars(PTy)
                                         .getQuantity());
     else if (getContext().getTargetAddressSpace(PTy) == 0)
       RetAttrs.addAttribute(llvm::Attribute::NonNull);
   }
 
-  if (RetAttrs.hasAttributes())
-    PAL.push_back(llvm::
-                  AttributeSet::get(getLLVMContext(),
-                                    llvm::AttributeSet::ReturnIndex,
-                                    RetAttrs));
+  // Attach return attributes.
+  if (RetAttrs.hasAttributes()) {
+    PAL.push_back(llvm::AttributeSet::get(
+        getLLVMContext(), llvm::AttributeSet::ReturnIndex, RetAttrs));
+  }
 
-  for (const auto &I : FI.arguments()) {
-    QualType ParamType = I.type;
-    const ABIArgInfo &AI = I.info;
+  // Attach attributes to sret.
+  if (IRFunctionArgs.hasSRetArg()) {
+    llvm::AttrBuilder SRETAttrs;
+    SRETAttrs.addAttribute(llvm::Attribute::StructRet);
+    if (RetAI.getInReg())
+      SRETAttrs.addAttribute(llvm::Attribute::InReg);
+    PAL.push_back(llvm::AttributeSet::get(
+        getLLVMContext(), IRFunctionArgs.getSRetArgNo() + 1, SRETAttrs));
+  }
+
+  // Attach attributes to inalloca argument.
+  if (IRFunctionArgs.hasInallocaArg()) {
     llvm::AttrBuilder Attrs;
+    Attrs.addAttribute(llvm::Attribute::InAlloca);
+    PAL.push_back(llvm::AttributeSet::get(
+        getLLVMContext(), IRFunctionArgs.getInallocaArgNo() + 1, Attrs));
+  }
 
-    // Skip over the sret parameter when it comes second.  We already handled it
-    // above.
-    if (Index == 2 && SwapThisWithSRet)
-      ++Index;
 
-    if (AI.getPaddingType()) {
+  unsigned ArgNo = 0;
+  for (CGFunctionInfo::const_arg_iterator I = FI.arg_begin(),
+                                          E = FI.arg_end();
+       I != E; ++I, ++ArgNo) {
+    QualType ParamType = I->type;
+    const ABIArgInfo &AI = I->info;
+    llvm::AttrBuilder Attrs;
+
+    // Add attribute for padding argument, if necessary.
+    if (IRFunctionArgs.hasPaddingArg(ArgNo)) {
       if (AI.getPaddingInReg())
-        PAL.push_back(llvm::AttributeSet::get(getLLVMContext(), Index,
-                                              llvm::Attribute::InReg));
-      // Increment Index if there is padding.
-      ++Index;
+        PAL.push_back(llvm::AttributeSet::get(
+            getLLVMContext(), IRFunctionArgs.getPaddingArgNo(ArgNo) + 1,
     }
 
     // 'restrict' -> 'noalias' is done in EmitFunctionProlog when we
     // have the corresponding parameter variable.  It doesn't make
     // sense to do it here because parameters are so messed up.
     switch (AI.getKind()) {
     case ABIArgInfo::Extend:
       if (ParamType->isSignedIntegerOrEnumerationType())
         Attrs.addAttribute(llvm::Attribute::SExt);
       else if (ParamType->isUnsignedIntegerOrEnumerationType())
         Attrs.addAttribute(llvm::Attribute::ZExt);
       // FALL THROUGH
-    case ABIArgInfo::Direct: {
+    case ABIArgInfo::Direct:
       if (AI.getInReg())
         Attrs.addAttribute(llvm::Attribute::InReg);
-
-      // FIXME: handle sseregparm someday...
-
-      llvm::StructType *STy =
-          dyn_cast<llvm::StructType>(AI.getCoerceToType());
-      if (!isAAPCSVFP(FI, getTarget()) && STy) {
-        unsigned Extra = STy->getNumElements()-1;  // 1 will be added below.
-        if (Attrs.hasAttributes())
-          for (unsigned I = 0; I < Extra; ++I)
-            PAL.push_back(llvm::AttributeSet::get(getLLVMContext(), Index + I,
-                                                  Attrs));
-        Index += Extra;
-      }
       break;
-    }
+
     case ABIArgInfo::Indirect:
       if (AI.getInReg())
         Attrs.addAttribute(llvm::Attribute::InReg);
 
       if (AI.getIndirectByVal())
         Attrs.addAttribute(llvm::Attribute::ByVal);
 
       Attrs.addAlignmentAttr(AI.getIndirectAlign());
 
       // byval disables readnone and readonly.
       FuncAttrs.removeAttribute(llvm::Attribute::ReadOnly)
         .removeAttribute(llvm::Attribute::ReadNone);
       break;
 
     case ABIArgInfo::Ignore:
-      // Skip increment, no matching LLVM parameter.
+    case ABIArgInfo::Expand:
       continue;
 
     case ABIArgInfo::InAlloca:
       // inalloca disables readnone and readonly.
       FuncAttrs.removeAttribute(llvm::Attribute::ReadOnly)
           .removeAttribute(llvm::Attribute::ReadNone);
-      // Skip increment, no matching LLVM parameter.
-      continue;
-
-    case ABIArgInfo::Expand: {
-      SmallVector<llvm::Type*, 8> types;
-      // FIXME: This is rather inefficient. Do we ever actually need to do
-      // anything here? The result should be just reconstructed on the other
-      // side, so extension should be a non-issue.
-      getTypes().GetExpandedTypes(ParamType, types);
-      Index += types.size();
       continue;
     }
-    }
 
     if (const auto *RefTy = ParamType->getAs<ReferenceType>()) {
       QualType PTy = RefTy->getPointeeType();
       if (!PTy->isIncompleteType() && PTy->isConstantSizeType())
         Attrs.addDereferenceableAttr(getContext().getTypeSizeInChars(PTy)
                                        .getQuantity());
       else if (getContext().getTargetAddressSpace(PTy) == 0)
         Attrs.addAttribute(llvm::Attribute::NonNull);
     }
 
-    if (Attrs.hasAttributes())
-      PAL.push_back(llvm::AttributeSet::get(getLLVMContext(), Index, Attrs));
-    ++Index;
+    if (Attrs.hasAttributes()) {
+      unsigned FirstIRArg, NumIRArgs;
+      std::tie(FirstIRArg, NumIRArgs) = IRFunctionArgs.getIRArgs(ArgNo);
+      for (unsigned i = 0; i < NumIRArgs; i++)
+        PAL.push_back(llvm::AttributeSet::get(getLLVMContext(),
+                                              FirstIRArg + i + 1, Attrs));
     }
-
-  // Add the inalloca attribute to the trailing inalloca parameter if present.
-  if (FI.usesInAlloca()) {
-    llvm::AttrBuilder Attrs;
-    Attrs.addAttribute(llvm::Attribute::InAlloca);
-    PAL.push_back(llvm::AttributeSet::get(getLLVMContext(), Index, Attrs));
   }
+  assert(ArgNo == FI.arg_size());
 
   if (FuncAttrs.hasAttributes())
     PAL.push_back(llvm::
                   AttributeSet::get(getLLVMContext(),
                                     llvm::AttributeSet::FunctionIndex,
                                     FuncAttrs));
 }
 
if (FD->hasImplicitReturnZero()) {
       llvm::Constant* Zero = llvm::Constant::getNullValue(LLVMTy);
       Builder.CreateStore(Zero, ReturnValue);
     }
   }
 
   // FIXME: We no longer need the types from FunctionArgList; lift up and
   // simplify.
 
-  // Emit allocs for param decls.  Give the LLVM Argument nodes names.
-  llvm::Function::arg_iterator AI = Fn->arg_begin();
+  ClangToLLVMArgMapping IRFunctionArgs(CGM, FI);
+  // Flattened function arguments.
+  SmallVector<llvm::Argument *, 16> FnArgs;
+  FnArgs.reserve(IRFunctionArgs.totalIRArgs());
+  for (auto &Arg : Fn->args()) {
+    FnArgs.push_back(&Arg);
+  }
+  assert(FnArgs.size() == IRFunctionArgs.totalIRArgs());
 
   // If we're using inalloca, all the memory arguments are GEPs off of the last
   // parameter, which is a pointer to the complete memory area.
   llvm::Value *ArgStruct = nullptr;
-  if (FI.usesInAlloca()) {
-    llvm::Function::arg_iterator EI = Fn->arg_end();
-    --EI;
-    ArgStruct = EI;
+  if (IRFunctionArgs.hasInallocaArg()) {
+    ArgStruct = FnArgs[IRFunctionArgs.getInallocaArgNo()];
     assert(ArgStruct->getType() == FI.getArgStruct()->getPointerTo());
   }
 
-  // Name the struct return parameter, which can come first or second.
-  const ABIArgInfo &RetAI = FI.getReturnInfo();
-  bool SwapThisWithSRet = false;
-  if (RetAI.isIndirect()) {
-    SwapThisWithSRet = RetAI.isSRetAfterThis();
-    if (SwapThisWithSRet)
-      ++AI;
+  // Name the struct return parameter.
+  if (IRFunctionArgs.hasSRetArg()) {
+    auto AI = FnArgs[IRFunctionArgs.getSRetArgNo()];
     AI->setName("agg.result");
     AI->addAttr(llvm::AttributeSet::get(getLLVMContext(), AI->getArgNo() + 1,
                                         llvm::Attribute::NoAlias));
-    if (SwapThisWithSRet)
-      --AI;  // Go back to the beginning for 'this'.
-    else
-      ++AI;  // Skip the sret parameter.
   }
 
   // Get the function-level nonnull attribute if it exists.
   const NonNullAttr *NNAtt =
     CurCodeDecl ? CurCodeDecl->getAttr<NonNullAttr>() : nullptr;
 
   // Track if we received the parameter as a pointer (indirect, byval, or
   // inalloca).  If already have a pointer, EmitParmDecl doesn't need to copy it
   // into a local alloca for us.
   enum ValOrPointer { HaveValue = 0, HavePointer = 1 };
   typedef llvm::PointerIntPair<llvm::Value *, 1> ValueAndIsPtr;
   SmallVector<ValueAndIsPtr, 16> ArgVals;
   ArgVals.reserve(Args.size());
 
   // Create a pointer value for every parameter declaration.  This usually
   // entails copying one or more LLVM IR arguments into an alloca.  Don't push
   // any cleanups or do anything that might unwind.  We do that separately, so
   // we can push the cleanups in the correct order for the ABI.
   assert(FI.arg_size() == Args.size() &&
          "Mismatch between function signature & arguments.");
-  unsigned ArgNo = 1;
+  unsigned ArgNo = 0;
   CGFunctionInfo::const_arg_iterator info_it = FI.arg_begin();
-  for (FunctionArgList::const_iterator i = Args.begin(), e = Args.end(); 
+  for (FunctionArgList::const_iterator i = Args.begin(), e = Args.end();
        i != e; ++i, ++info_it, ++ArgNo) {
     const VarDecl *Arg = *i;
     QualType Ty = info_it->type;
     const ABIArgInfo &ArgI = info_it->info;
 
     bool isPromoted =
       isa<ParmVarDecl>(Arg) && cast<ParmVarDecl>(Arg)->isKNRPromoted();
 
-    // Skip the dummy padding argument.
-    if (ArgI.getPaddingType())
-      ++AI;
+    unsigned FirstIRArg, NumIRArgs;
+    std::tie(FirstIRArg, NumIRArgs) = IRFunctionArgs.getIRArgs(ArgNo);
 
     switch (ArgI.getKind()) {
     case ABIArgInfo::InAlloca: {
+      assert(NumIRArgs == 0);
       llvm::Value *V = Builder.CreateStructGEP(
           ArgStruct, ArgI.getInAllocaFieldIndex(), Arg->getName());
       ArgVals.push_back(ValueAndIsPtr(V, HavePointer));
-      continue;  // Don't increment AI!
+      break;
     }
 
     case ABIArgInfo::Indirect: {
-      llvm::Value *V = AI;
+      assert(NumIRArgs == 1);
+      llvm::Value *V = FnArgs[FirstIRArg];
 
       if (!hasScalarEvaluationKind(Ty)) {
         // Aggregates and complex variables are accessed by reference.  All we
         // need to do is realign the value, if requested
         if (ArgI.getIndirectRealign()) {
           llvm::Value *AlignedTemp = CreateMemTemp(Ty, "coerce");
 
           // Copy from the incoming argument pointer to the temporary with the
for (FunctionArgList::const_iterator i = Args.begin(), e = Args.end();
 
     case ABIArgInfo::Extend:
     case ABIArgInfo::Direct: {
 
       // If we have the trivial case, handle it with no muss and fuss.
       if (!isa<llvm::StructType>(ArgI.getCoerceToType()) &&
           ArgI.getCoerceToType() == ConvertType(Ty) &&
           ArgI.getDirectOffset() == 0) {
-        assert(AI != Fn->arg_end() && "Argument mismatch!");
+        assert(NumIRArgs == 1);
+        auto AI = FnArgs[FirstIRArg];
         llvm::Value *V = AI;
 
         if (const ParmVarDecl *PVD = dyn_cast<ParmVarDecl>(Arg)) {
           if ((NNAtt && NNAtt->isNonNull(PVD->getFunctionScopeIndex())) ||
               PVD->hasAttr<NonNullAttr>())
             AI->addAttr(llvm::AttributeSet::get(getLLVMContext(),
                                                 AI->getArgNo() + 1,
                                                 llvm::Attribute::NonNull));
case ABIArgInfo::Direct: {
         uint64_t SrcSize = CGM.getDataLayout().getTypeAllocSize(STy);
         llvm::Type *DstTy =
           cast<llvm::PointerType>(Ptr->getType())->getElementType();
         uint64_t DstSize = CGM.getDataLayout().getTypeAllocSize(DstTy);
 
         if (SrcSize <= DstSize) {
           Ptr = Builder.CreateBitCast(Ptr, llvm::PointerType::getUnqual(STy));
 
+          assert(STy->getNumElements() == NumIRArgs);
           for (unsigned i = 0, e = STy->getNumElements(); i != e; ++i) {
-            assert(AI != Fn->arg_end() && "Argument mismatch!");
+            auto AI = FnArgs[FirstIRArg + i];
             AI->setName(Arg->getName() + ".coerce" + Twine(i));
             llvm::Value *EltPtr = Builder.CreateConstGEP2_32(Ptr, 0, i);
-            Builder.CreateStore(AI++, EltPtr);
+            Builder.CreateStore(AI, EltPtr);
           }
         } else {
           llvm::AllocaInst *TempAlloca =
             CreateTempAlloca(ArgI.getCoerceToType(), "coerce");
           TempAlloca->setAlignment(AlignmentToUse);
           llvm::Value *TempV = TempAlloca;
 
+          assert(STy->getNumElements() == NumIRArgs);
           for (unsigned i = 0, e = STy->getNumElements(); i != e; ++i) {
-            assert(AI != Fn->arg_end() && "Argument mismatch!");
+            auto AI = FnArgs[FirstIRArg + i];
             AI->setName(Arg->getName() + ".coerce" + Twine(i));
             llvm::Value *EltPtr = Builder.CreateConstGEP2_32(TempV, 0, i);
-            Builder.CreateStore(AI++, EltPtr);
+            Builder.CreateStore(AI, EltPtr);
           }
 
           Builder.CreateMemCpy(Ptr, TempV, DstSize, AlignmentToUse);
         }
       } else {
         // Simple case, just do a coerced store of the argument into the alloca.
-        assert(AI != Fn->arg_end() && "Argument mismatch!");
+        assert(NumIRArgs == 1);
+        auto AI = FnArgs[FirstIRArg];
         AI->setName(Arg->getName() + ".coerce");
-        CreateCoercedStore(AI++, Ptr, /*DestIsVolatile=*/false, *this);
+        CreateCoercedStore(AI, Ptr, /*DestIsVolatile=*/false, *this);
       }
 
 
       // Match to what EmitParmDecl is expecting for this type.
       if (CodeGenFunction::hasScalarEvaluationKind(Ty)) {
         V = EmitLoadOfScalar(V, false, AlignmentToUse, Ty, Arg->getLocStart());
         if (isPromoted)
           V = emitArgumentDemotion(*this, Arg, V);
         ArgVals.push_back(ValueAndIsPtr(V, HaveValue));
       } else {
         ArgVals.push_back(ValueAndIsPtr(V, HavePointer));
       }
-      continue;  // Skip ++AI increment, already done.
+      break;
     }
 
     case ABIArgInfo::Expand: {
       // If this structure was expanded into multiple arguments then
       // we need to create a temporary and reconstruct it from the
       // arguments.
       llvm::AllocaInst *Alloca = CreateMemTemp(Ty);
       CharUnits Align = getContext().getDeclAlign(Arg);
       Alloca->setAlignment(Align.getQuantity());
       LValue LV = MakeAddrLValue(Alloca, Ty, Align);
-      llvm::Function::arg_iterator End = ExpandTypeFromArgs(Ty, LV, AI);
       ArgVals.push_back(ValueAndIsPtr(Alloca, HavePointer));
 
-      // Name the arguments used in expansion and increment AI.
-      unsigned Index = 0;
-      for (; AI != End; ++AI, ++Index)
-        AI->setName(Arg->getName() + "." + Twine(Index));
-      continue;
+      auto FnArgIter = FnArgs.begin() + FirstIRArg;
+      ExpandTypeFromArgs(Ty, LV, FnArgIter);
+      assert(FnArgIter == FnArgs.begin() + FirstIRArg + NumIRArgs);
+      for (unsigned i = 0, e = NumIRArgs; i != e; ++i) {
+        auto AI = FnArgs[FirstIRArg + i];
+        AI->setName(Arg->getName() + "." + Twine(i));
+      }
+      break;
     }
 
     case ABIArgInfo::Ignore:
+      assert(NumIRArgs == 0);
       // Initialize the local variable appropriately.
       if (!hasScalarEvaluationKind(Ty)) {
         ArgVals.push_back(ValueAndIsPtr(CreateMemTemp(Ty), HavePointer));
       } else {
         llvm::Value *U = llvm::UndefValue::get(ConvertType(Arg->getType()));
         ArgVals.push_back(ValueAndIsPtr(U, HaveValue));
       }
-
-      // Skip increment, no matching LLVM parameter.
-      continue;
+      break;
     }
-
-    ++AI;
-
-    if (ArgNo == 1 && SwapThisWithSRet)
-      ++AI;  // Skip the sret parameter.
   }
 
-  if (FI.usesInAlloca())
-    ++AI;
-  assert(AI == Fn->arg_end() && "Argument mismatch!");
-
   if (getTarget().getCXXABI().areArgsDestroyedLeftToRightInCallee()) {
     for (int I = Args.size() - 1; I >= 0; --I)
       EmitParmDecl(*Args[I], ArgVals[I].getPointer(), ArgVals[I].getInt(),
                    I + 1);
   } else {
     for (unsigned I = 0, E = Args.size(); I != E; ++I)
       EmitParmDecl(*Args[I], ArgVals[I].getPointer(), ArgVals[I].getInt(),
                    I + 1);
CodeGenFunction::EmitCallOrInvoke(llvm::Value *Callee,
   // In ObjC ARC mode with no ObjC ARC exception safety, tell the ARC
   // optimizer it can aggressively ignore unwind edges.
   if (CGM.getLangOpts().ObjCAutoRefCount)
     AddObjCARCExceptionMetadata(Inst);
 
   return Inst;
 }
 
-static void checkArgMatches(llvm::Value *Elt, unsigned &ArgNo,
-                            llvm::FunctionType *FTy) {
-  if (ArgNo < FTy->getNumParams())
-    assert(Elt->getType() == FTy->getParamType(ArgNo));
-  else
-    assert(FTy->isVarArg());
-  ++ArgNo;
-}
-
-void CodeGenFunction::ExpandTypeToArgs(QualType Ty, RValue RV,
-                                       SmallVectorImpl<llvm::Value *> &Args,
-                                       llvm::FunctionType *IRFuncTy) {
+void CodeGenFunction::ExpandTypeToArgs(
+    QualType Ty, RValue RV, llvm::FunctionType *IRFuncTy,
+    SmallVectorImpl<llvm::Value *> &IRCallArgs, unsigned &IRCallArgPos) {
   if (const ConstantArrayType *AT = getContext().getAsConstantArrayType(Ty)) {
     unsigned NumElts = AT->getSize().getZExtValue();
     QualType EltTy = AT->getElementType();
     llvm::Value *Addr = RV.getAggregateAddr();
     for (unsigned Elt = 0; Elt < NumElts; ++Elt) {
       llvm::Value *EltAddr = Builder.CreateConstGEP2_32(Addr, 0, Elt);
       RValue EltRV = convertTempToRValue(EltAddr, EltTy, SourceLocation());
-      ExpandTypeToArgs(EltTy, EltRV, Args, IRFuncTy);
+      ExpandTypeToArgs(EltTy, EltRV, IRFuncTy, IRCallArgs, IRCallArgPos);
     }
   } else if (const RecordType *RT = Ty->getAs<RecordType>()) {
     RecordDecl *RD = RT->getDecl();
     assert(RV.isAggregate() && "Unexpected rvalue during struct expansion");
     LValue LV = MakeAddrLValue(RV.getAggregateAddr(), Ty);
 
     if (RD->isUnion()) {
       const FieldDecl *LargestFD = nullptr;
       CharUnits UnionSize = CharUnits::Zero();
 
       for (const auto *FD : RD->fields()) {
         assert(!FD->isBitField() &&
                "Cannot expand structure with bit-field members.");
         CharUnits FieldSize = getContext().getTypeSizeInChars(FD->getType());
         if (UnionSize < FieldSize) {
           UnionSize = FieldSize;
           LargestFD = FD;
         }
       }
       if (LargestFD) {
         RValue FldRV = EmitRValueForField(LV, LargestFD, SourceLocation());
-        ExpandTypeToArgs(LargestFD->getType(), FldRV, Args, IRFuncTy);
+        ExpandTypeToArgs(LargestFD->getType(), FldRV, IRFuncTy, IRCallArgs,
+                         IRCallArgPos);
       }
     } else {
       for (const auto *FD : RD->fields()) {
         RValue FldRV = EmitRValueForField(LV, FD, SourceLocation());
-        ExpandTypeToArgs(FD->getType(), FldRV, Args, IRFuncTy);
+        ExpandTypeToArgs(FD->getType(), FldRV, IRFuncTy, IRCallArgs, IRCallArgPos);
       }
     }
   } else if (Ty->isAnyComplexType()) {
     ComplexPairTy CV = RV.getComplexVal();
-    Args.push_back(CV.first);
-    Args.push_back(CV.second);
+    IRCallArgs[IRCallArgPos++] = CV.first;
+    IRCallArgs[IRCallArgPos++] = CV.second;
   } else {
     assert(RV.isScalar() &&
            "Unexpected non-scalar rvalue during struct expansion.");
 
     // Insert a bitcast as needed.
     llvm::Value *V = RV.getScalarVal();
-    if (Args.size() < IRFuncTy->getNumParams() &&
-        V->getType() != IRFuncTy->getParamType(Args.size()))
-      V = Builder.CreateBitCast(V, IRFuncTy->getParamType(Args.size()));
+    if (IRCallArgPos < IRFuncTy->getNumParams() &&
+        V->getType() != IRFuncTy->getParamType(IRCallArgPos))
+      V = Builder.CreateBitCast(V, IRFuncTy->getParamType(IRCallArgPos));
 
-    Args.push_back(V);
+    IRCallArgs[IRCallArgPos++] = V;
   }
 }
 
 /// \brief Store a non-aggregate value to an address to initialize it.  For
 /// initialization, a non-atomic store will be used.
 static void EmitInitStoreOfNonAggregate(CodeGenFunction &CGF, RValue Src,
                                         LValue Dst) {
   if (Src.isScalar())

 
 RValue CodeGenFunction::EmitCall(const CGFunctionInfo &CallInfo,
                                  llvm::Value *Callee,
                                  ReturnValueSlot ReturnValue,
                                  const CallArgList &CallArgs,
                                  const Decl *TargetDecl,
                                  llvm::Instruction **callOrInvoke) {
   // FIXME: We no longer need the types from CallArgs; lift up and simplify.
-  SmallVector<llvm::Value*, 16> Args;
 
   // Handle struct-return functions by passing a pointer to the
   // location that we would like to return into.
   QualType RetTy = CallInfo.getReturnType();
   const ABIArgInfo &RetAI = CallInfo.getReturnInfo();
 
-  // IRArgNo - Keep track of the argument number in the callee we're looking at.
-  unsigned IRArgNo = 0;
   llvm::FunctionType *IRFuncTy =
     cast<llvm::FunctionType>(
                   cast<llvm::PointerType>(Callee->getType())->getElementType());
 
   // If we're using inalloca, insert the allocation after the stack save.
   // FIXME: Do this earlier rather than hacking it in here!
   llvm::Value *ArgMemory = nullptr;
   if (llvm::StructType *ArgStruct = CallInfo.getArgStruct()) {
     llvm::Instruction *IP = CallArgs.getStackBase();
     llvm::AllocaInst *AI;
     if (IP) {
       IP = IP->getNextNode();
       AI = new llvm::AllocaInst(ArgStruct, "argmem", IP);
     } else {
       AI = CreateTempAlloca(ArgStruct, "argmem");
     }
     AI->setUsedWithInAlloca(true);
     assert(AI->isUsedWithInAlloca() && !AI->isStaticAlloca());
     ArgMemory = AI;
   }
 
+  ClangToLLVMArgMapping IRFunctionArgs(CGM, CallInfo);
+  SmallVector<llvm::Value *, 16> IRCallArgs(IRFunctionArgs.totalIRArgs());
+
   // If the call returns a temporary with struct return, create a temporary
   // alloca to hold the result, unless one is given to us.
   llvm::Value *SRetPtr = nullptr;
-  bool SwapThisWithSRet = false;
   if (RetAI.isIndirect() || RetAI.isInAlloca()) {
     SRetPtr = ReturnValue.getValue();
     if (!SRetPtr)
       SRetPtr = CreateMemTemp(RetTy);
-    if (RetAI.isIndirect()) {
-      Args.push_back(SRetPtr);
-      SwapThisWithSRet = RetAI.isSRetAfterThis();
-      if (SwapThisWithSRet)
-        IRArgNo = 1;
-      checkArgMatches(SRetPtr, IRArgNo, IRFuncTy);
-      if (SwapThisWithSRet)
-        IRArgNo = 0;
+    if (IRFunctionArgs.hasSRetArg()) {
+      IRCallArgs[IRFunctionArgs.getSRetArgNo()] = SRetPtr;
     } else {
       llvm::Value *Addr =
           Builder.CreateStructGEP(ArgMemory, RetAI.getInAllocaFieldIndex());
       Builder.CreateStore(SRetPtr, Addr);
     }
   }
 
   assert(CallInfo.arg_size() == CallArgs.size() &&
          "Mismatch between function signature & arguments.");
+  unsigned ArgNo = 0;
   CGFunctionInfo::const_arg_iterator info_it = CallInfo.arg_begin();
   for (CallArgList::const_iterator I = CallArgs.begin(), E = CallArgs.end();
-       I != E; ++I, ++info_it) {
+       I != E; ++I, ++info_it, ++ArgNo) {
     const ABIArgInfo &ArgInfo = info_it->info;
     RValue RV = I->RV;
 
-    // Skip 'sret' if it came second.
-    if (IRArgNo == 1 && SwapThisWithSRet)
-      ++IRArgNo;
-
     CharUnits TypeAlign = getContext().getTypeAlignInChars(I->Ty);
 
     // Insert a padding argument to ensure proper alignment.
-    if (llvm::Type *PaddingType = ArgInfo.getPaddingType()) {
-      Args.push_back(llvm::UndefValue::get(PaddingType));
-      ++IRArgNo;
-    }
+    if (IRFunctionArgs.hasPaddingArg(ArgNo))
+      IRCallArgs[IRFunctionArgs.getPaddingArgNo(ArgNo)] =
+          llvm::UndefValue::get(ArgInfo.getPaddingType());
+
+    unsigned FirstIRArg, NumIRArgs;
+    std::tie(FirstIRArg, NumIRArgs) = IRFunctionArgs.getIRArgs(ArgNo);
 
     switch (ArgInfo.getKind()) {
     case ABIArgInfo::InAlloca: {
+      assert(NumIRArgs == 0);
       assert(getTarget().getTriple().getArch() == llvm::Triple::x86);
       if (RV.isAggregate()) {
         // Replace the placeholder with the appropriate argument slot GEP.
         llvm::Instruction *Placeholder =
             cast<llvm::Instruction>(RV.getAggregateAddr());
         CGBuilderTy::InsertPoint IP = Builder.saveIP();
         Builder.SetInsertPoint(Placeholder);
         llvm::Value *Addr = Builder.CreateStructGEP(
case ABIArgInfo::InAlloca: {
         // There are some cases where a trivial bitcast is not avoidable.  The
         // definition of a type later in a translation unit may change it's type
         // from {}* to (%struct.foo*)*.
         if (Addr->getType() != MemType)
           Addr = Builder.CreateBitCast(Addr, MemType);
         LValue argLV = MakeAddrLValue(Addr, I->Ty, TypeAlign);
         EmitInitStoreOfNonAggregate(*this, RV, argLV);
       }
-      break; // Don't increment IRArgNo!
+      break;
     }
 
     case ABIArgInfo::Indirect: {
+      assert(NumIRArgs == 1);
       if (RV.isScalar() || RV.isComplex()) {
         // Make a temporary alloca to pass the argument.
         llvm::AllocaInst *AI = CreateMemTemp(I->Ty);
         if (ArgInfo.getIndirectAlign() > AI->getAlignment())
           AI->setAlignment(ArgInfo.getIndirectAlign());
-        Args.push_back(AI);
+        IRCallArgs[FirstIRArg] = AI;
 
-        LValue argLV = MakeAddrLValue(Args.back(), I->Ty, TypeAlign);
+        LValue argLV = MakeAddrLValue(AI, I->Ty, TypeAlign);
         EmitInitStoreOfNonAggregate(*this, RV, argLV);
-        
-        // Validate argument match.
-        checkArgMatches(AI, IRArgNo, IRFuncTy);
       } else {
         // We want to avoid creating an unnecessary temporary+copy here;
         // however, we need one in three cases:
         // 1. If the argument is not byval, and we are required to copy the
         //    source.  (This case doesn't occur on any common architecture.)
         // 2. If the argument is byval, RV is not sufficiently aligned, and
         //    we cannot force it to be sufficiently aligned.
         // 3. If the argument is byval, but RV is located in an address space
         //    different than that of the argument (0).
         llvm::Value *Addr = RV.getAggregateAddr();
         unsigned Align = ArgInfo.getIndirectAlign();
         const llvm::DataLayout *TD = &CGM.getDataLayout();
         const unsigned RVAddrSpace = Addr->getType()->getPointerAddressSpace();
-        const unsigned ArgAddrSpace = (IRArgNo < IRFuncTy->getNumParams() ?
-          IRFuncTy->getParamType(IRArgNo)->getPointerAddressSpace() : 0);
+        const unsigned ArgAddrSpace =
+            (FirstIRArg < IRFuncTy->getNumParams()
+                 ? IRFuncTy->getParamType(FirstIRArg)->getPointerAddressSpace()
+                 : 0);
         if ((!ArgInfo.getIndirectByVal() && I->NeedsCopy) ||
             (ArgInfo.getIndirectByVal() && TypeAlign.getQuantity() < Align &&
              llvm::getOrEnforceKnownAlignment(Addr, Align, TD) < Align) ||
              (ArgInfo.getIndirectByVal() && (RVAddrSpace != ArgAddrSpace))) {
           // Create an aligned temporary, and copy to it.
           llvm::AllocaInst *AI = CreateMemTemp(I->Ty);
           if (Align > AI->getAlignment())
             AI->setAlignment(Align);
-          Args.push_back(AI);
+          IRCallArgs[FirstIRArg] = AI;
           EmitAggregateCopy(AI, Addr, I->Ty, RV.isVolatileQualified());
-              
-          // Validate argument match.
-          checkArgMatches(AI, IRArgNo, IRFuncTy);
         } else {
           // Skip the extra memcpy call.
-          Args.push_back(Addr);
-          
-          // Validate argument match.
-          checkArgMatches(Addr, IRArgNo, IRFuncTy);
+          IRCallArgs[FirstIRArg] = Addr;
         }
       }
       break;
     }
 
     case ABIArgInfo::Ignore:
+      assert(NumIRArgs == 0);
       break;
 
     case ABIArgInfo::Extend:
     case ABIArgInfo::Direct: {
       if (!isa<llvm::StructType>(ArgInfo.getCoerceToType()) &&
           ArgInfo.getCoerceToType() == ConvertType(info_it->type) &&
           ArgInfo.getDirectOffset() == 0) {
+        assert(NumIRArgs == 1);
         llvm::Value *V;
         if (RV.isScalar())
           V = RV.getScalarVal();
         else
           V = Builder.CreateLoad(RV.getAggregateAddr());
-        
         // If the argument doesn't match, perform a bitcast to coerce it.  This
         // can happen due to trivial type mismatches.
-        if (IRArgNo < IRFuncTy->getNumParams() &&
-            V->getType() != IRFuncTy->getParamType(IRArgNo))
-          V = Builder.CreateBitCast(V, IRFuncTy->getParamType(IRArgNo));
-        Args.push_back(V);
-        
-        checkArgMatches(V, IRArgNo, IRFuncTy);
+        if (FirstIRArg < IRFuncTy->getNumParams() &&
+            V->getType() != IRFuncTy->getParamType(FirstIRArg))
+          V = Builder.CreateBitCast(V, IRFuncTy->getParamType(FirstIRArg));
+        IRCallArgs[FirstIRArg] = V;
         break;
       }
 
       // FIXME: Avoid the conversion through memory if possible.
       llvm::Value *SrcPtr;
       if (RV.isScalar() || RV.isComplex()) {
         SrcPtr = CreateMemTemp(I->Ty, "coerce");
         LValue SrcLV = MakeAddrLValue(SrcPtr, I->Ty, TypeAlign);
case ABIArgInfo::Direct: {
             = CreateTempAlloca(STy, SrcPtr->getName() + ".coerce");
           Builder.CreateMemCpy(TempAlloca, SrcPtr, SrcSize, 0);
           SrcPtr = TempAlloca;
         } else {
           SrcPtr = Builder.CreateBitCast(SrcPtr,
                                          llvm::PointerType::getUnqual(STy));
         }
 
+        assert(NumIRArgs == STy->getNumElements());
         for (unsigned i = 0, e = STy->getNumElements(); i != e; ++i) {
           llvm::Value *EltPtr = Builder.CreateConstGEP2_32(SrcPtr, 0, i);
           llvm::LoadInst *LI = Builder.CreateLoad(EltPtr);
           // We don't know what we're loading from.
           LI->setAlignment(1);
-          Args.push_back(LI);
-
-          // Validate argument match.
-          checkArgMatches(LI, IRArgNo, IRFuncTy);
+          IRCallArgs[FirstIRArg + i] = LI;
         }
       } else {
         // In the simple case, just pass the coerced loaded value.
-        Args.push_back(CreateCoercedLoad(SrcPtr, ArgInfo.getCoerceToType(),
-                                         *this));
-
-        // Validate argument match.
-        checkArgMatches(Args.back(), IRArgNo, IRFuncTy);
+        assert(NumIRArgs == 1);
+        IRCallArgs[FirstIRArg] =
+            CreateCoercedLoad(SrcPtr, ArgInfo.getCoerceToType(), *this);
       }
 
       break;
     }
 
     case ABIArgInfo::Expand:
-      ExpandTypeToArgs(I->Ty, RV, Args, IRFuncTy);
-      IRArgNo = Args.size();
+      unsigned IRArgPos = FirstIRArg;
+      ExpandTypeToArgs(I->Ty, RV, IRFuncTy, IRCallArgs, IRArgPos);
+      assert(IRArgPos == FirstIRArg + NumIRArgs);
       break;
     }
   }
 
-  if (SwapThisWithSRet)
-    std::swap(Args[0], Args[1]);
-
   if (ArgMemory) {
     llvm::Value *Arg = ArgMemory;
     if (CallInfo.isVariadic()) {
       // When passing non-POD arguments by value to variadic functions, we will
       // end up with a variadic prototype and an inalloca call site.  In such
       // cases, we can't do any parameter mismatch checks.  Give up and bitcast
       // the callee.
       unsigned CalleeAS =
#ifndef NDEBUG
                                                 DE = DeclaredTy->element_end(),
                                                 FI = FullTy->element_begin();
              DI != DE; ++DI, ++FI)
           assert(*DI == *FI);
 #endif
         Arg = Builder.CreateBitCast(Arg, LastParamTy);
       }
     }
-    Args.push_back(Arg);
+    assert(IRFunctionArgs.hasInallocaArg());
+    IRCallArgs[IRFunctionArgs.getInallocaArgNo()] = Arg;
   }
 
   if (!CallArgs.getCleanupsToDeactivate().empty())
     deactivateArgCleanupsBeforeCall(*this, CallArgs);
 
   // If the callee is a bitcast of a function to a varargs pointer to function
   // type, check to see if we can remove the bitcast.  This handles some cases
   // with unprototyped functions.
   if (llvm::ConstantExpr *CE = dyn_cast<llvm::ConstantExpr>(Callee))
     if (llvm::Function *CalleeF = dyn_cast<llvm::Function>(CE->getOperand(0))) {
       llvm::PointerType *CurPT=cast<llvm::PointerType>(Callee->getType());
       llvm::FunctionType *CurFT =
         cast<llvm::FunctionType>(CurPT->getElementType());
       llvm::FunctionType *ActualFT = CalleeF->getFunctionType();
 
       if (CE->getOpcode() == llvm::Instruction::BitCast &&
           ActualFT->getReturnType() == CurFT->getReturnType() &&
           ActualFT->getNumParams() == CurFT->getNumParams() &&
-          ActualFT->getNumParams() == Args.size() &&
+          ActualFT->getNumParams() == IRCallArgs.size() &&
           (CurFT->isVarArg() || !ActualFT->isVarArg())) {
         bool ArgsMatch = true;
         for (unsigned i = 0, e = ActualFT->getNumParams(); i != e; ++i)
           if (ActualFT->getParamType(i) != CurFT->getParamType(i)) {
             ArgsMatch = false;
             break;
           }
 
         // Strip the cast if we can get away with it.  This is a nice cleanup,
         // but also allows us to inline the function at -O0 if it is marked
         // always_inline.
         if (ArgsMatch)
           Callee = CalleeF;
       }
     }
 
+  assert(IRCallArgs.size() == IRFuncTy->getNumParams() || IRFuncTy->isVarArg());
+  for (unsigned i = 0; i < IRCallArgs.size(); ++i) {
+    // Inalloca argument can have different type.
+    if (IRFunctionArgs.hasInallocaArg() &&
+        i == IRFunctionArgs.getInallocaArgNo())
+      continue;
+    if (i < IRFuncTy->getNumParams())
+      assert(IRCallArgs[i]->getType() == IRFuncTy->getParamType(i));
+  }
+
   unsigned CallingConv;
   CodeGen::AttributeListType AttributeList;
   CGM.ConstructAttributeList(CallInfo, TargetDecl, AttributeList,
                              CallingConv, true);
   llvm::AttributeSet Attrs = llvm::AttributeSet::get(getLLVMContext(),
                                                      AttributeList);
 
   llvm::BasicBlock *InvokeDest = nullptr;
   if (!Attrs.hasAttribute(llvm::AttributeSet::FunctionIndex,
                           llvm::Attribute::NoUnwind))
     InvokeDest = getInvokeDest();
 
   llvm::CallSite CS;
   if (!InvokeDest) {
-    CS = Builder.CreateCall(Callee, Args);
+    CS = Builder.CreateCall(Callee, IRCallArgs);
   } else {
     llvm::BasicBlock *Cont = createBasicBlock("invoke.cont");
-    CS = Builder.CreateInvoke(Callee, Cont, InvokeDest, Args);
+    CS = Builder.CreateInvoke(Callee, Cont, InvokeDest, IRCallArgs);
     EmitBlock(Cont);
   }
   if (callOrInvoke)
     *callOrInvoke = CS.getInstruction();
 
   if (CurCodeDecl && CurCodeDecl->hasAttr<FlattenAttr>() &&
       !CS.hasFnAttr(llvm::Attribute::NoInline))
     Attrs =

lib/CodeGen/CodeGenFunction.h

private:
 
   llvm::SmallVector<std::pair<llvm::Instruction *, llvm::Value *>, 4>
   DeferredReplacements;
 
   /// ExpandTypeFromArgs - Reconstruct a structure of type \arg Ty
   /// from function arguments into \arg Dst. See ABIArgInfo::Expand.
   ///
   /// \param AI - The first function argument of the expansion.
-  /// \return The argument following the last expanded function
-  /// argument.
-  llvm::Function::arg_iterator
-  ExpandTypeFromArgs(QualType Ty, LValue Dst,
-                     llvm::Function::arg_iterator AI);
-
-  /// ExpandTypeToArgs - Expand an RValue \arg Src, with the LLVM type for \arg
-  /// Ty, into individual arguments on the provided vector \arg Args. See
-  /// ABIArgInfo::Expand.
-  void ExpandTypeToArgs(QualType Ty, RValue Src,
-                        SmallVectorImpl<llvm::Value *> &Args,
-                        llvm::FunctionType *IRFuncTy);
+  void ExpandTypeFromArgs(QualType Ty, LValue Dst,
+                          SmallVectorImpl<llvm::Argument *>::iterator &AI);
+
+  /// ExpandTypeToArgs - Expand an RValue \arg RV, with the LLVM type for \arg
+  /// Ty, into individual arguments on the provided vector \arg IRCallArgs,
+  /// starting at index \arg IRCallArgPos. See ABIArgInfo::Expand.
+  void ExpandTypeToArgs(QualType Ty, RValue RV, llvm::FunctionType *IRFuncTy,
+                        SmallVectorImpl<llvm::Value *> &IRCallArgs,
+                        unsigned &IRCallArgPos);
 
   llvm::Value* EmitAsmInput(const TargetInfo::ConstraintInfo &Info,
                             const Expr *InputExpr, std::string &ConstraintStr);
 
   llvm::Value* EmitAsmInputLValue(const TargetInfo::ConstraintInfo &Info,
                                   LValue InputValue, QualType InputType,
                                   std::string &ConstraintStr,
                                   SourceLocation Loc);