[ISEL][X86] Tracking of registers that forward call arguments

[djtodoro]

Summary

While lowering calls collect info about registers that forward arguments into following function frame. We store such info into MachineFunction of the call.
This is used very late when dumping DWARF info about call site parameters.

NOTE: Other architectures should follow similar procedure buy collecting call argument forwarding register in theirs LowerCallspecialization. Once the call site info is collected it should be bound to SDNode that represents the call instruction. Backend support should keep track about updating MachineInstr call instructions. For more info about backend call site support see https://reviews.llvm.org/D61062.

Authors: @asowda, @NikolaPrica, @djtodoro, @ivanbaev

[aprantl]

Looking through the file list, this seems to be missing FastISel and GlobalIsel support?

include/llvm/CodeGen/MIRYamlMapping.h

340

This could use some Doxygen comments :-)

[bjope]

Looks generally good. But...

I think the subject and summary of this patch is a little bit misleading. It talks about ISEL and lowering of calls, but that only seems to be a small part of it.

Afaict this patch only implements the "full" call lowering support for X86, right? What should happen for other targets when using -emit-call-site-info (afaict SelectionDAG still would generate some callsite information)?
Should probably elaborate a little bit about only supporting X86 and what other targets would need to do.

A substantial part of this patch is touching Codegen in a more general way to support storing the callsite info in the MachineFunction, and serializing that information in the MIR format. Not really related to ISEL IMHO :-)
Should perhaps MIRLangReg.rst be updated in this patch to describe the new additions (or is that covered by another patch in the stack)?.

The patch is also updating passes like InlineSpiller, PeepholeOptimizer and MachineOutliner (this is not about ISEL either) to support keeping the callsite info valid and up-to-date through the backend.
I guess this is putting some new requirements on all backend passes. It would be good if the summary (commit msg) would explain a little bit more about the implications with the solution (such as when MF.updateCallSiteInfo(...); should be used). If I for example want to use this functionality in my OOT target such information is extra helpful, as I need to make all such patches myself. But such information could also be useful for in-tree development, e.g. for reviewers to understand if there are more places to update that you have missed (or for other patches that would need to rebase onto this patch later).

lib/CodeGen/TargetInstrInfo.cpp

138–146

Not sure if it is worth the complexity to mix the update of call site info and erasing in the same loop here? Always tricky to remove elements while iterating over the same data structure (even if I think you got it correct here).

test/CodeGen/X86/call-site-info-output.ll

8

Afaict offset: 14 just matches what we currently get (so it is not important that we get 14 here). We should check for offset: {{[0-9]+}} instead, to make the testcase less sensitive to future unrelated changes in lowering, right?

12

This verifies that the parser accepts the new syntax. But not really that it does anything useful with it (it could just discard it and this test would still pass). Maybe that is supposed to be tested elsewhere? Otherwise one idea could be to stop before expand-isel-pseudos in the RUN above, then run expand-isel-pseudos and check that we still get the same callSites info in the output, and then start-after expand-isel-pseudos for the last part of the testing.

[NikolaPrica]

@bjope @aprantl Thank you for your comments!

In D60715#1469515, @aprantl wrote:

Looking through the file list, this seems to be missing FastISel and GlobalIsel support?

Yes. We taught that it might go as a separate patch? Or should we add it all together?

Firstly this patch was in two parts but for the first review we left it as this in order to see big picture. We will separate it on ISEL part and backend handling part for sure!

Should probably elaborate a little bit about only supporting X86 and what other targets would need to do.

We will do that.

A substantial part of this patch is touching Codegen in a more general way to support storing the callsite info in the MachineFunction, and serializing that information in the MIR format. Not really related to ISEL IMHO :-)
Should perhaps MIRLangReg.rst be updated in this patch to describe the new additions (or is that covered by another patch in the stack)?.

You are right about this but since we produce such data at ISEL phase we saw it as whole. We provide it like that in order to be able to provide tests that we actually changed something in MachineFunction. Like this we can add test that MIR could be parsed by reading .mir produced from test file. But we can separate it on general part that can read given MIR format and ISEL part for X86 that would test writing MIR? Do you think that it will look better like that?

include/llvm/CodeGen/MIRYamlMapping.h

340

Oh sorry about that. Will add those!

lib/CodeGen/TargetInstrInfo.cpp

138–146

This is good question since we have some doubts about this. All updateCallSiteInfo(MI) are used to inform MachineFunction.CallSitesInfo that referenced call instruction got deleted and that it needs update. Now such update can be moved to MachineFunction::DeleteMachineInstr to preserve valid state of CallSiteInfo but we are not quite sure about it since it is frequently used function.
There we left assertion (it should be warped by NDEBUG or removed) that is tested for X86 but will definitely be useful for adding call site info support for other architectures. That assertion brought us to fix this function.
General question is should we move preservation of call site information state to MachineFunction::DeleteMachineInstr or just use assertion in it to point us where should we insert updateCallSiteInfo(MI); ?

test/CodeGen/X86/call-site-info-output.ll

12

Yes, you are right. This only verifies new syntax. Usage of this information is tested in the latest patch. Here we test output for call-site. With new RUN we can test parsing of new syntax and by so there will be no need to test "offset" since parser will verify that information. I will add this as comment for the test.

[djtodoro]

-Rebase

[aprantl]

include/llvm/CodeGen/MachineFunction.h

381

How about using a struct with field names instead of a std::pair?

955

/p -> \p

include/llvm/CodeGen/SelectionDAG.h

1634

This function appears to reimplement the operator[] of DenseMap?
I guess it doesn't insert the value...

lib/CodeGen/MIRPrinter.cpp

476

Can we replace this bodyless loop with something from std::algorithm?

lib/CodeGen/MachineFunction.cpp

365

assert (!MI->isCall(MachineInstr::IgnoreBundle) || ...)

lib/CodeGen/MachineVerifier.cpp

2150

typo

lib/CodeGen/TargetInstrInfo.cpp

140

clang-format

145

I'd prefer writing two loops (or even better two std::algorithms) for updating and erasing and let the optimizer merge them.

lib/CodeGen/XRayInstrumentation.cpp

114

So how do we make sure that people will call this in their new passes form here on? Can we verify it?

[NikolaPrica]

Thanks for the comments! I've addressed most of them. This patch will be split on 3 parts once we decide how to address comment about updating call site in TargetInstrInfo.

lib/CodeGen/TargetInstrInfo.cpp

145

I've tried with std::remove_if and llvm::remove_if but they require deleted MachineInstr copy assignment operator. So I came up with two solutions:

Solution 1

// Update call site info and remove all the dead instructions
 // from the end of MBB.
 while (Tail != MBB->end()) {
   auto MI = Tail++;
   if (MI->isCall())
     MBB->getParent()->updateCallSiteInfo(&*MI);
   MBB->erase(MI);
 }

Or

Solution 2

// Update call site info.
 for (auto It = Tail; It != MBB->end(); ++It)
   if (It->isCall())
     MBB->getParent()->updateCallSiteInfo(&*It)

 // Remove all the dead instructions from the end of MBB.
 for (auto It = Tail; It != MBB->end();) {
   auto MI = It++;
   MBB->erase(MI);
 }

@aprantl First example seams better than existing one. I'm wondering whether second with two loops is what you aimed to?

[bjope]

lib/CodeGen/TargetInstrInfo.cpp

145

The second loop in "solution 2" can be reduced to MBB->erase(Tail, MBB->end()); (that is what it used to look like). No need to complicate that part ;-)

[bjope]

lib/CodeGen/MIRPrinter.cpp

490

From CodingStandards.rst:

As a rule of thumb, always make sure to use llvm::sort instead of std::sort.

[NikolaPrica]

lib/CodeGen/MIRPrinter.cpp

490

Thanks for pointing that!

lib/CodeGen/TargetInstrInfo.cpp

145

MBB->erase(Tail, MBB->end()); recursively deletes elements. I taught that Adrian aimed to usage of two loops because optimizer will be able to merge them. Not sure whether it will be able to merge loop and recursive deletion call.

[djtodoro]

-Split up of introduction, production and handling of call site info

[lebedev.ri]

include/llvm/CodeGen/SelectionDAG.h

1633

The function is marked as const, yet this is a std::move().
Do you mean to only get the CallSiteInfo once?

[NikolaPrica]

include/llvm/CodeGen/SelectionDAG.h

1633

Too bad I got no warning on that. Thanks for pointing that!

Do you mean to only get the `CallSiteInfo` once?

Yes. For now it is used only in that way.

[djtodoro]

-Remove const from getSDCallSiteInfo()

[aprantl]

LGTM from my high-level point of view, but please make sure to address everybody else's feedback, too.

[djtodoro]

-Rebase

[bjope]

FYI: I have no further comments.

[aprantl]

test/CodeGen/X86/call-site-info-output.ll

14

Is there *anything* we can check in the output here? Otherwise this will bitrot quickly.

[djtodoro]

-Verify that call site info is the same after loading and running a pass over it.

[djtodoro]

-Rebase

[djtodoro]

-Follow the new order of fields inside the ArgRegPair struct (the struct is now better packed))

include/llvm/CodeGen/MIRYamlMapping.h

YamlIO.mapOptional("debug-info-expression", Object.DebugExpr,
                        StringValue()); // Don't print it out when it's empty.
     YamlIO.mapOptional("debug-info-location", Object.DebugLoc,
                        StringValue()); // Don't print it out when it's empty.
   }
 
   static const bool flow = true;
 };
 
+struct CallSiteInfo {
+  struct ArgRegPair {
+    uint32_t ArgNo;
+    StringValue Reg;
+
+    bool operator==(const ArgRegPair &Other) const {
+      return ArgNo == Other.ArgNo && Reg == Other.Reg;
+    }
+  };
+
+  struct MachineInstrLoc {
+    unsigned BlockNum;
+    unsigned Offset;
+
+    bool operator==(const MachineInstrLoc &Other) const {
+      return BlockNum == Other.BlockNum && Offset == Other.Offset;
+    }
+  };
+
+  MachineInstrLoc CallLocation;
+  std::vector<ArgRegPair> ArgForwardingRegs;
+
+  bool operator==(const CallSiteInfo &Other) const {
+    return CallLocation.BlockNum == Other.CallLocation.BlockNum &&
+           CallLocation.Offset == Other.CallLocation.Offset;
+  }
+};
+
+template <> struct MappingTraits<CallSiteInfo::ArgRegPair> {
+  static void mapping(IO &YamlIO, CallSiteInfo::ArgRegPair &ArgReg) {
+    YamlIO.mapRequired("arg", ArgReg.ArgNo);
+    YamlIO.mapRequired("reg", ArgReg.Reg);
+  }
+
+  static const bool flow = true;
+};
+}
+}
+
+LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::yaml::CallSiteInfo::ArgRegPair)
+
+namespace llvm {
+namespace yaml {
+
+template <> struct MappingTraits<CallSiteInfo> {
+  static void mapping(IO &YamlIO, CallSiteInfo &CSInfo) {
+    YamlIO.mapRequired("bb", CSInfo.CallLocation.BlockNum);
+    YamlIO.mapRequired("offset", CSInfo.CallLocation.Offset);
+    YamlIO.mapOptional("fwdArgRegs", CSInfo.ArgForwardingRegs,
+                       std::vector<CallSiteInfo::ArgRegPair>());
+  }
+
+  static const bool flow = true;
+};
+
 struct MachineConstantPoolValue {
   UnsignedValue ID;
   StringValue Value;
   unsigned Alignment = 0;
   bool IsTargetSpecific = false;
 
   bool operator==(const MachineConstantPoolValue &Other) const {
     return ID == Other.ID && Value == Other.Value &&

 
 } // end namespace yaml
 } // end namespace llvm
 
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::yaml::MachineFunctionLiveIn)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::yaml::VirtualRegisterDefinition)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::yaml::MachineStackObject)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::yaml::FixedMachineStackObject)
+LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::yaml::CallSiteInfo)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::yaml::MachineConstantPoolValue)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::yaml::MachineJumpTable::Entry)
 
 namespace llvm {
 namespace yaml {
 
 template <> struct MappingTraits<MachineJumpTable> {
   static void mapping(IO &YamlIO, MachineJumpTable &JT) {
struct MachineFunction {
   Optional<std::vector<FlowStringValue>> CalleeSavedRegisters;
   // TODO: Serialize the various register masks.
   // Frame information
   MachineFrameInfo FrameInfo;
   std::vector<FixedMachineStackObject> FixedStackObjects;
   std::vector<MachineStackObject> StackObjects;
   std::vector<MachineConstantPoolValue> Constants; /// Constant pool.
   std::unique_ptr<MachineFunctionInfo> MachineFuncInfo;
+  std::vector<CallSiteInfo> CallSitesInfo;
   MachineJumpTable JumpTableInfo;
   BlockStringValue Body;
 };
 
 template <> struct MappingTraits<MachineFunction> {
   static void mapping(IO &YamlIO, MachineFunction &MF) {
     YamlIO.mapRequired("name", MF.Name);
     YamlIO.mapOptional("alignment", MF.Alignment, (unsigned)0);
YamlIO.mapOptional("liveins", MF.LiveIns,
                        std::vector<MachineFunctionLiveIn>());
     YamlIO.mapOptional("calleeSavedRegisters", MF.CalleeSavedRegisters,
                        Optional<std::vector<FlowStringValue>>());
     YamlIO.mapOptional("frameInfo", MF.FrameInfo, MachineFrameInfo());
     YamlIO.mapOptional("fixedStack", MF.FixedStackObjects,
                        std::vector<FixedMachineStackObject>());
     YamlIO.mapOptional("stack", MF.StackObjects,
                        std::vector<MachineStackObject>());
+    YamlIO.mapOptional("callSites", MF.CallSitesInfo,
+                       std::vector<CallSiteInfo>());
     YamlIO.mapOptional("constants", MF.Constants,
                        std::vector<MachineConstantPoolValue>());
     YamlIO.mapOptional("machineFunctionInfo", MF.MachineFuncInfo);
     if (!YamlIO.outputting() || !MF.JumpTableInfo.Entries.empty())
       YamlIO.mapOptional("jumpTable", MF.JumpTableInfo, MachineJumpTable());
     YamlIO.mapOptional("body", MF.Body, BlockStringValue());
   }
 };
 
 } // end namespace yaml
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_MIRYAMLMAPPING_H

include/llvm/CodeGen/MachineFunction.h

 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/ilist.h"
 #include "llvm/ADT/iterator.h"
 #include "llvm/Analysis/EHPersonalities.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
+#include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/ArrayRecycler.h"
 #include "llvm/Support/AtomicOrdering.h"
 #include "llvm/Support/Compiler.h"
public:
   public:
     virtual ~Delegate() = default;
     /// Callback after an insertion. This should not modify the MI directly.
     virtual void MF_HandleInsertion(MachineInstr &MI) = 0;
     /// Callback before a removal. This should not modify the MI directly.
     virtual void MF_HandleRemoval(MachineInstr &MI) = 0;
   };
 
+  /// Vector of call argument and its forwarding register.
+  using CallSiteInfo = SmallVector<std::pair<uint32_t, unsigned>, 1>;
+  using CallSiteInfoImpl = SmallVectorImpl<std::pair<uint32_t, unsigned>>;
+
 private:
   Delegate *TheDelegate = nullptr;
 
+  using CallSiteInfoMap = DenseMap<const MachineInstr *, CallSiteInfo>;
+  /// Map a call instruction to call site arguments forwarding info.
+  CallSiteInfoMap CallSitesInfo;
+
   // Callbacks for insertion and removal.
   void handleInsertion(MachineInstr &MI);
   void handleRemoval(MachineInstr &MI);
   friend struct ilist_traits<MachineInstr>;
 
 public:
   using VariableDbgInfoMapTy = SmallVector<VariableDbgInfo, 4>;
   VariableDbgInfoMapTy VariableDbgInfos;
void setVariableDbgInfo(const DILocalVariable *Var, const DIExpression *Expr,
                           int Slot, const DILocation *Loc) {
     VariableDbgInfos.emplace_back(Var, Expr, Slot, Loc);
   }
 
   VariableDbgInfoMapTy &getVariableDbgInfo() { return VariableDbgInfos; }
   const VariableDbgInfoMapTy &getVariableDbgInfo() const {
     return VariableDbgInfos;
   }
+
+  void addCallArgsForwardingRegs(const MachineInstr *CallI,
+                                 CallSiteInfoImpl &&CallInfo) {
+    assert(CallI->isCall());
+    CallSitesInfo[CallI] = std::move(CallInfo);
+  }
+
+  const CallSiteInfoMap &getCallSitesInfo() const {
+    return CallSitesInfo;
+  }
+
+  /// Update call sites info by deleting entry for /p Old call instruction.
+  /// If /p New is present then transfer /p Old call info to it.
+  void updateCallSiteInfo(const MachineInstr *Old,
+                          const MachineInstr *New = nullptr) {
+    assert(Old->isCall() && (!New || New->isCall()));
+    if (New)
+      CallSitesInfo[New] = std::move(CallSitesInfo[Old]);
+    CallSitesInfo.erase(Old);
+  }
 };
 
 //===--------------------------------------------------------------------===//
 // GraphTraits specializations for function basic block graphs (CFGs)
 //===--------------------------------------------------------------------===//
 
 // Provide specializations of GraphTraits to be able to treat a
 // machine function as a graph of machine basic blocks... these are

include/llvm/CodeGen/SelectionDAG.h

class SelectionDAG {
   ArrayRecycler<SDUse> OperandRecycler;
 
   /// Pool allocation for misc. objects that are created once per SelectionDAG.
   BumpPtrAllocator Allocator;
 
   /// Tracks dbg_value and dbg_label information through SDISel.
   SDDbgInfo *DbgInfo;
 
+  using CallSiteInfo = MachineFunction::CallSiteInfo;
+  using CallSiteInfoImpl = MachineFunction::CallSiteInfoImpl;
+  DenseMap<const SDNode *, CallSiteInfo> SDCallSiteInfo;
+
   uint16_t NextPersistentId = 0;
 
 public:
   /// Clients of various APIs that cause global effects on
   /// the DAG can optionally implement this interface.  This allows the clients
   /// to handle the various sorts of updates that happen.
   ///
   /// A DAGUpdateListener automatically registers itself with DAG when it is
public:
 
   /// \returns true if \p N is any kind of constant or build_vector of
   /// constants, int or float. If a vector, it may not necessarily be a splat.
   inline bool isConstantValueOfAnyType(SDValue N) {
     return isConstantIntBuildVectorOrConstantInt(N) ||
            isConstantFPBuildVectorOrConstantFP(N);
   }
 
+  void addCallSiteInfo(const SDNode *CallNode, CallSiteInfoImpl &CallInfo) {
+    SDCallSiteInfo[CallNode] = std::move(CallInfo);
+  }
+
+  CallSiteInfo getSDCallSiteInfo(const SDNode *CallNode) const {
+    auto I = SDCallSiteInfo.find(CallNode);
+    if (I != SDCallSiteInfo.end())
+      return std::move(I->second);
+    return CallSiteInfo();
+  }
+
 private:
   void InsertNode(SDNode *N);
   bool RemoveNodeFromCSEMaps(SDNode *N);
   void AddModifiedNodeToCSEMaps(SDNode *N);
   SDNode *FindModifiedNodeSlot(SDNode *N, SDValue Op, void *&InsertPos);
   SDNode *FindModifiedNodeSlot(SDNode *N, SDValue Op1, SDValue Op2,
                                void *&InsertPos);
   SDNode *FindModifiedNodeSlot(SDNode *N, ArrayRef<SDValue> Ops,

lib/CodeGen/InlineSpiller.cpp

for (MIBundleOperands MO(*MI); MO.isValid(); ++MO) {
     LIS.removePhysRegDefAt(Reg, Idx);
   }
 
   int FI;
   if (TII.isStoreToStackSlot(*MI, FI) &&
       HSpiller.rmFromMergeableSpills(*MI, FI))
     --NumSpills;
   LIS.ReplaceMachineInstrInMaps(*MI, *FoldMI);
+  if (MI->isCall())
+    MI->getMF()->updateCallSiteInfo(MI, FoldMI);
   MI->eraseFromParent();
 
   // Insert any new instructions other than FoldMI into the LIS maps.
   assert(!MIS.empty() && "Unexpected empty span of instructions!");
   for (MachineInstr &MI : MIS)
     if (&MI != FoldMI)
       LIS.InsertMachineInstrInMaps(MI);
 

lib/CodeGen/MIRParser/MIRParser.cpp

reportDiagnostic(
         diagFromBlockStringDiag(Error, YamlMF.Body.Value.SourceRange));
     return true;
   }
   PFS.SM = &SM;
 
   if (setupRegisterInfo(PFS, YamlMF))
     return true;
 
-  if (YamlMF.MachineFuncInfo) {
-    const LLVMTargetMachine &TM = MF.getTarget();
+  if (YamlMF.MachineFuncInfo) {
     // Note this is called after the initial constructor of the
     // MachineFunctionInfo based on the MachineFunction, which may depend on the
     // IR.
 
     SMRange SrcRange;
     if (TM.parseMachineFunctionInfo(*YamlMF.MachineFuncInfo, PFS, Error,
                                     SrcRange)) {
       return error(Error, SrcRange);
     }
   }
 
   // Set the reserved registers after parsing MachineFuncInfo. The target may
   // have been recording information used to select the reserved registers
   // there.
   // FIXME: This is a temporary workaround until the reserved registers can be
   // serialized.
   MachineRegisterInfo &MRI = MF.getRegInfo();
   MRI.freezeReservedRegs(MF);
 
   computeFunctionProperties(MF);
 
+  for (auto YamlCSInfo : YamlMF.CallSitesInfo) {
+    yaml::CallSiteInfo::MachineInstrLoc MILoc = YamlCSInfo.CallLocation;
+    auto CallB = std::next(MF.begin(), MILoc.BlockNum);
+    auto CallI = std::next(CallB->begin(), MILoc.Offset - 1);
+    assert(CallI->isCall() && "Call site info is related to call instruction");
+    MachineFunction::CallSiteInfo CSInfo;
+    for (auto ArgRegPair : YamlCSInfo.ArgForwardingRegs) {
+      unsigned Reg = 0;
+      if (parseNamedRegisterReference(PFS, Reg, ArgRegPair.Reg.Value,
+                                      Error))
+        return error(Error, ArgRegPair.Reg.SourceRange);
+      CSInfo.emplace_back(ArgRegPair.ArgNo, Reg);
+    }
+
+    if (TM.Options.EnableParamEntryValues)
+      MF.addCallArgsForwardingRegs(&*CallI, std::move(CSInfo));
+  }
+
   MF.getSubtarget().mirFileLoaded(MF);
 
   MF.verify();
   return false;
 }
 
 bool MIRParserImpl::parseRegisterInfo(PerFunctionMIParsingState &PFS,
                                       const yaml::MachineFunction &YamlMF) {

lib/CodeGen/MIRPrinter.cpp

public:
   void convert(ModuleSlotTracker &MST, yaml::MachineFrameInfo &YamlMFI,
                const MachineFrameInfo &MFI);
   void convert(yaml::MachineFunction &MF,
                const MachineConstantPool &ConstantPool);
   void convert(ModuleSlotTracker &MST, yaml::MachineJumpTable &YamlJTI,
                const MachineJumpTableInfo &JTI);
   void convertStackObjects(yaml::MachineFunction &YMF,
                            const MachineFunction &MF, ModuleSlotTracker &MST);
+  void convertCallSiteObjects(yaml::MachineFunction &YMF,
+                              const MachineFunction &MF,
+                              ModuleSlotTracker &MST);
 
 private:
   void initRegisterMaskIds(const MachineFunction &MF);
 };
 
 /// This class prints out the machine instructions using the MIR serialization
 /// format.
 class MIPrinter {
void MIRPrinter::print(const MachineFunction &MF) {
   YamlMF.FailedISel = MF.getProperties().hasProperty(
       MachineFunctionProperties::Property::FailedISel);
 
   convert(YamlMF, MF.getRegInfo(), MF.getSubtarget().getRegisterInfo());
   ModuleSlotTracker MST(MF.getFunction().getParent());
   MST.incorporateFunction(MF.getFunction());
   convert(MST, YamlMF.FrameInfo, MF.getFrameInfo());
   convertStackObjects(YamlMF, MF, MST);
+  convertCallSiteObjects(YamlMF, MF, MST);
   if (const auto *ConstantPool = MF.getConstantPool())
     convert(YamlMF, *ConstantPool);
   if (const auto *JumpTableInfo = MF.getJumpTableInfo())
     convert(MST, YamlMF.JumpTableInfo, *JumpTableInfo);
 
   const TargetMachine &TM = MF.getTarget();
   YamlMF.MachineFuncInfo =
       std::unique_ptr<yaml::MachineFunctionInfo>(TM.convertFuncInfoToYAML(MF));
if (StackObject.IsFixed) {
       printStackObjectDbgInfo(DebugVar, Object, MST);
     } else {
       auto &Object = YMF.StackObjects[StackObject.ID];
       printStackObjectDbgInfo(DebugVar, Object, MST);
     }
   }
 }
 
+void MIRPrinter::convertCallSiteObjects(yaml::MachineFunction &YMF,
+                                        const MachineFunction &MF,
+                                        ModuleSlotTracker &MST) {
+  const auto *TRI = MF.getSubtarget().getRegisterInfo();
+  for (auto CSInfo : MF.getCallSitesInfo()) {
+    yaml::CallSiteInfo YmlCS;
+    yaml::CallSiteInfo::MachineInstrLoc CallLocation;
+
+    // Prepare instruction position.
+    auto CallI = CSInfo.first->getReverseIterator();
+    unsigned Offset = 0;
+    CallLocation.BlockNum = CallI->getParent()->getNumber();
+    // Get call instruction offset from the beginning of block.
+    for (auto E = CallI->getParent()->rend(); CallI != E; CallI++, Offset++);
+    CallLocation.Offset = Offset;
+    YmlCS.CallLocation = CallLocation;
+    // Construct call arguments and theirs forwarding register info.
+    for (auto ArgReg : CSInfo.second) {
+      yaml::CallSiteInfo::ArgRegPair YmlArgReg;
+      YmlArgReg.ArgNo = ArgReg.first;
+      printRegMIR(ArgReg.second, YmlArgReg.Reg, TRI);
+      YmlCS.ArgForwardingRegs.emplace_back(YmlArgReg);
+    }
+    YMF.CallSitesInfo.push_back(YmlCS);
+  }
+
+  // Sort call info by position of call instructions.
+  std::sort(YMF.CallSitesInfo.begin(), YMF.CallSitesInfo.end(),
+            [](yaml::CallSiteInfo A, yaml::CallSiteInfo B) {
+              if (A.CallLocation.BlockNum == B.CallLocation.BlockNum)
+                return A.CallLocation.Offset < B.CallLocation.Offset;
+              return A.CallLocation.BlockNum < B.CallLocation.BlockNum;
+            });
+}
+
 void MIRPrinter::convert(yaml::MachineFunction &MF,
                          const MachineConstantPool &ConstantPool) {
   unsigned ID = 0;
   for (const MachineConstantPoolEntry &Constant : ConstantPool.getConstants()) {
     std::string Str;
     raw_string_ostream StrOS(Str);
     if (Constant.isMachineConstantPoolEntry()) {
       Constant.Val.MachineCPVal->print(StrOS);

lib/CodeGen/MachineFunction.cpp

 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/PseudoSourceValue.h"
 #include "llvm/CodeGen/TargetFrameLowering.h"
 #include "llvm/CodeGen/TargetLowering.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
-#include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/CodeGen/WasmEHFuncInfo.h"
 #include "llvm/CodeGen/WinEHFuncInfo.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DerivedTypes.h"

 }
 
 /// Delete the given MachineInstr.
 ///
 /// This function also serves as the MachineInstr destructor - the real
 /// ~MachineInstr() destructor must be empty.
 void
 MachineFunction::DeleteMachineInstr(MachineInstr *MI) {
+  if (MI->isCall(MachineInstr::IgnoreBundle))
+    assert(CallSitesInfo.find(MI) == CallSitesInfo.end() &&
+           "Call site info is not updated!");
   // Strip it for parts. The operand array and the MI object itself are
   // independently recyclable.
   if (MI->Operands)
     deallocateOperandArray(MI->CapOperands, MI->Operands);
   // Don't call ~MachineInstr() which must be trivial anyway because
   // ~MachineFunction drops whole lists of MachineInstrs wihout calling their
   // destructors.
   InstructionRecycler.Deallocate(Allocator, MI);

lib/CodeGen/MachineOutliner.cpp

for (Candidate &C : OF.Candidates) {
 
       // If the caller tracks liveness, then we need to make sure that
       // anything we outline doesn't break liveness assumptions. The outlined
       // functions themselves currently don't track liveness, but we should
       // make sure that the ranges we yank things out of aren't wrong.
       if (MBB.getParent()->getProperties().hasProperty(
               MachineFunctionProperties::Property::TracksLiveness)) {
         // Helper lambda for adding implicit def operands to the call
-        // instruction.
-        auto CopyDefs = [&CallInst](MachineInstr &MI) {
+        // instruction. It also updates call site information for moved
+        // code.
+        auto CopyDefsAndUpdateCalls = [&CallInst](MachineInstr &MI) {
           for (MachineOperand &MOP : MI.operands()) {
             // Skip over anything that isn't a register.
             if (!MOP.isReg())
               continue;
 
             // If it's a def, add it to the call instruction.
             if (MOP.isDef())
               CallInst->addOperand(MachineOperand::CreateReg(
                   MOP.getReg(), true, /* isDef = true */
                   true /* isImp = true */));
           }
+          if (MI.isCall())
+            MI.getMF()->updateCallSiteInfo(&MI);
         };
         // Copy over the defs in the outlined range.
         // First inst in outlined range <-- Anything that's defined in this
         // ...                           .. range has to be added as an
         // implicit Last inst in outlined range  <-- def to the call
-        // instruction.
-        std::for_each(CallInst, std::next(EndIt), CopyDefs);
+        // instruction. Also remove call site information for outlined block
+        // of code.
+        std::for_each(CallInst, std::next(EndIt), CopyDefsAndUpdateCalls);
       }
 
       // Erase from the point after where the call was inserted up to, and
       // including, the final instruction in the sequence.
       // Erase needs one past the end, so we need std::next there too.
       MBB.erase(std::next(StartIt), std::next(EndIt));
 
       // Keep track of what we removed by marking them all as -1.

lib/CodeGen/MachineVerifier.cpp

for (RegSet::iterator
       report_context_vreg(*I);
     }
   }
 
   if (LiveVars)
     verifyLiveVariables();
   if (LiveInts)
     verifyLiveIntervals();
+
+  for (auto CSInfo : MF->getCallSitesInfo())
+    if (!CSInfo.first->isCall())
+      report("Call site info refrenceining instruction that is not call", MF);
 }
 
 void MachineVerifier::verifyLiveVariables() {
   assert(LiveVars && "Don't call verifyLiveVariables without LiveVars");
   for (unsigned i = 0, e = MRI->getNumVirtRegs(); i != e; ++i) {
     unsigned Reg = TargetRegisterInfo::index2VirtReg(i);
     LiveVariables::VarInfo &VI = LiveVars->getVarInfo(Reg);
     for (const auto &MBB : *MF) {

lib/CodeGen/PeepholeOptimizer.cpp

for (MachineBasicBlock::iterator MII = MBB.begin(), MIE = MBB.end();
                     TII->optimizeLoadInstr(*MI, MRI, FoldAsLoadDefReg, DefMI)) {
               // Update LocalMIs since we replaced MI with FoldMI and deleted
               // DefMI.
               LLVM_DEBUG(dbgs() << "Replacing: " << *MI);
               LLVM_DEBUG(dbgs() << "     With: " << *FoldMI);
               LocalMIs.erase(MI);
               LocalMIs.erase(DefMI);
               LocalMIs.insert(FoldMI);
+              if (MI->isCall())
+                MI->getMF()->updateCallSiteInfo(MI, FoldMI);
               MI->eraseFromParent();
               DefMI->eraseFromParent();
               MRI->markUsesInDebugValueAsUndef(FoldedReg);
               FoldAsLoadDefCandidates.erase(FoldedReg);
               ++NumLoadFold;
 
               // MI is replaced with FoldMI so we can continue trying to fold
               Changed = true;

lib/CodeGen/SelectionDAG/ScheduleDAGSDNodes.cpp

auto EmitNode =
     MachineBasicBlock::iterator Before = GetPrevInsn(Emitter.getInsertPos());
     Emitter.EmitNode(Node, IsClone, IsCloned, VRBaseMap);
     MachineBasicBlock::iterator After = GetPrevInsn(Emitter.getInsertPos());
 
     // If the iterator did not change, no instructions were inserted.
     if (Before == After)
       return nullptr;
 
+    MachineInstr *MI;
     if (Before == BB->end()) {
       // There were no prior instructions; the new ones must start at the
       // beginning of the block.
-      return &Emitter.getBlock()->instr_front();
+      MI = &Emitter.getBlock()->instr_front();
     } else {
       // Return first instruction after the pre-existing instructions.
-      return &*std::next(Before);
+      MI = &*std::next(Before);
     }
+
+    if (MI->isCall() && DAG->getTarget().Options.EnableParamEntryValues)
+      MF.addCallArgsForwardingRegs(MI, DAG->getSDCallSiteInfo(Node));
+
+    return MI;
   };
 
   // If this is the first BB, emit byval parameter dbg_value's.
   if (HasDbg && BB->getParent()->begin() == MachineFunction::iterator(BB)) {
     SDDbgInfo::DbgIterator PDI = DAG->ByvalParmDbgBegin();
     SDDbgInfo::DbgIterator PDE = DAG->ByvalParmDbgEnd();
     for (; PDI != PDE; ++PDI) {
       MachineInstr *DbgMI= Emitter.EmitDbgValue(*PDI, VRBaseMap);

lib/CodeGen/TargetInstrInfo.cpp

TargetInstrInfo::ReplaceTailWithBranchTo(MachineBasicBlock::iterator Tail,
 
   // Remove all the old successors of MBB from the CFG.
   while (!MBB->succ_empty())
     MBB->removeSuccessor(MBB->succ_begin());
 
   // Save off the debug loc before erasing the instruction.
   DebugLoc DL = Tail->getDebugLoc();
 
-  // Remove all the dead instructions from the end of MBB.
-  MBB->erase(Tail, MBB->end());
+  // Remove all the dead instructions from the end of MBB and update call site
+  // info.
+  while(Tail != MBB->end()) {
+    MachineInstr *MI = &*Tail;
+    if (MI->isCall())
+      MBB->getParent()->updateCallSiteInfo(MI);
+    ++Tail;
+    MBB->erase(MI);
+  }
 
   // If MBB isn't immediately before MBB, insert a branch to it.
   if (++MachineFunction::iterator(MBB) != MachineFunction::iterator(NewDest))
     insertBranch(*MBB, NewDest, nullptr, SmallVector<MachineOperand, 0>(), DL);
   MBB->addSuccessor(NewDest);
 }
 
 MachineInstr *TargetInstrInfo::commuteInstructionImpl(MachineInstr &MI,

lib/CodeGen/TargetSubtargetInfo.cpp

 //===- TargetSubtargetInfo.cpp - General Target Information ----------------==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 /// \file This file describes the general parts of a Subtarget.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 
 using namespace llvm;
 
+static cl::opt<bool> EmitCallSiteInfo("emit-call-site-info", cl::Hidden,
+                                      cl::init(false));
+
 TargetSubtargetInfo::TargetSubtargetInfo(
     const Triple &TT, StringRef CPU, StringRef FS,
     ArrayRef<SubtargetFeatureKV> PF, ArrayRef<SubtargetSubTypeKV> PD,
     const MCWriteProcResEntry *WPR,
     const MCWriteLatencyEntry *WL, const MCReadAdvanceEntry *RA,
     const InstrStage *IS, const unsigned *OC, const unsigned *FP)
     : MCSubtargetInfo(TT, CPU, FS, PF, PD, WPR, WL, RA, IS, OC, FP) {
 }

lib/CodeGen/XRayInstrumentation.cpp

for (auto &T : MBB.terminators()) {
         Opc = TargetOpcode::PATCHABLE_TAIL_CALL;
       }
       if (Opc != 0) {
         auto MIB = BuildMI(MBB, T, T.getDebugLoc(), TII->get(Opc))
                        .addImm(T.getOpcode());
         for (auto &MO : T.operands())
           MIB.add(MO);
         Terminators.push_back(&T);
+        if (T.isCall())
+          MF.updateCallSiteInfo(&T);
       }
     }
   }
 
   for (auto &I : Terminators)
     I->eraseFromParent();
 }
 

lib/Target/X86/X86ExpandPseudo.cpp

case X86::TCRETURNmi64: {
     } else {
       JumpTarget.setIsKill();
       BuildMI(MBB, MBBI, DL, TII->get(X86::TAILJMPr))
           .add(JumpTarget);
     }
 
     MachineInstr &NewMI = *std::prev(MBBI);
     NewMI.copyImplicitOps(*MBBI->getParent()->getParent(), *MBBI);
+    MBB.getParent()->updateCallSiteInfo(&*MBBI, &NewMI);
 
     // Delete the pseudo instruction TCRETURN.
     MBB.erase(MBBI);
 
     return true;
   }
   case X86::EH_RETURN:
   case X86::EH_RETURN64: {

lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   bool HasNCSR = (CI && CI->hasFnAttr("no_caller_saved_registers")) ||
                  (Fn && Fn->hasFnAttribute("no_caller_saved_registers"));
   const auto *II = dyn_cast_or_null<InvokeInst>(CLI.CS.getInstruction());
   bool HasNoCfCheck =
       (CI && CI->doesNoCfCheck()) || (II && II->doesNoCfCheck());
   const Module *M = MF.getMMI().getModule();
   Metadata *IsCFProtectionSupported = M->getModuleFlag("cf-protection-branch");
 
+  MachineFunction::CallSiteInfo CSInfo;
+
   if (CallConv == CallingConv::X86_INTR)
     report_fatal_error("X86 interrupts may not be called directly");
 
   if (Attr.getValueAsString() == "true")
     isTailCall = false;
 
   if (Subtarget.isPICStyleGOT() &&
       !MF.getTarget().Options.GuaranteedTailCallOpt) {
for (unsigned I = 0, OutIndex = 0, E = ArgLocs.size(); I != E;
     if (VA.needsCustom()) {
       assert(VA.getValVT() == MVT::v64i1 &&
              "Currently the only custom case is when we split v64i1 to 2 regs");
       // Split v64i1 value into two registers
       Passv64i1ArgInRegs(dl, DAG, Chain, Arg, RegsToPass, VA, ArgLocs[++I],
                          Subtarget);
     } else if (VA.isRegLoc()) {
       RegsToPass.push_back(std::make_pair(VA.getLocReg(), Arg));
+      const TargetOptions &Options = DAG.getTarget().Options;
+      if (Options.EnableParamEntryValues)
+        CSInfo.emplace_back(I, VA.getLocReg());
       if (isVarArg && IsWin64) {
         // Win64 ABI requires argument XMM reg to be copied to the corresponding
         // shadow reg if callee is a varargs function.
         unsigned ShadowReg = 0;
         switch (VA.getLocReg()) {
         case X86::XMM0: ShadowReg = X86::RCX; break;
         case X86::XMM1: ShadowReg = X86::RDX; break;
         case X86::XMM2: ShadowReg = X86::R8; break;
X86TargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   if (isTailCall) {
     // We used to do:
     //// If this is the first return lowered for this function, add the regs
     //// to the liveout set for the function.
     // This isn't right, although it's probably harmless on x86; liveouts
     // should be computed from returns not tail calls.  Consider a void
     // function making a tail call to a function returning int.
     MF.getFrameInfo().setHasTailCall();
-    return DAG.getNode(X86ISD::TC_RETURN, dl, NodeTys, Ops);
+    SDValue Ret = DAG.getNode(X86ISD::TC_RETURN, dl, NodeTys, Ops);
+    DAG.addCallSiteInfo(Ret.getNode(), CSInfo);
+    return Ret;
   }
 
   if (HasNoCfCheck && IsCFProtectionSupported) {
     Chain = DAG.getNode(X86ISD::NT_CALL, dl, NodeTys, Ops);
   } else {
     Chain = DAG.getNode(X86ISD::CALL, dl, NodeTys, Ops);
   }
   InFlag = Chain.getValue(1);
+  DAG.addCallSiteInfo(Chain.getNode(), CSInfo);
 
   // Create the CALLSEQ_END node.
   unsigned NumBytesForCalleeToPop;
   if (X86::isCalleePop(CallConv, Is64Bit, isVarArg,
                        DAG.getTarget().Options.GuaranteedTailCallOpt))
     NumBytesForCalleeToPop = NumBytes;    // Callee pops everything
   else if (!Is64Bit && !canGuaranteeTCO(CallConv) &&
            !Subtarget.getTargetTriple().isOSMSVCRT() &&

test/CodeGen/X86/call-site-info-output.ll

+; Test call site info MIR printer and parser.Parser assertions and machine
+; verifier will check the rest;
+; RUN: llc -param-entry-values %s -stop-after=expand-isel-pseudos -o %t.mir
+; RUN: cat %t.mir | FileCheck %s
+; CHECK: name: fn2
+; CHECK: callSites:
+; CHECK-NEXT: bb: 0, offset: 14, fwdArgRegs:
+; CHECK-NEXT:   arg: 0, reg: '$edi'
+; CHECK-NEXT:   arg: 1, reg: '$esi'
+; CHECK-NEXT:   arg: 2, reg: '$edx'
+; RUN: llc %t.mir -start-after=expand-isel-pseudos
+
+; ModuleID = 'test/CodeGen/X86/call-site-info-output.c'
+source_filename = "test/CodeGen/X86/call-site-info-output.c"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; Function Attrs: noinline nounwind uwtable
+define dso_local i64 @fn2(i32 %a, i32 %b, i32 %c) local_unnamed_addr #0 {
+entry:
+  %call = tail call i32 (i32, i32, i32, ...) bitcast (i32 (...)* @fn1 to i32 (i32, i32, i32, ...)*)(i32 -50, i32 50, i32 -7) #2
+  %add = mul i32 %a, 3
+  %sub = sub i32 %add, %b
+  %add2 = add i32 %sub, %c
+  %conv4 = sext i32 %add2 to i64
+  ret i64 %conv4
+}
+
+declare dso_local i32 @fn1(...) local_unnamed_addr #1
+
+!llvm.module.flags = !{!0}
+!llvm.ident = !{!1}
+
+!0 = !{i32 1, !"wchar_size", i32 4}
+!1 = !{!"clang version 9.0.0"}