[PGO] Value profile for size of memory intrinsic calls

[xur]

Summary

This patch adds the value profile support to profile the size parameter of
memory intrinsic calls: memcpy, memcmp, and memmove.

We have two adjustable parameters: SmallVal and LargeVal size less than or equal to SmallVal will be precisely profiled. size greater than SmallVal but less than LargeVal will have one count size greater than or equal to LargeVale will have one count.

The patch has dependency on the profile value range support in compiler_rt:
https://reviews.llvm.org/D28964.

[davidxl]

See my comment in https://reviews.llvm.org/D28964. There is no need to introduce runtime change nor new clang intrinsics. The instrumentation lower can do the job of mapping the ranges. Another choice is to let the instrumenter does the range mapping work and keep the lowerer unchanged.

[xur]

Update the patch based on David's review comments.

(1) remove instrprof_value_rang_profile intrinsic and use existing instrprof_value_profile and used valuekind to distinguish different value profile.
(2) enclose most of the implementation details in value profile lowering.

[davidxl]

llvm-profdata needs test cases for profile reading and writing

1. show command
2. merge command in binary and in text format

lib/Transforms/Instrumentation/InstrProfiling.cpp

79

Perhaps make it a string option with both start and end values specified:

MemOpSizeValueRange

85

The description is not accurate.

tools/llvm-profdata/llvm-profdata.cpp

485

Add a test case to show text format dump works expected.

545

Collect summary statistics and dump it at the end -- similar to IC statistics.

653

This needs to be documented as other user facing options.

[xur]

lib/Transforms/Instrumentation/InstrProfiling.cpp

85

Fixed

[xur]

Integrated David's review comments.

Also fixed a bug in InstrProfReader when reading text format value profile.

[davidxl]

How to make the llvm-profdata 'show' command dump the range information as well?

include/llvm/Transforms/InstrProfiling.h

63

should it be 1?

66

Why 8? why not 64?

lib/ProfileData/InstrProfReader.cpp

185

Can you submit this separately as a bug fix?

lib/Transforms/Instrumentation/InstrProfiling.cpp

181

extract this into an option parsing helper.

546

Larger range does not mean on average the number of values tracked will be proportionally larger. Perhaps using a different heuristic for allocation size? Besides, TotalNS tracks the number of value sites, overloading it for number of values does not make sense

553

Here is the counter # estimation.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1301

unrelated change.

test/tools/llvm-profdata/memop-size-prof.proftext

3

also test 'merge' with -text option.

18

perhaps add a test with 2 value kinds

tools/llvm-profdata/llvm-profdata.cpp

468

TotalNumValueSitesICall

Or perhaps defined an array

TotalNumValueSites[..] and use value kind index to access.

469

TotalNumValuesSitesWithValueProfileICall

472

Make the var name consistent -- TotalNumValueSitesMemOpSize

509

NumMemOpCalls

511

Number of memory intrinsic calls :

546

extract this into to a common helper function? There are many duplications.

[xur]

include/llvm/Transforms/InstrProfiling.h

63

there are calls to memory intrinsics with size 0. There are quite many actually.

66

I think 8 is enough. In the optimization, I multi-version up to value of 8.
This is a tunable. 64 might be too big.

lib/Transforms/Instrumentation/InstrProfiling.cpp

546

I removed this change. Now MemOPSize profile uses the same heuristic as indirect-call profiling.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1301

reverted.

tools/llvm-profdata/llvm-profdata.cpp

468

Used a vector here.

[xur]

Updated the patch according on David's review comments.

[davidxl]

The profile annotation part of another patch should be merged in here.

include/llvm/Transforms/InstrProfiling.h

63

The value transformation patch does not seem to handle it . Do you see cases where value 0 is a frequent value in hot stringop sites?

lib/Transforms/Instrumentation/InstrProfiling.cpp

83

document the case when some value is missing in the option.

[davidxl]

lib/Transforms/Instrumentation/InstrProfiling.cpp

546

With this default heuristics, do you see any dropped value profiling at runtime?

tools/llvm-profdata/llvm-profdata.cpp

449

Can you split out this part as a refactoring patch?

[xur]

This is the patch that after splitting out the llvm-profdata refactor change.

[xur]

Update the patch: remove some unrelated changes.

[davidxl]

After the refactoring code lands, do you plan to add the site annotation part in this patch?

[xur]

In D28965#701857, @davidxl wrote:

After the refactoring code lands, do you plan to add the site annotation part in this patch?

I plan to do that part if a separated patch.

[davidxl]

lgtm

docs/CommandGuide/llvm-profdata.rst

  instrumentation-based profile data representation. By default, the profile
  information is dumped in a more human readable form (also in text) with
  annotations.
 
 .. option:: -sample
 
  Specify that the input profile is a sample-based profile.
 
+.. option:: -memop-sizes
+
+ Show the profiled sizes of the memory intrinsic calls for shown functions.
+
 EXIT STATUS
 -----------
 
 :program:`llvm-profdata` returns 1 if the command is omitted or is invalid,
 if it cannot read input files, or if there is a mismatch between their data.

include/llvm/ProfileData/InstrProf.h

 }
 
 /// Return the name profile runtime entry point to do value profiling
 /// for a given site.
 inline StringRef getInstrProfValueProfFuncName() {
   return INSTR_PROF_VALUE_PROF_FUNC_STR;
 }
 
+/// Return the name profile runtime entry point to do value range profiling.
+inline StringRef getInstrProfValueRangeProfFuncName() {
+  return INSTR_PROF_VALUE_RANGE_PROF_FUNC_STR;
+}
+
 /// Return the name of the section containing function coverage mapping
 /// data.
 inline StringRef getInstrProfCoverageSectionName(bool AddSegment) {
   return AddSegment ? "__LLVM_COV," INSTR_PROF_COVMAP_SECT_NAME_STR
                     : INSTR_PROF_COVMAP_SECT_NAME_STR;
 }
 
 /// Return the name prefix of variables containing instrumented function names.
for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind)
       getValueSitesForKind(Kind).clear();
   }
 
   /// Get the error contained within the record's soft error counter.
   Error takeError() { return SIPE.takeError(); }
 
 private:
   std::vector<InstrProfValueSiteRecord> IndirectCallSites;
+  std::vector<InstrProfValueSiteRecord> MemOPSizes;
   const std::vector<InstrProfValueSiteRecord> &
   getValueSitesForKind(uint32_t ValueKind) const {
     switch (ValueKind) {
     case IPVK_IndirectCallTarget:
       return IndirectCallSites;
+    case IPVK_MemOPSize:
+      return MemOPSizes;
     default:
       llvm_unreachable("Unknown value kind!");
     }
     return IndirectCallSites;
   }
 
   std::vector<InstrProfValueSiteRecord> &
   getValueSitesForKind(uint32_t ValueKind) {

include/llvm/ProfileData/InstrProfData.inc

 #define INSTR_PROF_COMMA
 #else
 #define INSTR_PROF_DATA_DEFINED
 #define INSTR_PROF_COMMA ,
 #endif
 VALUE_PROF_FUNC_PARAM(uint64_t, TargetValue, Type::getInt64Ty(Ctx)) \
                       INSTR_PROF_COMMA
 VALUE_PROF_FUNC_PARAM(void *, Data, Type::getInt8PtrTy(Ctx)) INSTR_PROF_COMMA
+#ifndef VALUE_RANGE_PROF
 VALUE_PROF_FUNC_PARAM(uint32_t, CounterIndex, Type::getInt32Ty(Ctx))
+#else /* VALUE_RANGE_PROF */
+VALUE_PROF_FUNC_PARAM(uint32_t, CounterIndex, Type::getInt32Ty(Ctx)) \
+                      INSTR_PROF_COMMA
+VALUE_PROF_FUNC_PARAM(uint64_t, PreciseRangeStart, Type::getInt64Ty(Ctx)) \
+                      INSTR_PROF_COMMA
+VALUE_PROF_FUNC_PARAM(uint64_t, PreciseRangeLast, Type::getInt64Ty(Ctx)) \
+                      INSTR_PROF_COMMA
+VALUE_PROF_FUNC_PARAM(uint64_t, LargeValue, Type::getInt64Ty(Ctx))
+#endif /*VALUE_RANGE_PROF */
 #undef VALUE_PROF_FUNC_PARAM
 #undef INSTR_PROF_COMMA
 /* VALUE_PROF_FUNC_PARAM end */
 
 /* VALUE_PROF_KIND start */
 #ifndef VALUE_PROF_KIND
 #define VALUE_PROF_KIND(Enumerator, Value)
 #else
 #define INSTR_PROF_DATA_DEFINED
 #endif
 /* For indirect function call value profiling, the addresses of the target
  * functions are profiled by the instrumented code. The target addresses are
  * written in the raw profile data and converted to target function name's MD5
  * hash by the profile reader during deserialization.  Typically, this happens
  * when the the raw profile data is read during profile merging.
  *
  * For this remapping the ProfData is used.  ProfData contains both the function
  * name hash and the function address.
  */
 VALUE_PROF_KIND(IPVK_IndirectCallTarget, 0)
+/* For memory intrinsic functions size profiling. */
+VALUE_PROF_KIND(IPVK_MemOPSize, 1)
 /* These two kinds must be the last to be
  * declared. This is to make sure the string
  * array created with the template can be
  * indexed with the kind value.
  */
 VALUE_PROF_KIND(IPVK_First, IPVK_IndirectCallTarget)
-VALUE_PROF_KIND(IPVK_Last, IPVK_IndirectCallTarget)
+VALUE_PROF_KIND(IPVK_Last, IPVK_MemOPSize)
 
 #undef VALUE_PROF_KIND
 /* VALUE_PROF_KIND end */
 
 /* COVMAP_FUNC_RECORD start */
 /* Definition of member fields of the function record structure in coverage
  * map.
  */
#define INSTR_PROF_SECT_START(Sect) \
         INSTR_PROF_CONCAT(__start_,Sect)
 #define INSTR_PROF_SECT_STOP(Sect) \
         INSTR_PROF_CONCAT(__stop_,Sect)
 
 /* Value Profiling API linkage name.  */
 #define INSTR_PROF_VALUE_PROF_FUNC __llvm_profile_instrument_target
 #define INSTR_PROF_VALUE_PROF_FUNC_STR \
         INSTR_PROF_QUOTE(INSTR_PROF_VALUE_PROF_FUNC)
+#define INSTR_PROF_VALUE_RANGE_PROF_FUNC __llvm_profile_instrument_range
+#define INSTR_PROF_VALUE_RANGE_PROF_FUNC_STR \
+        INSTR_PROF_QUOTE(INSTR_PROF_VALUE_RANGE_PROF_FUNC)
 
 /* InstrProfile per-function control data alignment.  */
 #define INSTR_PROF_DATA_ALIGNMENT 8
 
 /* The data structure that represents a tracked value by the
  * value profiler.
  */
 typedef struct InstrProfValueData {

include/llvm/Transforms/InstrProfiling.h

struct PerFunctionProfileData {
     }
   };
   DenseMap<GlobalVariable *, PerFunctionProfileData> ProfileDataMap;
   std::vector<GlobalValue *> UsedVars;
   std::vector<GlobalVariable *> ReferencedNames;
   GlobalVariable *NamesVar;
   size_t NamesSize;
 
+  // The start value of precise value profile range for memory intrinsic sizes.
+  const int64_t DefaultMemOPSizeRangeStart = 0;
+  int64_t MemOPSizeRangeStart;
+  // The end value of precise value profile range for memory intrinsic sizes.
+  const int64_t DefaultMemOPSizeRangeLast = 8;
+  int64_t MemOPSizeRangeLast;
+  int64_t MemOPSizeLargeVal;
+
   bool isMachO() const;
 
   /// Get the section name for the counter variables.
   StringRef getCountersSection() const;
 
   /// Get the section name for the name variables.
   StringRef getNameSection() const;
 

lib/ProfileData/InstrProfReader.cpp

for (uint32_t VK = 0; VK < NumValueKinds; VK++) {
     for (uint32_t S = 0; S < NumValueSites; S++) {
       VP_READ_ADVANCE(NumValueData);
 
       std::vector<InstrProfValueData> CurrentValues;
       for (uint32_t V = 0; V < NumValueData; V++) {
         CHECK_LINE_END(Line);
         std::pair<StringRef, StringRef> VD = Line->rsplit(':');
         uint64_t TakenCount, Value;
-        if (VK == IPVK_IndirectCallTarget) {
+        if (ValueKind == IPVK_IndirectCallTarget) {
           Symtab->addFuncName(VD.first);
           Value = IndexedInstrProf::ComputeHash(VD.first);
         } else {
           READ_NUM(VD.first, Value);
         }
         READ_NUM(VD.second, TakenCount);
         CurrentValues.push_back({Value, TakenCount});
         Line++;
       }
-      Record.addValueData(VK, S, CurrentValues.data(), NumValueData, nullptr);
+      Record.addValueData(ValueKind, S, CurrentValues.data(), NumValueData, nullptr);
     }
   }
   return success();
 
 #undef CHECK_LINE_END
 #undef READ_NUM
 #undef VP_READ_ADVANCE
 }

lib/Transforms/Instrumentation/InstrProfiling.cpp

cl::opt<double> NumCountersPerValueSite(
     "vp-counters-per-site",
     cl::desc("The average number of profile counters allocated "
              "per value profiling site."),
     // This is set to a very small value because in real programs, only
     // a very small percentage of value sites have non-zero targets, e.g, 1/30.
     // For those sites with non-zero profile, the average number of targets
     // is usually smaller than 2.
     cl::init(1.0));
 
+cl::opt<std::string> MemOPSizeRange(
+    "memop-size-range",
+    cl::desc("Set the range of size in memory intrinsic calls to be profiled "
+             "precisely, in a format of <start_val>:<end_val>"),
+    cl::init(""));
+cl::opt<unsigned> MemOPSizeLarge(
+    "memop-size-large",
+    cl::desc("Set large value thresthold in memory intrinsic size profiling. "
+             "Value of 0 disables the large value profiling."),
+    cl::init(8192));
+
 class InstrProfilingLegacyPass : public ModulePass {
   InstrProfiling InstrProf;
 
 public:
   static char ID;
 
   InstrProfilingLegacyPass() : ModulePass(ID) {}
   InstrProfilingLegacyPass(const InstrProfOptions &Options)
bool InstrProfiling::run(Module &M, const TargetLibraryInfo &TLI) {
 
   this->M = &M;
   this->TLI = &TLI;
   NamesVar = nullptr;
   NamesSize = 0;
   ProfileDataMap.clear();
   UsedVars.clear();
 
+  // Parse the value profile options.
+  MemOPSizeRangeStart = DefaultMemOPSizeRangeStart;
+  MemOPSizeRangeLast = DefaultMemOPSizeRangeLast;
+  if (!MemOPSizeRange.empty()) {
+    auto Pos = MemOPSizeRange.find(":");
+    if (Pos != std::string::npos) {
+      if (Pos > 0)
+        MemOPSizeRangeStart = std::stoi(MemOPSizeRange.substr(0, Pos));
+      if (Pos < MemOPSizeRange.size() - 1)
+        MemOPSizeRangeLast = std::stoi(MemOPSizeRange.substr(Pos + 1));
+    } else
+      MemOPSizeRangeLast = std::stoi(MemOPSizeRange);
+  }
+  assert(MemOPSizeRangeLast >= MemOPSizeRangeStart);
+
+  MemOPSizeLargeVal = MemOPSizeLarge;
+  if (MemOPSizeLargeVal == 0)
+    MemOPSizeLargeVal = INT64_MIN;
+
   // We did not know how many value sites there would be inside
   // the instrumented function. This is counting the number of instrumented
   // target value sites to enter it as field in the profile data variable.
   for (Function &F : M) {
     InstrProfIncrementInst *FirstProfIncInst = nullptr;
     for (BasicBlock &BB : F)
       for (auto I = BB.begin(), E = BB.end(); I != E; I++)
         if (auto *Ind = dyn_cast<InstrProfValueProfileInst>(I))
bool InstrProfiling::run(Module &M, const TargetLibraryInfo &TLI) {
   emitRegistration();
   emitRuntimeHook();
   emitUses();
   emitInitialization();
   return true;
 }
 
 static Constant *getOrInsertValueProfilingCall(Module &M,
-                                               const TargetLibraryInfo &TLI) {
+                                               const TargetLibraryInfo &TLI,
+                                               bool IsRange = false) {
   LLVMContext &Ctx = M.getContext();
   auto *ReturnTy = Type::getVoidTy(M.getContext());
+
+  Constant *Res;
+  if (!IsRange) {
     Type *ParamTypes[] = {
 #define VALUE_PROF_FUNC_PARAM(ParamType, ParamName, ParamLLVMType) ParamLLVMType
 #include "llvm/ProfileData/InstrProfData.inc"
     };
     auto *ValueProfilingCallTy =
         FunctionType::get(ReturnTy, makeArrayRef(ParamTypes), false);
-  Constant *Res = M.getOrInsertFunction(getInstrProfValueProfFuncName(),
-                                        ValueProfilingCallTy);
+    Res = M.getOrInsertFunction(getInstrProfValueProfFuncName(),
+  } else {
+    Type *RangeParamTypes[] = {
+#define VALUE_RANGE_PROF 1
+#define VALUE_PROF_FUNC_PARAM(ParamType, ParamName, ParamLLVMType) ParamLLVMType
+#include "llvm/ProfileData/InstrProfData.inc"
+#undef VALUE_RANGE_PROF
+    };
+    auto *ValueRangeProfilingCallTy =
+        FunctionType::get(ReturnTy, makeArrayRef(RangeParamTypes), false);
+    Res = M.getOrInsertFunction(getInstrProfValueRangeProfFuncName(),
+                                ValueRangeProfilingCallTy);
+  }
+
   if (Function *FunRes = dyn_cast<Function>(Res)) {
     if (auto AK = TLI.getExtAttrForI32Param(false))
       FunRes->addAttribute(3, AK);
   }
   return Res;
 }
 
 void InstrProfiling::computeNumValueSiteCounts(InstrProfValueProfileInst *Ind) {
void InstrProfiling::lowerValueProfileInst(InstrProfValueProfileInst *Ind) {
 
   GlobalVariable *DataVar = It->second.DataVar;
   uint64_t ValueKind = Ind->getValueKind()->getZExtValue();
   uint64_t Index = Ind->getIndex()->getZExtValue();
   for (uint32_t Kind = IPVK_First; Kind < ValueKind; ++Kind)
     Index += It->second.NumValueSites[Kind];
 
   IRBuilder<> Builder(Ind);
+  bool IsRange = (Ind->getValueKind()->getZExtValue() ==
+                  llvm::InstrProfValueKind::IPVK_MemOPSize);
+  CallInst *Call = nullptr;
+  if (!IsRange) {
     Value *Args[3] = {Ind->getTargetValue(),
                       Builder.CreateBitCast(DataVar, Builder.getInt8PtrTy()),
                       Builder.getInt32(Index)};
-  CallInst *Call = Builder.CreateCall(getOrInsertValueProfilingCall(*M, *TLI),
-                                      Args);
+    Call = Builder.CreateCall(getOrInsertValueProfilingCall(*M, *TLI), Args);
+  } else {
+    Value *Args[6] = {Ind->getTargetValue(),
+                      Builder.CreateBitCast(DataVar, Builder.getInt8PtrTy()),
+                      Builder.getInt32(Index),
+                      Builder.getInt64(MemOPSizeRangeStart),
+                      Builder.getInt64(MemOPSizeRangeLast),
+                      Builder.getInt64(MemOPSizeLargeVal)};
+    Call =
+        Builder.CreateCall(getOrInsertValueProfilingCall(*M, *TLI, true), Args);
+  }
   if (auto AK = TLI->getExtAttrForI32Param(false))
     Call->addAttribute(3, AK);
   Ind->replaceAllUsesWith(Call);
   Ind->eraseFromParent();
 }
 
 void InstrProfiling::lowerIncrement(InstrProfIncrementInst *Inc) {
   GlobalVariable *Counters = getOrCreateRegionCounters(Inc);
void InstrProfiling::emitVNodes() {
   // For now only support this on platforms that do
   // not require runtime registration to discover
   // named section start/end.
   if (needsRuntimeRegistrationOfSectionRange(*M))
     return;
 
   size_t TotalNS = 0;
   for (auto &PD : ProfileDataMap) {
-    for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind)
-      TotalNS += PD.second.NumValueSites[Kind];
+    for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind) {
+      auto I = PD.second.NumValueSites[Kind];
+      TotalNS += I;
+      // For value range profiles, such as IPVK_MemOPSize, we need preallocate
+      // more value sites.
+      if (Kind == IPVK_MemOPSize)
+        TotalNS += I * (MemOPSizeRangeLast + 1);
+    }
   }
 
   if (!TotalNS)
     return;
 
   uint64_t NumCounters = TotalNS * NumCountersPerValueSite;
 // Heuristic for small programs with very few total value sites.
 // The default value of vp-counters-per-site is chosen based on
 // the observation that large apps usually have a low percentage
 // of value sites that actually have any profile data, and thus
 // the average number of counters per site is low. For small
 // apps with very few sites, this may not be true. Bump up the
 // number of counters in this case.
 #define INSTR_PROF_MIN_VAL_COUNTS 10

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

 #include <vector>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "pgo-instrumentation"
 
 STATISTIC(NumOfPGOInstrument, "Number of edges instrumented.");
 STATISTIC(NumOfPGOSelectInsts, "Number of select instruction instrumented.");
+STATISTIC(NumOfPGOMemIntrinsics, "Number of mem intrinsics instrumented.");
 STATISTIC(NumOfPGOEdge, "Number of edges.");
 STATISTIC(NumOfPGOBB, "Number of basic-blocks.");
 STATISTIC(NumOfPGOSplit, "Number of critical edge splits.");
 STATISTIC(NumOfPGOFunc, "Number of functions having valid profile counts.");
 STATISTIC(NumOfPGOMismatch, "Number of functions having mismatch profile.");
 STATISTIC(NumOfPGOMissing, "Number of functions without profile.");
 STATISTIC(NumOfPGOICall, "Number of indirect call value instrumentations.");
 

 
 // Command line option to specify the name of the function for CFG dump
 static cl::opt<std::string>
     PGOViewFunction("pgo-view-function", cl::Hidden,
                     cl::desc("The option to specify "
                              "the name of the function "
                              "whose CFG will be displayed."));
 
+// Command line option to enable/disable memop intrinsic calls..
+static cl::opt<bool> PGOInstrMemOP("pgo-instr-memop", cl::init(true),
+                                    cl::Hidden);
+
 // Command line option to turn on CFG dot dump after profile annotation.
 extern cl::opt<bool> PGOViewCounts;
 
 namespace {
 
 /// The select instruction visitor plays three roles specified
 /// by the mode. In \c VM_counting mode, it simply counts the number of
 /// select instructions. In \c VM_instrument mode, it inserts code to count
 /// the number times TrueValue of select is taken. In \c VM_annotate mode,
 /// it reads the profile data and annotate the select instruction with metadata.
-enum VisitMode { VM_counting, VM_instrument, VM_annotate };
+enum VisitMode { VM_counting, VM_instrument, VM_annotate, VM_optimize };
 class PGOUseFunc;
 
 /// Instruction Visitor class to visit select instructions.
 struct SelectInstVisitor : public InstVisitor<SelectInstVisitor> {
   Function &F;
   unsigned NSIs = 0;             // Number of select instructions instrumented.
   VisitMode Mode = VM_counting;  // Visiting mode.
   unsigned *CurCtrIdx = nullptr; // Pointer to current counter index.
struct SelectInstVisitor : public InstVisitor<SelectInstVisitor> {
 
   void instrumentOneSelectInst(SelectInst &SI);
   void annotateOneSelectInst(SelectInst &SI);
   // Visit \p SI instruction and perform tasks according to visit mode.
   void visitSelectInst(SelectInst &SI);
   unsigned getNumOfSelectInsts() const { return NSIs; }
 };
 
+/// Instruction Visitor class to visit memory intrinsic calls.
+struct MemIntrinsicVisitor : public InstVisitor<MemIntrinsicVisitor> {
+  Function &F;
+  unsigned NMemIs = 0;           // Number of memIntrinsics instrumented.
+  VisitMode Mode = VM_counting;  // Visiting mode.
+  unsigned CurCtrId = 0;         // Current counter index.
+  unsigned TotalNumCtrs = 0;     // Total number of counters
+  GlobalVariable *FuncNameVar = nullptr;
+  uint64_t FuncHash = 0;
+  PGOUseFunc *UseFunc = nullptr;
+
+  MemIntrinsicVisitor(Function &Func) : F(Func) {}
+
+  void countMemIntrinsics(Function &Func) {
+    NMemIs = 0;
+    Mode = VM_counting;
+    visit(Func);
+  }
+  void instrumentMemIntrinsics(Function &Func, unsigned TotalNC,
+                               GlobalVariable *FNV, uint64_t FHash) {
+    Mode = VM_instrument;
+    TotalNumCtrs = TotalNC;
+    FuncHash = FHash;
+    FuncNameVar = FNV;
+    visit(Func);
+  }
+
+  // Visit the IR stream and annotate all mem intrinsic call instructions.
+  void optimizeMemIntrinsics(Function &Func, PGOUseFunc *UF);
+  void instrumentOneMemIntrinsic(MemIntrinsic &MI);
+  void optimizeOneMemIntrinsic(MemIntrinsic &MI);
+  // Visit \p MI instruction and perform tasks according to visit mode.
+  void visitMemIntrinsic(MemIntrinsic &SI);
+  unsigned getNumOfMemIntrinsics() const { return NMemIs; }
+};
+
 class PGOInstrumentationGenLegacyPass : public ModulePass {
 public:
   static char ID;
 
   PGOInstrumentationGenLegacyPass() : ModulePass(ID) {
     initializePGOInstrumentationGenLegacyPassPass(
         *PassRegistry::getPassRegistry());
   }
private:
   void computeCFGHash();
   void renameComdatFunction();
   // A map that stores the Comdat group in function F.
   std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers;
 
 public:
   std::vector<Instruction *> IndirectCallSites;
   SelectInstVisitor SIVisitor;
+  MemIntrinsicVisitor MIVisitor;
   std::string FuncName;
   GlobalVariable *FuncNameVar;
   // CFG hash value for this function.
   uint64_t FunctionHash;
 
   // The Minimum Spanning Tree of function CFG.
   CFGMST<Edge, BBInfo> MST;
 
MST.dumpEdges(dbgs(), Twine("Dump Function ") + FuncName + " Hash: " +
                               Twine(FunctionHash) + "\t" + Str);
   }
 
   FuncPGOInstrumentation(
       Function &Func,
       std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,
       bool CreateGlobalVar = false, BranchProbabilityInfo *BPI = nullptr,
       BlockFrequencyInfo *BFI = nullptr)
-      : F(Func), ComdatMembers(ComdatMembers), SIVisitor(Func), FunctionHash(0),
-        MST(F, BPI, BFI) {
+      : F(Func), ComdatMembers(ComdatMembers), SIVisitor(Func),
+        MIVisitor(Func), FunctionHash(0), MST(F, BPI, BFI) {
 
     // This should be done before CFG hash computation.
     SIVisitor.countSelects(Func);
+    MIVisitor.countMemIntrinsics(Func);
     NumOfPGOSelectInsts += SIVisitor.getNumOfSelectInsts();
+    NumOfPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();
     IndirectCallSites = findIndirectCallSites(Func);
 
     FuncName = getPGOFuncName(F);
     computeCFGHash();
     if (ComdatMembers.size())
       renameComdatFunction();
     DEBUG(dumpInfo("after CFGMST"));
 
Builder.CreateCall(
         Intrinsic::getDeclaration(M, Intrinsic::instrprof_value_profile),
         {llvm::ConstantExpr::getBitCast(FuncInfo.FuncNameVar, I8PtrTy),
          Builder.getInt64(FuncInfo.FunctionHash),
          Builder.CreatePtrToInt(Callee, Builder.getInt64Ty()),
          Builder.getInt32(llvm::InstrProfValueKind::IPVK_IndirectCallTarget),
          Builder.getInt32(NumIndirectCallSites++)});
   }
   NumOfPGOICall += NumIndirectCallSites;
+
+  // Now instrument memop instrinsic calls:
+  FuncInfo.MIVisitor.instrumentMemIntrinsics(F, NumCounters,
+                                             FuncInfo.FuncNameVar,
+                                             FuncInfo.FunctionHash);
 }
 
 // This class represents a CFG edge in profile use compilation.
 struct PGOUseEdge : public PGOEdge {
   bool CountValid;
   uint64_t CountValue;
   PGOUseEdge(const BasicBlock *Src, const BasicBlock *Dest, unsigned W = 1)
       : PGOEdge(Src, Dest, W), CountValid(false), CountValue(0) {}
static void setProfMetadata(Module *M, Instruction *TI,
   MDBuilder MDB(M->getContext());
   assert(MaxCount > 0 && "Bad max count");
   uint64_t Scale = calculateCountScale(MaxCount);
   SmallVector<unsigned, 4> Weights;
   for (const auto &ECI : EdgeCounts)
     Weights.push_back(scaleBranchCount(ECI, Scale));
 
   DEBUG(dbgs() << "Weight is: ";
-        for (const auto &W : Weights) { dbgs() << W << " "; } 
+        for (const auto &W : Weights) { dbgs() << W << " "; }
         dbgs() << "\n";);
   TI->setMetadata(llvm::LLVMContext::MD_prof, MDB.createBranchWeights(Weights));
 }
 
 // Assign the scaled count values to the BB with multiple out edges.
 void PGOUseFunc::setBranchWeights() {
   // Generate MD_prof metadata for every branch instruction.
   DEBUG(dbgs() << "\nSetting branch weights.\n");
void SelectInstVisitor::visitSelectInst(SelectInst &SI) {
   case VM_counting:
     return;
   case VM_instrument:
     instrumentOneSelectInst(SI);
     return;
   case VM_annotate:
     annotateOneSelectInst(SI);
     return;
+  default:
+    break;
   }
 
   llvm_unreachable("Unknown visiting mode");
 }
 
+void MemIntrinsicVisitor::instrumentOneMemIntrinsic(MemIntrinsic &MI) {
+  Module *M = F.getParent();
+  IRBuilder<> Builder(&MI);
+  Type *Int64Ty = Builder.getInt64Ty();
+  Type *I8PtrTy = Builder.getInt8PtrTy();
+  Value *Length = MI.getLength();
+  assert(!dyn_cast<ConstantInt>(Length));
+  Builder.CreateCall(
+      Intrinsic::getDeclaration(M, Intrinsic::instrprof_value_profile),
+      {llvm::ConstantExpr::getBitCast(FuncNameVar, I8PtrTy),
+       Builder.getInt64(FuncHash), Builder.CreatePtrToInt(Length, Int64Ty),
+       Builder.getInt32(llvm::InstrProfValueKind::IPVK_MemOPSize),
+       Builder.getInt32(CurCtrId)});
+  ++CurCtrId;
+}
+
+void MemIntrinsicVisitor::visitMemIntrinsic(MemIntrinsic &MI) {
+  if (!PGOInstrMemOP)
+    return;
+  Value *Length = MI.getLength();
+  // Not instrument constant length calls.
+  if (dyn_cast<ConstantInt>(Length))
+    return;
+
+  NMemIs++;
+  switch (Mode) {
+  case VM_counting:
+    return;
+  case VM_instrument:
+    instrumentOneMemIntrinsic(MI);
+    return;
+  default:
+    break;
+  }
+  llvm_unreachable("Unknown visiting mode");
+}
+
 // Traverse all the indirect callsites and annotate the instructions.
 void PGOUseFunc::annotateIndirectCallSites() {
   if (DisableValueProfiling)
     return;
 
   // Create the PGOFuncName meta data.
   createPGOFuncNameMetadata(F, FuncInfo.FuncName);
 
for (auto &F : M) {
     auto *BPI = LookupBPI(F);
     auto *BFI = LookupBFI(F);
     PGOUseFunc Func(F, &M, ComdatMembers, BPI, BFI);
     if (!Func.readCounters(PGOReader.get()))
       continue;
     Func.populateCounters();
     Func.setBranchWeights();
     Func.annotateIndirectCallSites();
+
     PGOUseFunc::FuncFreqAttr FreqAttr = Func.getFuncFreqAttr();
     if (FreqAttr == PGOUseFunc::FFA_Cold)
       ColdFunctions.push_back(&F);
     else if (FreqAttr == PGOUseFunc::FFA_Hot)
       HotFunctions.push_back(&F);
 #ifndef NDEBUG
     if (PGOViewCounts &&
         (PGOViewFunction.empty() || F.getName().equals(PGOViewFunction))) {

test/Instrumentation/InstrProfiling/PR23499.ll

 ; RUN: opt < %s -mtriple=x86_64-pc-win32-coff -passes=instrprof -S | FileCheck %s --check-prefix=COFF
 
 $_Z3barIvEvv = comdat any
 
 @__profn__Z3barIvEvv = linkonce_odr hidden constant [11 x i8] c"_Z3barIvEvv", align 1
 
 ; CHECK: @__profn__Z3barIvEvv = private constant [11 x i8] c"_Z3barIvEvv", align 1
 ; CHECK: @__profc__Z3barIvEvv = linkonce_odr hidden global [1 x i64] zeroinitializer, section "{{.*}}__llvm_prf_cnts", comdat($__profv__Z3barIvEvv), align 8
-; CHECK: @__profd__Z3barIvEvv = linkonce_odr hidden global { i64, i64, i64*, i8*, i8*, i32, [1 x i16] } { i64 4947693190065689389, i64 0, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__Z3barIvEvv, i32 0, i32 0), i8*{{.*}}, i8* null, i32 1, [1 x i16] zeroinitializer }, section "{{.*}}__llvm_prf_data{{.*}}", comdat($__profv__Z3barIvEvv), align 8
+; CHECK: @__profd__Z3barIvEvv = linkonce_odr hidden global { i64, i64, i64*, i8*, i8*, i32, [2 x i16] } { i64 4947693190065689389, i64 0, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__Z3barIvEvv, i32 0, i32 0), i8*{{.*}}, i8* null, i32 1, [2 x i16] zeroinitializer }, section "{{.*}}__llvm_prf_data{{.*}}", comdat($__profv__Z3barIvEvv), align 8
 ; CHECK: @__llvm_prf_nm = private constant [{{.*}} x i8] c"{{.*}}", section "{{.*}}__llvm_prf_names"
 
 
 ; COFF: @__profn__Z3barIvEvv = private constant [11 x i8] c"_Z3barIvEvv", align 1
 ; COFF: @__profc__Z3barIvEvv = linkonce_odr hidden global [1 x i64] zeroinitializer, section "{{.*}}__llvm_prf_cnts", comdat, align 8
-; COFF: @__profd__Z3barIvEvv = linkonce_odr hidden global { i64, i64, i64*, i8*, i8*, i32, [1 x i16] } { i64 4947693190065689389, i64 0, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__Z3barIvEvv, i32 0, i32 0), i8*{{.*}}, i8* null, i32 1, [1 x i16] zeroinitializer }, section "{{.*}}__llvm_prf_data{{.*}}", comdat($__profc__Z3barIvEvv), align 8
+; COFF: @__profd__Z3barIvEvv = linkonce_odr hidden global { i64, i64, i64*, i8*, i8*, i32, [2 x i16] } { i64 4947693190065689389, i64 0, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__Z3barIvEvv, i32 0, i32 0), i8*{{.*}}, i8* null, i32 1, [2 x i16] zeroinitializer }, section "{{.*}}__llvm_prf_data{{.*}}", comdat($__profc__Z3barIvEvv), align 8
 
 
 declare void @llvm.instrprof.increment(i8*, i64, i32, i32) #1
 
 define linkonce_odr void @_Z3barIvEvv() comdat {
 entry:
   call void @llvm.instrprof.increment(i8* getelementptr inbounds ([11 x i8], [11 x i8]* @__profn__Z3barIvEvv, i32 0, i32 0), i64 0, i32 1, i32 0)
   ret void
 }

test/Instrumentation/InstrProfiling/icall.ll

 attributes #0 = { nounwind }
 
 ; STATIC: @__profvp_foo
 ; STATIC: @__llvm_prf_vnodes
 
 ; DYN-NOT: @__profvp_foo
 ; DYN-NOT: @__llvm_prf_vnodes
 
-; STATIC: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [1 x i16] }* @__profd_foo to i8*), i32 0)
-; STATIC-EXT: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [1 x i16] }* @__profd_foo to i8*), i32 zeroext 0)
-; STATIC-SEXT: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [1 x i16] }* @__profd_foo to i8*), i32 signext 0)
+; STATIC: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [2 x i16] }* @__profd_foo to i8*), i32 0)
+; STATIC-EXT: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [2 x i16] }* @__profd_foo to i8*), i32 zeroext 0)
+; STATIC-SEXT: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [2 x i16] }* @__profd_foo to i8*), i32 signext 0)
 
 ; STATIC: declare void @__llvm_profile_instrument_target(i64, i8*, i32)
 ; STATIC-EXT: declare void @__llvm_profile_instrument_target(i64, i8*, i32 zeroext)
 ; STATIC-SEXT: declare void @__llvm_profile_instrument_target(i64, i8*, i32 signext)

test/Transforms/PGOProfile/comdat_internal.ll

 ; RUN: opt < %s -pgo-instr-gen -instrprof -S | FileCheck %s
 ; RUN: opt < %s -passes=pgo-instr-gen,instrprof -S | FileCheck %s
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 $foo = comdat any
 ; CHECK: $foo = comdat any
 
 ; CHECK: $__llvm_profile_raw_version = comdat any
 ; CHECK: $__profv__stdin__foo.[[FOO_HASH:[0-9]+]] = comdat any
 
 @bar = global i32 ()* @foo, align 8
 
 ; CHECK: @__llvm_profile_raw_version = constant i64 {{[0-9]+}}, comdat
 ; CHECK: @__profn__stdin__foo = private constant [11 x i8] c"<stdin>:foo"
 ; CHECK: @__profc__stdin__foo.[[FOO_HASH]] = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat($__profv__stdin__foo.[[FOO_HASH]]), align 8
-; CHECK: @__profd__stdin__foo.[[FOO_HASH]] = private global { i64, i64, i64*, i8*, i8*, i32, [1 x i16] } { i64 -5640069336071256030, i64 [[FOO_HASH]], i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__stdin__foo.[[FOO_HASH]], i32 0, i32 0), i8* null
+; CHECK: @__profd__stdin__foo.[[FOO_HASH]] = private global { i64, i64, i64*, i8*, i8*, i32, [2 x i16] } { i64 -5640069336071256030, i64 [[FOO_HASH]], i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__stdin__foo.[[FOO_HASH]], i32 0, i32 0), i8* null
 ; CHECK-NOT: bitcast (i32 ()* @foo to i8*)
-; CHECK-SAME: , i8* null, i32 1, [1 x i16] zeroinitializer }, section "__llvm_prf_data", comdat($__profv__stdin__foo.[[FOO_HASH]]), align 8
+; CHECK-SAME: , i8* null, i32 1, [2 x i16] zeroinitializer }, section "__llvm_prf_data", comdat($__profv__stdin__foo.[[FOO_HASH]]), align 8
 ; CHECK: @__llvm_prf_nm
 ; CHECK: @llvm.used
 
 define internal i32 @foo() comdat {
 entry:
   ret i32 1
 }

test/Transforms/PGOProfile/memcpy.ll

+; RUN: opt < %s -pgo-instr-gen -instrprof -S | FileCheck %s
+; RUN: opt <%s -passes=pgo-instr-gen,instrprof -S | FileCheck %s
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @foo(i8* %dst, i8* %src, i32* %a, i32 %n) {
+entry:
+  br label %for.cond
+
+for.cond:
+  %i.0 = phi i32 [ 0, %entry ], [ %add, %for.cond1 ]
+  %cmp = icmp slt i32 %i.0, %n
+  br i1 %cmp, label %for.cond1, label %for.end6
+
+for.cond1:
+  %j.0 = phi i32 [ %inc, %for.body3 ], [ 0, %for.cond ]
+  %idx.ext = sext i32 %i.0 to i64
+  %add.ptr = getelementptr inbounds i32, i32* %a, i64 %idx.ext
+  %0 = load i32, i32* %add.ptr, align 4
+  %cmp2 = icmp slt i32 %j.0, %0
+  %add = add nsw i32 %i.0, 1
+  br i1 %cmp2, label %for.body3, label %for.cond
+
+for.body3:
+  %conv = sext i32 %add to i64
+; CHECK: call void @__llvm_profile_instrument_range(i64 %conv, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [2 x i16] }* @__profd_foo to i8*), i32 0, i64 0, i64 8, i64 8192)
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %dst, i8* %src, i64 %conv, i32 1, i1 false)
+  %inc = add nsw i32 %j.0, 1
+  br label %for.cond1
+
+for.end6:
+  ret void
+}
+
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i32, i1)

test/tools/llvm-profdata/memop-size-prof.proftext

+# RUN: llvm-profdata show -memop-sizes -all-functions %s | FileCheck %s --check-prefix=MEMOP --check-prefix=MEMOP_SUM
+# RUN: llvm-profdata show -memop-sizes -counts -text -all-functions %s | FileCheck %s --check-prefix=MEMOP_TEXT
+# RUN: llvm-profdata merge -o %t.profdata  %s
+# RUN: llvm-profdata show -memop-sizes -all-functions %t.profdata | FileCheck %s --check-prefix=MEMOP --check-prefix=MEMOP_SUM
+
+# IR level Instrumentation Flag
+:ir
+foo
+# Func Hash:
[phone]
+# Num Counters:
+3
+# Counter Values:
+20
+556
+1
+# Num Value Kinds:
+1
+# ValueKind = IPVK_MemOPSize:
+1
+# NumValueSites:
+1
+9
+1:99
+2:88
+3:77
+9:72
+4:66
+5:55
+6:44
+7:33
+8:22
+
+#MEMOP: Memory Intrinsics Size: 1
+#MEMOP-NEXT: Memory Instrinsic Size Results:
+#MEMOP-NEXT:  [ 0, 1, 99 ]
+#MEMOP-NEXT:  [ 0, 2, 88 ]
+#MEMOP-NEXT:  [ 0, 3, 77 ]
+#MEMOP-NEXT:  [ 0, 9, 72 ]
+#MEMOP-NEXT:  [ 0, 4, 66 ]
+#MEMOP-NEXT:  [ 0, 5, 55 ]
+#MEMOP-NEXT:  [ 0, 6, 44 ]
+#MEMOP-NEXT:  [ 0, 7, 33 ]
+#MEMOP-NEXT:  [ 0, 8, 22 ]
+
+#MEMOP_SUM: Total Number of Memory Intrinsic Calls Profiled: 1
+#MEMOP_SUM: Total Number of Memory Intrinsic Calls With Values : 1
+#MEMOP_SUM: Total Number of Memory Intrinsic Calls Profiled Values : 9
+#MEMOP_SUM: MemOPSize Value histogram:
+#MEMOP_SUM: Size, SiteCount
+#MEMOP_SUM: 1, 0
+#MEMOP_SUM: 2, 0
+#MEMOP_SUM: 3, 0
+#MEMOP_SUM: 4, 0
+#MEMOP_SUM: 5, 0
+#MEMOP_SUM: 6, 0
+#MEMOP_SUM: 7, 0
+#MEMOP_SUM: 8, 0
+#MEMOP_SUM: 9, 1
+
+#MEMOP_TEXT: # Num Value Kinds:
+#MEMOP_TEXT: 1
+#MEMOP_TEXT: # ValueKind = IPVK_MemOPSize:
+#MEMOP_TEXT: 1
+#MEMOP_TEXT: # NumValueSites:
+#MEMOP_TEXT: 1
+#MEMOP_TEXT: 9
+#MEMOP_TEXT: 1:99
+#MEMOP_TEXT: 2:88
+#MEMOP_TEXT: 3:77
+#MEMOP_TEXT: 9:72
+#MEMOP_TEXT: 4:66
+#MEMOP_TEXT: 5:55
+#MEMOP_TEXT: 6:44
+#MEMOP_TEXT: 7:33
+#MEMOP_TEXT: 8:22

tools/llvm-profdata/llvm-profdata.cpp

if (ProfileKind == instr)
     mergeInstrProfile(WeightedInputs, OutputFilename, OutputFormat,
                       OutputSparse, NumThreads);
   else
     mergeSampleProfile(WeightedInputs, OutputFilename, OutputFormat);
 
   return 0;
 }
 
 static int showInstrProfile(const std::string &Filename, bool ShowCounts,
-                            bool ShowIndirectCallTargets,
+                            bool ShowIndirectCallTargets, bool ShowMemOPSizes,
                             bool ShowDetailedSummary,
                             std::vector<uint32_t> DetailedSummaryCutoffs,
                             bool ShowAllFunctions,
                             const std::string &ShowFunction, bool TextFormat,
                             raw_fd_ostream &OS) {
   auto ReaderOrErr = InstrProfReader::create(Filename);
   std::vector<uint32_t> Cutoffs = std::move(DetailedSummaryCutoffs);
   if (ShowDetailedSummary && Cutoffs.empty()) {
     Cutoffs = {800000, 900000, 950000, 990000, 999000, 999900, 999990};
   }
   InstrProfSummaryBuilder Builder(std::move(Cutoffs));
   if (Error E = ReaderOrErr.takeError())
     exitWithError(std::move(E), Filename);
 
   auto Reader = std::move(ReaderOrErr.get());
   bool IsIRInstr = Reader->isIRLevelProfile();
   size_t ShownFunctions = 0;
   uint64_t TotalNumValueSites = 0;
   uint64_t TotalNumValueSitesWithValueProfile = 0;
   uint64_t TotalNumValues = 0;
   std::vector<unsigned> ICHistogram;
+  uint64_t MemOPSizeTotalNumValueSites = 0;
+  uint64_t MemOPSizeTotalNumValueSitesWithValueProfile = 0;
+  uint64_t MemOPSizeTotalNumValues = 0;
+  std::vector<unsigned> MemOPSizeHistogram;
   for (const auto &Func : *Reader) {
     bool Show =
         ShowAllFunctions || (!ShowFunction.empty() &&
                              Func.Name.find(ShowFunction) != Func.Name.npos);
 
     bool doTextFormatDump = (Show && ShowCounts && TextFormat);
 
     if (doTextFormatDump) {
       InstrProfSymtab &Symtab = Reader->getSymtab();
       InstrProfWriter::writeRecordInText(Func, Symtab, OS);
       continue;
     }
 
     assert(Func.Counts.size() > 0 && "function missing entry counter");
     Builder.addRecord(Func);
 
     if (Show) {
 
       if (!ShownFunctions)
         OS << "Counters:\n";
 
       ++ShownFunctions;
 
       OS << "  " << Func.Name << ":\n"
          << "    Hash: " << format("0x%016" PRIx64, Func.Hash) << "\n"
          << "    Counters: " << Func.Counts.size() << "\n";
       if (!IsIRInstr)
         OS << "    Function count: " << Func.Counts[0] << "\n";
 
       if (ShowIndirectCallTargets)
         OS << "    Indirect Call Site Count: "
            << Func.getNumValueSites(IPVK_IndirectCallTarget) << "\n";
 
+      uint32_t MemOPSize = Func.getNumValueSites(IPVK_MemOPSize);
+      if (ShowMemOPSizes && MemOPSize > 0)
+        OS << "    Memory Intrinsics Size: " << MemOPSize << "\n";
+
       if (ShowCounts) {
         OS << "    Block counts: [";
         size_t Start = (IsIRInstr ? 0 : 1);
         for (size_t I = Start, E = Func.Counts.size(); I < E; ++I) {
           OS << (I == Start ? "" : ", ") << Func.Counts[I];
         }
         OS << "]\n";
       }
if (Show) {
           }
           for (uint32_t V = 0; V < NV; V++) {
             OS << "\t[ " << I << ", ";
             OS << Symtab.getFuncName(VD[V].Value) << ", " << VD[V].Count
                << " ]\n";
           }
         }
       }
+      if (ShowMemOPSizes && MemOPSize > 0) {
+        uint32_t NS = Func.getNumValueSites(IPVK_MemOPSize);
+        OS << "    Memory Instrinsic Size Results:\n";
+        MemOPSizeTotalNumValueSites += NS;
+        for (size_t I = 0; I < NS; ++I) {
+          uint32_t NV = Func.getNumValueDataForSite(IPVK_MemOPSize, I);
+          std::unique_ptr<InstrProfValueData[]> VD =
+              Func.getValueForSite(IPVK_MemOPSize, I);
+          MemOPSizeTotalNumValues += NV;
+          if (NV) {
+            MemOPSizeTotalNumValueSitesWithValueProfile++;
+            if (NV > MemOPSizeHistogram.size())
+              MemOPSizeHistogram.resize(NV, 0);
+            MemOPSizeHistogram[NV - 1]++;
+          }
+          for (uint32_t V = 0; V < NV; V++) {
+            OS << "\t[ " << I << ", ";
+            OS << VD[V].Value << ", " << VD[V].Count << " ]\n";
+          }
+        }
+      }
     }
   }
   if (Reader->hasError())
     exitWithError(Reader->getError(), Filename);
 
   if (ShowCounts && TextFormat)
     return 0;
   std::unique_ptr<ProfileSummary> PS(Builder.getSummary());
OS << "Total Number of Sites With Values : "
        << TotalNumValueSitesWithValueProfile << "\n";
     OS << "Total Number of Profiled Values : " << TotalNumValues << "\n";
 
     OS << "IC Value histogram : \n\tNumTargets, SiteCount\n";
     for (unsigned I = 0; I < ICHistogram.size(); I++) {
       OS << "\t" << I + 1 << ", " << ICHistogram[I] << "\n";
     }
   }
+  if (ShownFunctions && ShowMemOPSizes) {
+    OS << "Total Number of Memory Intrinsic Calls Profiled: "
+       << MemOPSizeTotalNumValueSites << "\n";
+    OS << "Total Number of Memory Intrinsic Calls With Values : "
+       << MemOPSizeTotalNumValueSitesWithValueProfile << "\n";
+    OS << "Total Number of Memory Intrinsic Calls Profiled Values : "
+       << MemOPSizeTotalNumValues << "\n";
+
+    OS << "MemOPSize Value histogram:\n\tSize, SiteCount\n";
+    for (unsigned I = 0; I < MemOPSizeHistogram.size(); I++) {
+      OS << "\t" << I + 1 << ", " << MemOPSizeHistogram[I] << "\n";
+    }
+  }
 
   if (ShowDetailedSummary) {
     OS << "Detailed summary:\n";
     OS << "Total number of blocks: " << PS->getNumCounts() << "\n";
     OS << "Total count: " << PS->getTotalCount() << "\n";
     for (auto Entry : PS->getDetailedSummary()) {
       OS << Entry.NumCounts << " blocks with count >= " << Entry.MinCount
          << " account for "
static int show_main(int argc, const char *argv[]) {
   cl::opt<bool> ShowCounts("counts", cl::init(false),
                            cl::desc("Show counter values for shown functions"));
   cl::opt<bool> TextFormat(
       "text", cl::init(false),
       cl::desc("Show instr profile data in text dump format"));
   cl::opt<bool> ShowIndirectCallTargets(
       "ic-targets", cl::init(false),
       cl::desc("Show indirect call site target values for shown functions"));
+  cl::opt<bool> ShowMemOPSizes(
+      "memop-sizes", cl::init(false),
+      cl::desc("Show the profiled sizes of the memory intrinsic calls "
+               "for shown functions"));
   cl::opt<bool> ShowDetailedSummary("detailed-summary", cl::init(false),
                                     cl::desc("Show detailed profile summary"));
   cl::list<uint32_t> DetailedSummaryCutoffs(
       cl::CommaSeparated, "detailed-summary-cutoffs",
       cl::desc(
           "Cutoff percentages (times 10000) for generating detailed summary"),
       cl::value_desc("800000,901000,999999"));
   cl::opt<bool> ShowAllFunctions("all-functions", cl::init(false),
static int show_main(int argc, const char *argv[]) {
 
   if (ShowAllFunctions && !ShowFunction.empty())
     errs() << "warning: -function argument ignored: showing all functions\n";
 
   std::vector<uint32_t> Cutoffs(DetailedSummaryCutoffs.begin(),
                                 DetailedSummaryCutoffs.end());
   if (ProfileKind == instr)
     return showInstrProfile(Filename, ShowCounts, ShowIndirectCallTargets,
-                            ShowDetailedSummary, DetailedSummaryCutoffs,
-                            ShowAllFunctions, ShowFunction, TextFormat, OS);
+                            ShowMemOPSizes, ShowDetailedSummary,
+                            DetailedSummaryCutoffs, ShowAllFunctions,
+                            ShowFunction, TextFormat, OS);
   else
     return showSampleProfile(Filename, ShowCounts, ShowAllFunctions,
                              ShowFunction, OS);
 }
 
 int main(int argc, const char *argv[]) {
   // Print a stack trace if we signal out.
   sys::PrintStackTraceOnErrorSignal(argv[0]);