[PGO] Value profile for size of memory intrinsic calls

[xur]

Summary

This patch adds the value profile support to profile the size parameter of
memory intrinsic calls: memcpy, memcmp, and memmove.

We have two adjustable parameters: SmallVal and LargeVal size less than or equal to SmallVal will be precisely profiled. size greater than SmallVal but less than LargeVal will have one count size greater than or equal to LargeVale will have one count.

The patch has dependency on the profile value range support in compiler_rt:
https://reviews.llvm.org/D28964.

[davidxl]

See my comment in https://reviews.llvm.org/D28964. There is no need to introduce runtime change nor new clang intrinsics. The instrumentation lower can do the job of mapping the ranges. Another choice is to let the instrumenter does the range mapping work and keep the lowerer unchanged.

[xur]

Update the patch based on David's review comments.

(1) remove instrprof_value_rang_profile intrinsic and use existing instrprof_value_profile and used valuekind to distinguish different value profile.
(2) enclose most of the implementation details in value profile lowering.

[davidxl]

llvm-profdata needs test cases for profile reading and writing

1. show command
2. merge command in binary and in text format

lib/Transforms/Instrumentation/InstrProfiling.cpp

79

Perhaps make it a string option with both start and end values specified:

MemOpSizeValueRange

85

The description is not accurate.

tools/llvm-profdata/llvm-profdata.cpp

482

Add a test case to show text format dump works expected.

542

Collect summary statistics and dump it at the end -- similar to IC statistics.

636

This needs to be documented as other user facing options.

[xur]

lib/Transforms/Instrumentation/InstrProfiling.cpp

85

Fixed

[xur]

Integrated David's review comments.

Also fixed a bug in InstrProfReader when reading text format value profile.

[davidxl]

How to make the llvm-profdata 'show' command dump the range information as well?

include/llvm/Transforms/InstrProfiling.h

63 ↗

(On Diff #86697)

should it be 1?

66 ↗

(On Diff #86697)

Why 8? why not 64?

lib/ProfileData/InstrProfReader.cpp

185 ↗

(On Diff #86697)

Can you submit this separately as a bug fix?

lib/Transforms/Instrumentation/InstrProfiling.cpp

178

extract this into an option parsing helper.

528

Larger range does not mean on average the number of values tracked will be proportionally larger. Perhaps using a different heuristic for allocation size? Besides, TotalNS tracks the number of value sites, overloading it for number of values does not make sense

535

Here is the counter # estimation.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1283

unrelated change.

test/tools/llvm-profdata/memop-size-prof.proftext

3 ↗

(On Diff #86697)

also test 'merge' with -text option.

18 ↗

(On Diff #86697)

perhaps add a test with 2 value kinds

tools/llvm-profdata/llvm-profdata.cpp

469

TotalNumValueSitesICall

Or perhaps defined an array

TotalNumValueSites[..] and use value kind index to access.

470

TotalNumValuesSitesWithValueProfileICall

473

Make the var name consistent -- TotalNumValueSitesMemOpSize

506

NumMemOpCalls

508

Number of memory intrinsic calls :

543

extract this into to a common helper function? There are many duplications.

[xur]

include/llvm/Transforms/InstrProfiling.h

63 ↗

(On Diff #86697)

there are calls to memory intrinsics with size 0. There are quite many actually.

66 ↗

(On Diff #86697)

I think 8 is enough. In the optimization, I multi-version up to value of 8.
This is a tunable. 64 might be too big.

lib/Transforms/Instrumentation/InstrProfiling.cpp

528

I removed this change. Now MemOPSize profile uses the same heuristic as indirect-call profiling.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1283

reverted.

tools/llvm-profdata/llvm-profdata.cpp

469

Used a vector here.

[xur]

Updated the patch according on David's review comments.

[davidxl]

The profile annotation part of another patch should be merged in here.

include/llvm/Transforms/InstrProfiling.h

63 ↗

(On Diff #86697)

The value transformation patch does not seem to handle it . Do you see cases where value 0 is a frequent value in hot stringop sites?

lib/Transforms/Instrumentation/InstrProfiling.cpp

91

document the case when some value is missing in the option.

[davidxl]

lib/Transforms/Instrumentation/InstrProfiling.cpp

528

With this default heuristics, do you see any dropped value profiling at runtime?

tools/llvm-profdata/llvm-profdata.cpp

449

Can you split out this part as a refactoring patch?

[xur]

This is the patch that after splitting out the llvm-profdata refactor change.

[xur]

Update the patch: remove some unrelated changes.

[davidxl]

After the refactoring code lands, do you plan to add the site annotation part in this patch?

[xur]

In D28965#701857, @davidxl wrote:

After the refactoring code lands, do you plan to add the site annotation part in this patch?

I plan to do that part if a separated patch.

[davidxl]

lgtm

include/llvm/IR/IntrinsicInst.h

static inline bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }
   };
 
   /// This represents the llvm.instrprof_value_profile intrinsic.
   class InstrProfValueProfileInst : public IntrinsicInst {
   public:
     static inline bool classof(const IntrinsicInst *I) {
-      return I->getIntrinsicID() == Intrinsic::instrprof_value_profile;
+      switch (I->getIntrinsicID()) {
+      case Intrinsic::instrprof_value_profile:
+      case Intrinsic::instrprof_value_range_profile:
+        return true;
+      default: return false;
+      }
     }
     static inline bool classof(const Value *V) {
       return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
     }
 
     GlobalVariable *getName() const {
       return cast<GlobalVariable>(
           const_cast<Value *>(getArgOperand(0))->stripPointerCasts());
public:
     }
 
     // Returns the value site index.
     ConstantInt *getIndex() const {
       return cast<ConstantInt>(const_cast<Value *>(getArgOperand(4)));
     }
   };
 
+  /// This represents the llvm.instrprof_value_range_profile intrinsic.
+  class InstrProfValueRangeProfileInst : public InstrProfValueProfileInst {
+  public:
+    static inline bool classof(const IntrinsicInst *I) {
+      return I->getIntrinsicID() == Intrinsic::instrprof_value_range_profile;
+    }
+  };
 } // end namespace llvm
 
 #endif // LLVM_IR_INTRINSICINST_H

include/llvm/IR/Intrinsics.td

 // A call to profile runtime for value profiling of target expressions
 // through instrumentation based profiling.
 def int_instrprof_value_profile : Intrinsic<[],
                                             [llvm_ptr_ty, llvm_i64_ty,
                                              llvm_i64_ty, llvm_i32_ty,
                                              llvm_i32_ty],
                                             []>;
 
+// A call to profile runtime for value range profiling of target expressions
+// through instrumentation based profiling.
+def int_instrprof_value_range_profile : Intrinsic<[],
+                                                  [llvm_ptr_ty, llvm_i64_ty,
+                                                   llvm_i64_ty, llvm_i32_ty,
+                                                   llvm_i32_ty],
+                                                  []>;
+
 //===------------------- Standard C Library Intrinsics --------------------===//
 //
 
 def int_memcpy  : Intrinsic<[],
                              [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
                               llvm_i32_ty, llvm_i1_ty],
                             [IntrArgMemOnly, NoCapture<0>, NoCapture<1>,
                              WriteOnly<0>, ReadOnly<1>]>;

include/llvm/ProfileData/InstrProf.h

 }
 
 /// Return the name profile runtime entry point to do value profiling
 /// for a given site.
 inline StringRef getInstrProfValueProfFuncName() {
   return INSTR_PROF_VALUE_PROF_FUNC_STR;
 }
 
+/// Return the name profile runtime entry point to do value range profiling.
+inline StringRef getInstrProfValueRangeProfFuncName() {
+  return INSTR_PROF_VALUE_RANGE_PROF_FUNC_STR;
+}
+
 /// Return the name of the section containing function coverage mapping
 /// data.
 inline StringRef getInstrProfCoverageSectionName(bool AddSegment) {
   return AddSegment ? "__LLVM_COV," INSTR_PROF_COVMAP_SECT_NAME_STR
                     : INSTR_PROF_COVMAP_SECT_NAME_STR;
 }
 
 /// Return the name prefix of variables containing instrumented function names.

 bool getValueProfDataFromInst(const Instruction &Inst,
                               InstrProfValueKind ValueKind,
                               uint32_t MaxNumValueData,
                               InstrProfValueData ValueData[],
                               uint32_t &ActualNumValueData, uint64_t &TotalC);
 
 inline StringRef getPGOFuncNameMetadataName() { return "PGOFuncName"; }
 
+/// Set the global variable property to be used in profile runtime.
+void setRuntimeVariableProperty(Module &M, GlobalVariable *GV);
+
 /// Return the PGOFuncName meta data associated with a function.
 MDNode *getPGOFuncNameMetadata(const Function &F);
 
 /// Create the PGOFuncName meta data if PGOFuncName is different from
 /// function's raw name. This should only apply to internal linkage functions
 /// declared by users only.
 void createPGOFuncNameMetadata(Function &F, StringRef PGOFuncName);
 
for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind)
       getValueSitesForKind(Kind).clear();
   }
 
   /// Get the error contained within the record's soft error counter.
   Error takeError() { return SIPE.takeError(); }
 
 private:
   std::vector<InstrProfValueSiteRecord> IndirectCallSites;
+  std::vector<InstrProfValueSiteRecord> MemOPSizes;
   const std::vector<InstrProfValueSiteRecord> &
   getValueSitesForKind(uint32_t ValueKind) const {
     switch (ValueKind) {
     case IPVK_IndirectCallTarget:
       return IndirectCallSites;
+    case IPVK_MemOPSize:
+      return MemOPSizes;
     default:
       llvm_unreachable("Unknown value kind!");
     }
     return IndirectCallSites;
   }
 
   std::vector<InstrProfValueSiteRecord> &
   getValueSitesForKind(uint32_t ValueKind) {

include/llvm/ProfileData/InstrProfData.inc

  * written in the raw profile data and converted to target function name's MD5
  * hash by the profile reader during deserialization.  Typically, this happens
  * when the the raw profile data is read during profile merging.
  *
  * For this remapping the ProfData is used.  ProfData contains both the function
  * name hash and the function address.
  */
 VALUE_PROF_KIND(IPVK_IndirectCallTarget, 0)
+/* For memory intrinsic functions size profiling. */
+VALUE_PROF_KIND(IPVK_MemOPSize, 1)
 /* These two kinds must be the last to be
  * declared. This is to make sure the string
  * array created with the template can be
  * indexed with the kind value.
  */
 VALUE_PROF_KIND(IPVK_First, IPVK_IndirectCallTarget)
-VALUE_PROF_KIND(IPVK_Last, IPVK_IndirectCallTarget)
+VALUE_PROF_KIND(IPVK_Last, IPVK_MemOPSize)
 
 #undef VALUE_PROF_KIND
 /* VALUE_PROF_KIND end */
 
 /* COVMAP_FUNC_RECORD start */
 /* Definition of member fields of the function record structure in coverage
  * map.
  */

  * bits (i.e. bit 56) to 1 to indicate if this is an IR-level instrumentaiton
  * generated profile, and 0 if this is a Clang FE generated profile.
  */
 #define VARIANT_MASKS_ALL 0xff00000000000000ULL
 #define GET_VERSION(V) ((V) & ~VARIANT_MASKS_ALL)
 #define VARIANT_MASK_IR_PROF (0x1ULL << 56)
 #define INSTR_PROF_RAW_VERSION_VAR __llvm_profile_raw_version
 #define INSTR_PROF_PROFILE_RUNTIME_VAR __llvm_profile_runtime
+#define INSTR_PROF_MEMOP_SMALLVAL_VAR __llvm_profile_memop_smallval
+#define INSTR_PROF_MEMOP_LARGEVAL_VAR __llvm_profile_memop_largeval
 
 /* The variable that holds the name of the profile data
  * specified via command line. */
 #define INSTR_PROF_PROFILE_NAME_VAR __llvm_profile_filename
 
 /* Runtime section names and name strings.  */
 #define INSTR_PROF_DATA_SECT_NAME __llvm_prf_data
 #define INSTR_PROF_NAME_SECT_NAME __llvm_prf_names
#define INSTR_PROF_SECT_START(Sect) \
         INSTR_PROF_CONCAT(__start_,Sect)
 #define INSTR_PROF_SECT_STOP(Sect) \
         INSTR_PROF_CONCAT(__stop_,Sect)
 
 /* Value Profiling API linkage name.  */
 #define INSTR_PROF_VALUE_PROF_FUNC __llvm_profile_instrument_target
 #define INSTR_PROF_VALUE_PROF_FUNC_STR \
         INSTR_PROF_QUOTE(INSTR_PROF_VALUE_PROF_FUNC)
+#define INSTR_PROF_VALUE_RANGE_PROF_FUNC __llvm_profile_instrument_range
+#define INSTR_PROF_VALUE_RANGE_PROF_FUNC_STR \
+        INSTR_PROF_QUOTE(INSTR_PROF_VALUE_RANGE_PROF_FUNC)
 
 /* InstrProfile per-function control data alignment.  */
 #define INSTR_PROF_DATA_ALIGNMENT 8
 
 /* The data structure that represents a tracked value by the
  * value profiler.
  */
 typedef struct InstrProfValueData {

lib/ProfileData/InstrProf.cpp

bool canRenameComdatFunc(const Function &F, bool CheckAddressTaken) {
 
   // For AvailableExternallyLinkage functions.
   if (!F.hasComdat()) {
     assert(F.getLinkage() == GlobalValue::AvailableExternallyLinkage);
     return true;
   }
   return true;
 }
+
+// Set the global variable property to be used in profile runtime.
+void setRuntimeVariableProperty(Module &M, GlobalVariable *GV) {
+  assert(GV);
+  GV->setVisibility(GlobalValue::DefaultVisibility);
+  Triple TT(M.getTargetTriple());
+  if (!TT.supportsCOMDAT())
+    GV->setLinkage(GlobalValue::WeakAnyLinkage);
+  else
+    GV->setComdat(M.getOrInsertComdat(
+        StringRef(INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR))));
+}
 } // end namespace llvm

lib/Transforms/Instrumentation/InstrProfiling.cpp

cl::opt<double> NumCountersPerValueSite(
     "vp-counters-per-site",
     cl::desc("The average number of profile counters allocated "
              "per value profiling site."),
     // This is set to a very small value because in real programs, only
     // a very small percentage of value sites have non-zero targets, e.g, 1/30.
     // For those sites with non-zero profile, the average number of targets
     // is usually smaller than 2.
     cl::init(1.0));
+cl::opt<unsigned> MemOPSizeSmallVal(
+    "memop-size-small",
+    cl::desc("Set the threshold value under which to be profiled precisely"),
+    cl::init(8));
+cl::opt<unsigned> MemOPSizeLargeVal(
+    "memop-size-large",
+    cl::desc("Set the threshold value to be profiled seperately"),
+    cl::init(8192));
 
 class InstrProfilingLegacyPass : public ModulePass {
   InstrProfiling InstrProf;
 
 public:
   static char ID;
 
   InstrProfilingLegacyPass() : ModulePass(ID) {}
   InstrProfilingLegacyPass(const InstrProfOptions &Options)
       : ModulePass(ID), InstrProf(Options) {}
 
   StringRef getPassName() const override {
     return "Frontend instrumentation-based coverage lowering";
bool InstrProfiling::run(Module &M, const TargetLibraryInfo &TLI) {
   this->M = &M;
   this->TLI = &TLI;
   NamesVar = nullptr;
   NamesSize = 0;
   ProfileDataMap.clear();
   UsedVars.clear();
 
   // We did not know how many value sites there would be inside
   // the instrumented function. This is counting the number of instrumented
   // target value sites to enter it as field in the profile data variable.
   for (Function &F : M) {
     InstrProfIncrementInst *FirstProfIncInst = nullptr;
     for (BasicBlock &BB : F)
       for (auto I = BB.begin(), E = BB.end(); I != E; I++)
         if (auto *Ind = dyn_cast<InstrProfValueProfileInst>(I))
           computeNumValueSiteCounts(Ind);
         else if (FirstProfIncInst == nullptr)
bool InstrProfiling::run(Module &M, const TargetLibraryInfo &TLI) {
   emitRegistration();
   emitRuntimeHook();
   emitUses();
   emitInitialization();
   return true;
 }
 
 static Constant *getOrInsertValueProfilingCall(Module &M,
-                                               const TargetLibraryInfo &TLI) {
+                                               const TargetLibraryInfo &TLI,
+                                               bool IsRange = false) {
   LLVMContext &Ctx = M.getContext();
   auto *ReturnTy = Type::getVoidTy(M.getContext());
   Type *ParamTypes[] = {
 #define VALUE_PROF_FUNC_PARAM(ParamType, ParamName, ParamLLVMType) ParamLLVMType
 #include "llvm/ProfileData/InstrProfData.inc"
   };
   auto *ValueProfilingCallTy =
       FunctionType::get(ReturnTy, makeArrayRef(ParamTypes), false);
-  Constant *Res = M.getOrInsertFunction(getInstrProfValueProfFuncName(),
+  StringRef ValurProfFuncName = (IsRange ? getInstrProfValueRangeProfFuncName()
+                                 : getInstrProfValueProfFuncName());
+  Constant *Res = M.getOrInsertFunction(ValurProfFuncName,
                                         ValueProfilingCallTy);
   if (Function *FunRes = dyn_cast<Function>(Res)) {
     if (auto AK = TLI.getExtAttrForI32Param(false))
       FunRes->addAttribute(3, AK);
   }
   return Res;
 }
 
 void InstrProfiling::computeNumValueSiteCounts(InstrProfValueProfileInst *Ind) {
   GlobalVariable *Name = Ind->getName();
   uint64_t ValueKind = Ind->getValueKind()->getZExtValue();
   uint64_t Index = Ind->getIndex()->getZExtValue();
   auto It = ProfileDataMap.find(Name);
   if (It == ProfileDataMap.end()) {
     PerFunctionProfileData PD;
     PD.NumValueSites[ValueKind] = Index + 1;
     ProfileDataMap[Name] = PD;
   } else if (It->second.NumValueSites[ValueKind] <= Index)
     It->second.NumValueSites[ValueKind] = Index + 1;
 }
 
+// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR for the profile runtime.
+static void createMemOPSizeSmallValVariable(Module *M) {
+  Type *IntTy32 = Type::getInt32Ty(M->getContext());
+  unsigned V = MemOPSizeSmallVal;
+  auto GV = new GlobalVariable(
+      *M, IntTy32, true, GlobalVariable::ExternalLinkage,
+      Constant::getIntegerValue(IntTy32, APInt(32, V)),
+      INSTR_PROF_QUOTE(INSTR_PROF_MEMOP_LARGELVAL_VAR));
+  setRuntimeVariableProperty(*M, GV);
+}
+
+// Create a COMDAT variable INSTR_PROF_RAW_VERSION_VAR for the profile runtime.
+static void createMemOPSizeLargeValVariable(Module *M) {
+  Type *IntTy32 = Type::getInt32Ty(M->getContext());
+  unsigned V = MemOPSizeLargeVal;
+  auto GV = new GlobalVariable(
+      *M, IntTy32, true, GlobalVariable::ExternalLinkage,
+      Constant::getIntegerValue(IntTy32, APInt(32, V)),
+      INSTR_PROF_QUOTE(INSTR_PROF_MEMOP_LARGEVAL_VAR));
+  setRuntimeVariableProperty(*M, GV);
+}
+
 void InstrProfiling::lowerValueProfileInst(InstrProfValueProfileInst *Ind) {
+  static bool EmitRangeVariable = false;
   GlobalVariable *Name = Ind->getName();
   auto It = ProfileDataMap.find(Name);
   assert(It != ProfileDataMap.end() && It->second.DataVar &&
          "value profiling detected in function with no counter incerement");
 
   GlobalVariable *DataVar = It->second.DataVar;
   uint64_t ValueKind = Ind->getValueKind()->getZExtValue();
   uint64_t Index = Ind->getIndex()->getZExtValue();
   for (uint32_t Kind = IPVK_First; Kind < ValueKind; ++Kind)
     Index += It->second.NumValueSites[Kind];
 
   IRBuilder<> Builder(Ind);
   Value *Args[3] = {Ind->getTargetValue(),
                     Builder.CreateBitCast(DataVar, Builder.getInt8PtrTy()),
                     Builder.getInt32(Index)};
-  CallInst *Call = Builder.CreateCall(getOrInsertValueProfilingCall(*M, *TLI),
+  bool IsRange = false;
+  if (dyn_cast<InstrProfValueRangeProfileInst>(Ind))
+    IsRange = true;
+  if (IsRange && !EmitRangeVariable) {
+    EmitRangeVariable = true;
+    createMemOPSizeSmallValVariable(Ind->getModule());
+    createMemOPSizeLargeValVariable(Ind->getModule());
+  }
+  CallInst *Call = Builder.CreateCall(getOrInsertValueProfilingCall(*M, *TLI, IsRange),
                                       Args);
   if (auto AK = TLI->getExtAttrForI32Param(false))
     Call->addAttribute(3, AK);
   Ind->replaceAllUsesWith(Call);
   Ind->eraseFromParent();
 }
 
 void InstrProfiling::lowerIncrement(InstrProfIncrementInst *Inc) {
void InstrProfiling::emitVNodes() {
   // For now only support this on platforms that do
   // not require runtime registration to discover
   // named section start/end.
   if (needsRuntimeRegistrationOfSectionRange(*M))
     return;
 
   size_t TotalNS = 0;
   for (auto &PD : ProfileDataMap) {
-    for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind)
-      TotalNS += PD.second.NumValueSites[Kind];
+    for (uint32_t Kind = IPVK_First; Kind <= IPVK_Last; ++Kind) {
+      auto I = PD.second.NumValueSites[Kind];
+      TotalNS += I;
+      // For value range profiles, such as IPVK_MemOPSize, we need preallocate
+      // more value sites.
+      if (Kind == IPVK_MemOPSize)
+        TotalNS += I * (MemOPSizeSmallVal + 1);
+    }
   }
 
   if (!TotalNS)
     return;
 
   uint64_t NumCounters = TotalNS * NumCountersPerValueSite;
 // Heuristic for small programs with very few total value sites.
 // The default value of vp-counters-per-site is chosen based on
 // the observation that large apps usually have a low percentage
 // of value sites that actually have any profile data, and thus
 // the average number of counters per site is low. For small
 // apps with very few sites, this may not be true. Bump up the
 // number of counters in this case.
 #define INSTR_PROF_MIN_VAL_COUNTS 10

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

 #include <vector>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "pgo-instrumentation"
 
 STATISTIC(NumOfPGOInstrument, "Number of edges instrumented.");
 STATISTIC(NumOfPGOSelectInsts, "Number of select instruction instrumented.");
+STATISTIC(NumOfPGOMemIntrinsics, "Number of mem intrinsics instrumented.");
 STATISTIC(NumOfPGOEdge, "Number of edges.");
 STATISTIC(NumOfPGOBB, "Number of basic-blocks.");
 STATISTIC(NumOfPGOSplit, "Number of critical edge splits.");
 STATISTIC(NumOfPGOFunc, "Number of functions having valid profile counts.");
 STATISTIC(NumOfPGOMismatch, "Number of functions having mismatch profile.");
 STATISTIC(NumOfPGOMissing, "Number of functions without profile.");
 STATISTIC(NumOfPGOICall, "Number of indirect call value instrumentations.");
 

 // the profile data for Comdat functions, which often turns out to be false
 // positive due to the pre-instrumentation inline.
 static cl::opt<bool> NoPGOWarnMismatchComdat("no-pgo-warn-mismatch-comdat",
                                              cl::init(true), cl::Hidden);
 
 // Command line option to enable/disable select instruction instrumentation.
 static cl::opt<bool> PGOInstrSelect("pgo-instr-select", cl::init(true),
                                     cl::Hidden);
+
+// Command line option to enable/disable memop intrinsic calls..
+static cl::opt<bool> PGOInstrMemOP("pgo-instr-memop", cl::init(true),
+                                    cl::Hidden);
+
 namespace {
 
 /// The select instruction visitor plays three roles specified
 /// by the mode. In \c VM_counting mode, it simply counts the number of
 /// select instructions. In \c VM_instrument mode, it inserts code to count
 /// the number times TrueValue of select is taken. In \c VM_annotate mode,
 /// it reads the profile data and annotate the select instruction with metadata.
-enum VisitMode { VM_counting, VM_instrument, VM_annotate };
+enum VisitMode { VM_counting, VM_instrument, VM_annotate, VM_optimize };
 class PGOUseFunc;
 
 /// Instruction Visitor class to visit select instructions.
 struct SelectInstVisitor : public InstVisitor<SelectInstVisitor> {
   Function &F;
   unsigned NSIs = 0;             // Number of select instructions instrumented.
   VisitMode Mode = VM_counting;  // Visiting mode.
   unsigned *CurCtrIdx = nullptr; // Pointer to current counter index.
struct SelectInstVisitor : public InstVisitor<SelectInstVisitor> {
 
   void instrumentOneSelectInst(SelectInst &SI);
   void annotateOneSelectInst(SelectInst &SI);
   // Visit \p SI instruction and perform tasks according to visit mode.
   void visitSelectInst(SelectInst &SI);
   unsigned getNumOfSelectInsts() const { return NSIs; }
 };
 
+/// Instruction Visitor class to visit memory intrinsic calls.
+struct MemIntrinsicVisitor : public InstVisitor<MemIntrinsicVisitor> {
+  Function &F;
+  unsigned NMemIs = 0;           // Number of memIntrinsics instrumented.
+  VisitMode Mode = VM_counting;  // Visiting mode.
+  unsigned CurCtrId = 0;         // Current counter index.
+  unsigned TotalNumCtrs = 0;     // Total number of counters
+  GlobalVariable *FuncNameVar = nullptr;
+  uint64_t FuncHash = 0;
+  PGOUseFunc *UseFunc = nullptr;
+
+  MemIntrinsicVisitor(Function &Func) : F(Func) {}
+
+  void countMemIntrinsics(Function &Func) {
+    NMemIs = 0;
+    Mode = VM_counting;
+    visit(Func);
+  }
+  void instrumentMemIntrinsics(Function &Func, unsigned TotalNC,
+                               GlobalVariable *FNV, uint64_t FHash) {
+    Mode = VM_instrument;
+    TotalNumCtrs = TotalNC;
+    FuncHash = FHash;
+    FuncNameVar = FNV;
+    visit(Func);
+  }
+
+  // Visit the IR stream and annotate all mem intrinsic call instructions.
+  void optimizeMemIntrinsics(Function &Func, PGOUseFunc *UF);
+  void instrumentOneMemIntrinsic(MemIntrinsic &MI);
+  void optimizeOneMemIntrinsic(MemIntrinsic &MI);
+  // Visit \p MI instruction and perform tasks according to visit mode.
+  void visitMemIntrinsic(MemIntrinsic &SI);
+  unsigned getNumOfMemIntrinsics() const { return NMemIs; }
+};
+
 class PGOInstrumentationGenLegacyPass : public ModulePass {
 public:
   static char ID;
 
   PGOInstrumentationGenLegacyPass() : ModulePass(ID) {
     initializePGOInstrumentationGenLegacyPassPass(
         *PassRegistry::getPassRegistry());
   }
private:
   void computeCFGHash();
   void renameComdatFunction();
   // A map that stores the Comdat group in function F.
   std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers;
 
 public:
   std::vector<Instruction *> IndirectCallSites;
   SelectInstVisitor SIVisitor;
+  MemIntrinsicVisitor MIVisitor;
   std::string FuncName;
   GlobalVariable *FuncNameVar;
   // CFG hash value for this function.
   uint64_t FunctionHash;
 
   // The Minimum Spanning Tree of function CFG.
   CFGMST<Edge, BBInfo> MST;
 
MST.dumpEdges(dbgs(), Twine("Dump Function ") + FuncName + " Hash: " +
                               Twine(FunctionHash) + "\t" + Str);
   }
 
   FuncPGOInstrumentation(
       Function &Func,
       std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers,
       bool CreateGlobalVar = false, BranchProbabilityInfo *BPI = nullptr,
       BlockFrequencyInfo *BFI = nullptr)
-      : F(Func), ComdatMembers(ComdatMembers), SIVisitor(Func), FunctionHash(0),
-        MST(F, BPI, BFI) {
+      : F(Func), ComdatMembers(ComdatMembers), SIVisitor(Func),
+        MIVisitor(Func), FunctionHash(0), MST(F, BPI, BFI) {
 
     // This should be done before CFG hash computation.
     SIVisitor.countSelects(Func);
+    MIVisitor.countMemIntrinsics(Func);
     NumOfPGOSelectInsts += SIVisitor.getNumOfSelectInsts();
+    NumOfPGOMemIntrinsics += MIVisitor.getNumOfMemIntrinsics();
     IndirectCallSites = findIndirectCallSites(Func);
 
     FuncName = getPGOFuncName(F);
     computeCFGHash();
     if (ComdatMembers.size())
       renameComdatFunction();
     DEBUG(dumpInfo("after CFGMST"));
 
Builder.CreateCall(
         Intrinsic::getDeclaration(M, Intrinsic::instrprof_value_profile),
         {llvm::ConstantExpr::getBitCast(FuncInfo.FuncNameVar, I8PtrTy),
          Builder.getInt64(FuncInfo.FunctionHash),
          Builder.CreatePtrToInt(Callee, Builder.getInt64Ty()),
          Builder.getInt32(llvm::InstrProfValueKind::IPVK_IndirectCallTarget),
          Builder.getInt32(NumIndirectCallSites++)});
   }
   NumOfPGOICall += NumIndirectCallSites;
+
+  // Now instrument memop instrinsic calls:
+  FuncInfo.MIVisitor.instrumentMemIntrinsics(F, NumCounters,
+                                             FuncInfo.FuncNameVar,
+                                             FuncInfo.FunctionHash);
 }
 
 // This class represents a CFG edge in profile use compilation.
 struct PGOUseEdge : public PGOEdge {
   bool CountValid;
   uint64_t CountValue;
   PGOUseEdge(const BasicBlock *Src, const BasicBlock *Dest, unsigned W = 1)
       : PGOEdge(Src, Dest, W), CountValid(false), CountValue(0) {}
static void setProfMetadata(Module *M, Instruction *TI,
   MDBuilder MDB(M->getContext());
   assert(MaxCount > 0 && "Bad max count");
   uint64_t Scale = calculateCountScale(MaxCount);
   SmallVector<unsigned, 4> Weights;
   for (const auto &ECI : EdgeCounts)
     Weights.push_back(scaleBranchCount(ECI, Scale));
 
   DEBUG(dbgs() << "Weight is: ";
-        for (const auto &W : Weights) { dbgs() << W << " "; } 
+        for (const auto &W : Weights) { dbgs() << W << " "; }
         dbgs() << "\n";);
   TI->setMetadata(llvm::LLVMContext::MD_prof, MDB.createBranchWeights(Weights));
 }
 
 // Assign the scaled count values to the BB with multiple out edges.
 void PGOUseFunc::setBranchWeights() {
   // Generate MD_prof metadata for every branch instruction.
   DEBUG(dbgs() << "\nSetting branch weights.\n");
void SelectInstVisitor::visitSelectInst(SelectInst &SI) {
   case VM_counting:
     return;
   case VM_instrument:
     instrumentOneSelectInst(SI);
     return;
   case VM_annotate:
     annotateOneSelectInst(SI);
     return;
+  default:
+    break;
   }
 
   llvm_unreachable("Unknown visiting mode");
 }
 
+void MemIntrinsicVisitor::instrumentOneMemIntrinsic(MemIntrinsic &MI) {
+  Module *M = F.getParent();
+  IRBuilder<> Builder(&MI);
+  Type *Int64Ty = Builder.getInt64Ty();
+  Type *I8PtrTy = Builder.getInt8PtrTy();
+  Value *Length = MI.getLength();
+  assert(!dyn_cast<ConstantInt>(Length));
+  Builder.CreateCall(
+      Intrinsic::getDeclaration(M, Intrinsic::instrprof_value_range_profile),
+      {llvm::ConstantExpr::getBitCast(FuncNameVar, I8PtrTy),
+       Builder.getInt64(FuncHash), Builder.CreatePtrToInt(Length, Int64Ty),
+       Builder.getInt32(llvm::InstrProfValueKind::IPVK_MemOPSize),
+       Builder.getInt32(CurCtrId)});
+  ++CurCtrId;
+}
+
+void MemIntrinsicVisitor::visitMemIntrinsic(MemIntrinsic &MI) {
+  if (!PGOInstrMemOP)
+    return;
+  Value *Length = MI.getLength();
+  // Not instrument constant length calls.
+  if (dyn_cast<ConstantInt>(Length))
+    return;
+
+  NMemIs++;
+  switch (Mode) {
+  case VM_counting:
+    return;
+  case VM_instrument:
+    instrumentOneMemIntrinsic(MI);
+    return;
+  default:
+    break;
+  }
+  llvm_unreachable("Unknown visiting mode");
+}
+
 // Traverse all the indirect callsites and annotate the instructions.
 void PGOUseFunc::annotateIndirectCallSites() {
   if (DisableValueProfiling)
     return;
 
   // Create the PGOFuncName meta data.
   createPGOFuncNameMetadata(F, FuncInfo.FuncName);
 

 // aware this is an ir_level profile so it can set the version flag.
 static void createIRLevelProfileFlagVariable(Module &M) {
   Type *IntTy64 = Type::getInt64Ty(M.getContext());
   uint64_t ProfileVersion = (INSTR_PROF_RAW_VERSION | VARIANT_MASK_IR_PROF);
   auto IRLevelVersionVariable = new GlobalVariable(
       M, IntTy64, true, GlobalVariable::ExternalLinkage,
       Constant::getIntegerValue(IntTy64, APInt(64, ProfileVersion)),
       INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR));
-  IRLevelVersionVariable->setVisibility(GlobalValue::DefaultVisibility);
-  Triple TT(M.getTargetTriple());
-  if (!TT.supportsCOMDAT())
-    IRLevelVersionVariable->setLinkage(GlobalValue::WeakAnyLinkage);
-  else
-    IRLevelVersionVariable->setComdat(M.getOrInsertComdat(
-        StringRef(INSTR_PROF_QUOTE(INSTR_PROF_RAW_VERSION_VAR))));
+  setRuntimeVariableProperty(M, IRLevelVersionVariable);
 }
 
 // Collect the set of members for each Comdat in module M and store
 // in ComdatMembers.
 static void collectComdatMembers(
     Module &M,
     std::unordered_multimap<Comdat *, GlobalValue *> &ComdatMembers) {
   if (!DoComdatRenaming)
for (auto &F : M) {
     auto *BPI = LookupBPI(F);
     auto *BFI = LookupBFI(F);
     PGOUseFunc Func(F, &M, ComdatMembers, BPI, BFI);
     if (!Func.readCounters(PGOReader.get()))
       continue;
     Func.populateCounters();
     Func.setBranchWeights();
     Func.annotateIndirectCallSites();
+
     PGOUseFunc::FuncFreqAttr FreqAttr = Func.getFuncFreqAttr();
     if (FreqAttr == PGOUseFunc::FFA_Cold)
       ColdFunctions.push_back(&F);
     else if (FreqAttr == PGOUseFunc::FFA_Hot)
       HotFunctions.push_back(&F);
   }
   M.setProfileSummary(PGOReader->getSummary().getMD(M.getContext()));
   // Set function hotness attribute from the profile.

test/Instrumentation/InstrProfiling/PR23499.ll

 ; RUN: opt < %s -mtriple=x86_64-pc-win32-coff -passes=instrprof -S | FileCheck %s --check-prefix=COFF
 
 $_Z3barIvEvv = comdat any
 
 @__profn__Z3barIvEvv = linkonce_odr hidden constant [11 x i8] c"_Z3barIvEvv", align 1
 
 ; CHECK: @__profn__Z3barIvEvv = private constant [11 x i8] c"_Z3barIvEvv", align 1
 ; CHECK: @__profc__Z3barIvEvv = linkonce_odr hidden global [1 x i64] zeroinitializer, section "{{.*}}__llvm_prf_cnts", comdat($__profv__Z3barIvEvv), align 8
-; CHECK: @__profd__Z3barIvEvv = linkonce_odr hidden global { i64, i64, i64*, i8*, i8*, i32, [1 x i16] } { i64 4947693190065689389, i64 0, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__Z3barIvEvv, i32 0, i32 0), i8*{{.*}}, i8* null, i32 1, [1 x i16] zeroinitializer }, section "{{.*}}__llvm_prf_data{{.*}}", comdat($__profv__Z3barIvEvv), align 8
+; CHECK: @__profd__Z3barIvEvv = linkonce_odr hidden global { i64, i64, i64*, i8*, i8*, i32, [2 x i16] } { i64 4947693190065689389, i64 0, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__Z3barIvEvv, i32 0, i32 0), i8*{{.*}}, i8* null, i32 1, [2 x i16] zeroinitializer }, section "{{.*}}__llvm_prf_data{{.*}}", comdat($__profv__Z3barIvEvv), align 8
 ; CHECK: @__llvm_prf_nm = private constant [{{.*}} x i8] c"{{.*}}", section "{{.*}}__llvm_prf_names"
 
 
 ; COFF: @__profn__Z3barIvEvv = private constant [11 x i8] c"_Z3barIvEvv", align 1
 ; COFF: @__profc__Z3barIvEvv = linkonce_odr hidden global [1 x i64] zeroinitializer, section "{{.*}}__llvm_prf_cnts", comdat, align 8
-; COFF: @__profd__Z3barIvEvv = linkonce_odr hidden global { i64, i64, i64*, i8*, i8*, i32, [1 x i16] } { i64 4947693190065689389, i64 0, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__Z3barIvEvv, i32 0, i32 0), i8*{{.*}}, i8* null, i32 1, [1 x i16] zeroinitializer }, section "{{.*}}__llvm_prf_data{{.*}}", comdat($__profc__Z3barIvEvv), align 8
+; COFF: @__profd__Z3barIvEvv = linkonce_odr hidden global { i64, i64, i64*, i8*, i8*, i32, [2 x i16] } { i64 4947693190065689389, i64 0, i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__Z3barIvEvv, i32 0, i32 0), i8*{{.*}}, i8* null, i32 1, [2 x i16] zeroinitializer }, section "{{.*}}__llvm_prf_data{{.*}}", comdat($__profc__Z3barIvEvv), align 8
 
 
 declare void @llvm.instrprof.increment(i8*, i64, i32, i32) #1
 
 define linkonce_odr void @_Z3barIvEvv() comdat {
 entry:
   call void @llvm.instrprof.increment(i8* getelementptr inbounds ([11 x i8], [11 x i8]* @__profn__Z3barIvEvv, i32 0, i32 0), i64 0, i32 1, i32 0)
   ret void
 }

test/Instrumentation/InstrProfiling/icall.ll

 attributes #0 = { nounwind }
 
 ; STATIC: @__profvp_foo
 ; STATIC: @__llvm_prf_vnodes
 
 ; DYN-NOT: @__profvp_foo
 ; DYN-NOT: @__llvm_prf_vnodes
 
-; STATIC: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [1 x i16] }* @__profd_foo to i8*), i32 0)
-; STATIC-EXT: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [1 x i16] }* @__profd_foo to i8*), i32 zeroext 0)
-; STATIC-SEXT: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [1 x i16] }* @__profd_foo to i8*), i32 signext 0)
+; STATIC: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [2 x i16] }* @__profd_foo to i8*), i32 0)
+; STATIC-EXT: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [2 x i16] }* @__profd_foo to i8*), i32 zeroext 0)
+; STATIC-SEXT: call void @__llvm_profile_instrument_target(i64 %3, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [2 x i16] }* @__profd_foo to i8*), i32 signext 0)
 
 ; STATIC: declare void @__llvm_profile_instrument_target(i64, i8*, i32)
 ; STATIC-EXT: declare void @__llvm_profile_instrument_target(i64, i8*, i32 zeroext)
 ; STATIC-SEXT: declare void @__llvm_profile_instrument_target(i64, i8*, i32 signext)

test/Transforms/PGOProfile/comdat_internal.ll

 ; RUN: opt < %s -pgo-instr-gen -instrprof -S | FileCheck %s
 ; RUN: opt < %s -passes=pgo-instr-gen,instrprof -S | FileCheck %s
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 $foo = comdat any
 ; CHECK: $foo = comdat any
 
 ; CHECK: $__llvm_profile_raw_version = comdat any
 ; CHECK: $__profv__stdin__foo.[[FOO_HASH:[0-9]+]] = comdat any
 
 @bar = global i32 ()* @foo, align 8
 
 ; CHECK: @__llvm_profile_raw_version = constant i64 {{[0-9]+}}, comdat
 ; CHECK: @__profn__stdin__foo = private constant [11 x i8] c"<stdin>:foo"
 ; CHECK: @__profc__stdin__foo.[[FOO_HASH]] = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat($__profv__stdin__foo.[[FOO_HASH]]), align 8
-; CHECK: @__profd__stdin__foo.[[FOO_HASH]] = private global { i64, i64, i64*, i8*, i8*, i32, [1 x i16] } { i64 -5640069336071256030, i64 [[FOO_HASH]], i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__stdin__foo.[[FOO_HASH]], i32 0, i32 0), i8* null
+; CHECK: @__profd__stdin__foo.[[FOO_HASH]] = private global { i64, i64, i64*, i8*, i8*, i32, [2 x i16] } { i64 -5640069336071256030, i64 [[FOO_HASH]], i64* getelementptr inbounds ([1 x i64], [1 x i64]* @__profc__stdin__foo.[[FOO_HASH]], i32 0, i32 0), i8* null
 ; CHECK-NOT: bitcast (i32 ()* @foo to i8*)
-; CHECK-SAME: , i8* null, i32 1, [1 x i16] zeroinitializer }, section "__llvm_prf_data", comdat($__profv__stdin__foo.[[FOO_HASH]]), align 8
+; CHECK-SAME: , i8* null, i32 1, [2 x i16] zeroinitializer }, section "__llvm_prf_data", comdat($__profv__stdin__foo.[[FOO_HASH]]), align 8
 ; CHECK: @__llvm_prf_nm
 ; CHECK: @llvm.used
 
 define internal i32 @foo() comdat {
 entry:
   ret i32 1
 }

test/Transforms/PGOProfile/memcpy.ll

+; RUN: opt < %s -pgo-instr-gen -instrprof -S | FileCheck %s
+; RUN: opt <%s -passes=pgo-instr-gen,instrprof -S | FileCheck %s
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @foo(i8* %dst, i8* %src, i32* %a, i32 %n) {
+entry:
+  br label %for.cond
+
+for.cond:
+  %i.0 = phi i32 [ 0, %entry ], [ %add, %for.cond1 ]
+  %cmp = icmp slt i32 %i.0, %n
+  br i1 %cmp, label %for.cond1, label %for.end6
+
+for.cond1:
+  %j.0 = phi i32 [ %inc, %for.body3 ], [ 0, %for.cond ]
+  %idx.ext = sext i32 %i.0 to i64
+  %add.ptr = getelementptr inbounds i32, i32* %a, i64 %idx.ext
+  %0 = load i32, i32* %add.ptr, align 4
+  %cmp2 = icmp slt i32 %j.0, %0
+  %add = add nsw i32 %i.0, 1
+  br i1 %cmp2, label %for.body3, label %for.cond
+
+for.body3:
+  %conv = sext i32 %add to i64
+; CHECK: call void @__llvm_profile_instrument_range(i64 %conv, i8* bitcast ({ i64, i64, i64*, i8*, i8*, i32, [2 x i16] }* @__profd_foo to i8*), i32 0)
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %dst, i8* %src, i64 %conv, i32 1, i1 false)
+  %inc = add nsw i32 %j.0, 1
+  br label %for.cond1
+
+for.end6:
+  ret void
+}
+
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i32, i1)

tools/llvm-profdata/llvm-profdata.cpp

if (ProfileKind == instr)
     mergeInstrProfile(WeightedInputs, OutputFilename, OutputFormat,
                       OutputSparse, NumThreads);
   else
     mergeSampleProfile(WeightedInputs, OutputFilename, OutputFormat);
 
   return 0;
 }
 
 static int showInstrProfile(const std::string &Filename, bool ShowCounts,
                             bool ShowIndirectCallTargets,
+                            bool ShowMemOPSizes,
                             bool ShowDetailedSummary,
                             std::vector<uint32_t> DetailedSummaryCutoffs,
                             bool ShowAllFunctions,
                             const std::string &ShowFunction, bool TextFormat,
                             raw_fd_ostream &OS) {
   auto ReaderOrErr = InstrProfReader::create(Filename);
   std::vector<uint32_t> Cutoffs = std::move(DetailedSummaryCutoffs);
   if (ShowDetailedSummary && Cutoffs.empty()) {
     Cutoffs = {800000, 900000, 950000, 990000, 999000, 999900, 999990};
   }
   InstrProfSummaryBuilder Builder(std::move(Cutoffs));
   if (Error E = ReaderOrErr.takeError())
     exitWithError(std::move(E), Filename);
 
   auto Reader = std::move(ReaderOrErr.get());
   bool IsIRInstr = Reader->isIRLevelProfile();
   size_t ShownFunctions = 0;
   uint64_t TotalNumValueSites = 0;
   uint64_t TotalNumValueSitesWithValueProfile = 0;
   uint64_t TotalNumValues = 0;
   std::vector<unsigned> ICHistogram;
   for (const auto &Func : *Reader) {
     bool Show =
         ShowAllFunctions || (!ShowFunction.empty() &&
                              Func.Name.find(ShowFunction) != Func.Name.npos);
 
     bool doTextFormatDump = (Show && ShowCounts && TextFormat);
 
     if (doTextFormatDump) {
       InstrProfSymtab &Symtab = Reader->getSymtab();
       InstrProfWriter::writeRecordInText(Func, Symtab, OS);
       continue;
     }
 
     assert(Func.Counts.size() > 0 && "function missing entry counter");
     Builder.addRecord(Func);
 
     if (Show) {
 
       if (!ShownFunctions)
         OS << "Counters:\n";
 
       ++ShownFunctions;
 
       OS << "  " << Func.Name << ":\n"
          << "    Hash: " << format("0x%016" PRIx64, Func.Hash) << "\n"
          << "    Counters: " << Func.Counts.size() << "\n";
       if (!IsIRInstr)
         OS << "    Function count: " << Func.Counts[0] << "\n";
 
       if (ShowIndirectCallTargets)
         OS << "    Indirect Call Site Count: "
            << Func.getNumValueSites(IPVK_IndirectCallTarget) << "\n";
 
+      uint32_t MemOPSize = Func.getNumValueSites(IPVK_MemOPSize);
+      if (ShowMemOPSizes && MemOPSize > 0)
+        OS << "    Memory Intrinsics Size: " << MemOPSize << "\n";
+
       if (ShowCounts) {
         OS << "    Block counts: [";
         size_t Start = (IsIRInstr ? 0 : 1);
         for (size_t I = Start, E = Func.Counts.size(); I < E; ++I) {
           OS << (I == Start ? "" : ", ") << Func.Counts[I];
         }
         OS << "]\n";
       }
if (Show) {
           }
           for (uint32_t V = 0; V < NV; V++) {
             OS << "\t[ " << I << ", ";
             OS << Symtab.getFuncName(VD[V].Value) << ", " << VD[V].Count
                << " ]\n";
           }
         }
       }
+      if (ShowMemOPSizes && MemOPSize > 0) {
+        uint32_t NS = Func.getNumValueSites(IPVK_MemOPSize);
+        OS << "    Memory Instrinsic Size Results:\n";
+        for (size_t I = 0; I < NS; ++I) {
+          uint32_t NV = Func.getNumValueDataForSite(IPVK_MemOPSize, I);
+          std::unique_ptr<InstrProfValueData[]> VD =
+              Func.getValueForSite(IPVK_MemOPSize, I);
+          TotalNumValues += NV;
+          for (uint32_t V = 0; V < NV; V++) {
+            OS << "\t[ " << I << ", ";
+            int64_t Size = VD[V].Value;
+            if (Size < 0)
+              OS << ">=" << -Size;
+            else
+              OS << Size;
+            OS << ", " << VD[V].Count
+               << " ]\n";
+          }
+        }
+      }
     }
   }
   if (Reader->hasError())
     exitWithError(Reader->getError(), Filename);
 
   if (ShowCounts && TextFormat)
     return 0;
   std::unique_ptr<ProfileSummary> PS(Builder.getSummary());
static int show_main(int argc, const char *argv[]) {
   cl::opt<bool> ShowCounts("counts", cl::init(false),
                            cl::desc("Show counter values for shown functions"));
   cl::opt<bool> TextFormat(
       "text", cl::init(false),
       cl::desc("Show instr profile data in text dump format"));
   cl::opt<bool> ShowIndirectCallTargets(
       "ic-targets", cl::init(false),
       cl::desc("Show indirect call site target values for shown functions"));
+  cl::opt<bool> ShowMemOPSizes(
+      "memop-sizes", cl::init(false),
+      cl::desc("Show the sizes of the memory intrinsic calls"));
   cl::opt<bool> ShowDetailedSummary("detailed-summary", cl::init(false),
                                     cl::desc("Show detailed profile summary"));
   cl::list<uint32_t> DetailedSummaryCutoffs(
       cl::CommaSeparated, "detailed-summary-cutoffs",
       cl::desc(
           "Cutoff percentages (times 10000) for generating detailed summary"),
       cl::value_desc("800000,901000,999999"));
   cl::opt<bool> ShowAllFunctions("all-functions", cl::init(false),
static int show_main(int argc, const char *argv[]) {
 
   if (ShowAllFunctions && !ShowFunction.empty())
     errs() << "warning: -function argument ignored: showing all functions\n";
 
   std::vector<uint32_t> Cutoffs(DetailedSummaryCutoffs.begin(),
                                 DetailedSummaryCutoffs.end());
   if (ProfileKind == instr)
     return showInstrProfile(Filename, ShowCounts, ShowIndirectCallTargets,
+                            ShowMemOPSizes,
                             ShowDetailedSummary, DetailedSummaryCutoffs,
                             ShowAllFunctions, ShowFunction, TextFormat, OS);
   else
     return showSampleProfile(Filename, ShowCounts, ShowAllFunctions,
                              ShowFunction, OS);
 }
 
 int main(int argc, const char *argv[]) {