[XRay][llvm+clang] Consolidate attribute list files

[dberris]

Summary

This change consolidates the always/never lists that may be provided to
clang to externally control which functions should be XRay instrumented
by imbuing attributes. The files follow the same format as defined in
https://clang.llvm.org/docs/SanitizerSpecialCaseList.html for the
sanitizer blacklist.

We also deprecate the existing -fxray-instrument-always= and
-fxray-instrument-never= flags, in favour of -fxray-attr-list=.

This fixes http://llvm.org/PR34721.

[vlad.tsyrklevich]

llvm/docs/XRayExample.rst

196

combined*

[dberris]

• fixup: fix typo

clang/include/clang/Basic/LangOptions.h

public:
 
   /// \brief Paths to blacklist files specifying which objects
   /// (files, functions, variables) should not be instrumented.
   std::vector<std::string> SanitizerBlacklistFiles;
 
   /// \brief Paths to the XRay "always instrument" files specifying which
   /// objects (files, functions, variables) should be imbued with the XRay
   /// "always instrument" attribute.
+  /// WARNING: This is a deprecated field and will go away in the future.
   std::vector<std::string> XRayAlwaysInstrumentFiles;
 
   /// \brief Paths to the XRay "never instrument" files specifying which
   /// objects (files, functions, variables) should be imbued with the XRay
   /// "never instrument" attribute.
+  /// WARNING: This is a deprecated field and will go away in the future.
   std::vector<std::string> XRayNeverInstrumentFiles;
 
+  /// \brief Paths to the XRay attribute list files, specifying which objects
+  /// (files, functions, variables) should be imbued with the appropriate XRay
+  /// attribute(s).
+  std::vector<std::string> XRayAttrListFiles;
+
   clang::ObjCRuntime ObjCRuntime;
 
   std::string ObjCConstantStringClass;
   
   /// \brief The name of the handler function to be called when -ftrapv is
   /// specified.
   ///
   /// If none is specified, abort (GCC-compatible behaviour).

clang/include/clang/Basic/XRayLists.h

 #include "llvm/Support/SpecialCaseList.h"
 #include <memory>
 
 namespace clang {
 
 class XRayFunctionFilter {
   std::unique_ptr<llvm::SpecialCaseList> AlwaysInstrument;
   std::unique_ptr<llvm::SpecialCaseList> NeverInstrument;
+  std::unique_ptr<llvm::SpecialCaseList> AttrList;
   SourceManager &SM;
 
 public:
   XRayFunctionFilter(ArrayRef<std::string> AlwaysInstrumentPaths,
                      ArrayRef<std::string> NeverInstrumentPaths,
-                     SourceManager &SM);
+                     ArrayRef<std::string> AttrListPaths, SourceManager &SM);
 
   enum class ImbueAttribute {
     NONE,
     ALWAYS,
     NEVER,
     ALWAYS_ARG1,
   };
 

clang/include/clang/Driver/Options.td

def fxray_instruction_threshold_EQ :
   HelpText<"Sets the minimum function size to instrument with XRay">;
 def fxray_instruction_threshold_ :
   JoinedOrSeparate<["-"], "fxray-instruction-threshold">,
   Group<f_Group>, Flags<[CC1Option]>;
 
 def fxray_always_instrument :
   JoinedOrSeparate<["-"], "fxray-always-instrument=">,
   Group<f_Group>, Flags<[CC1Option]>,
-  HelpText<"Filename defining the whitelist for imbuing the 'always instrument' XRay attribute.">;
+  HelpText<"DEPRECATED: Filename defining the whitelist for imbuing the 'always instrument' XRay attribute.">;
 def fxray_never_instrument :
   JoinedOrSeparate<["-"], "fxray-never-instrument=">,
   Group<f_Group>, Flags<[CC1Option]>,
-  HelpText<"Filename defining the whitelist for imbuing the 'never instrument' XRay attribute.">;
+  HelpText<"DEPRECATED: Filename defining the whitelist for imbuing the 'never instrument' XRay attribute.">;
+def fxray_attr_list :
+  JoinedOrSeparate<["-"], "fxray-attr-list=">,
+  Group<f_Group>, Flags<[CC1Option]>,
+  HelpText<"Filename defining the list of functions/types for imbuing XRay attributes.">;
 
 def fxray_always_emit_customevents : Flag<["-"], "fxray-always-emit-customevents">, Group<f_Group>,
   Flags<[CC1Option]>,
   HelpText<"Determine whether to always emit __xray_customevent(...) calls even if the function it appears in is not always instrumented.">;
 def fnoxray_always_emit_customevents : Flag<["-"], "fno-xray-always-emit-customevents">, Group<f_Group>,
   Flags<[CC1Option]>;
 
 def fxray_link_deps : Flag<["-"], "fxray-link-deps">, Group<f_Group>,

clang/include/clang/Driver/XRayArgs.h

 namespace clang {
 namespace driver {
 
 class ToolChain;
 
 class XRayArgs {
   std::vector<std::string> AlwaysInstrumentFiles;
   std::vector<std::string> NeverInstrumentFiles;
+  std::vector<std::string> AttrListFiles;
   std::vector<std::string> ExtraDeps;
   bool XRayInstrument = false;
   int InstructionThreshold = 200;
   bool XRayAlwaysEmitCustomEvents = false;
   bool XRayRT = true;
 
 public:
   /// Parses the XRay arguments from an argument list.

clang/lib/AST/ASTContext.cpp

 ASTContext::ASTContext(LangOptions &LOpts, SourceManager &SM,
                        IdentifierTable &idents, SelectorTable &sels,
                        Builtin::Context &builtins)
     : FunctionProtoTypes(this_()), TemplateSpecializationTypes(this_()),
       DependentTemplateSpecializationTypes(this_()),
       SubstTemplateTemplateParmPacks(this_()), SourceMgr(SM), LangOpts(LOpts),
       SanitizerBL(new SanitizerBlacklist(LangOpts.SanitizerBlacklistFiles, SM)),
       XRayFilter(new XRayFunctionFilter(LangOpts.XRayAlwaysInstrumentFiles,
-                                        LangOpts.XRayNeverInstrumentFiles, SM)),
+                                        LangOpts.XRayNeverInstrumentFiles,
+                                        LangOpts.XRayAttrListFiles, SM)),
       PrintingPolicy(LOpts), Idents(idents), Selectors(sels),
       BuiltinInfo(builtins), DeclarationNames(*this), Comments(SM),
       CommentCommandTraits(BumpAlloc, LOpts.CommentOpts), LastSDM(nullptr, 0) {
   TUDecl = TranslationUnitDecl::Create(*this);
 }
 
 ASTContext::~ASTContext() {
   ReleaseParentMapEntries();

clang/lib/Basic/XRayLists.cpp

 //
 //===----------------------------------------------------------------------===//
 #include "clang/Basic/XRayLists.h"
 
 using namespace clang;
 
 XRayFunctionFilter::XRayFunctionFilter(
     ArrayRef<std::string> AlwaysInstrumentPaths,
-    ArrayRef<std::string> NeverInstrumentPaths, SourceManager &SM)
+    ArrayRef<std::string> NeverInstrumentPaths,
+    ArrayRef<std::string> AttrListPaths, SourceManager &SM)
     : AlwaysInstrument(
           llvm::SpecialCaseList::createOrDie(AlwaysInstrumentPaths)),
       NeverInstrument(llvm::SpecialCaseList::createOrDie(NeverInstrumentPaths)),
-      SM(SM) {}
+      AttrList(llvm::SpecialCaseList::createOrDie(AttrListPaths)), SM(SM) {}
 
 XRayFunctionFilter::ImbueAttribute
 XRayFunctionFilter::shouldImbueFunction(StringRef FunctionName) const {
   // First apply the always instrument list, than if it isn't an "always" see
   // whether it's treated as a "never" instrument function.
+  // TODO: Remove these as they're deprecated; use the AttrList exclusively.
   if (AlwaysInstrument->inSection("xray_always_instrument", "fun", FunctionName,
-                                  "arg1"))
+                                  "arg1") ||
+      AttrList->inSection("always", "fun", FunctionName, "arg1"))
     return ImbueAttribute::ALWAYS_ARG1;
   if (AlwaysInstrument->inSection("xray_always_instrument", "fun",
-                                  FunctionName))
+                                  FunctionName) ||
+      AttrList->inSection("always", "fun", FunctionName))
     return ImbueAttribute::ALWAYS;
-  if (NeverInstrument->inSection("xray_never_instrument", "fun", FunctionName))
+
+  if (NeverInstrument->inSection("xray_never_instrument", "fun",
+                                 FunctionName) ||
+      AttrList->inSection("never", "fun", FunctionName))
     return ImbueAttribute::NEVER;
+
   return ImbueAttribute::NONE;
 }
 
 XRayFunctionFilter::ImbueAttribute
 XRayFunctionFilter::shouldImbueFunctionsInFile(StringRef Filename,
                                                StringRef Category) const {
   if (AlwaysInstrument->inSection("xray_always_instrument", "src", Filename,
-                                  Category))
+                                  Category) ||
+      AttrList->inSection("always", "src", Filename, Category))
     return ImbueAttribute::ALWAYS;
   if (NeverInstrument->inSection("xray_never_instrument", "src", Filename,
-                                 Category))
+                                 Category) ||
+      AttrList->inSection("never", "src", Filename, Category))
     return ImbueAttribute::NEVER;
   return ImbueAttribute::NONE;
 }
 
 XRayFunctionFilter::ImbueAttribute
 XRayFunctionFilter::shouldImbueLocation(SourceLocation Loc,
                                         StringRef Category) const {
   if (!Loc.isValid())
     return ImbueAttribute::NONE;
   return this->shouldImbueFunctionsInFile(SM.getFilename(SM.getFileLoc(Loc)),
                                           Category);
 }

clang/lib/Driver/XRayArgs.cpp

if (Args.hasFlag(options::OPT_fxray_instrument,
     for (const auto &Filename :
          Args.getAllArgValues(options::OPT_fxray_never_instrument)) {
       if (llvm::sys::fs::exists(Filename)) {
         NeverInstrumentFiles.push_back(Filename);
         ExtraDeps.push_back(Filename);
       } else
         D.Diag(clang::diag::err_drv_no_such_file) << Filename;
     }
+
+    for (const auto &Filename :
+         Args.getAllArgValues(options::OPT_fxray_attr_list)) {
+      if (llvm::sys::fs::exists(Filename)) {
+        AttrListFiles.push_back(Filename);
+        ExtraDeps.push_back(Filename);
+      } else
+        D.Diag(clang::diag::err_drv_no_such_file) << Filename;
+    }
   }
 }
 
 void XRayArgs::addArgs(const ToolChain &TC, const ArgList &Args,
                        ArgStringList &CmdArgs, types::ID InputType) const {
   if (!XRayInstrument)
     return;
 
void XRayArgs::addArgs(const ToolChain &TC, const ArgList &Args,
   }
 
   for (const auto &Never : NeverInstrumentFiles) {
     SmallString<64> NeverInstrumentOpt("-fxray-never-instrument=");
     NeverInstrumentOpt += Never;
     CmdArgs.push_back(Args.MakeArgString(NeverInstrumentOpt));
   }
 
+  for (const auto&AttrFile : AttrListFiles) {
+    SmallString<64> AttrListFileOpt("-fxray-attr-list=");
+    AttrListFileOpt += AttrFile;
+    CmdArgs.push_back(Args.MakeArgString(AttrListFileOpt));
+  }
+
   for (const auto &Dep : ExtraDeps) {
     SmallString<64> ExtraDepOpt("-fdepfile-entry=");
     ExtraDepOpt += Dep;
     CmdArgs.push_back(Args.MakeArgString(ExtraDepOpt));
   }
 }

clang/lib/Frontend/CompilerInvocation.cpp

Opts.XRayAlwaysEmitCustomEvents =
       Args.hasFlag(OPT_fxray_always_emit_customevents,
                    OPT_fnoxray_always_emit_customevents, false);
 
   // -fxray-{always,never}-instrument= filenames.
   Opts.XRayAlwaysInstrumentFiles =
       Args.getAllArgValues(OPT_fxray_always_instrument);
   Opts.XRayNeverInstrumentFiles =
       Args.getAllArgValues(OPT_fxray_never_instrument);
+  Opts.XRayAttrListFiles = Args.getAllArgValues(OPT_fxray_attr_list);
 
   // -fallow-editor-placeholders
   Opts.AllowEditorPlaceholders = Args.hasArg(OPT_fallow_editor_placeholders);
 
   if (Arg *A = Args.getLastArg(OPT_fclang_abi_compat_EQ)) {
     Opts.setClangABICompat(LangOptions::ClangABI::Latest);
 
     StringRef Ver = A->getValue();

clang/test/CodeGen/xray-always-instrument.cpp

 // RUN: echo "fun:*foo*" > %t.always-instrument
 // RUN: echo "src:*xray-always-instrument.cpp" >> %t.always-instrument
-// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 -fxray-always-instrument=%t.always-instrument -emit-llvm -o - %s -triple x86_64-unknown-linux-gnu | FileCheck %s
+// RUN: echo "[always]" > %t.xray-attrlist
+// RUN: echo "fun:*foo*" >> %t.xray-attrlist
+// RUN: echo "src:*xray-always-instrument.cpp" >> %t.xray-attrlist
+// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 \
+// RUN:     -fxray-always-instrument=%t.always-instrument -emit-llvm -o - %s \
+// RUN:     -triple x86_64-unknown-linux-gnu | FileCheck %s
+// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 \
+// RUN:     -fxray-attr-list=%t.xray-attrlist -emit-llvm -o - %s \
+// RUN:     -triple x86_64-unknown-linux-gnu | FileCheck %s
 
 void foo() {}
 
 [[clang::xray_never_instrument]] void bar() {}
 
 void baz() {}
 
 // CHECK: define void @_Z3foov() #[[ALWAYSATTR:[0-9]+]] {
 // CHECK: define void @_Z3barv() #[[NEVERATTR:[0-9]+]] {
 // CHECK: define void @_Z3bazv() #[[ALWAYSATTR:[0-9]+]] {
 // CHECK: attributes #[[ALWAYSATTR]] = {{.*}} "function-instrument"="xray-always" {{.*}}
 // CHECK: attributes #[[NEVERATTR]] = {{.*}} "function-instrument"="xray-never" {{.*}}

clang/test/CodeGen/xray-attr-list.cpp

+// RUN: echo "[always]" > %t.xray-attrlist
+// RUN: echo "fun:*always*" >> %t.xray-attrlist
+// RUN: echo "[never]" >> %t.xray-attrlist
+// RUN: echo "fun:*never*" >> %t.xray-attrlist
+// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 \
+// RUN:     -fxray-attr-list=%t.xray-attrlist -emit-llvm -o - %s \
+// RUN:     -triple x86_64-unknown-linux-gnu | FileCheck %s
+
+void always() {}
+void never() {}
+[[clang::xray_never_instrument]] void alwaysNever() {}
+[[clang::xray_always_instrument]] void neverAlways() {}
+
+// CHECK: define void @_Z6alwaysv() #[[ALWAYSATTR:[0-9]+]] {
+// CHECK: define void @_Z5neverv() #[[NEVERATTR:[0-9]+]] {
+// CHECK: define void @_Z11alwaysNeverv() #[[NEVERATTR]] {
+// CHECK: define void @_Z11neverAlwaysv() #[[ALWAYSATTR]] {
+// CHECK: attributes #[[ALWAYSATTR]] = {{.*}} "function-instrument"="xray-always" {{.*}}
+// CHECK: attributes #[[NEVERATTR]] = {{.*}} "function-instrument"="xray-never" {{.*}}

clang/test/CodeGen/xray-imbue-arg1.cpp

 // RUN: echo "fun:*arg1*=arg1" >> %t.always-instrument
-// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 -fxray-always-instrument=%t.always-instrument -emit-llvm -o - %s -triple x86_64-unknown-linux-gnu | FileCheck %s
+// RUN: echo "[always]" > %t.xray-attrlist
+// RUN: echo "fun:*arg1*=arg1" >> %t.xray-attrlist
+// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 \
+// RUN:     -fxray-always-instrument=%t.always-instrument -emit-llvm -o - %s \
+// RUN:     -triple x86_64-unknown-linux-gnu | FileCheck %s
+// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 \
+// RUN:     -fxray-attr-list=%t.xray-attrlist -emit-llvm -o - %s \
+// RUN:     -triple x86_64-unknown-linux-gnu | FileCheck %s
 
 void foo() {}
 
 void arg1(void*) {}
 
 // CHECK: define void @_Z3foov() #[[FOO:[0-9]+]] {
 // CHECK: define void {{.*}}arg1{{.*}} #[[ALWAYSARG1:[0-9]+]] {
 
 // CHECK: attributes #[[FOO]] = {{.*}}
 // CHECK: attributes #[[ALWAYSARG1]] = {{.*}} "function-instrument"="xray-always" {{.*}} "xray-log-args"="1"

clang/test/CodeGen/xray-never-instrument.cpp

+// RUN: echo "fun:*foo*" > %t.never-instrument
+// RUN: echo "src:*xray-never-instrument.cpp" >> %t.never-instrument
+// RUN: echo "[never]" > %t.xray-attrlist
+// RUN: echo "fun:*foo*" >> %t.xray-attrlist
+// RUN: echo "src:*xray-never-instrument.cpp" >> %t.xray-attrlist
+// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 \
+// RUN:     -fxray-never-instrument=%t.never-instrument -emit-llvm -o - %s \
+// RUN:     -triple x86_64-unknown-linux-gnu | FileCheck %s
+// RUN: %clang_cc1 -fxray-instrument -x c++ -std=c++11 \
+// RUN:     -fxray-attr-list=%t.xray-attrlist -emit-llvm -o - %s \
+// RUN:     -triple x86_64-unknown-linux-gnu | FileCheck %s
+
+void foo() {}
+
+[[clang::xray_always_instrument]] void bar() {}
+
+void baz() {}
+
+// CHECK: define void @_Z3foov() #[[NEVERATTR:[0-9]+]] {
+// CHECK: define void @_Z3barv() #[[ALWAYSATTR:[0-9]+]] {
+// CHECK: define void @_Z3bazv() #[[NEVERATTR:[0-9]+]] {
+// CHECK: attributes #[[NEVERATTR]] = {{.*}} "function-instrument"="xray-never" {{.*}}
+// CHECK: attributes #[[ALWAYSATTR]] = {{.*}} "function-instrument"="xray-always" {{.*}}
+

llvm/docs/XRay.rst

 it gets instrumented.
 
 .. code-block:: llvm
 
     define i32 @maybe_instrument() uwtable "xray-instruction-threshold"="2" {
       ; ...
     }
 
+Special Case File
+-----------------
+
+Attributes can be imbued through the use of special case files instead of
+adding them to the original source files. You can use this to mark certain
+functions and classes to be never, always, or instrumented with first-argument
+logging from a file. The file's format is described below:
+
+.. code-block:: bash
+
+    # Comments are supported
+    [always]
+    fun:always_instrument
+    fun:log_arg1=arg1 # Log the first argument for the function
+
+    [never]
+    fun:never_instrument
+
+These files can be provided through the ``-fxray-attr-list=`` flag to clang.
+You may have multiple files loaded through multiple instances of the flag.
+
 XRay Runtime Library
 --------------------
 
 The XRay Runtime Library is part of the compiler-rt project, which implements
 the runtime components that perform the patching and unpatching of inserted
 instrumentation points. When you use ``clang`` to link your binaries and the
 ``-fxray-instrument`` flag, it will automatically link in the XRay runtime.
 

llvm/docs/XRayExample.rst

 The other way is to use configuration files for selecting which functions
 should always be instrumented by the compiler. This gives us a way of ensuring
 that certain functions are either always or never instrumented by not having to
 add the attribute to the source.
 
 To use this feature, you can define one file for the functions to always
 instrument, and another for functions to never instrument. The format of these
 files are exactly the same as the SanitizerLists files that control similar
-things for the sanitizer implementations. For example, we can have two
-different files like below:
+things for the sanitizer implementations. For example:
 
 ::
 
-  # always-instrument.txt
+  # xray-attr-list.txt
   # always instrument functions that match the following filters:
+  [always]
   fun:main
 
-  # never-instrument.txt
   # never instrument functions that match the following filters:
+  [never]
   fun:__cxx_*
 
-Given the above two files we can re-build by providing those two files as
-arguments to clang as ``-fxray-always-instrument=always-instrument.txt`` or
-``-fxray-never-instrument=never-instrument.txt``.
+Given the file above we can re-build by providing it to the
+``-fxray-attr-list=`` flag to clang. You can have multiple files, each defining
+different sets of attribute sets, to be combined into a single list by clang.
 
 The XRay stack tool
 -------------------
 
 Given a trace, and optionally an instrumentation map, the ``llvm-xray stack``
 command can be used to analyze a call stack graph constructed from the function
 call timeline.