Port ObjCMTAction to new option parsing system

[jansvoboda11]

Summary

Merge existing marhsalling info kinds and add some primitives to
express flag options that contribute to a bitfield.

Depends on D82574

[dang]

clang/include/clang/Driver/Options.td

345–347

If this kind of combination of normalizer, value merger and extractor happen again, I will make a mixin that helps set this up. Or do you reckon it is worth doing it now for readability.

[dang]

Denormalizer should also use the extractor

[dang]

Remove the need for the merger and extractor to be macros

[dang]

Ensure that mergeForwardValue can accept default values whose type doesn't quite match the associated keypath.

[dang]

Make mergers use values directly instead of constant references.

[Bigcheese]

lgtm

[jansvoboda11]

Taking control of this revision, as Daniel is no longer involved.

[jansvoboda11]

Rebase on top of D82756.

[jansvoboda11]

WDYT @Bigcheese?

[Bigcheese]

LGTM.

[jpienaar]

This seems to breaking clang-5 builds:

utils/TableGen/CMakeFiles/llvm-tblgen.dir/OptParserEmitter.cpp.o: In function `llvm::EmitOptParser(llvm::RecordKeeper&, llvm::raw_ostream&)':
/var/lib/buildkite-agent/builds/buildkite-69fdf6c495-wt2bd-1/mlir/mlir-core/llvm/utils/TableGen/OptParserEmitter.cpp:(.text._ZN4llvm13EmitOptParserERNS_12RecordKeeperERNS_11raw_ostreamE+0x148a): undefined reference to `MarshallingInfo::MacroName'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

(https://buildkite.com/mlir/mlir-core/builds/9273#ae602c1f-8c21-4ac6-90bb-99c9a3ae473e)

[dexonsmith]

In D82860#2388916, @jpienaar wrote:

This seems to breaking clang-5 builds:

utils/TableGen/CMakeFiles/llvm-tblgen.dir/OptParserEmitter.cpp.o: In function `llvm::EmitOptParser(llvm::RecordKeeper&, llvm::raw_ostream&)':
/var/lib/buildkite-agent/builds/buildkite-69fdf6c495-wt2bd-1/mlir/mlir-core/llvm/utils/TableGen/OptParserEmitter.cpp:(.text._ZN4llvm13EmitOptParserERNS_12RecordKeeperERNS_11raw_ostreamE+0x148a): undefined reference to `MarshallingInfo::MacroName'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

(https://buildkite.com/mlir/mlir-core/builds/9273#ae602c1f-8c21-4ac6-90bb-99c9a3ae473e)

Jan is in Europe and may not see this until tomorrow; if you need this fixed sooner you might want to revert.

[mehdi_amini]

Reverted in f917356f9ce0 ; I suspect a static constexpr in a class missing a definition out of class (required pre-c++17).

[jansvoboda11]

Thank you for the heads-up. It seems like Clang 5 didn't like the usage of static constexpr member via an instance.

clang/include/clang/Driver/Options.td

def gen_cdb_fragment_path: Separate<["-"], "gen-cdb-fragment-path">, InternalDebugOpt,
   HelpText<"Emit a compilation database fragment to the specified directory">;
 
 def _migrate : Flag<["--"], "migrate">, Flags<[NoXarchOption]>,
   HelpText<"Run the migrator">;
 def ccc_objcmt_migrate : Separate<["-"], "ccc-objcmt-migrate">,
   InternalDriverOpt,
   HelpText<"Apply modifications and produces temporary files to migrate to "
    "modern ObjC syntax">;
+
 def objcmt_migrate_literals : Flag<["-"], "objcmt-migrate-literals">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to modern ObjC literals">;
+  HelpText<"Enable migration to modern ObjC literals">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_Literals">;
 def objcmt_migrate_subscripting : Flag<["-"], "objcmt-migrate-subscripting">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to modern ObjC subscripting">;
+  HelpText<"Enable migration to modern ObjC subscripting">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_Subscripting">;
 def objcmt_migrate_property : Flag<["-"], "objcmt-migrate-property">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to modern ObjC property">;
+  HelpText<"Enable migration to modern ObjC property">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_Property">;
 def objcmt_migrate_all : Flag<["-"], "objcmt-migrate-all">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to modern ObjC">;
+  HelpText<"Enable migration to modern ObjC">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_MigrateDecls">;
 def objcmt_migrate_readonly_property : Flag<["-"], "objcmt-migrate-readonly-property">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to modern ObjC readonly property">;
+  HelpText<"Enable migration to modern ObjC readonly property">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_ReadonlyProperty">;
 def objcmt_migrate_readwrite_property : Flag<["-"], "objcmt-migrate-readwrite-property">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to modern ObjC readwrite property">;
+  HelpText<"Enable migration to modern ObjC readwrite property">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_ReadwriteProperty">;
 def objcmt_migrate_property_dot_syntax : Flag<["-"], "objcmt-migrate-property-dot-syntax">, Flags<[CC1Option]>,
-  HelpText<"Enable migration of setter/getter messages to property-dot syntax">;
+  HelpText<"Enable migration of setter/getter messages to property-dot syntax">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_PropertyDotSyntax">;
 def objcmt_migrate_annotation : Flag<["-"], "objcmt-migrate-annotation">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to property and method annotations">;
+  HelpText<"Enable migration to property and method annotations">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_Annotation">;
 def objcmt_migrate_instancetype : Flag<["-"], "objcmt-migrate-instancetype">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to infer instancetype for method result type">;
+  HelpText<"Enable migration to infer instancetype for method result type">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_Instancetype">;
 def objcmt_migrate_nsmacros : Flag<["-"], "objcmt-migrate-ns-macros">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to NS_ENUM/NS_OPTIONS macros">;
+  HelpText<"Enable migration to NS_ENUM/NS_OPTIONS macros">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_NsMacros">;
 def objcmt_migrate_protocol_conformance : Flag<["-"], "objcmt-migrate-protocol-conformance">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to add protocol conformance on classes">;
+  HelpText<"Enable migration to add protocol conformance on classes">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_ProtocolConformance">;
 def objcmt_atomic_property : Flag<["-"], "objcmt-atomic-property">, Flags<[CC1Option]>,
-  HelpText<"Make migration to 'atomic' properties">;
+  HelpText<"Make migration to 'atomic' properties">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_AtomicProperty">;
 def objcmt_returns_innerpointer_property : Flag<["-"], "objcmt-returns-innerpointer-property">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to annotate property with NS_RETURNS_INNER_POINTER">;
+  HelpText<"Enable migration to annotate property with NS_RETURNS_INNER_POINTER">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_ReturnsInnerPointerProperty">;
 def objcmt_ns_nonatomic_iosonly: Flag<["-"], "objcmt-ns-nonatomic-iosonly">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to use NS_NONATOMIC_IOSONLY macro for setting property's 'atomic' attribute">;
+  HelpText<"Enable migration to use NS_NONATOMIC_IOSONLY macro for setting property's 'atomic' attribute">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_NsAtomicIOSOnlyProperty">;
 def objcmt_migrate_designated_init : Flag<["-"], "objcmt-migrate-designated-init">, Flags<[CC1Option]>,
-  HelpText<"Enable migration to infer NS_DESIGNATED_INITIALIZER for initializer methods">;
+  HelpText<"Enable migration to infer NS_DESIGNATED_INITIALIZER for initializer methods">,
+  MarshallingInfoBitfieldFlag<"FrontendOpts.ObjCMTAction", "FrontendOptions::ObjCMT_DesignatedInitializer">;
+
 def objcmt_whitelist_dir_path: Joined<["-"], "objcmt-whitelist-dir-path=">, Flags<[CC1Option]>,
   HelpText<"Only modify files with a filename contained in the provided directory path">;
 // The misspelt "white-list" [sic] alias is due for removal.
 def : Joined<["-"], "objcmt-white-list-dir-path=">, Flags<[CC1Option]>,
     Alias<objcmt_whitelist_dir_path>;
 
 // Make sure all other -ccc- options are rejected.
 def ccc_ : Joined<["-"], "ccc-">, Group<internal_Group>, Flags<[Unsupported]>;

clang/lib/Frontend/CompilerInvocation.cpp

 //===----------------------------------------------------------------------===//
 // Normalizers
 //===----------------------------------------------------------------------===//
 
 #define SIMPLE_ENUM_VALUE_TABLE
 #include "clang/Driver/Options.inc"
 #undef SIMPLE_ENUM_VALUE_TABLE
 
+static llvm::Optional<bool> normalizeSimpleFlag(OptSpecifier Opt,
+                                                unsigned TableIndex,
+                                                const ArgList &Args,
+                                                DiagnosticsEngine &Diags) {
+  if (Args.hasArg(Opt))
+    return true;
+  return None;
+}
+
+template <typename T, T Value>
+static llvm::Optional<T>
+normalizeFlagToValue(OptSpecifier Opt, unsigned TableIndex, const ArgList &Args,
+                     DiagnosticsEngine &Diags) {
+  if (Args.hasArg(Opt))
+    return Value;
+  return None;
+}
+
 static llvm::Optional<unsigned> normalizeSimpleEnum(OptSpecifier Opt,
                                                     unsigned TableIndex,
                                                     const ArgList &Args,
                                                     DiagnosticsEngine &Diags) {
   assert(TableIndex < SimpleEnumValueTablesSize);
   const SimpleEnumValueTable &Table = SimpleEnumValueTables[TableIndex];
 
   auto *Arg = Args.getLastArg(Opt);
   if (!Arg)
     return None;
 
   StringRef ArgValue = Arg->getValue();
   for (int I = 0, E = Table.Size; I != E; ++I)
     if (ArgValue == Table.Table[I].Name)
       return Table.Table[I].Value;
 
   Diags.Report(diag::err_drv_invalid_value)
       << Arg->getAsString(Args) << ArgValue;
   return None;
 }
 
-static const char *denormalizeSimpleEnum(CompilerInvocation::StringAllocator SA,
-                                         unsigned TableIndex, unsigned Value) {
+static void denormalizeSimpleEnum(SmallVectorImpl<const char *> &Args,
+                                  CompilerInvocation::StringAllocator SA,
   assert(TableIndex < SimpleEnumValueTablesSize);
   const SimpleEnumValueTable &Table = SimpleEnumValueTables[TableIndex];
-  for (int I = 0, E = Table.Size; I != E; ++I)
-    if (Value == Table.Table[I].Value)
-      return Table.Table[I].Name;
+  for (int I = 0, E = Table.Size; I != E; ++I) {
+    if (Value == Table.Table[I].Value) {
+      Args.push_back(Table.Table[I].Name);
+      return;
+    }
+  }
 
   llvm_unreachable("The simple enum value was not correctly defined in "
                    "the tablegen option description");
 }
 
-static const char *denormalizeString(CompilerInvocation::StringAllocator SA,
-                                     unsigned TableIndex,
-                                     const std::string &Value) {
-  return SA(Value);
+static void denormalizeString(SmallVectorImpl<const char *> &Args,
+                              CompilerInvocation::StringAllocator SA,
+                              unsigned TableIndex, const std::string &Value) {
+  Args.push_back(SA(Value));
 }
 
 static Optional<std::string> normalizeTriple(OptSpecifier Opt, int TableIndex,
                                              const ArgList &Args,
                                              DiagnosticsEngine &Diags) {
   auto *Arg = Args.getLastArg(Opt);
   if (!Arg)
     return None;
   return llvm::Triple::normalize(Arg->getValue());
 }
 
+template <typename T, typename U>
+static T mergeForwardValue(T KeyPath, U Value) {
+  return Value;
+}
+
+template <typename T, typename U> static T mergeMaskValue(T KeyPath, U Value) {
+  return KeyPath | Value;
+}
+
+template <typename T> static T extractForwardValue(T KeyPath) {
+  return KeyPath;
+}
+
+template <typename T, typename U, U Value>
+static T extractMaskValue(T KeyPath) {
+  return KeyPath & Value;
+}
+
 //===----------------------------------------------------------------------===//
 // Deserialization (from args)
 //===----------------------------------------------------------------------===//
 
 static unsigned getOptimizationLevel(ArgList &Args, InputKind IK,
                                      DiagnosticsEngine &Diags) {
   unsigned DefaultOpt = llvm::CodeGenOpt::None;
   if (IK.getLanguage() == Language::OpenCL && !Args.hasArg(OPT_cl_opt_disable))
static InputKind ParseFrontendArgs(FrontendOptions &Opts, ArgList &Args,
   }
   Opts.MTMigrateDir =
       std::string(Args.getLastArgValue(OPT_mt_migrate_directory));
   Opts.ARCMTMigrateReportOut =
       std::string(Args.getLastArgValue(OPT_arcmt_migrate_report_output));
   Opts.ARCMTMigrateEmitARCErrors
     = Args.hasArg(OPT_arcmt_migrate_emit_arc_errors);
 
-  if (Args.hasArg(OPT_objcmt_migrate_literals))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_Literals;
-  if (Args.hasArg(OPT_objcmt_migrate_subscripting))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_Subscripting;
-  if (Args.hasArg(OPT_objcmt_migrate_property_dot_syntax))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_PropertyDotSyntax;
-  if (Args.hasArg(OPT_objcmt_migrate_property))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_Property;
-  if (Args.hasArg(OPT_objcmt_migrate_readonly_property))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_ReadonlyProperty;
-  if (Args.hasArg(OPT_objcmt_migrate_readwrite_property))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_ReadwriteProperty;
-  if (Args.hasArg(OPT_objcmt_migrate_annotation))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_Annotation;
-  if (Args.hasArg(OPT_objcmt_returns_innerpointer_property))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_ReturnsInnerPointerProperty;
-  if (Args.hasArg(OPT_objcmt_migrate_instancetype))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_Instancetype;
-  if (Args.hasArg(OPT_objcmt_migrate_nsmacros))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_NsMacros;
-  if (Args.hasArg(OPT_objcmt_migrate_protocol_conformance))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_ProtocolConformance;
-  if (Args.hasArg(OPT_objcmt_atomic_property))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_AtomicProperty;
-  if (Args.hasArg(OPT_objcmt_ns_nonatomic_iosonly))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_NsAtomicIOSOnlyProperty;
-  if (Args.hasArg(OPT_objcmt_migrate_designated_init))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_DesignatedInitializer;
-  if (Args.hasArg(OPT_objcmt_migrate_all))
-    Opts.ObjCMTAction |= FrontendOptions::ObjCMT_MigrateDecls;
-
   Opts.ObjCMTWhiteListPath =
       std::string(Args.getLastArgValue(OPT_objcmt_whitelist_dir_path));
 
   if (Opts.ARCMTAction != FrontendOptions::ARCMT_None &&
       Opts.ObjCMTAction != FrontendOptions::ObjCMT_None) {
     Diags.Report(diag::err_drv_argument_not_allowed_with)
       << "ARC migration" << "ObjC migration";
   }
if (Version.tryParse(A->getValue()))
           << A->getAsString(Args) << A->getValue();
     else
       Opts.SDKVersion = Version;
   }
 }
 
 bool CompilerInvocation::parseSimpleArgs(const ArgList &Args,
                                          DiagnosticsEngine &Diags) {
-#define OPTION_WITH_MARSHALLING_FLAG(PREFIX_TYPE, NAME, ID, KIND, GROUP,       \
-                                     ALIAS, ALIASARGS, FLAGS, PARAM, HELPTEXT, \
-                                     METAVAR, VALUES, SPELLING, ALWAYS_EMIT,   \
-                                     KEYPATH, DEFAULT_VALUE, IS_POSITIVE)      \
-  this->KEYPATH = (Args.hasArg(OPT_##ID) && IS_POSITIVE) || (DEFAULT_VALUE);
-
-#define OPTION_WITH_MARSHALLING_STRING(                                        \
+#define OPTION_WITH_MARSHALLING(                                               \
     PREFIX_TYPE, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,        \
     HELPTEXT, METAVAR, VALUES, SPELLING, ALWAYS_EMIT, KEYPATH, DEFAULT_VALUE,  \
-    TYPE, NORMALIZER, DENORMALIZER, TABLE_INDEX)                               \
+    TYPE, NORMALIZER, DENORMALIZER, MERGER, EXTRACTOR, TABLE_INDEX)            \
   {                                                                            \
+    this->KEYPATH = MERGER(this->KEYPATH, DEFAULT_VALUE);                      \
     if (auto MaybeValue = NORMALIZER(OPT_##ID, TABLE_INDEX, Args, Diags))      \
-      this->KEYPATH = static_cast<TYPE>(*MaybeValue);                          \
-    else                                                                       \
-      this->KEYPATH = DEFAULT_VALUE;                                           \
+      this->KEYPATH = MERGER(this->KEYPATH, static_cast<TYPE>(*MaybeValue));   \
   }
 
 #include "clang/Driver/Options.inc"
-#undef OPTION_WITH_MARSHALLING_STRING
-#undef OPTION_WITH_MARSHALLING_FLAG
+#undef OPTION_WITH_MARSHALLING
   return true;
 }
 
 bool CompilerInvocation::CreateFromArgs(CompilerInvocation &Res,
                                         ArrayRef<const char *> CommandLineArgs,
                                         DiagnosticsEngine &Diags,
                                         const char *Argv0) {
   bool Success = true;
#include "clang/Basic/LangOptions.def"
   if (!SanHash.empty())
     code = hash_combine(code, SanHash.Mask);
 
   return llvm::APInt(64, code).toString(36, /*Signed=*/false);
 }
 
 void CompilerInvocation::generateCC1CommandLine(
     SmallVectorImpl<const char *> &Args, StringAllocator SA) const {
-#define OPTION_WITH_MARSHALLING_FLAG(PREFIX_TYPE, NAME, ID, KIND, GROUP,       \
-                                     ALIAS, ALIASARGS, FLAGS, PARAM, HELPTEXT, \
-                                     METAVAR, VALUES, SPELLING, ALWAYS_EMIT,   \
-                                     KEYPATH, DEFAULT_VALUE, IS_POSITIVE)      \
-  if ((FLAGS) & options::CC1Option &&                                          \
-      (ALWAYS_EMIT || this->KEYPATH != (DEFAULT_VALUE)))                       \
-    Args.push_back(SPELLING);
-
-#define OPTION_WITH_MARSHALLING_STRING(                                        \
+#define OPTION_WITH_MARSHALLING(                                               \
     PREFIX_TYPE, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,        \
     HELPTEXT, METAVAR, VALUES, SPELLING, ALWAYS_EMIT, KEYPATH, DEFAULT_VALUE,  \
-    NORMALIZER_RET_TY, NORMALIZER, DENORMALIZER, TABLE_INDEX)                  \
+    TYPE, NORMALIZER, DENORMALIZER, MERGER, EXTRACTOR, TABLE_INDEX)            \
   if (((FLAGS) & options::CC1Option) &&                                        \
-      (ALWAYS_EMIT || this->KEYPATH != DEFAULT_VALUE)) {                       \
+      (ALWAYS_EMIT || EXTRACTOR(this->KEYPATH) != (DEFAULT_VALUE))) {          \
+    if (Option::KIND##Class == Option::FlagClass) {                            \
+      Args.push_back(SPELLING);                                                \
+    }                                                                          \
     if (Option::KIND##Class == Option::SeparateClass) {                        \
       Args.push_back(SPELLING);                                                \
-      Args.push_back(DENORMALIZER(SA, TABLE_INDEX, this->KEYPATH));            \
+      DENORMALIZER(Args, SA, TABLE_INDEX, EXTRACTOR(this->KEYPATH));           \
     }                                                                          \
   }
 
 #include "clang/Driver/Options.inc"
-#undef OPTION_WITH_MARSHALLING_STRING
-#undef OPTION_WITH_MARSHALLING_FLAG
+#undef OPTION_WITH_MARSHALLING
 }
 
 namespace clang {
 
 IntrusiveRefCntPtr<llvm::vfs::FileSystem>
 createVFSFromCompilerInvocation(const CompilerInvocation &CI,
                                 DiagnosticsEngine &Diags) {
   return createVFSFromCompilerInvocation(CI, Diags,

llvm/include/llvm/Option/OptParser.td

class Option<list<string> prefixes, string name, OptionKind kind> {
   string HelpText = ?;
   string MetaVarName = ?;
   string Values = ?;
   code ValuesCode = ?;
   list<OptionFlag> Flags = [];
   OptionGroup Group = ?;
   Option Alias = ?;
   list<string> AliasArgs = [];
-  string MarshallingKind = ?;
   code KeyPath = ?;
   code DefaultValue = ?;
   bit ShouldAlwaysEmit = 0;
-  // Used by the Flag option kind.
-  bit IsPositive = 1;
-  // Used by the String option kind.
   code NormalizerRetTy = ?;
   code NormalizedValuesScope = "";
   code Normalizer = "";
   code Denormalizer = "";
+  code ValueMerger = "mergeForwardValue";
+  code ValueExtractor = "extractForwardValue";
   list<code> NormalizedValues = ?;
 }
 
 // Helpers for defining options.
 
 class Flag<list<string> prefixes, string name>
   : Option<prefixes, name, KIND_FLAG>;
 class Joined<list<string> prefixes, string name>

 class Group<OptionGroup group> { OptionGroup Group = group; }
 class HelpText<string text> { string HelpText = text; }
 class MetaVarName<string name> { string MetaVarName = name; }
 class Values<string value> { string Values = value; }
 class ValuesCode<code valuecode> { code ValuesCode = valuecode; }
 
 // Helpers for defining marshalling information.
 
-class DefaultAnyOf<list<Option> defaults> {
-  code DefaultValue = !foldl("false", defaults, accumulator, option,
-                             !strconcat(accumulator, " || ", !cast<string>(option.KeyPath)));
+class DefaultAnyOf<list<Option> options, string default = "false", string separator = " || "> {
+  code DefaultValue = !foldl(default, options, accumulator, option,
+                             !strconcat(accumulator, separator, !cast<string>(option.KeyPath)));
 }
 
 class MarshallingInfo<code keypath, code defaultvalue> {
   code KeyPath = keypath;
   code DefaultValue = defaultvalue;
 }
+
 class MarshallingInfoString<code keypath, code defaultvalue, code normalizerretty>
   : MarshallingInfo<keypath, defaultvalue> {
-  string MarshallingKind = "string";
   code NormalizerRetTy = normalizerretty;
 }
 
-class MarshallingInfoFlag<code keypath, DefaultAnyOf defaults = DefaultAnyOf<[]>>
+class MarshallingInfoFlag<code keypath, DefaultAnyOf defaults = DefaultAnyOf<[]>, code ty="unsigned">
   : MarshallingInfo<keypath, defaults.DefaultValue> {
-  string MarshallingKind = "flag";
+  code NormalizerRetTy = ty;
+  code Normalizer = "normalizeSimpleFlag";
+}
+
+class MarshallingInfoBitfieldFlag<code keypath, code value>
+  : MarshallingInfoFlag<keypath, DefaultAnyOf<[], "0u", " | ">, "unsigned"> {
+  code Normalizer = "(normalizeFlagToValue<unsigned, "#value#">)";
+  code ValueMerger = "mergeMaskValue";
+  code ValueExtractor = "(extractMaskValue<unsigned, decltype("#value#"), "#value#">)";
 }
 
 // Mixins for additional marshalling attributes.
 
-class IsNegative { bit IsPositive = 0; }
+class IsNegative {
+  // todo: create & apply a normalizer for negative flags
+}
 class AlwaysEmit { bit ShouldAlwaysEmit = 1; }
 class Normalizer<code normalizer> { code Normalizer = normalizer; }
 class Denormalizer<code denormalizer> { code Denormalizer = denormalizer; }
 class NormalizedValuesScope<code scope> { code NormalizedValuesScope = scope; }
 class NormalizedValues<list<code> definitions> { list<code> NormalizedValues = definitions; } 
 class DenormalizeString { code Denormalizer = "denormalizeString"; }
 class AutoNormalizeEnum {
   code Normalizer = "normalizeSimpleEnum";
   code Denormalizer = "denormalizeSimpleEnum";
 }
+class ValueMerger<code merger> { code ValueMerger = merger; }
+class ValueExtractor<code extractor> { code ValueExtractor = extractor; }
 
 // Predefined options.
 
 // FIXME: Have generator validate that these appear in correct position (and
 // aren't duplicated).
 def INPUT : Option<[], "<input>", KIND_INPUT>;
 def UNKNOWN : Option<[], "<unknown>", KIND_UNKNOWN>;

llvm/unittests/Option/OptionMarshallingTest.cpp

 
 struct OptionWithMarshallingInfo {
   const char *Name;
   const char *KeyPath;
   const char *DefaultValue;
 };
 
 static const OptionWithMarshallingInfo MarshallingTable[] = {
-#define OPTION_WITH_MARSHALLING_FLAG(PREFIX_TYPE, NAME, ID, KIND, GROUP,       \
-                                     ALIAS, ALIASARGS, FLAGS, PARAM, HELPTEXT, \
-                                     METAVAR, VALUES, SPELLING, ALWAYS_EMIT,   \
-                                     KEYPATH, DEFAULT_VALUE, IS_POSITIVE)      \
-  { NAME, #KEYPATH, #DEFAULT_VALUE },
+#define OPTION_WITH_MARSHALLING(                                               \
+    PREFIX_TYPE, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,        \
+    HELPTEXT, METAVAR, VALUES, SPELLING, ALWAYS_EMIT, KEYPATH, DEFAULT_VALUE,  \
+    TYPE, NORMALIZER, DENORMALIZER, MERGER, EXTRACTOR, TABLE_INDEX)            \
+  {NAME, #KEYPATH, #DEFAULT_VALUE},
 #include "Opts.inc"
-#undef OPTION_WITH_MARSHALLING_FLAG
+#undef OPTION_WITH_MARSHALLING
 };
 
 TEST(OptionMarshalling, EmittedOrderSameAsDefinitionOrder) {
-  ASSERT_STREQ(MarshallingTable[0].Name, "marshalled-flag-0");
-  ASSERT_STREQ(MarshallingTable[1].Name, "marshalled-flag-1");
-  ASSERT_STREQ(MarshallingTable[2].Name, "marshalled-flag-2");
-  ASSERT_STREQ(MarshallingTable[3].Name, "marshalled-flag-3");
+  ASSERT_STREQ(MarshallingTable[0].Name, "marshalled-flag-d");
+  ASSERT_STREQ(MarshallingTable[1].Name, "marshalled-flag-c");
+  ASSERT_STREQ(MarshallingTable[2].Name, "marshalled-flag-b");
+  ASSERT_STREQ(MarshallingTable[3].Name, "marshalled-flag-a");
 }
 
 TEST(OptionMarshalling, EmittedSpecifiedKeyPath) {
-  ASSERT_STREQ(MarshallingTable[0].KeyPath, "MarshalledFlag0");
-  ASSERT_STREQ(MarshallingTable[1].KeyPath, "MarshalledFlag1");
-  ASSERT_STREQ(MarshallingTable[2].KeyPath, "MarshalledFlag2");
-  ASSERT_STREQ(MarshallingTable[3].KeyPath, "MarshalledFlag3");
+  ASSERT_STREQ(MarshallingTable[0].KeyPath, "MarshalledFlagD");
+  ASSERT_STREQ(MarshallingTable[1].KeyPath, "MarshalledFlagC");
+  ASSERT_STREQ(MarshallingTable[2].KeyPath, "MarshalledFlagB");
+  ASSERT_STREQ(MarshallingTable[3].KeyPath, "MarshalledFlagA");
 }
 
 TEST(OptionMarshalling, DefaultAnyOfConstructedDisjunctionOfKeypaths) {
   ASSERT_STREQ(MarshallingTable[0].DefaultValue, "false");
-  ASSERT_STREQ(MarshallingTable[1].DefaultValue, "false || MarshalledFlag0");
-  ASSERT_STREQ(MarshallingTable[2].DefaultValue, "false || MarshalledFlag0");
+  ASSERT_STREQ(MarshallingTable[1].DefaultValue, "false || MarshalledFlagD");
+  ASSERT_STREQ(MarshallingTable[2].DefaultValue, "false || MarshalledFlagD");
   ASSERT_STREQ(MarshallingTable[3].DefaultValue,
-            "false || MarshalledFlag1 || MarshalledFlag2");
+            "false || MarshalledFlagC || MarshalledFlagB");
 }

llvm/unittests/Option/Opts.td

 
 def Glorrmp_eq : Flag<["--"], "glorrmp=">;
 
 def Blurmpq : Flag<["--"], "blurmp">;
 def Blurmpq_eq : Flag<["--"], "blurmp=">;
 
 def DashDash : Option<["--"], "", KIND_REMAINING_ARGS>;
 
-def marshalled_flag_0 : Flag<["-"], "marshalled-flag-0">,
-  MarshallingInfoFlag<"MarshalledFlag0", DefaultAnyOf<[]>>;
-def marshalled_flag_1 : Flag<["-"], "marshalled-flag-1">,
-  MarshallingInfoFlag<"MarshalledFlag1", DefaultAnyOf<[marshalled_flag_0]>>;
-def marshalled_flag_2 : Flag<["-"], "marshalled-flag-2">,
-  MarshallingInfoFlag<"MarshalledFlag2", DefaultAnyOf<[marshalled_flag_0]>>;
-def marshalled_flag_3 : Flag<["-"], "marshalled-flag-3">,
-  MarshallingInfoFlag<"MarshalledFlag3", DefaultAnyOf<[marshalled_flag_1, marshalled_flag_2]>>;
+def marshalled_flag_d : Flag<["-"], "marshalled-flag-d">,
+  MarshallingInfoFlag<"MarshalledFlagD", DefaultAnyOf<[]>>;
+def marshalled_flag_c : Flag<["-"], "marshalled-flag-c">,
+  MarshallingInfoFlag<"MarshalledFlagC", DefaultAnyOf<[marshalled_flag_d]>>;
+def marshalled_flag_b : Flag<["-"], "marshalled-flag-b">,
+  MarshallingInfoFlag<"MarshalledFlagB", DefaultAnyOf<[marshalled_flag_d]>>;
+def marshalled_flag_a : Flag<["-"], "marshalled-flag-a">,
+  MarshallingInfoFlag<"MarshalledFlagA", DefaultAnyOf<[marshalled_flag_c, marshalled_flag_b]>>;

llvm/utils/TableGen/OptParserEmitter.cpp

 
 static void emitNameUsingSpelling(raw_ostream &OS, const Record &R) {
   size_t PrefixLength;
   OS << "&";
   write_cstring(OS, StringRef(getOptionSpelling(R, PrefixLength)));
   OS << "[" << PrefixLength << "]";
 }
 
-class MarshallingKindInfo {
+class MarshallingInfo {
 public:
+  static constexpr const char *MacroName = "OPTION_WITH_MARSHALLING";
+
   const Record &R;
-  const char *MacroName;
   bool ShouldAlwaysEmit;
   StringRef KeyPath;
   StringRef DefaultValue;
   StringRef NormalizedValuesScope;
-
-  void emit(raw_ostream &OS) const {
-    write_cstring(OS, StringRef(getOptionSpelling(R)));
-    OS << ", ";
-    OS << ShouldAlwaysEmit;
-    OS << ", ";
-    OS << KeyPath;
-    OS << ", ";
-    emitScopedNormalizedValue(OS, DefaultValue);
-    OS << ", ";
-    emitSpecific(OS);
-  }
-
-  virtual Optional<StringRef> emitValueTable(raw_ostream &OS) const {
-    return None;
-  }
-
-  virtual ~MarshallingKindInfo() = default;
-
-  static std::unique_ptr<MarshallingKindInfo> create(const Record &R);
-
-protected:
-  void emitScopedNormalizedValue(raw_ostream &OS,
-                                 StringRef NormalizedValue) const {
-    if (!NormalizedValuesScope.empty())
-      OS << NormalizedValuesScope << "::";
-    OS << NormalizedValue;
-  }
-
-  virtual void emitSpecific(raw_ostream &OS) const = 0;
-  MarshallingKindInfo(const Record &R, const char *MacroName)
-      : R(R), MacroName(MacroName) {}
-};
-
-class MarshallingFlagInfo final : public MarshallingKindInfo {
-public:
-  bool IsPositive;
-
-  void emitSpecific(raw_ostream &OS) const override { OS << IsPositive; }
-
-  static std::unique_ptr<MarshallingKindInfo> create(const Record &R) {
-    std::unique_ptr<MarshallingFlagInfo> Ret(new MarshallingFlagInfo(R));
-    Ret->IsPositive = R.getValueAsBit("IsPositive");
-    // FIXME: This is a workaround for a bug in older versions of clang (< 3.9)
-    //   The constructor that is supposed to allow for Derived to Base
-    //   conversion does not work. Remove this if we drop support for such
-    //   configurations.
-    return std::unique_ptr<MarshallingKindInfo>(Ret.release());
-  }
-
-private:
-  MarshallingFlagInfo(const Record &R)
-      : MarshallingKindInfo(R, "OPTION_WITH_MARSHALLING_FLAG") {}
-};
-
-class MarshallingStringInfo final : public MarshallingKindInfo {
-public:
   StringRef NormalizerRetTy;
   StringRef Normalizer;
   StringRef Denormalizer;
+  StringRef ValueMerger;
+  StringRef ValueExtractor;
   int TableIndex = -1;
   std::vector<StringRef> Values;
   std::vector<StringRef> NormalizedValues;
   std::string ValueTableName;
 
   static constexpr const char *ValueTablePreamble = R"(
 struct SimpleEnumValue {
   const char *Name;
   unsigned Value;
 };
 
 struct SimpleEnumValueTable {
   const SimpleEnumValue *Table;
   unsigned Size;
 };
 )";
 
   static constexpr const char *ValueTablesDecl =
       "static const SimpleEnumValueTable SimpleEnumValueTables[] = ";
 
-  void emitSpecific(raw_ostream &OS) const override {
+  void emit(raw_ostream &OS) const {
+    write_cstring(OS, StringRef(getOptionSpelling(R)));
+    OS << ", ";
+    OS << ShouldAlwaysEmit;
+    OS << ", ";
+    OS << KeyPath;
+    OS << ", ";
+    emitScopedNormalizedValue(OS, DefaultValue);
+    OS << ", ";
     emitScopedNormalizedValue(OS, NormalizerRetTy);
     OS << ", ";
     OS << Normalizer;
     OS << ", ";
     OS << Denormalizer;
     OS << ", ";
+    OS << ValueMerger;
+    OS << ", ";
+    OS << ValueExtractor;
+    OS << ", ";
     OS << TableIndex;
   }
 
-  Optional<StringRef> emitValueTable(raw_ostream &OS) const override {
+  Optional<StringRef> emitValueTable(raw_ostream &OS) const {
     if (TableIndex == -1)
       return {};
     OS << "static const SimpleEnumValue " << ValueTableName << "[] = {\n";
     for (unsigned I = 0, E = Values.size(); I != E; ++I) {
       OS << "{";
       write_cstring(OS, Values[I]);
       OS << ",";
       OS << "static_cast<unsigned>(";
       emitScopedNormalizedValue(OS, NormalizedValues[I]);
       OS << ")},";
     }
     OS << "};\n";
     return StringRef(ValueTableName);
   }
 
-  static std::unique_ptr<MarshallingKindInfo> create(const Record &R) {
-    assert(!isa<UnsetInit>(R.getValueInit("NormalizerRetTy")) &&
-           "String options must have a type");
-
-    std::unique_ptr<MarshallingStringInfo> Ret(new MarshallingStringInfo(R));
-    Ret->NormalizerRetTy = R.getValueAsString("NormalizerRetTy");
-
-    Ret->Normalizer = R.getValueAsString("Normalizer");
-    Ret->Denormalizer = R.getValueAsString("Denormalizer");
+  static MarshallingInfo create(const Record &R) {
+    assert(!isa<UnsetInit>(R.getValueInit("KeyPath")) &&
+           !isa<UnsetInit>(R.getValueInit("DefaultValue")) &&
+           !isa<UnsetInit>(R.getValueInit("NormalizerRetTy")) &&
+           !isa<UnsetInit>(R.getValueInit("ValueMerger")) &&
+           "MarshallingInfo must have a type");
+
+    MarshallingInfo Ret(R);
+    Ret.ShouldAlwaysEmit = R.getValueAsBit("ShouldAlwaysEmit");
+    Ret.KeyPath = R.getValueAsString("KeyPath");
+    Ret.DefaultValue = R.getValueAsString("DefaultValue");
+    Ret.NormalizedValuesScope = R.getValueAsString("NormalizedValuesScope");
+    Ret.NormalizerRetTy = R.getValueAsString("NormalizerRetTy");
+
+    Ret.Normalizer = R.getValueAsString("Normalizer");
+    Ret.Denormalizer = R.getValueAsString("Denormalizer");
+    Ret.ValueMerger = R.getValueAsString("ValueMerger");
+    Ret.ValueExtractor = R.getValueAsString("ValueExtractor");
 
     if (!isa<UnsetInit>(R.getValueInit("NormalizedValues"))) {
       assert(!isa<UnsetInit>(R.getValueInit("Values")) &&
              "Cannot provide normalized values for value-less options");
-      Ret->TableIndex = NextTableIndex++;
-      Ret->NormalizedValues = R.getValueAsListOfStrings("NormalizedValues");
-      Ret->Values.reserve(Ret->NormalizedValues.size());
-      Ret->ValueTableName = getOptionName(R) + "ValueTable";
+      Ret.TableIndex = NextTableIndex++;
+      Ret.NormalizedValues = R.getValueAsListOfStrings("NormalizedValues");
+      Ret.Values.reserve(Ret.NormalizedValues.size());
+      Ret.ValueTableName = getOptionName(R) + "ValueTable";
 
       StringRef ValuesStr = R.getValueAsString("Values");
       for (;;) {
         size_t Idx = ValuesStr.find(',');
         if (Idx == StringRef::npos)
           break;
         if (Idx > 0)
-          Ret->Values.push_back(ValuesStr.slice(0, Idx));
+          Ret.Values.push_back(ValuesStr.slice(0, Idx));
         ValuesStr = ValuesStr.slice(Idx + 1, StringRef::npos);
       }
       if (!ValuesStr.empty())
-        Ret->Values.push_back(ValuesStr);
+        Ret.Values.push_back(ValuesStr);
 
-      assert(Ret->Values.size() == Ret->NormalizedValues.size() &&
+      assert(Ret.Values.size() == Ret.NormalizedValues.size() &&
              "The number of normalized values doesn't match the number of "
              "values");
     }
 
-    // FIXME: This is a workaround for a bug in older versions of clang (< 3.9)
-    //   The constructor that is supposed to allow for Derived to Base
-    //   conversion does not work. Remove this if we drop support for such
-    //   configurations.
-    return std::unique_ptr<MarshallingKindInfo>(Ret.release());
+    return Ret;
   }
 
 private:
-  MarshallingStringInfo(const Record &R)
-      : MarshallingKindInfo(R, "OPTION_WITH_MARSHALLING_STRING") {}
+  void emitScopedNormalizedValue(raw_ostream &OS,
+                                 StringRef NormalizedValue) const {
+    if (!NormalizedValuesScope.empty())
+      OS << NormalizedValuesScope << "::";
+    OS << NormalizedValue;
+  }
+
+  MarshallingInfo(const Record &R) : R(R){};
 
   static size_t NextTableIndex;
 };
 
-size_t MarshallingStringInfo::NextTableIndex = 0;
-
-std::unique_ptr<MarshallingKindInfo>
-MarshallingKindInfo::create(const Record &R) {
-  assert(!isa<UnsetInit>(R.getValueInit("KeyPath")) &&
-         !isa<UnsetInit>(R.getValueInit("DefaultValue")) &&
-         "Must provide at least a key-path and a default value for emitting "
-         "marshalling information");
-
-  std::unique_ptr<MarshallingKindInfo> Ret = nullptr;
-  StringRef MarshallingKindStr = R.getValueAsString("MarshallingKind");
-
-  if (MarshallingKindStr == "flag")
-    Ret = MarshallingFlagInfo::create(R);
-  else if (MarshallingKindStr == "string")
-    Ret = MarshallingStringInfo::create(R);
-
-  Ret->ShouldAlwaysEmit = R.getValueAsBit("ShouldAlwaysEmit");
-  Ret->KeyPath = R.getValueAsString("KeyPath");
-  Ret->DefaultValue = R.getValueAsString("DefaultValue");
-  if (!isa<UnsetInit>(R.getValueInit("NormalizedValuesScope")))
-    Ret->NormalizedValuesScope = R.getValueAsString("NormalizedValuesScope");
-  return Ret;
-}
+size_t MarshallingInfo::NextTableIndex = 0;
 
 /// OptParserEmitter - This tablegen backend takes an input .td file
 /// describing a list of options and emits a data structure for parsing and
 /// working with those options when given an input command line.
 namespace llvm {
 void EmitOptParser(RecordKeeper &Records, raw_ostream &OS) {
   // Get the option groups and options.
   const std::vector<Record*> &Groups =
auto WriteOptRecordFields = [&](raw_ostream &OS, const Record &R) {
     OS << ", ";
     if (!isa<UnsetInit>(R.getValueInit("Values")))
       write_cstring(OS, R.getValueAsString("Values"));
     else
       OS << "nullptr";
   };
 
   auto IsMarshallingOption = [](const Record &R) {
-    return !isa<UnsetInit>(R.getValueInit("MarshallingKind")) &&
+    return !isa<UnsetInit>(R.getValueInit("KeyPath")) &&
            !R.getValueAsString("KeyPath").empty();
   };
 
   std::vector<const Record *> OptsWithMarshalling;
   for (unsigned I = 0, E = Opts.size(); I != E; ++I) {
     const Record &R = *Opts[I];
 
     // Start a single option entry.
void EmitOptParser(RecordKeeper &Records, raw_ostream &OS) {
   };
   // The RecordKeeper stores records (options) in lexicographical order, and we
   // have reordered the options again when generating prefix groups. We need to
   // restore the original definition order of options with marshalling to honor
   // the topology of the dependency graph implied by `DefaultAnyOf`.
   array_pod_sort(OptsWithMarshalling.begin(), OptsWithMarshalling.end(),
                  CmpMarshallingOpts);
 
-  std::vector<std::unique_ptr<MarshallingKindInfo>> MarshallingKindInfos;
+  std::vector<MarshallingInfo> MarshallingInfos;
   for (const auto *R : OptsWithMarshalling)
-    MarshallingKindInfos.push_back(MarshallingKindInfo::create(*R));
+    MarshallingInfos.push_back(MarshallingInfo::create(*R));
 
-  for (const auto &KindInfo : MarshallingKindInfos) {
-    OS << "#ifdef " << KindInfo->MacroName << "\n";
-    OS << KindInfo->MacroName << "(";
-    WriteOptRecordFields(OS, KindInfo->R);
+  for (const auto &MI : MarshallingInfos) {
+    OS << "#ifdef " << MI.MacroName << "\n";
+    OS << MI.MacroName << "(";
+    WriteOptRecordFields(OS, MI.R);
     OS << ", ";
-    KindInfo->emit(OS);
+    MI.emit(OS);
     OS << ")\n";
-    OS << "#endif // " << KindInfo->MacroName << "\n";
+    OS << "#endif // " << MI.MacroName << "\n";
   }
 
   OS << "\n";
   OS << "#ifdef SIMPLE_ENUM_VALUE_TABLE";
   OS << "\n";
-  OS << MarshallingStringInfo::ValueTablePreamble;
+  OS << MarshallingInfo::ValueTablePreamble;
   std::vector<StringRef> ValueTableNames;
-  for (const auto &KindInfo : MarshallingKindInfos)
-    if (auto MaybeValueTableName = KindInfo->emitValueTable(OS))
+  for (const auto &MI : MarshallingInfos)
+    if (auto MaybeValueTableName = MI.emitValueTable(OS))
       ValueTableNames.push_back(*MaybeValueTableName);
 
-  OS << MarshallingStringInfo::ValueTablesDecl << "{";
+  OS << MarshallingInfo::ValueTablesDecl << "{";
   for (auto ValueTableName : ValueTableNames)
     OS << "{" << ValueTableName << ", sizeof(" << ValueTableName
        << ") / sizeof(SimpleEnumValue)"
        << "},\n";
   OS << "};\n";
   OS << "static const unsigned SimpleEnumValueTablesSize = "
         "sizeof(SimpleEnumValueTables) / sizeof(SimpleEnumValueTable);\n";