[yaml2obj] Add -D k=v to preprocess the input YAML

[MaskRay]

Summary

Examples:

yaml2obj -D MACHINE=EM_386 a.yaml -o a.o
yaml2obj -D MACHINE=0x1234 a.yaml -o a.o

where a.yaml contains:

--- !ELF
FileHeader:
  Class:   ELFCLASS64
  Data:    ELFDATA2MSB
  Type:    ET_REL
  Machine: [[MACHINE]]

[MaskRay]

drop a redundant test

[merge_guards_bot]

Unit tests: fail. 62372 tests passed, 2 failed and 839 were skipped.

failed: LLVM.tools/yaml2obj/ELF/unused-overridden.yaml
failed: libc++.std/thread/thread_mutex/thread_mutex_requirements/thread_mutex_requirements_mutex/thread_mutex_class/try_lock.pass.cpp

clang-tidy: fail. clang-tidy found 0 errors and 5 warnings. 0 of them are added as review comments below (why?).

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[merge_guards_bot]

Unit tests: pass. 62374 tests passed, 0 failed and 839 were skipped.

clang-tidy: fail. clang-tidy found 0 errors and 5 warnings. 0 of them are added as review comments below (why?).

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[MaskRay]

Improve tests

[merge_guards_bot]

Unit tests: fail. 62379 tests passed, 1 failed and 839 were skipped.

failed: libc++.std/thread/thread_mutex/thread_mutex_requirements/thread_mutex_requirements_mutex/thread_mutex_recursive/try_lock.pass.cpp

clang-tidy: fail. clang-tidy found 0 errors and 5 warnings. 0 of them are added as review comments below (why?).

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[grimar]

Sorry, but I'd probably prefer to have a different approach.
Something that works like a preprocessor. We could have:

--- !ELF
FileHeader:
  [[FOO]]:   ELFCLASS32
  Data:    ELFDATA2LSB
  Type:    ET_REL
  Machine: [[MACHINE]]

And then have a code that replaces `[[MACHINE]]', '[[FOO]]' strings in the input data
with something that is specified with -D option. e.g.

yaml2obj -DMACHINE="XXX" -DFOO="Class"

gives

--- !ELF
FileHeader:
  Class:   ELFCLASS32
  Data:    ELFDATA2LSB
  Type:    ET_REL
  Machine: XXX

It is much more generic (can be used for all targets, fields, etc), and it should be much simpler to implement.
As a bonus we will be able to modify any part of the YAML. It also makes inputs to be more clear,
because currently it is not obvious that some part is overridable.

And I do not have any concerns about the fact we modify the input YAML description here,
because the usage of such construction is limited to our test cases
and we should not met issues with it. When there is no -D option given, no preprocessing should happen. Should be safe enough.

[jhenderson]

I mostly agree with @grimar, although I don't think it's necessary to support arbitrary Key names, only Value names. Should the proposal (in whatever form it takes) go up on the mailing lists as an RFC? It feels like a useful feature that more people might want to talk about.

[MaskRay]

Repurpose.

Adopt grimar's suggestion

[merge_guards_bot]

Unit tests: fail. 62422 tests passed, 1 failed and 845 were skipped.

failed: LLVM.Analysis/ConstantFolding/vscale-shufflevector.ll

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[grimar]

I am going to debug this. My first impression is that the direction is right,
but it needs a bit of polishing.
Will add more comments soon.

llvm/test/tools/yaml2obj/ELF/emachine.yaml

1

This is a refactoring change, right? I'd leave such changes for a follow-up patch(es).

llvm/test/tools/yaml2obj/preprocess.yaml

10 ↗

(On Diff #242158)

It is probably usefull to add cases with more than one -D:

-D FOO=x -D BAR=y

-D FOO=x -D FOO=y

llvm/tools/yaml2obj/yaml2obj.cpp

35

Seems the description should start from the upper case letter and perhaps I'd reword it somehow.
It is not a list it seems? It is a single redefinition as far I understand.

45

Perhaps Substitutions or may be just Defines? Up to you,
but -D option is closer to the latter word I think.

47

What is F stands for? I`d rename to Name and Value..

[jhenderson]

llvm/test/tools/yaml2obj/preprocess.yaml

1 ↗

(On Diff #242158)

Perhaps also worth a test case that "-DNAME=..." is accepted, since that matches the compiler syntax, so would be a likely usage.

22 ↗

(On Diff #242158)

I'm not sure this comment line adds anything. My immediate reaction was "what's this test got to do with FileCheck?"

llvm/tools/yaml2obj/yaml2obj.cpp

35

I'd change this help text to "Defined the specified macros to their specified definition. The syntax is <macro>=<definition>".

47

Let's use Macro and Definition

The "field/value" naming doesn't really make sense since the new functionality is much more general purpose.

50

This error should a) mention the switch name, and b) probably not use the word "override" since nothing is being overridden as such. How about "invalid syntax for -D: xxx" (where xxx is the input string). If you think you need extra context, I'd split "missing '=' from "missing macro name".

60

I != StringRef::npos

[jhenderson]

(@grimar got in with comments as I was writing mine, so sorry for any duplication!)

[grimar]

llvm/test/tools/yaml2obj/preprocess.yaml

26 ↗

(On Diff #242158)

This does not work for me under windows.

echo -e 'a[[\n[[a\n[[a]\n[[a][[a]][[a]]' | cmp - %t.nosubst

I see the following when I run this line:

D:\Work3\LLVM\llvm-project\build\test\tools\yaml2obj\Output>echo -e 'a[[\n[[a\n[
[a]\n[[a][[a]][[a]]' | cmp - preprocess.yaml.tmp.nosubst
- preprocess.yaml.tmp.nosubst различаются: байт 1, строка 1

What means that byte 1 at the line 1 is different from the source. (My apologies, I do not know how to
switch to displaying errors in English for cmp).

I did not dig deeper because anyways I do not like this way of checking symbol names (sorry :),
so I'd ask you to rewrite in a more traditional way.

llvm/tools/yaml2obj/yaml2obj.cpp

53

Substitution[F] = V;

try_emplace is not needed to be used here, is't? StringRef is a cheap type for copying and
moving concept does not make sense at all I think.

75

I'd suggest something like the following

std::vector<std::pair<std::string, StringRef>> Substitution;
  for (StringRef Define : D) {
    StringRef F, V;
    std::tie(F, V) = Define.split('=');
    if (!Define.count('=') || F.empty()) {
      ErrHandler("bad override, missing field name: " + Define);
      return {};
    }
    Substitution.push_back({(Twine("[[") + F + "]]").str(), V});
  }

  std::string Document = Buf.str();
  if (Substitution.empty())
    return Document;

  for (std::pair<std::string, StringRef> &P : Substitution) {
    while (true) {
      auto I = Document.find(P.first);
      if (I == std::string::npos)
        break;
      Document = Document.replace(I, P.first.size(), P.second.str());
    }
  }

[MaskRay]

Address review comments.

llvm/test/tools/yaml2obj/preprocess.yaml

26 ↗

(On Diff #242158)

Switched to

echo -e '.........' > ...
diff -u ... %t.nosubst

Unfortunately we cannot use FileCheck because FileCheck does not like [[ which is not used as a FileCheck pattern.

llvm/tools/yaml2obj/yaml2obj.cpp

75

Maybe we should avoid quadratic complexity behavior, just in case that there are a long run of [[[[[[[[[[.

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[MaskRay]

Simplify with Buf.find_first_of

(OT: StringRef::find_first_of is O(n * 256) but the method can be imroved.)

[MaskRay]

Simplify...

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[grimar]

llvm/test/tools/yaml2obj/preprocess.yaml

26 ↗

(On Diff #242158)

Unfortunately we cannot use FileCheck because FileCheck does not like [[ which is not used as a FileCheck pattern.

Ah, I see. I did not know that. That is unfortunate behavior.

llvm/tools/yaml2obj/yaml2obj.cpp

75

just in case that there are a long run of [[[[[[[[[[.

It reminded me about D36307. I believe that [[[[[[ is a misuse of the feature and hence we probably should not care much.
The approach I suggested is just very simple and it is much easier to read. I think the common use case for this feature will be
"replace 1-2 defines in a short enough YAML", i.e. I am not sure we should think about an additional optimizations here.

Probably it is time to hear a third opinion though. @jhenderson, what do you think?

[MaskRay]

llvm/tools/yaml2obj/yaml2obj.cpp

75

Doing replace repeatedly may also have a non-convergence problem...

[jhenderson]

llvm/tools/yaml2obj/yaml2obj.cpp

75

Recursive replacement causes issues. For example -DFOO=[[FOO]] would result in an infinite loop. Convergence issues are only a problem if we try to do this. I personally don't think we should in the first version.

Another issue is what to do about variables that expand to other variables. One example is -D VAR1=FOO -D VAR2=[[VAR1]]BAR which will result in [[VAR2]] becoming [[VAR1]]BAR if specified in that order, or FOOBAR if VAR2 is defined first. I don't think we want this behaviour, as it is unstable and unobvious.

(Relatedly, I think we should have a test showing that variables that expand to a pattern that could be another variable aren't recursively expanded)

As for the quadratic complexity, I'm not too bothered by it in this case, as I doubt we'll ever get a YAML doc in a test where it really matters (and even if we do, there won't be many, so the overall slow-down in the testsuite will be minimal).

FWIW, I think we can probably forbid names containing '[' or ']' themselves.

[MaskRay]

Improve a test

[MaskRay]

llvm/tools/yaml2obj/yaml2obj.cpp

75

Added the following to macro.yaml

# RUN: yaml2obj --docnum=2 -D a0='[[a1]]' -D a1='[[a0]]' %s | llvm-nm --just-symbol-name --no-sort - > %t.recur
# RUN: echo -e '[[a1]]\n[[a0]]' > %t0.recur
# RUN: diff -u %t0.recur %t.recur

This patch avoids quadratic complexity and non-convergence with very little code, so I think sticking with the current version is fine.

[merge_guards_bot]

Unit tests: fail. 62504 tests passed, 8 failed and 844 were skipped.

failed: LLVM.tools/yaml2obj/help.test
failed: lld.ELF/compressed-input-alignment.test
failed: lld.ELF/invalid/bad-reloc-target.test
failed: lld.ELF/invalid/common-symbol-alignment.test
failed: lld.ELF/invalid/dynamic-section-broken.test
failed: lld.ELF/invalid/symtab-sh-info.s
failed: lld.ELF/mips-elf-flags-err.test
failed: lld.ELF/mips-fp-flags-err.test

clang-tidy: pass.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[grimar]

OK. I have 2 non-strong comments/suggestions about the current code.

llvm/tools/yaml2obj/yaml2obj.cpp

45

Should we exit early for a general case?

if (D.empty())
  return Buf.str();

It might simplify things for those who want to debug the code.

65

Here is a little logical issue. If we have input like "[[AAA[",
then you still assign "AAA" to Macro and perform a lookup,
but at fact "AAA" is not a macro and you do not need to do a lookup for it.

[MaskRay]

llvm/tools/yaml2obj/yaml2obj.cpp

45

When an early exit is just for the fast path (for the common case), I prefer not to add them.

For example,

if (thing.empty())
  return;
for (... : thing) {
  do something
}

I think if (thing.empty()) is unnecessary, unless the code is a critical path and the early return thing has significant performance improvement.

65

Yes. The if can be split to avoid a lookup. It will add two lines. I can do that.

[grimar]

llvm/tools/yaml2obj/yaml2obj.cpp

45

I see what you mean, though here the situation is a bit different probably: too many code is involved, i.e. it is not a trivial loop from your example. it is somehting much more. I do not insist here though, so up to you.

65

I am not concerned about the perfomance at all here, but I'd do that for code readability as it is a bit too noticable and IMO might look like a possible issue for a new readers.

[MaskRay]

Split an if

[grimar]

LGTM, thanks! Please wait for @jhenderson too.

[jhenderson]

LGTM too!

llvm/include/llvm/ObjectYAML/ELFYAML.h

 /// This file declares classes for handling the YAML representation
 /// of ELF.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_OBJECTYAML_ELFYAML_H
 #define LLVM_OBJECTYAML_ELFYAML_H
 
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ObjectYAML/YAML.h"
 #include "llvm/Support/YAMLTraits.h"
 #include <cstdint>
 #include <memory>
 #include <vector>
 
 namespace llvm {
struct Object {
 
   // Although in reality the symbols reside in a section, it is a lot
   // cleaner and nicer if we read them from the YAML as a separate
   // top-level key, which automatically ensures that invariants like there
   // being a single SHT_SYMTAB section are upheld.
   Optional<std::vector<Symbol>> Symbols;
   Optional<std::vector<Symbol>> DynamicSymbols;
 
+  DenseMap<StringRef, StringRef> Overridden;
+  DenseSet<StringRef> SeenOverridden;
+
+  Object(DenseMap<StringRef, StringRef> Overridden) : Overridden(Overridden) {}
+
   std::vector<Section *> getSections() {
     std::vector<Section *> Ret;
     for (const std::unique_ptr<Chunk> &Sec : Chunks)
       if (auto S = dyn_cast<ELFYAML::Section>(Sec.get()))
         Ret.push_back(S);
     return Ret;
   }
 };
struct ScalarEnumerationTraits<ELFYAML::ELF_ET> {
   static void enumeration(IO &IO, ELFYAML::ELF_ET &Value);
 };
 
 template <> struct ScalarEnumerationTraits<ELFYAML::ELF_PT> {
   static void enumeration(IO &IO, ELFYAML::ELF_PT &Value);
 };
 
 template <>
-struct ScalarEnumerationTraits<ELFYAML::ELF_EM> {
-  static void enumeration(IO &IO, ELFYAML::ELF_EM &Value);
+struct ScalarEnumerationContextTraits<ELFYAML::ELF_EM, ELFYAML::Object *> {
+  static void enumeration(IO &IO, ELFYAML::ELF_EM &Value,
+                          ELFYAML::Object *&Ctx);
 };
 
 template <>
 struct ScalarEnumerationTraits<ELFYAML::ELF_ELFCLASS> {
   static void enumeration(IO &IO, ELFYAML::ELF_ELFCLASS &Value);
 };
 
 template <>

llvm/include/llvm/ObjectYAML/ObjectYAML.h

 
 struct YamlObjectFile {
   std::unique_ptr<ELFYAML::Object> Elf;
   std::unique_ptr<COFFYAML::Object> Coff;
   std::unique_ptr<MachOYAML::Object> MachO;
   std::unique_ptr<MachOYAML::UniversalBinary> FatMachO;
   std::unique_ptr<MinidumpYAML::Object> Minidump;
   std::unique_ptr<WasmYAML::Object> Wasm;
+  DenseMap<StringRef, StringRef> Overridden;
 };
 
 template <> struct MappingTraits<YamlObjectFile> {
   static void mapping(IO &IO, YamlObjectFile &ObjectFile);
 };
 
 } // end namespace yaml
 } // end namespace llvm
 
 #endif // LLVM_OBJECTYAML_OBJECTYAML_H

llvm/include/llvm/ObjectYAML/yaml2obj.h

 //===--- yaml2obj.h - -------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file
 /// Common declarations for yaml2obj
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_TOOLS_YAML2OBJ_YAML2OBJ_H
 #define LLVM_TOOLS_YAML2OBJ_YAML2OBJ_H
 
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
 #include <memory>
+#include <string>
+#include <vector>
 
 namespace llvm {
 class raw_ostream;
 template <typename T> class SmallVectorImpl;
 template <typename T> class Expected;
 
 namespace object {
 class ObjectFile;

 bool yaml2coff(COFFYAML::Object &Doc, raw_ostream &Out, ErrorHandler EH);
 bool yaml2elf(ELFYAML::Object &Doc, raw_ostream &Out, ErrorHandler EH);
 bool yaml2macho(YamlObjectFile &Doc, raw_ostream &Out, ErrorHandler EH);
 bool yaml2minidump(MinidumpYAML::Object &Doc, raw_ostream &Out,
                    ErrorHandler EH);
 bool yaml2wasm(WasmYAML::Object &Doc, raw_ostream &Out, ErrorHandler EH);
 
 bool convertYAML(Input &YIn, raw_ostream &Out, ErrorHandler ErrHandler,
-                 unsigned DocNum = 1);
+                 unsigned DocNum = 1,
+                 std::vector<std::string> Substitutions = {});
 
 /// Convenience function for tests.
 std::unique_ptr<object::ObjectFile>
 yaml2ObjectFile(SmallVectorImpl<char> &Storage, StringRef Yaml,
                 ErrorHandler ErrHandler);
 
 } // namespace yaml
 } // namespace llvm
 
 #endif

llvm/include/llvm/Support/YAMLTraits.h

 ///           io.enumCase(value, "green", cGreen);
 ///         }
 ///       };
 template <typename T, typename Enable = void> struct ScalarEnumerationTraits {
   // Must provide:
   // static void enumeration(IO &io, T &value);
 };
 
+/// Similar to ScalarEnumerationTraits<T>, but allows you to pass in an
+/// additional context.
+template <class T, class Context> struct ScalarEnumerationContextTraits {};
+
 /// This class should be specialized by any integer type that is a union
 /// of bit values and the YAML representation is a flow sequence of
 /// strings.  For example:
 ///
 ///      struct ScalarBitSetTraits<MyFlags> {
 ///        static void bitset(IO &io, MyFlags &value) {
 ///          io.bitSetCase(value, "big",   flagBig);
 ///          io.bitSetCase(value, "flat",  flagFlat);
template <typename T> struct PolymorphicTraits {
   // static map_type &getAsMap(T &poly);
   // static sequence_type &getAsSequence(T &poly);
 };
 
 // Only used for better diagnostics of missing traits
 template <typename T>
 struct MissingTrait;
 
-// Test if ScalarEnumerationTraits<T> is defined on type T.
-template <class T>
-struct has_ScalarEnumerationTraits
-{
-  using Signature_enumeration = void (*)(class IO&, T&);
-
+// Test if ScalarEnumerationContextTraits<T> is defined on type T.
+template <class T, class Context> struct has_ScalarEnumerationTraits {
   template <typename U>
-  static char test(SameType<Signature_enumeration, &U::enumeration>*);
+  static char
+  test(SameType<void (*)(class IO &, T &, Context &), &U::enumeration> *);
+  template <typename U> static double test(...);
+
+  static bool const value =
+      (sizeof(test<ScalarEnumerationContextTraits<T, Context>>(nullptr)) == 1);
+};
 
+// Test if ScalarEnumerationCTraits<T> is defined on type T.
+template <class T> struct has_ScalarEnumerationTraits<T, EmptyContext> {
   template <typename U>
-  static double test(...);
+  static char test(SameType<void (*)(class IO &, T &), &U::enumeration> *);
+  template <typename U> static double test(...);
 
   static bool const value =
       (sizeof(test<ScalarEnumerationTraits<T>>(nullptr)) == 1);
 };
 
 // Test if ScalarBitSetTraits<T> is defined on type T.
 template <class T>
 struct has_ScalarBitSetTraits
 {
   using Signature_bitset = void (*)(class IO&, T&);
 
for (unsigned char C : S) {
     }
   }
 
   return MaxQuotingNeeded;
 }
 
 template <typename T, typename Context>
 struct missingTraits
-    : public std::integral_constant<bool,
-                                    !has_ScalarEnumerationTraits<T>::value &&
-                                        !has_ScalarBitSetTraits<T>::value &&
-                                        !has_ScalarTraits<T>::value &&
-                                        !has_BlockScalarTraits<T>::value &&
-                                        !has_TaggedScalarTraits<T>::value &&
-                                        !has_MappingTraits<T, Context>::value &&
-                                        !has_SequenceTraits<T>::value &&
-                                        !has_CustomMappingTraits<T>::value &&
-                                        !has_DocumentListTraits<T>::value &&
-                                        !has_PolymorphicTraits<T>::value> {};
+    : public std::integral_constant<
+          bool, !has_ScalarEnumerationTraits<T, Context>::value &&
 
 template <typename T, typename Context>
 struct validatedMappingTraits
     : public std::integral_constant<
           bool, has_MappingTraits<T, Context>::value &&
                     has_MappingValidateTraits<T, Context>::value> {};
 
 template <typename T, typename Context>
if (matchEnumFallback()) {
       EmptyContext Context;
       // FIXME: Force integral conversion to allow strong typedefs to convert.
       FBT Res = static_cast<typename FBT::BaseType>(Val);
       yamlize(*this, Res, true, Context);
       Val = static_cast<T>(static_cast<typename FBT::BaseType>(Res));
     }
   }
 
+  template <typename FBT, typename T>
+  void enumFallbackWithOverridden(StringRef Overridden, T &Val) {
+    if (matchEnumFallback()) {
+      FBT Res = static_cast<typename FBT::BaseType>(Val);
+      StringRef Err = ScalarTraits<FBT>::input(Overridden, getContext(), Res);
+      if (!Err.empty())
+        setError(Err);
+      Val = static_cast<T>(static_cast<typename FBT::BaseType>(Res));
+    }
+  }
+
   template <typename T>
   void bitSetCase(T &Val, const char* Str, const T ConstVal) {
     if ( bitSetMatch(Str, outputting() && (Val & ConstVal) == ConstVal) ) {
       Val = static_cast<T>(Val | ConstVal);
     }
   }
 
   // allow anonymous enum values to be used with LLVM_YAML_STRONG_TYPEDEF

 
 private:
   void *Ctxt;
 };
 
 namespace detail {
 
 template <typename T, typename Context>
+void doEnumeration(IO &io, T &Val, Context &Ctx) {
+  ScalarEnumerationContextTraits<T, Context>::enumeration(io, Val, Ctx);
+}
+
+template <typename T> void doEnumeration(IO &io, T &Val, EmptyContext &) {
+  ScalarEnumerationTraits<T>::enumeration(io, Val);
+}
+
+template <typename T, typename Context>
 void doMapping(IO &io, T &Val, Context &Ctx) {
   MappingContextTraits<T, Context>::mapping(io, Val, Ctx);
 }
 
 template <typename T> void doMapping(IO &io, T &Val, EmptyContext &Ctx) {
   MappingTraits<T>::mapping(io, Val);
 }
 
 } // end namespace detail
 
-template <typename T>
-typename std::enable_if<has_ScalarEnumerationTraits<T>::value, void>::type
-yamlize(IO &io, T &Val, bool, EmptyContext &Ctx) {
+template <typename T, typename Context = EmptyContext>
+typename std::enable_if<has_ScalarEnumerationTraits<T, Context>::value,
+                        void>::type
+yamlize(IO &io, T &Val, bool, Context &Ctx) {
   io.beginEnumScalar();
-  ScalarEnumerationTraits<T>::enumeration(io, Val);
+  detail::doEnumeration(io, Val, Ctx);
   io.endEnumScalar();
 }
 
 template <typename T>
 typename std::enable_if<has_ScalarBitSetTraits<T>::value, void>::type
 yamlize(IO &io, T &Val, bool, EmptyContext &Ctx) {
   bool DoClear;
   if ( io.beginBitSetScalar(DoClear) ) {
struct ScalarTraits<
   }
 };
 
 template <typename value_type, support::endianness endian, size_t alignment>
 struct ScalarEnumerationTraits<
     support::detail::packed_endian_specific_integral<value_type, endian,
                                                      alignment>,
     typename std::enable_if<
-        has_ScalarEnumerationTraits<value_type>::value>::type> {
+        has_ScalarEnumerationTraits<value_type, EmptyContext>::value>::type> {
   using endian_type =
       support::detail::packed_endian_specific_integral<value_type, endian,
                                                        alignment>;
 
   static void enumeration(IO &io, endian_type &E) {
     value_type V = E;
     ScalarEnumerationTraits<value_type>::enumeration(io, V);
     E = V;

llvm/lib/ObjectYAML/ELFYAML.cpp

#define ECase(X) IO.enumCase(Value, #X, ELF::X)
   ECase(PT_GNU_EH_FRAME);
   ECase(PT_GNU_STACK);
   ECase(PT_GNU_RELRO);
   ECase(PT_GNU_PROPERTY);
 #undef ECase
   IO.enumFallback<Hex32>(Value);
 }
 
-void ScalarEnumerationTraits<ELFYAML::ELF_EM>::enumeration(
-    IO &IO, ELFYAML::ELF_EM &Value) {
-#define ECase(X) IO.enumCase(Value, #X, ELF::X)
+void ScalarEnumerationContextTraits<
+    ELFYAML::ELF_EM, ELFYAML::Object *>::enumeration(IO &io,
+                                                     ELFYAML::ELF_EM &Value,
+                                                     ELFYAML::Object *&Ctx) {
+  bool Output = io.outputting();
+  Optional<StringRef> OverriddenStr;
+  auto It = Ctx->Overridden.find("e_machine");
+  if (It != Ctx->Overridden.end()) {
+    OverriddenStr = It->second;
+    Ctx->SeenOverridden.insert("e_machine");
+  }
+
+#define ECase(X)                                                               \
+  if (!Output && OverriddenStr && *OverriddenStr == #X) {                      \
+    Value = ELF::X;                                                            \
+    io.matchEnumFallback();                                                    \
+    return;                                                                    \
+  }                                                                            \
+  io.enumCase(Value, #X, ELF::X);
+
   ECase(EM_NONE);
   ECase(EM_M32);
   ECase(EM_SPARC);
   ECase(EM_386);
   ECase(EM_68K);
   ECase(EM_88K);
   ECase(EM_IAMCU);
   ECase(EM_860);
#define ECase(X)                                                               \
   ECase(EM_VIDEOCORE5);
   ECase(EM_78KOR);
   ECase(EM_56800EX);
   ECase(EM_AMDGPU);
   ECase(EM_RISCV);
   ECase(EM_LANAI);
   ECase(EM_BPF);
 #undef ECase
-  IO.enumFallback<Hex16>(Value);
+
+  if (!Output && OverriddenStr)
+    io.enumFallbackWithOverridden<Hex16>(*OverriddenStr, Value);
+  else
+    io.enumFallback<Hex16>(Value);
 }
 
 void ScalarEnumerationTraits<ELFYAML::ELF_ELFCLASS>::enumeration(
     IO &IO, ELFYAML::ELF_ELFCLASS &Value) {
 #define ECase(X) IO.enumCase(Value, #X, ELF::X)
   // Since the semantics of ELFCLASSNONE is "invalid", just don't accept it
   // here.
   ECase(ELFCLASS32);
void ScalarBitSetTraits<ELFYAML::MIPS_AFL_FLAGS1>::bitset(
     IO &IO, ELFYAML::MIPS_AFL_FLAGS1 &Value) {
 #define BCase(X) IO.bitSetCase(Value, #X, Mips::AFL_FLAGS1_##X)
   BCase(ODDSPREG);
 #undef BCase
 }
 
 void MappingTraits<ELFYAML::FileHeader>::mapping(IO &IO,
                                                  ELFYAML::FileHeader &FileHdr) {
+  auto *Ctx = static_cast<ELFYAML::Object *>(IO.getContext());
   IO.mapRequired("Class", FileHdr.Class);
   IO.mapRequired("Data", FileHdr.Data);
   IO.mapOptional("OSABI", FileHdr.OSABI, ELFYAML::ELF_ELFOSABI(0));
   IO.mapOptional("ABIVersion", FileHdr.ABIVersion, Hex8(0));
   IO.mapRequired("Type", FileHdr.Type);
-  IO.mapRequired("Machine", FileHdr.Machine);
+  IO.mapRequired("Machine", FileHdr.Machine, Ctx);
   IO.mapOptional("Flags", FileHdr.Flags, ELFYAML::ELF_EF(0));
   IO.mapOptional("Entry", FileHdr.Entry, Hex64(0));
 
   IO.mapOptional("SHEntSize", FileHdr.SHEntSize);
   IO.mapOptional("SHOff", FileHdr.SHOff);
   IO.mapOptional("SHNum", FileHdr.SHNum);
   IO.mapOptional("SHStrNdx", FileHdr.SHStrNdx);
 }

llvm/lib/ObjectYAML/ObjectYAML.cpp

if (IO.outputting()) {
     if (ObjectFile.Coff)
       MappingTraits<COFFYAML::Object>::mapping(IO, *ObjectFile.Coff);
     if (ObjectFile.MachO)
       MappingTraits<MachOYAML::Object>::mapping(IO, *ObjectFile.MachO);
     if (ObjectFile.FatMachO)
       MappingTraits<MachOYAML::UniversalBinary>::mapping(IO,
                                                          *ObjectFile.FatMachO);
   } else {
+    DenseMap<StringRef, StringRef> Overridden;
+    DenseSet<StringRef> SeenOverridden;
     Input &In = (Input &)IO;
     if (IO.mapTag("!ELF")) {
-      ObjectFile.Elf.reset(new ELFYAML::Object());
+      ObjectFile.Elf.reset(
+          new ELFYAML::Object(std::move(ObjectFile.Overridden)));
       MappingTraits<ELFYAML::Object>::mapping(IO, *ObjectFile.Elf);
+      Overridden = std::move(ObjectFile.Elf->Overridden);
+      SeenOverridden = std::move(ObjectFile.Elf->SeenOverridden);
     } else if (IO.mapTag("!COFF")) {
       ObjectFile.Coff.reset(new COFFYAML::Object());
       MappingTraits<COFFYAML::Object>::mapping(IO, *ObjectFile.Coff);
     } else if (IO.mapTag("!mach-o")) {
       ObjectFile.MachO.reset(new MachOYAML::Object());
       MappingTraits<MachOYAML::Object>::mapping(IO, *ObjectFile.MachO);
     } else if (IO.mapTag("!fat-mach-o")) {
       ObjectFile.FatMachO.reset(new MachOYAML::UniversalBinary());
       MappingTraits<MachOYAML::UniversalBinary>::mapping(IO,
                                                          *ObjectFile.FatMachO);
     } else if (IO.mapTag("!minidump")) {
       ObjectFile.Minidump.reset(new MinidumpYAML::Object());
       MappingTraits<MinidumpYAML::Object>::mapping(IO, *ObjectFile.Minidump);
     } else if (IO.mapTag("!WASM")) {
       ObjectFile.Wasm.reset(new WasmYAML::Object());
       MappingTraits<WasmYAML::Object>::mapping(IO, *ObjectFile.Wasm);
     } else if (const Node *N = In.getCurrentNode()) {
       if (N->getRawTag().empty())
         IO.setError("YAML Object File missing document type tag!");
       else
         IO.setError("YAML Object File unsupported document type tag '" +
                     N->getRawTag() + "'!");
     }
+
+    for (auto &It : Overridden)
+      if (!SeenOverridden.count(It.first)) {
+        IO.setError("'" + It.first + "' is overridden but not used");
+        break;
+      }
   }
 }

llvm/lib/ObjectYAML/yaml2obj.cpp

 #include "llvm/Support/Errc.h"
 #include "llvm/Support/WithColor.h"
 #include "llvm/Support/YAMLTraits.h"
 
 namespace llvm {
 namespace yaml {
 
 bool convertYAML(yaml::Input &YIn, raw_ostream &Out, ErrorHandler ErrHandler,
-                 unsigned DocNum) {
+                 unsigned DocNum, std::vector<std::string> Overridden) {
+  yaml::YamlObjectFile Doc;
+  for (StringRef O : Overridden) {
+    StringRef F, V;
+    std::tie(F, V) = O.split('=');
+    if (!O.count('=') || F.empty()) {
+      ErrHandler("bad override, missing field name: " + O);
+      return false;
+    }
+    Doc.Overridden.try_emplace(F, V);
+  }
+
   unsigned CurDocNum = 0;
   do {
     if (++CurDocNum != DocNum)
       continue;
 
-    yaml::YamlObjectFile Doc;
     YIn >> Doc;
     if (std::error_code EC = YIn.error()) {
       ErrHandler("failed to parse YAML input: " + EC.message());
       return false;
     }
 
     if (Doc.Elf)
       return yaml2elf(*Doc.Elf, Out, ErrHandler);

llvm/test/tools/yaml2obj/ELF/emachine.yaml

 ## Test how the Machine YAML field can be used to set the e_machine ELF header field.
 
 ## Test we can use an arbitrary value.
 
 # RUN: yaml2obj --docnum=1 %s -o %t1
 # RUN: llvm-readelf %t1 --file-headers | FileCheck %s --check-prefix=UNKNOWN
 
 # UNKNOWN: Machine: 1234
 

 # ERR2: error: invalid hex16 number
 
 --- !ELF
 FileHeader:
   Class:   ELFCLASS64
   Data:    ELFDATA2MSB
   Type:    ET_REL
   Machine: EM_UNKNOWN_FOO
+
+## Test we can override e_machine with -D.
+
+# RUN: yaml2obj --docnum=1 -D e_machine=0x4321 %s -o %t1.4321
+# RUN: llvm-readobj --file-headers %t1.4321 | FileCheck %s --check-prefix=4321
+
+# 4321: Machine: 0x4321
+
+# RUN: yaml2obj --docnum=1 -D e_machine=EM_386 %s -o %t1.386
+# RUN: llvm-readobj --file-headers %t1.386 | FileCheck %s --check-prefix=386
+
+# 386: Machine: EM_386
+
+## TODO Report that the overridden value is invalid.
+# RUN: not yaml2obj --docnum=1 -D e_machine=invalid %s -o /dev/null 2>&1 | FileCheck --check-prefix=INVALID %s
+
+# INVALID: error: invalid hex16 number
+# INVALID-NEXT: Machine: 0x1234

llvm/test/tools/yaml2obj/ELF/unused-overridden.yaml

+## Test that an unused -D option emits a diagnostic.
+
+# RUN: not yaml2obj -D unused=1 %s -o /dev/null 2>&1 | FileCheck --check-prefix=UNUSED %s
+
+# UNUSED: error: 'unused' is overridden but not used
+
+## Test that an unused -D option emits an error.
+
+## TODO Report that the overridden value is invalid.
+# RUN: not yaml2obj -D e_machine=invalid %s -o /dev/null 2>&1 | FileCheck --check-prefix=INVALID %s
+
+# INVALID: error: invalid hex16 number
+# INVALID-NEXT: Machine: EM_386
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS32
+  Data:    ELFDATA2LSB
+  Type:    ET_REL
+  Machine: EM_386

llvm/test/tools/yaml2obj/invalid-override.yaml

+# RUN: not yaml2obj -D field %s 2>&1 | FileCheck --check-prefix=ERR1 %s
+# ERR1: error: bad override, missing field name: field
+
+# RUN: not yaml2obj -D =value %s 2>&1 | FileCheck --check-prefix=ERR2 %s
+# ERR2: error: bad override, missing field name: =value

llvm/tools/obj2yaml/elf2yaml.cpp

else
       SymbolNames[Index] = std::string(Name);
     return SymbolNames[Index];
   }
 
   return Name;
 }
 
 template <class ELFT> Expected<ELFYAML::Object *> ELFDumper<ELFT>::dump() {
-  auto Y = std::make_unique<ELFYAML::Object>();
+  auto Y = std::make_unique<ELFYAML::Object>(DenseMap<StringRef, StringRef>());
 
   // Dump header. We do not dump SHEntSize, SHOff, SHNum and SHStrNdx fields.
   // When not explicitly set, the values are set by yaml2obj automatically
   // and there is no need to dump them here.
   Y->Header.Class = ELFYAML::ELF_ELFCLASS(Obj.getHeader()->getFileClass());
   Y->Header.Data = ELFYAML::ELF_ELFDATA(Obj.getHeader()->getDataEncoding());
   Y->Header.OSABI = Obj.getHeader()->e_ident[ELF::EI_OSABI];
   Y->Header.ABIVersion = Obj.getHeader()->e_ident[ELF::EI_ABIVERSION];

llvm/tools/yaml2obj/yaml2obj.cpp

 #include "llvm/Support/raw_ostream.h"
 #include <system_error>
 
 using namespace llvm;
 
 static cl::opt<std::string>
   Input(cl::Positional, cl::desc("<input>"), cl::init("-"));
 
+static cl::list<std::string> D("D", cl::desc("list of overridden fields: field=value"));
+
 static cl::opt<unsigned>
     DocNum("docnum", cl::init(1),
            cl::desc("Read specified document from input (default = 1)"));
 
 static cl::opt<std::string> OutputFilename("o", cl::desc("Output filename"),
                                            cl::value_desc("filename"));
 
 int main(int argc, char **argv) {
   InitLLVM X(argc, argv);
   cl::ParseCommandLineOptions(argc, argv);
 
   if (OutputFilename.empty())
     OutputFilename = "-";
 
   auto ErrHandler = [](const Twine &Msg) {
     WithColor::error(errs(), "yaml2obj") << Msg << "\n";
   };
 
   std::error_code EC;
   std::unique_ptr<ToolOutputFile> Out(
       new ToolOutputFile(OutputFilename, EC, sys::fs::OF_None));
   if (EC) {
     ErrHandler("failed to open '" + OutputFilename + "': " + EC.message());
     return 1;
   }
 
   ErrorOr<std::unique_ptr<MemoryBuffer>> Buf =
       MemoryBuffer::getFileOrSTDIN(Input);
   if (!Buf)
     return 1;
 
   yaml::Input YIn(Buf.get()->getBuffer());
-  if (!convertYAML(YIn, Out->os(), ErrHandler, DocNum))
+  if (!convertYAML(YIn, Out->os(), ErrHandler, DocNum, D))
     return 1;
 
   Out->keep();
   Out->os().flush();
   return 0;
 }