[mlir] Allow for attaching external resources to .mlir files

[rriddle]

Summary

This commit enables support for providing and processing external
resources within MLIR assembly formats. This is a mechanism with which
dialects, and external clients, may attach additional information when
printing IR without that information being encoded in the IR itself.
External resources are not uniqued within the MLIR context, are not
attached directly to any operation, and are solely intended to live and be
processed outside of the immediate IR. There are many potential uses of this
functionality, for example MLIR's pass crash reproducer could utilize this to
attach the pass resource executing when a crash occurs. Other types of
uses may be embedding large amounts of binary data, such as weights in ML
applications, that shouldn't be copied directly into the MLIR context, but
need to be kept adjacent to the IR.

External resources are encoded using a key-value pair nested within a
dictionary anchored by name either on a dialect, or an externally registered
entity. The key is an identifier used to disambiguate the data. The value
may be stored in various limited forms, but general encodings use a string
(human readable) or blob format (binary). Within the textual format, an
example may be of the form:

mlir
{-#
  // The `dialect_resources` section within the file-level metadata
  // dictionary is used to contain any dialect resource entries.
  dialect_resources: {
    // Here is a dictionary anchored on "foo_dialect", which is a dialect
    // namespace.
    foo_dialect: {
      // `some_dialect_resource` is a key to be interpreted by the dialect,
      // and used to initialize/configure/etc.
      some_dialect_resource: "Some important resource value"
    }
  },
  // The `external_resources` section within the file-level metadata
  // dictionary is used to contain any non-dialect resource entries.
  external_resources: {
    // Here is a dictionary anchored on "mlir_reproducer", which is an
    // external entity representing MLIR's crash reproducer functionality.
    mlir_reproducer: {
      // `pipeline` is an entry that holds a crash reproducer pipeline
      // resource.
      pipeline: "func.func(canonicalize,cse)"
    }
  }

[mehdi_amini]

What about using directly YAML for this section?

[Mogball]

Doing it ourselves would keep MLIR light. And the external config API would just be built on top of YAML.

[Mogball]

mlir/include/mlir/IR/AsmState.h

57

Just curious, but what else could be stored here other than "external config"?

69

Is there any guarantee that dialect names won't collide with any random config key?

[rriddle]

In D126446#3539427, @mehdi_amini wrote:

What about using directly YAML for this section?

I considered, but ultimately it felt like more complexity than the benefits it brought (especially for this draft). Having something
just for us felt much simpler and more expressive (we could use Attributes/Types for config values if we wanted). I could likely be
convinced otherwise though, using something like YAML would conceptually make supporting more complex config values
easier (nested structs and stuff) if we wanted to go that approach. There are tradeoffs though (e.g. error reporting, attribute/type values,
still need a generic interface to enable seamless bitcode integration, etc.).

mlir/include/mlir/IR/AsmState.h

57

The idea for the top-level dictionary was to provide a convenient place to put things in the future. For example, this would be a natural place to put the dialect_versions metadata in Mehdi's WIP versioning patch: https://reviews.llvm.org/D117761

69

Non-dialect config keys are always wrapped with <>, so there won't be collision between the two.

[stellaraccident]

In D126446#3540627, @rriddle wrote:

In D126446#3539427, @mehdi_amini wrote:

What about using directly YAML for this section?

I considered, but ultimately it felt like more complexity than the benefits it brought (especially for this draft). Having something
just for us felt much simpler and more expressive (we could use Attributes/Types for config values if we wanted). I could likely be
convinced otherwise though, using something like YAML would conceptually make supporting more complex config values
easier (nested structs and stuff) if we wanted to go that approach. There are tradeoffs though (e.g. error reporting, attribute/type values,
still need a generic interface to enable seamless bitcode integration, etc.).

If the AsmConfig* classes were a bit more POD/model-ish, then it would be pretty easy to wrap YAML around that, but the way the implementation is laid out would be a bit of a weird fit (and I think I see why you have it the way you do). In general, I agree with the tradeoffs: it will feel and act like a complete side-thing. From my point of view, how this grows towards bitcode is more salient to me, and binding the text format close to YAML will just make it weird (all my subjective opinion based on trying to do such things before).

[mehdi_amini]

mlir/test/IR/file-metadata-config.mlir

17

I do have some concerns with this example (to the extent that this will be an example in-tree).

There is a problem in that we can't load in the same context some piece of IR if there was already another piece of IR using "blob1" loaded (even if the IR is deleted since).

[rriddle]

mlir/test/IR/file-metadata-config.mlir

17

Oh yeah, thanks for pointing this out. I originally wrote this quickly as a simple example, but forgot to update it. Do you have any suggestions on what we want to do here to set a good example? Compare the data? Just compare hashes?

[jfurtek]

Have you considered using base64 instead of hex for binary data?

[tschuett]

https://reviews.llvm.org/D126254 will bring base64 to LLVM and LLDB.

[bzcheeseman]

In D126446#3548652, @jfurtek wrote:

Have you considered using base64 instead of hex for binary data?

I would argue against base64 personally - I don't know if the benefits outweigh the issue that it inflates all data by 33%. It does beg the question though, of 'should we allow folks to choose some other encoding for data in this config struct?' If this is the only use case - that some folks want b64 data, then maybe it's better to just wait for the bitcode format and b64 encode the whole file if you want it to be Javascript-safe?

[jfurtek]

The current implementation uses hexadecimal, which requires 16 bits (2 bytes) to encode 8 bits of binary data. base64 would be an improvement over the current implementation that uses hexadecimal, requiring 8 bits to encode 6 bits of binary data. (The other constraint here is that encoding needs to be handled by the parser, which in the original example, means it can't have a quotation mark.) IMO, base64 is:

• smaller than hex
• somewhat ubiquitous
• compatible with the existing parser (I think?)
• a reasonable default for situations where the size overhead is an acceptable tradeoff for having the MLIR file be text-based and self-contained.

For text-based MLIR file situations where the size increase is an issue, I could imagine the config data containing a string with the name of an external file, and an application-defined way to resolve the full path. That could be out-of-tree and/or dialect specific.

[bzcheeseman]

In D126446#3553562, @jfurtek wrote:

The current implementation uses hexadecimal, which requires 16 bits (2 bytes) to encode 8 bits of binary data. base64 would be an improvement over the current implementation that uses hexadecimal, requiring 8 bits to encode 6 bits of binary data. (The other constraint here is that encoding needs to be handled by the parser, which in the original example, means it can't have a quotation mark.)

Good point! I was thinking of just the raw binary data, forgot that hex is silly that way :)

[rriddle]

In D126446#3553562, @jfurtek wrote:

The current implementation uses hexadecimal, which requires 16 bits (2 bytes) to encode 8 bits of binary data. base64 would be an improvement over the current implementation that uses hexadecimal, requiring 8 bits to encode 6 bits of binary data. (The other constraint here is that encoding needs to be handled by the parser, which in the original example, means it can't have a quotation mark.) IMO, base64 is:

• smaller than hex
• somewhat ubiquitous
• compatible with the existing parser (I think?)
• a reasonable default for situations where the size overhead is an acceptable tradeoff for having the MLIR file be text-based and self-contained.

For text-based MLIR file situations where the size increase is an issue, I could imagine the config data containing a string with the name of an external file, and an application-defined way to resolve the full path. That could be out-of-tree and/or dialect specific.

Yeah, I'd gladly switch to base64. We've only been using hex because it has prior art/use in-tree.

[stellaraccident]

A few very minor nits, but otherwise lgtm.

mlir/include/mlir/IR/AsmState.h

45

s/used/use/

97

sp. dataAlignment

mlir/lib/Parser/Parser.cpp

307

Missing closing ascii art?

[mehdi_amini]

mlir/include/mlir/IR/OpImplementation.h

32

Why didn't you keep an instance of the dialect here? Seems like you have to pass it around in a few places.

62

I find the naming and setup confusing with an inner class "Base" which inherits from its enclosing class? What are you trying to convey with this?
Is this some common pattern?

mlir/lib/Parser/Parser.cpp

317

I'm not sure what "otherwise" refer to here? This is the first comment in this function

mlir/test/lib/Dialect/Test/TestDialect.h

52

Where does deallocation happens?

[rriddle]

mlir/include/mlir/IR/OpImplementation.h

32

No good reason, added a dialect field (which is likely cleaner/more general anyways).

62

We do this in a few other places as well. The intent is to provide a CRTP Base/Impl class that defines away all of the boiler plate for the given resource type. Users don't really interact with it, it's solely for simplifying the definition of concrete handle types. Do you have a suggestion on better naming here?

mlir/test/lib/Dialect/Test/TestDialect.h

52

The base AsmResourceBlob holds a deleter function which deallocates the data within the destructor.

[mehdi_amini]

mlir/include/mlir/IR/OpImplementation.h

62

Why is it an inner class?

mlir/test/lib/Dialect/Test/TestDialect.h

52

I was missing that you were inheriting the base constructor and that it is how we get the deleter...

[rriddle]

mlir/include/mlir/IR/OpImplementation.h

62

Just to make it more contained (i.e. not pollute the mlir namespace), I could just pull it out if you prefer.

[uweigand]

This commit has broken the mlir builder on s390x-ibm-linux: https://lab.llvm.org/buildbot/#/builders/199/builds/6417

The immediate error is this failing assertion:

mlir-opt: /home/uweigand/llvm/llvm-head/mlir/test/lib/Dialect/Test/TestDialect.cpp:191: {anonymous}::TestOpAsmInterface::parseResource(mlir::AsmParsedResourceEntry&) const::<lambda(unsigned int, unsigned int)>: Assertion `align == alignof(uint64_t) && "unexpected data alignment"' failed.

which occurs because the value of align is 0x08000000 instead of the expected 8. The alignment was taken from the binary blob, encoded in little-endian format, but decoded as uint32_t on the host platform, which in our case is big-endian.

Now this is easily fixed by using proper endian conversion when accessing the alignment, which fixes the assertion, but then the test cases still fail because they expect little-endian data in the generated blobs, while they will actually contain big-endian data when generated on a big-endian machine.

What is the intended design point here? Are those binary blobs intended to be portable across machine architectures? In this case, I think we'd have to define a particular binary format that needs to be used (e.g. the blobs are per definition always little-endian), and then the code would have to be updated to always respect that format. Or, instead, are those blobs intended to be machine-specific? In this case, at the least the tests need to be made machine specific as well.

Note that in general, there may of course be other issues apart from endianness when attempting to re-use binary blobs across machines, e.g. pointer and/or word size, or ABI struct alignment rules. To define an actually portable format, all of that would have to be addressed.

[rriddle]

In D126446#3639047, @uweigand wrote:

This commit has broken the mlir builder on s390x-ibm-linux: https://lab.llvm.org/buildbot/#/builders/199/builds/6417

The immediate error is this failing assertion:

mlir-opt: /home/uweigand/llvm/llvm-head/mlir/test/lib/Dialect/Test/TestDialect.cpp:191: {anonymous}::TestOpAsmInterface::parseResource(mlir::AsmParsedResourceEntry&) const::<lambda(unsigned int, unsigned int)>: Assertion `align == alignof(uint64_t) && "unexpected data alignment"' failed.

which occurs because the value of align is 0x08000000 instead of the expected 8. The alignment was taken from the binary blob, encoded in little-endian format, but decoded as uint32_t on the host platform, which in our case is big-endian.

Now this is easily fixed by using proper endian conversion when accessing the alignment, which fixes the assertion, but then the test cases still fail because they expect little-endian data in the generated blobs, while they will actually contain big-endian data when generated on a big-endian machine.

What is the intended design point here? Are those binary blobs intended to be portable across machine architectures? In this case, I think we'd have to define a particular binary format that needs to be used (e.g. the blobs are per definition always little-endian), and then the code would have to be updated to always respect that format. Or, instead, are those blobs intended to be machine-specific? In this case, at the least the tests need to be made machine specific as well.

Note that in general, there may of course be other issues apart from endianness when attempting to re-use binary blobs across machines, e.g. pointer and/or word size, or ABI struct alignment rules. To define an actually portable format, all of that would have to be addressed.

Thanks for bringing this up. I think for the sake of the textual format, we should just always use little endian (this matches what DenseElementsAttr stores as in hex form as well). For the binary format of MLIR, depending on what we want to do in the long term, we could craft something more intricate so that blobs are more optimal on non-little endian (so we wouldn't need to do conversions). I'm also happy with extending the API/serialized data for blobs here to cover any information necessary for endian conversions/etc.

[mehdi_amini]

Can we XFAIL the test on this platform in the meantime?

[mehdi_amini]

(bring the bot back to green should be a priority)

[uweigand]

We have to fix the alignment tag for real, otherwise there will be failing assertions all over the place. With that fix in, there's still a failing test case - that one could indeed be XFAILed for now.

I've submitted a patch to implement this here: https://reviews.llvm.org/D129483

mlir/include/mlir/IR/AsmState.h

-//===- AsmState.h - State class for AsmPrinter ------------------*- C++ -*-===//
+//===- AsmState.h - Assembly State Utilities --------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file defines the AsmState class.
+// This file defines various classes and utilites for interacting with the MLIR
+// assembly formats.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef MLIR_IR_ASMSTATE_H_
 #define MLIR_IR_ASMSTATE_H_
 
 #include "mlir/IR/OperationSupport.h"
 #include "mlir/Support/LLVM.h"
 
 #include <memory>
 
 namespace mlir {
+class AsmConfigPrinter;
 class Operation;
 
 namespace detail {
 class AsmStateImpl;
 } // namespace detail
 
+//===----------------------------------------------------------------------===//
+// Config
+//===----------------------------------------------------------------------===//
+
+/// The following classes enable support for parsing and printing configurations
+/// within MLIR assembly formats. Configurations are a mechanism by which
+/// dialects, and external clients, may attach additional information when
+/// parsing/printing IR without that information being encoded in the IR itself.
+/// Configurations are not uniqued within the MLIR context, are not attached
+/// directly to any operation, and are solely intended to live and be processed
+/// outside of the immediate IR.
+///
+/// External configurations are encoded using a key-value pair nested within
+/// dictionaries anchored either on a dialect, or an externally registered
+/// entity. Dictionaries anchored on dialects use the dialect namespace
+/// directly, and dictionaries anchored on external entities anchor on an
+/// identifier wrapped within `<>`. The configuration key is an identifier used
+/// to disambiguate the data. The configuration value may be stored in various
+/// limited forms, but general encodings use a string (human readable) or blob
+/// format (binary). Within the textual format, an example may be of the form:
+///
+/// {-#
+///   // The `config` section within the file-level metadata dictionary is used
+///   // to contain any configuration entries.
+///   config: {
+///     // Here is a dictionary anchored on "mlir_reproducer", which is an
+///     // external entity representing MLIR's crash reproducer functionality.
+///     // External entity anchors are wrapped in `<>` to differentiate them
+///     // from dialect names.
+///     <mlir_reproducer>: {
+///       // `pipeline` is an entry that holds a crash reproducer pipeline
+///       // configuration.
+///       pipeline: "func.func(canonicalize,cse)"
+///     },
+///     // Here is a dictionary anchored on "foo_dialect", which is a dialect
+///     // namespace.
+///     foo_dialect: {
+///       // `some_dialect_config` is a key to be interpreted by the dialect,
+///       // and used to initialize/configure/etc.
+///       some_dialect_config: "Some important config value"
+///     }
+///   }
+/// #-}
+///
+
+//===----------------------------------------------------------------------===//
+// Config Entry
+
+/// This class is used to build config entries for the printer. Each config
+/// entry is represented using a key/value pair. The provided key must be unique
+/// within the current context, which allows for a client to provide
+/// configuration entries without worrying about overlap with other clients.
+class AsmConfigBuilder {
+public:
+  virtual ~AsmConfigBuilder();
+
+  /// Build an config entry represented by the given bool.
+  virtual void buildBool(StringRef key, bool data) = 0;
+
+  /// Build an config entry represented by the given human-readable string
+  /// value.
+  virtual void buildString(StringRef key, StringRef data) = 0;
+
+  /// Build an config entry represented by the given binary blob data.
+  virtual void buildBlob(StringRef key, ArrayRef<char> data,
+                         unsigned dataAlignent) = 0;
+  /// A useful overload if the data type is known. Note that this does not
+  /// support `char` element types to avoid accidentally not providing the
+  /// expected alignment of data in situations that treat blobs generically.
+  template <typename T>
+  std::enable_if_t<!std::is_same<T, char>::value> buildBlob(StringRef key,
+                                                            ArrayRef<T> data) {
+    buildBlob(
+        key, ArrayRef<char>((const char *)data.data(), data.size() * sizeof(T)),
+        alignof(T));
+  }
+};
+
+/// This class represents a single parsed config entry.
+class AsmConfigEntry {
+public:
+  virtual ~AsmConfigEntry();
+
+  /// Return the key of the config entry.
+  virtual StringRef getKey() const = 0;
+
+  /// Emit an error at the location of this entry.
+  virtual InFlightDiagnostic emitError() const = 0;
+
+  /// Parse the config entry represented by a boolean. Returns failure if the
+  /// entry does not correspond to a bool.
+  virtual FailureOr<bool> parseAsBool() const = 0;
+
+  /// Parse the config entry represented by a human-readable string. Returns
+  /// failure if the entry does not correspond to a string.
+  virtual FailureOr<std::string> parseAsString() const = 0;
+
+  /// The type of an allocator function used to allocate memory for a blob when
+  /// required. The function is provided a size and alignment, and should return
+  /// an aligned allocation buffer.
+  using BlobAllocatorFn =
+      function_ref<MutableArrayRef<char>(unsigned size, unsigned align)>;
+
+  /// This class represents a processed blob of data.
+  struct Blob {
+    /// The raw, properly aligned, blob data.
+    MutableArrayRef<char> data;
+
+    /// A flag indicating if data was allocated using the provided allocation
+    /// function, false if the data did not need to be allocated (e.g. such as
+    /// when mmap'd).
+    bool dataWasAllocated = false;
+  };
+
+  /// Parse the config entry represented by a binary blob. Returns failure if
+  /// the entry does not correspond to a blob. If the blob needed to be
+  /// allocated, the given allocator function is invoked.
+  virtual FailureOr<Blob> parseAsBlob(BlobAllocatorFn allocator) const = 0;
+};
+
+//===----------------------------------------------------------------------===//
+// Config
+
+/// This class represents an instance of a configuration parser. This class
+/// should be implemented by non-dialect clients that want to inject
+/// additional configurations into MLIR assembly formats.
+class AsmConfigParser {
+public:
+  /// Create a new parser with the given identifying name. This name uniquely
+  /// identifies the entries of this parser, and differentiates them from other
+  /// contexts.
+  AsmConfigParser(StringRef name) : name(name.str()) {}
+  virtual ~AsmConfigParser();
+
+  /// Return the name of this parser.
+  StringRef getName() const { return name; }
+
+  /// Parse the given configuration entry. Returns failure if the key/data were
+  /// not valid, or could otherwise not be processed correctly. Any necessary
+  /// errors should be emitted with the provided entry.
+  virtual LogicalResult parseConfig(AsmConfigEntry &entry) = 0;
+
+  /// Return an configuration parser implemented via the given callable, whose
+  /// form should match that of `parseConfig` above.
+  template <typename CallableT>
+  static std::unique_ptr<AsmConfigParser> fromCallable(StringRef name,
+                                                       CallableT &&parseFn) {
+    struct Processor : public AsmConfigParser {
+      Processor(StringRef name, CallableT &&parseFn)
+          : AsmConfigParser(name), parseFn(std::move(parseFn)) {}
+      LogicalResult parseConfig(AsmConfigEntry &entry) override {
+        return parseFn(entry);
+      }
+
+      std::decay_t<CallableT> parseFn;
+    };
+    return std::make_unique<Processor>(name, std::forward<CallableT>(parseFn));
+  }
+
+private:
+  std::string name;
+};
+
+/// This class represents an instance of a configuration printer. This class
+/// should be implemented by non-dialect clients that want to inject additional
+/// configurations into MLIR assembly formats.
+class AsmConfigPrinter {
+public:
+  /// Create a new printer with the given identifying name. This name uniquely
+  /// identifies the entries of this printer, and differentiates them from
+  /// other contexts.
+  AsmConfigPrinter(StringRef name) : name(name.str()) {}
+  virtual ~AsmConfigPrinter();
+
+  /// Return the name of this printer.
+  StringRef getName() const { return name; }
+
+  /// Build any configurations to include during printing, utilizing the given
+  /// top-level root operation to help determine what information to include.
+  /// Provided data should be registered in the form of a key/data pair, to the
+  /// given builder.
+  virtual void buildConfigs(Operation *op, AsmConfigBuilder &builder) const = 0;
+
+  /// Return an configuration printer implemented via the given callable, whose
+  /// form should match that of `buildConfigs` above.
+  template <typename CallableT>
+  static std::unique_ptr<AsmConfigPrinter> fromCallable(StringRef name,
+                                                        CallableT &&printFn) {
+    struct Printer : public AsmConfigPrinter {
+      Printer(StringRef name, CallableT &&printFn)
+          : AsmConfigPrinter(name), printFn(std::move(printFn)) {}
+      void buildConfigs(Operation *op,
+                        AsmConfigBuilder &builder) const override {
+        printFn(op, builder);
+      }
+
+      std::decay_t<CallableT> printFn;
+    };
+    return std::make_unique<Printer>(name, std::forward<CallableT>(printFn));
+  }
+
+private:
+  std::string name;
+};
+
+//===----------------------------------------------------------------------===//
+// AsmState
+//===----------------------------------------------------------------------===//
+
 /// This class provides management for the lifetime of the state used when
 /// printing the IR. It allows for alleviating the cost of recomputing the
 /// internal state of the asm printer.
 ///
 /// The IR should not be mutated in-between invocations using this state, and
 /// the IR being printed must not be an parent of the IR originally used to
 /// initialize this state. This means that if a child operation is provided, a
 /// parent operation cannot reuse this state.
public:
 
   /// Get the printer flags.
   const OpPrintingFlags &getPrinterFlags() const;
 
   /// Return an instance of the internal implementation. Returns nullptr if the
   /// state has not been initialized.
   detail::AsmStateImpl &getImpl() { return *impl; }
 
+  //===--------------------------------------------------------------------===//
+  // Config
+  //===--------------------------------------------------------------------===//
+
+  /// Attach the given configuration printer to the AsmState.
+  void attachConfigPrinter(std::unique_ptr<AsmConfigPrinter> printer);
+
+  /// Attach an configuration printer, in the form of a callable, to the
+  /// AsmState.
+  template <typename CallableT>
+  std::enable_if_t<std::is_convertible<
+      CallableT, function_ref<void(Operation *, AsmConfigBuilder &)>>::value>
+  attachConfigPrinter(StringRef name, CallableT &&printFn) {
+    attachConfigPrinter(
+        AsmConfigPrinter::fromCallable(name, std::forward<CallableT>(printFn)));
+  }
+
 private:
   AsmState() = delete;
 
   /// A pointer to allocated storage for the impl state.
   std::unique_ptr<detail::AsmStateImpl> impl;
 };
 
 //===----------------------------------------------------------------------===//

mlir/include/mlir/IR/OpImplementation.h

 
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/DialectInterface.h"
 #include "mlir/IR/OpDefinition.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/SMLoc.h"
 
 namespace mlir {
-
+class AsmConfigEntry;
+class AsmConfigBuilder;
 class Builder;
 
 //===----------------------------------------------------------------------===//
 // AsmPrinter
 //===----------------------------------------------------------------------===//
 
 /// This base class exposes generic asm printer hooks, usable across the various
 /// derived printers.
 class AsmPrinter {
 public:
   /// This class contains the internal default implementation of the base
   /// printer methods.
   class Impl;
 
   /// Initialize the printer with the given internal implementation.
   AsmPrinter(Impl &impl) : impl(&impl) {}
public:
   /// Trait to check if `AttrType` provides a `print` method.
   template <typename AttrOrType>
   using has_print_method =
       decltype(std::declval<AttrOrType>().print(std::declval<AsmPrinter &>()));
   template <typename AttrOrType>
   using detect_has_print_method =
       llvm::is_detected<has_print_method, AttrOrType>;
 
   /// Print the provided attribute in the context of an operation custom
   /// printer/parser: this will invoke directly the print method on the
   /// attribute class and skip the `#dialect.mnemonic` prefix in most cases.
   template <typename AttrOrType,
             std::enable_if_t<detect_has_print_method<AttrOrType>::value>
                 *sfinae = nullptr>
   void printStrippedAttrOrType(AttrOrType attrOrType) {
     if (succeeded(printAlias(attrOrType)))
       return;

 
 /// A functor used to set the name of blocks in regions directly nested under
 /// an operation.
 using OpAsmSetBlockNameFn = function_ref<void(Block *, StringRef)>;
 
 class OpAsmDialectInterface
     : public DialectInterface::Base<OpAsmDialectInterface> {
 public:
+  OpAsmDialectInterface(Dialect *dialect) : Base(dialect) {}
+
+  //===------------------------------------------------------------------===//
+  // Aliases
+  //===------------------------------------------------------------------===//
+
   /// Holds the result of `getAlias` hook call.
   enum class AliasResult {
     /// The object (type or attribute) is not supported by the hook
     /// and an alias was not provided.
     NoAlias,
     /// An alias was provided, but it might be overriden by other hook.
     OverridableAlias,
     /// An alias was provided and it should be used
     /// (no other hooks will be checked).
     FinalAlias
   };
 
-  OpAsmDialectInterface(Dialect *dialect) : Base(dialect) {}
-
   /// Hooks for getting an alias identifier alias for a given symbol, that is
   /// not necessarily a part of this dialect. The identifier is used in place of
   /// the symbol when printing textual IR. These aliases must not contain `.` or
   /// end with a numeric digit([0-9]+).
   virtual AliasResult getAlias(Attribute attr, raw_ostream &os) const {
     return AliasResult::NoAlias;
   }
   virtual AliasResult getAlias(Type type, raw_ostream &os) const {
     return AliasResult::NoAlias;
   }
+
+  //===------------------------------------------------------------------===//
+  // Config
+  //===------------------------------------------------------------------===//
+
+  /// Hook for parsing configurations entries. Returns failure if the entry was
+  /// not valid, or could otherwise not be processed correctly. Any necessary
+  /// errors can be emitted via the provided entry.
+  virtual LogicalResult parseConfig(AsmConfigEntry &entry) const;
+
+  /// Hook for building configurations to use during printing. The given
+  /// top-level root operation may be inspected to help determine what
+  /// information to include.
+  virtual void buildConfigs(Operation *op, AsmConfigBuilder &builder) const {}
 };
 } // namespace mlir
 
 //===--------------------------------------------------------------------===//
 // Operation OpAsm interface.
 //===--------------------------------------------------------------------===//
 
 /// The OpAsmOpInterface, see OpAsmInterface.td for more details.
 #include "mlir/IR/OpAsmInterface.h.inc"
 
 #endif

mlir/include/mlir/Parser/Parser.h

 //===- Parser.h - MLIR Parser Library Interface -----------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is contains the interface to the MLIR parser library.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef MLIR_PARSER_PARSER_H
 #define MLIR_PARSER_PARSER_H
 
+#include "mlir/IR/AsmState.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/BuiltinOps.h"
 #include <cstddef>
 
 namespace llvm {
 class SourceMgr;
 class SMDiagnostic;
 class StringRef;
public:
   }
 
   /// Return the MLIRContext to be used when parsing.
   MLIRContext *getContext() const { return context; }
 
   /// Return asm parser state to be used when parsing.
   AsmParserState *getAsmParserState() const { return asmState; }
 
+  /// Return the configuration parser registered to the given name, or nullptr
+  /// if no parser with `name` is registered.
+  AsmConfigParser *getConfigParser(StringRef name) const {
+    auto it = configParsers.find(name);
+    return it == configParsers.end() ? nullptr : it->second.get();
+  }
+
+  /// Attach the given configuration parser.
+  void attachConfigParser(std::unique_ptr<AsmConfigParser> parser) {
+    StringRef name = parser->getName();
+    auto it = configParsers.try_emplace(name, std::move(parser));
+    assert(it.second && "config parser already registered with the given name");
+  }
+
+  /// Attach the given callable configuration parser with the given name.
+  template <typename CallableT>
+  std::enable_if_t<std::is_convertible<
+      CallableT, function_ref<LogicalResult(AsmConfigEntry &)>>::value>
+  attachConfigParser(StringRef name, CallableT &&parserFn) {
+    attachConfigParser(
+        AsmConfigParser::fromCallable(name, std::forward<CallableT>(parserFn)));
+  }
+
 private:
   MLIRContext *context;
   AsmParserState *asmState;
+  DenseMap<StringRef, std::unique_ptr<AsmConfigParser>> configParsers;
 };
 
 /// This parses the file specified by the indicated SourceMgr and appends parsed
 /// operations to the given block. If the block is non-empty, the operations are
 /// placed before the current terminator. If parsing is successful, success is
 /// returned. Otherwise, an error message is emitted through the error handler
 /// registered in the context, and failure is returned. If `sourceFileLoc` is
 /// non-null, it is populated with a file location representing the start of the

mlir/lib/IR/AsmPrinter.cpp

 
 //===----------------------------------------------------------------------===//
 // Operation OpAsm interface.
 //===----------------------------------------------------------------------===//
 
 /// The OpAsmOpInterface, see OpAsmInterface.td for more details.
 #include "mlir/IR/OpAsmInterface.cpp.inc"
 
+LogicalResult OpAsmDialectInterface::parseConfig(AsmConfigEntry &entry) const {
+  return entry.emitError() << "unknown 'config' key '" << entry.getKey()
+                           << "' for dialect '" << getDialect()->getNamespace()
+                           << "'";
+}
+
 //===----------------------------------------------------------------------===//
 // OpPrintingFlags
 //===----------------------------------------------------------------------===//
 
 namespace {
 /// This struct contains command line options that can be used to initialize
 /// various bits of the AsmPrinter. This uses a struct wrapper to avoid the need
 /// for global command line options.
if (!usedNames.count(name)) {
     }
   }
 
   usedNames.insert(name, char());
   return name;
 }
 
 //===----------------------------------------------------------------------===//
+// External Config
+//===----------------------------------------------------------------------===//
+
+AsmConfigEntry::~AsmConfigEntry() = default;
+AsmConfigBuilder::~AsmConfigBuilder() = default;
+AsmConfigParser::~AsmConfigParser() = default;
+AsmConfigPrinter::~AsmConfigPrinter() = default;
+
+//===----------------------------------------------------------------------===//
 // AsmState
 //===----------------------------------------------------------------------===//
 
 namespace mlir {
 namespace detail {
 class AsmStateImpl {
 public:
   explicit AsmStateImpl(Operation *op, const OpPrintingFlags &printerFlags,
                         AsmState::LocationMap *locationMap)
       : interfaces(op->getContext()), nameState(op, printerFlags),
         printerFlags(printerFlags), locationMap(locationMap) {}
 
   /// Initialize the alias state to enable the printing of aliases.
   void initializeAliases(Operation *op) {
     aliasState.initialize(op, printerFlags, interfaces);
   }
 
   /// Get the state used for aliases.
   AliasState &getAliasState() { return aliasState; }
 
   /// Get the state used for SSA names.
   SSANameState &getSSANameState() { return nameState; }
 
+  /// Return the dialects within the context that implement
+  /// OpAsmDialectInterface.
+  DialectInterfaceCollection<OpAsmDialectInterface> &getDialectInterfaces() {
+    return interfaces;
+  }
+
+  /// Return the non-dialect config printers.
+  auto getConfigPrinters() {
+    return llvm::make_pointee_range(externalConfigPrinters);
+  }
+
   /// Get the printer flags.
   const OpPrintingFlags &getPrinterFlags() const { return printerFlags; }
 
   /// Register the location, line and column, within the buffer that the given
   /// operation was printed at.
   void registerOperationLocation(Operation *op, unsigned line, unsigned col) {
     if (locationMap)
       (*locationMap)[op] = std::make_pair(line, col);
   }
 
 private:
   /// Collection of OpAsm interfaces implemented in the context.
   DialectInterfaceCollection<OpAsmDialectInterface> interfaces;
 
+  /// A collection of non-dialect configuration printers.
+  SmallVector<std::unique_ptr<AsmConfigPrinter>> externalConfigPrinters;
+
   /// The state used for attribute and type aliases.
   AliasState aliasState;
 
   /// The state used for SSA value names.
   SSANameState nameState;
 
   /// Flags that control op output.
   OpPrintingFlags printerFlags;
 
   /// An optional location map to be populated.
   AsmState::LocationMap *locationMap;
+
+  // Allow direct access to the impl fields.
+  friend AsmState;
 };
 } // namespace detail
 } // namespace mlir
 
 /// Verifies the operation and switches to generic op printing if verification
 /// fails. We need to do this because custom print functions may fail for
 /// invalid ops.
 static OpPrintingFlags verifyOpAndAdjustFlags(Operation *op,
AsmState::AsmState(Operation *op, const OpPrintingFlags &printerFlags,
     : impl(std::make_unique<AsmStateImpl>(
           op, verifyOpAndAdjustFlags(op, printerFlags), locationMap)) {}
 AsmState::~AsmState() = default;
 
 const OpPrintingFlags &AsmState::getPrinterFlags() const {
   return impl->getPrinterFlags();
 }
 
+void AsmState::attachConfigPrinter(std::unique_ptr<AsmConfigPrinter> printer) {
+  impl->externalConfigPrinters.emplace_back(std::move(printer));
+}
+
 //===----------------------------------------------------------------------===//
 // AsmPrinter::Impl
 //===----------------------------------------------------------------------===//
 
 namespace mlir {
 class AsmPrinter::Impl {
 public:
   Impl(raw_ostream &os, OpPrintingFlags flags = llvm::None,
public:
   /// Print the users of a value.
   void printValueUsers(Value value);
 
   /// Print either the ids of the result values or the id of the operation if
   /// the operation has no results.
   void printUserIDs(Operation *user, bool prefixComma = false);
 
 private:
+  /// This class represents a config builder implementation for the MLIR textual
+  /// assembly format.
+  class ConfigBuilder : public AsmConfigBuilder {
+  public:
+    using ValueFn = function_ref<void(raw_ostream &)>;
+    using PrintFn = function_ref<void(StringRef, ValueFn)>;
+
+    ConfigBuilder(OperationPrinter &p, PrintFn printFn)
+        : p(p), printFn(printFn) {}
+    ~ConfigBuilder() override = default;
+
+    void buildBool(StringRef key, bool data) final {
+      printFn(key, [&](raw_ostream &os) { p.os << (data ? "true" : "false"); });
+    }
+
+    void buildString(StringRef key, StringRef data) final {
+      printFn(key, [&](raw_ostream &os) { p.printEscapedString(data); });
+    }
+
+    void buildBlob(StringRef key, ArrayRef<char> data,
+                   unsigned dataAlignment) final {
+      printFn(key, [&](raw_ostream &os) {
+        // Store the blob in a hex string containing the alignment and the data.
+        os << "\"0x"
+           << llvm::toHex(StringRef(reinterpret_cast<char *>(&dataAlignment),
+                                    sizeof(dataAlignment)))
+           << llvm::toHex(StringRef(data.data(), data.size())) << "\"";
+      });
+    }
+
+  private:
+    OperationPrinter &p;
+    PrintFn printFn;
+  };
+
+  /// Print the metadata dictionary for the file, eliding it if it is empty.
+  void printFileMetadataDictionary(Operation *op);
+
+  /// Print the config section for the file metadata dictionary.
+  /// `checkAddMetadataDict` is used to indicate that metadata is going to be
+  /// added, and the file metadata dictionary should be started if it hasn't
+  /// yet.
+  void printConfigFileMetadata(function_ref<void()> checkAddMetadataDict,
+                               Operation *op);
+
   // Contains the stack of default dialects to use when printing regions.
   // A new dialect is pushed to the stack before parsing regions nested under an
   // operation implementing `OpAsmOpInterface`, and popped when done. At the
   // top-level we start with "builtin" as the default, so that the top-level
   // `module` operation prints as-is.
   SmallVector<StringRef> defaultDialectStack{"builtin"};
 
   /// The number of spaces used for indenting nested operations.
void OperationPrinter::printTopLevelOperation(Operation *op) {
   state->getAliasState().printNonDeferredAliases(os, newLine);
 
   // Print the module.
   print(op);
   os << newLine;
 
   // Output the aliases at the top level that can be deferred.
   state->getAliasState().printDeferredAliases(os, newLine);
+
+  // Output any file level metadata.
+  printFileMetadataDictionary(op);
+}
+
+void OperationPrinter::printFileMetadataDictionary(Operation *op) {
+  bool sawMetadataEntry = false;
+  auto checkAddMetadataDict = [&] {
+    if (!std::exchange(sawMetadataEntry, true))
+      os << newLine << "{-#" << newLine;
+  };
+
+  // Add the various types of metadata.
+  printConfigFileMetadata(checkAddMetadataDict, op);
+
+  // If the file dictionary exists, close it.
+  if (sawMetadataEntry)
+    os << newLine << "#-}" << newLine;
+}
+
+void OperationPrinter::printConfigFileMetadata(
+    function_ref<void()> checkAddMetadataDict, Operation *op) {
+  // Functor used to add data entries to the file metadata dictionary.
+  bool hadConfig = false;
+  auto processProvider = [&](const Twine &name, auto &provider) {
+    bool hadEntry = false;
+    auto printFn = [&](StringRef key, ConfigBuilder::ValueFn valueFn) {
+      checkAddMetadataDict();
+
+      // Emit the top-level config entry if we haven't yet.
+      if (!std::exchange(hadConfig, true))
+        os << "  config: {" << newLine;
+      // Emit the parent config entry if we haven't yet.
+      if (!std::exchange(hadEntry, true))
+        os << "    " << name << ": {" << newLine;
+      else
+        os << "," << newLine;
+
+      os << "      " << key << ": ";
+      valueFn(os);
+    };
+    ConfigBuilder entryBuilder(*this, printFn);
+    provider.buildConfigs(op, entryBuilder);
+
+    if (hadEntry)
+      os << newLine << "    }";
+  };
+
+  // Check each of the dialects for config data.
+  for (const OpAsmDialectInterface &interface : state->getDialectInterfaces())
+    processProvider(interface.getDialect()->getNamespace(), interface);
+  // Check for any non-dialect providers.
+  for (const auto &printer : state->getConfigPrinters())
+    processProvider("<" + printer.getName() + ">", printer);
+
+  if (hadConfig)
+    os << newLine << "  }";
 }
 
 /// Print a block argument in the usual format of:
 ///   %ssaName : type {attr1=42} loc("here")
 /// where location printing is controlled by the standard internal option.
 /// You may pass omitType=true to not print a type, and pass an empty
 /// attribute list if you don't care for attributes.
 void OperationPrinter::printRegionArgument(BlockArgument arg,

mlir/lib/Parser/AsmParserImpl.h

if (!parser.getToken().is(Token::string))
       return failure();
 
     if (string)
       *string = parser.getToken().getStringValue();
     parser.consumeToken();
     return success();
   }
 
-  /// Returns true if the current token corresponds to a keyword.
-  bool isCurrentTokenAKeyword() const {
-    return parser.getToken().isAny(Token::bare_identifier, Token::inttype) ||
-           parser.getToken().isKeyword();
-  }
-
   /// Parse the given keyword if present.
   ParseResult parseOptionalKeyword(StringRef keyword) override {
     // Check that the current token has the same spelling.
-    if (!isCurrentTokenAKeyword() || parser.getTokenSpelling() != keyword)
+    if (!parser.isCurrentTokenAKeyword() ||
+        parser.getTokenSpelling() != keyword)
       return failure();
     parser.consumeToken();
     return success();
   }
 
   /// Parse a keyword, if present, into 'keyword'.
   ParseResult parseOptionalKeyword(StringRef *keyword) override {
     // Check that the current token is a keyword.
-    if (!isCurrentTokenAKeyword())
+    if (!parser.isCurrentTokenAKeyword())
       return failure();
 
     *keyword = parser.getTokenSpelling();
     parser.consumeToken();
     return success();
   }
 
   /// Parse a keyword if it is one of the 'allowedKeywords'.
   ParseResult
   parseOptionalKeyword(StringRef *keyword,
                        ArrayRef<StringRef> allowedKeywords) override {
     // Check that the current token is a keyword.
-    if (!isCurrentTokenAKeyword())
+    if (!parser.isCurrentTokenAKeyword())
       return failure();
 
     StringRef currentKeyword = parser.getTokenSpelling();
     if (llvm::is_contained(allowedKeywords, currentKeyword)) {
       *keyword = currentKeyword;
       parser.consumeToken();
       return success();
     }

mlir/lib/Parser/Lexer.cpp

case ',':
       return formToken(Token::comma, tokStart);
     case '.':
       return lexEllipsis(tokStart);
     case '(':
       return formToken(Token::l_paren, tokStart);
     case ')':
       return formToken(Token::r_paren, tokStart);
     case '{':
+      if (*curPtr == '-' && *(curPtr + 1) == '#') {
+        curPtr += 2;
+        return formToken(Token::file_metadata_begin, tokStart);
+      }
       return formToken(Token::l_brace, tokStart);
     case '}':
       return formToken(Token::r_brace, tokStart);
     case '[':
       return formToken(Token::l_square, tokStart);
     case ']':
       return formToken(Token::r_square, tokStart);
     case '<':
case '/':
         skipComment();
         continue;
       }
       return emitError(tokStart, "unexpected character");
 
     case '@':
       return lexAtIdentifier(tokStart);
 
-    case '!':
-      LLVM_FALLTHROUGH;
-    case '^':
-      LLVM_FALLTHROUGH;
     case '#':
+      if (*curPtr == '-' && *(curPtr + 1) == '}') {
+        curPtr += 2;
+        return formToken(Token::file_metadata_end, tokStart);
+      }
       LLVM_FALLTHROUGH;
+    case '!':
+    case '^':
     case '%':
       return lexPrefixedIdentifier(tokStart);
     case '"':
       return lexString(tokStart);
 
     case '0':
     case '1':
     case '2':

mlir/lib/Parser/Parser.h

public:
   OptionalParseResult parseOptionalInteger(APInt &result);
 
   /// Parse a floating point value from an integer literal token.
   ParseResult parseFloatFromIntegerLiteral(Optional<APFloat> &result,
                                            const Token &tok, bool isNegative,
                                            const llvm::fltSemantics &semantics,
                                            size_t typeSizeInBits);
 
+  /// Returns true if the current token corresponds to a keyword.
+  bool isCurrentTokenAKeyword() const {
+    return getToken().isAny(Token::bare_identifier, Token::inttype) ||
+           getToken().isKeyword();
+  }
+
+  /// Parse a keyword, if present, into 'keyword'.
+  ParseResult parseOptionalKeyword(StringRef *keyword);
+
   //===--------------------------------------------------------------------===//
   // Type Parsing
   //===--------------------------------------------------------------------===//
 
   /// Invoke the `getChecked` method of the given Attribute or Type class, using
   /// the provided location to emit errors in the case of failure. Note that
   /// unlike `OpBuilder::getType`, this method does not implicitly insert a
   /// context parameter.

mlir/lib/Parser/Parser.cpp

 //===- Parser.cpp - MLIR Parser Implementation ----------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the parser for the MLIR textual form.
 //
 //===----------------------------------------------------------------------===//
 
 #include "Parser.h"
 #include "AsmParserImpl.h"
 #include "mlir/IR/AffineMap.h"
+#include "mlir/IR/AsmState.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/Dialect.h"
 #include "mlir/IR/Verifier.h"
 #include "mlir/Parser/AsmParserState.h"
 #include "mlir/Parser/Parser.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/StringSet.h"
ParseResult Parser::parseFloatFromIntegerLiteral(
   APInt apInt(typeSizeInBits, *value);
   if (apInt != *value)
     return emitError(loc, "hexadecimal float constant out of range for type");
   result = APFloat(semantics, apInt);
 
   return success();
 }
 
+ParseResult Parser::parseOptionalKeyword(StringRef *keyword) {
+  // Check that the current token is a keyword.
+  if (!isCurrentTokenAKeyword())
+    return failure();
+
+  *keyword = getTokenSpelling();
+  consumeToken();
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // OperationParser
 //===----------------------------------------------------------------------===//
 
 namespace {
 /// This class provides support for parsing operations and regions of
 /// operations.
 class OperationParser : public Parser {
 public:
   OperationParser(ParserState &state, ModuleOp topLevelOp);
   ~OperationParser();
 
   /// After parsing is finished, this function must be called to see if there
   /// are any remaining issues.
   ParseResult finalize();
 
   //===--------------------------------------------------------------------===//
   // SSA Value Handling
   //===--------------------------------------------------------------------===//
 
   using UnresolvedOperand = OpAsmParser::UnresolvedOperand;

 public:
   explicit TopLevelOperationParser(ParserState &state) : Parser(state) {}
 
   /// Parse a set of operations into the end of the given Block.
   ParseResult parse(Block *topLevelBlock, Location parserLoc);
 
 private:
   /// Parse an attribute alias declaration.
+  ///
+  ///   attribute-alias-def ::= '#' alias-name `=` attribute-value
+  ///
   ParseResult parseAttributeAliasDef();
 
-  /// Parse an attribute alias declaration.
+  /// Parse a type alias declaration.
+  ///
+  ///   type-alias-def ::= '!' alias-name `=` type
+  ///
   ParseResult parseTypeAliasDef();
-};
-} // namespace
 
-/// Parses an attribute alias declaration.
+  /// Parse a top-level file metadata dictionary.
   ///
-///   attribute-alias-def ::= '#' alias-name `=` attribute-value
+  ///   file-metadata-dict ::= '{-#' file-metadata-entry* `#-}'
   ///
+  ParseResult parseFileMetadataDictionary();
+
+  /// Parse an instance of the 'config' file metadata.
+  ParseResult parseConfigFileMetadata();
+};
+
+/// This class represents an implementation of a config entry for the MLIR
+/// textual format.
+class ParsedConfigEntry : public AsmConfigEntry {
+public:
+  ParsedConfigEntry(StringRef key, SMLoc keyLoc, Token value, Parser &p)
+      : key(key), keyLoc(keyLoc), value(value), p(p) {}
+  ~ParsedConfigEntry() override = default;
+
+  StringRef getKey() const final { return key; }
+
+  InFlightDiagnostic emitError() const final { return p.emitError(keyLoc); }
+
+  FailureOr<bool> parseAsBool() const final {
+    if (value.is(Token::kw_true))
+      return true;
+    if (value.is(Token::kw_false))
+      return false;
+    return p.emitError(value.getLoc(),
+                       "expected 'true' or 'false' value for key '" + key +
+                           "'");
+  }
+
+  FailureOr<std::string> parseAsString() const final {
+    if (value.isNot(Token::string))
+      return p.emitError(value.getLoc(),
+                         "expected string value for key '" + key + "'");
+    return value.getStringValue();
+  }
+
+  FailureOr<Blob> parseAsBlob(BlobAllocatorFn allocator) const final {
+    // Blob data within then textual format is represented as a hex string.
+    // TODO: We could avoid an additional alloc+copy here if we pre-allocated
+    // the buffer to use during hex processing.
+    Optional<std::string> blobData =
+        value.is(Token::string) ? value.getHexStringValue() : llvm::None;
+    if (!blobData)
+      return p.emitError(value.getLoc(),
+                         "expected hex string blob for key '" + key + "'");
+
+    // Extract the alignment of the blob data, which gets stored at the
+    // beginning of the string.
+    if (blobData->size() < sizeof(unsigned)) {
+      return p.emitError(value.getLoc(),
+                         "expected hex string blob for key '" + key +
+                             "' to encode alignment in first 4 bytes");
+    }
+    unsigned align = 0;
+    memcpy(&align, blobData->data(), sizeof(unsigned));
+
+    // Get the data portion of the blob.
+    StringRef data = StringRef(*blobData).drop_front(sizeof(unsigned));
+    if (data.empty())
+      return Blob{llvm::None, /*dataWasAllocated=*/false};
+
+    // Allocate memory for the blob using the provided allocator and copy the
+    // data into it.
+    Blob blob = {allocator(data.size(), align), /*dataWasAllocated=*/true};
+    assert(llvm::isAddrAligned(llvm::Align(align), blob.data.data()) &&
+           "blob allocator did not return a properly aligned address");
+    memcpy(blob.data.data(), data.data(), data.size());
+    return blob;
+  }
+
+private:
+  StringRef key;
+  SMLoc keyLoc;
+  Token value;
+  Parser &p;
+};
+} // namespace
+
 ParseResult TopLevelOperationParser::parseAttributeAliasDef() {
   assert(getToken().is(Token::hash_identifier));
   StringRef aliasName = getTokenSpelling().drop_front();
 
   // Check for redefinitions.
   if (state.symbols.attributeAliasDefinitions.count(aliasName) > 0)
     return emitError("redefinition of attribute alias id '" + aliasName + "'");
 
ParseResult TopLevelOperationParser::parseAttributeAliasDef() {
   Attribute attr = parseAttribute();
   if (!attr)
     return failure();
 
   state.symbols.attributeAliasDefinitions[aliasName] = attr;
   return success();
 }
 
-/// Parse a type alias declaration.
-///
-///   type-alias-def ::= '!' alias-name `=` type
-///
 ParseResult TopLevelOperationParser::parseTypeAliasDef() {
   assert(getToken().is(Token::exclamation_identifier));
   StringRef aliasName = getTokenSpelling().drop_front();
 
   // Check for redefinitions.
   if (state.symbols.typeAliasDefinitions.count(aliasName) > 0)
     return emitError("redefinition of type alias id '" + aliasName + "'");
 
ParseResult TopLevelOperationParser::parseTypeAliasDef() {
   if (!aliasedType)
     return failure();
 
   // Register this alias with the parser state.
   state.symbols.typeAliasDefinitions.try_emplace(aliasName, aliasedType);
   return success();
 }
 
+ParseResult TopLevelOperationParser::parseFileMetadataDictionary() {
+  consumeToken(Token::file_metadata_begin);
+  return parseCommaSeparatedListUntil(
+      Token::file_metadata_end, [&]() -> ParseResult {
+        // Parse the key of the metadata dictionary.
+        SMLoc keyLoc = getToken().getLoc();
+        StringRef key;
+        if (failed(parseOptionalKeyword(&key)))
+          return emitError("expected identifier key in file "
+                           "metadata dictionary");
+        if (parseToken(Token::colon, "expected ':'"))
+          return failure();
+
+        // Process the metadata entry.
+        if (key == "config")
+          return parseConfigFileMetadata();
+        return emitError(keyLoc, "unknown key '" + key +
+                                     "' in file metadata dictionary");
+      });
+}
+
+ParseResult TopLevelOperationParser::parseConfigFileMetadata() {
+  if (parseToken(Token::l_brace, "expected '{'"))
+    return failure();
+
+  return parseCommaSeparatedListUntil(Token::r_brace, [&]() -> ParseResult {
+    // Parse the name, which is either 'dialect-name' or '<external-name>'.
+    bool isExternalKey = consumeIf(Token::less);
+    SMLoc nameLoc = getToken().getLoc();
+    StringRef name;
+    if (failed(parseOptionalKeyword(&name)))
+      return emitError("expected identifier name for 'config' entry");
+    if (isExternalKey && parseToken(Token::greater, "expected '>'"))
+      return failure();
+    if (parseToken(Token::colon, "expected ':'") ||
+        parseToken(Token::l_brace, "expected '{'"))
+      return failure();
+
+    PointerUnion<const OpAsmDialectInterface *, AsmConfigParser *> handler;
+    if (isExternalKey) {
+      handler = state.config.getConfigParser(name);
+    } else {
+      // Lookup the dialect and check that it can handle the configuration
+      // entry.
+      Dialect *dialect = getContext()->getOrLoadDialect(name);
+      if (!dialect)
+        return emitError(nameLoc, "dialect '" + name + "' is unknown");
+
+      handler = dyn_cast<OpAsmDialectInterface>(dialect);
+      if (!handler) {
+        return emitError() << "unexpected 'config' section for dialect '"
+                           << dialect->getNamespace() << "'";
+      }
+    }
+
+    return parseCommaSeparatedListUntil(Token::r_brace, [&]() -> ParseResult {
+      SMLoc keyLoc = getToken().getLoc();
+      StringRef key;
+      if (failed(parseOptionalKeyword(&key)))
+        return emitError("expected identifier key for 'config' entry");
+      if (parseToken(Token::colon, "expected ':'"))
+        return failure();
+      Token valueTok = getToken();
+      consumeToken();
+
+      // FIXME: What is our policy on unknown configurations? The current code
+      // asserts that dialect configs can be handler, but external configs don't
+      // need to be. We should define a set policy here.
+      if (!handler)
+        return success();
+
+      ParsedConfigEntry entry(key, keyLoc, valueTok, *this);
+      if (const auto *iface = handler.dyn_cast<const OpAsmDialectInterface *>())
+        return iface->parseConfig(entry);
+      auto *processor = handler.get<AsmConfigParser *>();
+      return processor->parseConfig(entry);
+    });
+  });
+}
+
 ParseResult TopLevelOperationParser::parse(Block *topLevelBlock,
                                            Location parserLoc) {
   // Create a top-level operation to contain the parsed state.
   OwningOpRef<ModuleOp> topLevelOp(ModuleOp::create(parserLoc));
   OperationParser opParser(state, topLevelOp.get());
   while (true) {
     switch (getToken().getKind()) {
     default:
case Token::hash_identifier:
         return failure();
       break;
 
     // Parse a type alias.
     case Token::exclamation_identifier:
       if (parseTypeAliasDef())
         return failure();
       break;
+
+      // Parse a file-level metadata dictionary.
+    case Token::file_metadata_begin:
+      if (parseFileMetadataDictionary())
+        return failure();
+      break;
     }
   }
 }
 
 //===----------------------------------------------------------------------===//
 
 LogicalResult mlir::parseSourceFile(const llvm::SourceMgr &sourceMgr,
                                     Block *block, const AsmParserConfig &config,

mlir/lib/Parser/Token.cpp

 /// Given a token containing a hex string literal, return its value or None if
 /// the token does not contain a valid hex string.
 Optional<std::string> Token::getHexStringValue() const {
   assert(getKind() == string);
 
   // Get the internal string data, without the quotes.
   StringRef bytes = getSpelling().drop_front().drop_back();
 
-  // Try to extract the binary data from the hex string.
+  // Try to extract the binary data from the hex string. We expect the hex
+  // string to start with `0x` and have an even number of hex nibbles (nibbles
+  // should come in pairs).
   std::string hex;
-  if (!bytes.consume_front("0x") || !llvm::tryGetFromHex(bytes, hex))
+  if (!bytes.consume_front("0x") || (bytes.size() & 1) ||
+      !llvm::tryGetFromHex(bytes, hex))
     return llvm::None;
   return hex;
 }
 
 /// Given a token containing a symbol reference, return the unescaped string
 /// value.
 std::string Token::getSymbolReference() const {
   assert(is(Token::at_identifier) && "expected valid @-identifier");

mlir/lib/Parser/TokenKinds.def

 TOK_PUNCTUATION(plus, "+")
 TOK_PUNCTUATION(question, "?")
 TOK_PUNCTUATION(r_brace, "}")
 TOK_PUNCTUATION(r_paren, ")")
 TOK_PUNCTUATION(r_square, "]")
 TOK_PUNCTUATION(star, "*")
 TOK_PUNCTUATION(vertical_bar, "|")
 
+TOK_PUNCTUATION(file_metadata_begin, "{-#")
+TOK_PUNCTUATION(file_metadata_end, "#-}")
+
 // Keywords.  These turn "foo" into Token::kw_foo enums.
 
 // NOTE: Please key these alphabetized to make it easier to find something in
 // this list and to cater to OCD.
 TOK_KEYWORD(affine_map)
 TOK_KEYWORD(affine_set)
 TOK_KEYWORD(attributes)
 TOK_KEYWORD(bf16)

mlir/test/IR/elements-attr-interface.mlir

 // expected-error@below {{Test iterating `IntegerAttr`: unable to iterate type}}
 arith.constant opaque<"_", "0xDEADBEEF"> : tensor<5xi64>
 
 // Check that we don't crash on empty element attributes.
 // expected-error@below {{Test iterating `uint64_t`: }}
 // expected-error@below {{Test iterating `APInt`: }}
 // expected-error@below {{Test iterating `IntegerAttr`: }}
 arith.constant dense<> : tensor<0xi64>
+
+// Check that we handle an external constant parsed from the config.
+// expected-error@below {{Test iterating `uint64_t`: 1, 2, 3}}
+// expected-error@below {{Test iterating `APInt`: unable to iterate type}}
+// expected-error@below {{Test iterating `IntegerAttr`: unable to iterate type}}
+arith.constant #test.e1di64_elements<"blob1"> : tensor<3xi64>
+
+{-#
+  config: {
+    test: {
+      blob1: "0x08000000010000000000000002000000000000000300000000000000"
+    }
+  }
+#-}

mlir/test/IR/file-metadata-config.mlir

+// RUN: mlir-opt %s -split-input-file | FileCheck %s
+
+// Check that we only preserve the blob that got referenced.
+// CHECK:      test: {
+// CHECK-NEXT:   blob1: "0x08000000010000000000000002000000000000000300000000000000"
+// CHECK-NEXT: }
+
+module attributes { test.blob_ref = #test.e1di64_elements<"blob1"> } {}
+
+{-#
+  config: {
+    test: {
+      blob1: "0x08000000010000000000000002000000000000000300000000000000",
+      blob2: "0x08000000040000000000000005000000000000000600000000000000"
+    }
+  }
+#-}

mlir/test/IR/invalid-file-metadata.mlir

+// RUN: mlir-opt %s -split-input-file -verify-diagnostics
+
+// expected-error@+2 {{expected identifier key in file metadata dictionary}}
+{-#
+
+// -----
+
+// expected-error@+2 {{expected ':'}}
+{-#
+  key
+#-}
+
+// -----
+
+// expected-error@+2 {{unknown key 'some_key' in file metadata dictionary}}
+{-#
+  some_key: {}
+#-}
+
+// -----
+
+//===----------------------------------------------------------------------===//
+// `config`
+//===----------------------------------------------------------------------===//
+
+// expected-error@+2 {{expected '{'}}
+{-#
+  config: "value"
+#-}
+
+// -----
+
+// expected-error@+3 {{expected identifier name for 'config' entry}}
+{-#
+  config: {
+    10
+  }
+#-}
+
+// -----
+
+// expected-error@+3 {{expected '>'}}
+{-#
+  config: {
+    <entry "value"
+  }
+#-}
+
+// -----
+
+// expected-error@+3 {{expected ':'}}
+{-#
+  config: {
+    entry "value"
+  }
+#-}
+
+// -----
+
+// expected-error@+3 {{dialect 'foobar' is unknown}}
+{-#
+  config: {
+    foobar: {
+      entry: "foo"
+    }
+  }
+#-}
+
+// -----
+
+// expected-error@+4 {{unknown 'config' key 'unknown_entry' for dialect 'builtin'}}
+{-#
+  config: {
+    builtin: {
+      unknown_entry: "foo"
+    }
+  }
+#-}
+
+// -----
+
+// expected-error@+4 {{expected hex string blob for key 'invalid_blob'}}
+{-#
+  config: {
+    test: {
+      invalid_blob: 10
+    }
+  }
+#-}
+
+// -----
+
+// expected-error@+4 {{expected hex string blob for key 'invalid_blob'}}
+{-#
+  config: {
+    test: {
+      invalid_blob: ""
+    }
+  }
+#-}
+
+// -----
+
+// expected-error@+4 {{expected hex string blob for key 'invalid_blob' to encode alignment in first 4 bytes}}
+{-#
+  config: {
+    test: {
+      invalid_blob: "0x"
+    }
+  }
+#-}

mlir/test/lib/Dialect/Test/TestAttrDefs.td

 def TestAttrSelfTypeParameterFormat
     : Test_Attr<"TestAttrSelfTypeParameterFormat"> {
   let parameters = (ins "int":$a, AttributeSelfTypeParameter<"">:$type);
 
   let mnemonic = "attr_self_type_format";
   let assemblyFormat = "`<` $a `>`";
 }
 
+// Test simple extern 1D vector using ElementsAttrInterface.
+def TestExtern1DI64ElementsAttr : Test_Attr<"TestExtern1DI64Elements", [
+    ElementsAttrInterface
+  ]> {
+  let mnemonic = "e1di64_elements";
+  let parameters = (ins
+    AttributeSelfTypeParameter<"", "::mlir::ShapedType">:$type,
+
+    // A string key to the data, which is actually stored within the
+    // dialect.
+    StringRefParameter<>:$key
+  );
+  let extraClassDeclaration = [{
+    /// Return the elements referenced by this attribute.
+    llvm::ArrayRef<uint64_t> getElements() const;
+
+    /// The set of data types that can be iterated by this attribute.
+    using ContiguousIterableTypesT = std::tuple<uint64_t>;
+
+    /// Provide begin iterators for the various iterable types.
+    // * uint64_t
+    auto value_begin_impl(OverloadToken<uint64_t>) const {
+      return getElements().begin();
+    }
+  }];
+  let assemblyFormat = "`<` $key `>`";
+}
+
 #endif // TEST_ATTRDEFS

mlir/test/lib/Dialect/Test/TestAttributes.cpp

case 2:
       third = it.second;
       break;
     }
   }
   return get(getContext(), first, second, third);
 }
 
 //===----------------------------------------------------------------------===//
+// TestExtern1DI64ElementsAttr
+//===----------------------------------------------------------------------===//
+
+ArrayRef<uint64_t> TestExtern1DI64ElementsAttr::getElements() const {
+  auto &mgr = static_cast<TestDialect &>(getDialect()).getExternalDataManager();
+  const TestExternalElementsData *data = mgr.getData(getKey());
+
+  // TODO: If ElementsAttr had failable accessors, we could gracefully handle
+  // this case.
+  assert(data && "expected external data to be registered");
+  return data->data;
+}
+
+//===----------------------------------------------------------------------===//
 // Tablegen Generated Definitions
 //===----------------------------------------------------------------------===//
 
 #include "TestAttrInterfaces.cpp.inc"
 
 #define GET_ATTRDEF_CLASSES
 #include "TestAttrDefs.cpp.inc"
 

mlir/test/lib/Dialect/Test/TestDialect.h

 #include "mlir/Interfaces/SideEffectInterfaces.h"
 #include "mlir/Interfaces/ViewLikeInterface.h"
 
 namespace mlir {
 class DLTIDialect;
 class RewritePatternSet;
 } // namespace mlir
 
+//===----------------------------------------------------------------------===//
+// External Elements Data
+//===----------------------------------------------------------------------===//
+
+namespace test {
+/// This class represents a single external elements instance. It keeps track of
+/// the data, and handles deallocation when necessary.
+struct TestExternalElementsData {
+  TestExternalElementsData(llvm::MutableArrayRef<uint64_t> data,
+                           bool dataIsOwned)
+      : data(data), dataIsOwned(dataIsOwned) {}
+  ~TestExternalElementsData() {
+    if (dataIsOwned)
+      free(data.data());
+  }
+
+  /// The raw underlying data for the attribute.
+  llvm::MutableArrayRef<uint64_t> data;
+  /// A boolean indicating if the data is owned and should be freed.
+  bool dataIsOwned;
+};
+
+/// This class acts as a manager for external elements data. It provides API
+/// for creating and accessing registered elements data.
+class TestExternalElementsDataManager {
+public:
+  /// Return the data registered for the given name, or nullptr if no data is
+  /// registered.
+  const TestExternalElementsData *getData(llvm::StringRef name) const {
+    auto it = dataMap.find(name);
+    return it != dataMap.end() ? &*it->second : nullptr;
+  }
+
+  /// Register the provided data with the given name. Asserts that no data is
+  /// already registered with the name.
+  void insertData(llvm::StringRef name,
+                  std::unique_ptr<TestExternalElementsData> data) {
+    auto it = dataMap.try_emplace(name, std::move(data));
+    (void)it;
+    assert(it.second && "data already registered");
+  }
+
+private:
+  llvm::StringMap<std::unique_ptr<TestExternalElementsData>> dataMap;
+};
+} // namespace test
+
+//===----------------------------------------------------------------------===//
+// TestDialect
+//===----------------------------------------------------------------------===//
+
 #include "TestOpInterfaces.h.inc"
 #include "TestOpStructs.h.inc"
 #include "TestOpsDialect.h.inc"
 
 #define GET_OP_CLASSES
 #include "TestOps.h.inc"
 
 namespace test {
 void registerTestDialect(::mlir::DialectRegistry &registry);
 void populateTestReductionPatterns(::mlir::RewritePatternSet &patterns);
 } // namespace test
 
 #endif // MLIR_TESTDIALECT_H

mlir/test/lib/Dialect/Test/TestDialect.cpp

 //===- TestDialect.cpp - MLIR Dialect for Testing -------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "TestDialect.h"
 #include "TestAttributes.h"
 #include "TestInterfaces.h"
 #include "TestTypes.h"
 #include "mlir/Dialect/Arithmetic/IR/Arithmetic.h"
 #include "mlir/Dialect/DLTI/DLTI.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
+#include "mlir/IR/AsmState.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/DialectImplementation.h"
 #include "mlir/IR/ExtensibleDialect.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/IR/Verifier.h"
 #include "mlir/Reducer/ReductionPatternInterface.h"
 #include "mlir/Transforms/FoldUtils.h"
static_assert(OpTrait::hasSingleBlockImplicitTerminator<
                   SingleBlockImplicitTerminatorOp>::value,
               "hasSingleBlockImplicitTerminator does not match "
               "SingleBlockImplicitTerminatorOp");
 
 // Test support for interacting with the AsmPrinter.
 struct TestOpAsmInterface : public OpAsmDialectInterface {
   using OpAsmDialectInterface::OpAsmDialectInterface;
 
+  //===------------------------------------------------------------------===//
+  // Aliases
+  //===------------------------------------------------------------------===//
+
   AliasResult getAlias(Attribute attr, raw_ostream &os) const final {
     StringAttr strAttr = attr.dyn_cast<StringAttr>();
     if (!strAttr)
       return AliasResult::NoAlias;
 
     // Check the contents of the string attribute to see what the test alias
     // should be named.
     Optional<StringRef> aliasName =
if (auto intType = type.dyn_cast<TestIntegerType>()) {
               TestIntegerType::SignednessSemantics::Unsigned &&
           intType.getWidth() == 8) {
         os << "test_ui8";
         return AliasResult::FinalAlias;
       }
     }
     return AliasResult::NoAlias;
   }
+
+  //===------------------------------------------------------------------===//
+  // Config
+  //===------------------------------------------------------------------===//
+
+  LogicalResult parseConfig(AsmConfigEntry &entry) const final {
+    TestDialect *dialect = cast<TestDialect>(getDialect());
+    TestExternalElementsDataManager &mgr = dialect->getExternalDataManager();
+
+    // The config entries are external constant data.
+    auto blobAllocFn = [](unsigned size, unsigned align) {
+      assert(align == alignof(uint64_t) && "unexpected data alignment");
+      return MutableArrayRef<char>((char *)malloc(size), size);
+    };
+    FailureOr<AsmConfigEntry::Blob> blob = entry.parseAsBlob(blobAllocFn);
+    if (failed(blob))
+      return failure();
+
+    MutableArrayRef<uint64_t> data((uint64_t *)blob->data.data(),
+                                   blob->data.size() / sizeof(uint64_t));
+    mgr.insertData(entry.getKey(), std::make_unique<TestExternalElementsData>(
+                                       data, blob->dataWasAllocated));
+    return success();
+  }
+
+  void buildConfigs(Operation *op, AsmConfigBuilder &provider) const final {
+    SetVector<TestExtern1DI64ElementsAttr> usedExternalData;
+    op->walk([&](Operation *op) {
+      for (NamedAttribute attr : op->getAttrs())
+        if (auto data = attr.getValue().dyn_cast<TestExtern1DI64ElementsAttr>())
+          usedExternalData.insert(data);
+    });
+    for (TestExtern1DI64ElementsAttr data : usedExternalData)
+      provider.buildBlob(data.getKey(), data.getElements());
+  }
 };
 
 struct TestDialectFoldInterface : public DialectFoldInterface {
   using DialectFoldInterface::DialectFoldInterface;
 
   /// Registered hook to check if the given region, which is attached to an
   /// operation that is *not* isolated from above, should be used when
   /// materializing constants.

mlir/test/lib/Dialect/Test/TestDialect.td

::llvm::Optional<ParseOpHook>
       getParseOperationHook(::llvm::StringRef opName) const override;
     ::llvm::unique_function<void(::mlir::Operation *,
                                  ::mlir::OpAsmPrinter &printer)>
      getOperationPrinter(::mlir::Operation *op) const override;
 
      ::mlir::Type parseType(::mlir::DialectAsmParser &parser) const override;
      void printType(::mlir::Type type,
                     ::mlir::DialectAsmPrinter &printer) const override;
+
+    /// Returns the external elements data manager for this dialect.
+    TestExternalElementsDataManager &getExternalDataManager() {
+      return externalDataManager;
+    }
+
   private:
     // Storage for a custom fallback interface.
     void *fallbackEffectOpInterfaces;
 
     ::mlir::Type parseTestType(::mlir::AsmParser &parser,
                                ::llvm::SetVector<::mlir::Type> &stack) const;
     void printTestType(::mlir::Type type, ::mlir::AsmPrinter &printer,
                        ::llvm::SetVector<::mlir::Type> &stack) const;
+
+    /// An external data manager used to test external elements data.
+    TestExternalElementsDataManager externalDataManager;
   }];
 }
 
 #endif // TEST_DIALECT