Add an MLIR example of some mimimal example of MLIR setup

[mehdi_amini]

Summary

These may serve as example and showcase of the MLIR binary footprint.

Right now a release build of these tools on a linux machine shows:

• mlir-cat: 2MB This includes the Core IR, the builtin dialect, the textual parser/printer, the support for bytecode.
• mlir-minimal-opt: 3MB This adds all the tooling for an mlir-opt tool: the pass infrastructure and all the instrumentation associated with it.
• mlir-miminal-opt-canonicalize: 4.8MB This add the canonicalizer pass, which pulls in all the pattern/rewrite machinery, including the PDL compiler and intepreter.

[mehdi_amini]

update readme

[mehdi_amini]

Tweak readme

[Mogball]

Nice. This looks useful

[rriddle]

mlir/examples/minimal-opt/mlir-minimal-opt-canonicalize.cpp

1

Missed an update here?

mlir/examples/minimal-opt/mlir-minimal-opt.cpp

2

Missed an update here?

[jpienaar]

What compile config were used in measured sizes?

[mehdi_amini]

mlir/examples/minimal-opt/mlir-cat.cpp

1

You missed this one River ;)

[Meinersbur]

I think this broke the Windows build: https://lab.llvm.org/buildbot/#/builders/172/builds/29434. To investigate the issue, I took the buildbot offline to investigate. The error is:

[1/2] Building RC object tools\mlir\examples\minimal-opt\CMakeFiles\mlir-minimal-opt-can...zen-win\flang-x86_64-windows\llvm-project\llvm\resources\windows_version_resource.rc.res
FAILED: tools/mlir/examples/minimal-opt/CMakeFiles/mlir-minimal-opt-canonicalize.dir/C_/Users/buildbot-worker/minipc-ryzen-win/flang-x86_64-windows/llvm-project/llvm/resources/windows_version_resource.rc.res
C:/PROGRA~1/CMake/bin/cmcldeps.exe RC C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\llvm\resources\windows_version_resource.rc tools\mlir\examples\minimal-opt\CMakeFiles\mlir-minimal-opt-canonicalize.dir\C_\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\llvm\resources\windows_version_resource.rc.res.d tools\mlir\examples\minimal-opt\CMakeFiles\mlir-minimal-opt-canonicalize.dir\C_\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\llvm\resources\windows_version_resource.rc.res "Note: including file: " "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x64/cl.exe" C:\PROGRA~2\WI3CF2~1\10\bin\100220~1.0\x64\rc.exe -DGTEST_HAS_RTTI=0 -DMLIR_CUDA_CONVERSIONS_ENABLED=0 -DMLIR_INCLUDE_TESTS -DMLIR_ROCM_CONVERSIONS_ENABLED=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_GLIBCXX_ASSERTIONS -D_HAS_EXCEPTIONS=0 -D_LIBCPP_ENABLE_HARDENED_MODE -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DRC_FILE_VERSION=\"0\" -DRC_INTERNAL_NAME=\"mlir-minimal-opt-canonicalize\" -DRC_PRODUCT_NAME=\"LLVM\" -DRC_PRODUCT_VERSION=\"0\" -DRC_VERSION_FIELD_1=18 -DRC_VERSION_FIELD_2=0 -DRC_VERSION_FIELD_3=0 -DRC_VERSION_FIELD_4=0 -I C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\build\tools\mlir\examples\minimal-opt -I C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\mlir\examples\minimal-opt -I C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\build\include -I C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\llvm\include -I C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\mlir\include -I C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\build\tools\mlir\include -DWIN32 /nologo /fo tools\mlir\examples\minimal-opt\CMakeFiles\mlir-minimal-opt-canonicalize.dir\C_\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\llvm\resources\windows_version_resource.rc.res C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\llvm\resources\windows_version_resource.rc
C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\llvm\resources\windows_version_resource.rc: fatal error C1083: Cannot open compiler generated file: 'windows_version_resource.i': No such file or directory
ninja: build stopped: subcommand failed.`

The issue is that C:\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\build\tools\mlir\examples\minimal-opt\CMakeFiles\mlir-minimal-opt-canonicalize.dir\C_\Users\buildbot-worker\minipc-ryzen-win\flang-x86_64-windows\llvm-project\llvm\resources\windows_version_resource.i exceeds MAX_PATH and msvc cl.exe seems to not support long paths yet.

CMake appends the full path of source files if they do not lie in the CMakeLists.txt's source directory (at other times it is a relative path with __ for ..) so you get the LLVM source directory twice in the build directory. CMake has a workaround CMAKE_OBJECT_PATH_MAX. That seems to work when sufficiently low (to account for appended filenames such as windows_version_resource.i), but I also get lots of CMake warnings:

CMake Warning in examples/OrcV2Examples/LLJITWithObjectLinkingLayerPlugin/CMakeLists.txt:
  The object file directory

    C:/Users/buildbot-worker/minipc-ryzen-win/flang-x86_64-windows/build/examples/OrcV2Examples/LLJITWithObjectLinkingLayerPlugin/CMakeFiles/LLJITWithObjectLinkingLayerPlugin.dir/./

  has 177 characters.  The maximum full path to an object file is 200
  characters (see CMAKE_OBJECT_PATH_MAX).  Object file

    LLJITWithObjectLinkingLayerPlugin.cpp.obj

  cannot be safely placed under this directory.  The build may not work
  correctly.

At the moment if feels safest/easiest to shorten the build directory for my buildbot. Putting the report here for others to find.

---

NB. minimal-opt/CMakeLists.txt compiles three executables in the same directory, all of them get the same llvm/resources/windows_version_resource.rc added to it. That's itself not a problem, but AddLLVM.txt also set set_property(SOURCE ${resource_file} COMPILE_DEFINITIONS RC_INTERNAL_NAME=${name}) with different ${name} for each executable, but CMake only stores one COMPILE_DEFINITIONS per source file and add_subdirectory. Therefore, all but one executable get incorrect version information.

mlir/examples/CMakeLists.txt

 add_subdirectory(toy)
 add_subdirectory(transform)
+add_subdirectory(minimal-opt)

mlir/examples/minimal-opt/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Support
+  )
+
+
+set(LIBS
+  MLIRParser
+  MLIRSupport
+  MLIRIR
+)
+
+add_mlir_tool(mlir-cat
+  mlir-cat.cpp
+  PARTIAL_SOURCES_INTENDED
+
+  DEPENDS
+  ${LIBS}
+  )
+target_link_libraries(mlir-cat PRIVATE ${LIBS})
+llvm_update_compile_flags(mlir-cat)
+mlir_check_all_link_libraries(mlir-cat)
+
+list(APPEND LIBS
+  MLIROptLib
+  MLIRPass
+)
+add_mlir_tool(mlir-minimal-opt
+  mlir-minimal-opt.cpp
+  PARTIAL_SOURCES_INTENDED
+
+  DEPENDS
+  ${LIBS}
+  )
+target_link_libraries(mlir-minimal-opt PRIVATE ${LIBS})
+llvm_update_compile_flags(mlir-minimal-opt)
+mlir_check_all_link_libraries(mlir-minimal-opt)
+
+
+list(APPEND LIBS
+  MLIROptLib
+  MLIRPass
+  MLIRTransforms
+)
+add_mlir_tool(mlir-minimal-opt-canonicalize
+  mlir-minimal-opt-canonicalize.cpp
+  PARTIAL_SOURCES_INTENDED
+
+  DEPENDS
+  ${LIBS}
+  )
+target_link_libraries(mlir-minimal-opt-canonicalize PRIVATE ${LIBS})
+llvm_update_compile_flags(mlir-minimal-opt-canonicalize)
+mlir_check_all_link_libraries(mlir-minimal-opt-canonicalize)
+

mlir/examples/minimal-opt/README.md

+# Minmal MLIR binaries
+
+This folder contains example of minimal MLIR setup that can showcase the
+intended binary footprint of the framework.
+
+- mlir-cat: ~2MB
+  This includes the Core IR, the builtin dialect, the textual parser/printer,
+  the support for bytecode serialization.
+- mlir-minimal-opt: ~3MB
+  This adds all the tooling for an mlir-opt tool: the pass infrastructure
+  and all the instrumentation associated with it.
+- mlir-miminal-opt-canonicalize: ~4.8MB
+  This add the canonicalizer pass, which pulls in all the pattern/rewrite
+  machinery, including the PDL compiler and intepreter.
 No newline at end of file

mlir/examples/minimal-opt/mlir-cat.cpp

+//===- standalone-opt.cpp ---------------------------------------*- C++ -*-===//
+//
+// This file is licensed under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/IR/MLIRContext.h"
+#include "mlir/Parser/Parser.h"
+#include "mlir/Support/FileUtilities.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Support/LogicalResult.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/ToolOutputFile.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace mlir;
+
+/// This example parse its input, either from a file or its standard input (in
+/// bytecode or textual assembly) and print it back.
+
+int main(int argc, char **argv) {
+  // Set up the input file.
+  StringRef inputFile;
+  if (argc <= 1) {
+    llvm::errs() << "Reading from stdin...\n";
+    inputFile = "-";
+  } else {
+    inputFile = argv[1];
+  }
+  std::string errorMessage;
+  auto file = openInputFile(inputFile, &errorMessage);
+  if (!file) {
+    llvm::errs() << errorMessage << "\n";
+    exit(1);
+  }
+  llvm::SourceMgr sourceMgr;
+  sourceMgr.AddNewSourceBuffer(std::move(file), SMLoc());
+
+  auto output = openOutputFile("-", &errorMessage);
+  if (!output) {
+    llvm::errs() << errorMessage << "\n";
+    exit(1);
+  }
+
+  DialectRegistry registry;
+  MLIRContext context(registry, MLIRContext::Threading::DISABLED);
+  context.allowUnregisteredDialects(true);
+  OwningOpRef<Operation *> op = parseSourceFile(sourceMgr, &context);
+  if (!op) {
+    llvm::errs() << "Failed to parse input file";
+    exit(1);
+  }
+  output->os() << *(op.get()) << "\n";
+}

mlir/examples/minimal-opt/mlir-minimal-opt-canonicalize.cpp

+//===- standalone-opt.cpp ---------------------------------------*- C++ -*-===//
+//
+// This file is licensed under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Tools/mlir-opt/MlirOptMain.h"
+#include "mlir/Transforms/Passes.h"
+
+int main(int argc, char **argv) {
+  // Register only the canonicalize pass
+  // This pulls in the pattern rewrite engine as well as the whole PDL
+  // compiler/intepreter.
+  mlir::registerCanonicalizerPass();
+
+  mlir::DialectRegistry registry;
+  return mlir::asMainReturnCode(mlir::MlirOptMain(
+      argc, argv, "Minimal Standalone optimizer driver\n", registry));
+}

mlir/examples/minimal-opt/mlir-minimal-opt.cpp

+//===- standalone-opt.cpp ---------------------------------------*- C++ -*-===//
+//
+// This file is licensed under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Tools/mlir-opt/MlirOptMain.h"
+
+/// This test includes the minimal amount of components for mlir-opt, that is
+/// the CoreIR, the printer/parser, the bytecode reader/writer, the
+/// passmanagement infrastructure and all the instrumentation.
+int main(int argc, char **argv) {
+  mlir::DialectRegistry registry;
+  return mlir::asMainReturnCode(mlir::MlirOptMain(
+      argc, argv, "Minimal Standalone optimizer driver\n", registry));
+}

mlir/test/CMakeLists.txt

if(LLVM_BUILD_EXAMPLES)
   list(APPEND MLIR_TEST_DEPENDS
     toyc-ch1
     toyc-ch2
     toyc-ch3
     toyc-ch4
     toyc-ch5
     transform-opt-ch2
     transform-opt-ch3
+    mlir-minimal-opt
     )
   if(MLIR_ENABLE_EXECUTION_ENGINE)
     list(APPEND MLIR_TEST_DEPENDS
       toyc-ch6
       toyc-ch7
     )
   endif()
 endif()