[ms] [llvm-ml] Add placeholder for llvm-ml, based on llvm-mc

[epastor]

Summary

As discussed on the mailing list, I plan to introduce an ml-compatible MASM assembler as part of providing more of the Windows build tools. This will be similar to llvm-mc, but with different command-line parameters.

This placeholder is purely a stripped-down version of llvm-mc; we'll eventually add support for the Microsoft-style command-line flags, and back it with a MASM parser.

[merge_guards_bot]

Unit tests: pass. 61799 tests passed, 0 failed and 781 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[thakis]

Nice! Looks like a good start.

It might be nice to add a test that just runs llvm-ml --help to check that the binary runs.

I'd imagine that once this is mostly done, we want to busy-box into clang-cl so that clang-cl file.asm runs the ml-style asm parser by default, and we'd have some -fasm-mode flag to override this (so that .s files can be parsed in ml mode and .asm files in gas mode, if desired). So far, this is all compatible with that, but something to keep in mind going forward.

llvm/tools/llvm-ml/CMakeLists.txt

2

You should probably add this in this commit as well :)

llvm/tools/llvm-ml/Disassembler.h

11

Is this comment up-to-date? Should probably mention ml?

llvm/tools/llvm-ml/llvm-ml.cpp

102

If you intend to add these soon, add "// FIXME: Enable once $condition", else remove commented-out code for now

125

Even more important than this flag is -fdebug-compilation-dir (ctrl-f "fdebug-compilation" in http://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html )

211

Hm!

Elaborate, or remove

[epastor]

Fixing most feedback; basic functionality test still pending.

[epastor]

llvm/tools/llvm-ml/Disassembler.h

11

This is accurate; Disassembler.h and Disassembler.cpp are identical to that in llvm-mc. The only difference is that when run as part of llvm-ml, it will be passed different parameters by default. Is there a way I can share the code between them instead?

llvm/tools/llvm-ml/llvm-ml.cpp

125

Already in place; see above.

211

Removed.

[merge_guards_bot]

Unit tests: pass. 61799 tests passed, 0 failed and 781 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[epastor]

Add basic tests, confirming ENOENT for a non-existent file and that --help shows a usage message.

[epastor]

In D72679#1821730, @thakis wrote:

It might be nice to add a test that just runs llvm-ml --help to check that the binary runs.

Done.

I'd imagine that once this is mostly done, we want to busy-box into clang-cl so that clang-cl file.asm runs the ml-style asm parser by default, and we'd have some -fasm-mode flag to override this (so that .s files can be parsed in ml mode and .asm files in gas mode, if desired). So far, this is all compatible with that, but something to keep in mind going forward.

Something like that, once this is mostly done... though I'm not familiar with what you mean by "busy-box into clang-cl". I don't think that should be too tricky.

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt

[thakis]

Looks fine to me as a first step, with more uncommented code and a not very actionable FIXME removed :)

Maybe wait until EOD to see if rnk wants to chime in too.

llvm/tools/llvm-ml/llvm-ml.cpp

288

?

358

?

llvm/utils/gn/secondary/llvm/tools/llvm-ml/BUILD.gn

13

?

[epastor]

Fix an inconsistency apparently introduced in an earlier rebase.

[epastor]

Removing some more commented-code artifacts from development

[merge_guards_bot]

Unit tests: pass. 61801 tests passed, 0 failed and 781 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[merge_guards_bot]

Unit tests: pass. 61801 tests passed, 0 failed and 781 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[epastor]

Fix formatting where reasonable

[merge_guards_bot]

Unit tests: pass. 61801 tests passed, 0 failed and 781 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[RKSimon]

@epastor This is causing build bots failures, please can you take a look? http://lab.llvm.org:8011/builders/llvm-clang-win-x-armv7l/builds/3121

[epastor]

In D72679#1828049, @RKSimon wrote:

@epastor This is causing build bots failures, please can you take a look? http://lab.llvm.org:8011/builders/llvm-clang-win-x-armv7l/builds/3121

Looks real to me - this entire tool is premised on the availability of the (i386|x86_64)-pc-windows targets, since it's intended to imitate ml.exe & ml64.exe. Apparently at least one isn't available when building in this environment? Any reason for that?

Also, would there be a way to keep this tool from building (and exclude it from tests) if those targets aren't available?

[epastor]

Quick note: I've reverted this change in commit 0eeddf1ac59066567a096ad95344f43c38e6b04f due to the build breakages on all ARM platforms.

@RKSimon If you have any advice for relanding this safely, it'd be much appreciated!

[RKSimon]

In D72679#1828062, @epastor wrote:

Quick note: I've reverted this change in commit 0eeddf1ac59066567a096ad95344f43c38e6b04f due to the build breakages on all ARM platforms.

@RKSimon If you have any advice for relanding this safely, it'd be much appreciated!

Can you replicate the cmake settings in http://lab.llvm.org:8011/builders/llvm-clang-win-x-aarch64/builds/3114/steps/cmake-configure/logs/stdio ? The issue seems in part due to the available targets/triples:

llvm-ml.exe: error: : error: unable to get target for 'x86_64-pc-windows-msvc', see --version and --triple.

[epastor]

In D72679#1828069, @RKSimon wrote:

In D72679#1828062, @epastor wrote:

Quick note: I've reverted this change in commit 0eeddf1ac59066567a096ad95344f43c38e6b04f due to the build breakages on all ARM platforms.

@RKSimon If you have any advice for relanding this safely, it'd be much appreciated!

Can you replicate the cmake settings in http://lab.llvm.org:8011/builders/llvm-clang-win-x-aarch64/builds/3114/steps/cmake-configure/logs/stdio ? The issue seems in part due to the available targets/triples:

llvm-ml.exe: error: : error: unable to get target for 'x86_64-pc-windows-msvc', see --version and --triple.

As mentioned in my earlier comment - I think the issue is ENTIRELY due to the available targets/triples, since this tool forces to the (i386|x86_64)-pc-windows-msvc targets/triples. It's designed to (eventually) mimic Microsoft's ml.exe. I can try to broaden it, but I'm concerned about functionality - MASM is a pretty specific assembly dialect. No chance you know a standard way to set it up so this tool won't build unless those triples are available?

[RKSimon]

You might be able to fix it with the equivalent of rG80146fc13ada

[epastor]

Tag non-trivial llvm-ml tests as requiring x86-registered-target, since llvm-ml's first draft only targets x86.

[epastor]

Rebasing past the reversion

[epastor]

In D72679#1828069, @RKSimon wrote:

Can you replicate the cmake settings in http://lab.llvm.org:8011/builders/llvm-clang-win-x-aarch64/builds/3114/steps/cmake-configure/logs/stdio ? The issue seems in part due to the available targets/triples:

llvm-ml.exe: error: : error: unable to get target for 'x86_64-pc-windows-msvc', see --version and --triple.

@RKSimon I replicated the issue as you suggested. With the added "REQUIRES" line, I can confirm it's correctly identified as unsupported. This shouldn't cause the same buildbot breakage anymore.

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt

[merge_guards_bot]

Unit tests: fail. 61977 tests passed, 1 failed and 783 were skipped.

failed: LLVM.Bindings/Go/go.test

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[RKSimon]

Thanks - you might want to fix the "newline at end of file" and clang-format warnings but other than that LGTM for re-commit. Cheers.

[epastor]

Fix missing newline at end of file

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt

[thakis]

Only supporting x86 targets for llvm-ml makes sense to me. As far as I know, msvc doesn't have an ml.exe for arm64 either (?).

llvm/test/tools/llvm-ml/basic.test

2

This is an ok way of doing this, but since all tests in this directory are going to need it, it's a bit nicer to instead have a file called lit.local.cfg in this directory that contains:

if not ('X86' in config.root.targets):
    # We need support for X86.
    config.unsupported = True

Then all the tests in this dir will only run if X86 is in LLVM_TARGETS_TO_BUILD.

[mstorsjo]

In D72679#1829526, @thakis wrote:

Only supporting x86 targets for llvm-ml makes sense to me. As far as I know, msvc doesn't have an ml.exe for arm64 either (?).

That's correct. For arm/arm64, they have armasm.exe and armasm64.exe, which consumes assembly in the ARM RVCT armasm format (which is different from gas format). (And even more for the record, if some cares, there's a perl script called gas-preprocessor available, which can do on-the-fly conversion from gas to armasm format, for at least some subsets of cleaned up arm assembly out there.)

[kamaub]

@epastor The lastest commit for this patch: r5f6dfa800e0b causes build failures on:
http://lab.llvm.org:8011/builders/clang-ppc64be-linux-multistage/builds/23101
http://lab.llvm.org:8011/builders/clang-ppc64be-linux/builds/43182
http://lab.llvm.org:8011/builders/clang-ppc64be-linux-lnt/builds/34405

[epastor]

In D72679#1832296, @kamaub wrote:

@epastor The lastest commit for this patch: r5f6dfa800e0b causes build failures on:
http://lab.llvm.org:8011/builders/clang-ppc64be-linux-multistage/builds/23101
http://lab.llvm.org:8011/builders/clang-ppc64be-linux/builds/43182
http://lab.llvm.org:8011/builders/clang-ppc64be-linux-lnt/builds/34405

@kamaub The build failures you see should have been fixed by the followup commit r6ccebe004446.

EDIT: On inspection, (for example) http://lab.llvm.org:8011/builders/clang-ppc64be-linux/builds/43185 shows that you're now seeing a different error that appears unrelated?

[kamaub]

In D72679#1832424, @epastor wrote:

In D72679#1832296, @kamaub wrote:

@epastor The lastest commit for this patch: r5f6dfa800e0b causes build failures on:
http://lab.llvm.org:8011/builders/clang-ppc64be-linux-multistage/builds/23101
http://lab.llvm.org:8011/builders/clang-ppc64be-linux/builds/43182
http://lab.llvm.org:8011/builders/clang-ppc64be-linux-lnt/builds/34405

@kamaub The build failures you see should have been fixed by the followup commit r6ccebe004446.

EDIT: On inspection, (for example) http://lab.llvm.org:8011/builders/clang-ppc64be-linux/builds/43185 shows that you're now seeing a different error that appears unrelated?

Thank you, I have confirmed that this is true.

llvm/test/CMakeLists.txt

set(LLVM_TEST_DEPENDS
           llvm-jitlink
           llvm-lib
           llvm-link
           llvm-lipo
           llvm-locstats
           llvm-lto2
           llvm-mc
           llvm-mca
+          llvm-ml
           llvm-modextract
           llvm-mt
           llvm-nm
           llvm-objcopy
           llvm-objdump
           llvm-opt-fuzzer
           llvm-opt-report
           llvm-pdbutil

llvm/test/tools/llvm-ml/basic.test

+# RUN: not llvm-ml %t.blah -o /dev/null 2>&1 | FileCheck --check-prefix=ENOENT %s
+
+# ENOENT: {{.*}}.blah: {{[Nn]}}o such file or directory

llvm/test/tools/llvm-ml/run.test

+# RUN: llvm-ml --help | FileCheck %s
+
+# CHECK: USAGE: llvm-ml
 No newline at end of file

llvm/tools/llvm-ml/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  AllTargetsAsmPrinters
+  AllTargetsAsmParsers
+  AllTargetsDescs
+  AllTargetsDisassemblers
+  AllTargetsInfos
+  MC
+  MCParser
+  Support
+  )
+
+add_llvm_tool(llvm-ml
+  llvm-ml.cpp
+  Disassembler.cpp
+  )

llvm/tools/llvm-ml/Disassembler.h

+//===- Disassembler.h - Text File Disassembler ----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This class implements the disassembler of strings of bytes written in
+// hexadecimal, from standard input or from a file.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_MC_DISASSEMBLER_H
+#define LLVM_TOOLS_LLVM_MC_DISASSEMBLER_H
+
+#include <string>
+
+namespace llvm {
+
+class MemoryBuffer;
+class Target;
+class raw_ostream;
+class SourceMgr;
+class MCSubtargetInfo;
+class MCStreamer;
+
+class Disassembler {
+public:
+  static int disassemble(const Target &T,
+                         const std::string &Triple,
+                         MCSubtargetInfo &STI,
+                         MCStreamer &Streamer,
+                         MemoryBuffer &Buffer,
+                         SourceMgr &SM,
+                         raw_ostream &Out);
+};
+
+} // namespace llvm
+
+#endif

llvm/tools/llvm-ml/Disassembler.cpp

+//===- Disassembler.cpp - Disassembler for hex strings --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This class implements the disassembler of strings of bytes written in
+// hexadecimal, from standard input or from a file.
+//
+//===----------------------------------------------------------------------===//
+
+#include "Disassembler.h"
+#include "llvm/ADT/Triple.h"
+#include "llvm/MC/MCAsmInfo.h"
+#include "llvm/MC/MCContext.h"
+#include "llvm/MC/MCDisassembler/MCDisassembler.h"
+#include "llvm/MC/MCInst.h"
+#include "llvm/MC/MCRegisterInfo.h"
+#include "llvm/MC/MCStreamer.h"
+#include "llvm/MC/MCSubtargetInfo.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/TargetRegistry.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace llvm;
+
+typedef std::pair<std::vector<unsigned char>, std::vector<const char *>>
+    ByteArrayTy;
+
+static bool PrintInsts(const MCDisassembler &DisAsm,
+                       const ByteArrayTy &Bytes,
+                       SourceMgr &SM, raw_ostream &Out,
+                       MCStreamer &Streamer, bool InAtomicBlock,
+                       const MCSubtargetInfo &STI) {
+  ArrayRef<uint8_t> Data(Bytes.first.data(), Bytes.first.size());
+
+  // Disassemble it to strings.
+  uint64_t Size;
+  uint64_t Index;
+
+  for (Index = 0; Index < Bytes.first.size(); Index += Size) {
+    MCInst Inst;
+
+    MCDisassembler::DecodeStatus S;
+    S = DisAsm.getInstruction(Inst, Size, Data.slice(Index), Index,
+                              /*REMOVE*/ nulls(), nulls());
+    switch (S) {
+    case MCDisassembler::Fail:
+      SM.PrintMessage(SMLoc::getFromPointer(Bytes.second[Index]),
+                      SourceMgr::DK_Warning,
+                      "invalid instruction encoding");
+      // Don't try to resynchronise the stream in a block
+      if (InAtomicBlock)
+        return true;
+
+      if (Size == 0)
+        Size = 1;  // skip illegible bytes
+
+      break;
+
+    case MCDisassembler::SoftFail:
+      SM.PrintMessage(SMLoc::getFromPointer(Bytes.second[Index]),
+                      SourceMgr::DK_Warning,
+                      "potentially undefined instruction encoding");
+      LLVM_FALLTHROUGH;
+
+    case MCDisassembler::Success:
+      Streamer.EmitInstruction(Inst, STI);
+      break;
+    }
+  }
+
+  return false;
+}
+
+static bool SkipToToken(StringRef &Str) {
+  for (;;) {
+    if (Str.empty())
+      return false;
+
+    // Strip horizontal whitespace and commas.
+    if (size_t Pos = Str.find_first_not_of(" \t\r\n,")) {
+      Str = Str.substr(Pos);
+      continue;
+    }
+
+    // If this is the start of a comment, remove the rest of the line.
+    if (Str[0] == '#') {
+        Str = Str.substr(Str.find_first_of('\n'));
+      continue;
+    }
+    return true;
+  }
+}
+
+
+static bool ByteArrayFromString(ByteArrayTy &ByteArray,
+                                StringRef &Str,
+                                SourceMgr &SM) {
+  while (SkipToToken(Str)) {
+    // Handled by higher level
+    if (Str[0] == '[' || Str[0] == ']')
+      return false;
+
+    // Get the current token.
+    size_t Next = Str.find_first_of(" \t\n\r,#[]");
+    StringRef Value = Str.substr(0, Next);
+
+    // Convert to a byte and add to the byte vector.
+    unsigned ByteVal;
+    if (Value.getAsInteger(0, ByteVal) || ByteVal > 255) {
+      // If we have an error, print it and skip to the end of line.
+      SM.PrintMessage(SMLoc::getFromPointer(Value.data()), SourceMgr::DK_Error,
+                      "invalid input token");
+      Str = Str.substr(Str.find('\n'));
+      ByteArray.first.clear();
+      ByteArray.second.clear();
+      continue;
+    }
+
+    ByteArray.first.push_back(ByteVal);
+    ByteArray.second.push_back(Value.data());
+    Str = Str.substr(Next);
+  }
+
+  return false;
+}
+
+int Disassembler::disassemble(const Target &T,
+                              const std::string &Triple,
+                              MCSubtargetInfo &STI,
+                              MCStreamer &Streamer,
+                              MemoryBuffer &Buffer,
+                              SourceMgr &SM,
+                              raw_ostream &Out) {
+  std::unique_ptr<const MCRegisterInfo> MRI(T.createMCRegInfo(Triple));
+  if (!MRI) {
+    errs() << "error: no register info for target " << Triple << "\n";
+    return -1;
+  }
+
+  MCTargetOptions MCOptions;
+  std::unique_ptr<const MCAsmInfo> MAI(
+      T.createMCAsmInfo(*MRI, Triple, MCOptions));
+  if (!MAI) {
+    errs() << "error: no assembly info for target " << Triple << "\n";
+    return -1;
+  }
+
+  // Set up the MCContext for creating symbols and MCExpr's.
+  MCContext Ctx(MAI.get(), MRI.get(), nullptr);
+
+  std::unique_ptr<const MCDisassembler> DisAsm(
+    T.createMCDisassembler(STI, Ctx));
+  if (!DisAsm) {
+    errs() << "error: no disassembler for target " << Triple << "\n";
+    return -1;
+  }
+
+  // Set up initial section manually here
+  Streamer.InitSections(false);
+
+  bool ErrorOccurred = false;
+
+  // Convert the input to a vector for disassembly.
+  ByteArrayTy ByteArray;
+  StringRef Str = Buffer.getBuffer();
+  bool InAtomicBlock = false;
+
+  while (SkipToToken(Str)) {
+    ByteArray.first.clear();
+    ByteArray.second.clear();
+
+    if (Str[0] == '[') {
+      if (InAtomicBlock) {
+        SM.PrintMessage(SMLoc::getFromPointer(Str.data()), SourceMgr::DK_Error,
+                        "nested atomic blocks make no sense");
+        ErrorOccurred = true;
+      }
+      InAtomicBlock = true;
+      Str = Str.drop_front();
+      continue;
+    } else if (Str[0] == ']') {
+      if (!InAtomicBlock) {
+        SM.PrintMessage(SMLoc::getFromPointer(Str.data()), SourceMgr::DK_Error,
+                        "attempt to close atomic block without opening");
+        ErrorOccurred = true;
+      }
+      InAtomicBlock = false;
+      Str = Str.drop_front();
+      continue;
+    }
+
+    // It's a real token, get the bytes and emit them
+    ErrorOccurred |= ByteArrayFromString(ByteArray, Str, SM);
+
+    if (!ByteArray.first.empty())
+      ErrorOccurred |= PrintInsts(*DisAsm, ByteArray, SM, Out, Streamer,
+                                  InAtomicBlock, STI);
+  }
+
+  if (InAtomicBlock) {
+    SM.PrintMessage(SMLoc::getFromPointer(Str.data()), SourceMgr::DK_Error,
+                    "unclosed atomic block");
+    ErrorOccurred = true;
+  }
+
+  return ErrorOccurred;
+}

llvm/tools/llvm-ml/llvm-ml.cpp

+//===-- llvm-ml.cpp - masm-compatible assembler -----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// A simple driver around MasmParser; based on llvm-mc.
+//
+//===----------------------------------------------------------------------===//
+
+#include "Disassembler.h"
+
+#include "llvm/MC/MCAsmBackend.h"
+#include "llvm/MC/MCAsmInfo.h"
+#include "llvm/MC/MCCodeEmitter.h"
+#include "llvm/MC/MCContext.h"
+#include "llvm/MC/MCInstPrinter.h"
+#include "llvm/MC/MCInstrInfo.h"
+#include "llvm/MC/MCObjectFileInfo.h"
+#include "llvm/MC/MCObjectWriter.h"
+#include "llvm/MC/MCParser/AsmLexer.h"
+#include "llvm/MC/MCParser/MCTargetAsmParser.h"
+#include "llvm/MC/MCRegisterInfo.h"
+#include "llvm/MC/MCStreamer.h"
+#include "llvm/MC/MCSubtargetInfo.h"
+#include "llvm/MC/MCTargetOptionsCommandFlags.inc"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Compression.h"
+#include "llvm/Support/FileUtilities.h"
+#include "llvm/Support/FormattedStream.h"
+#include "llvm/Support/Host.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/TargetRegistry.h"
+#include "llvm/Support/TargetSelect.h"
+#include "llvm/Support/ToolOutputFile.h"
+#include "llvm/Support/WithColor.h"
+
+using namespace llvm;
+
+static cl::opt<std::string>
+InputFilename(cl::Positional, cl::desc("<input file>"), cl::init("-"));
+
+static cl::opt<std::string> OutputFilename("o", cl::desc("Output filename"),
+                                           cl::value_desc("filename"),
+                                           cl::init("-"));
+
+static cl::opt<bool>
+ShowEncoding("show-encoding", cl::desc("Show instruction encodings"));
+
+static cl::opt<bool>
+ShowInst("show-inst", cl::desc("Show internal instruction representation"));
+
+static cl::opt<bool>
+ShowInstOperands("show-inst-operands",
+                 cl::desc("Show instructions operands as parsed"));
+
+static cl::opt<bool>
+OutputATTAsm("output-att-asm", cl::desc("Use ATT syntax for output printing"));
+
+static cl::opt<bool>
+PrintImmHex("print-imm-hex", cl::init(false),
+            cl::desc("Prefer hex format for immediate values"));
+
+static cl::opt<bool>
+    PreserveComments("preserve-comments",
+                     cl::desc("Preserve Comments in outputted assembly"));
+
+enum OutputFileType {
+  OFT_Null,
+  OFT_AssemblyFile,
+  OFT_ObjectFile
+};
+static cl::opt<OutputFileType>
+FileType("filetype", cl::init(OFT_ObjectFile),
+  cl::desc("Choose an output file type:"),
+  cl::values(
+       clEnumValN(OFT_AssemblyFile, "asm",
+                  "Emit an assembly ('.s') file"),
+       clEnumValN(OFT_Null, "null",
+                  "Don't emit anything (for timing purposes)"),
+       clEnumValN(OFT_ObjectFile, "obj",
+                  "Emit a native object ('.o') file")));
+
+static cl::list<std::string>
+IncludeDirs("I", cl::desc("Directory of include files"),
+            cl::value_desc("directory"), cl::Prefix);
+
+enum BitnessType {
+  m32, m64,
+};
+cl::opt<BitnessType> Bitness(cl::desc("Choose bitness:"), cl::init(m64),
+                              cl::values(clEnumVal(m32, "32-bit"),
+                                         clEnumVal(m64, "64-bit (default)")));
+
+static cl::opt<std::string>
+TripleName("triple", cl::desc("Target triple to assemble for, "
+                              "see -version for available targets"));
+
+static cl::opt<std::string>
+DebugCompilationDir("fdebug-compilation-dir",
+                    cl::desc("Specifies the debug info's compilation dir"));
+
+static cl::list<std::string>
+DebugPrefixMap("fdebug-prefix-map",
+               cl::desc("Map file source paths in debug info"),
+               cl::value_desc("= separated key-value pairs"));
+
+static cl::opt<std::string>
+MainFileName("main-file-name",
+             cl::desc("Specifies the name we should consider the input file"));
+
+static cl::opt<bool> SaveTempLabels("save-temp-labels",
+                                    cl::desc("Don't discard temporary labels"));
+
+enum ActionType {
+  AC_AsLex,
+  AC_Assemble,
+  AC_Disassemble,
+  AC_MDisassemble,
+};
+
+static cl::opt<ActionType>
+Action(cl::desc("Action to perform:"),
+       cl::init(AC_Assemble),
+       cl::values(clEnumValN(AC_AsLex, "as-lex",
+                             "Lex tokens from a .asm file"),
+                  clEnumValN(AC_Assemble, "assemble",
+                             "Assemble a .asm file (default)"),
+                  clEnumValN(AC_Disassemble, "disassemble",
+                             "Disassemble strings of hex bytes"),
+                  clEnumValN(AC_MDisassemble, "mdis",
+                             "Marked up disassembly of strings of hex bytes")));
+
+static const Target *GetTarget(const char *ProgName) {
+  // Figure out the target triple.
+  if (TripleName.empty()) {
+    if (Bitness == m32)
+      TripleName = "i386-pc-windows";
+    else if (Bitness == m64)
+      TripleName = "x86_64-pc-windows";
+  }
+  Triple TheTriple(Triple::normalize(TripleName));
+
+  // Get the target specific parser.
+  std::string Error;
+  const Target *TheTarget = TargetRegistry::lookupTarget("", TheTriple, Error);
+  if (!TheTarget) {
+    WithColor::error(errs(), ProgName) << Error;
+    return nullptr;
+  }
+
+  // Update the triple name and return the found target.
+  TripleName = TheTriple.getTriple();
+  return TheTarget;
+}
+
+static std::unique_ptr<ToolOutputFile> GetOutputStream(StringRef Path) {
+  std::error_code EC;
+  auto Out = std::make_unique<ToolOutputFile>(Path, EC, sys::fs::F_None);
+  if (EC) {
+    WithColor::error() << EC.message() << '\n';
+    return nullptr;
+  }
+
+  return Out;
+}
+
+static int AsLexInput(SourceMgr &SrcMgr, MCAsmInfo &MAI,
+                      raw_ostream &OS) {
+  AsmLexer Lexer(MAI);
+  Lexer.setBuffer(SrcMgr.getMemoryBuffer(SrcMgr.getMainFileID())->getBuffer());
+
+  bool Error = false;
+  while (Lexer.Lex().isNot(AsmToken::Eof)) {
+    Lexer.getTok().dump(OS);
+    OS << "\n";
+    if (Lexer.getTok().getKind() == AsmToken::Error)
+      Error = true;
+  }
+
+  return Error;
+}
+
+static int AssembleInput(const char *ProgName, const Target *TheTarget,
+                         SourceMgr &SrcMgr, MCContext &Ctx, MCStreamer &Str,
+                         MCAsmInfo &MAI, MCSubtargetInfo &STI,
+                         MCInstrInfo &MCII, MCTargetOptions &MCOptions) {
+  std::unique_ptr<MCAsmParser> Parser(createMCAsmParser(SrcMgr, Ctx, Str, MAI));
+  std::unique_ptr<MCTargetAsmParser> TAP(
+      TheTarget->createMCAsmParser(STI, *Parser, MCII, MCOptions));
+
+  if (!TAP) {
+    WithColor::error(errs(), ProgName)
+        << "this target does not support assembly parsing.\n";
+    return 1;
+  }
+
+  Parser->setShowParsedOperands(ShowInstOperands);
+  Parser->setTargetParser(*TAP);
+  Parser->getLexer().setLexMasmIntegers(true);
+
+  int Res = Parser->Run(/*NoInitialTextSection=*/true);
+
+  return Res;
+}
+
+int main(int argc, char **argv) {
+  InitLLVM X(argc, argv);
+
+  // Initialize targets and assembly printers/parsers.
+  llvm::InitializeAllTargetInfos();
+  llvm::InitializeAllTargetMCs();
+  llvm::InitializeAllAsmParsers();
+  llvm::InitializeAllDisassemblers();
+
+  // Register the target printer for --version.
+  cl::AddExtraVersionPrinter(TargetRegistry::printRegisteredTargetsForVersion);
+
+  cl::ParseCommandLineOptions(argc, argv, "llvm machine code playground\n");
+  MCTargetOptions MCOptions = InitMCTargetOptionsFromFlags();
+
+  const char *ProgName = argv[0];
+  const Target *TheTarget = GetTarget(ProgName);
+  if (!TheTarget)
+    return 1;
+  // Now that GetTarget() has (potentially) replaced TripleName, it's safe to
+  // construct the Triple object.
+  Triple TheTriple(TripleName);
+
+  ErrorOr<std::unique_ptr<MemoryBuffer>> BufferPtr =
+      MemoryBuffer::getFileOrSTDIN(InputFilename);
+  if (std::error_code EC = BufferPtr.getError()) {
+    WithColor::error(errs(), ProgName)
+        << InputFilename << ": " << EC.message() << '\n';
+    return 1;
+  }
+  MemoryBuffer *Buffer = BufferPtr->get();
+
+  SourceMgr SrcMgr;
+
+  // Tell SrcMgr about this buffer, which is what the parser will pick up.
+  SrcMgr.AddNewSourceBuffer(std::move(*BufferPtr), SMLoc());
+
+  // Record the location of the include directories so that the lexer can find
+  // it later.
+  SrcMgr.setIncludeDirs(IncludeDirs);
+
+  std::unique_ptr<MCRegisterInfo> MRI(TheTarget->createMCRegInfo(TripleName));
+  assert(MRI && "Unable to create target register info!");
+
+  std::unique_ptr<MCAsmInfo> MAI(
+      TheTarget->createMCAsmInfo(*MRI, TripleName, MCOptions));
+  assert(MAI && "Unable to create target asm info!");
+
+  MAI->setPreserveAsmComments(PreserveComments);
+
+  // FIXME: This is not pretty. MCContext has a ptr to MCObjectFileInfo and
+  // MCObjectFileInfo needs a MCContext reference in order to initialize itself.
+  MCObjectFileInfo MOFI;
+  MCContext Ctx(MAI.get(), MRI.get(), &MOFI, &SrcMgr);
+  MOFI.InitMCObjectFileInfo(TheTriple, /*PIC=*/false, Ctx,
+                            /*LargeCodeModel=*/true);
+
+  if (SaveTempLabels)
+    Ctx.setAllowTemporaryLabels(false);
+
+  if (!DebugCompilationDir.empty()) {
+    Ctx.setCompilationDir(DebugCompilationDir);
+  } else {
+    // If no compilation dir is set, try to use the current directory.
+    SmallString<128> CWD;
+    if (!sys::fs::current_path(CWD))
+      Ctx.setCompilationDir(CWD);
+  }
+  for (const auto &Arg : DebugPrefixMap) {
+    const auto &KV = StringRef(Arg).split('=');
+    Ctx.addDebugPrefixMapEntry(KV.first, KV.second);
+  }
+  if (!MainFileName.empty())
+    Ctx.setMainFileName(MainFileName);
+
+  // Package up features to be passed to target/subtarget
+  std::string FeaturesStr;
+  //if (MAttrs.size()) {
+    //SubtargetFeatures Features;
+    //for (unsigned i = 0; i != MAttrs.size(); ++i)
+      //Features.AddFeature(MAttrs[i]);
+    //FeaturesStr = Features.getString();
+  //}
+
+  std::unique_ptr<ToolOutputFile> Out = GetOutputStream(OutputFilename);
+  if (!Out)
+    return 1;
+
+  std::unique_ptr<buffer_ostream> BOS;
+  raw_pwrite_stream *OS = &Out->os();
+  std::unique_ptr<MCStreamer> Str;
+
+  std::unique_ptr<MCInstrInfo> MCII(TheTarget->createMCInstrInfo());
+  std::unique_ptr<MCSubtargetInfo> STI(
+      TheTarget->createMCSubtargetInfo(TripleName, /*CPU=*/"", FeaturesStr));
+
+  MCInstPrinter *IP = nullptr;
+  if (FileType == OFT_AssemblyFile) {
+    const unsigned OutputAsmVariant = OutputATTAsm ? 0U   // ATT dialect
+                                                   : 1U;  // Intel dialect
+    IP = TheTarget->createMCInstPrinter(Triple(TripleName), OutputAsmVariant,
+                                        *MAI, *MCII, *MRI);
+
+    if (!IP) {
+      WithColor::error()
+          << "unable to create instruction printer for target triple '"
+          << TheTriple.normalize() << "' with "
+          << (OutputATTAsm ? "ATT" : "Intel") << " assembly variant.\n";
+      return 1;
+    }
+
+    // Set the display preference for hex vs. decimal immediates.
+    IP->setPrintImmHex(PrintImmHex);
+
+    // Set up the AsmStreamer.
+    std::unique_ptr<MCCodeEmitter> CE;
+    if (ShowEncoding)
+      CE.reset(TheTarget->createMCCodeEmitter(*MCII, *MRI, Ctx));
+
+    std::unique_ptr<MCAsmBackend> MAB(
+        TheTarget->createMCAsmBackend(*STI, *MRI, MCOptions));
+    auto FOut = std::make_unique<formatted_raw_ostream>(*OS);
+    Str.reset(
+        TheTarget->createAsmStreamer(Ctx, std::move(FOut), /*asmverbose*/ true,
+                                     /*useDwarfDirectory*/ true, IP,
+                                     std::move(CE), std::move(MAB), ShowInst));
+
+  } else if (FileType == OFT_Null) {
+    Str.reset(TheTarget->createNullStreamer(Ctx));
+  } else {
+    assert(FileType == OFT_ObjectFile && "Invalid file type!");
+
+    // Don't waste memory on names of temp labels.
+    Ctx.setUseNamesOnTempLabels(false);
+
+    if (!Out->os().supportsSeeking()) {
+      BOS = std::make_unique<buffer_ostream>(Out->os());
+      OS = BOS.get();
+    }
+
+    MCCodeEmitter *CE = TheTarget->createMCCodeEmitter(*MCII, *MRI, Ctx);
+    MCAsmBackend *MAB = TheTarget->createMCAsmBackend(*STI, *MRI, MCOptions);
+    Str.reset(TheTarget->createMCObjectStreamer(
+        TheTriple, Ctx, std::unique_ptr<MCAsmBackend>(MAB),
+        MAB->createObjectWriter(*OS), std::unique_ptr<MCCodeEmitter>(CE), *STI,
+        MCOptions.MCRelaxAll, MCOptions.MCIncrementalLinkerCompatible,
+        /*DWARFMustBeAtTheEnd*/ false));
+    //if (NoExecStack)
+      //Str->InitSections(true);
+  }
+
+  // Use Assembler information for parsing.
+  Str->setUseAssemblerInfoForParsing(true);
+
+  int Res = 1;
+  bool disassemble = false;
+  switch (Action) {
+  case AC_AsLex:
+    Res = AsLexInput(SrcMgr, *MAI, Out->os());
+    break;
+  case AC_Assemble:
+    Res = AssembleInput(ProgName, TheTarget, SrcMgr, Ctx, *Str, *MAI, *STI,
+                        *MCII, MCOptions);
+    break;
+  case AC_MDisassemble:
+    assert(IP && "Expected assembly output");
+    IP->setUseMarkup(1);
+    disassemble = true;
+    break;
+  case AC_Disassemble:
+    disassemble = true;
+    break;
+  }
+  if (disassemble)
+    Res = Disassembler::disassemble(*TheTarget, TripleName, *STI, *Str,
+                                    *Buffer, SrcMgr, Out->os());
+
+  // Keep output if no errors.
+  if (Res == 0)
+    Out->keep();
+  return Res;
+}

llvm/utils/gn/secondary/llvm/test/BUILD.gn

deps = [
     "//llvm/tools/llvm-isel-fuzzer",
     "//llvm/tools/llvm-jitlink",
     "//llvm/tools/llvm-link",
     "//llvm/tools/llvm-lipo",
     "//llvm/tools/llvm-lto",
     "//llvm/tools/llvm-lto2",
     "//llvm/tools/llvm-mc",
     "//llvm/tools/llvm-mca",
+    "//llvm/tools/llvm-ml",
     "//llvm/tools/llvm-modextract",
     "//llvm/tools/llvm-mt",
     "//llvm/tools/llvm-nm",
     "//llvm/tools/llvm-objcopy:symlinks",
     "//llvm/tools/llvm-objdump",
     "//llvm/tools/llvm-opt-fuzzer",
     "//llvm/tools/llvm-opt-report",
     "//llvm/tools/llvm-pdbutil",

llvm/utils/gn/secondary/llvm/tools/llvm-ml/BUILD.gn

+executable("llvm-ml") {
+  deps = [
+    "//llvm/lib/MC",
+    "//llvm/lib/MC/MCParser",
+    "//llvm/lib/Support",
+    "//llvm/lib/Target:AllTargetsAsmParsers",
+    "//llvm/lib/Target:AllTargetsAsmPrinters",
+    "//llvm/lib/Target:AllTargetsDescs",
+    "//llvm/lib/Target:AllTargetsDisassemblers",
+    "//llvm/lib/Target:AllTargetsInfos",
+  ]
+  sources = [
+    # FIXME: duplicating this is stupid
+    "Disassembler.cpp",
+    "llvm-ml.cpp",
+  ]
+}