[Object][DX] Initial DXContainer parsing support

[beanz]

Summary

This patch begins adding DXContainer parsing support to libObject.
Following the pattern used by ELFFile my goal here is to write a
standalone DXContainer parser and later write an adapter interface to
support a subset of the ObjectFile interfaces so that we can add
limited objdump support. I will also be adding ObjectYAML support to
help drive testing of the object tools and MC-level object writers as
those come together.

DXContainer is a slightly odd format. It is arranged in "parts" that
are semantically similar to sections, but it doesn't support symbol
listing.

[MaskRay]

llvm/include/llvm/BinaryFormat/DXContainer.h

66

This comment just seems to duplicate what the code says.

If you just write uint8_t Magic[4]; // "DXBC", it seems to cover all the reader needs.

96

The comment doesn't convey more information than the code says.

llvm/lib/Object/DXContainer.cpp

38

Omit llvm::

[beanz]

Updates based on PR feedback

[beanz]

One more set of updates. Thanks @MaskRay!

[MaskRay]

llvm/include/llvm/BinaryFormat/DXContainer.h

89

delete blank line

llvm/include/llvm/Object/DXContainer.h

22

Remove as the file doesn't use std::vector

llvm/lib/Object/DXContainer.cpp

17

Use static instead of anonymous namespace.

See https://llvm.org/docs/CodingStandards.html#anonymous-namespaces

43

llvm::Error => Error everyewhere

llvm/unittests/Object/DXContainerTest.cpp

19

static

[nikic]

llvm/include/llvm/BinaryFormat/DXContainer.h

68

This breaks the build with some versions of GCC:

In file included from /root/llvm-compile-time-tracker/llvm-project/llvm/include/llvm/Object/DXContainer.h:19,
                 from /root/llvm-compile-time-tracker/llvm-project/llvm/lib/Object/DXContainer.cpp:9:
/root/llvm-compile-time-tracker/llvm-project/llvm/include/llvm/BinaryFormat/DXContainer.h:67:8: error: declaration of ‘llvm::dxbc::Hash llvm::dxbc::Header::Hash’ changes meaning of ‘Hash’ [-fpermissive]
   67 |   Hash Hash;
      |        ^~~~
/root/llvm-compile-time-tracker/llvm-project/llvm/include/llvm/BinaryFormat/DXContainer.h:38:8: note: ‘Hash’ declared here as ‘struct llvm::dxbc::Hash’
   38 | struct Hash {

[dyung]

The PS4 linux buildbot which runs gcc is not building any longer after this change, @beanz can you take a look?

https://lab.llvm.org/buildbot/#/builders/139/builds/21149

[beanz]

Bots should be fixed with rG55e13a6bc0d6a31afc258a012184253041b0eb8e.

[nemanjai]

In D124643#3486603, @beanz wrote:

Bots should be fixed with rG55e13a6bc0d6a31afc258a012184253041b0eb8e.

This isn't the only problem. There are also failures due to:

llvm-project/llvm/lib/Object/DXContainer.cpp:42:12: error: call to deleted constructor of 'llvm::Error'
    return Err;
           ^~~

See https://lab.llvm.org/buildbot/#/builders/21/builds/40034/steps/5/logs/stdio

[delcypher]

In D124643#3486838, @nemanjai wrote:

In D124643#3486603, @beanz wrote:

Bots should be fixed with rG55e13a6bc0d6a31afc258a012184253041b0eb8e.

This isn't the only problem. There are also failures due to:

llvm-project/llvm/lib/Object/DXContainer.cpp:42:12: error: call to deleted constructor of 'llvm::Error'
    return Err;
           ^~~

See https://lab.llvm.org/buildbot/#/builders/21/builds/40034/steps/5/logs/stdio

@beanz

I'm also seeing this build error when we try to merge this change into Apple's llvm-project fork.

/am_github_build_tester/llvm-project/llvm/lib/Object/DXContainer.cpp:42:12: error: call to deleted constructor of 'llvm::Error'
    return Err;
           ^~~
/am_github_build_tester/llvm-project/llvm/include/llvm/Support/Error.h:184:3: note: 'Error' has been explicitly marked deleted here
  Error(const Error &Other) = delete;
  ^
/am_github_build_tester/llvm-project/llvm/include/llvm/Support/Error.h:491:18: note: passing argument to parameter 'Err' here
  Expected(Error Err)
                 ^
1 error generated.

Could you take a look? It looks like the copy constructor and assignment operator are deliberately deleted. However the move constructor and move assignment operator are implemented so I'm guessing you might be able to do

if (Error Err = Container.parseHeader())
   return std::move(Err);

I'm a little surprised by this error though because the pattern you're using is what's in the llvm:Error documentation: https://llvm.org/doxygen/classllvm_1_1Error.html . It kind of suggests those comments are out-of-date.

[beanz]

The other issue should be fixed in rGb26e44e623c75c084e865084b18541c6a1736df2.

[delcypher]

In D124643#3487122, @beanz wrote:

The other issue should be fixed in rGb26e44e623c75c084e865084b18541c6a1736df2.

@beanz Thanks for fixing so quickly :)

llvm/include/llvm/BinaryFormat/DXContainer.h

+//===-- llvm/BinaryFormat/DXContainer.h - The DXBC file format --*- C++/-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines manifest constants for the DXContainer object file format.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_BINARYFORMAT_DXCONTAINER_H
+#define LLVM_BINARYFORMAT_DXCONTAINER_H
+
+#include "llvm/Support/SwapByteOrder.h"
+
+#include <stdint.h>
+
+namespace llvm {
+
+// The DXContainer file format is arranged as a header and "parts". Semantically
+// parts are similar to sections in other object file formats. The File format
+// structure is roughly:
+
+// ┌────────────────────────────────┐
+// │             Header             │
+// ├────────────────────────────────┤
+// │              Part              │
+// ├────────────────────────────────┤
+// │              Part              │
+// ├────────────────────────────────┤
+// │              ...               │
+// └────────────────────────────────┘
+
+namespace dxbc {
+
+struct Hash {
+  uint8_t Digest[16];
+};
+
+enum class HashFlags : uint32_t {
+  None = 0,           // No flags defined.
+  IncludesSource = 1, // This flag indicates that the shader hash was computed
+                      // taking into account source information (-Zss)
+};
+
+struct ShaderHash {
+  uint32_t Flags; // DxilShaderHashFlags
+  uint8_t Digest[16];
+
+  void byteSwap() { sys::swapByteOrder(Flags); }
+};
+
+struct ContainerVersion {
+  uint16_t Major;
+  uint16_t Minor;
+
+  void byteSwap() {
+    sys::swapByteOrder(Major);
+    sys::swapByteOrder(Minor);
+  }
+};
+
+struct Header {
+  uint8_t Magic[4]; // "DXBC"
+  Hash Hash;
+  ContainerVersion Version;
+  uint32_t FileSize;
+  uint32_t PartCount;
+
+  void byteSwap() {
+    Version.byteSwap();
+    sys::swapByteOrder(FileSize);
+    sys::swapByteOrder(PartCount);
+  }
+  // Structure is followed by part offsets: uint32_t PartOffset[PartCount];
+  // The offset is to a PartHeader, which is followed by the Part Data.
+};
+
+/// Use this type to describe the size and type of a DXIL container part.
+struct PartHeader {
+  uint8_t Name[4];
+  uint32_t Size;
+  // Structure is followed directly by part data: uint8_t PartData[PartSize].
+};
+
+} // namespace dxbc
+
+} // namespace llvm
+
+#endif // LLVM_BINARYFORMAT_DXCONTAINER_H

llvm/include/llvm/Object/DXContainer.h

+//===- DXContainer.h - DXContainer file implementation ----------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the DXContainerFile class, which implements the ObjectFile
+// interface for DXContainer files.
+//
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_OBJECT_DXCONTAINER_H
+#define LLVM_OBJECT_DXCONTAINER_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/BinaryFormat/DXContainer.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/MemoryBufferRef.h"
+#include <vector>
+
+namespace llvm {
+namespace object {
+class DXContainer {
+private:
+  DXContainer(MemoryBufferRef O);
+
+  MemoryBufferRef Data;
+  dxbc::Header Header;
+
+  Error parseHeader();
+
+public:
+  StringRef getData() const { return Data.getBuffer(); }
+  static Expected<DXContainer> create(MemoryBufferRef Object);
+
+  const dxbc::Header &getHeader() const { return Header; }
+};
+
+} // namespace object
+} // namespace llvm
+
+#endif // LLVM_OBJECT_DXCONTAINERFILE_H

llvm/lib/Object/CMakeLists.txt

 add_llvm_component_library(LLVMObject
   Archive.cpp
   ArchiveWriter.cpp
   Binary.cpp
   COFFImportFile.cpp
   COFFModuleDefinition.cpp
   COFFObjectFile.cpp
   Decompressor.cpp
+  DXContainer.cpp
   ELF.cpp
   ELFObjectFile.cpp
   Error.cpp
   FaultMapParser.cpp
   IRObjectFile.cpp
   IRSymtab.cpp
   MachOObjectFile.cpp
   MachOUniversal.cpp

llvm/lib/Object/DXContainer.cpp

+//===- DXContainer.cpp - DXContainer object file implementation -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Object/DXContainer.h"
+#include "llvm/BinaryFormat/DXContainer.h"
+#include "llvm/Object/Error.h"
+
+using namespace llvm;
+using namespace llvm::object;
+
+namespace {
+Error parseFailed(const Twine &Msg) {
+  return make_error<GenericBinaryError>(Msg.str(), object_error::parse_failed);
+}
+
+template <typename T>
+Error readStruct(StringRef Buffer, const char *P, T &Struct) {
+  // Don't read before the beginning or past the end of the file
+  if (P < Buffer.begin() || P + sizeof(T) > Buffer.end())
+    return parseFailed("Reading structure out of file bounds");
+
+  memcpy(&Struct, P, sizeof(T));
+  // DXContainer is always BigEndian
+  if (sys::IsBigEndianHost)
+    Struct.byteSwap();
+  return llvm::Error::success();
+}
+} // namespace
+
+DXContainer::DXContainer(MemoryBufferRef O) : Data(O) {}
+
+Error DXContainer::parseHeader() {
+  return readStruct(Data.getBuffer(), Data.getBuffer().data(), Header);
+}
+
+Expected<DXContainer> DXContainer::create(MemoryBufferRef Object) {
+  DXContainer Container(Object);
+  if (llvm::Error Err = Container.parseHeader())
+    return Err;
+  return Container;
+}

llvm/unittests/Object/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   BinaryFormat
   Object
   ObjectYAML
   )
 
 add_llvm_unittest(ObjectTests
   ArchiveTest.cpp
+  DXContainerTest.cpp
   ELFObjectFileTest.cpp
   ELFTypesTest.cpp
   ELFTest.cpp
   MinidumpTest.cpp
   ObjectFileTest.cpp
   OffloadingTest.cpp
   SymbolSizeTest.cpp
   SymbolicFileTest.cpp
   XCOFFObjectFileTest.cpp
   )
 
 target_link_libraries(ObjectTests PRIVATE LLVMTestingSupport)

llvm/unittests/Object/DXContainerTest.cpp

+//===- DXContainerTest.cpp - Tests for DXContainerFile --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Object/DXContainer.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/BinaryFormat/Magic.h"
+#include "llvm/Support/MemoryBufferRef.h"
+#include "llvm/Testing/Support/Error.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+using namespace llvm::object;
+
+template <std::size_t X> MemoryBufferRef getMemoryBuffer(uint8_t Data[X]) {
+  StringRef Obj(reinterpret_cast<char *>(&Data[0]), X);
+  return MemoryBufferRef(Obj, "");
+}
+
+TEST(DXCFile, ParseHeaderErrors) {
+  uint8_t Buffer[] = {0x44, 0x58, 0x42, 0x43};
+  EXPECT_THAT_EXPECTED(
+      DXContainer::create(getMemoryBuffer<4>(Buffer)),
+      FailedWithMessage("Reading structure out of file bounds"));
+}
+
+TEST(DXCFile, ParseHeader) {
+  uint8_t Buffer[] = {0x44, 0x58, 0x42, 0x43, 0x00, 0x00, 0x00, 0x00,
+                      0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+                      0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00,
+                      0x70, 0x0D, 0x00, 0x00, 0x07, 0x00, 0x00, 0x00};
+  DXContainer C =
+      llvm::cantFail(DXContainer::create(getMemoryBuffer<32>(Buffer)));
+  EXPECT_TRUE(memcmp(C.getHeader().Magic, "DXBC", 4) == 0);
+  EXPECT_TRUE(memcmp(C.getHeader().Hash.Digest,
+                     "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) == 0);
+  EXPECT_EQ(C.getHeader().Version.Major, 1u);
+  EXPECT_EQ(C.getHeader().Version.Minor, 0u);
+}