[AMDGPU] llvm-objdump: disassembling amdgcn object file

[vpykhtin]

Summary

Minimal ISA dumper for amdgcn object file.

.hsatext ELF kernel symbol references amd_kernel_code_t runtime control structure (256 bytes) followed by instructions, so skip it.

[• tstellarAMD]

Is there some reason why you can't parse the code object in from the AMDGPU implementation of getInstruction()?

[vpykhtin]

In D16998#346610, @tstellarAMD wrote:

Is there some reason why you can't parse the code object in from the AMDGPU implementation of getInstruction()?

MCDisassembler::getInstruction is declared as follows:

/// Returns the disassembly of a single instruction.
///
/// \param Instr    - An MCInst to populate with the contents of the
///                   instruction.
/// \param Size     - A value to populate with the size of the instruction, or
///                   the number of bytes consumed while attempting to decode
///                   an invalid instruction.
/// ...
virtual DecodeStatus getInstruction(MCInst &Instr, uint64_t &Size,
                                    ArrayRef<uint8_t> Bytes, uint64_t Address,
                                    raw_ostream &VStream,
                                    raw_ostream &CStream) const = 0;

Its supposed to return filled MCInst as a result and return consumed bytes num.

Current DisassemblyObject just passes what it consider as 'code' to the disassembler and therefore inside getInstruction we facing the following:

1. We need to know whether we're actually inside an amd_kernel_code_t. Yes we can analize Address but llvm-mc calls dissassembler for a raw stream of bytes and we need to care of it too.
2. amd_kernel_code_t cannot be stored to MCInst but its desirable to dump it too as part of disassembly.
3. it would be a hack to try reposition reader specifying negative size in the case when amd_kernel_code_t is actually before amd_kernel_code_t
4. we need to process all code symbols in .hsasection to find all kernels ends as kernel size is not specified by design.
5. DisassemblyObject knows nothing about ELF::STT_AMDGPU_HSA_KERNEL

[• tstellarAMD]

Hi,

In the loop that iterates over symbols and emits them, there is already some target specific code for AARCH64.
I would recommend adding a callback to MCDisassembler called something like disassembleSymbolStart() and call that function for the AMDGPU target, and then move the disassembly code for the amd_kernel_code_t into the AMDGPU backend.

[vpykhtin]

In D16998#347911, @tstellarAMD wrote:

Hi,

In the loop that iterates over symbols and emits them, there is already some target specific code for AARCH64.
I would recommend adding a callback to MCDisassembler called something like disassembleSymbolStart() and call that function for the AMDGPU target, and then move the disassembly code for the amd_kernel_code_t into the AMDGPU backend.

For now I just moved all my code to AMDGPUDisassembler library (links with llvm-objdump already) and call DisassembleHSACodeObject in the beginning of llvm-objdump's DisassembleObject if Obj->getArch() == Triple::amdgcn. It still lefts llvm-objdump slightly AMDGPU target dependent but now I think it's not worth to touch MCDisassembler interface until we know all the requirements. It's likely that we might want to disassemble HSA code object completely with AMDGPU code not depending on llvm-objdump.

[vpykhtin]

• AMDGPU dependent code moved to AMDGPUDisassembler library.
• Dissassembly code slightly enhanced to be ready assembled again. A lot of TODO though.
• Skipping unussembled code bytes nicely

[arsenm]

lib/Target/AMDGPU/Disassembler/HSAObjCodeDisassembler.cpp

113 ↗

(On Diff #47672)

Would this be useful in the MCContext header?

150–152 ↗

(On Diff #47672)

This is dead code

158 ↗

(On Diff #47672)

The triple should be amdgcn-unknown-amdhsa. Why do you need to hardcode this here?

166–167 ↗

(On Diff #47672)

I don't think this should be able to happen. These checks should maybe all be asserts

208 ↗

(On Diff #47672)

Put declarations on separate lines

225 ↗

(On Diff #47672)

Single quotes

228 ↗

(On Diff #47672)

Single quotes

tools/llvm-objdump/llvm-objdump.cpp

834 ↗

(On Diff #47672)

Function should start with lowercase letter

[vpykhtin]

lib/Target/AMDGPU/Disassembler/HSAObjCodeDisassembler.cpp

113 ↗

(On Diff #47672)

Yep, I would move it there.

150–152 ↗

(On Diff #47672)

Well, when NDEBUG there is no DebugFlag, but it is tested below

if (!DisAsm->getInstruction(Inst, EatenBytesNum,

Code.slice(Index), Index,
DebugFlag ? dbgs() : nulls(),
nulls())) {

I just dont like interleave code with macros a lot.

158 ↗

(On Diff #47672)

probably not need to hardcode, but I haven't found yet how to construct it properly.

166–167 ↗

(On Diff #47672)

ok.

[• tstellarAMD]

tools/llvm-objdump/llvm-objdump.cpp

840 ↗

(On Diff #47672)

You aren't going to be able to call target specific functions from a common tool like this. The only way this is going to work is with some kind of generic callback.

[nhaustov]

tools/llvm-objdump/llvm-objdump.cpp

840 ↗

(On Diff #47672)

llvm-objdump actually has custom code for AArch64 that tries to determine whether bytes being disassembled is code or data:

// AArch64 ELF binaries can interleave data and text in the
// same section. We rely on the markers introduced to
// understand what we need to dump.

...
To me, proposed code in separate file looks cleaner and does not clutter llvm-objdump.cpp.

Anyway, another question is whether it is really only AMDGPU that needs custom handling of parts of .text segment? How do other platforms handle metadata needed for code startup? Any pointers?

One reason to keep amd_kernel_code_t (metadata needed for kernel startup) close to actual ISA is effiiency and cache concerns. We can still probably change it in new format.

[vpykhtin]

• HSACodeObjectDisassembler class introduced
• ELFNote support added
• Target ISA determined based on amdgpu_hsa_note_isa_t ELF note (spec)
• printing for assembler directives

[• rafael]

Testcase?

[vpykhtin]

Added binary test

[• rafael]

How has the test created. What is a ".co"?

[vpykhtin]

In D16998#371942, @rafael wrote:

How has the test created. What is a ".co"?

".co" means CodeObject. It was created using out of llvm tree utility. It is of HSA Code Object v1.0 format which is slightly different from what AMDGPU assembler currently emitting, the main difference is - code symbol has zero size and amd_kernel_code_t (runtime control structure) and ISA could be non-adjacent. The dissasembler is supposed to accept both versions.

lib/Target/AMDGPU/Disassembler/HSAObjCodeDisassembler.cpp

113 ↗

(On Diff #47672)

Let's do it in separate change later.

[vpykhtin]

Adding quote from summary so it can be seen in notification emails:

Problem: this change contains direct call for disassembleHSACodeObject from llvm-objdump which is actually located in the AMDGPU library. This is incorrect as require llvm-objdump always link with AMDGPU target. There should be a target callback somewhere that allows custom object file disassembly. One thought was to add such callback (something like disassembleObjectFile(ObjFile, OutStream)) to the MCDisassembler interface but it is a bit late because MCDisassembler is created when the target ISA specification is already known (it depends on MCSubtargetInfo) but the target ISA spec is yet to be read from the object file really.

[llvm-commits]

In other words, this is a Elf_Rel file? Why do you need a new extension?

Right Elf_Rel. Disassembling of this format requires different features that is currently suggested by llvm-objdump and these are really target specific. In particular we would like to print all custom directives and structures so it can be fed directly to assembler input, for example amd_kernel_code_t runtime control structure definitions. Dumping this structure without llvm-objdump->AMDGPU link dependency would require copy-paste of the target code. I'm not sure it's possible to create sufficient number of target callbacks to accomplish this.

Another issue is with the HSA Code Object 1.0 format itself - zero sized kernel code symbol require to preprocess the whole .hsatext section to find start markers of every item in this section to calculate the kernel code size. This particular issue is fixed with what AMDGPU assembler currently emits but we would like disassembler could accept old files too.

---

llvm-commits mailing list
[e-mail]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits

[vpykhtin]

Made minimal possible change.

Limitations:

• accepts object file produced by current AMDGPU assembler (no older formats support)
• only ISA dumping (no directives (notes), amd_kernel_code_t etc.)
• mcpu should be explicitly specified upon llvm-objdump run (no autodetection)

Added test using llvm-mc | llvm-objdump check.

[Bigcheese]

tools/llvm-objdump/llvm-objdump.cpp

395 ↗

(On Diff #50625)

This should be using a endian specific type. Either llvm::support::ulittle32_t or ubig32_t.

1077 ↗

(On Diff #50625)

Same with the endianness.

[vpykhtin]

Thanks for the review!

Updated the diff with issues fixed.

[vpykhtin]

ping.

[vpykhtin]

ping.

[Bigcheese]

lgtm

[vpykhtin]

I reverted it for a while, instruction bytes are printed in bigendian order on ppc despite on support::ulittle32_t usage. Need to fix.

llvm/trunk/test/Object/AMDGPU/objdump.s

+// RUN: llvm-mc -arch=amdgcn -mcpu=tonga %s -filetype=obj | llvm-objdump -disassemble -arch-name=amdgcn -mcpu=tonga - | FileCheck %s
+
+  .text
+
+  .amdgpu_hsa_kernel hello_world
+hello_world:
+  .amd_kernel_code_t
+  .end_amd_kernel_code_t
+
+  s_mov_b32 m0, 0x10000
+  s_load_dwordx2 s[0:1], s[4:5], 0x8
+  s_waitcnt lgkmcnt(0)
+  s_add_u32 s0, s7, s0
+  v_add_i32_e32 v1, vcc, s0, v1
+  s_movk_i32 s0, 0x483
+  v_cmp_ge_i32_e32 vcc, s0, v0
+  s_and_saveexec_b64 s[0:1], vcc
+  v_lshlrev_b32_e32 v4, 2, v0
+  s_cbranch_execz 21
+  s_mov_b64 s[2:3], exec
+  s_mov_b64 s[10:11], exec
+  v_mov_b32_e32 v3, v0
+        s_endpgm
+
+  .amdgpu_hsa_kernel hello_world2
+hello_world2:
+  .amd_kernel_code_t
+  .end_amd_kernel_code_t
+
+  s_and_saveexec_b64 s[0:1], vcc
+  s_cbranch_execz 85
+  s_load_dwordx4 s[8:11], s[4:5], 0x40
+  v_ashrrev_i32_e32 v77, 31, v76
+  v_lshlrev_b64 v[10:11], 2, v[76:77]
+  s_waitcnt lgkmcnt(0)
+  v_add_i32_e32 v10, vcc, s8, v10
+  v_mov_b32_e32 v6, s9
+  v_addc_u32_e32 v11, vcc, v6, v11, vcc
+  flat_load_dword v0, v[10:11]
+  v_lshlrev_b32_e32 v6, 5, v8
+  v_lshlrev_b32_e32 v7, 2, v7
+        s_endpgm
+
+// CHECK:  file format ELF64-amdgpu-hsacobj
+// CHECK:  Disassembly of section .hsatext:
+// CHECK:  hello_world:
+// CHECK:  s_mov_b32 m0, 0x10000                                      // 000000000100: BEFC00FF 00010000
+// CHECK:  s_load_dwordx2 s[0:1], s[4:5], 0x8                         // 000000000108: C0060002 00000008
+// CHECK:  s_waitcnt lgkmcnt(0)                                       // 000000000110: BF8C007F
+// CHECK:  s_add_u32 s0, s7, s0                                       // 000000000114: 80000007
+// CHECK:  v_add_i32_e32 v1, vcc, s0, v1                              // 000000000118: 32020200
+// CHECK:  s_movk_i32 s0, 0x483                                       // 00000000011C: B0000483
+// CHECK:  v_cmp_ge_i32_e32 vcc, s0, v0                               // 000000000120: 7D8C0000
+// CHECK:  s_and_saveexec_b64 s[0:1], vcc                             // 000000000124: BE80206A
+// CHECK:  v_lshlrev_b32_e32 v4, 2, v0                                // 000000000128: 24080082
+// CHECK:  s_cbranch_execz 21                                         // 00000000012C: BF880015
+// CHECK:  s_mov_b64 s[2:3], exec                                     // 000000000130: BE82017E
+// CHECK:  s_mov_b64 s[10:11], exec                                   // 000000000134: BE8A017E
+// CHECK:  v_mov_b32_e32 v3, v0                                       // 000000000138: 7E060300
+// CHECK:  s_endpgm                                                   // 00000000013C: BF810000
+
+// CHECK:  hello_world2:
+// CHECK:  s_and_saveexec_b64 s[0:1], vcc                             // 000000000240: BE80206A
+// CHECK:  s_cbranch_execz 85                                         // 000000000244: BF880055
+// CHECK:  s_load_dwordx4 s[8:11], s[4:5], 0x40                       // 000000000248: C00A0202 00000040
+// CHECK:  v_ashrrev_i32_e32 v77, 31, v76                             // 000000000250: 229A989F
+// CHECK:  v_lshlrev_b64 v[10:11], 2, v[76:77]                        // 000000000254: D28F000A 00029882
+// CHECK:  s_waitcnt lgkmcnt(0)                                       // 00000000025C: BF8C007F
+// CHECK:  v_add_i32_e32 v10, vcc, s8, v10                            // 000000000260: 32141408
+// CHECK:  v_mov_b32_e32 v6, s9                                       // 000000000264: 7E0C0209
+// CHECK:  v_addc_u32_e32 v11, vcc, v6, v11, vcc                      // 000000000268: 38161706
+// CHECK:  flat_load_dword v0, v[10:11]                               // 00000000026C: DC500000 0000000A
+// CHECK:  v_lshlrev_b32_e32 v6, 5, v8                                // 000000000274: 240C1085
+// CHECK:  v_lshlrev_b32_e32 v7, 2, v7                                // 000000000278: 240E0E82
+// CHECK:  s_endpgm                                                   // 00000000027C: BF810000

llvm/trunk/tools/llvm-objdump/llvm-objdump.cpp

while(!HeadTail.first.empty()) {
       Bytes = Bytes.slice(4);
       Address += 4;
       HeadTail = HeadTail.second.split('\n');
     }
     OS << " } " << PacketBundle.second;
   }
 };
 HexagonPrettyPrinter HexagonPrettyPrinterInst;
+
+class AMDGCNPrettyPrinter : public PrettyPrinter {
+public:
+  void printInst(MCInstPrinter &IP,
+                 const MCInst *MI,
+                 ArrayRef<uint8_t> Bytes,
+                 uint64_t Address,
+                 raw_ostream &OS,
+                 StringRef Annot,
+                 MCSubtargetInfo const &STI) override {
+    SmallString<40> InstStr;
+    raw_svector_ostream IS(InstStr);
+
+    IP.printInst(MI, IS, "", STI);
+
+    OS << left_justify(IS.str(), 60) << format("// %012X: ", Address);
+    typedef support::ulittle32_t U32;
+    for (auto D : makeArrayRef(reinterpret_cast<const U32*>(Bytes.data()),
+                               Bytes.size() / sizeof(U32)))
+      OS << format("%08X ", D);
+
+    if (!Annot.empty())
+      OS << "// " << Annot;
+  }
+};
+AMDGCNPrettyPrinter AMDGCNPrettyPrinterInst;
+
 PrettyPrinter &selectPrettyPrinter(Triple const &Triple) {
   switch(Triple.getArch()) {
   default:
     return PrettyPrinterInst;
   case Triple::hexagon:
     return HexagonPrettyPrinterInst;
+  case Triple::amdgcn:
+    return AMDGCNPrettyPrinterInst;
   }
 }
 }
 
 template <class ELFT>
 static std::error_code getRelocationValueString(const ELFObjectFile<ELFT> *Obj,
                                                 const RelocationRef &RelRef,
                                                 SmallVectorImpl<char> &Result) {
for (unsigned si = 0, se = Symbols.size(); si != se; ++si) {
           (si == se - 1) ? SectSize : Symbols[si + 1].first - SectionAddr;
       // Don't try to disassemble beyond the end of section contents.
       if (End > SectSize)
         End = SectSize;
       // If this symbol has the same address as the next symbol, then skip it.
       if (Start >= End)
         continue;
 
+      if (Obj->isELF() && Obj->getArch() == Triple::amdgcn) {
+        // make size 4 bytes folded
+        End = Start + ((End - Start) & ~0x3ull);
+        Start += 256; // add sizeof(amd_kernel_code_t)
+        // cut trailing zeroes - up to 256 bytes (align)
+        const uint64_t EndAlign = 256;
+        const auto Limit = End - (std::min)(EndAlign, End - Start);
+        while (End > Limit &&
+          *reinterpret_cast<const support::ulittle32_t*>(&Bytes[End - 4]) == 0)
+          End -= 4;
+      }
+
       outs() << '\n' << Symbols[si].second << ":\n";
 
 #ifndef NDEBUG
       raw_ostream &DebugOut = DebugFlag ? dbgs() : nulls();
 #else
       raw_ostream &DebugOut = nulls();
 #endif