Add MachO signature verification test

[nuriamari]

Summary

Add a test to ensure that MachO files including
a LC_CODE_SIGNATURE load command produced by lld
are signed correctly.

[int3]

lld/test/MachO/Inputs/code-signature-check.py

11

can we have a docstring describing what this script does?

[thevinster]

lld/test/MachO/Inputs/code-signature-check.py

11

+1 - Usage of the script would also be nice for future reader.

[int3]

lld/test/MachO/adhoc-codesign-hash.s

17

nit: try to vertically align where possible (same for lines below)

[oontvoo]

lld/test/MachO/Inputs/code-signature-check.py

2

Any chance this could be omitted for portability? (since you've already invoked the script with %python below)?
(also is python3 a must?)

lld/test/MachO/adhoc-codesign-hash.s

17–34

Perhaps consider using variables here to avoid duplication? (also easier for future updates?)

(suggested diff is in-complete -- only included one eg)

[nuriamari]

Address shebang, comment and spacing related feeback

[nuriamari]

lld/test/MachO/Inputs/code-signature-check.py

2

I'll remove the shebang, but the script does require at least python3.5 for the typing module.

lld/test/MachO/adhoc-codesign-hash.s

17–34

I haven't been able to get what you are suggesting working. Are FileCheck capture variables usable within a run command?

[int3]

lgtm

lld/test/MachO/adhoc-codesign-hash.s

17–34

What @oontvoo left out was how to define variables for FileCheck :) you'll want something like -D#DATAOFF=16400 here for the substitution to work. Then you can reference it via [[#DATAOFF]]. (The # indicates that the value is an integer. It only matters if you want to do arithmetic on the values, which doesn't apply in this case, but it's nice to use # anyway for clarity.)

[nuriamari]

Correct docstring argument description

[oontvoo]

In D109840#3002816, @int3 wrote:

What @oontvoo left out was how to define variables for FileCheck :) you'll want something like -D#DATAOFF=16400 here for the substitution to work. Then you can reference it via #DATAOFF. (The # indicates that the value is an integer. It only matters if you want to do arithmetic on the values, which doesn't apply in this case, but it's nice to use # anyway for clarity.)

Ah, yes - I'd forgot that part. Thanks for clarifying it!

not sure why this comment is not in-lined anymore. 🤔 @nuriamari : Do you use arc to update this patch?

[nuriamari]

In D109840#3003027, @oontvoo wrote:

In D109840#3002816, @int3 wrote:

What @oontvoo left out was how to define variables for FileCheck :) you'll want something like -D#DATAOFF=16400 here for the substitution to work. Then you can reference it via #DATAOFF. (The # indicates that the value is an integer. It only matters if you want to do arithmetic on the values, which doesn't apply in this case, but it's nice to use # anyway for clarity.)

Ah, yes - I'd forgot that part. Thanks for clarifying it!

not sure why this comment is not in-lined anymore. 🤔 @nuriamari : Do you use arc to update this patch?

Yes, I am using arc to update the patch, I did notice inline comments from previous revisions disappear, but if you change the diff viewer to show the first revision, the inline comments remain. In any case, I will add the variables to the FileCheck lines. Thanks!

[nuriamari]

Use FileCheck variables in test

lld/test/MachO/Inputs/code-signature-check.py

+"""Checks the validity of MachO binary signatures
+
+MachO binaries sometimes include a LC_CODE_SIGNATURE load command
+and corresponding section in the __LINKEDIT segment that together
+work to "sign" the binary. This script is used to check the validity
+of this signature.
+
+Usage:
+    ./code-signature-check.py my_binary 800 300 0 800
+
+Arguments:
+   binary - The MachO binary to be tested
+   offset - The offset from the start of the binary to where the code signature section begins
+   size - The size of the code section in the binary
+   code_offset - The point in the binary to begin hashing
+   code_size - The length starting from code_offset to hash
+"""
+
+import argparse
+import collections
+import hashlib
+import itertools
+import struct
+import sys
+import typing
+
+class CodeDirectoryVersion:
+    SUPPORTSSCATTER = 0x20100
+    SUPPORTSTEAMID = 0x20200
+    SUPPORTSCODELIMIT64 = 0x20300
+    SUPPORTSEXECSEG = 0x20400
+
+class CodeDirectory:
+    @staticmethod
+    def make(buf: memoryview) -> typing.Union['CodeDirectoryBase', 'CodeDirectoryV20100', 'CodeDirectoryV20200', 'CodeDirectoryV20300', 'CodeDirectoryV20400']:
+        _magic, _length, version = struct.unpack_from(">III", buf, 0)
+        subtype = {
+            CodeDirectoryVersion.SUPPORTSSCATTER: CodeDirectoryV20100,
+            CodeDirectoryVersion.SUPPORTSTEAMID: CodeDirectoryV20200,
+            CodeDirectoryVersion.SUPPORTSCODELIMIT64: CodeDirectoryV20300,
+            CodeDirectoryVersion.SUPPORTSEXECSEG: CodeDirectoryV20400,
+        }.get(version, CodeDirectoryBase)
+
+        return subtype._make(struct.unpack_from(subtype._format(), buf, 0))
+
+class CodeDirectoryBase(typing.NamedTuple):
+    magic: int
+    length: int
+    version: int
+    flags: int
+    hashOffset: int
+    identOffset: int
+    nSpecialSlots: int
+    nCodeSlots: int
+    codeLimit: int
+    hashSize: int
+    hashType: int
+    platform: int
+    pageSize: int
+    spare2: int
+
+    @staticmethod
+    def _format() -> str:
+        return ">IIIIIIIIIBBBBI"
+
+class CodeDirectoryV20100(typing.NamedTuple):
+    magic: int
+    length: int
+    version: int
+    flags: int
+    hashOffset: int
+    identOffset: int
+    nSpecialSlots: int
+    nCodeSlots: int
+    codeLimit: int
+    hashSize: int
+    hashType: int
+    platform: int
+    pageSize: int
+    spare2: int
+
+    scatterOffset: int
+
+    @staticmethod
+    def _format() -> str:
+        return CodeDirectoryBase._format() + "I"
+
+class CodeDirectoryV20200(typing.NamedTuple):
+    magic: int
+    length: int
+    version: int
+    flags: int
+    hashOffset: int
+    identOffset: int
+    nSpecialSlots: int
+    nCodeSlots: int
+    codeLimit: int
+    hashSize: int
+    hashType: int
+    platform: int
+    pageSize: int
+    spare2: int
+
+    scatterOffset: int
+
+    teamOffset: int
+
+    @staticmethod
+    def _format() -> str:
+        return CodeDirectoryV20100._format() + "I"
+
+class CodeDirectoryV20300(typing.NamedTuple):
+    magic: int
+    length: int
+    version: int
+    flags: int
+    hashOffset: int
+    identOffset: int
+    nSpecialSlots: int
+    nCodeSlots: int
+    codeLimit: int
+    hashSize: int
+    hashType: int
+    platform: int
+    pageSize: int
+    spare2: int
+
+    scatterOffset: int
+
+    teamOffset: int
+
+    spare3: int
+    codeLimit64: int
+
+    @staticmethod
+    def _format() -> str:
+        return CodeDirectoryV20200._format() + "IQ"
+
+class CodeDirectoryV20400(typing.NamedTuple):
+    magic: int
+    length: int
+    version: int
+    flags: int
+    hashOffset: int
+    identOffset: int
+    nSpecialSlots: int
+    nCodeSlots: int
+    codeLimit: int
+    hashSize: int
+    hashType: int
+    platform: int
+    pageSize: int
+    spare2: int
+
+    scatterOffset: int
+
+    teamOffset: int
+
+    spare3: int
+    codeLimit64: int
+
+    execSegBase: int
+    execSegLimit: int
+    execSegFlags: int
+
+    @staticmethod
+    def _format() -> str:
+        return CodeDirectoryV20300._format() + "QQQ"
+
+class CodeDirectoryBlobIndex(typing.NamedTuple):
+    type_: int
+    offset: int
+
+    @staticmethod
+    def make(buf: memoryview) -> 'CodeDirectoryBlobIndex':
+        return CodeDirectoryBlobIndex._make(struct.unpack_from(CodeDirectoryBlobIndex.__format(), buf, 0))
+
+    @staticmethod
+    def bytesize() -> int:
+        return struct.calcsize(CodeDirectoryBlobIndex.__format())
+
+    @staticmethod
+    def __format() -> str:
+        return ">II"
+
+class CodeDirectorySuperBlob(typing.NamedTuple):
+    magic: int
+    length: int
+    count: int
+    blob_indices: typing.List[CodeDirectoryBlobIndex]
+
+    @staticmethod
+    def make(buf: memoryview) -> 'CodeDirectorySuperBlob':
+        super_blob_layout = ">III"
+        super_blob = struct.unpack_from(super_blob_layout, buf, 0)
+
+        offset = struct.calcsize(super_blob_layout)
+        blob_indices = []
+        for idx in range(super_blob[2]):
+            blob_indices.append(CodeDirectoryBlobIndex.make(buf[offset:]))
+            offset += CodeDirectoryBlobIndex.bytesize()
+
+        return CodeDirectorySuperBlob(*super_blob, blob_indices)
+
+def unpack_null_terminated_string(buf: memoryview) -> str:
+    b = bytes(itertools.takewhile(lambda b: b != 0, buf))
+    return b.decode()
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('binary', type=argparse.FileType('rb'), help='The file to analyze')
+    parser.add_argument('offset', type=int, help='Offset to start of Code Directory data')
+    parser.add_argument('size', type=int, help='Size of Code Directory data')
+    parser.add_argument('code_offset', type=int, help='Offset to start of code pages to hash')
+    parser.add_argument('code_size', type=int, help='Size of the code pages to hash')
+
+    args = parser.parse_args()
+
+    args.binary.seek(args.offset)
+    super_blob_bytes = args.binary.read(args.size)
+    super_blob_mem = memoryview(super_blob_bytes)
+
+    super_blob = CodeDirectorySuperBlob.make(super_blob_mem)
+    print(super_blob)
+
+    for blob_index in super_blob.blob_indices:
+        code_directory_offset = blob_index.offset
+        code_directory = CodeDirectory.make(super_blob_mem[code_directory_offset:])
+        print(code_directory)
+
+        ident_offset = code_directory_offset + code_directory.identOffset
+        print("Code Directory ID: " + unpack_null_terminated_string(super_blob_mem[ident_offset:]))
+
+        code_offset = args.code_offset
+        code_end = code_offset + args.code_size
+        page_size = 1 << code_directory.pageSize
+        args.binary.seek(code_offset)
+
+        hashes_offset = code_directory_offset + code_directory.hashOffset
+        for idx in range(code_directory.nCodeSlots):
+            hash_bytes = bytes(super_blob_mem[hashes_offset:hashes_offset+code_directory.hashSize])
+            hashes_offset += code_directory.hashSize
+
+            hasher = hashlib.sha256()
+            read_size = min(page_size, code_end - code_offset)
+            hasher.update(args.binary.read(read_size))
+            calculated_hash_bytes = hasher.digest()
+            code_offset += read_size
+
+            print("%s <> %s" % (hash_bytes.hex(), calculated_hash_bytes.hex()))
+
+            if hash_bytes != calculated_hash_bytes:
+                sys.exit(-1)
+
+
+if __name__ == '__main__':
+    main()

lld/test/MachO/adhoc-codesign-hash.s

+# REQUIRES: x86, aarch64
+# RUN: rm -rf %t; mkdir -p %t
+
+# RUN: llvm-mc -filetype=obj -triple=arm64-apple-macos -o %t/empty-arm64-macos.o %s
+# RUN: llvm-mc -filetype=obj -triple=arm64-apple-iossimulator -o %t/empty-arm64-iossimulator.o %s
+# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-macos -o %t/empty-x86_64-macos.o %s
+
+# RUN: %lld -arch arm64 -dylib -adhoc_codesign -o %t/empty-arm64-macos.dylib %t/empty-arm64-macos.o
+# RUN: %lld -arch arm64 -dylib -adhoc_codesign -o %t/empty-arm64-iossimulator.dylib %t/empty-arm64-iossimulator.o
+# RUN: %lld -arch x86_64 -dylib -adhoc_codesign -o %t/empty-x86_64-macos.dylib %t/empty-x86_64-macos.o
+
+# RUN: obj2yaml %t/empty-arm64-macos.dylib | FileCheck %s --check-prefix=ARM64-MACOS
+# RUN: obj2yaml %t/empty-arm64-iossimulator.dylib | FileCheck %s --check-prefix=ARM64-IOS
+# RUN: obj2yaml %t/empty-x86_64-macos.dylib | FileCheck %s --check-prefix=X86_64-MACOS
+
+# ARM64-MACOS:    - cmd:             LC_CODE_SIGNATURE
+# ARM64-MACOS-NEXT: cmdsize:         16
+# ARM64-MACOS-NEXT: dataoff:         16400
+# ARM64-MACOS-NEXT: datasize:        304
+
+# ARM64-IOS:    - cmd:             LC_CODE_SIGNATURE
+# ARM64-IOS-NEXT: cmdsize:         16
+# ARM64-IOS-NEXT: dataoff:         16400
+# ARM64-IOS-NEXT: datasize:        304
+
+# X86_64-MACOS:    - cmd:             LC_CODE_SIGNATURE
+# X86_64-MACOS-NEXT: cmdsize:         16
+# X86_64-MACOS-NEXT: dataoff:         4112
+# X86_64-MACOS-NEXT: datasize:        208
+
+# RUN: %python %p/Inputs/code-signature-check.py %t/empty-arm64-macos.dylib 16400 304 0 16400
+# RUN: %python %p/Inputs/code-signature-check.py %t/empty-arm64-iossimulator.dylib 16400 304 0 16400
+# RUN: %python %p/Inputs/code-signature-check.py %t/empty-x86_64-macos.dylib 4112 208 0 4112