[BOLT] Check for FileOffset equality when aligned

[sebpop]

Summary

Running bolt on arm64-linux envoy fails:

git clone https://github.com/envoyproxy/envoy.git
cd envoy
./ci/run_envoy_docker.sh 'BAZEL_BUILD_EXTRA_OPTIONS="--copt=-gdwarf-4 --linkopt=-Wl,--emit-relocs" ./ci/do_ci.sh bazel.release.server_only'
cd /tmp/envoy-docker-build/envoy/source/exe/envoy
perf2bolt -p ~/perf.data -o ~/perf.fdata --nl envoy

This fails in lib/Profile/DataAggregator.cpp with:

PERF2BOLT-ERROR: could not find a profile matching binary "envoy". Profile for the following binary name(s) is available:

The following condition does not match:

if (alignDown(SegInfo.FileOffset, SegInfo.Alignment) == FileOffset)

(gdb) p /x FileOffset
$1 = 0x14ec000
(gdb) p /x SegInfo.FileOffset
$2 = 0x14ec000
(gdb) p /x SegInfo.Alignment
$3 = 0x10000

The patch checks the equality of SegInfo.FileOffset and FileOffset when both are
alignDown by SegInfo.Alignment:

(gdb) p /x alignDown(SegInfo.FileOffset, SegInfo.Alignment)
$4 = 0x14e0000
(gdb) p /x alignDown(FileOffset, SegInfo.Alignment)
$5 = 0x14e0000

[maksfb]

bolt/lib/Rewrite/RewriteInstance.cpp

426 ↗

(On Diff #499828)

Obj.base() will give you an address of internal mapping of the file by llvm-bolt, right? It doesn't necessarily translate to the mapping of the file at runtime and the alignment could be different too.

[sebpop]

bolt/lib/Rewrite/RewriteInstance.cpp

426 ↗

(On Diff #499828)

You are right, Obj.base() will have the alignment of the object file as opened by llvm-bolt.

The fail is when the phdr alignment is 0x10000, here's the SegInfo:

{Address = 0x14fc000, Size = 0x2305800, FileOffset = 0x14ec000, FileSize = 0x2305800, Alignment = 0x10000}

The Alignment does not match the alignment for FileOffset and Address.
Alignment would need to be 0x1000 for the condition to match:

if (alignDown(SegInfo.FileOffset, SegInfo.Alignment) == FileOffset)

I am wondering why the linker does not write a correct alignment in the program headers.

[maksfb]

bolt/lib/Rewrite/RewriteInstance.cpp

426 ↗

(On Diff #499828)

The "Alignment" in our dump is a bit misleading. It corresponds to p_align field in the program header, and the matching segment should be aligned to this value while loaded in memory (with a padding). Here's an excerpt from "Linkers and Libraries Guide":

Loadable process segments must have congruent values for p_vaddr and p_offset, modulo
the page size. This member gives the value to which the segments are aligned in memory and
in the file. Values 0 and 1 mean no alignment is required. Otherwise, p_align should be a
positive, integral power of 2, and p_vaddr should equal p_offset, modulo p_align.

I.e., the SegInfo looks correct, but my guess is the value of MMapAddress passed to getBaseAddressForMapping is not aligned to the required value?

[sebpop]

The updated patch checks the equality of SegInfo.FileOffset and FileOffset when both are alignDown by SegInfo.Alignment:

(gdb) p /x FileOffset
$1 = 0x14ec000
(gdb) p /x SegInfo.FileOffset
$2 = 0x14ec000
(gdb) p /x SegInfo.Alignment
$3 = 0x10000
(gdb) p /x alignDown(SegInfo.FileOffset, SegInfo.Alignment)
$4 = 0x14e0000
(gdb) p /x alignDown(FileOffset, SegInfo.Alignment)
$5 = 0x14e0000

[sebpop]

bolt/lib/Rewrite/RewriteInstance.cpp

426 ↗

(On Diff #499828)

Thanks for the pointer to the "Linkers and Libraries Guide". That's a useful book.

The value of MMapAddress does not get used in the search algorithm that fails.
It only gets used at the end of the computation on return MMapAddress - MemOffset;

[sebpop]

Ping patch.

[Amir]

I'm having the same issue with 64k page system but it's not addressed by this fix: https://github.com/llvm/llvm-project/issues/61370

bolt/lib/Core/BinaryContext.cpp

 }
 
 std::optional<uint64_t>
 BinaryContext::getBaseAddressForMapping(uint64_t MMapAddress,
                                         uint64_t FileOffset) const {
   // Find a segment with a matching file offset.
   for (auto &KV : SegmentMapInfo) {
     const SegmentInfo &SegInfo = KV.second;
-    if (alignDown(SegInfo.FileOffset, SegInfo.Alignment) == FileOffset) {
+    if (alignDown(SegInfo.FileOffset, SegInfo.Alignment) ==
+        alignDown(FileOffset, SegInfo.Alignment)) {
       // Use segment's aligned memory offset to calculate the base address.
       const uint64_t MemOffset = alignDown(SegInfo.Address, SegInfo.Alignment);
       return MMapAddress - MemOffset;
     }
   }
 
   return std::nullopt;
 }