Implement some functions in NativeSession.

[akhuang]

Summary

This change implements readFromExe, and calculating VA and RVA, which
are some of the functionalities that will be used for native PDB reading
for llvm symbolizer.

bug: https://bugs.llvm.org/show_bug.cgi?id=41795

[akhuang]

I'll also add unit tests for these

[amccarth]

The load address stuff looks fine.

I think the bit that tries to find the PDB file probably needs more flexibility or a bypass for the LLDB and Crash folks. I think we should include Pavel (labath) for his opinion on that bit. I recall there was lots of discussion about this in the Crash/Lexan meetings long ago.

llvm/lib/DebugInfo/PDB/Native/NativeSession.cpp

163

I think the LLDB folks might be concerned about searching the file system for the corresponding PDB file. The debugger folks are probably going to want a way to override that for remote debugging, symbol servers, virtual file systems, etc.

Perhaps the right thing to do is to have an API that takes paths to both the EXE and the PDB. If everything loads, great, otherwise return an error. Then you could have an API like this that tries to find the PDB, but rather than having it then load the session, it just forwards the PDB path to the two-path API.

That would make it possible for a client to get the automatic behavior you have here while also enabling LLDB to be explicit about where the PDB should be.

[amccarth]

Per my previous comment, I've added Pavel as a reviewer.

[akhuang]

Add unit tests

[labath]

In D78128#1982184, @amccarth wrote:

The load address stuff looks fine.

I think the bit that tries to find the PDB file probably needs more flexibility or a bypass for the LLDB and Crash folks. I think we should include Pavel (labath) for his opinion on that bit. I recall there was lots of discussion about this in the Crash/Lexan meetings long ago.

If I am not mistaken the "native" PDB plugin (which is the "future" for lldb ) does not use this code path at all -- it goes directly for the llvm::pdb::***Stream classes. As such, I don't think any special hooks need to be added here. Which isn't to say that searching the CWD unconditionally is a particularly nice interface for a library, but it is pretty similar to what the llvm::DWARFContext does...

[rnk]

llvm/lib/DebugInfo/PDB/Native/NativeSession.cpp

82–141

Maybe rename Path to ExePath so the code below that replaces the exe filename with the pdb filename is a bit more readable.

98

I see what you mean that it was mostly unimplemented. :)

163

Building on Adrian's idea, I think the API might look kind of like this:

struct PdbSearchOptions {
  std::string ExePath;
  // Options we can add later:
  // std::vector<std::string> SearchDirs;
  // bool UseEnvNtSymbolPath = false;
  // llvm::VirtualFileSystem *SearchVFS = nullptr; // Way to remap paths
};
Expected<std::string> searchForPdb(const PdbSearchOptions &Opts) {
  ...
}

The search would return the first existing file with matching PDB magic.

The process of searching for a PDB file can be quite involved. Here are some things that we may implement one day:

• _NT_SYMBOL_PATH: an environment variable with PDB search paths
• Symbol servers: Maybe we will want to learn to talk to MS symbol servers to download PDBs

Even if we don't implement the symsrv protocol, it would be useful to teach LLVM how to find PDBs in a symbol server cache. For example, I have these kernel32.dll PDBs in C:\symbols, which windbg put them:

$ ls /c/symbols/kernel32.pdb/*
/c/symbols/kernel32.pdb/548CAA89F1FCF0193D12D2DC02F8920E1:
kernel32.pdb

/c/symbols/kernel32.pdb/63816243EC704DC091BC31470BAC48A31:
kernel32.pdb

If llvm-symbolizer wants to be able to symbolize system DLLs, that will be our best bet.

213

Checking the sign bit makes sense, but these seem like simpler ways to express that:

• (int32_t) RVA < 0
• RVA > INT32_MAX (unsure on macro name)

221

If we can subtract the virtual address out of it, are you sure it is an RVA? It seems like it's supposed to be a virtual address... If it's a virtual address, though, I would expect it to require a uint64_t. Hm.

llvm/unittests/DebugInfo/PDB/Inputs/SimpleTest.cpp

5

I tried to think of how to use yaml2pdb to avoid the binary files, but I don't like any of my solutions. I'm OK with the binary files.

[akhuang]

Address comments about adding a separate path to search for the PDB.

[akhuang]

llvm/lib/DebugInfo/PDB/Native/NativeSession.cpp

221

I think the section header virtual addresses are also relative to the load address.

[akhuang]

Fix loadPdbFile function to take reference to Allocator

[amccarth]

Thanks. I appreciate the separation of concerns with regard to finding and loading of the PDB.

[MaskRay]

Hi, llvm/object/COFF.h does not work on a case-sensitive filesystem. I fixed it in 4927ae085807731eb4052e0a682443fe9399b512. BTW, you can strip unneeded Phabricator tags (Subscribers: Tags: Reviewers:) with

arcfilter () {
        arc amend
        git log -1 --pretty=%B | awk '/Reviewers:|Subscribers:/{p=1} /Reviewed By:|Differential Revision:/{p=0} !p && !/^Summary:$/ {sub(/^Summary: /,"");print}' | git commit --amend -F -
}

[akhuang]

@MaskRay Oh, whoops. Thanks for fixing!

[MaskRay]

http://45.33.8.238/linux/15836/step_12.txt

After fixing the -DLLVM_ENABLE_ABI_BREAKING_CHECKS=on problem locally, the unittest (unittests/DebugInfo/PDB/DebugInfoPDBTests) still fails.

(
An erroring Expected must be explicitly checked to avoid -DLLVM_ENABLE_ABI_BREAKING_CHECKS=on failures.

if (Expected<std::unique_ptr<PDBFile>> File = loadPdbFile(PdbPath, Allocator))
  return std::string(PdbPath);
else
  return File.takeError();

)

[hliao]

In D78128#1995209, @MaskRay wrote:

http://45.33.8.238/linux/15836/step_12.txt

After fixing the -DLLVM_ENABLE_ABI_BREAKING_CHECKS=on problem locally, the unittest (unittests/DebugInfo/PDB/DebugInfoPDBTests) still fails.

(
An erroring Expected must be explicitly checked to avoid -DLLVM_ENABLE_ABI_BREAKING_CHECKS=on failures.

if (Expected<std::unique_ptr<PDBFile>> File = loadPdbFile(PdbPath, Allocator))
  return std::string(PdbPath);
else
  return File.takeError();

)

Even more, as the test reads the PDB file path from that EXE file (llvm-project/llvm/unittests/DebugInfo/PDB/Inputs/SimpleTest.exe), that path is an absolute path in Windows format (C:\\src\\llvm-project\\llvm\\unittests\\DebugInfo\\PDB\\Inputs\\SimpleTest.pdb). On Linux builds, that always fail and, on Windows, we are only luck if we have the same dir layout by chance. Could you fix that by taking relative path and disable that test on non-Windows platforms?

[akhuang]

Actually, I will just revert for now.

[akhuang]

Relanded in 23609331472be22be3be134f6facd8f086c5ea49 so that tests pass on Linux. I changed the PDB searching part to specify the path style, so that it correctly finds the filename on Linux.

llvm/include/llvm/DebugInfo/PDB/Native/NativeSession.h

 
 namespace llvm {
 class MemoryBuffer;
 namespace pdb {
 class PDBFile;
 class NativeExeSymbol;
 
 class NativeSession : public IPDBSession {
+  struct PdbSearchOptions {
+    StringRef ExePath;
+    // FIXME: Add other PDB search options (_NT_SYMBOL_PATH, symsrv)
+  };
+
 public:
   NativeSession(std::unique_ptr<PDBFile> PdbFile,
                 std::unique_ptr<BumpPtrAllocator> Allocator);
   ~NativeSession() override;
 
   static Error createFromPdb(std::unique_ptr<MemoryBuffer> MB,
                              std::unique_ptr<IPDBSession> &Session);
+  static Error createFromPdbPath(StringRef PdbPath,
+                                 std::unique_ptr<IPDBSession> &Session);
   static Error createFromExe(StringRef Path,
                              std::unique_ptr<IPDBSession> &Session);
+  static Expected<std::string> searchForPdb(const PdbSearchOptions &Opts);
 
   uint64_t getLoadAddress() const override;
   bool setLoadAddress(uint64_t Address) override;
   std::unique_ptr<PDBSymbolExe> getGlobalScope() override;
   std::unique_ptr<PDBSymbol> getSymbolById(SymIndexId SymbolId) const override;
 
   bool addressForVA(uint64_t VA, uint32_t &Section,
                     uint32_t &Offset) const override;

 private:
   void initializeExeSymbol();
 
   std::unique_ptr<PDBFile> Pdb;
   std::unique_ptr<BumpPtrAllocator> Allocator;
 
   SymbolCache Cache;
   SymIndexId ExeSymbol = 0;
+  uint64_t LoadAddress = 0;
 };
 } // namespace pdb
 } // namespace llvm
 
 #endif

llvm/lib/DebugInfo/PDB/Native/NativeSession.cpp

 //===- NativeSession.cpp - Native implementation of IPDBSession -*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/PDB/Native/NativeSession.h"
 
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/DebugInfo/CodeView/TypeIndex.h"
 #include "llvm/DebugInfo/PDB/IPDBEnumChildren.h"
 #include "llvm/DebugInfo/PDB/IPDBSourceFile.h"
+#include "llvm/DebugInfo/PDB/Native/DbiStream.h"
 #include "llvm/DebugInfo/PDB/Native/NativeCompilandSymbol.h"
 #include "llvm/DebugInfo/PDB/Native/NativeEnumInjectedSources.h"
 #include "llvm/DebugInfo/PDB/Native/NativeEnumTypes.h"
 #include "llvm/DebugInfo/PDB/Native/NativeExeSymbol.h"
 #include "llvm/DebugInfo/PDB/Native/NativeTypeBuiltin.h"
 #include "llvm/DebugInfo/PDB/Native/NativeTypeEnum.h"
 #include "llvm/DebugInfo/PDB/Native/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Native/RawError.h"
 #include "llvm/DebugInfo/PDB/Native/SymbolCache.h"
 #include "llvm/DebugInfo/PDB/Native/TpiStream.h"
 #include "llvm/DebugInfo/PDB/PDBSymbolCompiland.h"
 #include "llvm/DebugInfo/PDB/PDBSymbolExe.h"
 #include "llvm/DebugInfo/PDB/PDBSymbolTypeEnum.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/BinaryByteStream.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorOr.h"
+#include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/object/COFF.h"
 
 #include <algorithm>
 #include <cassert>
 #include <memory>
 #include <utility>
 
 using namespace llvm;
 using namespace llvm::msf;
if (auto EC = File->parseStreamData())
     return EC;
 
   Session =
       std::make_unique<NativeSession>(std::move(File), std::move(Allocator));
 
   return Error::success();
 }
 
-Error NativeSession::createFromExe(StringRef Path,
+static Expected<std::unique_ptr<PDBFile>>
+loadPdbFile(StringRef PdbPath, std::unique_ptr<BumpPtrAllocator> &Allocator) {
+  ErrorOr<std::unique_ptr<MemoryBuffer>> ErrorOrBuffer =
+      MemoryBuffer::getFile(PdbPath, /*FileSize=*/-1,
+                            /*RequiresNullTerminator=*/false);
+  if (!ErrorOrBuffer)
+    return make_error<RawError>(ErrorOrBuffer.getError());
+  std::unique_ptr<llvm::MemoryBuffer> Buffer = std::move(*ErrorOrBuffer);
+
+  PdbPath = Buffer->getBufferIdentifier();
+  file_magic Magic;
+  auto EC = identify_magic(PdbPath, Magic);
+  if (EC || Magic != file_magic::pdb)
+    return make_error<RawError>(EC);
+
+  auto Stream = std::make_unique<MemoryBufferByteStream>(std::move(Buffer),
+                                                         llvm::support::little);
+
+  auto File = std::make_unique<PDBFile>(PdbPath, std::move(Stream), *Allocator);
+  if (auto EC = File->parseFileHeaders())
+    return std::move(EC);
+
+  if (auto EC = File->parseStreamData())
+    return std::move(EC);
+
+  return File;
+}
+
+Error NativeSession::createFromPdbPath(StringRef PdbPath,
                                        std::unique_ptr<IPDBSession> &Session) {
-  return make_error<RawError>(raw_error_code::feature_unsupported);
+  auto Allocator = std::make_unique<BumpPtrAllocator>();
+  auto PdbFile = loadPdbFile(PdbPath, Allocator);
+  if (!PdbFile)
+    return PdbFile.takeError();
+
+  Session = std::make_unique<NativeSession>(std::move(PdbFile.get()),
+                                            std::move(Allocator));
+  return Error::success();
+}
+
+static Expected<std::string> getPdbPathFromExe(StringRef ExePath) {
+  Expected<object::OwningBinary<object::Binary>> BinaryFile =
+      object::createBinary(ExePath);
+  if (!BinaryFile)
+    return BinaryFile.takeError();
+
+  const object::COFFObjectFile *ObjFile =
+      dyn_cast<object::COFFObjectFile>(BinaryFile->getBinary());
+  if (!ObjFile)
+    return make_error<RawError>(raw_error_code::invalid_format);
+
+  StringRef PdbPath;
+  const llvm::codeview::DebugInfo *PdbInfo = nullptr;
+  if (auto EC = ObjFile->getDebugPDBInfo(PdbInfo, PdbPath))
+    return make_error<RawError>(EC);
+
+  return std::string(PdbPath);
 }
 
-uint64_t NativeSession::getLoadAddress() const { return 0; }
+Error NativeSession::createFromExe(StringRef ExePath,
+                                   std::unique_ptr<IPDBSession> &Session) {
+  Expected<std::string> PdbPath = getPdbPathFromExe(ExePath);
+  if (!PdbPath)
+    return PdbPath.takeError();
+
+  file_magic Magic;
+  auto EC = identify_magic(PdbPath.get(), Magic);
+  if (EC || Magic != file_magic::pdb)
+    return make_error<RawError>(EC);
+
+  auto Allocator = std::make_unique<BumpPtrAllocator>();
+  auto File = loadPdbFile(PdbPath.get(), Allocator);
+  if (!File)
+    return File.takeError();
 
-bool NativeSession::setLoadAddress(uint64_t Address) { return false; }
+  Session = std::make_unique<NativeSession>(std::move(File.get()),
+                                            std::move(Allocator));
+
+  return Error::success();
+}
+
+Expected<std::string>
+NativeSession::searchForPdb(const PdbSearchOptions &Opts) {
+  Expected<std::string> PathOrErr = getPdbPathFromExe(Opts.ExePath);
+  if (!PathOrErr)
+    return PathOrErr.takeError();
+  StringRef PdbName = sys::path::filename(PathOrErr.get());
+
+  // Check if pdb exists in the executable directory.
+  SmallString<128> PdbPath = StringRef(Opts.ExePath);
+  sys::path::remove_filename(PdbPath);
+  sys::path::append(PdbPath, PdbName);
+
+  auto Allocator = std::make_unique<BumpPtrAllocator>();
+  if (loadPdbFile(PdbPath, Allocator))
+    return std::string(PdbPath);
+
+  return make_error<RawError>("PDB not found");
+}
+
+uint64_t NativeSession::getLoadAddress() const { return LoadAddress; }
+
+bool NativeSession::setLoadAddress(uint64_t Address) {
+  LoadAddress = Address;
+  return true;
+}
 
 std::unique_ptr<PDBSymbolExe> NativeSession::getGlobalScope() {
   return PDBSymbol::createAs<PDBSymbolExe>(*this, getNativeGlobalScope());
 }
 
 std::unique_ptr<PDBSymbol>
 NativeSession::getSymbolById(SymIndexId SymbolId) const {
   return Cache.getSymbolById(SymbolId);
 }
 
 bool NativeSession::addressForVA(uint64_t VA, uint32_t &Section,
                                  uint32_t &Offset) const {
-  return false;
+  uint32_t RVA = VA - getLoadAddress();
+  return addressForRVA(RVA, Section, Offset);
 }
 
-bool NativeSession::addressForRVA(uint32_t VA, uint32_t &Section,
+bool NativeSession::addressForRVA(uint32_t RVA, uint32_t &Section,
                                   uint32_t &Offset) const {
+  auto Dbi = Pdb->getPDBDbiStream();
+  if (!Dbi)
     return false;
+
+  Section = 0;
+  Offset = 0;
+
+  if ((int32_t)RVA < 0)
+    return true;
+
+  Offset = RVA;
+  for (; Section < Dbi->getSectionHeaders().size(); ++Section) {
+    auto &Sec = Dbi->getSectionHeaders()[Section];
+    if (RVA < Sec.VirtualAddress)
+      return true;
+    Offset = RVA - Sec.VirtualAddress;
+  }
+  return true;
 }
 
 std::unique_ptr<PDBSymbol>
 NativeSession::findSymbolByAddress(uint64_t Address, PDB_SymType Type) const {
   return nullptr;
 }
 
 std::unique_ptr<PDBSymbol>

llvm/lib/DebugInfo/PDB/PDB.cpp

 #include "llvm/Support/MemoryBuffer.h"
 
 using namespace llvm;
 using namespace llvm::pdb;
 
 Error llvm::pdb::loadDataForPDB(PDB_ReaderType Type, StringRef Path,
                                 std::unique_ptr<IPDBSession> &Session) {
   // Create the correct concrete instance type based on the value of Type.
-  if (Type == PDB_ReaderType::Native) {
-    ErrorOr<std::unique_ptr<MemoryBuffer>> ErrorOrBuffer =
-        MemoryBuffer::getFileOrSTDIN(Path, /*FileSize=*/-1,
-                                     /*RequiresNullTerminator=*/false);
-    if (!ErrorOrBuffer)
-      return errorCodeToError(ErrorOrBuffer.getError());
-
-    return NativeSession::createFromPdb(std::move(*ErrorOrBuffer), Session);
-  }
+  if (Type == PDB_ReaderType::Native)
+    return NativeSession::createFromPdbPath(Path, Session);
 
 #if LLVM_ENABLE_DIA_SDK
   return DIASession::createFromPdb(Path, Session);
 #else
   return make_error<PDBError>(pdb_error_code::dia_sdk_not_present);
 #endif
 }
 
 Error llvm::pdb::loadDataForEXE(PDB_ReaderType Type, StringRef Path,
                                 std::unique_ptr<IPDBSession> &Session) {
   // Create the correct concrete instance type based on the value of Type.
-  if (Type == PDB_ReaderType::Native)
-    return NativeSession::createFromExe(Path, Session);
+  if (Type == PDB_ReaderType::Native) {
+    if (auto Err = NativeSession::createFromExe(Path, Session)) {
+      consumeError(std::move(Err));
+
+      Expected<std::string> PdbPath = NativeSession::searchForPdb({Path});
+      if (!PdbPath)
+        return PdbPath.takeError();
+      return NativeSession::createFromPdbPath(PdbPath.get(), Session);
+    }
+    return Error::success();
+  }
 
 #if LLVM_ENABLE_DIA_SDK
   return DIASession::createFromExe(Path, Session);
 #else
   return make_error<PDBError>(pdb_error_code::dia_sdk_not_present);
 #endif
 }

llvm/unittests/DebugInfo/PDB/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   DebugInfoCodeView
   DebugInfoMSF
   DebugInfoPDB
   )
 
 add_llvm_unittest_with_input_files(DebugInfoPDBTests
   HashTableTest.cpp
+  NativeSessionTest.cpp
   NativeSymbolReuseTest.cpp
   StringTableBuilderTest.cpp
   PDBApiTest.cpp
   )
 
 target_link_libraries(DebugInfoPDBTests PRIVATE LLVMTestingSupport)

llvm/unittests/DebugInfo/PDB/Inputs/SimpleTest.cpp

+// Compile with "cl /c /Zi SimpleTest.cpp"
+// Link with "link SimpleTest.obj /debug /nodefaultlib /entry:main"
+
+int main() { return 0; }

llvm/unittests/DebugInfo/PDB/NativeSessionTest.cpp

+//===- llvm/unittest/DebugInfo/PDB/NativeSessionTest.cpp ------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DebugInfo/PDB/Native/NativeSession.h"
+#include "llvm/DebugInfo/PDB/IPDBSession.h"
+#include "llvm/DebugInfo/PDB/PDB.h"
+#include "llvm/Support/Path.h"
+
+#include "llvm/Testing/Support/Error.h"
+
+#include "gtest/gtest.h"
+
+#include <vector>
+
+using namespace llvm;
+using namespace llvm::pdb;
+
+extern const char *TestMainArgv0;
+
+static std::string getExePath() {
+  SmallString<128> InputsDir = unittest::getInputFileDirectory(TestMainArgv0);
+  llvm::sys::path::append(InputsDir, "SimpleTest.exe");
+  return std::string(InputsDir);
+}
+
+TEST(NativeSessionTest, TestCreateFromExe) {
+  std::unique_ptr<IPDBSession> S;
+  Error E = pdb::loadDataForEXE(PDB_ReaderType::Native, getExePath(), S);
+  ASSERT_THAT_ERROR(std::move(E), Succeeded());
+}
+
+TEST(NativeSessionTest, TestSetLoadAddress) {
+  std::unique_ptr<IPDBSession> S;
+  Error E = pdb::loadDataForEXE(PDB_ReaderType::Native, getExePath(), S);
+  ASSERT_THAT_ERROR(std::move(E), Succeeded());
+
+  S->setLoadAddress(123);
+  EXPECT_EQ(S->getLoadAddress(), 123U);
+}
+
+TEST(NativeSessionTest, TestAddressForVA) {
+  std::unique_ptr<IPDBSession> S;
+  Error E = pdb::loadDataForEXE(PDB_ReaderType::Native, getExePath(), S);
+  ASSERT_THAT_ERROR(std::move(E), Succeeded());
+
+  uint64_t LoadAddr = S->getLoadAddress();
+  uint32_t Section;
+  uint32_t Offset;
+  ASSERT_TRUE(S->addressForVA(LoadAddr + 5000, Section, Offset));
+  EXPECT_EQ(1U, Section);
+  EXPECT_EQ(904U, Offset);
+
+  ASSERT_TRUE(S->addressForVA(-1, Section, Offset));
+  EXPECT_EQ(0U, Section);
+  EXPECT_EQ(0U, Offset);
+
+  ASSERT_TRUE(S->addressForVA(4, Section, Offset));
+  EXPECT_EQ(0U, Section);
+  EXPECT_EQ(4U, Offset);
+
+  ASSERT_TRUE(S->addressForVA(LoadAddr + 100000, Section, Offset));
+  EXPECT_EQ(3U, Section);
+  EXPECT_EQ(83616U, Offset);
+}
+
+TEST(NativeSessionTest, TestAddressForRVA) {
+  std::unique_ptr<IPDBSession> S;
+  Error E = pdb::loadDataForEXE(PDB_ReaderType::Native, getExePath(), S);
+  ASSERT_THAT_ERROR(std::move(E), Succeeded());
+
+  uint32_t Section;
+  uint32_t Offset;
+  ASSERT_TRUE(S->addressForVA(5000, Section, Offset));
+  EXPECT_EQ(1U, Section);
+  EXPECT_EQ(904U, Offset);
+
+  ASSERT_TRUE(S->addressForVA(-1, Section, Offset));
+  EXPECT_EQ(0U, Section);
+  EXPECT_EQ(0U, Offset);
+
+  ASSERT_TRUE(S->addressForVA(4, Section, Offset));
+  EXPECT_EQ(0U, Section);
+  EXPECT_EQ(4U, Offset);
+
+  ASSERT_TRUE(S->addressForVA(100000, Section, Offset));
+  EXPECT_EQ(3U, Section);
+  EXPECT_EQ(83616U, Offset);
+}