LLVM's JIT is now the foundation of dynamic-compilation features for many languages. Clang also has low-level support for dynamic compilation (ASTImporter and ExternalASTSource, notably). How the compiler is set up for dynamic parsing is generally left up to individual clients, for example LLDB's C/C++/Objective-C expression parser and the ROOT project.
Although this arrangement offers external clients the flexibility to implement dynamic features as they see fit, the lack of an in-tree client means that subtle bugs can be introduced that cause regressions in the external clients but aren't caught by tests (or users) until much later. LLDB for example regularly encounters complicated ODR violation scenarios where it is not immediately clear who is at fault.
I propose a simple expression parser be added to Clang. I aim to have it encompass two main features:
- It should be able to look up external declarations from a variety of sources (e.g., from previous dynamic compilations, from modules, or from DWARF) and have clear conflict resolution rules with easily understood errors. This functionality will be supported by in-tree tests.
- It should work hand in hand with the LLVM JIT to resolve the locations of external declarations so that e.g. variables can be redeclared and (for high-performance applications like DTrace) external variables can be accessed directly from the registers where they reside.
I have attached a tester that parses a sequence of source files and then uses them as source data for an expression. External references are resolved using an ExternalASTSource that responds to name queries using an ASTImporter. This is the setup that LLDB uses, and the motivating reason for MinimalImport in ASTImporter.
Over time my intention is to make this more complete and support the many scenarios that LLDB can support. I also want to identify places where LLDB does the wrong thing and we can make this functionality more correct, hopefully eliminating frustrating and opaque errors. I also want to spot where Clang might get things wrong, and be able to point Clang developers to an easy-to-reproduce, in-tree test that doesn't require external projects or patches. Finally, I want to identify how we can make this functionality as generic as possible for the current clients besides LLDB, and for potential future clients.
@beanz recently went on a crusade to add dependencies to intrinsics_gen to a bunch of stuff. Chances are, this tool probably depends on intrinsics_gen, so I'd look into that and add the dep if needed.