[expression evaluator] Allow runtimes to execute custom LLVM ModulePasses over the generated IR at various stages after expression compilation.

[ldrumm]

Summary

During expression evaluation, the ClangExpressionParser preforms a number of hard-coded fixups on the expression's IR before the module is assembled and dispatched to be run in a ThreadPlan.

This patch allows the runtimes to register LLVM passes to be run over the generated IR, that they may perform language or architechture-specfic fixups or analyses over the generated expression.

This makes expression evaluation for plugins more flexible and allows language-specific fixes to reside in their own module, rather than littering the expression evaluator itself with language-specific fixes.

[jingham]

This seems fine as a generic instrumentation point. Obviously, the onus in on the passes, since they could totally ruin the expression evaluation if they don't do their job right... But I'm not sure there's any good way to make this not a sharp tool. But you should wait on Sean to weigh in as well...

One question, what is the force of the "legacy" in "llvm::legacy::PassManager? Is there some more "modern" way to do this, is this going to go away at some point?

[ldrumm]

In D17027#347487, @jingham wrote:

This seems fine as a generic instrumentation point. Obviously, the onus in on the passes, since they could totally ruin the expression evaluation if they don't do their job right... But I'm not sure there's any good way to make this not a sharp tool. But you should wait on Sean to weigh in as well...

I appreciate this, and I think that if a runtime provides passes, the expectation should be that the runtime knows what it's doing because this really is a powertool - as you say. I think it's a generally useful mechanism for better modularity/encapsulation of language-specific behaviours that are required during expression evaluation. For example, there are presently a large number of language-specific fixups in the clang-based expression evaluator, that could be factored out into individual EarlyPasses/LatePasses provided by their runtime, leaving IRForTarget and friends a lot cleaner and more maintainable.

One question, what is the force of the "legacy" in "llvm::legacy::PassManager? Is there some more "modern" way to do this, is this going to go away at some point?

The legacy pass manager is fully functional and tested. Using the new version does not work unfortunately in this instance. We can certainly move to the new version as it matures and becomes ready for general use, but I tried this implementation with both llvm::PassManager and the the llvm::legacy::PassManager, and unfortunately the new PassManager did not work here.

[ldrumm]

Hi Jim

<snip>

Did you file bugs with llvm on these issues?  It would be good to file a "move to the new pass manager when ready" bug on lldb, but it should be blocked on the relevant llvm bugs.

Not yet. I believe it's best to wait this until this revision is accepted before reporting bugs on its implementation. Though I'm happy to do so, I'll prefer to wait for yours and Sean's final comments.

Let me know your thoughts.

Best

Luke

[ldrumm]

Hi @spyffe @jingham

If you have the time, I'd appreciate your comments.

Thanks

Luke

[ldrumm]

Hi Sean

Did you get a chance to look over this? It'd be great to have your input.

Best

Luke

[ldrumm]

rebased on current HEAD

[ldrumm]

gentle ping, if @spyffe or @jingham is available to take another pass at this.

[ldrumm]

I'd like to give this another *bump*, and commit this soon assuming positive review.

@spyffe

[ldrumm]

committed in http://reviews.llvm.org/rL272800

include/lldb/Expression/LLVMUserExpression.h

 #define liblldb_LLVMUserExpression_h
 
 // C Includes
 // C++ Includes
 #include <string>
 #include <map>
 #include <vector>
 
+// Other libraries and framework includes
+#include "llvm/IR/LegacyPassManager.h"
+
 // Project includes
 #include "lldb/Expression/UserExpression.h"
 
 namespace lldb_private
 {
 
 //----------------------------------------------------------------------
 /// @class LLVMUserExpression LLVMUserExpression.h "lldb/Expression/LLVMUserExpression.h"
 /// @brief Encapsulates a one-time expression for use in lldb.
 ///
 /// LLDB uses expressions for various purposes, notably to call functions
 /// and as a backend for the expr command.  LLVMUserExpression is a virtual base
 /// class that encapsulates the objects needed to parse and JIT an expression.
 /// The actual parsing part will be provided by the specific implementations
 /// of LLVMUserExpression - which will be vended through the appropriate TypeSystem.
 //----------------------------------------------------------------------
 class LLVMUserExpression : public UserExpression
 {
 public:
+    // The IRPasses struct is filled in by a runtime after an expression is compiled and can be used to to run
+    // fixups/analysis passes as required. EarlyPasses are run on the generated module before lldb runs its own IR
+    // fixups and inserts instrumentation code/pointer checks. LatePasses are run after the module has been processed by
+    // llvm, before the module is assembled and run in the ThreadPlan.
+    struct IRPasses
+    {
+        IRPasses() : EarlyPasses(nullptr), LatePasses(nullptr){};
+        std::shared_ptr<llvm::legacy::PassManager> EarlyPasses;
+        std::shared_ptr<llvm::legacy::PassManager> LatePasses;
+    };
+
     LLVMUserExpression(ExecutionContextScope &exe_scope, const char *expr, const char *expr_prefix,
                        lldb::LanguageType language, ResultType desired_type,
                        const EvaluateExpressionOptions &options);
     ~LLVMUserExpression() override;
 
     bool
     FinalizeJITExecution(DiagnosticManager &diagnostic_manager, ExecutionContext &exe_ctx,
                          lldb::ExpressionVariableSP &result,

include/lldb/Target/LanguageRuntime.h

 #include "lldb/lldb-public.h"
 #include "lldb/Breakpoint/BreakpointResolver.h"
 #include "lldb/Breakpoint/BreakpointResolverName.h"
 #include "lldb/Core/PluginInterface.h"
 #include "lldb/lldb-private.h"
 #include "lldb/Core/ValueObject.h"
 #include "lldb/Core/Value.h"
 #include "lldb/Target/ExecutionContextScope.h"
+#include "lldb/Expression/LLVMUserExpression.h"
+
 #include "clang/Basic/TargetOptions.h"
 
 namespace lldb_private {
 
 class LanguageRuntime :
     public PluginInterface
 {
 public:
public:
     // the compiler.
     // If the options prototype is modified, runtimes must return true, false otherwise.
     virtual bool
     GetOverrideExprOptions(clang::TargetOptions &prototype)
     {
         return false;
     }
 
+    // Called by ClangExpressionParser::PrepareForExecution to query for any custom LLVM IR passes
+    // that need to be run before an expression is assembled and run.
+    virtual bool
+    GetIRPasses(LLVMUserExpression::IRPasses &custom_passes)
+    {
+        return false;
+    }
 protected:
     //------------------------------------------------------------------
     // Classes that inherit from LanguageRuntime can see and modify these
     //------------------------------------------------------------------
     
     LanguageRuntime(Process *process);
     Process *m_process;
 private:
 
     DISALLOW_COPY_AND_ASSIGN (LanguageRuntime);
 };
 
 } // namespace lldb_private
 
 #endif // liblldb_LanguageRuntime_h_

source/Plugins/ExpressionParser/Clang/ClangExpressionParser.cpp

func_end = LLDB_INVALID_ADDRESS;
     {
         sc = frame_sp->GetSymbolContext(lldb::eSymbolContextEverything);
     }
     else if (lldb::TargetSP target_sp = exe_ctx.GetTargetSP())
     {
         sc.target_sp = target_sp;
     }
 
+    LLVMUserExpression::IRPasses custom_passes;
+    {
+        auto lang = m_expr.Language();
+        if (log)
+            log->Printf("%s - Currrent expression language is %s\n", __FUNCTION__,
+                        Language::GetNameForLanguageType(lang));
+
+        if (lang != lldb::eLanguageTypeUnknown)
+        {
+            auto runtime = exe_ctx.GetProcessSP()->GetLanguageRuntime(lang);
+            if (runtime)
+                runtime->GetIRPasses(custom_passes);
+        }
+    }
+
+    if (custom_passes.EarlyPasses)
+    {
+        if (log)
+            log->Printf("%s - Running Early IR Passes from LanguageRuntime on expression module '%s'", __FUNCTION__,
+                        m_expr.FunctionName());
+
+        custom_passes.EarlyPasses->run(*llvm_module_ap);
+    }
+
     execution_unit_sp.reset(new IRExecutionUnit (m_llvm_context, // handed off here
                                                  llvm_module_ap, // handed off here
                                                  function_name,
                                                  exe_ctx.GetTargetSP(),
                                                  sc,
                                                  m_compiler->getTargetOpts().Features));
 
     ClangExpressionHelper *type_system_helper = dyn_cast<ClangExpressionHelper>(m_expr.GetTypeSystemHelper());
if (decl_map)
                     process->SetDynamicCheckers(dynamic_checkers);
 
                     if (log)
                         log->Printf("== [ClangUserExpression::Evaluate] Finished installing dynamic checkers ==");
                 }
 
                 IRDynamicChecks ir_dynamic_checks(*process->GetDynamicCheckers(), function_name.AsCString());
 
-                if (!ir_dynamic_checks.runOnModule(*execution_unit_sp->GetModule()))
+                llvm::Module *module = execution_unit_sp->GetModule();
+                if (!module || !ir_dynamic_checks.runOnModule(*module))
                 {
                     err.SetErrorToGenericError();
                     err.SetErrorString("Couldn't add dynamic checks to the expression");
                     return err;
                 }
+
+                if (custom_passes.LatePasses)
+                {
+                    if (log)
+                        log->Printf("%s - Running Late IR Passes from LanguageRuntime on expression module '%s'",
+                                    __FUNCTION__, m_expr.FunctionName());
+
+                    custom_passes.LatePasses->run(*module);
+                }
             }
         }
 
         if (execution_policy == eExecutionPolicyAlways || execution_policy == eExecutionPolicyTopLevel ||
             !can_interpret)
         {
             execution_unit_sp->GetRunnableInfo(err, func_addr, func_end);
         }