Add Documentation for Execution Results Handling in Clang-REPL

[Krishna-13-cyber]

Summary

This patch adds documentation for execution results handling in Clang-REPL with the below features:

• Automatic Printf feature
• Value Synthesis feature
• Pretty Printing feature

I am issuing this patch on behalf of @QuillPusher

[QuillPusher]

added some comments for minor changes. Two sections to be removed since they are not yet merged to upstream code:

• Complex Data Types
• Users can create their own types

clang/docs/ExecutionResultsHandling.rst

50–52

@Krishna-13-cyber This paragraph is exceeding 80 columns. Please reformat to limit to 80 columns

55

Please remove the hyphen:

right-away -> right away

(must have carried through from previous tool's preview)

257–259

Formatting for this note seems different, it is enclosed in single quotes. Did you mean Note: or " " ?

268

Please remove the question mark, this is an expression, not a question (common mistake)

How it works? -> How it works

Note that another heading has a question mark (Where is the captured result stored?). That question mark is OK, since that is actually mimicking a user's question. So leave that one as it is.

283

@Krishna-13-cyber please add a note/comment above the code block:

Note: Following is a sample code snippet. Actual code may vary over time.

329–356

@Krishna-13-cyber please remove the "Complex Data Types" section, since this is not merged into the upstream LLVM code yet.
@junaire can confirm.

357–371

@Krishna-13-cyber please remove the "Users can create their own types" section, since this is not merged into the upstream LLVM code yet.
@junaire can confirm.

[v.g.vassilev]

clang/docs/index.rst

96

We should probably move that under ClangRepl.

[Krishna-13-cyber]

• Addressed the comments

[QuillPusher]

Thanks @Krishna-13-cyber for the updates, look OK to me

Thanks @junaire for sharing feature knowledge to help create this document

[junaire]

I want to clarify: We offer two features: 1. capture the execution results and bring it back to the compiled program. 2. dump the captured value (value printing/automatic printf) and all the parsing, ast transform and balabala, these are all implementation details. The doc looks fine but may need a bit of restructuring. I'd suggest you split them into:

1. what do we offer? what are the new features? why are they important? The two I mentioned above
2. how these are implemented? what does the underhood look like? annotation token, code synthesis, etc...
3. further information, RFC and etc...

[QuillPusher]

Thanks @junaire for the clarification, following is the updated document structure:

1. Capture Execution Results
   • How the Feature works
   • more details
   • Implementation Details

2. Dump Captured Execution Results¶
   • How- the Feature works
   • more details
   • Implementation Details

@Krishna-13-cyber Please use the attached file named ExecutionResultsHandling.rst as the latest doc, so Jun can review the updated document.

ExecutionResultsHandling.rst12 KBDownload

clang/docs/ClangRepl.rst

217–232 ↗

(On Diff #547231)

Execution Results Handling in Clang-Repl

:doc:ExecutionResultsHandling helps extend the Clang-REPL functionality by
creating an interface between the execution results of a program and the compiled
program. Following are its main components:

1 - Capture Execution Results: This feature helps capture the execution results
of a program and bring them back to the compiled program.

2 - Dump Captured Execution Results: This feature helps create a temporary dump
for Value Printing/Automatic Printf, that is, to display the value and type of
the captured data.

[Krishna-13-cyber]

• Address the comments

[Krishna-13-cyber]

• Update section of Execution Results Handling in Clang-REPL

[QuillPusher]

Thanks @Krishna-13-cyber for the prompt changes.

@v.g.vassilev If no further changes are required by @junaire , then this should be ready to merge

[v.g.vassilev]

clang/docs/ExecutionResultsHandling.rst

81

cppyy seems undefined here.

[QuillPusher]

added minor changes based on Vassil's review during docs meeting

clang/docs/ClangRepl.rst

220–230 ↗

(On Diff #547574)

As per Vassil's comment below, please move all the Execution Results handling text in the main Clang-Repl doc instead of a separate ExecutionResultsHandling document

So this paragraph can be removed and all the content from the ExecutionResultsHandling.rst can be included here in this ClangRepl.rst document

clang/docs/ExecutionResultsHandling.rst

120–128

Add note before code snippet

Note: Following is a sample code snippet. Actual code may vary over time.

238–257

please remove this code block, it is not needed

clang/docs/index.rst

96

We should probably move that under ClangRepl.

@Krishna-13-cyber I have commented above, where the excerpt paragraph in CalngRepl.rst can be removed and all content from ExecutionResultsHandling.rst can be included (and remove file ExecutionResultsHandling.rst after that)

[Krishna-13-cyber]

• Address the comments

[QuillPusher]

accepting current revision, one minor change is still required, adding that separately

[QuillPusher]

Added minor changes requested by Vassil

clang/docs/ClangRepl.rst

296 ↗

(On Diff #549034)

@Krishna-13-cyber

Please change CPPYY to cppyy

I just saw they use small caps on their official website

298 ↗

(On Diff #549034)

@Krishna-13-cyber

Please add the following note under this paragraph, as requested by Vassil in previous comment:

Note: cppyy_ is an automatic, run-time, Python-to-C++ bindings
generator, for calling C++ from Python and Python from C++. It uses LLVM along
with a C++ interpreter (e.g., Cling) to enable features like run-time
instantiation of C++ templates, cross-inheritance, callbacks, auto-casting,
transparent use of smart pointers, etc.

.. _cppyy: https://github.com/wlav/cppyy/

[QuillPusher]

clang/docs/ClangRepl.rst

393–396 ↗

(On Diff #549034)

Please replace the image reference with the following Graphviz code (do not delete the image .png for now, we need to commit and see if graphviz works out of the box).

.. graphviz::

:name: automaticprintf
:caption: Automatic PrintF
:alt: Shows how Automatic PrintF can be used
:align: center

 digraph "AutomaticPrintF" {
     size="6,4";
     rankdir="LR";
     graph [fontname="Verdana", fontsize="12"];
     node [fontname="Verdana", fontsize="12"];
     edge [fontname="Sans", fontsize="9"];

     manual [label=" Manual PrintF ", shape="box"];
     int1 [label=" int ( &) 42 ", shape="box"]
     auto [label=" Automatic PrintF ", shape="box"];
     int2 [label=" int ( &) 42 ", shape="box"]

     auto -> int2 [label="int x = 42; \n x"];
     manual -> int1 [label="int x = 42; \n printf("(int &) %d \\n", x);"];
 }

[v.g.vassilev]

clang/docs/ClangRepl.rst

221 ↗

(On Diff #549034)

We should probably spell consistently Clang-Repl.

248 ↗

(On Diff #549034)

Can we replace that with its graphviz version?

264 ↗

(On Diff #549034)

265 ↗

(On Diff #549034)

266–267 ↗

(On Diff #549034)

269–274 ↗

(On Diff #549034)

276–278 ↗

(On Diff #549034)

284 ↗

(On Diff #549034)

296–298 ↗

(On Diff #549034)

352 ↗

(On Diff #549034)

404 ↗

(On Diff #549034)

419 ↗

(On Diff #549034)

461 ↗

(On Diff #549034)

[QuillPusher]

@Krishna-13-cyber added Graphviz code snippets to replace all .png images

clang/docs/ClangRepl.rst

248–250 ↗

(On Diff #549034)

.. graphviz::

:name: valuesynthesis
:caption: Value Synthesis
:alt: Shows how an object of type 'Value' is synthesized
:align: center

 digraph "valuesynthesis" {
     rankdir="LR";
     graph [fontname="Verdana", fontsize="12"];
     node [fontname="Verdana", fontsize="12"];
     edge [fontname="Sans", fontsize="9"];

     start [label=" Create an Object \n 'Last Value' \n of type 'Value' ", shape="note", fontcolor=white, fillcolor="#3333ff", style=filled];
     assign [label=" Assign the result \n to the 'LastValue' \n (based on respective \n Memory Allocation \n scenario) ", shape="box"]
     print [label=" Pretty Print \n the Value Object ", shape="Msquare", fillcolor="yellow", style=filled];
     start -> assign;
     assign -> print;

       subgraph SynthesizeExpression {
         synth [label=" SynthesizeExpr() ", shape="note", fontcolor=white, fillcolor="#3333ff", style=filled];
         mem [label=" New Memory \n Allocation? ", shape="diamond"];
         withaloc [label=" SetValueWithAlloc() ", shape="box"];
         noaloc [label=" SetValueNoAlloc() ", shape="box"];
         right [label=" 1. RValue Structure \n (a temporary value)", shape="box"];
         left2 [label=" 2. LValue Structure \n (a variable with \n an address)", shape="box"];
         left3 [label=" 3. Built-In Type \n (int, float, etc.)", shape="box"];
         output [label=" move to 'Assign' step ", shape="box"];

         synth -> mem;
         mem -> withaloc [label="Yes"];
         mem -> noaloc [label="No"];
         withaloc -> right;
         noaloc -> left2;
         noaloc -> left3;
         right -> output;
         left2 -> output;
         left3 -> output;
 }
     output -> assign;

}

424–427 ↗

(On Diff #549034)

@Krishna-13-cyber please replace the image reference with the following code:

.. graphviz::

:name: parsing
:caption: Parsing Mechanism
:alt: Shows the Parsing Mechanism for Pretty Printing
:align: center

 digraph "prettyprint" {
     rankdir="LR";
     graph [fontname="Verdana", fontsize="12"];
     node [fontname="Verdana", fontsize="12"];
     edge [fontname="Verdana", fontsize="9"];

     parse [label=" ParseAndExecute() \n in Clang ", shape="box"];
     capture [label=" Capture 'Value' parameter \n for processing? ", shape="diamond"];
     use [label="  Use for processing  ", shape="box"];
     dump [label="  Validate and push  \n to dump()", shape="box"];
     callp [label="  call print() function ", shape="box"];
     type [label="  Print the Type \n ReplPrintTypeImpl()", shape="box"];
     data [label="  Print the Data \n ReplPrintDataImpl() ", shape="box"];
     output [label="  Output Pretty Print \n to the user  ", shape="box", fontcolor=white, fillcolor="#3333ff", style=filled];

     parse -> capture [label="Optional 'Value' Parameter"];
     capture -> use [label="Yes"];
     use -> End;
     capture -> dump [label="No"];
     dump -> callp;
     callp -> type;
     callp -> data;
     type -> output;
     data -> output;

}

[Krishna-13-cyber]

• Address the comments

[QuillPusher]

Thanks @Krishna-13-cyber for the prompt efforts
@v.g.vassilev it is ready to commit as far as I can see

[v.g.vassilev]

The pre-merge check seems to fail at clang-format for some reason. This patch does not change anything that should be formatted by clang-format.

We will try to add the images as editable graphviz content. We need to see if the infrastructure allows for this as suggested here: https://discourse.llvm.org/t/any-tool-for-creating-editable-diagrams-in-llvm-clang-repl-documentation-similar-to-mermaid-ingithub/72729/2

clang/docs/ExecutionResultsHandling.rst

+=========================================
+Execution Results Handling in Clang-REPL
+=========================================
+
+Execution Results Handling features discussed below help extend the Clang-REPL 
+functionality by creating an interface between the execution results of a 
+program and the compiled program.
+
+- The Automatic Printf feature makes it easy to display variable values 
+  during program execution.
+
+- The Value Synthesis feature helps store the execution results 
+  (to be able to bring them back to the compiled program).
+
+- The Pretty Printing feature helps create a temporary dump to display the value
+  and type (pretty print) of the desired data. 
+
+1. Automatic Printf
+===================
+
+The `Automatic Printf` feature makes it easy to display variable values during 
+program execution. Using the `printf` function repeatedly is not required. 
+This is achieved using an extension in the `libclangInterpreter` library.
+
+To automatically print the value of an expression, simply write the expression 
+in the global scope **without a semicolon**.
+
+.. image:: autoprint.png
+   :align: center
+   :alt: autoprint design
+
+Examples
+--------
+.. code-block:: console
+
+    clang-repl> int x = 42;
+    clang-repl> x       // equivalent to calling printf("(int &) %d\n", x);
+    (int &) 42
+
+    clang-repl> std::vector<int> v = {1,2,3};
+    clang-repl> v  // This syntax is fine after D127284 
+    (std::vector<int> &) {1,2,3}
+
+    clang-repl> "Hello, interactive C++!"
+    (const char [24]) "Hello, interactive C++!"
+
+Significance of this feature
+----------------------------
+
+Inspired by a similar implementation in `Cling <https://github.com/root-project/cling>`_,
+this feature added to upstream Clang repo has essentially extended the syntax of C++,
+so that it can be more helpful for people that are writing code for data science applications.
+ 
+This is useful, for example, when you want to experiment with a set of values 
+against a set of functions, and you'd like to know the results right-away. 
+This is similar to how Python works (hence its popularity in data science 
+research), but the superior performance of C++, along with this flexibility 
+makes it a more attractive option.
+
+Annotation Token (annot_repl_input_end)
+---------------------------------------
+
+This feature uses a new token (annot_repl_input_end) to consider printing the 
+value of an expression if it doesn't end with a semicolon. When parsing an 
+Expression Statement, if the last semicolon is missing, then the code will 
+pretend that there one and set a marker there for later utilization, and 
+continue parsing.
+
+A semicolon is normally required in C++, but this feature expands the C++ 
+syntax to handle cases where a missing semicolon is expected (i.e., when 
+handling an expression statement). It also makes sure that an error is not 
+generated for the missing semicolon in this specific case. 
+
+This is accomplished by identifying the end position of the user input 
+(expression statement). This helps store and return the expression statement 
+effectively, so that it can be printed (displayed to the user automatically).
+
+**Note:** This logic is only available for C++ for now, since part of the 
+implementation itself requires C++ features. Future versions may support more 
+languages.
+
+.. code-block:: console
+
+  Token *CurTok = nullptr;
+  // If the semicolon is missing at the end of REPL input, consider if
+  // we want to do value printing. Note this is only enabled in C++ mode
+  // since part of the implementation requires C++ language features.
+  // Note we shouldn't eat the token since the callback needs it.
+  if (Tok.is(tok::annot_repl_input_end) && Actions.getLangOpts().CPlusPlus)
+    CurTok = &Tok;
+  else
+    // Otherwise, eat the semicolon.
+    ExpectAndConsumeSemi(diag::err_expected_semi_after_expr);
+
+  StmtResult R = handleExprStmt(Expr, StmtCtx);
+  if (CurTok && !R.isInvalid())
+    CurTok->setAnnotationValue(R.get());
+
+  return R;
+    }
+
+AST Transformation
+-------------------
+
+When Sema encounters the `annot_repl_input_end` token, it knows to transform 
+the AST before the real CodeGen process. It will consume the token and set a 
+'semi missing' bit in the respective decl.
+
+.. code-block:: console
+
+    if (Tok.is(tok::annot_repl_input_end) &&
+        Tok.getAnnotationValue() != nullptr) {
+        ConsumeAnnotationToken();
+        cast<TopLevelStmtDecl>(DeclsInGroup.back())->setSemiMissing();
+    }
+
+In the AST Consumer, traverse all the Top Level Decls, to look for expressions 
+to synthesize. If the current Decl is the Top Level Statement 
+Decl(`TopLevelStmtDecl`) and has a semicolon missing, then ask the interpreter 
+to synthesize another expression (an internal function call) to replace this 
+original expression.
+
+2. 'Value' Synthesis
+====================
+
+In many cases, it is useful to bring back the program execution result to the 
+compiled program. This result can be stored in an object of type **Value**.
+
+Incremental AST Consumer
+-------------------------
+
+The `IncrementalASTConsumer` class wraps the original code generator 
+`ASTConsumer` and it performs a hook, to traverse all the top-level decls, to 
+look for expressions to synthesize, based on the `isSemiMissing()` condition.
+
+If this condition is found to be true, then `Interp.SynthesizeExpr()` will be 
+invoked. 
+
+.. code-block:: console
+
+    for (Decl *D : DGR)
+      if (auto *TSD = llvm::dyn_cast<TopLevelStmtDecl>(D);
+          TSD && TSD->isSemiMissing())
+        TSD->setStmt(Interp.SynthesizeExpr(cast<Expr>(TSD->getStmt())));
+
+    return Consumer->HandleTopLevelDecl(DGR);
+
+The synthesizer will then choose the relevant expression, based on its type.
+
+How Execution Results are captured:
+-----------------------------------
+
+The synthesizer chooses which expression to synthesize, and then it replaces 
+the original expression with the synthesized expression. Depending on the 
+expression type, it may choose to save an object (`LastValue`) of type 'value'
+while allocating memory to it (`SetValueWithAlloc()`), or not (
+`SetValueNoAlloc()`).
+
+.. image:: valuesynth.png
+   :align: center
+   :alt: valuesynth design
+
+Where is the captured result stored?
+------------------------------------
+
+`LastValue` holds the last result of the value printing. It is a class member 
+because it can be accessed even after subsequent inputs. 
+
+**Note:** If no value printing happens, then it is in an invalid state. 
+
+Interpreter as a REPL vs. as a Library
+---------------------------------------
+
+1 - If we're using the interpreter in interactive (REPL) mode, it will dump 
+the value (i.e., value printing).
+
+.. code-block:: console
+
+  if (LastValue.isValid()) {
+    if (!V) {
+      LastValue.dump();
+      LastValue.clear();
+    } else
+      *V = std::move(LastValue);
+  }
+
+
+2 - If we're using the interpreter as a library, then it will pass the value 
+to the user.
+
+Improving Efficiency and User Experience
+-----------------------------------------
+
+The Value object is essentially used to create a mapping between an expression 
+'type' and the 'memory' to be allocated. Built-in types (bool, char, int, 
+float, double, etc.) are simpler, since their memory allocation size is known. 
+In case of objects, a pointer can be saved, since the size of the object is 
+not known.
+
+For further improvement, the underlying Clang Type is also identified. For 
+example, `X(char, Char_S)`, where `Char_S` is the Clang type. Clang types are 
+very efficient, which is important since these will be used in hotspots (high 
+utilization areas of the program). The `Value.h` header file has a very low 
+token count and was developed with strict constraints in mind, since it can 
+affect the performance of the interpreter.
+
+This also enables the user to receive the computed 'type' back in their code 
+and then transform the type into something else (e.g., transform a double into 
+a float). Normally, the compiler can handle these conversions transparently, 
+but in interpreter mode, the compiler cannot see all the 'from' and 'to' types,
+so it cannot implicitly do the conversions. So this logic enables providing 
+these conversions on request. 
+
+On-request conversions can help improve the user experience, by allowing 
+conversion to a desired 'to' type, when the 'from' type is unknown or unclear
+
+Significance of this Feature
+-----------------------------
+
+The 'Value' object enables wrapping a memory region that comes from the 
+JIT, and bringing it back to the compiled code (and vice versa). 
+This is a very useful functionality when:
+
+- connecting an interpreter to the compiled code, or
+- connecting an interpreter in another language.
+
+For example, the `CPPYY` code makes use of this feature to enable running 
+C++ within Python. It enables transporting values/information between C++ 
+and Python.
+
+In a nutshell, this feature enables a new way of developing code, paving the 
+way for language interoperability and easier interactive programming.
+
+Communication between Compiled Code and Interpreted Code
+--------------------------------------------------------
+
+In Clang-REPL there is **interpreted code**, and this feature adds a 'value' 
+runtime that can talk to the **compiled code**.
+
+Following is an example where the compiled code interacts with the interpreter 
+code. The execution results of an expression are stored in the object 'V' of 
+type Value. This value is then printed, effectively helping the interpreter 
+use a value from the compiled code.
+
+.. code-block:: console
+
+    int Global = 42;
+    void setGlobal(int val) { Global = val; }
+    int getGlobal() { return Global; }
+    Interp.ParseAndExecute(“void setGlobal(int val);”);
+    Interp.ParseAndExecute(“int getGlobal();”);
+    Value V;
+    Interp.ParseAndExecute(“getGlobal()”, &V);
+    std::cout << V.getAs<int>() << “\n”; // Prints 42
+
+
+`Above is an example of interoperability between the compiled code and the 
+interpreted code. Interoperability between languages (e.g., C++ and Python) 
+works similarly.`
+
+3. Pretty Printing
+===================
+
+This feature helps create a temporary dump to display the value and type 
+(pretty print) of the desired data. This is a good way to interact with the 
+interpreter during interactive programming.
+
+How it works?
+---------------
+.. image:: prettyprint.png
+   :align: center
+   :alt: prettyprint design
+
+Parsing mechanism:
+------------------
+
+The Interpreter in Clang-REPL (`interpreter.cpp`) includes the function 
+`ParseAndExecute()` that can accept a 'Value' parameter to capture the result. 
+But if the value parameter is made optional and it is omitted (i.e., that the 
+user does not want to utilize it elsewhere), then the last value can be 
+validated and pushed into the `dump()` function. 
+
+.. code-block:: console
+
+    llvm::Error Interpreter::ParseAndExecute(llvm::StringRef Code, Value *V) {
+
+    auto PTU = Parse(Code);
+    if (!PTU)
+        return PTU.takeError();
+    if (PTU->TheModule)
+        if (llvm::Error Err = Execute(*PTU))
+        return Err;
+
+    if (LastValue.isValid()) {
+        if (!V) {
+        LastValue.dump();
+        LastValue.clear();
+        } else
+        *V = std::move(LastValue);
+    }
+    return llvm::Error::success();
+    }
+
+The `dump()` function (in `value.cpp`) calls the `print()` function.
+
+.. code-block:: console
+
+    void Value::print(llvm::raw_ostream &Out) const {
+    assert(OpaqueType != nullptr && "Can't print default Value");
+
+    if (getType()->isVoidType() || !isValid())
+        return;
+
+    std::string Str;
+    llvm::raw_string_ostream SS(Str);
+
+    //Print the Type and Data
+    
+    SS << "(";
+    printType(SS);
+    SS << ") ";
+    printData(SS);
+    SS << "\n";
+    Out << Str;
+    }
+
+Printing the Data and Type are handled in their respective functions: 
+`ReplPrintDataImpl()` and `ReplPrintTypeImpl()`
+
+Complex Data Types:
+-------------------
+
+This feature can print out primitive types (int, char, bool, etc.) easily. 
+For more complex types (e.g., `std::vector`), it falls back to a runtime 
+function call using the following helper function.
+
+.. code-block:: console
+
+    static std::string SynthesizeRuntimePrint(const Value &V) {
+    Interpreter &Interp = const_cast<Interpreter &>(V.getInterpreter());
+    Sema &S = Interp.getCompilerInstance()->getSema();
+    ASTContext &Ctx = S.getASTContext();
+
+    static bool Included = false;
+    if (!Included) {
+        Included = true;
+        llvm::cantFail(
+            Interp.Parse("#include <__clang_interpreter_runtime_printvalue.h>"));
+    }
+
+
+The included header file `__clang_interpreter_runtime_printvalue.h` includes 
+functions that can be called to handle complex types (e.g., STL components).
+
+**Note here:** This header is only included on-demand, where needed, since it is
+an expensive runtime operation.
+
+Users can create their own types:
+---------------------------------
+
+All overloads live in a header, which are included at runtime. So **print a 
+std::vector** is equivalent to `PrintValueRuntime(&v);`.
+
+This means users can write their own overload for their types:
+
+.. code-block:: console
+
+    clang-repl> struct S{};
+    clang-repl> std::string PrintValueRuntime(const S* s) {return “My printer!”;}
+    clang-repl> S{}
+    (S) “My Printer!”
+
+Detailed RFC and Discussion:
+----------------------------
+
+For more technical details, community discussion and links to patches related 
+to these features, 
+Please visit: `RFC on LLVM Discourse <https://discourse.llvm.org/t/rfc-handle-execution-results-in-clang-repl/68493>`_.
+
+Some logic presented in the RFC (e.g. ValueGetter()) may be outdated, 
+compared to the final developed solution.
 No newline at end of file

clang/docs/index.rst

.. toctree::
    ClangCheck
    ClangFormat
    ClangFormatStyleOptions
    ClangFormattedStatus
    ClangLinkerWrapper
    ClangOffloadBundler
    ClangOffloadPackager
    ClangRepl
+   ExecutionResultsHandling
 
 Design Documents
 ================
 
 .. toctree::
    :maxdepth: 1
 
    CodeOwners