[Clang][docs]JSONLinkDatabase

[Glebuska]

Summary

Hello,

We provide you with a possible standard for link commands. We would be glad if this format, albeit with minor changes, reaches the approval stage. This format is needed for cmake developers, but most likely they are not the only ones. Check out the discussion with cmake developers for more information here.

Also, while we are not sure that the format will be preserved, we are thinking about using the first field to store meta information. Example here.

[Glebuska]

Updating D102216: JSONLinkDatabase

[Glebuska]

Updating D102216: [Clang][docs]JSONLinkDatabase

[sammccall]

Hi, thanks for writing this up! This is an interesting idea and whole-program information is something our tooling libraries don't cover.

A few high-level thoughts:

• this has wider scope and probably have wider discussion than a code review, an RFC on the cfe-dev mailing list is the usual starting point
• it's unusual for LLVM to define file-formats without tools for working with them. compile_commands.json was developed in tandem with a library to use it for clang-based analysis. Do you have plans to develop such tools in the LLVM tree?
• whether in or out of the LLVM tree, what are the most important use cases for such a format? Why was this design picked over others to satisfy them?
• is command/flags the right abstraction for information about how source code is ultimately linked? It follows the compile_commands.json precedent, but the underlying reasons seem much weaker. (With parsing commands there are a huge diversity of options, it's critical we preserve them ~all exactly, and there's a clear impl strategy for doing so)
• if the database can contain arbitrarily mixed linker/object file manipulation/non-linker commands and the tool is responsible for guessing/interpreting them seems to mean this format will be hard to consume robustly.
• one motivating case is in the readme (resolving one of multiple definitions in the code base). IIUC the suggested implementation is to index the codebase using compile_commands.json, and then use link info to choose between multiple definitions. This seems quite narrow (only helps when there are multiple defintions, but not if they're each linked to the caller in different binaries), doesn't yield a particularly *efficient* implementation, and seems to admit less invasive solutions (e.g. a subset compile_commands.json that only covers sources for one binary).

(Sorry about the delay, I've been off work for a few weeks)

clang/docs/JSONLinkDatabase.rst

+==============================================
+JSON Compilation Database Format Specification
+==============================================
+
+This document describes a format for specifying how to replay single link commands independently of the build system.
+
+Background
+==============================================
+
+Currently, there is `compilation database format <https://clang.llvm.org/docs/JSONCompilationDatabase.html>`_ , which specified by Clang. Compilation commands are actively used for syntax analysis of code, 
+but there are cases, when the compilation commands cannot provide all the necessary information, for example, look into simple C/C++ code:  
+
+ ::
+
+    a.cpp:
+
+    int f(int a) {
+        return g(a);
+     }
+
+    x.cpp:
+
+    int g(int a) {
+      return a + 1;
+    }
+
+    y.cpp:
+
+    int g(int a) {
+      return a - 1;
+    }
+
+
+A tool may need to know which function is actually called, but this information is only available at linking step as soon as there may be several functions with same name in different units.  
+Using of link database is providing more information for advance software analysis.
+More examples:
+
+1) Unit testing tools which use code analysis need to understand which function is called in order to get correct test result.
+2) IDEs need it to understand which file to go when you click "Go to definition" on a function call.
+
+Format
+================
+
+A link database is a JSON file, which consist of an array of “command objects”, where each command object specifies one way a target and first element is a version of format instead of link's command. 
+The contracts for each field in the command object are:
+
+- **command**: The link command executed. The field contains all the flags and files needed to create a target or library. As result of executing all commands, it's a full-build project.
+- **directory** The working directory of the link's command. All paths specified in the command or file fields must be either absolute or relative to this directory.
+- **files** An array which described all the object files, dynamic and static libraries needed for linking.
+- **arguments** The link command executed as list of strings. Either arguments or command is required.
+- **output** The name of the output created by this linking step. This field is optional. It can be used to distinguish different processing modes of the same input.
+
+Using link commands from file can give 3 different types of output:  
+- executable file 
+- static library
+- dynamic library
+
+Example:
+
+ :: 
+
+        
+            [   
+              {
+                "command" : "/usr/bin/ar qc libcmstd.a CMakeFiles/cmstd.dir/cm/bits/fs_path.cxx.o CMakeFiles/cmstd.dir/cm/bits/string_view.cxx.o",
+                  "directory" : "/home/myuser/exampleProject/build/Utilities/std",
+                "files" : 
+                [
+                    "/home/myuser/exampleProject/build/Utilities/std/CMakeFiles/cmstd.dir/cm/bits/fs_path.cxx.o", 
+                    "/home/myuser/exampleProject/build/Utilities/std/CMakeFiles/cmstd.dir/cm/bits/string_view.cxx.o"
+                ]
+              },
+              {
+                "command" : "/usr/bin/cc  -Wcast-align -Werror-implicit-function-declaration -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings -Wformat-security -Wmissing-format-attribute -fno-common -Wundef CMakeFiles/pseudonl_purify.dir/ret0.c.o -o purify",
+                  "directory" : "/home/myuser/exampleProject/build/Tests/CMakeLib/PseudoMemcheck/NoLog",
+                "files" : 
+                [
+                      "/home/myuser/exampleProject/build/Tests/CMakeLib/PseudoMemcheck/NoLog/CMakeFiles/pseudonl_purify.dir/ret0.c.o"
+                ]
+              },
+              ...
+            ]                    
+    
+
+
+
+
+
+This example demonstrates command **ar** to create **dynamic library libcmstd.a** and **cc** command to create **purify target**  
+
+In addition to the commands responsible for linking, link database can contain commands specific to the build system:
+Example for CMake build system:
+
+ 1) CMake command for deleting library
+
+     ::
+    
+                [   
+                    ...
+                  {
+                    "command" : "/usr/local/bin/cmake -E rm -f CMakeFiles/library.dir/lib.a",
+                      "directory" : "/home/myuser/exampleProject/build/Utilities/std",
+                    "files" : 
+                    [
+                        "/home/myuser/exampleProject/build/Utilities/std/CMakeFiles/cmstd.dir/cm/bits/file1.cxx.o", 
+                        "/home/myuser/exampleProject/build/Utilities/std/CMakeFiles/cmstd.dir/cm/bits/file2.cxx.o"
+                    ]
+                  },
+                  ...
+                ]
+    
+ 2) `CMake lwyu static checker <https://blog.kitware.com/static-checks-with-cmake-cdash-iwyu-clang-tidy-lwyu-cpplint-and-cppcheck/>`_
+
+     ::
+
+            [
+                ...
+                {
+                    "command": "/usr/local/bin/cmake -E __run_co_compile --lwyu=some_target",
+                    "directory": "/home/myuser/exampleProject/build",
+                    "files":
+                    [
+                        "/home/myuser/exampleProject/build/Utilities/std/CMakeFiles/cmstd.dir/cm/bits/file3.c.o"
+                    ]
+                },
+                ...
+            ]
+        
+ 3)  `ranlib command <https://man7.org/linux/man-pages/man1/ranlib.1.html>`_
+
+    ::
+        
+        [   
+            ...
+          {
+            "command" : "/usr/bin/ranlib library.a",
+              "directory" : "/home/myuser/exampleProject/build",
+            "files" : []
+          },
+          ...
+        ]
+