This is an archive of the discontinued LLVM Phabricator instance.

Differential D145178

[clang][NFC] reformats the SARIF diagnostic test so it's human readable
AcceptedPublic

Authored by cjdb on Mar 2 2023, 12:38 PM.

Details

Reviewers
aaron.ballman
shafik
erichkeane
Summary

The current FileCheck test output is very difficult to read, which makes
adding new tests to SARIF output rather painful. I'd ideally like to
replace this test with a far more robust form of JSON testing, but am
unable to allocate resources to paying down the tech debt at present.

Diff Detail

Event Timeline

cjdb created this revision.Mar 2 2023, 12:38 PM
Herald added a project: Restricted Project. · View Herald TranscriptMar 2 2023, 12:38 PM
cjdb requested review of this revision.Mar 2 2023, 12:38 PM
Herald added a project: Restricted Project. · View Herald TranscriptMar 2 2023, 12:38 PM
Herald added a subscriber: cfe-commits. · View Herald Transcript
cjdb added inline comments.Mar 2 2023, 12:41 PM
clang/test/Frontend/sarif-diagnostics.cpp
31

This was split out for future tests where the COMMON stuff isn't written to file. Perhaps I should check it in as STDERR?

43

Aside from the reformatting, the only change to this file is the use of regex for the URIs.

cjdb added inline comments.Mar 2 2023, 12:47 PM
clang/test/Frontend/sarif-diagnostics.cpp
31

Further thoughts: everything is common, it's a case of where it's output that's not the same. I think with this in mind I should absolutely change COMMON to STDERR, and possibly CHECK to SARIF.

cjdb updated this revision to Diff 501955.Mar 2 2023, 1:00 PM

renames check identifiers

Harbormaster completed remote builds in B217037: Diff 501955.Mar 2 2023, 2:06 PM
cjdb updated this revision to Diff 501982.Mar 2 2023, 2:09 PM

removes redundant line

Harbormaster completed remote builds in B217053: Diff 501982.Mar 2 2023, 3:12 PM
cjdb added a child revision: D145201: [clang] fixes header processing for `-fdiagnostics-format=sarif`.Mar 2 2023, 3:16 PM
cjdb added a subscriber: denik.Mar 2 2023, 3:25 PM
cjdb updated this revision to Diff 502019.Mar 2 2023, 4:10 PM

removes something from a future commit

This should be the final alteration pre-review.

Harbormaster completed remote builds in B217085: Diff 502019.Mar 2 2023, 4:57 PM
aaron.ballman accepted this revision.Mar 3 2023, 6:53 AM

LGTM, this is incremental progress. Hopefully we won't be adding too many more RUN lines to this file though (sticking too many tests into one file is also tech debt).

This revision is now accepted and ready to land.Mar 3 2023, 6:53 AM
cjdb added a comment.Mar 3 2023, 10:32 AM
In D145178#4167103, @aaron.ballman wrote:

LGTM, this is incremental progress. Hopefully we won't be adding too many more RUN lines to this file though (sticking too many tests into one file is also tech debt).

Thanks! 100% of the RUN lines I intend to add are bug regressions, but there will be more RUN lines in the future.

shafik accepted this revision.Mar 6 2023, 10:52 AM
cjdb added a child revision: D146654: [clang] replaces numeric SARIF ids with heirarchical names.Mar 22 2023, 2:08 PM
cjdb removed a child revision: D145201: [clang] fixes header processing for `-fdiagnostics-format=sarif`.Mar 22 2023, 2:26 PM

Revision Contents

PathSize
clang/
test/
Frontend/
428 lines

Diff 502019

clang/test/Frontend/sarif-diagnostics.cpp

// RUN: %clang -fsyntax-only -Wall -Wextra -fdiagnostics-format=sarif %s > %t 2>&1 || true
// RUN: FileCheck -dump-input=always %s --input-file=%t
// FIXME: this test is incredibly fragile because the `main()` function // FIXME: this test is incredibly fragile because the `main()` function
// must be on line 12 in order for the CHECK lines to get the correct line // must be on line 12 in order for the CHECK lines to get the correct line
// number values. // number values.
// //
// So these comment lines are being used to ensure the code below happens // So these comment lines are being used to ensure the code below happens
// to work properly for the test coverage, which as you can imagine, is not // to work properly for the test coverage, which as you can imagine, is not
// the best way to structure the test. We really need to introduce a better // the best way to structure the test. We really need to introduce a better
// tool than FileCheck for diff'ing JSON output like SARIF. // tool than FileCheck for diff'ing JSON output like SARIF.
//
// In order to add more tests over time, we've put the RUN lines **underneath**
// the code.
void main() { void main() {
int i = hello; int i = hello;
float test = 1a.0; float test = 1a.0;
if (true) if (true)
bool Yes = true; bool Yes = true;
return; return;
bool j = hi; bool j = hi;
} }
} }
struct t1 { }; struct t1 { };
void f1(t1 x, t1 y) { void f1(t1 x, t1 y) {
x + y; x + y;
} }
// CHECK: warning: diagnostic formatting in SARIF mode is currently unstable [-Wsarif-format-unstable] // RUN: %clang -fsyntax-only -Wall -Wextra -fdiagnostics-format=sarif %s > %t.txt 2>&1 || true
// CHECK: {"$schema":"https://docs.oasis-open.org/sarif/sarif/v2.1.0/cos02/schemas/sarif-schema-2.1.0.json","runs":[{"artifacts":[{"length": // RUN: FileCheck -dump-input=always %s --input-file=%t.txt --check-prefixes=STDERR,SARIF
cjdbAuthorUnsubmitted
Done
ReplyInline Actions

This was split out for future tests where the COMMON stuff isn't written to file. Perhaps I should check it in as STDERR?

cjdb: This was split out for future tests where the `COMMON` stuff isn't written to file. Perhaps I…
cjdbAuthorUnsubmitted
Done
ReplyInline Actions

Further thoughts: everything is common, it's a case of where it's output that's not the same. I think with this in mind I should absolutely change COMMON to STDERR, and possibly CHECK to SARIF.

cjdb: Further thoughts: everything is common, it's a case of where it's output that's not the same. I…
// Omit exact length of this file
// CHECK: ,"location":{"index":0,"uri":"file:// // STDERR: warning: diagnostic formatting in SARIF mode is currently unstable [-Wsarif-format-unstable]
// Omit filepath to llvm project directory // SARIF: {
// CHECK: test/Frontend/sarif-diagnostics.cpp"},"mimeType":"text/plain","roles":["resultFile"]}],"columnKind":"unicodeCodePoints","results": // SARIF: "$schema":"https://docs.oasis-open.org/sarif/sarif/v2.1.0/cos02/schemas/sarif-schema-2.1.0.json",
// CHECK: [{"level":"error","locations":[{"physicalLocation":{"artifactLocation":{"index":0,"uri":"file:// // SARIF: "runs":[
// CHECK: {"endColumn":1,"startColumn":1,"startLine":12}}}],"message":{"text":"'main' must return 'int'"},"ruleId":"{{[0-9]+}}","ruleIndex":0}, // SARIF: {
// CHECK: {"level":"error","locations":[{"physicalLocation":{"artifactLocation":{"index":0,"uri":"file:// // SARIF: "artifacts":[
// CHECK: {"endColumn":11,"startColumn":11,"startLine":13}}}],"message":{"text":"use of undeclared identifier // SARIF: {
// CHECK: 'hello'"},"ruleId":"{{[0-9]+}}","ruleIndex":1},{"level":"error","locations":[{"physicalLocation":{"artifactLocation": // SARIF: "length":{{[0-9]+}},
// CHECK: {"index":0,"uri":"file://{{.+}}"},"region":{"endColumn":17,"startColumn":17,"startLine":15}}}],"message":{"text":"invalid digit 'a' in decimal // SARIF: "location":{
// CHECK: constant"},"ruleId":"{{[0-9]+}}","ruleIndex":2},{"level":"warning","locations":[{"physicalLocation":{"artifactLocation": // SARIF: "index":0,
// CHECK: {"index":0,"uri":"file://{{.+}}"},"region":{"endColumn":5,"startColumn":5,"startLine":19}}}],"message":{"text":"misleading indentation; statement is not part // SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
cjdbAuthorUnsubmitted
Done
ReplyInline Actions

Aside from the reformatting, the only change to this file is the use of regex for the URIs.

cjdb: Aside from the reformatting, the only change to this file is the use of regex for the URIs.
// CHECK: of the previous 'if'"},"ruleId":"{{[0-9]+}}","ruleIndex":3},{"level":"note","locations":[{"physicalLocation":{"artifactLocation": // SARIF: },
// CHECK: {"index":0,"uri":"file://{{.+}}"},"region":{"endColumn":3,"startColumn":3,"startLine":17}}}],"message":{"text":"previous statement is // SARIF: "mimeType":"text/plain",
// CHECK: here"},"ruleId":"{{[0-9]+}}","ruleIndex":4},{"level":"warning","locations":[{"physicalLocation":{"artifactLocation": // SARIF: "roles":[
// CHECK: {"index":0,"uri":"file://{{.+}}"},"region":{"endColumn":10,"startColumn":10,"startLine":18}}}],"message":{"text":"unused variable // SARIF: "resultFile"
// CHECK: 'Yes'"},"ruleId":"{{[0-9]+}}","ruleIndex":5},{"level":"error","locations":[{"physicalLocation":{"artifactLocation":{"index":0,"uri":"file:// // SARIF: ]
// CHECK: {"endColumn":12,"startColumn":12,"startLine":21}}}],"message":{"text":"use of undeclared identifier // SARIF: }
// CHECK: 'hi'"},"ruleId":"{{[0-9]+}}","ruleIndex":6},{"level":"error","locations":[{"physicalLocation":{"artifactLocation":{"index":0,"uri":"file:// // SARIF: ],
// CHECK: {"endColumn":1,"startColumn":1,"startLine":23}}}],"message":{"text":"extraneous closing brace // SARIF: "columnKind":"unicodeCodePoints",
// CHECK: ('}')"},"ruleId":"{{[0-9]+}}","ruleIndex":7},{"level":"error","locations":[{"physicalLocation":{"artifactLocation":{"index":0,"uri":"file:// // SARIF: "results":[
// CHECK: {"endColumn":6,"endLine":27,"startColumn":5,"startLine":27}}},{"physicalLocation":{"artifactLocation":{"index":0,"uri":"file:// // SARIF: {
// CHECK: {"endColumn":10,"endLine":27,"startColumn":9,"startLine":27}}},{"physicalLocation":{"artifactLocation":{"index":0,"uri":"file:// // SARIF: "level":"error",
// CHECK: {"endColumn":7,"startColumn":7,"startLine":27}}}],"message":{"text":"invalid operands to binary expression ('t1' and // SARIF: "locations":[
// CHECK: 't1')"},"ruleId":"{{[0-9]+}}","ruleIndex":8}],"tool":{"driver":{"fullName":"","informationUri":"https://clang.llvm.org/docs/ // SARIF: {
// CHECK: UsersManual.html","language":"en-US","name":"clang","rules":[{"defaultConfiguration": // SARIF: "physicalLocation":{
// CHECK: {"enabled":true,"level":"error","rank":50},"fullDescription":{"text":""},"id":"{{[0-9]+}}","name":""},{"defaultConfiguration": // SARIF: "artifactLocation":{
// CHECK: {"enabled":true,"level":"error","rank":50},"fullDescription":{"text":""},"id":"{{[0-9]+}}","name":""},{"defaultConfiguration": // SARIF: "index":0,
// CHECK: {"enabled":true,"level":"error","rank":50},"fullDescription":{"text":""},"id":"{{[0-9]+}}","name":""},{"defaultConfiguration": // SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// CHECK: {"enabled":true,"level":"warning","rank":-1},"fullDescription":{"text":""},"id":"{{[0-9]+}}","name":""},{"defaultConfiguration": // SARIF: },
// CHECK: {"enabled":true,"level":"note","rank":-1},"fullDescription":{"text":""},"id":"{{[0-9]+}}","name":""},{"defaultConfiguration": // SARIF: "region":{
// CHECK: {"enabled":true,"level":"warning","rank":-1},"fullDescription":{"text":""},"id":"{{[0-9]+}}","name":""},{"defaultConfiguration": // SARIF: "endColumn":1,
// CHECK: {"enabled":true,"level":"error","rank":50},"fullDescription":{"text":""},"id":"{{[0-9]+}}","name":""},{"defaultConfiguration": // SARIF: "startColumn":1,
// CHECK: {"enabled":true,"level":"error","rank":50},"fullDescription":{"text":""},"id":"{{[0-9]+}}","name":""},{"defaultConfiguration": // SARIF: "startLine":12
// CHECK: {"enabled":true,"level":"error","rank":50},"fullDescription": // SARIF: }
// CHECK: {"text":""},"id":"{{[0-9]+}}","name":""}],"version":"{{[0-9]+\.[0-9]+\.[0-9]+}}"}}}],"version":"2.1.0"} // SARIF: }
// CHECK: 2 warnings and 6 errors generated. // SARIF: }
No newline at end of file // SARIF: ],
// SARIF: "message":{
// SARIF: "text":"'main' must return 'int'"
// SARIF: },
// SARIF: "ruleId":"3485",
// SARIF: "ruleIndex":0
// SARIF: },
// SARIF: {
// SARIF: "level":"error",
// SARIF: "locations":[
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":11,
// SARIF: "startColumn":11,
// SARIF: "startLine":13
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "message":{
// SARIF: "text":"use of undeclared identifier 'hello'"
// SARIF: },
// SARIF: "ruleId":"4632",
// SARIF: "ruleIndex":1
// SARIF: },
// SARIF: {
// SARIF: "level":"error",
// SARIF: "locations":[
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":17,
// SARIF: "startColumn":17,
// SARIF: "startLine":15
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "message":{
// SARIF: "text":"invalid digit 'a' in decimal constant"
// SARIF: },
// SARIF: "ruleId":"898",
// SARIF: "ruleIndex":2
// SARIF: },
// SARIF: {
// SARIF: "level":"warning",
// SARIF: "locations":[
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":5,
// SARIF: "startColumn":5,
// SARIF: "startLine":19
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "message":{
// SARIF: "text":"misleading indentation; statement is not part of the previous 'if'"
// SARIF: },
// SARIF: "ruleId":"1826",
// SARIF: "ruleIndex":3
// SARIF: },
// SARIF: {
// SARIF: "level":"note",
// SARIF: "locations":[
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":3,
// SARIF: "startColumn":3,
// SARIF: "startLine":17
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "message":{
// SARIF: "text":"previous statement is here"
// SARIF: },
// SARIF: "ruleId":"1746",
// SARIF: "ruleIndex":4
// SARIF: },
// SARIF: {
// SARIF: "level":"warning",
// SARIF: "locations":[
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":10,
// SARIF: "startColumn":10,
// SARIF: "startLine":18
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "message":{
// SARIF: "text":"unused variable 'Yes'"
// SARIF: },
// SARIF: "ruleId":"6593",
// SARIF: "ruleIndex":5
// SARIF: },
// SARIF: {
// SARIF: "level":"error",
// SARIF: "locations":[
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":12,
// SARIF: "startColumn":12,
// SARIF: "startLine":21
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "message":{
// SARIF: "text":"use of undeclared identifier 'hi'"
// SARIF: },
// SARIF: "ruleId":"4632",
// SARIF: "ruleIndex":6
// SARIF: },
// SARIF: {
// SARIF: "level":"error",
// SARIF: "locations":[
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":1,
// SARIF: "startColumn":1,
// SARIF: "startLine":23
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "message":{
// SARIF: "text":"extraneous closing brace ('}')"
// SARIF: },
// SARIF: "ruleId":"1400",
// SARIF: "ruleIndex":7
// SARIF: },
// SARIF: {
// SARIF: "level":"error",
// SARIF: "locations":[
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":6,
// SARIF: "endLine":27,
// SARIF: "startColumn":5,
// SARIF: "startLine":27
// SARIF: }
// SARIF: }
// SARIF: },
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":10,
// SARIF: "endLine":27,
// SARIF: "startColumn":9,
// SARIF: "startLine":27
// SARIF: }
// SARIF: }
// SARIF: },
// SARIF: {
// SARIF: "physicalLocation":{
// SARIF: "artifactLocation":{
// SARIF: "index":0,
// SARIF: "uri":{{"file://[^"]+/clang/test/Frontend/sarif-diagnostics.cpp"}}
// SARIF: },
// SARIF: "region":{
// SARIF: "endColumn":7,
// SARIF: "startColumn":7,
// SARIF: "startLine":27
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "message":{
// SARIF: "text":"invalid operands to binary expression ('t1' and 't1')"
// SARIF: },
// SARIF: "ruleId":"4567",
// SARIF: "ruleIndex":8
// SARIF: }
// SARIF: ],
// SARIF: "tool":{
// SARIF: "driver":{
// SARIF: "fullName":"",
// SARIF: "informationUri":"https://clang.llvm.org/docs/UsersManual.html",
// SARIF: "language":"en-US",
// SARIF: "name":"clang",
// SARIF: "rules":[
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"error",
// SARIF: "rank":50
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"3485",
// SARIF: "name":""
// SARIF: },
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"error",
// SARIF: "rank":50
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"4632",
// SARIF: "name":""
// SARIF: },
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"error",
// SARIF: "rank":50
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"898",
// SARIF: "name":""
// SARIF: },
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"warning",
// SARIF: "rank":-1
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"1826",
// SARIF: "name":""
// SARIF: },
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"note",
// SARIF: "rank":-1
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"1746",
// SARIF: "name":""
// SARIF: },
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"warning",
// SARIF: "rank":-1
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"6593",
// SARIF: "name":""
// SARIF: },
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"error",
// SARIF: "rank":50
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"4632",
// SARIF: "name":""
// SARIF: },
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"error",
// SARIF: "rank":50
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"1400",
// SARIF: "name":""
// SARIF: },
// SARIF: {
// SARIF: "defaultConfiguration":{
// SARIF: "enabled":true,
// SARIF: "level":"error",
// SARIF: "rank":50
// SARIF: },
// SARIF: "fullDescription":{
// SARIF: "text":""
// SARIF: },
// SARIF: "id":"4567",
// SARIF: "name":""
// SARIF: }
// SARIF: ],
// SARIF: "version":"17.0.0"
// SARIF: }
// SARIF: }
// SARIF: }
// SARIF: ],
// SARIF: "version":"2.1.0"
// SARIF: }
// STDERR: 2 warnings and 6 errors generated.