[test-suite] Unit tests to improve code coverage

[afd]

Summary

Adds a batch of C tests that have been found to cover several hundred
lines of Clang/LLVM that are not covered by the unit and regression
tests of the main LLVM project, nor by the test suite when run with the
-O3 configuration.

The tests were originally generated using Csmith, and were then reduced
using C-Reduce. They have been checked for undefined behaviour-freedom
using Frama-C and CompCert, and manually checked to eliminate
implementation-defined behaviour.

Most of the new coverage achieved by these tests is in:

clang/lib/Analysis/CFG.cpp

[afd]

Hi @anemet and @lenary - I am keen to contribute these tests that improve on the code coverage achieved by the existing test suite. It's my first LLVM-related patch so I wasn't sure who the right folks to ask for reviews were, but I saw that you'd each done some recent reviews in the test suite so I wondered whether you might be suitable to review this, or to suggest other reviewers.

Thanks!

[lenary]

There's one warning to fix (which should be trivial), and then I'm happy for this to land. Given the warning, the fix will hopefully be obvious?

I looked at the compilation and execution of all of these under clang and gcc trunk (for x86-64), and they all agree with your provided outputs.

SingleSource/UnitTests/2020-01-06-coverage-003.c

8

Clang gives a warning here:

warning: result of comparison of constant 4073709551607 with expression of type 'int8_t' (aka 'signed char') is always true [-Wtautological-constant-out-of-range-compare]

[afd]

Fixed a compiler warning as spotted by @lenary

[afd]

@lenary Thanks for the review! I have fixed the compiler warning that you spotted. I hope I did the right thing when I uploaded a revised patch (this is my first time working with the differential/arcanist workflow).

Are you able to land this for me?

[lenary]

Yeah, I can land this for you, I'll try and do that today.

[afd]

Ace, thank you.

[yroux]

Hi,

this commit broke ARM bots, because their data model is ILP32, thus it is not safe to use a long format specifier to print a 64 bits type.

los are available here: http://lab.llvm.org:8011/#/builders/59/builds/1227

Ithink you should either use inttypes.h PRId64 macro to print a int64_t or use %ji with a cast to intmax_t

[lenary]

I'll commit a fix that uses the inttypes macros.

[lenary]

@afd Some nits about changing the printfs. I have so far presumed that where a number is unsigned, you want me to print it as unsigned (using "%u" or equivalent via inttypes.h), and where it is signed, you want me to print it using "%i" (or equivalent). It raised some questions, below.

I'll have to get work to approve my updated patch before I can post it publicly, but that should be quick if I'm lucky.

SingleSource/UnitTests/2020-01-06-coverage-005.c

86

Here you format an unsigned number with %i (a signed format specifier), did you mean to? I'd sort of expect an explicit cast if you did.

There are other places where this is also the case, I'll comment on them.

SingleSource/UnitTests/2020-01-06-coverage-007.c

28

Also here

[afd]

@lenary Thanks for looking at this. Indeed, those uses of %i where it should be %u were oversights.

Something I just realised is that I named the tests according to the date on which I created them, but actually they were created in October, not January - I must have made a typo when I named them 2020-01-06 instead of 2020-10-06. I don't suppose this matters much but if you're updating this patch and if it's easy to change the 01s to 10s that would be grand.

SingleSource/UnitTests/2020-01-06-coverage-003.c

8

I suppose this warning could be silenced by casting a to uint64_t. That has the potential to affect the new coverage that this test ended up hitting, and I don't see that it matters if a test leads to warnings like this being issued - we still need to know that the compiler behaves sensibly on code that triggers such warnings.

SingleSource/UnitTests/2020-01-06-coverage-005.c

86

Ah yes, that was an oversight thanks - so changing to %u woudl be great.

SingleSource/UnitTests/2020-01-06-coverage-007.c

28

Also an oversight - should be %u - thanks.

[lenary]

Cool, I have a patch to update to the right specifiers, and to update the reference output where it has changed (005 and 007 have unsigned numbers being printed as signed, and showing up in the current output as negative).

I'm not going to do the rename. I figured something was up when checking this stuff out, because I didn't think the patch was that old, but reasonably it just needs to be a unique name grouping, the date doesn't have to be 100% accurate, I feel.

[yroux]

@lenary it'd be great if you can commit your fix today, the bots are red for long time

[lenary]

Follow-up patch is https://reviews.llvm.org/D97686

SingleSource/UnitTests/2020-01-06-coverage-001.c

+#include <stdint.h>
+#include <stdio.h>
+
+int32_t *a;
+uint8_t b = 7;
+static int32_t **c = &a;
+uint8_t *p;
+int32_t x;
+
+void d() {
+  **c &= 8;
+  uint8_t *e = &b;
+  p = e;
+  x = (*e &= 1) && *a && (*e = 0);
+}
+
+int main() {
+  int32_t val_3;
+  int8_t val_2;
+  uint8_t temp_2;
+  int32_t val_1;
+  int32_t temp_1;
+
+  val_1 = 1149798655;
+  val_2 = -125;
+  val_3 = -67108864;
+  temp_1 = val_1;
+  temp_2 = val_2;
+
+  x = val_3;
+  p = &temp_2;
+  a = &temp_1;
+  d();
+
+  printf("b = %i\n", b);
+  printf("x = %i\n", x);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-001.reference_output

+b = 0
+x = 0
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-002.c

+#include <stdint.h>
+#include <stdio.h>
+
+int32_t a, c, e;
+int8_t b;
+int16_t d;
+int32_t x;
+int64_t y;
+
+int64_t f() {
+  int32_t g = 0;
+  x = g;
+  for (; g < 5; g++)
+    ;
+  for (; g < 3; (void)g)
+  h:
+    for (;; (void)a)
+      ;
+  for (;; (void)b) {
+    int32_t i;
+    if (a)
+      goto h;
+    if (d) {
+      if (e)
+        return 0;
+      break;
+    }
+    if (c)
+      continue;
+  }
+  {
+    y = 0;
+    return y;
+  };
+}
+
+int64_t z;
+
+int32_t main() {
+  int64_t val_8;
+  int64_t val_7;
+  int32_t val_6;
+  int16_t val_5;
+  int8_t val_4;
+  int32_t val_3;
+  int32_t val_2;
+  int32_t val_1;
+
+  val_1 = 0;
+  val_2 = -2;
+  val_3 = 0;
+  val_4 = 0;
+  val_5 = 10;
+  val_6 = 4;
+  val_7 = 0;
+  val_8 = 0;
+
+  z = val_8;
+  y = val_7;
+  x = val_6;
+  d = val_5;
+  b = val_4;
+  e = val_3;
+  c = val_2;
+  a = val_1;
+  z = f();
+
+  printf("a = %i\n", a);
+  printf("b = %i\n", b);
+  printf("c = %i\n", c);
+  printf("d = %i\n", d);
+  printf("x = %i\n", x);
+  printf("e = %i\n", e);
+  printf("z = %li\n", z);
+  printf("y = %li\n", y);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-002.reference_output

+a = 0
+b = 0
+c = -2
+d = 10
+x = 0
+e = 0
+z = 0
+y = 0
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-003.c

+#include <stdint.h>
+#include <stdio.h>
+
+int8_t a;
+int32_t x;
+
+void b() { x = (uint64_t)4073709551607 > a || a != 0; }
+
+int main() {
+  int32_t val_2;
+  int8_t val_1;
+
+  val_1 = -7;
+  val_2 = 1695154176;
+
+  x = val_2;
+  a = val_1;
+  b();
+
+  printf("a = %i\n", a);
+  printf("x = %i\n", x);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-003.reference_output

+a = -7
+x = 1
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-004.c

+#include <stdint.h>
+#include <stdio.h>
+
+int32_t a, b;
+
+union u {
+  uint16_t c;
+} d;
+
+static int32_t *e = &b;
+int64_t *f;
+static int64_t **g = &f;
+int64_t x;
+
+int64_t h(int32_t i) {
+  {
+    x = i && (2036854775807 / a) && i;
+    return x;
+  };
+}
+
+void j() {
+  for (; d.c; ++d.c)
+    *e = h(**g);
+}
+
+int32_t h_call_argument_1;
+int64_t y;
+
+int main() {
+  int64_t val_7;
+  int32_t val_6;
+  int64_t val_5;
+  int64_t val_4;
+  int64_t temp_1;
+  uint16_t val_3;
+  int32_t val_2;
+  int32_t val_1;
+
+  val_1 = 251;
+  val_2 = -2113601536;
+  val_3 = 12027;
+  val_4 = 0;
+  val_5 = 70651297207679744;
+  val_6 = -7;
+  val_7 = 2089739496336261183;
+  temp_1 = val_4;
+
+  y = val_7;
+  h_call_argument_1 = val_6;
+  x = val_5;
+  f = &temp_1;
+  d.c = val_3;
+  b = val_2;
+  a = val_1;
+  j();
+  y = h(h_call_argument_1);
+
+  printf("a = %i\n", a);
+  printf("b = %i\n", b);
+  printf("d.c = %i\n", d.c);
+  printf("y = %li\n", y);
+  printf("h_call_argument_1 = %i\n", h_call_argument_1);
+  printf("x = %li\n", x);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-004.reference_output

+a = 251
+b = 0
+d.c = 0
+y = 1
+h_call_argument_1 = -7
+x = 1
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-005.c

+#include <stdint.h>
+#include <stdio.h>
+
+uint32_t a, c, d;
+uint64_t b;
+int16_t e;
+uint64_t *p;
+int16_t *q;
+void g();
+uint64_t *r;
+
+void h() {
+  uint64_t *f = &b;
+  r = f;
+  *f = 6;
+  g();
+}
+
+uint64_t *s;
+int16_t *t;
+
+void g() {
+i : {
+  uint64_t *j = &b;
+  s = j;
+  p = j;
+  int16_t *k = &e;
+  t = k;
+  q = k;
+  --*k;
+  *j &= c;
+}
+  for (; d; a = d)
+    if (e)
+      goto i;
+}
+
+int main() {
+  int16_t val_10;
+  int16_t temp_5;
+  int64_t val_9;
+  uint64_t temp_4;
+  int64_t val_8;
+  uint64_t temp_3;
+  int16_t val_7;
+  int16_t temp_2;
+  int64_t val_6;
+  uint64_t temp_1;
+  int16_t val_5;
+  int64_t val_4;
+  uint32_t val_3;
+  uint32_t val_2;
+  uint32_t val_1;
+
+  val_1 = 16777101;
+  val_2 = 0xFFFFFFFA;
+  val_3 = 0;
+  val_4 = 256;
+  val_5 = -7;
+  val_6 = -9223372032559808512;
+  val_7 = 0;
+  val_8 = -72318388763361280;
+  val_9 = 0;
+  val_10 = 0;
+  temp_1 = val_6;
+  temp_2 = val_7;
+  temp_3 = val_8;
+  temp_4 = val_9;
+  temp_5 = val_10;
+
+  t = &temp_5;
+  s = &temp_4;
+  r = &temp_3;
+  q = &temp_2;
+  p = &temp_1;
+  e = val_5;
+  b = val_4;
+  d = val_3;
+  c = val_2;
+  a = val_1;
+  h();
+  g();
+
+  printf("a = %i\n", a);
+  printf("b = %li\n", b);
+  printf("c = %i\n", c);
+  printf("d = %i\n", d);
+  printf("e = %i\n", e);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-005.reference_output

+a = 16777101
+b = 2
+c = -6
+d = 0
+e = -9
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-006.c

+#include <stdint.h>
+#include <stdio.h>
+
+#pragma pack(1)
+
+struct s {
+  int : 28;
+  unsigned : 17;
+  unsigned a : 25;
+} b;
+
+volatile int32_t c;
+int32_t x;
+
+void d() {
+  if (8 < b.a)
+    x = c;
+}
+
+int main() {
+  int32_t val_3;
+  int32_t val_2;
+  uint32_t val_1;
+
+  val_1 = 2097152;
+  val_2 = 0;
+  val_3 = 0;
+
+  x = val_3;
+  c = val_2;
+  b.a = val_1;
+  d();
+
+  printf("b.a = %i\n", b.a);
+  printf("c = %i\n", c);
+  printf("x = %i\n", x);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-006.reference_output

+b.a = 2097152
+c = 0
+x = 0
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-007.c

+#include <stdint.h>
+#include <stdio.h>
+
+uint32_t a;
+const uint32_t b = -1L, c;
+int16_t d;
+
+void e() {
+  uint32_t f;
+  uint32_t g;
+  f = b / a;
+  g = f;
+  d = g < c;
+}
+
+int main() {
+  int16_t val_2;
+  uint32_t val_1;
+
+  val_1 = 3;
+  val_2 = -32767;
+
+  d = val_2;
+  a = val_1;
+  e();
+
+  printf("a = %i\n", a);
+  printf("b = %i\n", b);
+  printf("c = %i\n", c);
+  printf("d = %i\n", d);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-007.reference_output

+a = 3
+b = -1
+c = 0
+d = 0
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-008.c

+#include <stdint.h>
+#include <stdio.h>
+
+struct a {
+  int64_t b;
+  int64_t c;
+} d;
+
+int64_t e;
+
+void f(struct a *g) { d = *g; }
+
+struct a s;
+
+int main() {
+  int64_t val_5;
+  int64_t val_4;
+  int64_t val_3;
+  int64_t val_2;
+  int64_t val_1;
+
+  val_1 = -7988785259004943837;
+  val_2 = -7;
+  val_3 = -7;
+  val_4 = -7988785259004943837;
+  val_5 = -6;
+
+  s.b = val_4;
+  s.c = val_5;
+  e = val_3;
+  d.b = val_1;
+  d.c = val_2;
+  struct a h = {0, 9};
+  s = h;
+  e = h.c;
+  f(&h);
+
+  printf("d.b = %li\n", d.b);
+  printf("d.c = %li\n", d.c);
+  printf("s.b = %li\n", s.b);
+  printf("s.c = %li\n", s.c);
+  printf("e = %li\n", e);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-008.reference_output

+d.b = 0
+d.c = 9
+s.b = 0
+s.c = 9
+e = 9
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-009.c

+#include <stdint.h>
+#include <stdio.h>
+
+uint32_t a, b, h;
+int16_t c, g;
+uint8_t d = 8;
+uint8_t *const e = &d;
+int32_t *f;
+
+void i() {
+  for (; c; c--) {
+    g = 0;
+    for (; g <= 8; g++) {
+      h = ((b & a) ^ a ^ (b & a) ^ b) < b;
+      *e ^= h;
+      *f = 0;
+    }
+  }
+}
+
+int main() {
+  int32_t val_6;
+  int32_t temp_1;
+  int16_t val_5;
+  int16_t val_4;
+  uint32_t val_3;
+  uint32_t val_2;
+  uint32_t val_1;
+
+  val_1 = 0;
+  val_2 = 1;
+  val_3 = 0xFFFFFFFF;
+  val_4 = -32767;
+  val_5 = -32767;
+  val_6 = -2130706432;
+  temp_1 = val_6;
+
+  f = &temp_1;
+  g = val_5;
+  c = val_4;
+  h = val_3;
+  b = val_2;
+  a = val_1;
+  i();
+
+  printf("a = %i\n", a);
+  printf("b = %i\n", b);
+  printf("c = %i\n", c);
+  printf("d = %i\n", d);
+  printf("g = %i\n", g);
+  printf("h = %i\n", h);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-009.reference_output

+a = 0
+b = 1
+c = 0
+d = 8
+g = 9
+h = 0
+exit 0

SingleSource/UnitTests/2020-01-06-coverage-010.c

+#include <stdint.h>
+#include <stdio.h>
+
+int32_t *a;
+int32_t b, d, f;
+int8_t c;
+volatile int32_t e;
+int32_t x;
+int8_t *p;
+int32_t y;
+int32_t z;
+int32_t w;
+
+static int32_t j() {
+  int32_t h = 5;
+  x = h;
+  for (; b;) {
+    f = 0;
+    for (; f <= 9; f++) {
+      int32_t *i = &h;
+      int8_t *g = &c;
+      p = g;
+      y = (*i)++ && (*g = 0);
+      z = e;
+    }
+    *a = 0;
+  }
+  {
+    w = d;
+    return w;
+  };
+}
+
+int32_t v;
+
+void k() { v = j(); }
+
+int main() {
+  int32_t val_12;
+  int32_t val_11;
+  int32_t val_10;
+  int32_t val_9;
+  int8_t val_8;
+  int8_t temp_2;
+  int32_t val_7;
+  int32_t val_6;
+  int8_t val_5;
+  int32_t val_4;
+  int32_t val_3;
+  int32_t val_2;
+  int32_t val_1;
+  int32_t temp_1;
+
+  val_1 = 0;
+  val_2 = 0;
+  val_3 = 0;
+  val_4 = 0;
+  val_5 = 0;
+  val_6 = 0;
+  val_7 = 0;
+  val_8 = 5;
+  val_9 = 0;
+  val_10 = 0;
+  val_11 = 0;
+  val_12 = 0;
+  temp_1 = val_1;
+  temp_2 = val_8;
+
+  v = val_12;
+  w = val_11;
+  z = val_10;
+  y = val_9;
+  p = &temp_2;
+  x = val_7;
+  e = val_6;
+  c = val_5;
+  f = val_4;
+  d = val_3;
+  b = val_2;
+  a = &temp_1;
+  k();
+
+  printf("b = %i\n", b);
+  printf("c = %i\n", c);
+  printf("d = %i\n", d);
+  printf("x = %i\n", x);
+  printf("e = %i\n", e);
+  printf("y = %i\n", y);
+  printf("z = %i\n", z);
+  printf("v = %i\n", v);
+  printf("f = %i\n", f);
+  printf("w = %i\n", w);
+
+  return 0;
+}

SingleSource/UnitTests/2020-01-06-coverage-010.reference_output

+b = 0
+c = 0
+d = 0
+x = 5
+e = 0
+y = 0
+z = 0
+v = 0
+f = 0
+w = 0
+exit 0