Make sancov.py py3-compatible

[chapuni]

Summary

This patch is squashed from a few commits.

sancov.py: [Py3] Get rid of "print" statement. Use print() or write() instead.

sancov.py: [Py3] Use '//' instead of '/' as division operator.

Py3 emits float with '/'.

Py3 has '//' as integer division operator.

sancov.py: [Py3] Use sys.stdout.buffer for bytes.

Py2 doesn't have sys.stdout.buffer. An idea from http://stackoverflow.com/questions/23932332

sancov.py: [Py3] Use bytes.decode() explicitly.

Or bogus filename like "b'foo'" would be generated.

[kcc]

ok, makes sense. I'll let Mike review this change.
Thanks!

[mgorny]

compiler-rt/trunk/lib/sanitizer_common/scripts/sancov.py

73–76

That's Python 2.7+ but I guess we don't have to care about 2.6 anymore.

140

I think you ate a newline here.

223

Newline?

[chapuni]

@mgorny Thanks to review.

compiler-rt/trunk/lib/sanitizer_common/scripts/sancov.py

73–76

We require Py>=2.7. http://llvm.org/docs/GettingStarted.html#id8

140

Yes. I missed since check-asan didn't complain.

223

It ought to emit a newline.

[mgorny]

compiler-rt/trunk/lib/sanitizer_common/scripts/sancov.py

223

You are correct. My mistake, sorry.

[chapuni]

compiler-rt/trunk/lib/sanitizer_common/scripts/sancov.py

93

I will add \n here later, too.

[aizatsky]

Does this script actually run with python3 after these changes, or are these lint-driven errors? I'm not an expert in python3 (only used python2), but afaik it doesn't have % operator (you are supposed to use "".format()).

[chapuni]

@aizatsky, I didn't know operator '%' will be obsoleted in future. I will rewrite them with format().

I am familiar to printf-like formatter, though.

[aizatsky]

Could you please clarify if you test it with python3 and if it works?

[chapuni]

@aizatsky, Yes, my patches aim to check-asan passing on py3.
I wrote such a clarification to https://reviews.llvm.org/D27404
but I forgot to write one here during creating diffs.

Tested on x86_64-ubuntu.

trunk,

Failing Tests (30):
    AddressSanitizer-i386-linux :: TestCases/Linux/coverage-missing.cc
    AddressSanitizer-i386-linux :: TestCases/Posix/asan-symbolize-bad-path.cc
    AddressSanitizer-i386-linux :: TestCases/Posix/asan-symbolize-sanity-test.cc
    AddressSanitizer-i386-linux :: TestCases/Posix/coverage-direct-activation.cc
    AddressSanitizer-i386-linux :: TestCases/Posix/coverage-direct-large.cc
    AddressSanitizer-i386-linux :: TestCases/Posix/coverage-direct.cc
    AddressSanitizer-i386-linux :: TestCases/Posix/coverage-fork-direct.cc
    AddressSanitizer-i386-linux :: TestCases/Posix/coverage-sandboxing.cc
    AddressSanitizer-i386-linux :: TestCases/Posix/coverage.cc
    AddressSanitizer-i386-linux :: TestCases/coverage-and-lsan.cc
    AddressSanitizer-i686-linux :: TestCases/Linux/coverage-missing.cc
    AddressSanitizer-i686-linux :: TestCases/Posix/asan-symbolize-bad-path.cc
    AddressSanitizer-i686-linux :: TestCases/Posix/asan-symbolize-sanity-test.cc
    AddressSanitizer-i686-linux :: TestCases/Posix/coverage-direct-activation.cc
    AddressSanitizer-i686-linux :: TestCases/Posix/coverage-direct-large.cc
    AddressSanitizer-i686-linux :: TestCases/Posix/coverage-direct.cc
    AddressSanitizer-i686-linux :: TestCases/Posix/coverage-fork-direct.cc
    AddressSanitizer-i686-linux :: TestCases/Posix/coverage-sandboxing.cc
    AddressSanitizer-i686-linux :: TestCases/Posix/coverage.cc
    AddressSanitizer-x86_64-linux :: TestCases/Linux/coverage-missing.cc
    AddressSanitizer-x86_64-linux :: TestCases/Posix/asan-symbolize-bad-path.cc
    AddressSanitizer-x86_64-linux :: TestCases/Posix/asan-symbolize-sanity-test.cc
    AddressSanitizer-x86_64-linux :: TestCases/Posix/coverage-direct-activation.cc
    AddressSanitizer-x86_64-linux :: TestCases/Posix/coverage-direct-large.cc
    AddressSanitizer-x86_64-linux :: TestCases/Posix/coverage-direct.cc
    AddressSanitizer-x86_64-linux :: TestCases/Posix/coverage-fork-direct.cc
    AddressSanitizer-x86_64-linux :: TestCases/Posix/coverage-sandboxing.cc
    AddressSanitizer-x86_64-linux :: TestCases/Posix/coverage.cc
    AddressSanitizer-x86_64-linux :: TestCases/coverage-and-lsan.cc
    AddressSanitizer-x86_64-linux :: TestCases/coverage-order-pcs.cc

  Expected Passes    : 1411
  Expected Failures  : 3
  Unsupported Tests  : 443
  Unexpected Failures: 30

With patches,

Testing Time: 52.02s
  Expected Passes    : 1441
  Expected Failures  : 3
  Unsupported Tests  : 443

[aizatsky]

If this script working under python3 is your goal, then I suggest you make the script work first, rather then try to fix lint issues separately. I'd rather we do python3-ification in one change if possible.

[chapuni]

@aizatsky, Yes, I can let sancov (and symbolize) work on both py2 and py3. May I commit changes (and pull them into release_40)?

After my investigation, String.format() is just *strongly suggested*.

[aizatsky]

LG then.

[chapuni]

Thanks @aizatsky, check-asan is py3-compatible now.

compiler-rt/trunk/lib/sanitizer_common/scripts/sancov.py

 #!/usr/bin/env python
 # Merge or print the coverage data collected by asan's coverage.
 # Input files are sequences of 4-byte integers.
 # We need to merge these integers into a set and then
 # either print them (as hex) or dump them into another file.
 import array
 import bisect
 import glob
 import os.path
 import struct
 import subprocess
 import sys
 
 prog_name = ""
 
 def Usage():
-  print >> sys.stderr, "Usage: \n" + \
-      " " + prog_name + " merge FILE [FILE...] > OUTPUT\n" \
-      " " + prog_name + " print FILE [FILE...]\n" \
-      " " + prog_name + " unpack FILE [FILE...]\n" \
-      " " + prog_name + " rawunpack FILE [FILE ...]\n" \
-      " " + prog_name + " missing BINARY < LIST_OF_PCS\n"
+  sys.stderr.write(
+    "Usage: \n" + \
+    " " + prog_name + " missing BINARY < LIST_OF_PCS\n" \
+    "\n")
   exit(1)
 
 def CheckBits(bits):
   if bits != 32 and bits != 64:
     raise Exception("Wrong bitness: %d" % bits)
 
 def TypeCodeForBits(bits):
   CheckBits(bits)
def ReadOneFile(path):
   with open(path, mode="rb") as f:
     f.seek(0, 2)
     size = f.tell()
     f.seek(0, 0)
     if size < 8:
       raise Exception('File %s is short (< 8 bytes)' % path)
     bits = ReadMagicAndReturnBitness(f, path)
     size -= 8
     w = size * 8 // bits
     s = struct.unpack_from(TypeCodeForStruct(bits) * (w), f.read(size))
   sys.stderr.write(
     "%s: read %d %d-bit PCs from %s\n" % (prog_name, w, bits, path))
   return s
 
 def Merge(files):
   s = set()
   for f in files:
     s = s.union(set(ReadOneFile(f)))
-  print >> sys.stderr, "%s: %d files merged; %d PCs total" % \
-    (prog_name, len(files), len(s))
+  sys.stderr.write(
+    "%s: %d files merged; %d PCs total\n" % (prog_name, len(files), len(s))
+  )
   return sorted(s)
 
 def PrintFiles(files):
   if len(files) > 1:
     s = Merge(files)
   else:  # If there is just on file, print the PCs in order.
     s = ReadOneFile(files[0])
-    print >> sys.stderr, "%s: 1 file merged; %d PCs total" % \
-      (prog_name, len(s))
+    sys.stderr.write("%s: 1 file merged; %d PCs total\n" % (prog_name, len(s)))
   for i in s:
-    print "0x%x" % i
+    print("0x%x" % i)
 
 def MergeAndPrint(files):
   if sys.stdout.isatty():
     Usage()
   s = Merge(files)
   bits = 32
   if max(s) > 0xFFFFFFFF:
     bits = 64
   stdout_buf = getattr(sys.stdout, 'buffer', sys.stdout)
   array.array('I', MagicForBits(bits)).tofile(stdout_buf)
   a = struct.pack(TypeCodeForStruct(bits) * len(s), *s)
   stdout_buf.write(a)
 
 
 def UnpackOneFile(path):
   with open(path, mode="rb") as f:
-    print >> sys.stderr, "%s: unpacking %s" % (prog_name, path)
+    sys.stderr.write("%s: unpacking %s\n" % (prog_name, path))
     while True:
       header = f.read(12)
       if not header: return
       if len(header) < 12:
         break
       pid, module_length, blob_size = struct.unpack('iII', header)
       module = f.read(module_length).decode('utf-8')
       blob = f.read(blob_size)
       assert(len(module) == module_length)
       assert(len(blob) == blob_size)
       extracted_file = "%s.%d.sancov" % (module, pid)
-      print >> sys.stderr, "%s: extracting %s" % \
-        (prog_name, extracted_file)
+      sys.stderr.write("%s: extracting %s\n" % (prog_name, extracted_file))
       # The packed file may contain multiple blobs for the same pid/module
       # pair. Append to the end of the file instead of overwriting.
       with open(extracted_file, 'ab') as f2:
         f2.write(blob)
     # fail
     raise Exception('Error reading file %s' % path)
 
 
 def Unpack(files):
   for f in files:
     UnpackOneFile(f)
 
 def UnpackOneRawFile(path, map_path):
   mem_map = []
   with open(map_path, mode="rt") as f_map:
-    print >> sys.stderr, "%s: reading map %s" % (prog_name, map_path)
+    sys.stderr.write("%s: reading map %s\n" % (prog_name, map_path))
     bits = int(f_map.readline())
     if bits != 32 and bits != 64:
       raise Exception('Wrong bits size in the map')
     for line in f_map:
       parts = line.rstrip().split()
       mem_map.append((int(parts[0], 16),
                   int(parts[1], 16),
                   int(parts[2], 16),
                   ' '.join(parts[3:])))
   mem_map.sort(key=lambda m : m[0])
   mem_map_keys = [m[0] for m in mem_map]
 
   with open(path, mode="rb") as f:
-    print >> sys.stderr, "%s: unpacking %s" % (prog_name, path)
+    sys.stderr.write("%s: unpacking %s\n" % (prog_name, path))
 
     f.seek(0, 2)
     size = f.tell()
     f.seek(0, 0)
     pcs = struct.unpack_from(TypeCodeForStruct(bits) * (size * 8 // bits), f.read(size))
     mem_map_pcs = [[] for i in range(0, len(mem_map))]
 
     for pc in pcs:
       if pc == 0: continue
       map_idx = bisect.bisect(mem_map_keys, pc) - 1
       (start, end, base, module_path) = mem_map[map_idx]
       assert pc >= start
       if pc >= end:
-        print >> sys.stderr, "warning: %s: pc %x outside of any known mapping" % (prog_name, pc)
+        sys.stderr.write("warning: %s: pc %x outside of any known mapping\n" % (prog_name, pc))
         continue
       mem_map_pcs[map_idx].append(pc - base)
 
     for ((start, end, base, module_path), pc_list) in zip(mem_map, mem_map_pcs):
       if len(pc_list) == 0: continue
       assert path.endswith('.sancov.raw')
       dst_path = module_path + '.' + os.path.basename(path)[:-4]
-      print >> sys.stderr, "%s: writing %d PCs to %s" % (prog_name, len(pc_list), dst_path)
+      sys.stderr.write("%s: writing %d PCs to %s\n" % (prog_name, len(pc_list), dst_path))
       sorted_pc_list = sorted(pc_list)
       pc_buffer = struct.pack(TypeCodeForStruct(bits) * len(pc_list), *sorted_pc_list)
       with open(dst_path, 'ab+') as f2:
         array.array('I', MagicForBits(bits)).tofile(f2)
         f2.seek(0, 2)
         f2.write(pc_buffer)
 
 def RawUnpack(files):
def GetInstrumentedPCs(binary):
   # beginning of the callq instruction. Empirically this is true on x86 and
   # x86_64.
   return set(int(line.strip(), 16) + 4 for line in proc.stdout)
 
 def PrintMissing(binary):
   if not os.path.isfile(binary):
     raise Exception('File not found: %s' % binary)
   instrumented = GetInstrumentedPCs(binary)
-  print >> sys.stderr, "%s: found %d instrumented PCs in %s" % (prog_name,
-                                                                len(instrumented),
-                                                                binary)
+  sys.stderr.write("%s: found %d instrumented PCs in %s\n" % (prog_name,
+                                                              binary))
   covered = set(int(line, 16) for line in sys.stdin)
-  print >> sys.stderr, "%s: read %d PCs from stdin" % (prog_name, len(covered))
+  sys.stderr.write("%s: read %d PCs from stdin\n" % (prog_name, len(covered)))
   missing = instrumented - covered
-  print >> sys.stderr, "%s: %d PCs missing from coverage" % (prog_name, len(missing))
+  sys.stderr.write("%s: %d PCs missing from coverage\n" % (prog_name, len(missing)))
   if (len(missing) > len(instrumented) - len(covered)):
-    print >> sys.stderr, \
-        "%s: WARNING: stdin contains PCs not found in binary" % prog_name
+    sys.stderr.write(
+      "%s: WARNING: stdin contains PCs not found in binary\n" % prog_name
+    )
   for pc in sorted(missing):
-    print "0x%x" % pc
+    print("0x%x" % pc)
 
 if __name__ == '__main__':
   prog_name = sys.argv[0]
   if len(sys.argv) <= 2:
     Usage();
 
   if sys.argv[1] == "missing":
     if len(sys.argv) != 3: