[LNT] Python 3 support: adapt to map returning an iterator

[thopre]

Summary

Adapt calls to map() to the fact it returns an iterator in Python 3 when
necessary. This was produced by running futurize's stage2
lib2to3.fixes.fix_map, with manual clean up to remove unecessary list
wrapping and simplify the code.

[hubert.reinterpretcast]

lnt/external/stats/pstat.py

514

max(map(lambda item: len(makestr(item)), items))?

lnt/external/stats/stats.py

1633

See above.

lnt/lnttool/main.py

173 ↗

(On Diff #220975)

I believe the change to this line is not necessary.

lnt/server/db/migrate.py

83 ↗

(On Diff #220975)

The unpacking works fine without the change even with Python 3.

[thopre]

Remove list wrapping when not necessary and simplify code whenever possible

[thopre]

lnt/external/stats/stats.py

1569–1573

These do not seem to be used later in the function, how can I check if these are global variables?

lnt/server/db/migrate.py

83 ↗

(On Diff #220975)

Indeed, https://docs.python.org/3/reference/simple_stmts.html#grammar-token-assignment-stmt says:

Assignment of an object to a target list, optionally enclosed in parentheses or square brackets, is recursively defined as follows.

• If the target list is a single target with no trailing comma, optionally in parentheses, the object is assigned to that target.
• Else: The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets.

lnt/testing/__init__.py

383

Undoing useless change I made just before committing https://reviews.llvm.org/D65751

[hubert.reinterpretcast]

lnt/external/stats/stats.py

1569

tmp is consumed twice immediately below, so the list application would still be appropriate if we aren't removing this block altogether.

1569–1573

From https://docs.python.org/3/faq/programming.html#what-are-the-rules-for-local-and-global-variables-in-python:

If a variable is assigned a value anywhere within the function’s body, it’s assumed to be a local unless explicitly declared as global.

1575

Drive by:
alldata = N.array(list(itertools.chain.from_iterable(lists)))

[hubert.reinterpretcast]

lnt/external/stats/stats.py

1575

Hmm, since we need a list anyway, maybe:

for list in lists:
    alldata.extend(list)

I don't know enough about the way the implementation works to know that the chain version would be handled as well.

[thopre]

Remove unused variables defined from calling map rather than fixing map call

lnt/external/stats/stats.py

1569

Yes but map accepts an iterable so if the only uses are below (as the name tmp would imply) that change should be fine. I grepped LNT around for vars and means and couldn't find use elsewhere and git history isn't very informative. I've thus decided to remove the entire block.

[hubert.reinterpretcast]

LGTM with comments on further changes that could be made.

lnt/external/stats/pstat.py

965

Not the subject of this patch, but the cmp might be a problem.

1025

Ditto.

lnt/external/stats/stats.py

1266

Another opportunity to do clean-up in another patch. It seems this is some attempt to preallocate n (and it happens in a few other places), but I haven't found anything that indicates that this isn't just dead code.

1569

I agree with the removal of the block. On the now-moot point (in case it comes up in the future), by "consume" I did mean destructively consume. That is:

tmp = map(str, [0, 1])
a = list(tmp)
b = list(tmp)  # Empty list!

[thopre]

lnt/external/stats/pstat.py

965

Good catch again! I'll grep to see if there's more. I'll update the cmp patch accordingly.

lnt/external/stats/stats.py

1569

Ah right silly me, of course. Thanks for pointing this out.

lnt/external/stats/pstat.py

 Usage:   colex (listoflists,cnums)
 Returns: a list-of-lists corresponding to the columns from listoflists
          specified by cnums, in the order the column numbers appear in cnums
 """
     global index
     column = 0
     if isinstance(cnums, (list, tuple)):   # if multiple columns to get
         index = cnums[0]
-        column = map(lambda x: x[index], listoflists)
+        column = [x[index] for x in listoflists]
         for col in cnums[1:]:
             index = col
-            column = abut(column,map(lambda x: x[index], listoflists))
+            column = abut(column, [x[index] for x in listoflists])
     elif isinstance(cnums, str):              # if an 'x[3:]' type expr.
-        evalstring = 'map(lambda x: x'+cnums+', listoflists)'
+        evalstring = 'list(map(lambda x: x'+cnums+', listoflists))'
         column = eval(evalstring)
     else:                                     # else it's just 1 col to get
         index = cnums
-        column = map(lambda x: x[index], listoflists)
+        column = [x[index] for x in listoflists]
     return column
 
 
 def collapse (listoflists,keepcols,collapsecols,fcn1=None,fcn2=None,cfcn=None):
      """
 Averages data in collapsecol, keeping all unique items in keepcols
 (using unique, which keeps unique LISTS of column numbers), retaining the
 unique sets of values in keepcols, the mean for each.  Setting fcn1

 def list2string (inlist,delimit=' '):
     """
 Converts a 1D list to a single long string for file output, using
 the string.join function.
 
 Usage:   list2string (inlist,delimit=' ')
 Returns: the string created from inlist
 """
-    stringlist = map(makestr,inlist)
-    return string.join(stringlist,delimit)
+    stringlist = map(makestr, inlist)
+    return delimit.join(stringlist)
 
 
 def makelol(inlist):
     """
 Converts a 1D list to a 2D list (i.e., a list-of-lists).  Useful when you
 want to use put() to write a 1D list one item per line in the file.
 
 Usage:   makelol(inlist)
for i in range(len(lst)):
         if lst[i] == ['\n'] or lst[i]=='\n' or lst[i]=='dashes' or lst[i]=='' or lst[i]==['']:
             rowstokill = rowstokill + [i]
     rowstokill.reverse()   # delete blank rows from the end
     for row in rowstokill:
         del list2print[row]
     maxsize = [0]*len(list2print[0])
     for col in range(len(list2print[0])):
         items = colex(list2print,col)
-        items = map(makestr,items)
-        maxsize[col] = max(map(len,items)) + extra
+        maxsize[col] = max(map(lambda item: len(makestr(item)), items)) + extra
     for row in lst:
         if row == ['\n'] or row == '\n' or row == '' or row == ['']:
             print()
         elif row == ['dashes'] or row == 'dashes':
             dashes = [0]*len(maxsize)
             for j in range(len(maxsize)):
                 dashes[j] = '-'*(maxsize[j]-2)
             print(lineincustcols(dashes, maxsize))
         else:
def remap (listoflists,criterion):
     """
 Remaps values in a given column of a 2D list (listoflists).  This requires
 a criterion as a function of 'x' so that the result of the following is
 returned ... map(lambda x: 'criterion',listoflists).  
 
 Usage:   remap(listoflists,criterion)    criterion=string
 Returns: remapped version of listoflists
 """
-    function = 'map(lambda x: '+criterion+',listoflists)'
+    function = 'list(map(lambda x: '+criterion+',listoflists))'
     lines = eval(function)
     return lines
 
 
 def roundlist (inlist,digits):
     """
 Goes through each element in a 1D or 2D inlist, and applies the following
 function to all elements of float ... round(element,digits).

 @@@PURPOSE? 2007-11-26
 
 Usage:   arowcompare(row1,row2)
 Returns: an array of equal length containing 1s where the two rows had
          identical elements and 0 otherwise
 """
     return 
     if row1.dtype.char=='O' or row2.dtype=='O':
-        cmpvect = N.logical_not(abs(N.array(map(cmp,row1,row2)))) # cmp fcn gives -1,0,1
+        cmpvect = N.logical_not(abs(N.array(list(map(cmp, row1, row2))))) # cmp fcn gives -1,0,1
     else:
         cmpvect = N.equal(row1,row2)
     return cmpvect
 
 
  def arowsame(row1, row2):
     """
 Compares two rows from an array, regardless of whether it is an
else:                                  # IT MUST BE A 2+D ARRAY
                     except TypeError:    # the item to add isn't a list
                         uniques = N.concatenate([uniques,N.array([item])])
                 else:
                     pass  # this item is already in the uniques array
         else:   # must be an Object array, alltrue/equal functions don't work
             for item in inarray[1:]:
                 newflag = 1
                 for unq in uniques:  # NOTE: cmp --> 0=same, -1=<, 1=>
-                    test = N.sum(abs(N.array(map(cmp,item,unq))))
+                    test = N.sum(abs(N.array(list(map(cmp, item, unq)))))
                     if test == 0:   # if item identical to any 1 row in uniques
                         newflag = 0 # then not a novel item to add
                         break
                 if newflag == 1:
                     try:
                         uniques = N.concatenate( [uniques,item[N.newaxis,:]] )
                     except TypeError:    # the item to add isn't a list
                         uniques = N.concatenate([uniques,N.array([item])])

lnt/external/stats/stats.py

 
 Usage:   lpearsonr(x,y)      where x and y are equal-length lists
 Returns: Pearson's r value, two-tailed p-value
 """
     TINY = 1.0e-30
     if len(x) != len(y):
         raise ValueError('Input values not paired in pearsonr.  Aborting.')
     n = len(x)
-    x = map(float,x)
-    y = map(float,y)
+    x = list(map(float, x))
+    y = list(map(float, y))
     xmean = mean(x)
     ymean = mean(y)
     r_num = n*(summult(x,y)) - sum(x)*sum(y)
     r_den = math.sqrt((n*ss(x) - square_of_sums(x))*(n*ss(y)-square_of_sums(y)))
     r = (r_num / r_den)  # denominator already a float
     df = n-2
     t = r*math.sqrt(df/((1.0-r+TINY)*(1.0+r+TINY)))
     prob = betai(0.5*df,0.5,df/float(df+t*t))

 
 Usage:   llinregress(x,y)      x,y are equal-length lists of x-y coordinates
 Returns: slope, intercept, r, two-tailed prob, sterr-of-estimate
 """
     TINY = 1.0e-20
     if len(x) != len(y):
         raise ValueError('Input values not paired in linregress.  Aborting.')
     n = len(x)
-    x = map(float,x)
-    y = map(float,y)
+    x = list(map(float, x))
+    y = list(map(float, y))
     xmean = mean(x)
     ymean = mean(y)
     r_num = float(n*(summult(x,y)) - sum(x)*sum(y))
     r_den = math.sqrt((n*ss(x) - square_of_sums(x))*(n*ss(y)-square_of_sums(y)))
     r = r_num / r_den
     z = 0.5*math.log((1.0+r+TINY)/(1.0-r+TINY))
     df = n-2
     t = r*math.sqrt(df/((1.0-r+TINY)*(1.0+r+TINY)))
"""
     if len(x) != len(y):
         raise ValueError('Unequal N in wilcoxont.  Aborting.')
     d=[]
     for i in range(len(x)):
         diff = x[i] - y[i]
         if diff != 0:
             d.append(diff)
     count = len(d)
-    absd = map(abs,d)
+    absd = list(map(abs, d))
     absranked = rankdata(absd)
     r_plus = 0.0
     r_minus = 0.0
     for i in range(len(absd)):
         if d[i] < 0:
             r_minus = r_minus + absranked[i]
         else:
             r_plus = r_plus + absranked[i]

 groups, requiring at least 5 subjects in each group.  This function
 calculates the Kruskal-Wallis H-test for 3 or more independent samples
 and returns the result.  
 
 Usage:   lkruskalwallish(*args)
 Returns: H-statistic (corrected for ties), associated p-value
 """
     args = list(args)
     n = [0]*len(args)
     all = []
-    n = map(len,args)
+    n = list(map(len, args))
     for i in range(len(args)):
         all = all + args[i]
     ranked = rankdata(all)
     T = tiecorrect(ranked)
     for i in range(len(args)):
         args[i] = ranked[0:n[i]]
         del ranked[0:n[i]]
     rsums = []
Usage:   F_oneway(*lists)    where *lists is any number of lists, one per
                                   treatment group
 Returns: F value, one-tailed p-value
 """
     a = len(lists)           # ANOVA on 'a' groups, each in it's own list
     means = [0]*a
     vars = [0]*a
     ns = [0]*a
     alldata = []
-    tmp = map(N.array,lists)
-    means = map(amean,tmp)
-    vars = map(avar,tmp)
-    ns = map(len,lists)
+    tmp = map(N.array, lists)
+    means = list(map(amean, tmp))
+    vars = list(map(avar, tmp))
+    ns = list(map(len, lists))
     for i in range(len(lists)):
         alldata = alldata + lists[i]
     alldata = N.array(alldata)
     bign = len(alldata)
     sstot = ass(alldata)-(asquare_of_sums(alldata)/float(bign))
     ssbn = 0
     for list in lists:
         ssbn = ssbn + asquare_of_sums(N.array(list))/float(len(list))
     ssbn = ssbn - (asquare_of_sums(alldata)/float(bign))
     sswn = sstot-ssbn
     dfbn = a-1
for i in range(len(listoflists)):
         if listoflists[i] == ['\n'] or listoflists[i]=='\n' or listoflists[i]=='dashes':
             rowstokill = rowstokill + [i]
     rowstokill.reverse()
     for row in rowstokill:
         del list2print[row]
     maxsize = [0]*len(list2print[0])
     for col in range(len(list2print[0])):
         items = pstat.colex(list2print,col)
-        items = map(pstat.makestr,items)
-        maxsize[col] = max(map(len,items)) + extra
+        maxsize[col] = (max(map(lambda item: len(pstat.makestr(item)), items))
+                        + extra)
     for row in listoflists:
         if row == ['\n'] or row == '\n':
             outfile.write('\n')
         elif row == ['dashes'] or row == 'dashes':
             dashes = [0]*len(maxsize)
             for j in range(len(maxsize)):
                 dashes[j] = '-'*(maxsize[j]-2)
             outfile.write(pstat.lineincustcols(dashes,maxsize))

 independent samples.
 
 Usage:   akruskalwallish(*args)     args are separate arrays for 3+ conditions
 Returns: H-statistic (corrected for ties), associated p-value
 """
     assert len(args) == 3, "Need at least 3 groups in stats.akruskalwallish()"
     args = list(args)
     n = [0]*len(args)
-    n = map(len,args)
+    n = list(map(len, args))
     all = []
     for i in range(len(args)):
         all = all + args[i].tolist()
     ranked = rankdata(all)
     T = tiecorrect(ranked)
     for i in range(len(args)):
         args[i] = ranked[0:n[i]]
         del ranked[0:n[i]]
Usage:   aF_oneway (*args)    where *args is 2 or more arrays, one per
                                   treatment group
 Returns: f-value, probability
 """
     na = len(args)            # ANOVA on 'na' groups, each in it's own array
     means = [0]*na
     vars = [0]*na
     ns = [0]*na
     alldata = []
-    tmp = map(N.array,args)
-    means = map(amean,tmp)
-    vars = map(avar,tmp)
-    ns = map(len,args)
+    tmp = map(N.array, args)
+    means = list(map(amean, tmp))
+    vars = list(map(avar, tmp))
+    ns = list(map(len, args))
     alldata = N.concatenate(args)
     bign = len(alldata)
     sstot = ass(alldata)-(asquare_of_sums(alldata)/float(bign))
     ssbn = 0
     for a in args:
         ssbn = ssbn + asquare_of_sums(N.array(a))/float(len(a))
     ssbn = ssbn - (asquare_of_sums(alldata)/float(bign))
     sswn = sstot-ssbn

lnt/server/ui/views.py

def v4_global_status():
     # We use periods in our machine names. css does not like this
     # since it uses periods to demark classes. Thus we convert periods
     # in the names of our machines to dashes for use in css. It is
     # also convenient for our computations in the jinja page to have
     # access to
     def get_machine_keys(m):
         m.css_name = m.name.replace('.', '-')
         return m
-    recent_machines = map(get_machine_keys, recent_machines)
+    recent_machines = list(map(get_machine_keys, recent_machines))
 
     # For each machine, build a table of the machine, the baseline run, and the
     # most recent run. We also computed a list of all the runs we are reporting
     # over.
     machine_run_info = []
     reported_run_ids = []
 
     for machine in recent_machines:

lnt/testing/__init__.py

def check(self):
         assert (
             self.report_version == 2
         ), "Only v2 supported for LNT report format MetricSamples objects."
 
     def add_samples(self, new_samples, conv_f=float):
         """Add samples for this metric, converted to float by calling
         function conv_f.
         """
-        self.data.extend(list(map(conv_f, new_samples)))
+        self.data.extend(map(conv_f, new_samples))
 
     def render(self):
         """Return info from this instance in a dictionary that respects
         the LNT report format in the version specified at construction
         when printed as json.
         """
         return {self.metric: self.data if len(self.data) > 1 else self.data[0]}
 

lnt/testing/util/valgrind.py

def fromfile(file, path):
         num_samples = len(positions) + len(events)
         current_file = None
         current_function = None
         summary_samples = None
         for ln in it:
             # Check if this is the closing summary line.
             if ln.startswith('summary'):
                 key, value = ln.split(':', 1)
-                summary_samples = map(int, value.split())
+                summary_samples = list(map(int, value.split()))
                 break
 
             # Check if this is an update to the current file or function.
             if ln.startswith('fl='):
                 current_file = ln[3:-1]
             elif ln.startswith('fn='):
                 current_function = ln[3:-1]
             else:
                 # Otherwise, this is a data record.
-                samples = map(int, ln.split())
+                samples = list(map(int, ln.split()))
                 if len(samples) != num_samples:
                     raise CalltreeParseError(
                         "invalid record line, unexpected sample count")
                 data.records.append((current_file,
                                      current_function,
                                      samples))
 
         # Validate that there are no more remaining records.

lnt/tests/nt.py

def _execute_test_again(config, test_name, test_path, test_relative_path,
     assert os.path.exists(result_path), "Missing results file."
 
     results, no_errors = load_nt_report_file(result_path, config)
     assert len(results) > 0
     return results, no_errors
 
 
 def _unix_quote_args(s):
-    return map(pipes.quote, shlex.split(s))
+    return list(map(pipes.quote, shlex.split(s)))
 
 
 # When set to true, all benchmarks will be rerun.
 # TODO: remove me when rerun patch is done.
 NUMBER_OF_RERUNS = 4
 
 SERVER_FAIL = u'FAIL'
 SERVER_PASS = u'PASS'