[analyzer] Support spaces in compiler path and arguments.

[ayartsev]

Summary

The patch fixes errors that occur if a path to default compiler has spaces or if an argument with spaces is given to compiler (e.g. via -I).

The precedent: running scan-build that uses clang compiled by VS ends up with
error: error reading 'Files\Microsoft'
error: error reading 'Visual'
error: error reading 'Studio'
error: error reading '11.0\VC\include'

Ok to commit?

[jordan_rose]

Thanks, good catch.

tools/scan-build/ccc-analyzer

182–188

I think we can just pass true to quotewords' 'keep' parameter.

[ayartsev]

Ok to commit the solution with passing true to quotewords?

tools/scan-build/ccc-analyzer

182–188

Yes, it works.

[jordan_rose]

Yes, thanks!

[ayartsev]

The patch, committed as r236423 caused at least two types of regressions:

1. (not reproduced on Windows) :

clang: error: no such file or directory: '"-cc1"'
clang: error: no such file or directory: '"-triple"'
...

2. Macro definitions with quotes were corrupted ( e.g. -DVAL_CONFIGURE="\"'CFLAGS=-fno-vectorize' 'CC=ccc-analyzer'\"" )

In the updated patch I restricted wrapping an argument with quotes to cases when an argument has spaces and arguments first symbol is not a quotation mark.
Ok to commit?

[ayartsev]

Ping.

[jordan_rose]

This just seems wrong. What if I have a relative path that starts with a directory name that begins with a quote character? Maybe I deserve what I get, but we should figure out why this is happening and fix it rather than just ignore arguments beginning with quotes.

[ayartsev]

Another try: we preserve all quotes and backslashes as is by passing 'true' to quotewords() and only remove quotes from macro definitions and arguments without spaces.

[jordan_rose]

What about macro definitions that appear as one argument? "-DFOO=abc"?

I think we need to take a step back here. Why are some arguments being quoted and others not? Is that something we can control in scan-build itself? Is it something the build system does wrong?

[ayartsev]

The $line that hold arguments is the raw output from Clang with -### option passed. An output is consistent: each item in the output is double-quoted, each macro definition is split into "-D" and "definition".
In scan-build we call 'quotewords' to extract items. If we call 'quotewords' with the false 'keep' parameter we "remove all quotes and backslashes that are not themselves backslash-escaped or inside of single quotes" (perldoc). Here the problem with paths that have spaces arises (at least under Windows).
If we pass true to 'quotewords' 'keep' parameter we preserve all quotes and backslashes and this worked for me under Windows, but the regressions were reported in comments to r236423 like the following:
clang: error: no such file or directory: '"-cc1"'
clang: error: no such file or directory: '"-analyze"'
...

and regressions caused by extra quotes around macro definitions.
That's how matters stand.
So I came to the following solution: pass true to 'quotewords' to preserve arguments unmodified and then strip quotes from every macro definition and from all arguments that have no spaces (I've looked through the gcc options manual, it looks like macro definitions and paths are the only arguments that can have spaces if I am not missing something).

[jordan_rose]

It's really not a good idea to hardcode some list of commands that behave a certain way.

What you're telling me is that @items is correctly separated, but that when we actually spawn a subprocess we fail to preserve that. Alternately, perhaps @items is different on Windows vs. Linux, and then we should address that. I really think we shouldn't try to come up with something ad hoc.

[ayartsev]

Finally addressed issues with arguments. I had a patch for D8774 (Prevent ccc/c++-analyzer from hanging on Windows) applied locally (because I was unable to use scan-build without that patch) and that was the reason for the arguments being processed differently .
I replaced all calls to 'system' and 'exec' (that pass arguments to the system's command shell for processing) with calls to pipe form of 'open' and this solution worked consistently under Windows 7, OS X, Ubuntu and different Windows perl ports. Merged this patch with D8774. Please review!

[ayartsev]

Merged with D8774.

tools/scan-build/ccc-analyzer

sub ProcessClangFailure {
 
   # Create the info file.
   open (OUT, ">", "$PPFile.info.txt") or die "Cannot open $PPFile.info.txt\n";
   print OUT abs_path($file), "\n";
   print OUT "$ErrorType\n";
   print OUT "@$Args\n";
   close OUT;
   `uname -a >> $PPFile.info.txt 2>&1`;
-  `$Compiler -v >> $PPFile.info.txt 2>&1`;
+  `"$Compiler" -v >> $PPFile.info.txt 2>&1`;
   rename($ofile, "$PPFile.stderr.txt");
   return (basename $PPFile);
 }
 
 ##----------------------------------------------------------------------------##
 #  Running the analyzer.
 ##----------------------------------------------------------------------------##
 
sub GetCCArgs {
   }
 
   waitpid($pid,0);
   close(FROM_CHILD);
 
   die "could not find clang line\n" if (!defined $line);
   # Strip leading and trailing whitespace characters.
   $line =~ s/^\s+|\s+$//g;
-  my @items = quotewords('\s+', 0, $line);
+  my @items = quotewords('\s+', 1, $line);
+  my $IsMacroDef = 0;
+  foreach (@items) {
+    $_ = substr($_, 1, -1) if (!/\s+/ || $IsMacroDef);
+    $IsMacroDef = $_ eq "-D";
+  }
   my $cmd = shift @items;
   die "cannot find 'clang' in 'clang' command\n" if (!($cmd =~ /clang/));
   return \@items;
 }
 
 sub Analyze {
   my ($Clang, $OriginalArgs, $AnalyzeArgs, $Lang, $Output, $Verbose, $HtmlDir,
       $file) = @_;