[docs][Bugpoint]Add notes about multiple crashes

[jsji]

Summary

When reducing case for a CodeGenCrash, bugpoint may generate a new
reduced
testcase that exposes/causes another crash or break something due to
limitation.

Bugpoint does not distiguish different crashes currently,
so when this happens, bugpoint will go on reducing for the new crash,
or just abort, we can't get the case reduced for the origial crash.

An advice is added into usage doc to connect to recommend checking error
message with scripts and `-compile-command`.

[jsji]

Fix typos in testcase.

[jsji]

Ping ...

[jsji]

Ping..
and add the most recent code modifier & patch reviewers . Thanks!

[vsk]

I think you've identified a real, pernicious problem, but am not sure that this is the best approach for addressing it. One potential problem here is that slight variations in the error message can confuse the reducer. If e.g. some location information changes in between messages, the reduction will stop too early.

Maybe we need better documentation/examples about how to write check scripts that validate the error message?

[jsji]

Thanks @vsk for the valuable comments.

In D66832#1664105, @vsk wrote:

I think you've identified a real, pernicious problem, but am not sure that this is the best approach for addressing it. One potential problem here is that slight variations in the error message can confuse the reducer. If e.g. some location information changes in between messages, the reduction will stop too early.

Maybe we need better documentation/examples about how to write check scripts that validate the error message?

Yes, you are absolutely right!
In some complex cases, the change might looks trivial, eg: location information, or call stack change.
And, yes, we should be able to use -custom-command and write a simple script to validate.

eg, for the testcase in llvm/test/BugPoint/crash-sameerror.ll, we should be able to use a custom-command line like:

$ cat t1.sh
#!/bin/bash
!(llc < $1 2>&1|grep "LLVM ERROR: Cannot select: intrinsic %llvm.ppc.altivec.dss" )

$ bugpoint -compile-custom -compile-command ./t1.sh t.ll

I think the major point of this patch is to make it easier for beginners, for common scenarios:
for most simple code gen crash, we can simply use follow command to reduce the testcases.

$ bugpoint --llc-safe t.ll -same-error 

[jsji]

I think it would be great to make the powerful bugpoint tool easier to use, especially for beginners, for common simple scenarios.

But if we don't think this is really necessary or in the wrong direction of design, I am OK to abandon this patch.

[vsk]

In D66832#1664974, @jsji wrote:

I think it would be great to make the powerful bugpoint tool easier to use, especially for beginners, for common simple scenarios.

But if we don't think this is really necessary or in the wrong direction of design, I am OK to abandon this patch.

I see the need to make error validation with bugpoint more beginner-friendly, but am worried about situations in which the mode proposed here could confuse beginners (what if the first error isn't the interesting one? or reductions stop too early?). I don't have a better alternative to suggest. Perhaps we could repost @mehdi_amini's writeup in the "Search for a string in the output" section within http://blog.llvm.org/2015/11/reduce-your-testcases-with-bugpoint-and.html, or link to it from the bugpoint docs.

[jsji]

Abandon the --same-error change, update docs instead.

[jsji]

> I see the need to make error validation with bugpoint more beginner-friendly, but am worried about situations in which the mode proposed here could confuse beginners (what if the first error isn't the interesting one? or reductions stop too early?). I don't have a better alternative to suggest. Perhaps we could repost @mehdi_amini's writeup in the "Search for a string in the output" section within http://blog.llvm.org/2015/11/reduce-your-testcases-with-bugpoint-and.html, or link to it from the bugpoint docs.

Make sense, the more flag, the more confusion. So I abandon the original change.

I am not sure whether it is appropriate to point to a writeup in blog, so I just simple refer it to the command in the official documents.
@vsk Any comments?

[jsji]

Ping? @vsk Do you think it worth updating Bugpoint.rst? If so, any comments? Thanks.

[vsk]

Thanks, this is quite helpful.

llvm/trunk/docs/Bugpoint.rst

 
 * In the code generator and miscompilation debuggers, ``bugpoint`` only works
   with programs that have deterministic output.  Thus, if the program outputs
   ``argv[0]``, the date, time, or any other "random" data, ``bugpoint`` may
   misinterpret differences in these data, when output, as the result of a
   miscompilation.  Programs should be temporarily modified to disable outputs
   that are likely to vary from run to run.
 
+* In the `crash debugger`_, ``bugpoint`` does not distiguish different crashes
+  during reduction. Thus, if new crash or miscompilation happens, ``bugpoint``
+  will continue with the new crash instead. If you would like to stick to
+  particular crash, you should write check scripts to validate the error
+  message, see ``-compile-command`` in :doc:`/Bugpoint`_.
+
 * In the code generator and miscompilation debuggers, debugging will go faster
   if you manually modify the program or its inputs to reduce the runtime, but
   still exhibit the problem.
 
 * ``bugpoint`` is extremely useful when working on a new optimization: it helps
   track down regressions quickly.  To avoid having to relink ``bugpoint`` every
   time you change your optimization however, have ``bugpoint`` dynamically load
   your optimization with the ``-load`` option.