In D32272#789006, @hfinkel wrote:

In D32272#788932, @dberris wrote:

LGTM for the most part, thanks for making the changes, it's much simpler now.

Thoughts @MatzeB? I think we can work on running/reporting the microbenchmarks data later, in changes to LNT.

Can you please summarize the state of this and the eventual goal? It looks like:

1. Eventually, we need a new LNT plugin (i.e. something like what we have in LNTBased/*) that understands how to interpret the output of the Google benchmark library.

I hope this could all be done within the test-suite, probably extending test-suite/litsupport. I hope/think that no changes to LNT will be needed, but I haven't investigated myself.
For example, I'm thinking that test-suite/litsupport could end up reporting the execution time of a single iteration of the code benchmarked with the Google Benchmark library, and that will map neatly to concepts already understood by LNT.

In D34368#784700, @kongyi wrote:

In D34368#784693, @efriedma wrote:

Testcase? (I think this affects inline asm.)

This change is actually laying ground for fixing an inline asm bug (PR33134). Test cases will be added along with rest of the fix in a follow up change.

In D34362#788285, @Meinersbur wrote:

I think nt.py is deprecated, and meant to be replaced by test_suite.py.

Maybe someone who already contributed to lnt should be necessary for acceptance.

Thanks for sharing your thoughts - that makes things quite a bit clearer to me.

There is quite a bit of code that is duplicated from LNT to test-suite as part of this patch.
I'm not entirely sure if having this much code duplication between LNT and test-suite is desirable, unless this is part of a bigger redesign, like we discussed in January in the thread starting roughly at http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20170123/423073.html, with a nicer design in the end?

In D33758#769748, @dsanders wrote:

In D33758#769689, @kristof.beyls wrote:

Replace the matcher if-statements for each rule with a state-machine. This
significantly reduces compile time, memory allocations, and cumulative memory
allocation when compiling AArch64InstructionSelector.cpp.o after r303259 is
recommitted.

My guess is that the compiler compile time goes down simply because tablegen produces far less code?

That's right. Instead of generating C++ and having the compiler optimize a single massive function, it's compiling a single copy of each predicate and chaining them together with a table.

Do you also have an insight of how using a state machine "interpreter" changes the run-time of the compiler itself compared to using the "directly-compiled" generated C++ code?
I can't predict really how the different tradeoffs in both approaches will affect the compiler's run-time performance in the end, so it would be nice to share insights you may have for this.

I haven't measured it yet since the priority so far has been to get the compile times back down to a reasonable level. Once I've finished posting the patches for review I'll take a look at the run-time effect.

My intuition is that the compiler was unlikely to have optimized the previous implementation very well (particularly since it didn't inline the single-use lambdas) and assuming that's the case, the overhead of the interpreter is likely to be limited to the cost of the table lookups. It's possible that it will turn out to be faster since the previous implementation had no chance of fitting into an instruction cache whereas the new code may fit on some machines. On machines where it doesn't fit, some paths are more common than others so it may still be able to use the instruction cache better than the old implementation did.

Also, if we end up using a state machine for this, I think it'd be a good idea to, after the design settles, write a blog post or something similar to explain how the state machine works, as I've heard from many people that reverse engineering the DAGISel-produced state machine was hard due to limited or no documentation on it.

Ok.

Replace the matcher if-statements for each rule with a state-machine. This
significantly reduces compile time, memory allocations, and cumulative memory
allocation when compiling AArch64InstructionSelector.cpp.o after r303259 is
recommitted.

Abandoning this as my motivation for this (fixing PR30324) has gone away, as that PR got fixed in another way, in r303360.
Splitting the StackProtector pass into a separate analysis and transformation is probably still nice, but probably also better pushed forward by someone with a stronger interest in the stack protector pass and/or someone who is more of an expert in this area.

committed in r304196.

Thanks all for the review comments!

LGTM, modulo considering a call instruction to be ending a basic block.
I don't know enough details about the X86_64 instruction set to judge whether the regular expressions are fully correct, but they look plausible to me.

Looks like a reasonable start to me (with one more nit-pick).
Let's get this in as it's valuable as it is.
It can be improved further incrementally.

Just wanted to point to https://reviews.llvm.org/D29494, which is a patch that isn't fully polished to split the StackProtector pass into an Analysis pass and a Transformation pass.
My motivation for that was also fixing PR30324.

Thanks for the review on the ARM-specific part, Diana!
@t.p.northover : would you have time for a quick glance at the target-independent parts in LegalizerInfo.h, LegalizerInfo.cpp and LegalizerInfoTest.cpp?

LGTM, even if you don't implement the nitpicky suggestions I made.

In D32706#743682, @mssimpso wrote:

Addressed Kristof's comments.

• Changed wording to refer to "widening" instructions rather than "lengthening" instructions.
• Changed function name to is isWideningFree.
• Updated the logic to consider the W instruction variants in addition to the L variants.
• Added code generation tests along side the cost test cases.
• Added additional tests for the W variants.

In D31965#743954, @anemet wrote:

Hey Matt,

In D31965#742295, @mssimpso wrote:

Hi Adam,

I'm not sure where the conversation about this patch landed, but I'm fine with it being a Cyclone only change for now if that's what you prefer. I haven't had a chance to evaluate it on our cores yet. But when I do, I can easily set MinVectorRegisterBitWith if there's any benefit. How does compile-time look?

There is no measurable compile-time change for AArch64 (testsuite, ctmark, spec).

I believe the idea was to try to enable this for all subtargets, assuming you and @evandro can test this. On the other hand, it's been almost a month and I'd like to wrap this up. So perhaps we should enable this for Cyclone and A57 for now and then perhaps file bugs for the remaining subtargets to evaluate this.

How does that sound, @rengolin and others?

Could you explain the rationale for adding libs/benchmarks-1.1.0?
Also, since that seems to have a different license, you probably need to add that directory to the top-level LICENSE.TXT file as one of the sub-directories with "additional or alternate copyrights,licenses, and/or restrictions".

In D32529#738192, @rovka wrote:

Thanks for splitting this out! The ARM part looks excellent :)
I think you could add some target-independent tests for this in unittests/CodeGen/GlobalISel/LegalizerInfoTest.cpp, to make sure it works to WidenScalar/MoreElements etc to a type that has a Libcall or Custom action. That way we won't have to depend on ARM for testing this functionality.

Adding a unit test.

I've also run the latest version of this patch with command line options '-O3 -DNDEBUG -mcpu=cortex-a57 -mthumb -fomit-frame-pointer' on the test-suite and a range of other benchmarks.
I do see the same few performance regressions as reported by Andrew, for the same reasons; and many performance improvements.
Overall, I see a geomean performance improvement of 1.70% for the benchmarks reporting execution time and a 1.63% performance improvement for the benchmarks reporting scores.

Adding big.LITTLE mcpu options for clang to be more command-line compatible with gcc is a good idea.
Mapping the code generation for it to what is done for the little core also is a defendable position.
I'm just wondering if in some of the implementation of this there could be less copy-paste? I've made a few specific comments in some places of the code where ideally there should be less copy-paste.

Committed in r300664

In D32048#728054, @MatzeB wrote:

In D32048#727043, @kristof.beyls wrote:

If a non-trivial amount of time is spent in zlib for clamav and we would want to retain benchmarking of the zlib library, maybe it'd be better to add zlib (with appropriate test input) as a separate benchmark to the test-suite?

There is already spec2000/164.gzip which has pretty much the same inflation/deflation code as zlib. Also this patch is not about adding a new benchmark but simply getting rid of an external dependency!

Move public methods to all be in the same location, as suggested by Ahmed, now that the implementation details are hidden more in private methods.

In D32015#727113, @a.elovikov wrote:

I have to note that after http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20170410/444886.html
[llvm] r300200 - Re-apply "[GVNHoist] Move GVNHoist to function simplification part of pipeline."
that test starts failing.

LGTM, Thanks!

In D30521#726797, @chandlerc wrote:

In D30521#698089, @MatzeB wrote:

In D30521#698048, @chandlerc wrote:

Currently, I really don't understand why this is the right approach...

We sometimes want to test a specific codegen API rather than a whole pass.

This adds a special pass to llc that features a minimalistic scripting language to
call some predefined API functions so we can test the API with the usual lit+FileCheck tools.

If you're actually trying to test a specific API, why aren't unittests the correct approach?

If the problem is that the unittests are hard to write, why not write libraries to make authoring the unittest easier?

I think there is something fundamental about how we write tests in llvm, and why we do not write pass tests in a unit testing framework because hey, who stops us implementing llc/opt as some functions in a unit testing library, they mostly have trivial APIs:

I think there is a beauty and we are hitting a sweet spot in the way most of our testing works in llvm: Forcing tests to be a simple human readable text file keeps things approachable and understandable. Understanding the basics of lit, llc, opt and FileCheck is easy. Having the full power of C++ in a unit test library on the other hand makes it way too easy do "fancy" stuff that is harder to understand because it happens to abstracted away in generic code and multiple layers of APIs that you have to learn and dig through when you actually have to understand a bug.

I don't see how things being abstracted away in a scripting language and complex fake "pass" infrastructure is going to be substantially better here.

Essentially, I totally understand prefering FileCheck and friends when testing something that is reasonably cohesive as a *pass*. But I feel like the point at which a scripting language or other things are being used to handle the fact that we're actually testing a specific API, I would *much rather* just write a unittest against that API directly.

I don't see why wanting FileCheck for things that aren't a pass isn't valid.

I don't think that pass vs. non-pass is the right distinction.

I think FileCheck makes sense when there is already a fundamental reason to have textual output that is sufficiently stable to use in tests. This can be everything from having an IR to having a clear and precise printing structure for an analysis. Those have utility *outside* of tests and also are great tools for testing.

But when the thing we are testing, fundamentally, is *an API*, I think we should test it *by calling that API*.

Admittedly I am not happy to introduce a scripting language here, but it seemed like the pragmatic choice compared with the alternative of building up a pile of small passes for every test or putting .mir code into C++ string literals in a unittest and still lacking all the llc mechanics for bringing up a target or having the nice matching capabilities of FileCheck.

Again, I am very much in favor of having the facilities from 'llc' or 'FileCheck' in useful APIs for unittests to leverage. But I don't think we should be adding a scripting language in order to write tests in a language other than C++ to exercise C++ API calls. I think this is significantly more burdensome than just writing unittests.

Sorry for the slow response, I think we may just fundamentally disagree about the right approach here. Currently, the approach you are proposing would make the resulting tests nearly incomprehensible to me... On the other hand, I think unittests (which are "just" C++ code even if sometimes non-obvious C++ code) *when used well* are much more friendly to people outside of the particular area than a MIR-specific scripting framework.

Avoid the external zlib dependency in ClamAV by copying zlib-1.2.11
source into the benchmark.

External dependencies are problematic in benchmarks because:

They are not compiled with the same compiler/flags as the rest of the benchmarks.
They are an additional burden to setup when running the test-suite. While zlib is a really popular and ubiquitous library it is still sometimes missing in cross-compilation settings.
No external dependencies simplifies the buildsystem.

Further tweaks based on Ahmed's feedback.

Thanks very much for the review Ahmed! I think I took all feedback into account in the new version of the patch. Please have another look.

update patch taking into account Ahmed's review feedback.

In D31965#724869, @anemet wrote:

Hi Kristof,

In D31965#724860, @kristof.beyls wrote:

In D31965#724804, @anemet wrote:

Hi Renato,

In D31965#724619, @rengolin wrote:

Hi Adam,

Interesting results! But it doesn't sound like this is Cyclone specific.

Sure it's not, it is just a deployment strategy for this change. See the FIXME in the code.

Rolling it out for Cyclone-only is just a way to get this going in a controllable manner. Other subtargets can roll it this out as people find the time to benchmark and tune this.

As the results section shows I had and still doing some tuning on this. This mostly allows 2-lane vectorization for 32-bit types so they benefit of vectorization is not so great thus the accuracy of the cost model is really tested by enabling this.

I also got the impression that this is a change that is somewhat (but only somewhat) independent of micro-architecture, as I assume this is mostly about trading off the overhead that may be introduced to get data into the vector registers vs the gain by doing arithmetic in a SIMD fashion.
Of course, the cost of getting data in vector registers and the gain of doing arithmetic in a SIMD fashion is somewhat micro-architecture dependent.

I noticed that Adam points to a number of other patches improving things - I'm assuming these other patches lower the cost of getting data into the vector registers?

Yes, do you have rL299482?

In D31965#724804, @anemet wrote:

Hi Renato,

In D31965#724619, @rengolin wrote:

Hi Adam,

Interesting results! But it doesn't sound like this is Cyclone specific.

Sure it's not, it is just a deployment strategy for this change. See the FIXME in the code.

Rolling it out for Cyclone-only is just a way to get this going in a controllable manner. Other subtargets can roll it this out as people find the time to benchmark and tune this.

As the results section shows I had and still doing some tuning on this. This mostly allows 2-lane vectorization for 32-bit types so they benefit of vectorization is not so great thus the accuracy of the cost model is really tested by enabling this.

It'd be nice if there could be a test case for the fallback mechanism in-tree for a case like this. It's not immediately obvious to me how such an in-tree test could be created at the moment.
All in all, my impression so far is that patch has 2 changes:

1. Having a mechanism to not blow up reportGISelFailure when a multi-step legalization fails half-way through
2. Allowing custom legalization of intrinsics. If I've understood correctly, maybe best to commit them separately once there's agreement?

Yes - you are correct with the above 2 impressions.
Please let me know if I should create two separate reviews instead of this one.

. Added a few short tests for a few more cores.

In D31359#712612, @aditya_nandakumar wrote:

In D31359#712574, @t.p.northover wrote:

The other (easier) approach is to reportGlobalISelFailure on intermediate instructions whenever we're unable to legalize/Lower them.

Is this all accounting for the fact that you may not know if you can legalize an intrinsic before you've already erased it? Seems like a bit of an odd situation to be in if so.

Not sure if I understand the question correctly. My view is the Intrinsic was legalized by the backend (created some machine instruction, erased the generic intrinsic and created intermediate instructions). It's entirely possible that the intermediate instructions may fail legalization (for e.g. missing LegalizeActions/custom lowering of other intrinsics etc).

• By refactoring a little bit more, it became possible to test using a regular unit test, not having to use a mock, which makes the test a lot cleaner.
• During review discussions with Hal, it become clear that it's probably best to stick to unit testing rather than using lit-based testing for this feature.

In D31236#707462, @hfinkel wrote:

I'm curious why you chose to take this approach rather than add some option that allows us to change the file name being read? If we do that, then we can test this with lit tests. I generally think of our practice as using mocking, and unit tests in general, for cases where lit tests aren't practical (or, to put it another way, the infrastructure necessary to enable them is more complicated than unit testing in C++). This does not seem to be the case here. It seems straightforward to make -proc-cpuinfo-file=/foo/bar/cpuinfo.txt (modulo bikeshed) work.

As I recall, there are several other places in Clang where this is also a problem (we have hard-coded file names for /etc/lsb-release, /etc/redhat-release, etc.).

2. Would this allow merging data?
   
   When sub-arch specific decisions are concerned, having a way to override a base default case would reduce the amount of code on both table-gen and c++ parts.
   
   For example, we could have a default catch-all case like {1,widen; 32,legal; 33,unsupp}, and later on, based on sub-arch decisions, increment legality, lib calls, etc.

With the patch as is, you indeed need to re-specify the info for all bit sizes.
It could be that I pushed the current patch way too far in breaking existing APIs and if I turned back the patch to only change the internal representations used in LegalizerInfo.cpp,
this would work. So, the idea being that tablegen info and targets specify for which data types an action can be taken that doesn't require changing bitsize, such as Legal, Lower, Libcall.
And then the target-independent logic can hopefully make decisions on most/all operations on how to adapt types to different bitsizes when that's needed.
I'll look into that.

• Made the API change to LegalizerInfo::setAction much smaller: the setAction API is largely unchanged now. The only difference is that it no longer allows legalizationActions that change the size to be specified this way.
• Instead, specifying how to legalize when the size of the type legalized needs to change is specified using a SizeChangeStrategy. In follow-up work, I think that these size-changing strategies will turn out to be largely target-independent, and therefore can be shared between all targets, and not need to be respecified for each target separately.
• Split setAction into setScalarAction and setPointerAction to avoid having to specify an LLT just to indicate whether the type is a scalar or a pointer(with address space).
• To keep this patch as NFC as possible, for AArch64, I had to come up with some complicated SizeChangeStrategies. While not pretty, it demonstrates that it is possible to create very custom SizeChangeStrategies. These ugly SizeChangeStrategies are also only expect to be there for a short while, until we make functional-change-changes to allow all non-power-of-two-sized types
• Moved some of the implementation code from LegalizerInfo.h to LegalizerInfo.cpp
• A lot of smaller cleanups.

• Split LegalizerInfo::setAction into setScalarAction and setPointerAction to avoid having to specify a mostly meaningless LLT as an argument just to indicate whether the type is a scalar or a pointer(with address space).

In D30529#699569, @dsanders wrote:

I haven't read the actual code yet, but I've got a couple questions and a comment based on the description and the conversation so far.

I hope the above makes sense?

It does, but there are two issues here:

In D30529#698648, @rengolin wrote:

Hi Kristof,

I haven't seen the patch at all, but what about situations where 64-bit is done with lib calls? For example, ldivmod takes 64-bit arguments and you wouldn't want to narrow them to 32-bits.

If this patch is intended to just simplify the legal vs. others, it shouldn't have a narrow-all that spans to +inf. Makes sense?

cheers,
--renato

• Rebased to top-of-trunk.
• Correctly store Action info for pointer types for targets with multiple address spaces.
• Added a few basic checks to verify correctness of a targets legalization specification.
• Added one more static function to LegalizerInfo to make specifications shorter and more readable. (UnsupportedButFor).
• Adapted setAction API to be able to specify if a particular setAction should be the first specification on the operation, or if it could be a refinement.
• Introduced a few typedefs to improve readability.
• Made legalization info identical to current ToT for all backends, so that this patch becomes NFCi.

Thanks for the updates, Daniel!
I don't have further remarks, but I think it's best if someone who understands the details better (like @ab ?) would also have a look.

In D30313#690811, @MatzeB wrote:

Sorry to be this guy: This benchmark is running for too long! We should aim for 0.5-1s runtimes for our benchmarks and the 1000000 looks arbitrary to me. (This takes nearly 3x the time of salsa20, the next slowest benchmark in SingleSource/Benchmarks/Misc for me).

ping

I think that there are 2 main reasons why a table could be empty for a particular metric on the daily report page: either the test run just didn't collect the metric, or it did collect the metric, but there were no significant deltas that need to be reported.
I'd say that the ideal behaviour would be to not show an empty table when the test run didn't collect the metric; but to show the empty table when test results were collected, but there were no significant deltas.

Thanks for all the feedback and the interesting discussion!
I've adapted the patch accordingly, going for using cmake features more directly rather than adding ever more lnt runtest command line options.
I've also added documentation that should help with using 'lnt runtest test-suite' in a cross-compiling setup.

ping

Thanks for all the feedback! I've adapted the patch accordingly and committed in r295039.

I also think this is an improvement.
Probably best to hear from Chris M to see if he thinks so too.