This is an archive of the discontinued LLVM Phabricator instance.

Properly hook debuginfo-tests up to lit and CMake
Needs ReviewPublic

Authored by zturner on Sep 7 2017, 4:15 PM.

Details

Summary

This is all a bit hairy and magical, and I don't entirely understand how this works, but here is what I am trying to achieve:

  1. debuginfo-tests needs to be able to depend on both clang and lld.
  2. It needs to work both with and without the monorepo
  3. It should be opt-in

Currently the assumption is that you clone it under the clang/test folder, but the fact that it needs to depend on lld means that clang isn't really the best place for it. Besides, this is already broken in the mono-repo unless you manually symlink it from clang/test which seems odd.

It currently runs automatically as part of ninja check-clang, and this satisfies the opt in requirement by saying that you opt in by cloning the repo into the clang tree the first place. With this patch, the opt-in requirement is satisfied by creating a new cmake target for it, so if you want to run it you just run ninja check-debuginfo.

In order for all of this to work, I had to give it its own lit configs, and this is where things start going crazy. These lit configs are a bit of magic to me, so I'm not sure if I did this right. I copied the clang lit configs as the starting poitn, and deleted a bunch of stuff from them that I don't think we need for the purposes of debug info tests. sanitizer stuff, clang tool stuff, etc. in any case, on windows I can now run ninja check-debuginfo and it actually runs the test suite.

They all fail for me, but this is because it's trying to run a perl script to kick off the test, and my machine doesn't have perl on it.

What i need help with is:

  1. Am I going about this all wrong? Am I close?
  2. Am I missing anything obvious with the lit configs? Or not obvious?
  3. I don't have a non mono-repo anymore, I would appreciate it if someone could help me test the non mono-repo case.

Diff Detail

Event Timeline

zturner created this revision.Sep 7 2017, 4:15 PM
zturner updated this revision to Diff 114278.Sep 7 2017, 4:17 PM

Remove some debug print statements from cmake

zturner updated this revision to Diff 114279.Sep 7 2017, 4:18 PM

print statements didn't get removed, try again.

amccarth added inline comments.
debuginfo-tests/CMakeLists.txt
29

I'm concerned that the difference between ninja check-debuginfo and ninja check-llvm-debuginfo isn't very obvious. Can we tweak the name of this one, like check-debuginfo-integration?

aprantl edited edge metadata.Sep 7 2017, 4:41 PM

Currently the assumption is that you clone it under the clang/test folder, but the fact that it needs to depend on lld means that clang isn't really the best place for it. Besides, this is already broken in the mono-repo unless you manually symlink it from clang/test which seems odd.

  • It is supposed to test clang (the product), so being (logically) part of the clang tests seems reasonable to me.
  • If you are planning to change this assumption, you might also want to change the Readme to reflect the new situation.
  • There is also code/configuration in clang/test/lit.cfg that is specific to these tests that would need to be (re)moved.

All in all I am really not sure whether this is a good idea. This patch is duplicating a lot of code from clang's lit configuration and it seems likely that the two will diverge in the future when people update one but forget to update the other, leading to potentially very confusing bugs. What problem is this solving that it is worth the extra maintenance cost?

debuginfo-tests/CMakeLists.txt
22

why does it need lld? Shouldn't the system linker be good enough?

aprantl added inline comments.Sep 7 2017, 4:43 PM
debuginfo-tests/CMakeLists.txt
29

the target should be called check-clang-debuginfo. It is testing clang, not (or only indirectly) LLVM.

rnk added inline comments.Sep 7 2017, 4:43 PM
debuginfo-tests/CMakeLists.txt
29

That's the first I've heard of check-llvm-debuginfo. It looks like @beanz added that in 2015. Personally I like check-debuginfo as is.

zturner added inline comments.Sep 7 2017, 4:52 PM
debuginfo-tests/CMakeLists.txt
22

No, lld is a hard requirement. Part of what we're trying to do is test that we emit PDBs that can work with Microsoft tools. If we use the system linker, that defeats the entire purpose.

As an aside, "system linker" on Windows isn't really a thing, since systems don't really have linkers unless you install them. Which in this case would be MSVC's linker, which we already know works for obvious reasons

rnk added inline comments.Sep 7 2017, 4:53 PM
debuginfo-tests/CMakeLists.txt
22

Don't make this unconditional, do it like ASan does: add the dependency iff LLD is checked out. See how compiler-rt tests for it. We can feed lld's existence as a variable into the test suite. Besides, LLD doesn't have a functioning MachO port today. We need to disable any LLD-depending tests on Darwin.

debuginfo-tests/lit.cfg
330

This duplication sucks. We should install it in llvm's share directory next to its cmake or something, so that we can import it.

zturner added inline comments.Sep 7 2017, 4:53 PM
debuginfo-tests/CMakeLists.txt
29

In our case, it is also testing LLD's debug info, since LLD is what emits the PDB

zturner added inline comments.Sep 7 2017, 4:54 PM
debuginfo-tests/lit.cfg
330

Indeed it does. But at the same time, debug info tests don't need half of the stuff clang's is doing. Is there a good way to extract out the useful bits into something that is more general than just "these two specific projects need to share some specific functions"? Is there a way to make it useful for all projects?

probinson added inline comments.Sep 7 2017, 5:08 PM
debuginfo-tests/CMakeLists.txt
22

Don't see why lld is a "hard requirement." Surely you want clang's concept of CodeView output to work with the MSVC linker as well as with lld. Or is clang-cl no longer viewed as a drop-in replacement for cl?

debuginfo-tests/lit.cfg
289

See D37604.

jroelofs edited edge metadata.Sep 7 2017, 5:14 PM

Unless you're going for "make this lit.cfg as close as possible to clang's", then...

debuginfo-tests/lit.cfg
141

Probably don't need %llvmshlibdir.

142

Probably don't need %pluginext.

188

Probably don't need clang_analyze_cc1.

282

Probably don't need libgcc as an available feature.

292

Probably don't need def-fd-fs as an available feature.

aprantl added inline comments.Sep 7 2017, 5:17 PM
debuginfo-tests/CMakeLists.txt
22

The bots on green dragon, for example, run the debuginfo-tests and don't even check out the lld repository. It may be a hard requirement for windows, but certainly not on every platform.

aprantl added inline comments.Sep 7 2017, 5:19 PM
debuginfo-tests/lit.cfg
330

Is it possible to import the clang configuration somehow?

zturner added inline comments.Sep 7 2017, 5:19 PM
debuginfo-tests/CMakeLists.txt
22

Definitely we want that. For example, if you look back in the thread on llvm-dev, I posted this as an "example":

// RUN: %clangcl /Z7 %i /c /Fo%t.obj
// RUN: %lld-link /DEBUG %t.obj /out:%t.lld.exe
// RUN: %run_windbg %t.lld.exe %s
// RUN: %ms-link /DEBUG:FASTLINK %t.obj /out:%t.fastlink.exe
// RUN: %run_windbg %t.fastlink.exe %s
// DEBUGGER: bp 22
// DEBUGGER: g
// DEBUGGER: dt v
// CHECK: Local var {{.*}} Type SVal
// CHECK:  +0x000 Data : (null) 
// CHECK: +0x004 Kind : 0x85e

In this hypothetical example, I am testing clang-cl's codeview with MSVC's linker, but I'm *also* testing clang-cl's codeview with lld. In all honesty, CodeView is not that complicated. PDB is, and because of that it is actually the primary motivating factor behind getting debuginfo-tests working on Windows. Without LLD, it's impossible to test that LLVM's PDBs are valid.

So again, yes we want to test clang-cl's CodeView with MS's linker, but it's still a hard requirement that we need to test LLD's PDBs against MS's tools.

zturner added inline comments.Sep 7 2017, 5:20 PM
debuginfo-tests/CMakeLists.txt
22

That's a fair point. Reid's suggestion of making the lld dependency conditional on whether it's checked out seems like it would address that.

zturner updated this revision to Diff 114422.Sep 8 2017, 1:14 PM
zturner added a reviewer: chandlerc.

I think this addresses most of the concerns raised so far. The logic to conditionally add a dependency on CMake depends on D37637. I took rnk's advice and looked at what compiler-rt is doing, and was a little bit horrified. I can always go back to that if needed, but I decided to generalize what compiler-rt was doing and add it to LLVM's main CMake library instead, and then build off of that. I think it makes for cleaner / more easily understandable code this way.

I also talked with rnk about how to mitigate the effects of the code duplication. He can correct me if I'm misrepresenting him, but I believe we both came to the agreement that this duplication is something that has been ongoing with LLVM's lit infrastructure and configuration files for some time. That said, we both agreed that it should be possible to move a lot of this functionality into the core lit utility library. I think that's a good improvement which I'm willing to attempt in a follow-up.

In any case, +chandlerc because it seems like the # of people who understand this lit stuff is pretty small, so hopefully this elicits some additional feedback.

s/conditionally add a dependency on CMake/conditionally add a dependency on LLD/

beanz added inline comments.Sep 8 2017, 1:25 PM
debuginfo-tests/CMakeLists.txt
29

In LLVM & Clang we auto-generate lots of sub-check targets so they follow a convention check-${project}-${test subdirectory}. That's where the check-llvm-debuginfo target comes from, it is the <llvm>/test/DebugInfo directory of lit tests.

This is all done with the CMake function add_lit_testsuites.

I think this addresses most of the concerns raised so far.

Here's a couple of concerns that were mentioned before that are outstanding:

  • The Readme.txt needs to be updated, it currently states that the repository is supposed to be cloned into tools/clang/test.
  • The clang lit configuration needs to be updated to remove the debuginfo-tests-specific parts.
  • This patch is adding >300 lines of extra configuration. I still don't understand what problem this patch is solving that is worth the extra maintenance effort.

I think this addresses most of the concerns raised so far.

Here's a couple of concerns that were mentioned before that are outstanding:

  • The Readme.txt needs to be updated, it currently states that the repository is supposed to be cloned into tools/clang/test.
  • The clang lit configuration needs to be updated to remove the debuginfo-tests-specific parts.

Thanks, I'll take a look.

  • This patch is adding >300 lines of extra configuration. I still don't understand what problem this patch is solving that is worth the extra maintenance effort.

Part of what I was hoping to achieve with this code review is someone who understands clang's lit config to come along and say "oh you can delete all of this stuff, it's not necessary for this." I also agree there's too much. I'm currently thinking about ways to sink some of it into llvm/utils/lit, since much of it is already duplicated across multiple projects.

modocache resigned from this revision.Nov 21 2019, 8:10 PM
Herald added a project: Restricted Project. · View Herald TranscriptNov 21 2019, 8:10 PM