This is an archive of the discontinued LLVM Phabricator instance.

Differential D90578

[llvm] New worker for ThinLTO whole program devirtualization
ClosedPublic

Authored by tejohnson on Nov 1 2020, 6:05 PM.

Details

Reviewers
gkistanova
mtrofin
Commits
rZORGdd1e514bf8aa: [llvm] New worker for ThinLTO whole program devirtualization
Summary

Adds a new x86_64 Ubuntu worker to test ThinLTO bootstrap with
whole program devirtualization enabled.

Diff Detail

Event Timeline

tejohnson created this revision.Nov 1 2020, 6:05 PM
Herald added subscribers: pengfei, inglorion, Prazek. · View Herald TranscriptNov 1 2020, 6:05 PM
tejohnson requested review of this revision.Nov 1 2020, 6:05 PM
Harbormaster completed remote builds in B77212: Diff 302196.Nov 1 2020, 6:05 PM
mtrofin accepted this revision.Nov 2 2020, 11:48 AM

lgtm

buildbot/osuosl/master/config/builders.py
1073

Consider adding -DLLVM_CCACHE_BUILD=ON, too, to speed up compilation. (ccache is installed on the bot, afaik)

This revision is now accepted and ready to land.Nov 2 2020, 11:48 AM
tejohnson added inline comments.Nov 2 2020, 1:55 PM
buildbot/osuosl/master/config/builders.py
1073

I can do that. Do you know why this isn't just on by default for all bots? I checked and there's only a few that have it enabled.

I assume this helps only the first stage compile of a multi-stage build, since for the later stages the compiler will have changed?

mtrofin added inline comments.Nov 2 2020, 1:58 PM
buildbot/osuosl/master/config/builders.py
1073

I can do that. Do you know why this isn't just on by default for all bots? I checked and there's only a few that have it enabled.

You need to have ccache installed, which is up to the bot administrator. We install it.

I assume this helps only the first stage compile of a multi-stage build, since for the later stages the compiler will have changed?

At minimum, yes.

tejohnson added inline comments.Nov 2 2020, 2:03 PM
buildbot/osuosl/master/config/builders.py
1073

You need to have ccache installed, which is up to the bot administrator. We install it.

Got it, thanks.

tejohnson updated this revision to Diff 302403.Nov 2 2020, 2:03 PM

Enable ccache

Harbormaster completed remote builds in B77325: Diff 302403.Nov 2 2020, 2:03 PM
tejohnson added a comment.Nov 10 2020, 8:22 AM

@gkistanova did you want to look at this change? I'll go ahead and commit tomorrow since it has been accepted, unless I hear otherwise.

Closed by commit rZORGdd1e514bf8aa: [llvm] New worker for ThinLTO whole program devirtualization (authored by tejohnson). · Explain WhyNov 11 2020, 3:02 PM
This revision was automatically updated to reflect the committed changes.
tejohnson added a commit: rZORGdd1e514bf8aa: [llvm] New worker for ThinLTO whole program devirtualization.

Revision Contents

PathSize
buildbot/
osuosl/
master/
config/
13 lines
8 lines
3 lines

Diff 304665

buildbot/osuosl/master/config/builders.py

Show First 20 Lines • Show All 1,054 Lines • ▼ Show 20 Lines
# LTO and ThinLTO builders. # LTO and ThinLTO builders.
{'name' : "clang-with-thin-lto-ubuntu", {'name' : "clang-with-thin-lto-ubuntu",
'tags' : ["clang","lld","LTO"], 'tags' : ["clang","lld","LTO"],
'workernames' : ["as-worker-92"], 'workernames' : ["as-worker-92"],
'builddir': "clang-with-thin-lto-ubuntu", 'builddir': "clang-with-thin-lto-ubuntu",
'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72, lto='thin')}, 'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72, lto='thin')},
{'name' : "clang-with-thin-lto-wpd-ubuntu",
'tags' : ["clang","lld","LTO"],
'workernames' : ["thinlto-x86-64-bot1"],
'builddir': "clang-with-thin-lto-wpd-ubuntu",
'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(
jobs=72,
lto='thin',
extra_configure_args=[
'-DLLVM_CCACHE_BUILD=ON',
'-DCMAKE_CXX_FLAGS="-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit"',
'-DCMAKE_C_FLAGS="-O3 -Xclang -fwhole-program-vtables -fno-split-lto-unit"',
mtrofinUnsubmitted
Not Done
ReplyInline Actions

Consider adding -DLLVM_CCACHE_BUILD=ON, too, to speed up compilation. (ccache is installed on the bot, afaik)

mtrofin: Consider adding -DLLVM_CCACHE_BUILD=ON, too, to speed up compilation. (ccache is installed on…
tejohnsonAuthorUnsubmitted
Done
ReplyInline Actions

I can do that. Do you know why this isn't just on by default for all bots? I checked and there's only a few that have it enabled.

I assume this helps only the first stage compile of a multi-stage build, since for the later stages the compiler will have changed?

tejohnson: I can do that. Do you know why this isn't just on by default for all bots? I checked and…
mtrofinUnsubmitted
Not Done
ReplyInline Actions

I can do that. Do you know why this isn't just on by default for all bots? I checked and there's only a few that have it enabled.

You need to have ccache installed, which is up to the bot administrator. We install it.

I assume this helps only the first stage compile of a multi-stage build, since for the later stages the compiler will have changed?

At minimum, yes.

mtrofin: > I can do that. Do you know why this isn't just on by default for all bots? I checked and…
tejohnsonAuthorUnsubmitted
Done
ReplyInline Actions

You need to have ccache installed, which is up to the bot administrator. We install it.

Got it, thanks.

tejohnson: > You need to have ccache installed, which is up to the bot administrator. We install it. Got…
'-DCMAKE_EXE_LINKER_FLAGS="-Wl,--lto-whole-program-visibility"'])},
{'name' : "clang-with-lto-ubuntu", {'name' : "clang-with-lto-ubuntu",
'tags' : ["clang","lld","LTO"], 'tags' : ["clang","lld","LTO"],
'workernames' : ["as-worker-91"], 'workernames' : ["as-worker-91"],
'builddir': "clang-with-lto-ubuntu", 'builddir': "clang-with-lto-ubuntu",
'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72)}, 'factory' : ClangLTOBuilder.getClangWithLTOBuildFactory(jobs=72)},
# Builders for MLIR. # Builders for MLIR.
▲ Show 20 Lines • Show All 822 Lines • Show Last 20 Lines

buildbot/osuosl/master/config/status.py

Show First 20 Lines • Show All 232 Lines • ▼ Show 20 Lines reporters.MailNotifier(
sendToInterestedUsers = False, sendToInterestedUsers = False,
extraRecipients = [ extraRecipients = [
"caroline.concatto@arm.com", "flang_llvm_buildbots@arm.com"], "caroline.concatto@arm.com", "flang_llvm_buildbots@arm.com"],
subject = "Build %(builder)s Failure", subject = "Build %(builder)s Failure",
mode = "failing", mode = "failing",
builders = [ builders = [
"flang-aarch64-ubuntu", "flang-aarch64-ubuntu-clang", "flang-aarch64-ubuntu", "flang-aarch64-ubuntu-clang",
"flang-aarch64-ubuntu-gcc10"]), "flang-aarch64-ubuntu-gcc10"]),
reporters.MailNotifier(
fromaddr="llvm.buildmaster@lab.llvm.org",
sendToInterestedUsers = False,
extraRecipients=[
"tejohnson@google.com"],
subject = "ThinLTO WPD Failure: %(builder)s",
mode = "failing",
builders = ["thinlto-x86-64-bot1"]),
] ]
# Returns a list of Status Targets. The results of each build will be # Returns a list of Status Targets. The results of each build will be
# pushed to these targets. buildbot.plugins reporters has a variety # pushed to these targets. buildbot.plugins reporters has a variety
# to choose from, including email senders, and IRC bots. # to choose from, including email senders, and IRC bots.
def getReporters(): def getReporters():
if config.options.getboolean('Master Options', 'is_production'): if config.options.getboolean('Master Options', 'is_production'):
return all return all
else: else:
# Staging buildbot does not report issues. # Staging buildbot does not report issues.
return [] return []

buildbot/osuosl/master/config/workers.py

Show First 20 Lines • Show All 225 Lines • ▼ Show 20 Lines return [
# ML-Driven Compiler Optimizations build workers (Ubuntu x86_64) # ML-Driven Compiler Optimizations build workers (Ubuntu x86_64)
create_worker("ml-opt-dev-x86-64-b1", create_worker("ml-opt-dev-x86-64-b1",
properties={'jobs': 64}, max_builds=1), properties={'jobs': 64}, max_builds=1),
create_worker("ml-opt-rel-x86-64-b1", create_worker("ml-opt-rel-x86-64-b1",
properties={'jobs': 64}, max_builds=1), properties={'jobs': 64}, max_builds=1),
create_worker("ml-opt-devrel-x86-64-b1", create_worker("ml-opt-devrel-x86-64-b1",
properties={'jobs': 64}, max_builds=1), properties={'jobs': 64}, max_builds=1),
# Ubuntu x86_64
create_worker("thinlto-x86-64-bot1", properties={'jobs': 64}, max_builds=1),
] ]