This is an archive of the discontinued LLVM Phabricator instance.

Paths

Table of Contentst

-
buildbot/google/docker/buildbot-mlir-nvidia/
-
google/
-
docker/
-
buildbot-mlir-nvidia/
4/9
Dockerfile

Differential D93960

Install Numpy in the MLIR Docker image, this is required for the python bindings
ClosedPublic

Authored by mehdi_amini on Dec 31 2020, 5:56 PM.

Download Raw Diff

Details

Reviewers

stellaraccident
gkistanova

Commits

rZORGe41a7c5e8e47: Install Numpy in the MLIR Docker image, this is required for the python bindings

Diff Detail

Repository: rZORG LLVM Github Zorg

Event Timeline

mehdi_amini created this revision.Dec 31 2020, 5:56 PM

Herald added a reviewer: gkistanova. · View Herald TranscriptDec 31 2020, 5:56 PM

Herald added subscribers: teijeong, rdzhabarov, tatianashp and 13 others. · View Herald Transcript

mehdi_amini requested review of this revision.Dec 31 2020, 5:56 PM

Herald added subscribers: stephenneuendorffer, nicolasvasilache. · View Herald TranscriptDec 31 2020, 5:56 PM

Harbormaster completed remote builds in B83795: Diff 314205.Dec 31 2020, 5:56 PM

You are going to also want the pybind11 package. That will be the next error after the numpy error :)

mehdi_amini added inline comments.Dec 31 2020, 6:00 PM

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile
21	pybind11-dev is installed here, isn't it enough?

stellaraccident added inline comments.Dec 31 2020, 7:10 PM

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile
21	Probably not because that is going to be the system pybind11 package, which for this OS, is quite old. The version we require is from 2020. We can just remove the install of the system version and switch to pip install, which is the most reliable way to stay current (we upped the min version after the above change was made). The cmake scripts will autodetect and use a pip version -- and this is how I've set everything else up.

Use pip to install pybind11

Harbormaster completed remote builds in B83798: Diff 314210.Dec 31 2020, 11:14 PM

mehdi_amini added inline comments.Dec 31 2020, 11:14 PM

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile
30	I suspect we should pin exact versions if we want reproducibility of the Docker image build, WDYT?

tschuett added a subscriber: tschuett.Jan 1 2021, 2:06 PM

tschuett added inline comments.

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile
18	Why do you use apt-get update, if you are interested in reproducibility? It actually hurts for non-pinned packages.

mehdi_amini added inline comments.Jan 1 2021, 4:49 PM

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile
18	That a good point as well, I was hoping that Ubuntu packages without experimental / PPA are fairly stable and should get only bug fixes? The problem I had in the past with pinning on exact package version was that they wouldn't stay available forever.
18	That a good point as well, I was hoping that Ubuntu packages without experimental / PPA are fairly stable and should get only bug fixes? I'm contrasting with `python pip` which is always pulling recent versions independently of the OS version.

stellaraccident added inline comments.Jan 1 2021, 8:03 PM

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile
30	Let's change it to: pip install numpy==1.19.3 pybind11==2.6.0 The pybind11 version is our minimum required in the cmake, and that specific numpy version avoids some incompatibilities on Windows that affect head versions on some patch levels. I verified that both work on Windows 10.

Pin pip-install package versions

Harbormaster completed remote builds in B83815: Diff 314238.Jan 1 2021, 8:10 PM

tschuett added inline comments.Jan 2 2021, 12:34 AM

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile
18	You pinned the base image. So you should be fine.

stellaraccident accepted this revision.Jan 2 2021, 12:41 AM

This revision is now accepted and ready to land.Jan 2 2021, 12:41 AM

Closed by commit rZORGe41a7c5e8e47: Install Numpy in the MLIR Docker image, this is required for the python bindings (authored by mehdi_amini). · Explain WhyJan 2 2021, 11:37 AM

This revision was automatically updated to reflect the committed changes.

mehdi_amini added a commit: rZORGe41a7c5e8e47: Install Numpy in the MLIR Docker image, this is required for the python bindings.

kuhnel added a subscriber: kuhnel.Jan 5 2021, 12:16 PM

kuhnel added inline comments.

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile
18	Why do you use apt-get update, if you are interested in reproducibility? It actually hurts for non-pinned packages. AFAIK we need to run an apt-update before we can install packages. Otherwise apt will not find them.

Revision Contents

Path

Size

buildbot/

google/

docker/

buildbot-mlir-nvidia/

Dockerfile

5 lines

Diff 314256

buildbot/google/docker/buildbot-mlir-nvidia/Dockerfile

	Show All 9 Lines
	# BUILDBOT_PORT - server port to connect to			# BUILDBOT_PORT - server port to connect to
	#===----------------------------------------------------------------------===//			#===----------------------------------------------------------------------===//

	# Use the image from NVIDIA as base			# Use the image from NVIDIA as base
	FROM nvidia/cuda:10.2-base-ubuntu18.04			FROM nvidia/cuda:10.2-base-ubuntu18.04


	# install build tools			# install build tools
	RUN apt-get update; \			RUN apt-get update; \
				tschuettUnsubmitted Not Done Reply Inline Actions Why do you use apt-get update, if you are interested in reproducibility? It actually hurts for non-pinned packages. tschuett: Why do you use apt-get update, if you are interested in reproducibility? It actually hurts for…
				mehdi_aminiAuthorUnsubmitted Done Reply Inline Actions That a good point as well, I was hoping that Ubuntu packages without experimental / PPA are fairly stable and should get only bug fixes? The problem I had in the past with pinning on exact package version was that they wouldn't stay available forever. mehdi_amini: That a good point as well, I was hoping that Ubuntu packages without experimental / PPA are…
				mehdi_aminiAuthorUnsubmitted Done Reply Inline Actions That a good point as well, I was hoping that Ubuntu packages without experimental / PPA are fairly stable and should get only bug fixes? I'm contrasting with `python pip` which is always pulling recent versions independently of the OS version. mehdi_amini: > That a good point as well, I was hoping that Ubuntu packages without experimental / PPA are…
				tschuettUnsubmitted Not Done Reply Inline Actions You pinned the base image. So you should be fine. tschuett: You pinned the base image. So you should be fine.
				kuhnelUnsubmitted Not Done Reply Inline Actions Why do you use apt-get update, if you are interested in reproducibility? It actually hurts for non-pinned packages. AFAIK we need to run an apt-update before we can install packages. Otherwise apt will not find them. kuhnel: > Why do you use apt-get update, if you are interested in reproducibility? It actually hurts…
	apt-get install -y software-properties-common apt-transport-https ca-certificates \			apt-get install -y software-properties-common apt-transport-https ca-certificates \
	clang-8 lld-8 ninja-build git wget gnupg ccache \			clang-8 lld-8 ninja-build git wget gnupg ccache \
	python3 python3-pip python3-psutil pybind11-dev \			python3 python3-pip python3-psutil \
				mehdi_aminiAuthorUnsubmitted Done Reply Inline Actions pybind11-dev is installed here, isn't it enough? mehdi_amini: pybind11-dev is installed here, isn't it enough?
				stellaraccidentUnsubmitted Not Done Reply Inline Actions Probably not because that is going to be the system pybind11 package, which for this OS, is quite old. The version we require is from 2020. We can just remove the install of the system version and switch to pip install, which is the most reliable way to stay current (we upped the min version after the above change was made). The cmake scripts will autodetect and use a pip version -- and this is how I've set everything else up. stellaraccident: Probably not because that is going to be the system pybind11 package, which for this OS, is…
	# dumb-init recommended in			# dumb-init recommended in
	# https://hub.docker.com/r/buildbot/buildbot-worker/dockerfile			# https://hub.docker.com/r/buildbot/buildbot-worker/dockerfile
	dumb-init;\			dumb-init;\
	update-alternatives --install /usr/bin/clang clang /usr/bin/clang-8 100 ;\			update-alternatives --install /usr/bin/clang clang /usr/bin/clang-8 100 ;\
	update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-8 100 ;\			update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-8 100 ;\
	update-alternatives --install /usr/bin/lld lld /usr/bin/lld-8 100			update-alternatives --install /usr/bin/lld lld /usr/bin/lld-8 100

				# PyBind11 and Numpy are required for MLIR bindings
				RUN python3 -m pip install numpy==1.19.3 pybind11==2.6.0
				mehdi_aminiAuthorUnsubmitted Done Reply Inline Actions I suspect we should pin exact versions if we want reproducibility of the Docker image build, WDYT? mehdi_amini: I suspect we should pin exact versions if we want reproducibility of the Docker image build…
				stellaraccidentUnsubmitted Not Done Reply Inline Actions Let's change it to: pip install numpy==1.19.3 pybind11==2.6.0 The pybind11 version is our minimum required in the cmake, and that specific numpy version avoids some incompatibilities on Windows that affect head versions on some patch levels. I verified that both work on Windows 10. stellaraccident: Let's change it to: ``` pip install numpy==1.19.3 pybind11==2.6.0 ``` The pybind11 version is…

	# install cuda			# install cuda
	# avoid popups for keyboard configurations			# avoid popups for keyboard configurations
	RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cuda			RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cuda

	# Ubuntu ships with old cmake version, install the latest one			# Ubuntu ships with old cmake version, install the latest one
	# from https://apt.kitware.com/			# from https://apt.kitware.com/
	RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null \| \			RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null \| \
	gpg --dearmor - \| \			gpg --dearmor - \| \
	▲ Show 20 Lines • Show All 51 Lines • Show Last 20 Lines