A logic to copy LLVM licences into docker images.

[mehdi_amini]

Can you elaborate why this script is desirable/needed?

[ilya-biryukov]

In D38298#881628, @mehdi_amini wrote:

Can you elaborate why this script is desirable/needed?

Sure, sorry for leaving out the context.

Including licences is technically required in any binary distribution of LLVM, including ones inside Docker images.
Copying the licenses could be made optional, but always putting the licenses into Docker images should not hurt either.

[klimek]

utils/docker/scripts/build_install_llvm.sh

200

Somewhat unrelated, but I just noticed this - if we're more pieces from LLVM than Clang in that directory, perhaps call it LLVM_BUILD_DIR? (different patch is obviously fine)

utils/docker/scripts/llvm_checksum/collect_licenses.py

1 ↗

(On Diff #116736)

Isn't that actually the same as
find . -iname '*license*' |while read f; do mkdir -p "$TARGET_DIR/$(dirname "$f")" && cp "$f" "$TARGET_DIR/$f" ; done

[ilya-biryukov]

utils/docker/scripts/build_install_llvm.sh

200

Makes sense, will do in a separate commit.

utils/docker/scripts/llvm_checksum/collect_licenses.py

1 ↗

(On Diff #116736)

Almost.
LLVM also embeds some licences into the source tree (MD5, Copyright.regex).
I was initially thinking of parsing references to other licences inside LICENSE.txt, so I chose Python. But then decided to go with hard-coded list of files for now.

Python is probably too much for this, will replace with a bash script :-)

[ilya-biryukov]

• Copy licenses in bash instead of Python.

[mehdi_amini]

In D38298#881699, @ilya-biryukov wrote:

In D38298#881628, @mehdi_amini wrote:

Can you elaborate why this script is desirable/needed?

Sure, sorry for leaving out the context.

Including licences is technically required in any binary distribution of LLVM, including ones inside Docker images.
Copying the licenses could be made optional, but always putting the licenses into Docker images should not hurt either.

OK, but why is this logic applicable to clang/llvm built with the docker script and not for any build?

It seems to me that if this is deemed useful/necessary, that should be part of make install and controlled by a CMake flag (-DLLVM_INSTALL_LICENCES=ON) which could be enabled by default or not.

Docker is great for managing dependencies and providing build reproducibility: it is great to provide scripts that makes it easier to use to build clang/LLVM. However I'm worried about embedding any information in these scripts about build / release / packaging that wouldn't be Docker specific.

[klimek]

In D38298#882638, @mehdi_amini wrote:

In D38298#881699, @ilya-biryukov wrote:

In D38298#881628, @mehdi_amini wrote:

Can you elaborate why this script is desirable/needed?

Sure, sorry for leaving out the context.

Including licences is technically required in any binary distribution of LLVM, including ones inside Docker images.
Copying the licenses could be made optional, but always putting the licenses into Docker images should not hurt either.

OK, but why is this logic applicable to clang/llvm built with the docker script and not for any build?

It seems to me that if this is deemed useful/necessary, that should be part of make install and controlled by a CMake flag (-DLLVM_INSTALL_LICENCES=ON) which could be enabled by default or not.

Docker is great for managing dependencies and providing build reproducibility: it is great to provide scripts that makes it easier to use to build clang/LLVM. However I'm worried about embedding any information in these scripts about build / release / packaging that wouldn't be Docker specific.

"make install" is not primarily a distribution mechanism, docker is (most folks upload their docker images to public registries without thinking much about this).
Docker to me is more similar to having debian or rpm generating rules; if we had those, I'd also argue that we should include the licenses in the right places of these.

[mehdi_amini]

In D38298#883025, @klimek wrote:

"make install" is not primarily a distribution mechanism, docker is (most folks upload their docker images to public registries without thinking much about this).

I'm not sure why "primarily a distribution mechanism matters? How is one suppose to distribute?

Docker to me is more similar to having debian or rpm generating rules; if we had those, I'd also argue that we should include the licenses in the right places of these.

I'm using Docker in my CI purely for reproducibility. Again separation of concerns: mixing the tool (Docker) with the purpose (distribution) does not seem right to me.

I just checked and the pre-built releases on llvm.org don't ship with the license files (but one, by accident I guess).

So again a CMake option with make install seems a far better place to me. Second to this, release scripts in llvm, decoupled from Docker.

[klimek]

In D38298#883091, @mehdi_amini wrote:

In D38298#883025, @klimek wrote:

"make install" is not primarily a distribution mechanism, docker is (most folks upload their docker images to public registries without thinking much about this).

I'm not sure why "primarily a distribution mechanism matters? How is one suppose to distribute?

docker push to a public registry?

Docker to me is more similar to having debian or rpm generating rules; if we had those, I'd also argue that we should include the licenses in the right places of these.

I'm using Docker in my CI purely for reproducibility. Again separation of concerns: mixing the tool (Docker) with the purpose (distribution) does not seem right to me.

Well, we definitely want a license label in the docker image / want the right license field set for .deb / other distribution mechanisms, so I don't think there's full separation of concerns anyway.

I just checked and the pre-built releases on llvm.org don't ship with the license files (but one, by accident I guess).

So again a CMake option with make install seems a far better place to me. Second to this, release scripts in llvm, decoupled from Docker.

Where would license files go on a make install?

Putting it into release scripts is obviously fine, especially as you pointed out that we'll probably want to include them somehow in the prebuilt binary distributions.

[mehdi_amini]

In D38298#883177, @klimek wrote:

In D38298#883091, @mehdi_amini wrote:

In D38298#883025, @klimek wrote:

"make install" is not primarily a distribution mechanism, docker is (most folks upload their docker images to public registries without thinking much about this).

I'm not sure why "primarily a distribution mechanism matters? How is one suppose to distribute?

docker push to a public registry?

Not much different from pushing a .tar.gz resulting from the install to llvm.org (which is actually that how I package my internal distribution right now, luckily I don't distribute outside the company, so I haven't hit the license problem...).

Docker to me is more similar to having debian or rpm generating rules; if we had those, I'd also argue that we should include the licenses in the right places of these.

I'm using Docker in my CI purely for reproducibility. Again separation of concerns: mixing the tool (Docker) with the purpose (distribution) does not seem right to me.

Well, we definitely want a license label in the docker image / want the right license field set for .deb / other distribution mechanisms, so I don't think there's full separation of concerns anyway.

We want a license in every distribution: using docker, debian or not. I'm not sure why duplicating the logic everywhere would be a good thing?

I just checked and the pre-built releases on llvm.org don't ship with the license files (but one, by accident I guess).

So again a CMake option with make install seems a far better place to me. Second to this, release scripts in llvm, decoupled from Docker.

Where would license files go on a make install?

"$CLANG_INSTALL_DIR/share/llvm/licenses"? Eventually make the path it an option?

[klimek]

In D38298#883472, @mehdi_amini wrote:

In D38298#883177, @klimek wrote:

In D38298#883091, @mehdi_amini wrote:

In D38298#883025, @klimek wrote:

"make install" is not primarily a distribution mechanism, docker is (most folks upload their docker images to public registries without thinking much about this).

I'm not sure why "primarily a distribution mechanism matters? How is one suppose to distribute?

docker push to a public registry?

Not much different from pushing a .tar.gz resulting from the install to llvm.org (which is actually that how I package my internal distribution right now, luckily I don't distribute outside the company, so I haven't hit the license problem...).

True. I'm surprised every time I see somebody distributing tar.gz's with binaries :), but given that people do it, a script that puts the licenses somewhere makes sense; it does only replace 5 lines of code, though, so I'm not sure how urgent that is (but I'm totally in agreement it's an improvement :)

Docker to me is more similar to having debian or rpm generating rules; if we had those, I'd also argue that we should include the licenses in the right places of these.

I'm using Docker in my CI purely for reproducibility. Again separation of concerns: mixing the tool (Docker) with the purpose (distribution) does not seem right to me.

Well, we definitely want a license label in the docker image / want the right license field set for .deb / other distribution mechanisms, so I don't think there's full separation of concerns anyway.

We want a license in every distribution: using docker, debian or not. I'm not sure why duplicating the logic everywhere would be a good thing?

I just checked and the pre-built releases on llvm.org don't ship with the license files (but one, by accident I guess).

So again a CMake option with make install seems a far better place to me. Second to this, release scripts in llvm, decoupled from Docker.

Where would license files go on a make install?

"$CLANG_INSTALL_DIR/share/llvm/licenses"? Eventually make the path it an option?

[ilya-biryukov]

I agree that putting this into CMake and enforcing that every llvm projects lists its licenses there is probably right approach.
While easy to duplicate, keeping license-copy scripts up-to-date is somewhat hard.

That said, I think doing this properly in CMake would require some discussion and changes to all LLVM projects.
And it would be nice to have this change right now as a quick workaround.

[mehdi_amini]

In D38298#883561, @ilya-biryukov wrote:

And it would be nice to have this change right now as a quick workaround.

That isn't how LLVM development work AFAIK: we merge things when they are ready. And when we identify the right way of doing things, we try to do build an incremental path in this direction instead of building technical debt.
I can send an RFC to LLVM if needed.

[ilya-biryukov]

In D38298#883584, @mehdi_amini wrote:

That isn't how LLVM development work AFAIK: we merge things when they are ready. And when we identify the right way of doing things, we try to do build an incremental path in this direction instead of building technical debt.

Point taken, your proposed solution would certainly help every LLVM distribution, not only Docker images.
I don't see how a workaround like that building up that much technical debt, though. But we can do that in our particular distribution if you're opposed to having that upstream now.

I can send an RFC to LLVM if needed.

That would certainly be very helpful, if you have spare cycles to work on that, of course.

utils/docker/debian8/release/Dockerfile

 #===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===//
 #
 #                     The LLVM Compiler Infrastructure
 #
 # This file is distributed under the University of Illinois Open Source
 # License. See LICENSE.TXT for details.
 #
 #===----------------------------------------------------------------------===//
 # A release image, containing clang installation, produced by the 'build/' image
 # and adding libstdc++ and binutils.
 FROM launcher.gcr.io/google/debian8:latest
 
 LABEL maintainer "LLVM Developers"
+LABEL licences "LLVM licenses are under /usr/local/share/llvm/licences"
 
 # Install packages for minimal useful image.
 RUN apt-get update && \
     apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
     rm -rf /var/lib/apt/lists/*
 
 # Unpack clang installation into this image.
 ADD clang.tar.gz /usr/local/

utils/docker/example/release/Dockerfile

 # container.
 # Replace FIXMEs to prepare your own image.
 
 # FIXME: Replace 'ubuntu' with your base image.
 FROM ubuntu
 
 # FIXME: Change maintainer name.
 LABEL maintainer "Maintainer <maintainer@email>"
+LABEL licences "LLVM licenses are under /usr/local/share/llvm/licences"
 
 # FIXME: Install all packages you want to have in your release container.
 # A minimal useful installation must include libstdc++ and binutils.
 
 # Unpack clang installation into this container.
 # It is copied to this directory by build_docker_image.sh script.
 ADD clang.tar.gz /usr/local/

utils/docker/nvidia-cuda/release/Dockerfile

 # minimal size.
 # Replace FIXMEs to prepare a new Dockerfile.
 
 # FIXME: Replace 'ubuntu' with your base image.
 FROM nvidia/cuda:8.0-devel
 
 # FIXME: Change maintainer name.
 LABEL maintainer "LLVM Developers"
+LABEL licences "LLVM licenses are under /usr/local/share/llvm/licences"
 
 # Unpack clang installation into this container.
 ADD clang.tar.gz /usr/local/
 
 # C++ standard library and binutils are already included in the base package.

utils/docker/scripts/build_install_llvm.sh

 if [ -f "$CHECKSUMS_FILE" ]; then
   echo "Validating checksums for LLVM checkout..."
   python "$(dirname $0)/llvm_checksum/llvm_checksum.py" -c "$CHECKSUMS_FILE" \
     --partial --multi_dir "$CLANG_BUILD_DIR/src"
 else
   echo "Skipping checksumming checks..."
 fi
 
+# Copy licences into docker image.
+LICENCES_DIR="$CLANG_INSTALL_DIR/share/llvm/licenses"
+mkdir -p "$LICENCES_DIR"
+
+function copy_license_file() {
+  local SOURCE_RELPATH="$1"
+  local SOURCE_DIR_RELPATH="$(dirname "$SOURCE_RELPATH")"
+
+  echo "Copying licencese file $SOURCE_RELPATH"
+  mkdir -p "$LICENCES_DIR/$SOURCE_DIR_RELPATH"
+  cp "$SOURCE_RELPATH" "$LICENCES_DIR/$SOURCE_RELPATH"
+}
+
+echo "Copying licences for the Docker images."
+echo "Please note that the list of licenses may be incomplete or outdated."
+
+pushd "$CLANG_BUILD_DIR/src"
+
+for PROJECT in $LLVM_PROJECTS; do
+  find "$PROJECT" -iname "license*" | while read SOURCE_FILE; do
+    copy_license_file "$SOURCE_FILE"
+  done
+
+  if [ "$PROJECT" == "llvm" ]; then
+    copy_license_file "llvm/lib/Support/MD5.cpp"
+    copy_license_file "llvm/include/llvm/Support/MD5.h"
+  fi
+done
+
+popd # "$CLANG_BUILD_DIR/src"
+
 mkdir "$CLANG_BUILD_DIR/build"
 pushd "$CLANG_BUILD_DIR/build"
 
 # Run the build as specified in the build arguments.
 echo "Running build"
 cmake -GNinja \
   -DCMAKE_INSTALL_PREFIX="$CLANG_INSTALL_DIR" \
   -DLLVM_ENABLE_PROJECTS="$CMAKE_LLVM_ENABLE_PROJECTS" \