Caching Docker Rails images - supplement
In my previous post, Caching Docker Rails images. Please read it first. This post is a continuation describing new exciting Docker (aka Moby) addition: volume mounts during build, added in buildkit#442. It’s part of Buildkit project, future replacement for current Docker build engine. You can learn more about Buildkit by reading Tõnis Tiigi’s post, Introducing BuildKit.
Proper syncing
To update source code in the repository my initial approach from previous post uses COPY
command.
COPY
command has one fundamental problem for this use case.
COPY
works similiarly to Unix cp
command - it only adds new files.
It is not capable of doing full sync and removing files from image when they are removed in the source.
Thanks to the latest update to Buildkit in buildkit#442 it is now possible to mount a volume during mount, including local filesystem. Combined with rsync
real syncing is finally available.
This feature is still in the proposal stage and there’s no stable version available. In order to build images you need to use custom dockerd
version. The good news is that images built this way are fully compatible with current stable Docker. This means that you only need the custom setup on a machine that creates images. However as it potentially is unstable use it carefully.
Preparing Docker for building images
In order to build Docker image using new feature you have to install Docker version that includes the latest change from buildkit#655.
The easiest way to do that is to build Docker yourself from source code. I use master
branch version to include the latest patches. Building process itself is very easy as Moby actually uses Docker to bootstrap itself and make the process even easier.
- Clone
moby
repository. What is Moby? The Moby Project is to Docker what Fedora is to Red Hat Enterprise Linux. - Solomon Hykes, Docker CTO/Foundergit clone git@github.com:moby/moby.git
- Run make to compile dockerd and all related processes. Make sure that you have running Docker. Moby actually uses Docker to build itself, so you don’t have to worry about any dependencies.
make
After the process finishes new binaries are available at bundles/binary-daemon
. These are portable go binray files, so you can install them by copying or adding to $PATH
. It’s important to copy all the binaries, not only dockerd
as they are direct dependencies.
The final step is to start dockerd
in experimental mode. In order to do that add --experimental
flag.
To save your time I created a Vagrantfile with all necessary steps included.
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provider "virtualbox" do |v|
v.memory = 2048 # additional memory is required
end
config.vm.provision "shell", privileged: false, inline: <<-SHELL
sudo apt-get update
sudo apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
software-properties-common \
make
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
sudo apt-get -y install docker-ce
sudo gpasswd -a vagrant docker
git clone https://github.com/moby/moby.git
cd moby
sg docker -c "make"
sudo service docker stop
sudo bundles/binary-daemon/* /usr/bin
sudo mkdir -p /etc/systemd/system/docker.service.d/
echo -e "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// --experimental" | sudo tee /etc/systemd/system/docker.service.d/override.conf > /dev/null
sudo systemctl daemon-reload
sudo service docker start
SHELL
end
Building images using RUN –mount
Buildkit is not yet default Docker’s builder. To use it you need set DOCKER_BUILDKIT
flag when running docker build
.
Additionally as new RUN --mount
syntax is still in experimental phase Dockerfile
has to be prepended with a directive:
# syntax = tonistiigi/dockerfile:runmount20180618
To check that it works correctly create a Dockerfile
with content:
# syntax = tonistiigi/dockerfile:runmount20180618
FROM alpine
RUN --mount=type=bind,source=.,target=/mounted-source cp /mounted-source/Dockerfile /copied-Dockerfile
Build it:
DOCKER_BUILDKIT=1 docker build .
You will notice slightly different output of buildkit compared to standard Docker build engine.
Cached Dockerfile update
Finally, with everything set up, we are able to tune Dockerfile.production.simplified.cached
from the previous post.
Old COPY . .
can be replaced with:
RUN apk add --no-cache --update rsync
RUN --mount=type=bind,source=.,target=/mounted-app rsync -ra /mounted-app/ /app