DockerHub Autobuilds for Multiple Architectures (cross-compilation)

What a mouthful that title is. This post is a WIP.

I recently discovered the joys (😉) of running docker containers on armhf and arm64 machines. This is a quick guide, mostly for myself, so I can reproduce the steps to creating dockerhub autobuilding images for multiple architectures.

AKA If you have a project, hosted in a public repository like Github or Bitbucket and your project may be run in a docker container on hosts with different CPU architectures, this is how you can get DockerHub to Autobuild your project.

Start by enabling "experimental" CLI features in your docker client : (add the "experimental" key and value)

cat ~/.docker/config.json 
        "auths": {
                "": {}
        "HttpHeaders": {
                "User-Agent": "Docker-Client/17.12.1-ce (linux)"
        "credsStore": "secretservice",
        "experimental": "enabled"

and your docker daemon : (and again, add "experimental")

cat /etc/docker/daemon.json 
    "experimental": true,
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []

Either create a new repository on DockerHub using the web interface or push an existing image to DockerHub (which automatically creates the repository) :

docker push aquarat/volantmq:amd64

In your repository, create the file structure described below and populate them accordingly. The documentation for this structure can be found here.
File structure : (largely lifted from this awesome Github answer)

├── Dockerfile
├── Dockerfile.aarch64
├── Dockerfile.armhf
└── hooks
    ├── build
    ├── post_checkout
    └── pre_build

hooks/build :


docker build 
    --file "${DOCKERFILE_PATH}" 
    --build-arg BUILD_DATE="$(date -u +"%Y-%m-%dT%H:%M:%SZ")" 
    --build-arg VCS_REF="$(git rev-parse --short HEAD)" 
    --tag "$IMAGE_NAME" 



BUILD_ARCH=$(echo "${DOCKERFILE_PATH}" | cut -d '.' -f 2)

[ "${BUILD_ARCH}" == "Dockerfile" ] && 
    { echo 'qemu-user-static: Download not required for current arch'; exit 0; }

QEMU_USER_STATIC_ARCH=$([ "${BUILD_ARCH}" == "armhf" ] && echo "${BUILD_ARCH::-2}" || echo "${BUILD_ARCH}")
    | grep 'name.*v[0-9]' 
    | head -n 1 
    | cut -d '"' -f 4)

    | tar xzv



BUILD_ARCH=$(echo "${DOCKERFILE_PATH}" | cut -d '.' -f 2)

[ "${BUILD_ARCH}" == "Dockerfile" ] && 
    { echo 'qemu-user-static: Registration not required for current arch'; exit 0; }

docker run --rm --privileged multiarch/qemu-user-static:register --reset

Dockerfile -> Your standard amd64 Dockerfile.
An example of the start of this would be VolantMQ’s Dockerfile :

cat Dockerfile.armhf 

FROM golang:1.11.1 as builder
LABEL stage=intermediate

and now Dockerfile.armhf, our armhf build :

cat Dockerfile.armhf 

FROM golang:1.11.1 as builder
LABEL stage=intermediate

COPY qemu-arm-static /usr/bin/

"qemu-arm-static" is a binary executable that acts as an emulator for armhf executables. It is downloaded by the pre_build script, which is called by DockerHub during the autobuild.


cat Dockerfile.aarch64 
FROM golang:1.11.1 as builder
LABEL stage=intermediate

COPY qemu-aarch64-static /usr/bin/

In order to allow the docker container to use this emulator you’ll need to register it as a binary executable handler (this tells the kernel how to deal with specific files). This should be covered by pre_build, but in case it isn’t: In Ubuntu install qemu-user-static :


or execute a docker image :

docker run --rm --privileged vicamo/binfmt-qemu:latest

Once you’ve got this done, you can test your builds locally, like so :

DOCKERFILE_PATH=Dockerfile.aarch64 IMAGE_NAME=aquarat/volantmq:latest-aarch64 bash -c "hooks/post_checkout && hooks/build"
DOCKERFILE_PATH=Dockerfile.armhf IMAGE_NAME=aquarat/volantmq:latest-arm bash -c "hooks/post_checkout && hooks/build"
DOCKERFILE_PATH=Dockerfile IMAGE_NAME=aquarat/volantmq:latest-amd64 bash -c "hooks/post_checkout && hooks/build"

If that works, you can get pave the way for the dockerhub manifest by pushing your newly-created images to dockerhub:

docker push aquarat/volantmq:latest-amd64
docker push aquarat/volantmq:latest-arm64
docker push aquarat/volantmq:latest-arm

You may need to log your docker client in : docker login

You should then commit your changes to your repository and push.

You’ll need to annotate your manifest images :

# Create a manifest that describes your DockerHub repository
# This takes the form of the multi-arch "virtual" image and then its constituent images.
docker manifest create aquarat/volantmq:latest aquarat/volantmq:aarch64 aquarat/volantmq:armhf aquarat/volantmq:amd64

# Tag each non-amd64 image apropriately
docker manifest annotate aquarat/volantmq:latest aquarat/volantmq:armhf --os linux --arch arm
docker manifest annotate aquarat/volantmq:latest aquarat/volantmq:aarch64 --os linux --arch arm64 --variant armv8

# and then push your changes to DockerHub
docker manifest push aquarat/volantmq

# and then to inspect the result :
docker run --rm mplatform/mquery aquarat/volantmq

Connect your dockerhub account to your Bitbucket/Github account. This can be found in your dockerhub profile page :

Go back to your repository, click the “Builds” tab and click “Configure Automated Builds”.

Set up the source repository.

and then set up some build rules :

dockerhub’s build rules page

Click “Save and Build” and watch what happens. It takes a while to build.

ESKOM-friendly home hosting on 64bit ARM SBCs

How to host websites and services on a fibre line while enduring regular power failures.

This website was hosted on an Intel NUC, sporting an Intel i7 CPU and a luxurious 32GBs of RAM. Serving websites from your home is viable when you have 100mbit symmetric fibre (You are awesome Vumatel). Unfortunately, South Africa occasionally can’t supply enough power to meet the demand of its public at which point South Africans experience load shedding.

My home was recently load shed for 5 hours a day on several days during the course of a week – and that got me thinking; why am I hosting relatively static content on a machine that uses around 200W of electricity when I could probably cut down on electricity costs by switching to a lower power machine and SSDs ? (I can’t switch everything, but small websites are a good target)

This seemed like the perfect time to try out Debian BUSTER for 64-bit ARM rawr. Running docker on a Pi with 1GB of RAM is probably a ridiculous, but it’s surprisingly usable. Better yet, you can run a Pi from a USB power bank for several hours, and UPS-like switch-over functionality is included as part of the deal (most of the time…) It’s got to be the cheapest way to reliably host anything and it reduces the power bill.

The first step is getting your routers to stay powered during a power failure. Decent routers usually have a Power-over-Ethernet capability and Mikrotik is no exception. Mikrotik makes a relatively inexpensive POE UPS for their routers called the mups. The mups is small, cheap and simply plugs in between the router and the existing POE source. It charges a 12V battery (you choose the size) and seamlessly switches to it in the event of a power failure.

The way a Mikrotik MUPS is supposed to look.

You might ask “Why can’t I use a normal UPS to power my routers ?” – you can, but a normal UPS has a battery and in order to power your equipment it has to take the battery power (DC), modulate it, send it through a step-up transformer and out to your device. Your device will generally take that AC 240V, step it down, rectify it (demodulate it) to DC and then use it. By stepping up and back down again you’re introducing a lot of inefficiency into the process, which translates into bigger batteries and big equipment. Mikrotik routers (like many routers) expect 10V-30V input – so when the power goes out and a MUPS is in use, the MUPS simply directly connects the battery to the router. The product is a simple battery can power a small Mikrotik router for several hours with almost no heat output and complete silence.

A 12V 7AH battery, Mikrotik MUPS with cover removed and Mikrotik HAP 802.11AC dual band router.

This thing is a beast – and works well on a MUPS despite the datasheet indicating otherwise. (Mikrotik RB4011)

Installing the Debian Buster preview image is easy, their wiki pretty-much does everything for you :

$ wget
$ xzcat 20190206-raspberry-pi-3-buster-PREVIEW.img.xz | dd of=/dev/sdX bs=64k oflag=dsync status=progress```

I found I had to do a few things to get things running smoothly :
Set a timezone : tzselect
Set the hostname and associated hosts entry : /etc/hosts and /etc/hostname
Install locales : apt install locales
Install dpkg-reconfigure : apt install debconf
Reconfigure locales : dpkg-reconfigure locales (this gets rid of the missing locale error message)
Install some other stuff : apt install ntp build-essential byobu atop htop sudo

That’s what’s hosting this website.

If your Pi is on a reliable battery-backup* you can enable write-caching :

In /etc/fstab :

LABEL=RASPIROOT / ext4 rw,async,commit=500,noatime,discard,nodiratime 0 1
rw – read/write
async – asynchronously read and write (dangerous with a battery)
commit=500 – the amount of time the fs waits before forcicbly flushing buffers to disk (500 seconds)
noatime – don’t update access times on files
discard – use the TRIM command to tell the SSD what blocks are no longer in use (this often doesn’t work, but I’ve found it works on high-end modern Sandisk and Samsung SD/MicroSD cards)
nodiratime – don’t write the access time for directories

You’ll want to create a user account for yourself with sudo privileges :

usermod -aG sudo username

And add your ssh key: (from desktop) $ ssh-copy-id username@rpi
Test the login and then don’t forget to disable root login via ssh.

Install docker-ce for Debian – don’t forget to add your new user to the docker group :

sudo usermod -aG docker your-user

To install docker-compose you’ll need python3 and pip : apt install python3-pip python3

and then “pip3 install docker-compose”. It works beautifully.

And that’s about it. You may find that some common images don’t have variants available for arm64, but rebuilding them is educational in itself 🙂

Often cloning the repository associated with the image you want and then running “docker build -t mynickname/a-project:version .” is enough to generate arm arm64 variant of the project. You can then push the image to docker-hub for use with docker-compose by going “docker push mynickname/a-project:version”. You may need to log in first though : “docker login”.

It’s amazing what low-end hardware can do. And this is behind a firewall in a DMZ – so don’t worry about those

And yes, one might argue that publishing the above is a security risk… but then one might counter with “obfuscation isn’t security”.

Not bad for a machine hosting 9 websites. The SSD is an “endurance” SSD, so a bit of swapping shouldn’t be a problem. 

A side effect of this process was the discovery that Ghost is a real RAM-hog and CPU-inefficient. WordPress uses < 10% of the RAM Ghost uses… and the WordPress sites are a lot more complex. WordPress also responds faster than Ghost, so it may be time to switch.