ESKOM-friendly home hosting on 64bit ARM SBCs

How to host websites and services on a fibre line while enduring regular power failures.

This website was hosted on an Intel NUC, sporting an Intel i7 CPU and a luxurious 32GBs of RAM. Serving websites from your home is viable when you have 100mbit symmetric fibre (You are awesome Vumatel). Unfortunately, South Africa occasionally can’t supply enough power to meet the demand of its public at which point South Africans experience load shedding.

My home was recently load shed for 5 hours a day on several days during the course of a week – and that got me thinking; why am I hosting relatively static content on a machine that uses around 200W of electricity when I could probably cut down on electricity costs by switching to a lower power machine and SSDs ? (I can’t switch everything, but small websites are a good target)

This seemed like the perfect time to try out Debian BUSTER for 64-bit ARM rawr. Running docker on a Pi with 1GB of RAM is probably a ridiculous, but it’s surprisingly usable. Better yet, you can run a Pi from a USB power bank for several hours, and UPS-like switch-over functionality is included as part of the deal (most of the time…) It’s got to be the cheapest way to reliably host anything and it reduces the power bill.

The first step is getting your routers to stay powered during a power failure. Decent routers usually have a Power-over-Ethernet capability and Mikrotik is no exception. Mikrotik makes a relatively inexpensive POE UPS for their routers called the mups. The mups is small, cheap and simply plugs in between the router and the existing POE source. It charges a 12V battery (you choose the size) and seamlessly switches to it in the event of a power failure.

The way a Mikrotik MUPS is supposed to look.

You might ask “Why can’t I use a normal UPS to power my routers ?” – you can, but a normal UPS has a battery and in order to power your equipment it has to take the battery power (DC), modulate it, send it through a step-up transformer and out to your device. Your device will generally take that AC 240V, step it down, rectify it (demodulate it) to DC and then use it. By stepping up and back down again you’re introducing a lot of inefficiency into the process, which translates into bigger batteries and big equipment. Mikrotik routers (like many routers) expect 10V-30V input – so when the power goes out and a MUPS is in use, the MUPS simply directly connects the battery to the router. The product is a simple battery can power a small Mikrotik router for several hours with almost no heat output and complete silence.

A 12V 7AH battery, Mikrotik MUPS with cover removed and Mikrotik HAP 802.11AC dual band router.

This thing is a beast – and works well on a MUPS despite the datasheet indicating otherwise. (Mikrotik RB4011)

Installing the Debian Buster preview image is easy, their wiki pretty-much does everything for you :

$ wget https://people.debian.org/~gwolf/raspberrypi3/20190206/20190206-raspberry-pi-3-buster-PREVIEW.img.xz
$ xzcat 20190206-raspberry-pi-3-buster-PREVIEW.img.xz | dd of=/dev/sdX bs=64k oflag=dsync status=progress```

I found I had to do a few things to get things running smoothly :
Set a timezone : tzselect
Set the hostname and associated hosts entry : /etc/hosts and /etc/hostname
Install locales : apt install locales
Install dpkg-reconfigure : apt install debconf
Reconfigure locales : dpkg-reconfigure locales (this gets rid of the missing locale error message)
Install some other stuff : apt install ntp build-essential byobu atop htop sudo

That’s what’s hosting this website.

If your Pi is on a reliable battery-backup* you can enable write-caching :

In /etc/fstab :

LABEL=RASPIROOT / ext4 rw,async,commit=500,noatime,discard,nodiratime 0 1
rw – read/write
async – asynchronously read and write (dangerous with a battery)
commit=500 – the amount of time the fs waits before forcicbly flushing buffers to disk (500 seconds)
noatime – don’t update access times on files
discard – use the TRIM command to tell the SSD what blocks are no longer in use (this often doesn’t work, but I’ve found it works on high-end modern Sandisk and Samsung SD/MicroSD cards)
nodiratime – don’t write the access time for directories

You’ll want to create a user account for yourself with sudo privileges :

usermod -aG sudo username

And add your ssh key: (from desktop) $ ssh-copy-id username@rpi
Test the login and then don’t forget to disable root login via ssh.

Install docker-ce for Debian – don’t forget to add your new user to the docker group :

sudo usermod -aG docker your-user

To install docker-compose you’ll need python3 and pip : apt install python3-pip python3

and then “pip3 install docker-compose”. It works beautifully.

And that’s about it. You may find that some common images don’t have variants available for arm64, but rebuilding them is educational in itself 🙂

Often cloning the repository associated with the image you want and then running “docker build -t mynickname/a-project:version .” is enough to generate arm arm64 variant of the project. You can then push the image to docker-hub for use with docker-compose by going “docker push mynickname/a-project:version”. You may need to log in first though : “docker login”.

It’s amazing what low-end hardware can do. And this is behind a firewall in a DMZ – so don’t worry about those 0.0.0.0s.

And yes, one might argue that publishing the above is a security risk… but then one might counter with “obfuscation isn’t security”.

Not bad for a machine hosting 9 websites. The SSD is an “endurance” SSD, so a bit of swapping shouldn’t be a problem. 

A side effect of this process was the discovery that Ghost is a real RAM-hog and CPU-inefficient. WordPress uses < 10% of the RAM Ghost uses… and the WordPress sites are a lot more complex. WordPress also responds faster than Ghost, so it may be time to switch.

Reverse Engineering the Ubbey Box

Some time ago I came across an ICO (distressed choir wails in the background as if welcoming lucifer to a dark cathedral) and said ICO was aiming to build a decentralised storage system. “Cool”, I thought, eager to experiment with a small amount of Ethereum on a diverse set of ICOs. I gave them some ETH… but before doing so I discovered that decentralised storage isn’t a new thing. Several projects have tried (and somewhat succeeded) in building dencentralised storage networks already; big examples being BitTorrent and IPFS being (neither offer in-band remuneration). In addition, cryptocurrency-supported examples include SiaCoin and StorJ – both of which have serious issues. After contributing to the Ubbey/Universal Labs ICO I decided to give Sia hosting a try… but that’s another story.

Ubbey’s 2000 Ethereum ICO was ultimately successful and I got some tokens. Shortly thereafter Ubbey allowed ICO participants to use some of their tokens to order their “Ubbey Box” at discounted rates. By this stage I had acquired additional Ubbey tokens at a fraction of the ICO price on decentralised exchanges (I love DEXs). Yesterday, the Ubbey box I ordered arrived. I only recently got a chance to play with it.

As with most boxes I don’t control, the Ubbey box has gone onto a DMZ network, which means it can’t access my home security cameras, storage machines, geyser controller, irrigation controllers… you get the idea.

First things first, unpacking :

Very nice Ubbey… the packaging looks good, even after being bashed around by DHL.

An Ubbey Box, by Universal Labs

This box is very obviously a general purpose media player that’s been rebranded to fit Ubbey’s needs. There’s nothing wrong with that, it’s sensible. They inadvertently hint at it being a general purpose media box with their specs, which quote the device as having an ARM CPU in it with 2 GBs of RAM, a VERY common configuration for ARM dev boards from around 2 years ago. This is begging to be disassembled…

So, onto the DMZ it goes. I installed the Ubbey app from the Google Play Store and soon enough it found the box (through what looked like an IP scan of the local subnet). The scan was followed by a login prompt and then a firmware update (forced) :

Unfortunately, it’s been about 2 hours now and it’s still updating…

While we wait for it to update, let’s see what’s inside :

This is how the device looks inside with the lower cover removed (it’s upside down in this shot).
The Ubbey logo is illuminated by an over-spec’ed 7-segment LED display.
The top-side of the main board of the Ubbey Box.

The top side of the main board of the Ubbey Box is -very- interesting. The device is equipped with a dual-chain/MIMO wifi module and the box has antennas for it installed. The top cover of the box has a large metal mass, which is squeezed against the top of that shielded cage/case with a thermal pad sandwiched inbetween. The board is clearly designed to have multiple display outputs with one set of traces unpopulated. The FORESEE module is probably flash “ROM” storage for the OS, which is OpenWRT. Also present are traces/holes for a TTL UART, an SD Card module and two USB ports (one of which appears to be USB 3).

It should be noted that the Ubbey Box runs an SSH daemon by default – attempts to log in to it using root/root, root/password, root/etc. proved fruitless.

The top side of the Ubbey Box PCB with the CPU/RAM shield/case and thermal pad removed.

A quick Google search for SEA Beelink SEA I found me this Amazon page :

Wireshark packet sniffing indicates that the device spends a lot of time communicating with 47.100.119.151, which seems to reverse-resolve to api.yqtc.co. Visiting the server with https indicates that the server’s HTTP certificate expired in May… so I thought the client device may be less than strict with it’s certificate acceptance. Some DNAT rules on my router and mitmproxy later and I got this :

mitmproxy is very cool software. Unfortunately the Ubbey Box’s Go client, although permissive, isn’t that permission 😉

It would appear that the best way of gaining access to this device is via the TTL UART.

Update 22 September 2018 : Serial Debug Console

Not my neatest work, but I now have access to the device’s debug console…
Part of the Ubbey Box’s boot process and the OpenWrt failsafe mode entry point.
The Ubbey Box’s labeled partitions.
The contents of description.xml inside what looks like an overlay filesystem called “nasetec”

More to follow…