ESKOM-friendly home hosting on 64bit ARM SBCs

How to host websites and services on a fibre line while enduring regular power failures.

This website was hosted on an Intel NUC, sporting an Intel i7 CPU and a luxurious 32GBs of RAM. Serving websites from your home is viable when you have 100mbit symmetric fibre (You are awesome Vumatel). Unfortunately, South Africa occasionally can’t supply enough power to meet the demand of its public at which point South Africans experience load shedding.

My home was recently load shed for 5 hours a day on several days during the course of a week – and that got me thinking; why am I hosting relatively static content on a machine that uses around 200W of electricity when I could probably cut down on electricity costs by switching to a lower power machine and SSDs ? (I can’t switch everything, but small websites are a good target)

This seemed like the perfect time to try out Debian BUSTER for 64-bit ARM rawr. Running docker on a Pi with 1GB of RAM is probably a ridiculous, but it’s surprisingly usable. Better yet, you can run a Pi from a USB power bank for several hours, and UPS-like switch-over functionality is included as part of the deal (most of the time…) It’s got to be the cheapest way to reliably host anything and it reduces the power bill.

The first step is getting your routers to stay powered during a power failure. Decent routers usually have a Power-over-Ethernet capability and Mikrotik is no exception. Mikrotik makes a relatively inexpensive POE UPS for their routers called the mups. The mups is small, cheap and simply plugs in between the router and the existing POE source. It charges a 12V battery (you choose the size) and seamlessly switches to it in the event of a power failure.

The way a Mikrotik MUPS is supposed to look.

You might ask “Why can’t I use a normal UPS to power my routers ?” – you can, but a normal UPS has a battery and in order to power your equipment it has to take the battery power (DC), modulate it, send it through a step-up transformer and out to your device. Your device will generally take that AC 240V, step it down, rectify it (demodulate it) to DC and then use it. By stepping up and back down again you’re introducing a lot of inefficiency into the process, which translates into bigger batteries and big equipment. Mikrotik routers (like many routers) expect 10V-30V input – so when the power goes out and a MUPS is in use, the MUPS simply directly connects the battery to the router. The product is a simple battery can power a small Mikrotik router for several hours with almost no heat output and complete silence.

A 12V 7AH battery, Mikrotik MUPS with cover removed and Mikrotik HAP 802.11AC dual band router.

This thing is a beast – and works well on a MUPS despite the datasheet indicating otherwise. (Mikrotik RB4011)

Installing the Debian Buster preview image is easy, their wiki pretty-much does everything for you :

$ wget https://people.debian.org/~gwolf/raspberrypi3/20190206/20190206-raspberry-pi-3-buster-PREVIEW.img.xz
$ xzcat 20190206-raspberry-pi-3-buster-PREVIEW.img.xz | dd of=/dev/sdX bs=64k oflag=dsync status=progress```

I found I had to do a few things to get things running smoothly :
Set a timezone : tzselect
Set the hostname and associated hosts entry : /etc/hosts and /etc/hostname
Install locales : apt install locales
Install dpkg-reconfigure : apt install debconf
Reconfigure locales : dpkg-reconfigure locales (this gets rid of the missing locale error message)
Install some other stuff : apt install ntp build-essential byobu atop htop sudo

That’s what’s hosting this website.

If your Pi is on a reliable battery-backup* you can enable write-caching :

In /etc/fstab :

LABEL=RASPIROOT / ext4 rw,async,commit=500,noatime,discard,nodiratime 0 1
rw – read/write
async – asynchronously read and write (dangerous with a battery)
commit=500 – the amount of time the fs waits before forcicbly flushing buffers to disk (500 seconds)
noatime – don’t update access times on files
discard – use the TRIM command to tell the SSD what blocks are no longer in use (this often doesn’t work, but I’ve found it works on high-end modern Sandisk and Samsung SD/MicroSD cards)
nodiratime – don’t write the access time for directories

You’ll want to create a user account for yourself with sudo privileges :

usermod -aG sudo username

And add your ssh key: (from desktop) $ ssh-copy-id username@rpi
Test the login and then don’t forget to disable root login via ssh.

Install docker-ce for Debian – don’t forget to add your new user to the docker group :

sudo usermod -aG docker your-user

To install docker-compose you’ll need python3 and pip : apt install python3-pip python3

and then “pip3 install docker-compose”. It works beautifully.

And that’s about it. You may find that some common images don’t have variants available for arm64, but rebuilding them is educational in itself ๐Ÿ™‚

Often cloning the repository associated with the image you want and then running “docker build -t mynickname/a-project:version .” is enough to generate arm arm64 variant of the project. You can then push the image to docker-hub for use with docker-compose by going “docker push mynickname/a-project:version”. You may need to log in first though : “docker login”.

It’s amazing what low-end hardware can do. And this is behind a firewall in a DMZ – so don’t worry about those 0.0.0.0s.

And yes, one might argue that publishing the above is a security risk… but then one might counter with “obfuscation isn’t security”.

Not bad for a machine hosting 9 websites. The SSD is an “endurance” SSD, so a bit of swapping shouldn’t be a problem. 

A side effect of this process was the discovery that Ghost is a real RAM-hog and CPU-inefficient. WordPress uses < 10% of the RAM Ghost uses… and the WordPress sites are a lot more complex. WordPress also responds faster than Ghost, so it may be time to switch.

Obsessive Home Automation

This is a quick deep dive into home automation with Home Assistant. Home automation is a very wide and complex topic, this post is mostly an overview of what I’ve personally found possible so far.

I bought my house back in 2011. The garden came with an irrigation system, but no valves and no automation. I went looking for valves and a suitable controller… but they were terrible; they had horrible LCD displays that required cryptographic experience to interpret and they cost a ton. The house’s alarm system also sucked. I never figured out how to operate it.

Back before Raspberry Pis if you wanted TCP/IP you needed a Phidget component.

Irritrol. It’s disgusting.

This was 2011 and that meant there were no Raspberry Pis. I automated the garden irrigation system by using Arduino (Atmel Atmega328P) MCUs, controlling relays on a long 100m I2C line around the garden. Using some tricks I managed to get the Arduinos down to 500 Hz and that was reliable enough.

Fortunately, things have changed; 2012 came, the Raspberry Pi 1 hit the market and suddenly these devices could be networked using Ethernet for a fee that wasn’t insane. Then came 2016 and the WiFi-enabled Espressif esp8266 MCU graced us with its incredible price point of ~$4.00 and tiny size. Initially your coding choices were Arduino or Lua but eventually MicroPython took away that world of hurt. MicroPython RAWKS.

Circa 2016, a very untidy ESP8266 borehole controller, temperature/humidity sensor and irrigation valve controller.

Olimex’s take on an esp8266 – this one controls lighting, a pool pump and monitors sand filter input pressure.

At this point I had automated some parts of my home, but all using bespoke code, communicating through an MQTT broker/server. This worked, but it wasn’t user-friendly. Great for cron-based irrigation control, crap for turning on lights during an evening with friends.

The pool pump never turns on at the wrong time.

Then, per chance, I came across Home Assistant. Initially I wanted to use it with a hacked Xiaomi Air Purifier (that’s a story on its own), but as I experimented with it I realise it had the ability to radically improve the usability of my existing home automation. I decided to give it a proper try and started configuring it to talk to my existing devices.


The first Hass experiment was a lonely Raspberry Pi 3B with a super crap 16GB Sandisk MicroSD card. Even in this configuration both Hass and InfluxDB were completely usable.

All my stuff was JSON-MQTT based in that my home-grown devices emitted (and ingested) JSON payloads via the MQTT broker. This was trivial to hand to Hass thanks to Hass’s “value_template” configuration directive:

A sample of Home Assistant’s JSONPath value templating for an electricity sensor that measure utility voltage.

Hass’s web-frontend representation of the compound results of the above sensor.

The sensor above is an EKM power meter, running on an RS422 bridge. A Golang application I wrote, running in a docker container on Hass talks to the meter via the bridge and sends the results to Hass via MQTT as a JSON object. Home Assistant is a collection of docker containers running on a machine (in my case an Arm single board computer).

A neat graph showing power consumption over the last 24 hours.

Hass has basic graphing functionality built-in, but for SUPER COOL FUN I installed the InfluxDB/Chronograf “add-on”.

I had never heard of InfluxDB… damn it’s cool:

Interactive graphs rock. This one shows power usage in watts.

I could sing InfluxDB’s praises for a long time… it’s unbelievably cool… but I’ll leave that for another day.

I neeeed moar GRAPHz…


It’s like eating candy and has similar health issues.

A Soil Moisture Sensor rendered in Grafana from data in InfluxDB.

Did I mention the soil moisture sensors ?


A small subset of soil moisture sensors in the house and garden. Also, gotta love the poop emoji.

For ages I had been looking for a soil moisture sensor solution, but they were all terrible and relatively expensive. I started with the Chirp from Tindie… but they aren’t wireless, they’re bare boards and as such they don’t last.


Dope.

I then tried an awesome solution by “ProtoArt” on Tindie called the GreenThumb (these are no longer for sale). These are esp8266 based. They worked really well and had some cleverness built in (frequency-resonance sensing of soil constituents), but it was also a bare board and esp8266s aren’t known to be light on battery use.

It’s 2019, enter…

The HHCC (also branded as Xiaomi Flora) soil moisture sensor. This is Bluetooth Low Energy-based. It does moisture, conductivity, light and temperature. It is rain-proof (it has internal seals), relatively affordable, aesthetically unobtrusive and critically the capacitance plates are embedded in the PCB, so should last a long time (compared to the Chirp probe where the sensing plates are traces on the outside of the PCB). Some awesome people have teared down the unit and the protocol is well understood.

I use a FriendlyArm NanoPi Air in a 3D printed case as a bridge/gateway between the sensors and the MQTT broker (and therefore both Hass and InfluxDB).

A NanoPi Air by FriendlyArm. This is a quad-core armhf machine with embedded WiFi, Bluetooth and an onboard eMMC device.

It runs nicely on a Mikrotik Router’s USB port. WiFi and soil moisture sensing.

Spot the soil moisture sensor.

All of these things can (and are) beautifully abstracted out into the Home Assistant web frontend, which runs nicely in both desktop Chrome and my phone’s browser. This got me thinking, maybe I should bring EVERYTHING into the MQTT broker?!? It’d have to be done properly because visions of the Mr Robot scene where a house goes bezerk are a real possibility where everything can be hacked and remotely controlled. Securing a home IOT network is an interesting topic on its own… maybe I’ll write about that next.

Back to apocalyptic home automation: A company called Itead makes a variety of switches which are meant to replace normal wall light switches. Crucially, these devices all use esp8266s inside, which means, with some finger gymnastics, they can be reprogrammed with custom firmware.

Yeah, you have to short R19 to ground during flashing, lots of fun. Also be sure to clear the RF MCU memory to avoid “ghost” switching ๐Ÿ˜€

The result of abstracting physical lights using MQTT as an API of sorts and Hass as a frontend.

Unfortunately, my house was built in the 80s and Sonoff devices need a neutral line to function. The idea of putting a neutral line in a light switch wall unit in the 80s would have seemed silly because switches only switch loads, they aren’t loads in and of themselves. Rewiring my house to have smart light switches has been a trial.

And some of this required work in the roof…

This was really quite painful.

But, now we have this:

Yes, the air-conditioning is also controlled by Hass. It gives me tremendous pleasure to find a use for a tablet computer from 2013 that has sat gathering dust for years. The light plate behind it used to house 5 switches, all of which have been abstracted and reconfigured in various ways. The tablet itself is stuck on using velcro and is powered by an over-specced PSU in the wall. The tablet is set up to limit the charge of its battery to extend the life of the unit. The interface is a web-app being run on the tablet by WallPanel (an open source app by ThanksMister). The web part of the interface is called HADashboard. This is what the YAML for it looks like :

The layout portion of the HADashboard YAML, which relies on additional definitions.

Some of the definitions used in the dashboard.

This system knows when my phone is on the network, it knows when there’s movement in the house (by virtue of being the alarm system – more on that later) and it can switch pretty much everything. The result is, I don’t have to turn on or turn off lights and when I do want to turn things on an off, I can do it from my phone, anywhere there’s internet connectivity. Once again, this is a complex subject suited to it’s own post.

An example auto-lighting automation. It needs some work, but it’s functional.

I hope you enjoyed this quick dive into obsessive home automation with Home Assistant.

Low Latency HDMI Streaming on the Cheap

Getting HD video from one point to another, wirelessly and with low latency/delay isn’t cheap. The best-known player in the market, currently, for these kinds of tasks, is Teradek with their BOLT range. Unfortunately, an entry-level Teradek Bolt goes for around R 37500 in South Africa (about $2690). This isn’t affordable in a number of contexts, and as such I tried my hand at finding a cheaper solution.

A “cheaper solution” invariably involves commodity hardware, specifically commodity hardware that is also modular and modifiable – so open source. It’d need to be something supporting a wireless connection option. WiFi is ubiquitous, cheap and highly flexible.

Enter the Raspberry Pi Zero W…

Raspberry Pi Zero WCC BY 2.0

This tiny little PCB runs Linux and handily has a built-in H.264 encoder as well as Bluetooth and WiFi – cool! The RPI Zero W also sports a camera connector (Camera Serial Interface or CSI) ย and that got me wondering: had anyone found a way of getting video from an SDI or HDMI cable into a Raspberry Pi via the CSI interface ? The CSI interface runs directly to the GPU (which does the encoding) and therefore cuts out common CPU-intensive issues that arise when using USB interfaces.

Ah yes, the B101 HDMI to CSI adapter, made by Auvidea. This board handily converts an HDMI stream into a stream that looks like a CSI camera. This board looks like it’s Plug ‘n Play but I soon found out that that wasn’t the case.

Tons of trawling through various forums resulted in me eventually coming up with a partial solution.

You’ll need a specific copy of Yet Another Video4Linux Test Application (yavta). This Yavta sets some registers on the video encoder, starts the pipeline and reads out the results to stdout. That stdout can be redirected easily, I used socat (like netcat) to send the output out to another machine via UDP. This is the final command :

Run ./yavta -c -f UYVY -n 3 --encode-to=- -m -T /dev/video0 | socat - udp-sendto:10.0.0.20:5000 on the Pi and
ffplay -probesize 32 -sync ext -fflags nobuffer -flags low_delay -framedrop -strict experimental -i udp://10.0.0.20:5000 on the receiver

But, before running this command you paradoxically have to provide an EDID definition to the V4L drivers, like so :

v4l2-ctl --set-edid=file=1080P30EDID.txt --fix-edid-checksums 

and the contents of the EDID file above :

00ffffffffffff005262888800888888
1c150103800000780aEE91A3544C9926
0F505400000001010101010101010101
010101010101011d007251d01e206e28
5500c48e2100001e8c0ad08a20e02d10
103e9600138e2100001e000000fc0054
6f73686962612d4832430a20000000FD
003b3d0f2e0f1e0a2020202020200100
020321434e041303021211012021a23c
3d3e1f2309070766030c00300080E300
7F8c0ad08a20e02d10103e9600c48e21
0000188c0ad08a20e02d10103e960013
8e210000188c0aa01451f01600267c43
00138e21000098000000000000000000
00000000000000000000000000000000
00000000000000000000000000000000

The weird part is that this works for any resolution and frame rate provided it is progressive (not interlaced) and is part of the HD-family of resolutions (namely 1920×1080 and 1280×720, I haven’t tested the ugly sister 1440×1080).

Audio, via I2S requires a whole new realm of heartache and I found it to be generally unreliable.

The result is a feed which shows a delay of 10 frames on a 1080P 25fps stream. This is about 400ms – which isn’t great, but considering it’s going from a camera, through an encoder, out via WiFi to an access point, through a switch, through a router, through a switch and then being decoded on another machine, I think the result is a decent first start.

The next step is to experiment with low latency options in the Pi’s H.264 encoder and also test the latency when the link is peer-to-peer.

The most interesting indication I’ve found of low-latency GOP options on the encoder is the register

MMAL_PARAMETER_VIDEO_ENCODE_H264_LOW_LATENCY

in mmal_parameters_video.h but so far it doesn’t seem to have any effect.