Lessons in DR, Part 2

2025-01-26 · 4 min read
docker incus

Alpine Container Pain... Again. Oda Clan Emblem

A solution to last post's problem is identified and partially in place.

No big surprise, a change in a PVE component in the handling of Docker within an Alpine Linux LXC has broken networking in my 5 docker containers.

My first attempt to dig into resolving this was to setup Docker in a Debian 12 container, which worked. There doesn't seem to be anything obvious in what broke docker networking in the Alpine Linux LXC containers, but I've now stopped looking for a root cause. Since I intend to leave PVE behind to setup an Incus testing host, I don't want to burn any more energy solving Docker issues on PVE-backed alpine LXC.

For now, migrating docker images to Debian containers and non-docker solutions works well in the interests of time and effort.

Thanks to the official Immich backup/restore solution, my photos are back online:

  • Backup the postgres db:
docker exec -t immich_postgres pg_dumpall --clean --if-exists --username=postgres | gzip > ./immich.dump.sql.gz
  • Copy your compose.yml and .env files, modify for new environment, fire it up to test
  • Stop the new docker and trash the data
  • Start the postgres component, restore the db, and then start the remaining components:
docker compose down -v
docker compose pull
docker compose create
docker start immich_postgres
sleep 10
gunzip < "./immich.dump.sql.gz" | sed "s/SELECT pg_catalog.set_config('search_path', '', false);/SELECT pg_catalog.set_config('search_path', 'public, pg_catalog', true);/g" | docker exec -i immich_postgres psql --dbname=postgres --username=<USERNAME>
docker compose up -d

The Technitium DNS servers have been rebuilt using the non-docker instructions. For now, this is a better solution from a service restoration perspective.

What is left is Frigate and Uptime Kuma. Uptime Kuma also has non-docker instructions, however I'm using this opportunity to manually migrate the docker image and persistent data to a new instance. Updates on that when it happens, likely tonight.

As for Frigate, I am waiting for a new B+M key Google Coral Accelerator for its restoration. My current Mini PCIE version was not the ideal choice for migrating to new hardware.

*
Jules