Refactor Infrastructure

As we develop, we sometimes gain new insights in how things should be arranged. For some, these insights are inspired by conversations with peers, sudden moments of clarity, or divine inspiration. For me, my insights come from being annoyed.

One of my chief design goals has always been to minimize server configuration. My app should be deployable to a random, clean server without having to install a bunch of things. The only exception is Docker (which is already pre-installed on many VPS products), because I make extensive use of docker compose.

This meant that I could not install a central proxy/load-balancer on the server. It had to be part of the Docker arrangement. So I used Caddy inside a docker.

Last year's situation

• production (docker compose)
  • Caddy, exposed ports 80:443
    • TLS for all domains and subdomains
    • reverse proxies to frontend and backend, internal network
    • reverse proxies to develop frontend and backend, through exposed ports
  • frontend
  • backend
  • Postgres database
• develop (docker compose)
  • frontend, exposed port
  • backend, exposed port
  • Postgres database

Another rule I have is that the docker-compose.yml must be identical for all environments: local, develop, production. I accomplish this by putting many vars in docker-compose.yml, that are automatically loaded from the .env file (which is obviously different in each environment). For example, I disabled Caddy in develop with something like this:

  caddy:
    image: caddy:latest
    deploy:
      replicas: ${CADDY_INSTANCES:-1}

Then you can add CADDY_INSTANCES=0 to your .env and Caddy is gone.

At this time, I had not yet figured out how to configure a setup without exposing ports. But I knew it was suboptimal and it was driving me nuts. Then one day, I found out that docker networks could be shared between different docker-compose groups.

networks:
  app-network:
    name: app-shared-network #you can pick whatever name
    external: true

If you define this network (with the same name) in another docker-compose.yml, suddenly the two groups can 'see' each other.

One caveat here is that for this to work, the network has to exist first. You can generate it with docker network create my-network.

With this new insight, I was able to further separate the develop environment, and put it under its own Caddy server.

Improved situation

• shared docker network
  • production
    • Caddy, exposed ports 80:443
      • TLS for all domains and subdomains
      • reverse proxies to frontend and backend, internal network
      • reverse proxy to Umami, internal network
      • reverse proxy to develop caddy, shared network
    • frontend
    • backend (blue, green)
    • Postgres database
    • Umami web analytics
  • develop
    • Caddy
      • only localhost:80
      • reverse proxies to frontend and backend, internal network
    • frontend
    • backend (blue, green)
    • Postgres database

This change was also partly to facilitate Blue-Green deployments, a method to achieve seamless deployments with zero downtime that I wrote about.

The exposed ports were finally closed. But a new annoyance was already taking shape. I wanted web analytics to see what visitors I'm getting so I installed Umami. But I had no need for Umami on develop so I had to use the replicas trick again. I don't like it when when I need to use tricks to work around the difference. I want both environments to be similar. And in the future, I will add more tools like Grafana and Redis, which will add even more bloat to the production environment.

After having this in the back of my head for a while, I came up with an architecture that pleased me:

Desired situation

• shared docker network
  • infra
    • Caddy 80:443
      • TLS for all domains and subdomains
      • reverse proxy to production caddy, shared network
      • reverse proxy to develop caddy, shared network
      • reverse proxy to umami, internal network
    • Umami web analytics
    • room for more monitoring tools
  • production
    • Caddy
      • only localhost:80
      • reverse proxies to frontend and backend, internal network
    • frontend
    • backend (blue, green)
    • Postgres
  • develop
    • Caddy
      • only localhost:80
      • reverse proxies to frontend and backend, internal network
    • frontend
    • backend (blue, green)
    • Postgres

I created a new git repository 'infra' that contains a docker-compose.yml (plus support files) that only has Caddy and Umami. It connects to the shared network that's already there (or creates it if it doesn't exist), and it relays everything to the develop and production Caddies. Those in turn relay everything to the proper endpoints.

All of this still works on localhost by the way! Here's the Caddyfile for localhost infra:

localhost {
	tls internal
	encode gzip

	reverse_proxy local-docker-caddy-1:80 {
		header_up Host {http.request.host}
		header_up X-Forwarded-Port {http.request.port}
	}
}

umami.localhost {
	tls internal
	encode gzip

	reverse_proxy local-infra-umami-1:3000
}

It turns out that you can define subdomains for localhost. With this setup, you can visit Umami by pointing your browser to https://umami.localhost

... and here's part of the infra compose.yml:

services:

  caddy:
    image: caddy:latest
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy-data:/data
      - caddy-config:/config
    restart: unless-stopped
    networks:
      - app-network
      - local-network
    dns:
      - 8.8.8.8
      - 1.1.1.1

  umami:
    image: ghcr.io/umami-software/umami:postgresql-latest
    init: true
    restart: always
    networks:
      - local-network

volumes:
  caddy-data:
  caddy-config:

networks:
  local-network:
  app-network:
    name: app-shared-network
    external: true

Docker network

If you add COMPOSE_PROJECT_NAME values to your envs, Docker will generate predictable container names that follow the convention: ${COMPOSE_PROJECT_NAME}-${service}-${replica}.

$ docker ps --format "table {{.Image}}\t{{.Names}}"
IMAGE                                            NAMES
adikhoff/eventapp-backend:local                  local-docker-backend-blue-1
adikhoff/eventapp-frontend:local                 local-docker-frontend-1
caddy:latest                                     local-infra-caddy-1
ghcr.io/umami-software/umami:postgresql-latest   local-infra-umami-1
caddy:latest                                     local-docker-caddy-1
postgres:18-alpine                               local-docker-postgres-1

Important: Make sure the project name is unique, otherwise Docker will treat them as the same project, and will rearrange and overwrite things you didn't expect.

These names will DNS resolve to the service you target, so when I configure this in my infra Caddy:

	reverse_proxy local-docker-caddy-1:80 {
		header_up Host {http.request.host}
		header_up X-Forwarded-Port {http.request.port}
	}

It relays to my local-docker Caddy, which has:

	handle_path /api/v2/* {
		reverse_proxy local-docker-backend-blue-1:8080 {
			header_up Host {http.request.host}
			header_up X-Forwarded-Proto https
			header_up X-Forwarded-Port 443
		}
	}

	handle {
		reverse_proxy local-docker-frontend-1:9090 {
			header_up Host {http.request.host}
			header_up X-Forwarded-Proto https
			header_up X-Forwarded-Port 443
		}
	}

Relaying the request to its proper recipient.

Future improvements

Currently, every container is on the same external network, so everyone can reach everyone, which can also lead to naming collisions. This can be further refined by defining internal and external networks. In my case, only the Caddy servers need to see outside their group, so they'll be on both internal and external networks. The rest only needs to see their dependencies. So that will be the next step.