Blue Green deployments

When you're a solo developer, you have to do all the OPS work yourself. At real companies, these things are handled by specialists, but since I don't have a specialist, I have to do my own thing.

On the flip side, this gives me the freedom to architect my infra exactly the way I want, so here's the story of how I implemented blue-green deployment with Docker, Caddy, and Bash.

My Setup

I never liked Kubernetes, so I use Docker Compose. I don't like how Nginx does TLS, so I use Caddy. For DB migrations I use Flyway. My main concern is simplicity. The professional packages offer a whole lot of stuff I don't need, and a ton of configuration options that I don't want to learn or specify.

I have a docker-compose.yml that's 132 lines long, with two small Dockerfiles for the Angular frontend and the Webflux backend. I have Caddyfiles for my three environments (local, dev, prod). Caddy magically negotiates certificates with letsencrypt so I get free TLS with ZERO configuration or preparation. And Docker Compose lets me define an internal network so the different parts can securely talk to each other.

In my local setup, I run Angular and Webflux in my IDE, with Postgres on Docker. I can run a local build with Docker Compose to test the full package with local TLS, on https://localhost. When I push to Github, my ci/cd pipeline creates the build images for frontend and backend, tags and pushes the images to Docker Hub, and publishes the changes to the develop environment on my VPS. When I'm ready to release, I create a pull request to merge to main, a release tag is created, and my changes are pushed to production. If successful, version numbers are bumped on develop and a new cycle starts. This is all automated.

Blue-Green deployments

This kind of setup is not without challenges. A while back, I was getting tired of the backend being offline for 5-10 seconds during a deploy (because it has to restart). This is especially annoying since I'm serving live data through SSE, and that umbilical cord gets cut which is noticeable in connected clients. In professional settings, there's usually multiple servers and an intelligent load balancer with a rolling deploy mechanism. I don't want multiple servers. It costs money and adds complexity. It's overkill for where I'm at.

The AI suggested adding a plugin to docker that does the thing. Two reasons why I didn't want to do that: 1. One of my chief design goals is to be able to deploy the entire stack to a clean, random server with minimal configuration. 2. It goes against the spirit of Independency Injection, which is my personal mission to replace and eliminate all dependencies that exist.

So I hand-implemented my own version of blue-green deployment using Docker, Caddy, and a little bit of Bash.

The general flow is like this:

• backend-blue is live and leading
• deploy new version to backend-green
• point Caddy to backend-green
• shutdown backend-blue
  • backend has a shutdown hook that sends a warning message to all connected clients
  • connected clients receive the warning, cut the SSE connection and immediately reconnect
  • Caddy sends the reconnect signal to backend-green. Clients are able to reconnect without loss
  • All this happens in milliseconds, the user doesn't notice anything
• deploy new version to backend-blue
• point Caddy back to backend-blue
• shutdown backend-green
  • clients perform the same dance in reverse, with zero interruption

Here's part of the deploy script

echo "🚀 Deploy backend-green, wait for health check"
export TAG_BACKEND_GREEN=${TAG_BACKEND}
docker compose up -d --remove-orphans --wait backend-green

echo "👉 Redirect Caddy to green"
cp Caddyfile-green Caddyfile
docker compose exec caddy sh -c 'caddy reload --config /etc/caddy/Caddyfile'

Shutdown hook

    @EventListener(ContextClosedEvent.class)
    public void onShutdown(ContextClosedEvent event) {
        log.info("Application shutdown initiated. Broadcasting warning to all SSE clients...");
        liveDataController.setShuttingDown(true);

        String warningPayload = "Server is shutting down. This stream will close shortly. Timestamp %s".formatted(Instant.now());

        notificationDispatcher.getAllActiveSinks().forEach((slug, sink) -> {
            try {
                DbNotification shutdownNotification = new DbNotification("shutdown-warning", warningPayload);
                sink.tryEmitNext(shutdownNotification);
                log.debug("Sent shutdown warning to event '{}'", slug);
            } catch (Exception e) {
                log.warn("Failed to send shutdown warning to {}", slug, e);
            }
        });

        try {
            Thread.sleep(1500);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

Result

Now deployments run invisible to the end user. They only see a pleasant message pop up that a new version is available with a refresh button.