Moving to Kubernetes in one flip

ClickFortify started life on a single VM running docker-compose. It now runs on Kubernetes on GCP and deploys through ArgoCD. This post is about the move between the two, and why I ended up doing it as one flip instead of a gradual migration.

The product can't pause

ClickFortify collects ad clicks on live traffic, scores them, and writes exclusions back into Google Ads and Meta. Clicks arrive around the clock. A migration plan that needs a maintenance window, or that leaves the tracker half on one stack and half on the other, costs real advertisers real budget.

So the question was not "how do we move everything?" It was "how do we make the switch a single, small, reversible step?"

Get the deploy path right first

Before any traffic moved, the cluster needed a deploy path I trusted. The GitOps setup uses three repositories: the application, the manifests ArgoCD deploys, and the platform infrastructure. The original plan kept the manifests inside the application monorepo. I split them out because ArgoCD re-clones the repository on every refresh and the monorepo is large, because the platform has its own lifecycle and reviewers, and because the app's CI then has no access to the platform repository at all.

That matters for a cutover. If the new side is deployed the same way it will be deployed forever after, then verifying it on day one also verifies the release process.

A parallel set of stage hostnames

The plan I settled on had three steps:

1. Deploy the full new stack on the cluster behind a parallel set of stage hostnames, such as stage.example.com.
2. Verify it there.
3. Repoint the production hostnames at the cluster in one flip.

Nothing about production changed while the new side was being tested. The old VM kept serving every real request, and the stage hostnames gave me a place to exercise the new stack end to end.

No data migration, and the catch that comes with it

Both sides used the same databases. That is what made a single flip possible: there was no copy to run, no sync to keep consistent, and no moment where the two stacks disagreed about the data. When DNS moved, the new stack was already reading and writing the same rows the old one had.

The catch is that the stage side was never really a sandbox. Anything it wrote landed in production data. Testing used a dedicated space set aside for it, and no real customer's script was ever pointed at the stage tracker.

Why the worker stayed at zero

The worker is the service that runs scheduled jobs. On a shared database, two workers means every scheduled job runs twice: emails go out twice, cleanups delete twice. So the worker on the new side stayed at zero replicas for the whole staging period.

# Illustrative: keep the new side's worker off until the flip
kubectl scale deployment worker --replicas=0

The flip, and checking it held

The flip itself was a DNS change: the production hostnames were pointed at the cluster. Before that, I checked that the image digests running on the cluster matched what I expected to ship.

Afterwards, the check that mattered most was on the tracker. It sits behind a CDN and a reverse proxy, and working out the visitor's real IP behind that is exactly the kind of thing that breaks when the topology changes. Live tracker traffic after the flip showed real visitor IPs and no CDN edge addresses. Once that held, the stage hostnames were removed.

Right after the flip, rollback was the same move in reverse: point the hostnames back at the old VM, which I stopped rather than deleted.

What I would tell someone doing the same move

1. If both sides can share the databases, do it. It removes the data migration and makes the cutover one DNS change.
2. List every scheduled job and make sure it runs in exactly one place during staging, even if that means a service cannot be rehearsed.
3. Drain or export queued and delayed jobs before the flip. They live outside the database.
4. Decide the verification check before the flip, and make it about real traffic, not health endpoints.