Migrations that fail closed

A schema change is the riskiest part of most deploys. New code can be rolled back in seconds. A database that is halfway between two shapes cannot. At ClickFortify, where I am the lead architect, the services run on Kubernetes and ArgoCD deploys them through GitOps. I wanted one property from the migration step: if it fails, nothing new runs.

That is what I mean by failing closed. A failed migration should stop the release, not let new pods start against a schema they don't expect.

Where the migration runs

ClickFortify's NestJS services use PostgreSQL through Prisma. Migrations are applied with prisma migrate deploy, which runs every pending migration in order and changes nothing else.

The question was where to run it. Running it inside the application container at startup means every replica races to migrate, and a failure shows up as crash-looping pods. Running it by hand from a laptop means it depends on someone remembering. Neither fails closed.

Instead, the migration is a Kubernetes Job that ArgoCD runs as a PreSync hook. ArgoCD runs PreSync hooks before it applies the rest of the manifests. If the hook Job fails, the sync aborts, and the new Deployment is never applied.

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/api:sha-abc1234
          command: ['npx', 'prisma', 'migrate', 'deploy']

This manifest is illustrative, not copied from a real one.

Same image as the release

The Job runs the same image as the release it is guarding. Deploying means committing a new image tag to the repository ArgoCD watches, and that tag is applied to the migration Job as well as to the api Deployment. So the migrations that run are exactly the ones shipped with the code about to go live. There is no separate migrations image that can drift out of step.

The deploy state lives in its own repository, separate from the application code. That keeps ArgoCD from re-cloning a large monorepo on every refresh, and it means what ArgoCD deploys is a small, reviewable history of tags and manifests.

No blind retries

backoffLimit: 0 is deliberate. Kubernetes would happily retry a failed Job, but a migration that failed partway may have left the schema half-applied. Running it again without looking can make that worse. One attempt, then stop and let a person read the logs.

activeDeadlineSeconds puts a ceiling on how long the Job can hang, for example on a lock, before it counts as failed.

Logs that survive

The hook delete policy is BeforeHookCreation. ArgoCD deletes the old Job only when it is about to create the next one. Until then, the failed Job and its pod logs stay in the cluster, which is exactly when you need them. A policy that deletes the hook on failure would throw away the evidence.

A healthy run is easy to recognise in those logs: Prisma reports the migrations it found, then either the list it applied or that nothing was pending, and the Job completes.

Order can matter

Running migrations before the new pods is the right default, and sometimes it is required. Consider a release where the new code stops writing a column, and the column's default is now wrong for the new world. The migration that fixes the existing rows and the default has to land before the new code starts, or new rows get the stale default with nothing to correct it.

A PreSync hook gives that ordering for free.

What I would tell someone setting this up

1. Run migrations as a pre-deploy hook that can abort the release, not at application startup.
2. Build the migration Job from the same image tag as the release, so code and schema can't drift.
3. Set backoffLimit: 0 and keep failed Job logs around; a half-applied schema needs a person, not a retry.
4. Deploy by sync only, because a restart skips the hooks.