Database migrations can be tricky, but they’re a crucial part of deploying new versions of applications that rely on an evolving schema. The last thing you want is to roll out an update without making sure the database is in sync with your app. Luckily, ArgoCD's PreSync hooks are perfect for managing these migrations. They make sure everything runs smoothly by ensuring the migration happens first, before your new application version gets deployed.

You might initially think that you can "just put" the database migrations in the CI stage. However, with GitOps you are meant to separate the build process from the release process. By not separating it, you compromise your ability to fully manage your release process solely with the Git tooling.

You might also think that you should just execute these migrations manually, as a work-around, but that risks you making an important process vulnerable to human error. What happens if someone forgets to migrate properly before pushing something to production?

Why use the PreSync hook?

The PreSync phase is designed to run tasks, like database migrations, before any other resources are applied in ArgoCD.

  1. Protects Your Application: PreSync hooks make sure that your app only gets deployed if the database migration completes successfully. If the migration fails, your app won’t be deployed, saving you from potential crashes or data corruption.
  2. Keeps Your App Stable: ArgoCD won’t move forward with deploying your application until the PreSync tasks (like your migration) finish successfully. This gives you peace of mind that your app won’t try to run with an outdated database schema.
  3. Fully Automated: With ArgoCD’s GitOps model, everything is automated. You just push your changes to your Git repository, and ArgoCD handles the rest, including your database migration. No need to manually intervene.

How to set it up

To use PreSync hooks for database migrations, you just need to define a Kubernetes Job that runs your migration and annotate it so ArgoCD knows to run it during the PreSync phase. You’ll also need to include any resources the migration job depends on—like secrets or configmaps—in the PreSync phase.

Here’s a simple example of how you can set up a database migration job with PreSync:

lang yaml

Copy code
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: your-database-migration-image
        command: ["sh", "-c"]
        args: ["your-database-migration-script.sh"]
      restartPolicy: Never

In this example:

  • The argocd.argoproj.io/hook: PreSync annotation tells ArgoCD to run this job before deploying the application.
  • The argocd.argoproj.io/hook-delete-policy: HookSucceeded ensures that the job is cleaned up after it successfully completes.

Depending on what your preferred flow is, you will likely want to change the hook-delete-policy.

  • HookSucceeded: The hook resource is deleted after the hook succeeded
  • HookFailed: The hook resource is deleted after the hook failed.
  • BeforeHookCreation: (Default) Any existing hook resource is deleted before the new one is created. It is meant to be used with /metadata/name.

Don’t forget the dependencies

It’s important to remember that any resources your migration job needs—like database credentials—also need to be part of the PreSync phase. If you forget to do this, your migration job might fail because it doesn’t have the required resources available at the right time.

For instance, if your migration job needs a secret to connect to the database, you should annotate that secret as PreSync:

lang yaml

Copy code
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  annotations:
    argocd.argoproj.io/hook: PreSync

This ensures that the secret is created before the migration job runs.

Tip: Keep resources separate

One thing to watch out for is resource overlap between the PreSync phase and the main application deployment. For example, if your migration job and your app share the same database credentials, the PreSync hook might accidentally delete or recreate the secret before your app gets deployed, causing issues.

A good practice is to create separate resources for the PreSync phase and the main application deployment. This way, the resources your migration job uses won’t interfere with the resources your app needs.

Here’s an example:

PreSync Secret: Used by the migration job.

lang yaml

Copy code
apiVersion: v1
kind: Secret
metadata:
  name: db-migration-credentials
  annotations:
    argocd.argoproj.io/hook: PreSync

Application Secret: Used by your app to connect to the database.

lang yaml

Copy code
apiVersion: v1
kind: Secret
metadata:
  name: app-db-credentials

By separating these secrets, you avoid any conflicts between the PreSync job and your application, ensuring a smoother deployment process.

If you are using Helm, I recommend you put all your presync resources into their own clearly marked folder, that way you will treat the presync process as its own application.

Get notified when something goes wrong

If something goes wrong during the migration, you’ll want to know about it right away. ArgoCD has a built-in notifications feature that can send alerts when things like a migration failure happen.

You can set up ArgoCD notifications to send alerts via Slack, email, or other messaging platforms. This gives you real-time updates on the status of your deployments, so you can jump in and fix any issues if the PreSync job fails.

You can follow the Triggers documentation on how to handle this: here.

With this setup, you’ll get notified immediately if a migration fails, allowing you to take action and resolve the issue before the application is deployed.