GitOps

From Canasta Wiki

Canasta supports git-based configuration management through the canasta gitops command group. Your instance's configuration files are stored in a private Git repository with encrypted secrets, providing change history, easy rollback, and optional multi-server deployments.

Overview

In the simplest case, a single server uses gitops purely for version-controlled configuration backup β€” every change is committed and pushed to a remote repository. No pull requests, no multi-server coordination β€” just canasta gitops add and canasta gitops push after making changes.

The same architecture extends to multi-server deployments. Changes are made on a source server, tested, pushed to the repo with an optional pull request for peer review, and then pulled onto production servers.

ℹ️ Note: Gitops manages configuration only β€” it does not back up databases or uploaded files. Use canasta backup separately for that.

Prerequisites

What has to be installed depends on the orchestrator, because the two store secrets in completely different ways.

Docker Compose

  • git-crypt β€” transparent encryption of the secrets that live in the repo. Install it with canasta install git-crypt (add -H <host> for a remote target), or with your package manager: brew install git-crypt, sudo apt install git-crypt, sudo dnf install git-crypt.
  • Optional: GnuPG (gpg) on the target host, only if you unlock the repo with GPG keys instead of a shared key file β€” see Key management below. Canasta does not install or check for it.

Kubernetes

  • git-crypt is not used. Argo CD cannot decrypt a git-crypt repo, so a Kubernetes gitops repo is cleartext and carries no secrets β€” they live in Kubernetes Secrets in the cluster.
  • Optional: to commit encrypted Secrets to the repo anyway, pass --encrypt-secrets to canasta gitops init and install the SOPS toolchain on both the controller and the target host β€” see Kubernetes: encrypted secrets with SOPS below.
canasta install sops              # controller (where the CLI runs)
canasta install -H node1 sops     # the gitops target host

Both orchestrators

  • gh (GitHub CLI) β€” only required when pull_requests: true is set in hosts.yaml
  • SSH access to the gitops repository β€” the host needs an SSH key with push permission to the repo. If git is not already configured on this host, see Set up a deploy key below.

canasta gitops init checks for the tools above and provides instructions if any are missing. It does not verify SSH access to the gitops repository β€” if that is not configured, the initial push will fail with Permission denied (publickey).

What gets tracked

The git repository contains:

  • Configuration files β€” config/ directory (Caddyfile customizations, PHP settings)
  • Environment template β€” env.template with {{placeholders}} for host-specific values
  • Wiki farm template β€” wikis.yaml.template with {{wiki_url_<id>}} placeholders for host-specific wiki URLs
  • Other templates β€” my.cnf.template and media-hosts.yaml.template, when the instance uses them
  • Per-host variables β€” hosts/{name}/vars.yaml with secrets and host-specific values (encrypted by git-crypt)
  • Shared variables β€” hosts/_shared/vars.yaml with the account-wide credentials every host inherits (encrypted by git-crypt)
  • Host inventory β€” hosts/hosts.yaml defining all servers and their roles
  • Extensions and skins β€” tracked as git submodules pinned to specific versions, or as regular files for custom extensions without their own repository
  • Orchestrator overrides β€” docker-compose.override.yml and any Dockerfile* at the top of the instance directory (if present)
  • Scripts β€” scripts/ directory (if present)
  • Public assets β€” public_assets/ directory (logos, favicons)

Which paths are committed automatically

Commits that Canasta makes without you choosing the files stage an explicit list of paths rather than everything in the instance directory, so stray files β€” a scratch file, a database dump, an editor's backup copy of .env β€” never reach the remote. Within those paths, new, changed, and deleted files are all staged, and .gitignore still applies.

Orchestrator When Paths staged
Docker Compose The initial commit made by canasta gitops init .gitignore, .gitattributes, .gitmodules, env.template, wikis.yaml.template, my.cnf.template, media-hosts.yaml.template, docker-compose.override.yml, Dockerfile* (top level only), hosts/, config/, extensions/, skins/, public_assets/, scripts/
Kubernetes The initial commit made by canasta gitops init, and every canasta gitops push .gitignore, .gitmodules, .sops.yaml, Chart.yaml, templates/, values.yaml, values.template.yaml, argocd/, hosts/, config/, extensions/, skins/, public_assets/

On Docker Compose, canasta gitops push commits only what you staged with canasta gitops add (plus commits other Canasta commands already made), so after the initial commit the list above no longer matters there.

To track anything outside these paths β€” for example a custom/ directory of your own β€” stage it explicitly and push:

canasta gitops add custom/
canasta gitops push -m "Track custom/ directory"

Once a file is tracked, later changes to it still have to be staged with canasta gitops add on both orchestrators; the Kubernetes push only stages the paths in the table on its own.

What is NOT tracked (gitignored)

  • .env β€” generated from env.template + vars at deploy time
  • config/wikis.yaml β€” generated from wikis.yaml.template + vars at deploy time
  • admin-password_* β€” generated from vars at deploy time
  • docker-compose.yml β€” managed by the Canasta CLI
  • docker-compose.edge.yml: generated on each start, joining the HAProxy edge (Compose only)
  • docker-compose.sidecars.yml: rendered on each start from config/sidecars.yaml (Compose only)
  • config/Caddyfile β€” auto-generated from wikis.yaml on restart
  • config/backup/ β€” database dumps created by canasta backup
  • images/ β€” uploaded files (covered by canasta backup)
  • Editor and tooling backup files β€” *.bak, *.bak.*, *~, *.swp, *.swo, and .#*

canasta upgrade adds any ignore rules shipped by a newer CLI to an existing instance's .gitignore, keeping lines you added yourself.

Repository structure

The tree below shows the full directory layout of a gitops-managed instance. Most files and directories are part of every Canasta instance. Files marked † are created by gitops init and only exist when gitops is in use. Files marked ‑ are not pushed to the gitops repository β€” they are either generated locally from templates, managed by the CLI, or excluded because they contain host-specific data.

The tree below is the Docker Compose layout. A Kubernetes repo is organized around the Helm chart instead β€” values.template.yaml, Chart.yaml, templates/, hosts/hosts.yaml, per-host hosts/{name}/vars.yaml plus the shared hosts/_shared/vars.yaml, and rendered-values.yaml β€” with no .gitattributes, and (with --encrypt-secrets) a .sops.yaml plus encrypted hosts/{name}/secrets/*.enc.yaml.

canasta-config/
β”œβ”€β”€ .gitattributes †
β”œβ”€β”€ .gitignore †
β”œβ”€β”€ .git-crypt/ †               # GPG mode only: the git-crypt key, encrypted to each GPG recipient
β”œβ”€β”€ .env ‑                      # rendered from env.template + vars
β”œβ”€β”€ env.template †              # .env template with {{placeholders}}
β”œβ”€β”€ wikis.yaml.template †       # wikis.yaml template (wiki farms only)
β”œβ”€β”€ hosts/ †                    # host inventory, per-host and shared variables
β”‚   β”œβ”€β”€ hosts.yaml †            # host inventory and settings
β”‚   β”œβ”€β”€ _shared/ †
β”‚   β”‚   └── vars.yaml †         # account-wide credentials, encrypted
β”‚   └── myserver/ †
β”‚       └── vars.yaml †         # encrypted by git-crypt
β”œβ”€β”€ admin-password_* ‑      # rendered from vars
β”œβ”€β”€ config/
β”‚   β”œβ”€β”€ wikis.yaml ‑            # rendered from template + vars
β”‚   β”œβ”€β”€ Caddyfile ‑             # generated on restart
β”‚   β”œβ”€β”€ backup/ ‑               # database dumps
β”‚   β”œβ”€β”€ Caddyfile.site
β”‚   β”œβ”€β”€ Caddyfile.global
β”‚   └── settings/
β”‚       β”œβ”€β”€ global/
β”‚       β”‚   └── *.php
β”‚       └── wikis/
β”‚           └── {wiki-id}/
β”‚               └── *.php
β”œβ”€β”€ custom/                     # your own files; tracked only once staged with gitops add
β”œβ”€β”€ extensions/                 # git submodules (or regular files for custom extensions)
β”œβ”€β”€ skins/                      # git submodules (or regular files for custom skins)
β”œβ”€β”€ public_assets/
β”œβ”€β”€ images/ ‑                   # uploaded files
β”œβ”€β”€ docker-compose.yml ‑        # managed by CLI
└── docker-compose.override.yml # if used

Initial setup

Gitops works with any existing Canasta instance β€” single wiki or wiki farm. You don't need to reinstall or recreate anything. The init command examines your running instance and builds the gitops repository around it.

All gitops commands accept -i <id> to specify the instance by its Canasta ID. If omitted, the command uses the current working directory.

1. Verify your instance

Make sure your instance is working and that .env, config/, and any extensions/skins are in their final state. Gitops will snapshot the current configuration as its starting point.

If you have a wiki farm, ensure config/wikis.yaml and all per-wiki settings under config/settings/ are in place. Admin passwords (admin-password_*) are automatically captured into the encrypted per-host vars.

If you don't have an instance yet:

canasta create -i mywiki -w main -n wiki.example.com

2. Check for custom secrets

Before initializing gitops, review your .env file for any secrets or host-specific values beyond the built-in set. Gitops does not work from a fixed list of key names β€” it classifies every key and extracts anything it judges secret into encrypted variables. A key counts as a secret when any of the following is true:

  • Its name contains PASSWORD, SECRET, TOKEN, KEY, or CREDENTIAL. This covers MYSQL_PASSWORD, WIKI_DB_PASSWORD, MW_SECRET_KEY, and RESTIC_PASSWORD.
  • Its name contains PASS, PASSWD, PWD, DSN, or PRIVATE as a whole underscore-separated word, such as DB_PASS or SENTRY_DSN (Canasta CLI 4.21.0 or later).
  • Its name starts with a credential prefix: AWS_, AZURE_, B2_, GOOGLE_, OS_, ST_, RCLONE_, or SMTP_.
  • It is RESTIC_REPOSITORY, which is listed explicitly because a repository URL can embed credentials.

A second group of keys is host-specific but not secret. Gitops templates these per host and leaves them readable: MW_SITE_SERVER, MW_SITE_FQDN, HTTP_PORT, HTTPS_PORT, TLS_MODE, TRUSTED_PROXIES, ACME_DNS_PROVIDER, CLOUDFLARE_ZONE_ID, and the CANASTA_ENABLE_* feature flags. CLOUDFLARE_API_TOKEN is a secret by its name.

Where extracted values live

Most extracted values go to the pushing host's own hosts/{name}/vars.yaml. A small set of account-wide credentials β€” AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, SMTP_USER, SMTP_PASSWORD β€” is migrated to a shared hosts/_shared/vars.yaml on push instead, so that every host in the repo inherits it.

Values identifying which infrastructure a host talks to stay per host even when they are secret, so dev, staging, and production can address different backends under one repo. RESTIC_REPOSITORY, RESTIC_PASSWORD, and AWS_BUCKET_NAME are per host for this reason. Where a key appears in both files, the host's value wins.

Custom keys the rules do not match

Any key matching neither group above is committed as a literal value in env.template, which is not encrypted. Most custom secrets are matched with no action from you β€” MY_API_KEY qualifies on KEY, and any SMTP_* key on its prefix.

For a genuine secret whose name carries no such hint (say MYSERVICE_USERNAME), set it with canasta config set --secret. On Docker Compose the value goes into .env and the key name is recorded in config/secret-keys, so gitops treats the key as a secret whatever it is called: env.template gets a placeholder and the value is stored in this host's encrypted hosts/{name}/vars.yaml. On Kubernetes the value goes into the gitignored config/secrets.env, which feeds a per-instance Secret. Either way the value belongs to this host and is not shared with other hosts, so set it on each host. Add --web when MediaWiki itself needs the value.

canasta config set --secret MYSERVICE_USERNAME=...

A sidecar receives a secret by referencing it as ${KEY} in its env (see Sidecars), and --web passes it to the web container, so a secret needs no override file.

A custom variable that is not secret needs nothing special. canasta config set rejects a key it does not recognize, so pass --force; the value then travels in env.template like any other:

canasta config set --force VENDOR_ACCOUNT_ID=...

To pass such a variable into the web container, add a docker-compose.override.yml file:

services:
  web:
    environment:
      - VENDOR_ACCOUNT_ID

If you don't have any custom secrets beyond the built-in set, you can skip this step.

3. Set up a deploy key

Skip this step if SSH access to the gitops repository is already working from this host β€” either because the user has a personal account key registered with the repo on github.com, or because a deploy key is already installed at a default path like ~/.ssh/id_ed25519 or ~/.ssh/id_rsa. To check:

ssh -T git@github.com

A successful greeting from github.com means this step is unnecessary. Permission denied (publickey) means SSH is not yet wired up β€” continue below.

Generate an ed25519 keypair on the host:

ssh-keygen -t ed25519 -C "canasta-gitops@$(hostname)" -f ~/.ssh/gitops_deploy -N ''

The empty passphrase (-N ) is required β€” canasta gitops init and canasta gitops push run unattended and cannot answer a key passphrase prompt.

Display the public key and copy it:

cat ~/.ssh/gitops_deploy.pub

On github.com, open the gitops repository's Settings β†’ Deploy keys β†’ Add deploy key. Paste the public key, use the hostname as the title, and tick "Allow write access" β€” without write access, the initial push performed by canasta gitops init will fail with Permission denied.

Tell SSH to use this key for github.com by adding the following block to ~/.ssh/config (create the file with mode 0600 if it does not exist):

Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/gitops_deploy
  IdentitiesOnly yes

For GitLab, Bitbucket, or other forges, replace both occurrences of github.com with the forge's hostname.

Verify before continuing:

ssh -T git@github.com

You should see a greeting addressed to your repository. If the message says you have read-only access, fix the deploy key permission on github.com before continuing.

ℹ️ Note: Compose-orchestrated gitops uses plain ssh for the initial push, so the ~/.ssh/config block above is required whenever the deploy key is stored at a non-default path. (Kubernetes-orchestrated gitops generates and uses its own deploy key via GIT_SSH_COMMAND and does not consult ~/.ssh/config. Once the instance is set up, canasta gitops push and canasta gitops pull authenticate with the copy of that key staged in the instance directory, so they do not depend on an SSH agent on the host; pass --ssh-key to use a different key, which is also used to fetch submodules.)

ℹ️ Note: Self-hosted and non-forge repositories: the gitops repo does not have to live on a public forge β€” a bare repository on your own server works too. There is no ssh -T greeting to test against in that case, but canasta gitops init and join still work. On Kubernetes, when the repository is on a self-hosted host Canasta adds that host's key to Argo CD's known-hosts and restarts the Argo CD repo-server, so Argo CD can sync from it instead of rejecting it as an unknown host. On Kubernetes, gitops init first checks whether the deploy key already works. If it does, init doesn't stop; if it doesn't, init shows the public key and pauses until you press Enter after adding it, or, with --non-interactive, fails right away with the key and instructions instead of waiting, so it can be driven from automation. gitops join doesn't pause.

4. Initialize gitops

canasta gitops init -n myserver --repo git@github.com:yourorg/mywiki-config.git --key /path/to/gitops-key

To require pull requests for all changes instead of pushing directly to main:

canasta gitops init -n myserver --repo git@github.com:yourorg/mywiki-config.git --key /path/to/gitops-key --pull-requests

The remote repository must be empty (no commits, no README). Create an empty repository on GitHub/GitLab first, then pass its URL with --repo. What --key means depends on the orchestrator: on Docker Compose it is where the git-crypt symmetric key is exported (nothing is written there in GPG mode, though the flag is still required); on Kubernetes it is the path basename of the SSH deploy key (<key>.ssh / <key>.ssh.pub), and with --encrypt-secrets it also receives a portable copy of the SOPS operator age key at <key>.age.

If git is not configured with a user identity on this host β€” i.e., git config --global user.name returns nothing β€” pass --git-user-name and --git-user-email so the gitops commits made by canasta gitops init have an author:

canasta gitops init -n myserver --repo git@github.com:yourorg/mywiki-config.git --key /path/to/gitops-key \
  --git-user-name "Your Name" --git-user-email "you@example.com"

These flags run git config --global user.name and user.email on the host, so they only need to be passed once. Subsequent gitops commands reuse the configured identity. If git is already configured, the flags are unnecessary.

This bootstraps a new gitops repository from the existing instance:

  1. Initializes a git repo in the instance directory
  2. Sets up .gitignore and .gitattributes
  3. Initializes git-crypt and exports the symmetric key to --key β€” or, with --git-crypt-gpg-user, encrypts the key to that GPG recipient instead (see Key management)
  4. Creates env.template by extracting the current .env and replacing host-specific values with {{placeholders}}
  5. Creates wikis.yaml.template by extracting wiki URLs from config/wikis.yaml and replacing them with {{wiki_url_<id>}} placeholders
  6. Creates hosts.yaml with this server as the first entry
  7. Creates hosts/myserver/vars.yaml with the actual values extracted from .env, wiki URLs, and admin password files
  8. Converts user-installed extensions and skins to git submodules
  9. Makes an initial commit of the paths listed in Which paths are committed automatically
  10. Pushes to the remote

Store the exported git-crypt key securely β€” it is needed to unlock the repo on other servers and must never be committed to the repo. (In GPG mode there is no key file to store; the GPG private key of a recipient unlocks the repo instead.)

ℹ️ Note: canasta upgrade keeps these gitops artifacts current on an already-initialized instance: it refreshes the instance's .gitignore from the CLI's shipped defaults, and for older gitops instances created before wikis.yaml.template existed it backfills that template so wiki-URL templating works.

Repairing broken submodules

If canasta gitops init fails partway through (e.g., due to push errors or missing credentials), extensions may be left in a broken state β€” committed as regular directories instead of proper submodules. Running init again is blocked because the .git directory already exists.

Use canasta gitops fix-submodules to repair submodule registration without re-initializing:

canasta gitops fix-submodules

This reads .gitmodules, detects extensions that were committed as regular trees instead of submodules, and re-registers them properly. It is also useful when adding a new extension to an existing gitops repo β€” if the extension was cloned manually, fix-submodules converts it to a proper submodule. After repairing, push the fix with canasta gitops push.

Environment template and variables

The env.template is the single source of truth for what configuration exists. Host-specific values (secrets, domain names, ports) are replaced with {{placeholders}}:

MW_SITE_SERVER={{mw_site_server}}
MW_SITE_FQDN={{mw_site_fqdn}}
MYSQL_PASSWORD={{mysql_password}}
MW_SECRET_KEY={{mw_secret_key}}
MW_DB_NAME=mediawiki
MW_SITE_NAME=My Wiki

Each host's vars.yaml supplies the actual values:

# hosts/myserver/vars.yaml
mw_site_fqdn: wiki.example.com
mysql_password: "my-db-pass"
mw_secret_key: "abc123..."
admin_password_main: "my-admin-pass"

At deploy time, canasta gitops pull renders the template with the host's vars to produce .env, and writes admin-password_* files from the corresponding vars.

ℹ️ Note: The .env file is rendered from env.template + vars.yaml, but on Docker Compose an edit made directly to .env is not lost. canasta reconcile and canasta gitops pull first write it back into its source the way canasta config set does β€” into env.template, or into hosts/{name}/vars.yaml for a host-specific or secret key β€” and stage the change. A pull then refuses to run until you push it with canasta gitops push (or discard it). In the other direction, run canasta reconcile to apply an edit you made to env.template or hosts/{name}/vars.yaml on this host. If the same key was changed in both .env and its source, to different values, the command stops without changing either.

Database account keys. As of Canasta CLI 4.22.0, a Docker Compose .env also holds WIKI_DB_USER and WIKI_DB_PASSWORD, the account MediaWiki connects with (see Database accounts). WIKI_DB_PASSWORD is a secret, so it becomes a per-host encrypted variable like MYSQL_PASSWORD. When canasta upgrade moves an instance onto MediaWiki's own account, it records both keys in this host's vars; run canasta gitops push afterwards so the next pull keeps them.

On a Kubernetes instance (as of Canasta CLI 4.25.2), this host's hosts/{name}/vars.yaml records the account as wiki_db_user instead, and its password stays in the instance's db Secret. A host whose vars do not name an account keeps MediaWiki on root. canasta upgrade stages the move for the next push and sync, and canasta gitops push refuses to publish values that select the account while the db Secret has no WIKI_DB_PASSWORD key; see Upgrading.

Wiki URL template

For wiki farms, config/wikis.yaml contains per-wiki URLs that differ between hosts (e.g., production.example.com vs localhost). The wikis.yaml.template works the same way as env.template β€” wiki URLs are replaced with {{wiki_url_<id>}} placeholders:

# wikis.yaml.template
wikis:
- id: main
  url: "<nowiki>{{</nowiki>wiki_url_main<nowiki>}}</nowiki>"
  name: Main Wiki
- id: docs
  url: "<nowiki>{{</nowiki>wiki_url_docs<nowiki>}}</nowiki>"
  name: Documentation

Each host's vars.yaml supplies the actual URLs:

# hosts/production/vars.yaml (in addition to .env vars)
wiki_url_main: production.example.com
wiki_url_docs: production.example.com/docs
# hosts/devbox/vars.yaml
wiki_url_main: localhost
wiki_url_docs: localhost/docs

At deploy time, canasta gitops pull renders wikis.yaml.template with the host's vars to produce config/wikis.yaml. This prevents local development from accidentally overwriting production URLs when pushing configuration changes.

ℹ️ Note: Like .env, config/wikis.yaml is rendered from wikis.yaml.template + vars.yaml, and on Docker Compose a direct edit to it is written back rather than overwritten: canasta reconcile and canasta gitops pull capture an edited url into this host's wiki_url_<id> in hosts/{name}/vars.yaml, and any other edit into wikis.yaml.template. As with .env, a pull then refuses until the change is pushed, and canasta reconcile applies edits made to the template or the host's vars on this host. To add or remove wikis, use canasta add and canasta remove, which update wikis.yaml.template for you.

A pull that stopped partway. A pull that fetched a commit but failed before applying it leaves .env and config/wikis.yaml holding the previous values. Until a pull applies that commit, the write-back above is skipped, so the old values are not mistaken for local edits. canasta gitops status reports that the last pull stopped before applying the commit, and canasta gitops add config/wikis.yaml refuses. Fix what stopped the pull, then run canasta gitops pull again.

To capture an edit you made directly in config/wikis.yaml β€” for example renaming a wiki's display name β€” run canasta gitops add config/wikis.yaml. Rather than tracking the rendered file, this reconciles your edit back into wikis.yaml.template while preserving the URL placeholders, so the change survives the next render and reaches the other hosts.

Adding and removing wikis. As of Canasta CLI 4.22.0, canasta add records the new wiki in wikis.yaml.template and its URL as wiki_url_<id> in this host's vars.yaml, and stages both along with the wiki's tracked files. canasta remove takes the wiki out of the template, drops its URL from this host's vars, and stages the deletion of its tracked files. Run canasta gitops push to share either change. On Kubernetes, neither command deploys the change itself: add creates the new wiki's database and remove drops the wiki's databases right away, but the configuration change reaches the cluster only through canasta gitops push and canasta gitops sync (see Applying changes on Kubernetes).

Another host that pulls a newly added wiki needs its own wiki_url_<id> for it. Until that variable is in the host's vars.yaml, the pull refuses to write config/wikis.yaml and names the missing variable.

Gitops carries configuration, not databases. On Docker Compose, after a pull the CLI prints a WARNING for each wiki whose database is not on this host's database server. Give the wiki its database with canasta import -w <wiki> -d <dump>, or with canasta backup restore -s <snapshot> -w <wiki>, which also replaces that wiki's settings and files with the snapshot's.

Built-in placeholder keys

For the rules deciding which .env keys are converted to placeholders (secrets, backup credentials, host-specific values), see Check for custom secrets. A custom secret the rules do not match is best set with canasta config set --secret, which marks the key as a secret whatever its name.

Host inventory

The hosts.yaml file defines the deployment targets:

Single-server (simplest case):

canasta_id: mywiki
hosts:
  myserver:
    role: both

When there is only one host, the role defaults to both and can be omitted.

Multi-server with pull requests:

canasta_id: mywiki
pull_requests: true
hosts:
  staging:
    role: source
  production:
    role: sink

Roles

Each host has a role that controls the direction of git flow:

Role Can push Can pull Use case
source Yes No Staging, dev β€” where changes originate
sink No Yes Production β€” receives config from the repo
both Yes Yes Single server, or dual-purpose server

Roles act as a safety guardrail β€” a sink host will refuse to push, preventing accidental commits of local drift on production.

Pull requests setting

The pull_requests setting controls how canasta gitops push behaves:

  • false (default) β€” commits push directly to main. Good for single-server setups or small teams.
  • true β€” push creates a branch and opens a pull request for review. Requires the gh CLI. If the push or the pull request fails, the checkout is switched back to main, so later pulls and pushes are not left on the branch.

Common operations

Changing a setting

  1. Edit the settings file
  2. Test the change
  3. Stage and push:
canasta gitops add config/settings/global/MySettings.php
canasta gitops push -m "Enable VisualEditor by default"

ℹ️ Note: canasta gitops add requires at least one file path β€” a bare canasta gitops add with no arguments is rejected β€” and accepts several at once, e.g. canasta gitops add config/settings/global/MySettings.php config/default.vcl. Staging config/wikis.yaml is handled specially; see #Wiki URL template.

Removing a file

To remove a tracked file from the repository:

canasta gitops rm config/settings/global/OldSettings.php
canasta gitops push -m "Remove obsolete settings file"

If pull requests are enabled, review and merge the PR. Then on sink hosts:

canasta gitops pull

Checking status

canasta gitops status

Shows the current host, role, commit info, staged and unstaged changes, untracked files, and ahead/behind remote status. It also flags edits made directly in config/wikis.yaml that have not yet been captured into wikis.yaml.template (see #Wiki URL template), so a per-wiki change such as a display-name edit isn't silently dropped on the next render.

On Kubernetes, canasta gitops push also renders gitignored local files β€” such as .env, config/wikis.yaml, the Caddyfile and config/default.vcl β€” into the values Argo CD deploys. Git cannot see an edit to those files until a push renders it, so status compares them with the deployed values and lists each one that differs under Changes the next push will render. A canasta config set or canasta backup databases add shows up there, for example.

Previewing changes before pulling

canasta gitops diff

Fetches without applying and shows what would change, organized into three sections: local changes (not yet pushed), remote changes (would be applied on pull), and potential conflicts (changed in both). Restart and maintenance hints are based on remote changes only.

Resolving conflicts after a pull

When this host's commits and the remote's change the same lines, canasta gitops pull cannot replay them. It rolls back, leaves the instance exactly as it was, and lists each conflicting file. To resolve them:

canasta gitops resolve

This replays the pull and stops at the conflicts, listing each file by what it needs:

  • A text file: edit out the conflict markers (<<<<<<<, =======, >>>>>>>). An encrypted file under hosts/ that git-crypt's merge driver merged has ordinary markers too, and is committed encrypted.
  • A binary file, or an encrypted file the merge driver could not merge: keep one version whole, this host's (--keep local <file>) or the remote's (--keep remote <file>).

Then finish with canasta gitops resolve --continue. It stops again if a later local commit also conflicts, and otherwise applies the result as gitops pull does, reporting whether a restart is needed. canasta gitops resolve --abort goes back to where the pull started, with this host's commits intact. Until the resolution finishes or is abandoned, commands that read or write the instance's configuration refuse to run.

Applying changes on Kubernetes

How a pushed change reaches a running instance differs by orchestrator:

  • Docker Compose has no continuous reconciler, so each sink host applies repository changes by pulling them β€” canasta gitops pull renders the templates, updates the tracked files, and reports whether a restart or maintenance update is needed.
  • Kubernetes is reconciled by Argo CD, which watches the gitops repository and applies changes to the cluster on its own. After canasta gitops push the change is live in Git immediately, but it only reaches the cluster at Argo CD's next sync interval. On Kubernetes, canasta gitops pull just re-renders the per-host values locally β€” it does not apply them to the cluster; that is Argo CD's job.

To apply a just-pushed change immediately rather than waiting for that interval, force a reconciliation:

canasta gitops sync

This triggers Argo CD to refresh and reconcile the instance now, and waits until the Application reports Synced and Healthy; if it times out, it fails with the last sync and health state Argo CD reported. Use it during a deploy, a hotfix, or while troubleshooting. canasta gitops sync is Kubernetes-only; on Docker Compose it has no effect, because there is no reconciler to trigger β€” there, canasta gitops pull is what applies a change.

Commands that stage instead of deploying

Argo CD deploys a gitops-managed Kubernetes instance from its repository and self-heals back to it, so a change deployed directly to the cluster would be reverted until it was pushed. As of Canasta CLI 4.25.0, these commands therefore stage their change in the gitops repository and do not deploy it:

  • canasta upgrade β€” stages the refreshed chart and image tag, along with the move to MediaWiki's own database account. An instance built from source still has its image rebuilt, and its web pods restart when the new build differs from the image they run.
  • canasta add and canasta remove β€” create or drop the wiki's databases right away and stage the configuration change. The web pods run update.php for an added wiki when the sync restarts them.
  • canasta config set and canasta config unset β€” save the change without restarting the instance, with or without --no-restart.
  • canasta extension enable and disable, and canasta skin enable and disable (as of Canasta CLI 4.25.1) β€” commit the settings change without deploying it. enable does not run update.php itself; the web pods run it for the affected wikis when the sync restarts them, even with --skip-update.

Each one ends by naming the commands that deploy the change:

canasta gitops push -i mywiki
canasta gitops sync -i mywiki

Docker Compose instances, and Kubernetes instances that are not gitops-managed, are unaffected; these commands still restart them.

Stopped instances. A stopped instance is left stopped. Run canasta gitops push; the change deploys the next time the instance starts. (canasta remove is the exception: it refuses on a stopped instance, because it drops the wiki's databases from inside the running instance.) canasta gitops sync on a stopped instance says that it is stopped, syncs nothing, and exits successfully.

Starting and restarting. canasta start, canasta restart and canasta reconcile still deploy the instance's local state directly, and Argo CD then puts back the state that is committed and pushed. As of Canasta CLI 4.25.1, when the local state holds changes that are not pushed β€” staged or unstaged edits, local commits, or inputs the next push would render β€” these commands print a warning that names them and the canasta gitops push and canasta gitops sync commands that keep them, then continue. A restart warns before it stops anything.

Changed secret values. Pods do not restart on their own when a Secret they read changes. canasta gitops push therefore records a hash of the Secret values each workload reads β€” the web pods, Caddy (for the CrowdSec bouncer key and agent password), the CrowdSec local API, and each sidecar β€” so after a secret changes, for example with canasta config set --secret, the sync rolls only the pods that read it, and an unchanged value rolls nothing. Only hashes are written, never the values. On an instance with SOPS-encrypted secrets, the push also commits the changed value to the repository.

Enabling CrowdSec takes two more steps after the sync; see CrowdSec.

Argo CD project. Each instance's Argo CD Application belongs to its own AppProject, canasta-<id>, which admits only the instance's gitops repository as a source and only its canasta-<id> namespace as a destination, with no cluster-scoped resources. canasta gitops init and canasta gitops join create it. canasta upgrade moves an existing instance's Application out of Argo CD's default project into its own without redeploying anything, and stages the updated Application manifest for a push. An Application you moved to another project yourself is left alone. canasta delete removes the project.

CLI versions. Run the same Canasta CLI version on every host that shares a gitops repository. A push from an older CLI version clears the per-workload secret checksums that the newer version commits, so the pods roll back and forth with each host's push.

Extensions and skins

Extensions and skins that have their own git repositories are tracked as git submodules, pinning each one to an exact commit. This ensures every server runs the same version and makes updates explicit and reviewable.

How init handles extensions and skins

During canasta gitops init, the CLI scans the extensions/ and skins/ directories. Each subdirectory that contains a .git folder is automatically converted to a git submodule using its origin remote URL. The original directory is removed and re-added via git submodule add.

Subdirectories that are not git repositories β€” such as custom extensions you wrote yourself or copied in without cloning β€” are left as regular files and committed directly to the repo. They are tracked like any other file, not as submodules. See Custom extensions below.

If a directory has a .git folder but no origin remote configured, it is skipped with a warning.

Adding a new extension or skin

On Docker Compose, canasta extension add and canasta skin add (Canasta CLI 4.21.0 or later) do this in one step: they add the extension or skin as a submodule, register any composer requirements, enable it, and commit the result to the local repository:

canasta extension add DarkMode --clone
canasta gitops push

See Adding with canasta extension add for how the repository URL and branch are chosen.

To add the submodule by hand instead:

git submodule add https://github.com/wikimedia/mediawiki-extensions-Cite.git extensions/Cite

Then enable it in the appropriate settings file (e.g., config/settings/global/extensions.php), test, stage, and push:

canasta gitops add extensions/Cite config/settings/global/extensions.php
canasta gitops push -m "Add Cite extension"

On sink hosts after pulling, run canasta restart -i mywiki and then canasta maintenance update -i mywiki to run any database migrations.

Enabling and disabling

On a gitops instance, canasta extension enable, canasta extension disable, canasta skin enable, and canasta skin disable commit the settings.yaml they change (global, or the wiki's own with --wiki) to the local repository. The commit contains only that file β€” anything else you have staged is left for its own commit β€” and nothing is committed when the file did not change. Run canasta gitops push to distribute it. On Kubernetes the change is not deployed until the push and a sync; see Commands that stage instead of deploying:

canasta extension enable VisualEditor
canasta gitops push

Updating an extension or skin

For an extension, canasta extension set-version moves the submodule and stages the change in one step:

canasta extension set-version MyExtension --ref v2.0.0
canasta gitops push -m "Update MyExtension to v2.0.0"

--ref takes a branch, tag, or commit SHA β€” pass the release branch matching your MediaWiki version (REL1_43) rather than master. Add --run-update if the new version changes the database schema. See Pinning a user extension to a version for the full behavior.

Skins have no equivalent command, so they are still moved by hand inside the submodule:

cd skins/MySkin
git fetch && git checkout v2.0.0
cd ../..
canasta gitops add skins/MySkin
canasta gitops push -m "Update MySkin to v2.0.0"

Either way the submodule reference in the parent repo now points at the new commit. On sink hosts after pulling, run canasta maintenance update if the update has schema changes.

Removing an extension or skin

On Docker Compose, canasta extension remove and canasta skin remove disable the extension or skin in every settings.yaml, unregister and delete the submodule, unregister its composer requirements, and commit the removal. Run canasta gitops push afterward to distribute it:

canasta extension remove MyExtension
canasta gitops push

To remove the submodule by hand instead:

git submodule deinit -f extensions/MyExtension
git rm -f extensions/MyExtension

Remove the wfLoadExtension or wfLoadSkin call from the settings file, then push.

On Docker Compose, when another host pulls the removal, canasta gitops pull deletes that host's checkout of the submodule too (see below).

Submodule initialization on new servers

When a new server joins the gitops repo via canasta gitops join, or when an existing server runs canasta gitops pull, the CLI runs git submodule update --init --recursive. This clones all submodule repositories and checks out the exact commits recorded in the repo.

This is necessary because git does not automatically clone submodules when cloning or pulling a repository β€” the submodule directories would otherwise be empty.

The reverse also holds: git drops a removed submodule from the repository but leaves its checkout on disk. On Docker Compose, canasta gitops pull deletes the checkout of each submodule the pull removed, and reports what it deleted. A checkout with local changes, or one that is not a git checkout, is kept instead, with a warning naming it β€” delete it yourself once anything in it you need is saved elsewhere. (Until it is deleted, it shows up as untracked in canasta gitops status.)

Converting a cloned extension to a submodule

If you cloned an extension directly with git clone rather than adding it as a submodule, you need to convert it before gitops can track it properly. If you haven't run canasta gitops init yet, the init command handles this automatically. If gitops is already initialized:

# Note the remote URL and current commit
cd extensions/MyExtension
git remote get-url origin
git rev-parse HEAD
cd ../..

# Remove the cloned directory and re-add as a submodule
rm -rf extensions/MyExtension
git submodule add https://github.com/org/MyExtension.git extensions/MyExtension

# Check out the same commit you were on
cd extensions/MyExtension
git checkout <commit-hash>
cd ../..

canasta gitops add extensions/MyExtension .gitmodules
canasta gitops push -m "Convert MyExtension to submodule"

Custom extensions without a git repo

Some extensions are custom code that doesn't live in a separate git repository β€” for example, a small extension you wrote specifically for your wiki. These are committed directly to the gitops repo as regular files, not submodules.

Since they are regular files in the repo, they are automatically synced to all servers on canasta gitops pull without any submodule commands. However, they cannot be independently versioned or pinned to a specific commit the way submodules can.

Adding a server

Backup and gitops are complementary systems. Backup (canasta backup) captures everything β€” databases, uploaded files, and configuration. Gitops tracks configuration only β€” settings files, extensions, skins, environment template, and wiki farm structure. When adding a server (whether a production replica or a local dev copy), you need both: the backup provides the database and files, while gitops gives you the shared configuration repo.

To add a new server to an existing managed wiki farm:

1. Back up the existing wiki farm

On the existing server:

canasta backup create

2. Create and restore on the new server

canasta create -i mywiki -w main -n production.example.com
canasta backup restore -s <snapshot-id>

This ensures the new server has all wikis and their databases. The restore gives the new server its own generated passwords (in .env and admin-password_*), which are then captured as that host's vars during the join step below.

3. Set any custom environment variables

Per-host values do not arrive with the join. A value provisioned with canasta config set --secret belongs to the host it was set on β€” on Docker Compose it is stored in that host's encrypted hosts/{name}/vars.yaml, and on Kubernetes in the gitignored config/secrets.env β€” so set it again on the new server before joining:

canasta config set --secret MYSERVICE_USERNAME=...

4. Join the gitops repo

canasta gitops join -n production --repo git@github.com:yourorg/mywiki-config.git --key /path/to/gitops-key

On Docker Compose, --key is the git-crypt key file on the controller, as exported by gitops init. For a repo set up in GPG mode you can leave --key out; the join then unlocks git-crypt through GPG on the new server:

canasta gitops join -n production --repo git@github.com:yourorg/mywiki-config.git

Leaving out --key for a repo without GPG recipients fails with a message asking for it. On Kubernetes, --key is always required.

This clones the repo, unlocks git-crypt, adds the host to hosts/hosts.yaml, extracts host-specific values (including wiki URLs) into vars.yaml, renders .env and config/wikis.yaml from templates, updates submodules, and pushes the new host entry back to the repo.

ℹ️ Note: Backup credentials are not inherited. RESTIC_PASSWORD unlocks the repository named by the host's own RESTIC_REPOSITORY, so it stays with the host that generated it rather than being shared through the repo. If every host here backs up to one restic repository, set the joining host's password to match it β€” otherwise its backups fail against a repository it cannot open:

canasta config set -i mywiki RESTIC_PASSWORD=...

Setting up a local dev copy

The steps above also apply when setting up a local development copy of a production wiki. The key differences are:

  1. Restore from a remote backup instead of copying a file. Add the production server's backup credentials (RESTIC_REPOSITORY, RESTIC_PASSWORD, and any AWS_*/AZURE_*/B2_* keys) to your local instance's .env, then:
    canasta backup list
    canasta backup restore -s <snapshot-id>
  1. Edit config/wikis.yaml before joining to replace production URLs with local addresses (e.g., localhost). The join command captures these URLs into your host's vars.yaml, so they must be correct before you run it.
  1. Join the instance to the gitops repo. gitops join registers the host with the both role, so you can pull updates and push your local changes back. Per-host role selection (source/sink/both) is only available when you first bootstrap the repo with gitops init:
    canasta gitops join -n devbox \
      --repo git@github.com:yourorg/mywiki-config.git --key /path/to/gitops-key

After making and testing changes locally, push them to the repo and pull on the production server:

# Local
canasta gitops add config/settings/global/MyChange.php
canasta gitops push -m "Description of change"

# Production
canasta gitops pull

Removing a server

On a source host:

  1. Remove the host entry from hosts.yaml
  2. Optionally remove the hosts/{name}/ directory
  3. Push: canasta gitops push -i mywiki -m "Remove production-2"

The instance on the removed server continues to function β€” it simply is no longer managed through gitops.

What needs a restart?

Change Restart needed?
PHP settings files No β€” takes effect on next request
wikis.yaml Yes β€” Caddyfile must be regenerated
Caddyfile.site / Caddyfile.global Yes β€” Caddy reloads on restart
docker-compose.override.yml Yes
.env changes Yes
my.cnf or config/media-hosts.yaml (rendered from their templates) Yes
New extension/skin Yes β€” run canasta maintenance update
Extension version update (no schema change) No
Extension version update (with schema change) Run canasta maintenance update

canasta gitops pull and canasta gitops diff automatically report whether a restart or maintenance update is needed.

Secret management

Docker Compose: git-crypt

On Docker Compose, secrets are encrypted transparently using git-crypt. Files under hosts/ are configured in .gitattributes to be encrypted on push and decrypted on pull:

hosts/** filter=git-crypt diff=git-crypt

On servers with the key, vars files are readable as plain text. On GitHub or for users without the key, they appear as encrypted blobs.

Key management

A Docker Compose gitops repo is unlocked in one of two ways, chosen when you run canasta gitops init.

Symmetric key (the default; recommended for small teams):

A single key file is generated during canasta gitops init and written to the --key path on the controller. Distribute it securely (e.g., via scp or a secrets manager) to each controller that needs access, and pass it as --key to canasta gitops join. Store the key outside the repo (e.g., /etc/canasta/gitops-key).

GPG-based (for larger teams; requires Canasta CLI 4.21.0 or later):

Instead of exporting a shared key file, canasta gitops init encrypts the git-crypt key to a GPG recipient (an email address or key ID) and commits the encrypted copy under .git-crypt/. Anyone holding that recipient's private key can unlock the repo. Before running init, import the recipient's public key into the GPG keyring on the target host β€” init runs git-crypt add-gpg-user there:

canasta gitops init -n myserver --repo git@github.com:yourorg/mywiki-config.git --key /path/to/gitops-key \
  --git-crypt-gpg-user ops@example.com

--key is still required, but no key file is written to it in this mode.

Canasta detects the mode from the repo itself: a repo with GPG recipients (encrypted keys under .git-crypt/keys/) is in GPG mode. Whenever Canasta has to unlock a GPG-mode checkout without --key β€” in canasta gitops join, or when push or pull find the checkout locked β€” it runs git-crypt unlock on the target host, which needs a recipient's private key there. Rather than copying private keys to servers, you can forward your own GPG agent to the target over SSH (an SSH RemoteForward to the target's S.gpg-agent socket); see the GnuPG guide to agent forwarding. When the CLI runs through canasta-docker, the host's GPG agent directory ($XDG_RUNTIME_DIR/gnupg, or /run/user/<uid>/gnupg) is mounted into the CLI container when it exists, so such a forward works from there too.

Only the recipient named at init is added by Canasta. To grant access to more people or servers, add their GPG identities with git-crypt's add-gpg-user in the instance directory and push the resulting commit with canasta gitops push.

When --key is given, Canasta always unlocks with that key file, in either mode β€” a git-crypt key file unlocks any git-crypt repo, including one with GPG recipients.

Concern Symmetric key GPG-based
Setup complexity Low β€” one key file Higher β€” GPG keys for every user/server
Key distribution Must securely copy the key file No shared secret
Revoking access Change key and redistribute Remove identity and re-key
Best for Small teams, few servers Larger teams, frequent access changes

Locked checkouts

A checkout that was cloned, reset, or restored without running git-crypt unlock holds the encrypted files under hosts/ as ciphertext. On Docker Compose, canasta gitops push and canasta gitops pull detect this and unlock the checkout first: with the key file given as --key (a path on the controller), or, without it, through GPG when the repo has GPG recipients. If the unlock fails, the command stops and says why. canasta gitops status does not unlock; it reports the locked state and the matching git-crypt unlock command to run.

canasta gitops pull --key /path/to/gitops-key

Kubernetes: encrypted secrets with SOPS

By default a Kubernetes gitops repo holds no secrets at all: MYSQL_PASSWORD, MW_SECRET_KEY and the rest live in Kubernetes Secrets in the cluster, and the repo stays cleartext because Argo CD has no git-crypt support and cannot render an encrypted values file. That means the repo alone is not enough to rebuild an instance β€” the cluster's Secrets have to survive independently.

Pass --encrypt-secrets to put them in the repo, encrypted with SOPS and an age key:

canasta gitops init --id mywiki --name dev \
  --repo git@github.com:yourorg/mywiki-config.git \
  --key ~/mywiki-deploy-key \
  --encrypt-secrets

Argo CD's repo-server decrypts at render time through a helm-sops sidecar, so git β€” not cluster state β€” becomes the source of truth for the instance's Secrets.

How the key is managed

  • The operator age key is cluster-global. One key per cluster decrypts every instance on it, because a single sops-age Secret in the argocd namespace serves them all. It is generated once and reused.
  • Its canonical home is on the controller, at $CANASTA_CONFIG_DIR/sops/<host>.age (defaulting to ~/.config/canasta/sops/, or /etc/canasta/sops/ for root). The private key is never written to the target host β€” only the public recipient reaches it, in .sops.yaml.
  • --key doubles as the portable disaster-recovery copy: init also exports the age key to <key>.age alongside the SSH deploy key.
  • Only the Secret payload is encrypted (data / stringData); the surrounding manifest stays readable, so diffs remain reviewable.

ℹ️ Note: canasta backup does not capture the operator age key β€” it lives in the controller's config directory, not the instance. Store the exported <key>.age off-host. Without it, a rebuilt controller cannot decrypt the repo and cannot re-provision the cluster's sops-age Secret.

Joining or recovering

A second cluster joining an already-encrypted repo, or a rebuilt controller that lost its key, adopts the existing key rather than minting a new one:

canasta gitops join --id mywiki --name prod \
  --repo git@github.com:yourorg/mywiki-config.git \
  --key /path/to/mywiki-deploy-key

The <key>.age file next to it is imported as the canonical operator key and checked against the recipient recorded in the repo's .sops.yaml. A mismatch is refused rather than silently re-keying: re-running gitops init on a controller whose key does not match the repo would re-encrypt future Secrets to a key the cluster cannot decrypt, so init stops and points you at join instead.

Because one sops-age Secret serves every instance on a cluster, gitops init and gitops join also refuse when that Secret already holds a different operator key. init's message names the key file to copy into place: the <host-name>.age file of the instance first initialized on that cluster. join's message asks you to join from a cluster whose sops-age Secret holds the repository's key, or from one that has none yet.

Workflow diagrams

Single server or small team (pull requests disabled):

[server] β†’ edit & test β†’ canasta gitops add β†’ canasta gitops push β†’ [git repo]

Multi-server with review (pull requests enabled):

[source] β†’ edit & test β†’ canasta gitops add β†’ canasta gitops push β†’ [PR] β†’ review & merge β†’ canasta gitops pull β†’ [sink hosts]

Further reading