Backup and restore

What to copy, and how to put it back. The owner has not chosen a backup schedule or a recovery procedure yet (PROJECT.md, “Open decisions”). PROJECT.md asks for SQLite, the worker’s private key, the host policy and the controller’s backups to be kept as one recovery set.

Verified on dev on 2026-10-05 on a production-style install in a throwaway container (install.md steps 1 to 12): a backup of the running site, then, after an upgrade and the way back to the previous images, a restore of that set, each step as written here. The site came back with the set’s administrators, settings and actions, and without what was changed after the backup; all three services were healthy and Caddy served the site. The parts this did not cover are marked below.

Run the commands from a folder project can enter, such as /. From root’s home, find run as project warns that it can’t return to the folder it started in.

Placeholders, besides those in Install:

• <backup-dir>: where finished sets are kept, outside /opt/ProjectStart; root:root, mode 0700.
• <staging-dir>: an empty folder outside /opt/ProjectStart that project can write; project:project, mode 0700. The SQLite copies are made there as project.
• <set>: the name of one set, for example the date.

What to keep

Path	What it holds	Secret
/opt/ProjectStart/var/db/app.db	projects, operations, history, administrators, sessions, sealed Git tokens and passwords, agents	yes, together with secret.key
/opt/ProjectStart/var/db/secret.pub	public sealing key	no
/opt/ProjectStart/var/db/files	one folder per agent: its chat database db/chat.db, kept files files/, shown pages sites/	may hold anything admins and agents put in a chat
/opt/ProjectStart/var/worker/secret.key	private key that opens every sealed secret in app.db	yes
/opt/ProjectStart/var/worker/remote.key	the worker’s own key for remote hosts’ reverse connections; the hosts know it by its public key, which their install command carried	yes
/opt/ProjectStart/var/worker/push.key	the worker’s Web Push key; its public key is the setting push_public_key in app.db, which the browsers subscribed with	yes
/opt/ProjectStart/var/spool	inputs the worker leaves for the controller, including short-lived Git credentials and agent logins	yes
/opt/ProjectStart/ctl	project container backups (backups/), journals of unfinished changes (journal/)	yes: a container backup holds its Git credentials and agent logins
/opt/ProjectStart/caddy	project route files	no
/opt/ProjectStart/etc	host policy, agents.json, sentry.env	sentry.env holds a DSN
/opt/ProjectStart/deploy	compose.yaml, .env with the image digests	no

Not needed: /opt/ProjectStart/var/db/files/uploads (the worker empties it when it starts), /opt/ProjectStart/var/run, /opt/ProjectStart/var/ctl and /opt/ProjectStart/web (sockets only), /opt/ProjectStart/var/work (Git work).

SQLite files

app.db and every chat.db are SQLite databases in WAL mode, written while the services run. Never copy them with cp while web or worker run. Copy them with sqlite3, as project: run as root, SQLite can leave root-owned -wal and -shm files that the services can’t write.

Backup

The services keep running. Agent turns and operations go on during the copy, so the files may be a few seconds newer or older than app.db. For a set taken at one instant, stop web and worker first (docker compose -f /opt/ProjectStart/deploy/compose.yaml stop web worker); that ends every agent run.

1. Make the set’s folders:

   install -d -o root -g root -m 0700 <backup-dir>/<set>
   install -d -o project -g project -m 0700 <staging-dir>/<set>

2. Copy app.db:

   runuser -u project -- sqlite3 /opt/ProjectStart/var/db/app.db "VACUUM INTO '<staging-dir>/<set>/app.db'"

3. Copy the agents’ folders, then replace each copied chat database with a consistent copy. The folder files exists once the first agent has been made; skip this step on a site without one.

   runuser -u project -- cp -a /opt/ProjectStart/var/db/files <staging-dir>/<set>/files
   runuser -u project -- rm -rf <staging-dir>/<set>/files/uploads
   runuser -u project -- find <staging-dir>/<set>/files -name 'chat.db-*' -type f -delete
   for db in /opt/ProjectStart/var/db/files/*/db/chat.db; do
     [ -f "$db" ] || continue
     agent=${db#/opt/ProjectStart/var/db/files/}
     agent=${agent%%/*}
     runuser -u project -- sqlite3 "$db" ".backup '<staging-dir>/<set>/files/$agent/db/chat.db'"
   done

   Not verified on dev: the test site had no agent (no Claude or Codex sign-in), so no chat database was copied.

4. Copy the keys and the spool:

   runuser -u project -- cp -p /opt/ProjectStart/var/db/secret.pub <staging-dir>/<set>/secret.pub
   runuser -u project -- cp -p /opt/ProjectStart/var/worker/secret.key <staging-dir>/<set>/secret.key
   runuser -u project -- cp -p /opt/ProjectStart/var/worker/remote.key <staging-dir>/<set>/remote.key
   runuser -u project -- cp -p /opt/ProjectStart/var/worker/push.key <staging-dir>/<set>/push.key
   runuser -u project -- cp -a /opt/ProjectStart/var/spool <staging-dir>/<set>/spool

5. Move the staged files into the set, and add the root-owned folders:

   mv <staging-dir>/<set> <backup-dir>/<set>/var
   cp -a /opt/ProjectStart/ctl <backup-dir>/<set>/ctl
   cp -a /opt/ProjectStart/caddy <backup-dir>/<set>/caddy
   cp -a /opt/ProjectStart/etc <backup-dir>/<set>/etc
   cp -a /opt/ProjectStart/deploy <backup-dir>/<set>/deploy

6. Check each SQLite copy:

   sqlite3 -readonly <backup-dir>/<set>/var/app.db "PRAGMA integrity_check;"
   for db in <backup-dir>/<set>/var/files/*/db/chat.db; do [ -f "$db" ] || continue; sqlite3 -readonly "$db" "PRAGMA integrity_check;"; done

   Each line must say ok.

On dev, with the services running and a site of a few records, each step took under a second.

Restore

Restores a whole set. Restore app.db, the agents’ folders, the keys and ctl together: the database names the container backups in ctl/backups, and its sealed secrets open only with the secret.key of the same set.

The project containers in LXD are not part of a set. A restore puts the records back to the set’s time; the containers stay as they are now.

1. Stop all three services:

   docker compose -f /opt/ProjectStart/deploy/compose.yaml stop web worker controller

2. Put the database and the agents’ folders back, removing the current WAL files first:

   rm -f /opt/ProjectStart/var/db/app.db-wal /opt/ProjectStart/var/db/app.db-shm
   install -o project -g project -m 0660 <backup-dir>/<set>/var/app.db /opt/ProjectStart/var/db/app.db
   install -o project -g project -m 0640 <backup-dir>/<set>/var/secret.pub /opt/ProjectStart/var/db/secret.pub
   rm -rf /opt/ProjectStart/var/db/files

   When the set has a var/files folder (it has none when the site had no agent), put it back:

   cp -a <backup-dir>/<set>/var/files /opt/ProjectStart/var/db/files
   chown -R project:project /opt/ProjectStart/var/db/files

3. Put the keys and the spool back. The worker refuses a secret.key that group or others can read.

   install -o project -g project -m 0600 <backup-dir>/<set>/var/secret.key /opt/ProjectStart/var/worker/secret.key
   install -o project -g project -m 0600 <backup-dir>/<set>/var/remote.key /opt/ProjectStart/var/worker/remote.key
   install -o project -g project -m 0600 <backup-dir>/<set>/var/push.key /opt/ProjectStart/var/worker/push.key
   rm -rf /opt/ProjectStart/var/spool
   cp -a <backup-dir>/<set>/var/spool /opt/ProjectStart/var/spool
   chown -R project:project /opt/ProjectStart/var/spool

4. Put the root-owned folders back:

   rm -rf /opt/ProjectStart/ctl /opt/ProjectStart/caddy
   cp -a <backup-dir>/<set>/ctl /opt/ProjectStart/ctl
   cp -a <backup-dir>/<set>/caddy /opt/ProjectStart/caddy

   Restore etc and deploy only when the host policy or settings were lost; compare first with diff -r <backup-dir>/<set>/etc /opt/ProjectStart/etc and diff -r <backup-dir>/<set>/deploy /opt/ProjectStart/deploy. After an upgrade, deploy also holds .env.previous (and .env.rejected after the way back); the diff lists them, which is expected.

5. Check the folders, then start in the usual order. Give the folder check the project account’s ids, as at the install; without them it checks production’s ids, 993 and 983, and stops on another host.

   PS_UID="$(id -u project)" PS_GID="$(id -g project)" /usr/local/sbin/projectstart-install-folders
   docker compose -f /opt/ProjectStart/deploy/compose.yaml up -d --wait

   On dev, up -d --wait took 92 seconds.

6. The route files in /opt/ProjectStart/caddy are now those of the set. Reload Caddy so it serves them. Caddy 2.6.2 can stop during a reload; start it again then:

   systemctl reload caddy || systemctl start caddy

Not verified on dev: that a sealed secret (a Git token, a password) opens after a restore. The test set held none; the worker started on the set’s keys without an error. A restore onto another host, and container backups in ctl/backups, were not covered either: the restore ran on the same host, and its one project had no backup.