Operations

Once ATHELOS is installed and running, this page covers the ongoing operational work — updates, backups, restoring from backup, and general maintenance.

Updating ATHELOS

ATHELOS releases follow semantic versioning. Updates are pulled as new Docker images.

Standard update

cd /home/kali/soc-solo-os
git pull
pnpm install
cd packages/db
pnpm drizzle-kit migrate
cd ../..
pnpm build

Then restart the services:

docker compose restart

Your data persists. Migrations are idempotent — running them twice is safe.

Update strategy

• Security patches — apply within days
• Minor versions — apply weekly or biweekly
• Major versions — read the release notes first; some may have breaking changes

The changelog documents every release. Breaking changes are flagged explicitly.

Rolling back

If an update breaks something:

cd /home/kali/soc-solo-os
git log --oneline -5
git checkout <previous-commit>
pnpm install
pnpm build
docker compose restart

If the update included a migration, you'll need to restore from backup first (see below). Migrations are not always reversible.

Backups

ATHELOS ships with automated encrypted backups to Backblaze B2 using restic.

What gets backed up

• The Postgres database (full dump, compressed)
• The MinIO data directory (evidence files)
• Environment files (apps/web/.env.local, docker/.env, root .env)
• Backup scripts and configs

Nothing else is backed up. The repo lives on GitHub; images can be pulled from the registry.

Automated backups

A systemd timer runs the backup daily at 03:00 with a randomized delay of up to 15 minutes.

Check status:

sudo systemctl status soc-backup.timer
systemctl list-timers | grep soc-backup

View recent runs:

journalctl -u soc-backup.service -n 50

Manual backup

To run a backup outside the schedule:

cd backups
./backup.sh

Configuring backups

Create backups/.env:

RESTIC_REPOSITORY=b2:athelos-backups
RESTIC_PASSWORD=<your restic password>
B2_ACCOUNT_ID=<key id>
B2_ACCOUNT_KEY=<application key>

Generate the restic password with:

openssl rand -base64 32

Store it in your password manager and on paper. Without it, backups cannot be decrypted.

Full setup instructions are in Scheduled backups.

Retention

Old snapshots are pruned automatically:

• 7 daily
• 4 weekly
• 12 monthly
• 1 yearly

24 snapshots at any time.

Restoring from backup

An untested backup is not a backup. Test the restore monthly.

List snapshots

cd backups
source .env
restic snapshots

Restore the database

./restore.sh db <snapshot-id>

Warning: this replaces the current database. Only run against a test instance or after confirming you want to lose current data.

Restore evidence files

./restore.sh files <snapshot-id>

Verify integrity

restic check

Reads every snapshot and confirms nothing is corrupted. Run monthly.

Database maintenance

Vacuum and analyze

Postgres accumulates dead rows over time. Run monthly:

docker exec soc_postgres psql -U soc -d soc_solo -c "VACUUM ANALYZE;"

Check database size

docker exec soc_postgres psql -U soc -d soc_solo -c "
SELECT pg_size_pretty(pg_database_size('soc_solo'));"

Slow queries

Enable pg_stat_statements:

docker exec soc_postgres psql -U soc -d soc_solo \\
  -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements;"

Top 10 slowest queries:

docker exec soc_postgres psql -U soc -d soc_solo -c "
SELECT query, calls, mean_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;"

Redis maintenance

Redis holds sessions, rate limits, and background job queues. It should not grow large.

Check memory

docker exec soc_redis redis-cli -a <password> INFO memory

Clear rate limits

If you're locked out of login:

docker exec soc_redis redis-cli -a <password> \\
  --scan --pattern "ratelimit:*" | xargs -r \\
  docker exec -i soc_redis redis-cli -a <password> DEL

Warning: don't clear the whole Redis database unless you intend to end all sessions.

MinIO maintenance

MinIO stores evidence files. It grows with every evidence upload.

Check disk usage

docker exec soc_minio du -sh /data

Filesystem check

MinIO includes a health endpoint:

curl http://localhost:9000/minio/health/live

Expected: empty response with HTTP 200.

Meilisearch

Meilisearch powers full-text search across the knowledge base.

Reindex

If search returns stale results, reindex:

pnpm reindex:kb

Check index stats

curl -H "Authorization: Bearer $MEILI_MASTER_KEY" \\
  http://localhost:7700/indexes

Logs

Every service logs to stdout, captured by Docker.

View all logs

docker compose logs -f

View one service

docker compose logs -f postgres
docker compose logs -f redis
docker compose logs -f minio
docker compose logs -f meilisearch

Application logs

The web app and worker log to their respective terminals. For systemd-managed services:

journalctl -u athelos-web -n 100
journalctl -u athelos-worker -n 100

Common maintenance tasks

Monthly

• Test the restore procedure
• Run restic check
• Run VACUUM ANALYZE on Postgres
• Review disk usage

Quarterly

• Review the changelog and update
• Rotate API keys if any external integrations exist
• Review audit log for unexpected activity

Yearly

• Review and rotate secrets if required by your policy
• Refresh the backup password if using password manager rotation
• Review the OS for updates

Where to go next

• Configuration — environment variables and tuning
• Scheduled backups — full backup guide
• Common issues — when something breaks