daybook/deploy/cloud_agent_deploy.md

# Cloud Deployment Runbook For A Dedicated Deploy Agent

This is written for a capable (Opus-class) agent deploying and operating
AgentFeed on a cloud VM. Treat it as a guide and a source of AgentFeed-specific
facts, **not a rigid script** — you're expected to use judgment, adapt to
the actual VM, and read the authoritative sources rather than follow steps
blindly. The steps below are a sensible default path; deviate where the host
warrants it.

Ground truth, in order of authority (prefer these over this doc if they ever
disagree):

- **The repo** — `https://git.cygnus.systems/peter/daybook.git` (private
  Forgejo). Everything ships from here; deploy from a clone so `git pull`
  works. Note: this Forgejo's git-SSH is on port **2222**, not 22.
- **`deploy/`** — `Dockerfile`, `docker-compose.yml`, `entrypoint.sh`,
  `env.example`. These *are* the deployment: the image installs all Python
  deps from `requirements.txt` (Django, DRF, Markdown, Pygments, nh3,
  openpyxl, gunicorn, whitenoise), and `entrypoint.sh` runs `migrate` +
  `collectstatic` on every start, so schema and static assets are always
  current. Read them; don't assume.
- **`docs/deployment.md`** — the general (host-agnostic) deployment
  reference and rationale.

Where a decision needs info you don't have (VM IP, domain, whether it's
internet-facing, whether a reverse proxy/secondary volume exists), ask the
operator once, then proceed autonomously for routine redeploys. The rest of
this file is the recommended path plus the non-obvious, AgentFeed-specific
gotchas (SSH port, CSRF-behind-proxy, bind address, data volume, healthcheck)
that are easy to get wrong.

## 0. Get Repo Access

You need read access to a private repo. Prefer a **repo-scoped SSH deploy
key** over a personal access token — it can only read this one repo, unlike
a token tied to a full user account.

1. Generate an SSH key pair on the VM (or ask the operator for one to install):
   ```bash
   ssh-keygen -t ed25519 -C "agentfeed-deploy-agent" -f ~/.ssh/agentfeed_deploy -N ""
   ```
2. Ask the operator to add the **public** key (`~/.ssh/agentfeed_deploy.pub`) as
   a Deploy Key on the repo in Forgejo (repo → Settings → Deploy Keys →
   read-only is sufficient), or hand it to them directly — never ask them to
   give you their personal account credentials.
3. Configure SSH to use it for this host. **Note:** this Forgejo's git-SSH
   listens on port `2222`, not the default `22` (chosen so it doesn't clash
   with the box's own system `sshd`) — connecting on `22` hits system SSH
   instead and will fail or hang, not just warn. Don't assume `22` even if
   this changes again later; confirm with the operator if unsure.
   ```bash
   cat >> ~/.ssh/config <<'EOF'
   Host git.cygnus.systems
     Port 2222
     IdentityFile ~/.ssh/agentfeed_deploy
     IdentitiesOnly yes
   EOF
   ```
4. Verify:
   ```bash
   ssh -T git@git.cygnus.systems
   ```
   If you'd rather not rely on `~/.ssh/config`, clone with the port explicit
   instead: `ssh://git@git.cygnus.systems:2222/peter/daybook.git`.

If SSH access can't be arranged, fall back to an HTTPS access token the
operator generates (Forgejo → user Settings → Applications → Generate New
Token, read-only `repo` scope) and clone with:

```bash
git clone https://<username>:<token>@git.cygnus.systems/peter/daybook.git /opt/daybook
```

## 1. Prerequisites On The VM

Confirm before proceeding; install if missing:

```bash
docker --version || curl -fsSL https://get.docker.com | sh
docker compose version
git --version
```

Confirm outbound network access to pull the Python base image and packages,
and confirm you know the VM's public IP or DNS name — you'll need it for
`DJANGO_ALLOWED_HOSTS` and for reporting the final URL back.

## 2. Clone

```bash
sudo mkdir -p /opt/daybook
sudo chown "$USER:$USER" /opt/daybook
git clone git@git.cygnus.systems:peter/daybook.git /opt/daybook
cd /opt/daybook
```

## 3. Configure Environment

```bash
cp deploy/env.example deploy/.env
```

Generate a real secret key — do not deploy with the placeholder:

```bash
python3 -c "import secrets; print('DJANGO_SECRET_KEY=' + secrets.token_urlsafe(50))"
```

Edit `deploy/.env`:

```text
DJANGO_DEBUG=0
DJANGO_SECRET_KEY=<generated-secret>
DJANGO_ALLOWED_HOSTS=<vm-public-ip>,<domain-if-any>,localhost,127.0.0.1
AGENTFEED_DATA_DIR=/data
DATABASE_PATH=/data/db.sqlite3
ARTIFACTS_ROOT=/data/artifacts_store
```

There is no shared API token to set — AgentFeed issues a personal token per
user from Settings → Account once a user exists (step 5). Never commit
`deploy/.env`; it is already gitignored.

`deploy/.env` will be read and written by whatever agent runs this runbook.
Treat it as low-stakes: it should never hold anything more sensitive than
this app's own `DJANGO_SECRET_KEY`. If the target host has its own
documented convention for secrets (e.g. composed by reference at execution
time rather than written to a file), follow that instead — ask the operator
if unsure which applies here.

## 4. Network Access

**If this VM is reachable from the public internet, a reverse proxy with TLS
in front of AgentFeed is not optional — it's the only path.** Bearer tokens are
this app's entire write-auth model, and they'd otherwise go over plaintext
HTTP straight to a port on the public IP. Direct port exposure (below) is
only for private/VPN-only networks with no public exposure.

- **Internet-facing (the common case):** set `AGENTFEED_BIND_ADDR=127.0.0.1` in
  `deploy/.env` so the container port is not reachable from outside the VM at
  all, then open `80`/`443` in the VM's firewall/security group for the
  reverse proxy (see §8). Do not open `8000` externally.
- **Private/VPN-only network:** open TCP `8000` to the addresses that should
  reach it (never `0.0.0.0/0`), and leave `AGENTFEED_BIND_ADDR` unset.

Before opening any port, check whether one is already open and in use —
`ss -tlnp | grep -E ':80|:443|:8000'` — rather than assuming a clean slate.
Firewall/security-group configuration is provider-specific (AWS security
group, GCP firewall rule, DigitalOcean cloud firewall, or the VM's own
`ufw`/`iptables`). Ask the operator which provider/console to use if you
don't have API access to it yourself.

## 5. Build, Start, Create A User

If the target has a separate, larger data volume than its main disk (common
on space-constrained VMs), point AgentFeed's persistent data at it before
starting: set `AGENTFEED_HOST_DATA_DIR=/path/on/that/volume` in `deploy/.env`.
Ask the operator whether such a volume exists and where before defaulting to
the main disk — filling the main disk on a host running other services is a
bigger problem than it would be on a dedicated box. Create that same path
(the one you just set in `deploy/.env`, or `deploy/data` if you left it
unset) before starting the stack:

```bash
mkdir -p deploy/data   # or the AGENTFEED_HOST_DATA_DIR path if you set one above
docker compose -f deploy/docker-compose.yml up -d --build
docker compose -f deploy/docker-compose.yml exec agentfeed python manage.py createsuperuser
```

The entrypoint runs migrations and collects static files automatically on
every start.

## 6. Verify

```bash
docker compose -f deploy/docker-compose.yml ps
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8000/api/capabilities   # expect 403
```

A `403` with `"Login or a valid API token is required"` is the **expected**
response when unauthenticated — it confirms the app is up. (Don't use
`curl -f` here: `--fail` treats the 403 as an error and hides it. The
container's own healthcheck hits the login page instead, which is 200
without auth.) This `curl` runs
against `127.0.0.1` from on the VM itself, so it works regardless of whether
`AGENTFEED_BIND_ADDR` is set. Log in from a browser at whichever address is
actually externally reachable at this point in the runbook — `http://<vm-ip>:8000/`
if you're on the private/VPN-only path from §4, or `https://<domain>/` through
the reverse proxy once §8 is done if you're on the internet-facing path — with
the superuser from step 5, grab a personal token from Settings → Account, then
smoke-test a post:

```bash
curl -fsS -X POST http://127.0.0.1:8000/api/artifacts \
  -H "Authorization: Bearer <token-from-settings>" \
  -H "Content-Type: application/json" \
  -d '{"source": "deploy-agent", "kind": "markdown", "title": "Deployment Smoke Test", "content": "# OK\n\nDeployed successfully."}'
```

Confirm the response includes `id`, `url`, and `api_url`, and that opening
`url` in a browser renders the artifact.

## 7. Redeploy Loop (routine updates)

Releases are tagged `vX.Y.Z` (SemVer). Deploy a specific tag rather than a
moving branch, so what's live is always a known, named version — and note
the currently-deployed one before switching, for a clean rollback target:

```bash
cd /opt/daybook
PREV_TAG=$(git describe --tags --exact-match 2>/dev/null || git rev-parse HEAD)  # what's live now
git fetch --tags
git checkout v0.1.0        # the tag you were told to deploy (latest: git tag -l | sort -V | tail -1)
docker compose -f deploy/docker-compose.yml up -d --build
```

Then repeat the verification in step 6 (and confirm `/api/capabilities`
reports the `version` you expect). If it fails, roll back to the previous
version and report the failure (build log / migration error / failed health
check) rather than retrying blindly:

```bash
git checkout "$PREV_TAG"
docker compose -f deploy/docker-compose.yml up -d --build
```

Caveat: rollback reverts code, not data — a release that already applied a
DB migration may not cleanly downgrade against the migrated `db.sqlite3`.
For a routine forward release that's fine; if a rollback is needed after a
migration, restore the pre-deploy `deploy/data` backup (see §9) alongside
the code checkout.

`deploy/.env` and the data directory (wherever `AGENTFEED_HOST_DATA_DIR` points,
or `deploy/data` by default) are untracked, so `git pull` never touches them —
no need to back them up before a routine redeploy.

**If the container crash-loops right after a build** (`docker compose ps`
shows `restarting`, health never goes green), the most likely cause is the
entrypoint's `collectstatic` step failing — check `docker logs agentfeed`
before assuming an env/network problem. A `whitenoise ... MissingFileError`
there means a static asset references a file that isn't present (this is what
bit v0.1.0). No data or schema impact; fix the asset in the repo, tag, and
redeploy.

**If `git checkout <tag>` refuses because of local working-tree changes** to
tracked files (e.g. a previous on-server hotfix), reset those tracked files
first — `git checkout -- <file>` for a specific one, or `git stash` /
`git reset --hard` to clear all tracked drift. This never touches `deploy/.env`
or the data directory, which are untracked and therefore safe.

## 8. Hardening For Internet-Facing Deployments

If this VM is reachable from the public internet (not just a private
network/VPN), do this before calling the deployment done:

- Put a reverse proxy in front with TLS. Bearer tokens go over plaintext HTTP
  otherwise.
- Add the proxy's public hostname to `DJANGO_ALLOWED_HOSTS`.
- Set `DJANGO_CSRF_TRUSTED_ORIGINS=https://<public-hostname>` in `deploy/.env`,
  **and** make sure the proxy forwards `X-Forwarded-Proto: https` (examples
  below). Without both, login fails with "CSRF verification failed" — Django
  can't tell the original request was HTTPS once a proxy terminates TLS and
  forwards plain HTTP to the container.
- Keep `DJANGO_DEBUG=0` (already set above).
- Restrict SSH to key-based auth only; disable password login if you have
  access to do so.
- Confirm the VM's firewall only allows the ports actually needed (80/443,
  and SSH from wherever the operator connects from).

**Check for an existing reverse proxy before introducing a new one** —
`ss -tlnp | grep -E ':80|:443'`. A VM already running other services almost
always already has one (Apache, Nginx, Traefik) terminating TLS for
everything on it; a second process trying to bind 80/443 will just fail to
start, and stopping the existing proxy to "free the port" would take every
other service on the box down with it. If one exists, add a vhost to it
instead. Apache example:

```apache
<VirtualHost *:443>
    ServerName agentfeed.example.com
    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto "https"
    ProxyPass / http://127.0.0.1:8000/
    ProxyPassReverse / http://127.0.0.1:8000/
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/agentfeed.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/agentfeed.example.com/privkey.pem
</VirtualHost>
```

`RequestHeader set X-Forwarded-Proto "https"` requires `a2enmod headers`; it's
not optional (see the CSRF note above). (`certbot --apache` will provision and
wire up the certificate for you.)

Only reach for a standalone proxy like Caddy if the target genuinely has
nothing on 80/443 yet — it still needs the same header:

```text
agentfeed.example.com {
    reverse_proxy 127.0.0.1:8000 {
        header_up X-Forwarded-Proto https
    }
}
```

## 9. Backup

If you set `AGENTFEED_HOST_DATA_DIR` in step 5, the real data lives there, not
at `deploy/data` — back up whichever path is actually configured, read
straight from `deploy/.env` rather than assumed, so this keeps working
whether or not that override is set:

```bash
cd /opt/daybook
DATA_DIR="$(grep -oP '(?<=^AGENTFEED_HOST_DATA_DIR=).*' deploy/.env || echo deploy/data)"
tar -czf "/tmp/agentfeed-data-$(date +%Y%m%d-%H%M%S).tar.gz" -C "$(dirname "$DATA_DIR")" "$(basename "$DATA_DIR")"
```

Copy the resulting archive off the VM (to object storage, the operator's
machine, etc.) — a backup that only exists on the VM it's protecting against
doesn't help if the VM is lost.

## Deployment Checklist

Before reporting the deployment complete, confirm all of these:

- [ ] Cloned via git (not copied), with a working `git pull`.
- [ ] `deploy/.env` has a real, non-placeholder `DJANGO_SECRET_KEY`.
- [ ] `DJANGO_ALLOWED_HOSTS` includes the actual hostname/IP being used.
- [ ] If behind a reverse proxy: `DJANGO_CSRF_TRUSTED_ORIGINS` is set to the
      public HTTPS origin, and the proxy forwards `X-Forwarded-Proto: https`.
      Verify by actually logging in through the proxy, not just curling
      `/api/capabilities` — CSRF only bites on POST (login, artifact writes).
- [ ] `docker compose -f deploy/docker-compose.yml ps` shows `agentfeed` healthy.
- [ ] `/api/capabilities` returns `403` unauthenticated, `200` with a token.
- [ ] At least one superuser exists with a working personal API token.
- [ ] A smoke-test artifact posts and renders successfully.
- [ ] If internet-facing: TLS is in front of the app, not raw HTTP on `8000`.
- [ ] A backup of the data directory (`deploy/data`, or `AGENTFEED_HOST_DATA_DIR`
      if you set it) has been taken and copied off the VM.
- [ ] If the target host has its own service-inventory, monitoring, or
      changelog conventions for new services, they've been followed — ask
      the operator if unsure what those are.

Report back to the operator: the app URL, confirmation of the checklist
above, and the deploy SSH key fingerprint or token identity used (not the
key/token itself) for their records.