Commit graph

26 commits

Author SHA1 Message Date
6d2fe2147c Archive resolved issues; add open issues #014-#016
Moves the 11 Fixed issues (#001-006, #008-012) into issues/_archive_/,
which is gitignored — they stay on local disk and in git history but drop
out of the committed tree, keeping the issues/ dir focused on what's open.
The sibling test project reads this dir by local path, so it still sees
the archived files. Adds the three new open issues found in round-3
testing: #014 (stored XSS in markdown/deck/html_fragment — High), #015
(RecursionError -> 500 on deeply nested JSON), #016 (ragged CSV rows lose
data in the rendered table). Documents the archive convention in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-03 10:27:45 +01:00
292c386209 Add round-2 handover doc for the LLM test project
Covers everything shipped since the #007 handoff: the json/csv kinds,
.xlsx conversion, public sharing, syntax highlighting/table styling, and
the #008-#012 fixes. Prioritizes round-3 probe suggestions by risk, with
sharing (the new unauthenticated surface) and the #010 quota race (no
automated coverage) called out as the highest-value targets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 20:53:12 +01:00
9a3b381f39 Add light/dark mode toggle for the UI
The UI stylesheet already centralized its palette in a :root block, so this
mostly meant (1) migrating the ~30 colors still hardcoded in rules into
variables so they flip in one place, and (2) adding a [data-theme="dark"]
override palette. Artifact rendering is untouched — those use the separate
static/themes/*.css and the preview backdrop stays white by design, since
artifacts (charts, reports) are authored light.

- Toggle lives in the gear-menu dropdown (Dark mode / Light mode), persists
  to localStorage.
- A tiny inline script in each page <head> (feed, settings, login) applies
  the stored theme before first paint, so dark-mode users never see a white
  flash on load.
- color-scheme: dark on the dark palette makes native form controls
  (inputs, buttons) render dark too.
- Semantic accents (kept=green, shared=blue, today, danger) got
  dark-appropriate variants; status/data-viz colors (usage bar, tag chips)
  and the white artifact backdrop are intentionally left as-is.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 17:14:56 +01:00
c9edac62cf Review docs/api.md and docs/llm_posting_helper.md for accuracy and completeness
Same pass as the SKILL.md review. Fixed factually-wrong content and filled
gaps against the current code.

docs/api.md:
- Capabilities example response was stale: artifact_kinds missing json/csv,
  raw_supported_for missing json/csv. Corrected to match live output, with
  a note that the endpoint is the authoritative source.
- "Office document conversion is not implemented" is now false - replaced
  with the accurate scope (.xlsx -> csv only; .docx/.pptx/.xls not).
- Source Fields overview table omitted source_prompt entirely; added it,
  plus a note that source_name/source_id/source_prompt must be strings
  (400, not coerced) and metadata must be an object.
- Kinds table: noted tables now render styled in markdown/html_fragment,
  and made explicit that html_full is served verbatim with no styling.
- MVP constraints: noted the public-share-link exception to "feeds private
  per user" and clarified PATCH edits metadata only.

docs/llm_posting_helper.md:
- Added .xlsx upload, --upload-image image embedding, a Source & Theme
  Flags table (all six source-*/theme flags were undocumented here), and a
  note that --meta/--meta-json merge and are distinct from tags.
- Added a scope note that the helper only creates - update/delete need the
  API, sharing is feed-UI-only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 16:49:19 +01:00
d2d970f33d Review and correct SKILL.md for accuracy and completeness
Found six real flags with zero mention anywhere in the doc:
--source-type, --source-name, --source-id, --source-prompt, --meta-json,
--dry-run. Added a new "Source & Provenance Fields" section for the
source-* flags and covered the rest inline/in Rules.

Fixed:
- html_full's cheat-sheet entry overstated the requirement as needing
  doctype/html/head/body; the actual check only requires <html>.
- The "Images Inside Content" section had grown to cover four unrelated
  topics (images, CSV/JSON, xlsx, code-as-markdown) under a header
  describing only the first one. Split into properly-titled sections:
  Images Inside Content, CSV/JSON/Spreadsheets, Other Code.
- An unclosed parenthesis in the xlsx limitations paragraph, rewritten as
  a bulleted list for clarity while fixing it.
- Nothing anywhere mentioned that this script only creates artifacts -
  update/delete need the raw API, sharing is feed-UI-only with no API
  path at all. Added an explicit scope note.
- --source-name's actual default (the resolved source, which can come
  from DAYBOOK_SOURCE, not just --source) was previously undocumented
  entirely, so "same as --source" would have been imprecise once added.

Added: available --theme values (report/tasks/base) and which kinds
ignore it, and a note that markdown/html_fragment tables now render
styled automatically (from this session's earlier CSS pass).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 16:42:15 +01:00
4b5ec09ae2 Convert uploaded .xlsx files to csv artifacts server-side
Uploading a .xlsx file no longer becomes a generic downloadable binary -
it's read server-side with openpyxl and converted to a csv artifact,
rendering in the same interactive grid as a native CSV post. Detected by
content type or .xlsx extension (some clients send generic/incorrect
content types for multipart uploads). Only the active sheet converts;
formula cells use whatever cached result the spreadsheet software last
saved (openpyxl doesn't evaluate formulas itself). Corrupt/unreadable
files return 400.

Since the existing posting script already forwards any non-text file via
multipart with no special-casing, `daybook_post.py balance_sheet.xlsx`
already works with zero script changes - verified live.

Two real bugs found and fixed while building this, both now covered by
regression tests:
- {**a_querydict, ...} silently wrapped every value (including title) in
  a list - Python's ** spread on a dict subclass bypasses QueryDict's
  __getitem__ override and exposes its internal one-list-per-key storage
  directly. Fixed by building the delegated dict explicitly via .get().
- Pre-setting None for absent optional fields (e.g. source_prompt) defeated
  create_text_artifact's own data.get(field, default) calls, since .get()
  only falls through to its default when the key is *missing*, not when
  the value is None - caused a NOT NULL constraint failure. Fixed by only
  setting keys that were actually provided.
- csv.writer's default "\r\n" terminator was getting doubled by
  Path.write_text()'s own newline translation on Windows, producing a
  blank line between every row. Fixed with lineterminator="\n".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 16:29:39 +01:00
4145c37dfe Note that html_full can reference Daybook's own vendored JS libraries
Grid.js and renderjson (vendored for the csv/json kinds) are reachable as
plain static files from any artifact, including html_full - a better
default than an external CDN for table/JSON rendering specifically, since
it works offline and has no third-party dependency.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 16:04:53 +01:00
91ea19d26b Add per-artifact public sharing
Any artifact can now be made publicly viewable via a share link, no login
required - opt-in per artifact, toggled from the feed UI next to Keep.

- Artifact.share_token: nullable, unique, high-entropy (secrets.token_urlsafe(32)).
  NULL = not shared; SQL treats each NULL as distinct so uniqueness still
  holds for the many not-shared rows.
- New public route GET /s/<token> (shared_artifact_view), completely
  unauthenticated, serving the same rendered/raw content as the owner
  route via a shared serve_artifact_file() helper.
- toggle_artifact_share: generates/clears the token: mirrors
  toggle_artifact_keep's pattern.
- Fixed a real gap this surfaced: embedded images in a shared artifact's
  content would otherwise break for an anonymous visitor, since asset_view
  was owner-only. It now also allows access when the asset's parent
  artifact is currently shared - unattached (orphaned) assets stay private
  regardless.
- serialize_artifact now reports is_shared/share_url so the API reflects
  sharing status (read-only for now; toggling stays UI-only).
- Feed UI: Share button (mirrors Keep's toggle interaction), a copyable
  share-link box that appears when shared, and a "shared" badge on the
  artifact row - distinct blue accent from Keep's green so both can be
  visible on the same row without confusion.

Verified live end-to-end (toggle on/off, public access, fake-token 404,
old-link-dies-on-unshare, embedded-asset access before/after sharing)
before writing the 12 automated tests in tests/views/test_sharing.py.
Also fixed two unrelated stale spots noticed in docs/api.md while adding
the Sharing section: the example response was missing is_kept/size_bytes,
and the raw-content table didn't list json/csv (added earlier this
session but never documented there).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 16:00:39 +01:00
e137725cab Add json/csv support to the posting script (issue #012)
TEXT_KINDS and TEXT_SUFFIXES now include json/csv, with matching
sniff_text_kind branches so both explicit --kind and positional auto-mode
work. Added validate_payload's JSON shape check mirroring the server.
Updated SKILL.md/docs/llm_posting_helper.md with real usage examples,
replacing the stale --as-markdown workaround guidance (kept only for
genuinely unstructured code, which still has no dedicated kind). Also
fixed an unrelated stale doc line found in the same table: html_full's
validation was documented as requiring <!doctype AND <html, but only
<html is actually required.

Verified via dry-run (explicit --kind, auto-mode, invalid-JSON rejection)
and one real end-to-end post through the script against the live dev
instance.

Added tests/scripts/test_daybook_post.py - 17 tests, plain unittest, zero
new dependencies, loaded via importlib since the script isn't a package.
Closes the test-coverage question the issue raised.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 15:38:47 +01:00
532b4bd4eb Document #009's substring-check limitation; fix #011's CDN docs and theme typo
#009: html_fragment's content-shape check is substring-based, so it can
false-positive on inert text inside <script>/<pre> (e.g. a JS string
literal that happens to contain "<html>"). Documented as a known
limitation in docs/api.md rather than implementing real HTML parsing - a
proper fix is a much bigger change than the narrow trigger justifies, and
substring matching is the same approach the client helper already uses.

#011: added a curated CDN Snippets section to SKILL.md for html_full
(Tailwind, Chart.js, with an offline/air-gapped caveat), scoped explicitly
to that kind only. Fixed daybook_post.py's --theme help text, which named
a non-existent "bootstrap" theme (written ahead of a vendored-Bootstrap
feature that was planned but never built, not a typo).

Also updated SKILL.md's kind cheat sheet, which never mentioned the
json/csv kinds and still pointed at the old --as-markdown workaround for
them - and filed issue #012 for a real gap that surfaced doing so: the
posting script itself doesn't support --kind json/csv yet, only the raw API
does.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 15:32:22 +01:00
09ea0a8b4f Reject non-string source_name/source_id/source_prompt (issue #008)
These accepted any JSON type on both POST and PATCH, silently mangled
into Python repr() strings at save time (a dict became the literal text
"{'a': 1}", unparseable as JSON) - and the immediate response echoed the
raw pre-save value, while a later GET returned the mangled one, an
internally inconsistent API.

Added validate_string_fields(), rejecting non-string values with 400
before they reach save(), matching #005's approach for tags. Verified live
and with new tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 15:27:07 +01:00
d1b6a19eb3 Fix concurrent posts racing past quota (issue #010)
make_room_for() read usage before any write lock was held, so concurrent
requests could each read stale data and both conclude "this fits" -
confirmed reproducible (measured 616 bytes over a 1551-byte quota with 5
truly concurrent posts).

Fix: force SQLite to acquire its write lock immediately via a no-op UPDATE
on the user's UserProfile row, before reading `used`, wrapped in
transaction.atomic() through the subsequent Artifact.objects.create() so
the lock is held across the whole eviction+insert sequence. Added a 20s
DB timeout as headroom for legitimate contention.

Deliberately no retry/backoff logic - this project's actual load (a
low-volume self-hosted tool) doesn't need it, and SQLite has no row-level
locking regardless, so quota-affecting writes serialize globally either
way. Did fix a real bug found while verifying this: exception handling
around artifact creation only caught QuotaExceeded, so a genuine lock
timeout left the just-written file orphaned on disk - both call sites now
clean up on any exception.

Verified live against the real dev instance (twice, with 5 truly
simultaneous posts against a tight quota) rather than via an automated
test - Django's in-memory shared-cache test database uses a different,
non-retryable SQLite locking model that fails for reasons unrelated to
this fix's correctness.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 15:22:37 +01:00
696cfe7d65 Style tables in markdown/html_fragment content
Markdown pipe-tables and hand-written html_fragment <table> markup already
rendered - the only existing rule was 'max-width: 100%', so they came out
as bare, unbordered browser-default tables. Adds borders, header
background, cell padding, and zebra striping, plus a display:block +
overflow-x:auto pattern so wide tables scroll horizontally instead of
squeezing columns or breaking page layout.

Lives in base.css, which report.css and tasks.css both @import, so it
applies to every existing and future markdown/html_fragment artifact
immediately - no agent-side change needed, since this is rendered live
from CSS rather than baked into stored HTML.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 15:01:26 +01:00
e5d2f192df Add json and csv artifact kinds with interactive viewers
Gap 1 from planning/2026-09-02-artifact_kind_gaps_for_frequent_agent_output.md.
Chose new first-class kinds over riding on html_fragment, resolving that
doc's open design question in favor of filtering support, raw-source
preservation (like markdown's .md sidecar), and server-owned validation.

- json: validated with json.loads at post time (400 on invalid JSON),
  rendered with renderjson (collapsible tree), vendored locally.
- csv: parsed server-side with the stdlib csv module into columns/rows and
  handed to Grid.js as plain data - no client-side CSV-parsing plugin
  needed, keeping normalization server-owned. Vendored locally (MIT).
- Both preserve their original text as a sidecar file, generalizing what
  was previously markdown-only handling (storage.py's SIDECAR_SUFFIXES) so
  disk-size accounting, eviction cleanup, and ?raw=1 all cover json/csv too.
- Raw JSON is embedded in a <script type="application/json"> block with
  `</script` occurrences escaped (safe because JSON's \/ escape round-trips
  correctly) - verified with a live </script>-injection payload against the
  actual running server, not just unit tests.

New migration (0012) for the KIND_CHOICES change - SQLite doesn't enforce
choices at the schema level, but Django's migration state tracks it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 14:56:20 +01:00
af9ecf28a9 Add syntax highlighting for fenced code blocks in markdown artifacts
Enables codehilite on the markdown renderer (Pygments, 'friendly' style,
vendored as static/themes/pygments.css) so fenced code blocks get real
syntax highlighting with zero client-side JS - matching the "themes are
CSS-only" stance in framework_artifact_rendering.md. Verified live against
the running dev instance: a fenced python block renders with highlighted
keyword/string/function spans.

This was Gap 2 from planning/2026-09-02-artifact_kind_gaps_for_frequent_agent_output.md,
picked as the first fix since it needed no new kind or design decision -
just enabling an existing markdown extension.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 14:39:02 +01:00
d9fc28aec0 Add handover doc summarizing issues #002-#006 fixes for the LLM test project
Confirms all five are live on the local dev instance without a restart
(autoreload picked them up), and suggests where to probe next - boundary
cases on the new length/content-shape validation, other wrong-typed PATCH
fields beyond tags, and eviction race conditions #006 doesn't cover.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 13:56:43 +01:00
07328a3412 Fix issues #002-#006 found by the LLM test harness
- #006 (high, data loss): make_room_for() no longer deletes unkept
  artifacts on a doomed request - checks whether eviction can possibly
  succeed before deleting anything. Also fixed a related latent bug found
  while testing: the function returned evicted Artifact objects after
  calling .delete() on them, so their .id was always None (Django clears
  pk on delete); now returns ids captured beforehand.
- #003: added server-side field-length validation (source/title/
  source_name/source_id/theme) on POST and PATCH, matching the model's
  own max_length limits, which were previously unenforced on the ingest
  path entirely.
- #004: ported daybook_post.py's content-shape validation
  (html_fragment/html_full/deck/markdown) to the server, closing the gap
  for any client that doesn't go through the helper script.
- #005: PATCH now rejects wrong-typed `tags` (previously silently wiped
  existing tags) and rejects immutable fields (content/source/kind/day)
  with 400 instead of silently no-op'ing them.
- #002: deleted the misleading skills/daybook-posting/.env.example (a
  zero-dependency script that never reads .env files) and clarified
  SKILL.md/docs/llm_posting_helper.md that these must be real
  process/shell env vars.

Adds tests/unit/ for storage-layer tests. 46 tests passing (was 25).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 13:38:59 +01:00
c1ea5c7990 Note the permanent test/testpwd fixture user for LLM API testing
AGENT_DAYBOOK__LLM_TEST (a sibling project) always uses this account to
test the API. Documenting it here so a future reset of the local dev
database gets it recreated instead of silently breaking that project.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 12:59:14 +01:00
edacb86800 Add admin-only Settings -> Users page for creating accounts
Peter needed a second user for throwaway testing without going through
Django admin each time. Superusers now get a "Users" tab in Settings
(hidden for regular users) with a create-user form and a list of existing
accounts. New users are regular (non-staff, non-superuser) by default, with
an optional "Grant admin access" checkbox; creating one immediately issues
their API token and displays it once, since that's the credential an agent
account needs and a human account can otherwise regenerate for themselves
later from their own Settings -> Account.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 12:33:38 +01:00
f2f2a9c96b Close out issue 001: document CSRF/reverse-proxy requirement
The app-side fix (CSRF_TRUSTED_ORIGINS, SECURE_PROXY_SSL_HEADER in
settings.py) was already committed as d458cd3 after the first Cloud-VM-1
deployment hit "CSRF verification failed" on login behind Apache. This
finishes the rest of that issue's checklist: documents
DJANGO_CSRF_TRUSTED_ORIGINS in deploy/env.example, adds the required
X-Forwarded-Proto header to the Apache/Caddy examples in
cloud_agent_deploy.md (plus a checklist item that verifies via an actual
login, not just an unauthenticated capabilities curl), and adds the same
proxy-agnostic guidance to docs/deployment.md.

Also introduces issues/, a tracked one-file-per-issue log for the project
going forward.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 12:28:41 +01:00
d458cd3673 Fix CSRF failure behind a TLS-terminating reverse proxy
Django had no CSRF_TRUSTED_ORIGINS and no SECURE_PROXY_SSL_HEADER, so
it always treated requests as plain HTTP. Login over HTTPS behind
Apache/Nginx failed CSRF verification as a result.
2026-09-02 12:17:16 +01:00
f7dd98f6e5 Fix backup step to follow the configurable data path (F1)
A follow-up review of the previous fix caught that M1's DAYBOOK_HOST_DATA_DIR
override wasn't carried through to the backup step or checklist - both still
hardcoded deploy/data. Exactly when the override matters (a host with a
separate/larger volume), the backup would silently tar the wrong directory
while the checklist still read as complete.

Backup and checklist now derive the path from deploy/.env instead of
restating it. Also documents in docker-compose.yml why the .env
interpolation only resolves correctly when invoked as
`docker compose -f deploy/docker-compose.yml`, per a non-blocking note from
the same review.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-01 22:45:18 +01:00
441d000d33 Fix deploy runbook against a real, already-crowded target host
Addresses a review of deploy/cloud_agent_deploy.md against Cloud-VM-1 (a
shared Hetzner VPS already running ~10 other services, including the
Forgejo instance this repo is hosted on):

- Blocking: git.cygnus.systems' git-SSH listens on port 2222, not 22 -
  fixed the SSH config example and clone fallback, with a note not to
  assume 22 if it changes again.
- High: container port now binds via configurable DAYBOOK_BIND_ADDR
  (defaults to all interfaces, set to 127.0.0.1 when a reverse proxy fronts
  it) instead of always publishing to 0.0.0.0; reverse-proxy+TLS is now the
  only path presented for internet-facing hosts, not a coequal option with
  direct exposure.
- High: replaced the Caddy-only hardening example with an Apache vhost +
  certbot path (matching how this specific host already terminates TLS for
  everything else on it), plus a check-before-assuming-80/443-are-free step.
- High: added mem_limit/mem_reservation to docker-compose.yml so this
  doesn't risk OOM-killing other services on a memory-tight box.
- Medium: added configurable DAYBOOK_HOST_DATA_DIR so persistent data can
  land on a separate/larger volume instead of a space-constrained main disk.
- Medium/Low: checklist now prompts checking the target's own
  service-inventory/monitoring/changelog conventions, and .env guidance
  notes it should stay low-stakes since a deploy agent reads/writes it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-01 22:38:03 +01:00
d84753c839 Add cloud deployment runbook for the dedicated deploy agent
Self-contained operational SOP for an autonomous agent deploying and
operating Daybook on a cloud VM: bootstrapping its own scoped SSH deploy
key, cloud-specific network/firewall setup, the git-pull redeploy loop with
rollback, TLS hardening for internet-facing hosts, and a completion
checklist.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-01 22:23:55 +01:00
4a5ea8944b Add generic deployment docs and fix stale references
Adds docs/deployment.md, a platform-agnostic guide (any Linux host with
Docker) built around a git-pull based fast-redeploy workflow, replacing the
Proxmox-specific deploy/proxmox_llm_deploy.md which was also stale (old
`persona` field, single-shared-token model predating multi-user auth).

Also: drops the unused DAYBOOK_API_TOKEN from deploy/env.example, documents
the new DELETE endpoint and capabilities `methods` field in docs/api.md and
docs/agent_quickstart.md, and fixes CLAUDE.md's OpenCode note now that
opencode.json is untracked (machine-specific MCP config).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-01 22:14:34 +01:00
940a703cd7 Initial public commit
Daybook: a self-hosted, multi-user Django + DRF app where agents, scripts,
or people publish artifacts and each logged-in user browses their own
time-ordered, label-filterable feed.

This squashes prior local development history (kept privately on a separate
branch) into a single clean starting point for the public repository.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-01 21:55:02 +01:00