v0.5.1: automatic Trash purge (cron-less background thread)
Trashed artifacts past a ~3-day retention window are now purged automatically. - storage.purge_expired_trash(): removes trash older than TRASH_RETENTION (all users), tolerant of already-gone rows (concurrent-safe). - spawn_trash_purge(): runs it on a daemon thread (manages/closes its own DB connection), triggered by each delete so it's cron-less and off the request thread. Inline fallback when AGENTFEED_ASYNC_PURGE is off (tests). - Wired into both delete paths (API + session). Manual Empty Trash unchanged. - Tests: purge removes expired-only; delete triggers the purge. 152 passing; live delete verified with the thread enabled (204, server healthy). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
4254164bef
commit
e5fa1b5e86
8 changed files with 87 additions and 4 deletions
13
CHANGELOG.md
13
CHANGELOG.md
|
|
@ -9,6 +9,16 @@ former name, "Daybook".)
|
||||||
|
|
||||||
_Nothing yet._
|
_Nothing yet._
|
||||||
|
|
||||||
|
## [0.5.1] — 2026-09-07
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- **Automatic Trash purge.** Trashed artifacts older than a ~3-day retention
|
||||||
|
window are now permanently removed automatically. It's cron-less: a delete
|
||||||
|
triggers a sweep that runs on a **background thread** (off the request, so
|
||||||
|
the click never blocks), tolerant of concurrent runs. Manual "Empty Trash"
|
||||||
|
still works for immediate reclaim. Controlled by `AGENTFEED_ASYNC_PURGE`
|
||||||
|
(on by default; tests run it inline).
|
||||||
|
|
||||||
## [0.5.0] — 2026-09-07
|
## [0.5.0] — 2026-09-07
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
@ -185,7 +195,8 @@ First tagged release.
|
||||||
- Dockerised (Gunicorn + whitenoise), SQLite, `git`-based deploy with a
|
- Dockerised (Gunicorn + whitenoise), SQLite, `git`-based deploy with a
|
||||||
runbook (`deploy/cloud_agent_deploy.md`) and health check.
|
runbook (`deploy/cloud_agent_deploy.md`) and health check.
|
||||||
|
|
||||||
[Unreleased]: https://git.cygnus.systems/peter/daybook/compare/v0.5.0...HEAD
|
[Unreleased]: https://git.cygnus.systems/peter/daybook/compare/v0.5.1...HEAD
|
||||||
|
[0.5.1]: https://git.cygnus.systems/peter/daybook/compare/v0.5.0...v0.5.1
|
||||||
[0.5.0]: https://git.cygnus.systems/peter/daybook/compare/v0.4.1...v0.5.0
|
[0.5.0]: https://git.cygnus.systems/peter/daybook/compare/v0.4.1...v0.5.0
|
||||||
[0.4.1]: https://git.cygnus.systems/peter/daybook/compare/v0.4.0...v0.4.1
|
[0.4.1]: https://git.cygnus.systems/peter/daybook/compare/v0.4.0...v0.4.1
|
||||||
[0.4.0]: https://git.cygnus.systems/peter/daybook/compare/v0.3.2...v0.4.0
|
[0.4.0]: https://git.cygnus.systems/peter/daybook/compare/v0.3.2...v0.4.0
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ The core product principle is: agents publish content, not files. The server own
|
||||||
- Exception to the above: an artifact can be explicitly opted into public sharing (per-artifact, owner-toggled from the feed UI) via a high-entropy `share_token`, served unauthenticated at `GET /s/{share_token}`. Not exposed via the ingest API yet — UI-only. Turning sharing off immediately invalidates the old link. Images embedded in a shared artifact are also served unauthenticated while it's shared, so inline content doesn't break for a link recipient.
|
- Exception to the above: an artifact can be explicitly opted into public sharing (per-artifact, owner-toggled from the feed UI) via a high-entropy `share_token`, served unauthenticated at `GET /s/{share_token}`. Not exposed via the ingest API yet — UI-only. Turning sharing off immediately invalidates the old link. Images embedded in a shared artifact are also served unauthenticated while it's shared, so inline content doesn't break for a link recipient.
|
||||||
- User accounts can be created from Settings → Users (superuser-only in-app page) or via Django admin. Regular users are not staff and cannot access `/admin/`. Creating a user via Settings → Users also issues their API token immediately, shown once.
|
- User accounts can be created from Settings → Users (superuser-only in-app page) or via Django admin. Regular users are not staff and cannot access `/admin/`. Creating a user via Settings → Users also issues their API token immediately, shown once.
|
||||||
- `TagColor` (label color overrides) stays global/shared across all users.
|
- `TagColor` (label color overrides) stays global/shared across all users.
|
||||||
- Deleting an artifact is a **soft-delete to Trash** (`deleted_at`), not a hard removal — recoverable via the "Deleted" sidebar folder (undelete), off-quota and eviction-protected while trashed, and purged permanently via "Empty Trash". `DELETE /api/artifacts/{id}` is soft too (idempotent); trashed artifacts are hidden from all API reads. A **Keep** sidebar folder is a view over `is_kept`. Both folders hide when empty. (Auto purge-on-delete is a planned fast-follow; currently purge is manual.)
|
- Deleting an artifact is a **soft-delete to Trash** (`deleted_at`), not a hard removal — recoverable via the "Deleted" sidebar folder (undelete), off-quota and eviction-protected while trashed, and purged permanently via "Empty Trash". `DELETE /api/artifacts/{id}` is soft too (idempotent); trashed artifacts are hidden from all API reads. A **Keep** sidebar folder is a view over `is_kept`. Both folders hide when empty. Trash auto-purges after a ~3-day retention window via a cron-less background thread triggered by deletes (`AGENTFEED_ASYNC_PURGE`, on by default); "Empty Trash" purges immediately.
|
||||||
- Use random 8-character artifact IDs.
|
- Use random 8-character artifact IDs.
|
||||||
- Start as a plain Django project; add Docker after core ingest and viewer work.
|
- Start as a plain Django project; add Docker after core ingest and viewer work.
|
||||||
- Preserve original markdown as `.md` alongside rendered `.html`.
|
- Preserve original markdown as `.md` alongside rendered `.html`.
|
||||||
|
|
|
||||||
|
|
@ -7,9 +7,11 @@ SUM rather than a stat of every file.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
|
import threading
|
||||||
from datetime import timedelta
|
from datetime import timedelta
|
||||||
|
|
||||||
from django.conf import settings
|
from django.conf import settings
|
||||||
|
from django.db import connection
|
||||||
from django.db.models import F, Sum
|
from django.db.models import F, Sum
|
||||||
from django.utils import timezone
|
from django.utils import timezone
|
||||||
|
|
||||||
|
|
@ -154,6 +156,45 @@ def purge_trashed(user):
|
||||||
return len(trashed)
|
return len(trashed)
|
||||||
|
|
||||||
|
|
||||||
|
TRASH_RETENTION = timedelta(days=3)
|
||||||
|
|
||||||
|
|
||||||
|
def purge_expired_trash(now=None):
|
||||||
|
"""Permanently remove trashed artifacts older than the retention window
|
||||||
|
(all users). Tolerant of already-gone rows so concurrent purges are safe.
|
||||||
|
Returns the number purged."""
|
||||||
|
now = now or timezone.now()
|
||||||
|
cutoff = now - TRASH_RETENTION
|
||||||
|
expired = list(Artifact.objects.filter(deleted_at__isnull=False, deleted_at__lt=cutoff))
|
||||||
|
purged = 0
|
||||||
|
for artifact in expired:
|
||||||
|
try:
|
||||||
|
delete_artifact_files(artifact)
|
||||||
|
artifact.delete()
|
||||||
|
purged += 1
|
||||||
|
except Exception:
|
||||||
|
# Another concurrent purge got there first, or a file's already
|
||||||
|
# gone — a purge error must never surface to anyone.
|
||||||
|
pass
|
||||||
|
return purged
|
||||||
|
|
||||||
|
|
||||||
|
def spawn_trash_purge():
|
||||||
|
"""Run purge_expired_trash off the request thread (cron-less: triggered by
|
||||||
|
deletes). Falls back to inline when async is disabled (tests)."""
|
||||||
|
if not getattr(settings, "AGENTFEED_ASYNC_PURGE", True):
|
||||||
|
purge_expired_trash()
|
||||||
|
return
|
||||||
|
|
||||||
|
def _run():
|
||||||
|
try:
|
||||||
|
purge_expired_trash()
|
||||||
|
finally:
|
||||||
|
connection.close() # threads get their own connection; don't leak it
|
||||||
|
|
||||||
|
threading.Thread(target=_run, daemon=True).start()
|
||||||
|
|
||||||
|
|
||||||
ASSET_ORPHAN_GRACE = timedelta(hours=24)
|
ASSET_ORPHAN_GRACE = timedelta(hours=24)
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -45,6 +45,7 @@ from .storage import (
|
||||||
disk_free_bytes,
|
disk_free_bytes,
|
||||||
make_room_for,
|
make_room_for,
|
||||||
purge_trashed,
|
purge_trashed,
|
||||||
|
spawn_trash_purge,
|
||||||
sweep_orphan_assets,
|
sweep_orphan_assets,
|
||||||
total_allocated_bytes,
|
total_allocated_bytes,
|
||||||
user_quota_bytes,
|
user_quota_bytes,
|
||||||
|
|
@ -305,6 +306,7 @@ class ArtifactDetailView(APIView):
|
||||||
if artifact.deleted_at is None:
|
if artifact.deleted_at is None:
|
||||||
artifact.deleted_at = timezone.now()
|
artifact.deleted_at = timezone.now()
|
||||||
artifact.save(update_fields=["deleted_at"])
|
artifact.save(update_fields=["deleted_at"])
|
||||||
|
spawn_trash_purge() # sweep expired trash off the request thread
|
||||||
return Response(status=204)
|
return Response(status=204)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -669,6 +671,7 @@ def delete_artifact(request, artifact_id):
|
||||||
if artifact.deleted_at is None:
|
if artifact.deleted_at is None:
|
||||||
artifact.deleted_at = timezone.now()
|
artifact.deleted_at = timezone.now()
|
||||||
artifact.save(update_fields=["deleted_at"])
|
artifact.save(update_fields=["deleted_at"])
|
||||||
|
spawn_trash_purge() # sweep expired trash off the request thread
|
||||||
return JsonResponse({"id": artifact.id, "deleted": True})
|
return JsonResponse({"id": artifact.id, "deleted": True})
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
# Single source of truth for the app version. Bump this, commit, then tag the
|
# Single source of truth for the app version. Bump this, commit, then tag the
|
||||||
# commit `vX.Y.Z` and deploy. Surfaced on Settings -> About and in
|
# commit `vX.Y.Z` and deploy. Surfaced on Settings -> About and in
|
||||||
# GET /api/capabilities ("version").
|
# GET /api/capabilities ("version").
|
||||||
__version__ = "0.5.0"
|
__version__ = "0.5.1"
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,10 @@ AGENTFEED_KEY_PROVISIONING_API = os.environ.get(
|
||||||
"AGENTFEED_KEY_PROVISIONING_API", ""
|
"AGENTFEED_KEY_PROVISIONING_API", ""
|
||||||
).lower() in {"1", "true", "yes", "on"}
|
).lower() in {"1", "true", "yes", "on"}
|
||||||
|
|
||||||
|
# Purge expired Trash off the request thread (triggered by deletes, no cron).
|
||||||
|
# Tests override this to run inline so purge behavior is deterministic.
|
||||||
|
AGENTFEED_ASYNC_PURGE = True
|
||||||
|
|
||||||
# Needed when running behind a TLS-terminating reverse proxy (Apache/Nginx/etc.):
|
# Needed when running behind a TLS-terminating reverse proxy (Apache/Nginx/etc.):
|
||||||
# Django otherwise thinks every request is plain HTTP, which breaks CSRF checks
|
# Django otherwise thinks every request is plain HTTP, which breaks CSRF checks
|
||||||
# on login/POST. The proxy must forward X-Forwarded-Proto: https.
|
# on login/POST. The proxy must forward X-Forwarded-Proto: https.
|
||||||
|
|
|
||||||
|
|
@ -17,6 +17,7 @@ from artifacts.models import ApiKey, Artifact
|
||||||
User = get_user_model()
|
User = get_user_model()
|
||||||
|
|
||||||
|
|
||||||
|
@override_settings(AGENTFEED_ASYNC_PURGE=False) # DELETE triggers purge; run it inline in tests
|
||||||
class ArtifactAPITestCase(APITestCase):
|
class ArtifactAPITestCase(APITestCase):
|
||||||
"""Exercises the ingest/list/detail API against a real (temp) filesystem
|
"""Exercises the ingest/list/detail API against a real (temp) filesystem
|
||||||
and a real (test) database, the way an agent posting over HTTP would."""
|
and a real (test) database, the way an agent posting over HTTP would."""
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,6 @@
|
||||||
import shutil
|
import shutil
|
||||||
import tempfile
|
import tempfile
|
||||||
|
from datetime import timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from django.contrib.auth import get_user_model
|
from django.contrib.auth import get_user_model
|
||||||
|
|
@ -8,11 +9,13 @@ from django.urls import reverse
|
||||||
from django.utils import timezone
|
from django.utils import timezone
|
||||||
|
|
||||||
from artifacts.models import Artifact
|
from artifacts.models import Artifact
|
||||||
from artifacts.storage import user_usage_bytes
|
from artifacts.storage import purge_expired_trash, user_usage_bytes
|
||||||
|
|
||||||
User = get_user_model()
|
User = get_user_model()
|
||||||
|
|
||||||
|
|
||||||
|
# Run the purge inline (no background thread) so its effects are deterministic.
|
||||||
|
@override_settings(AGENTFEED_ASYNC_PURGE=False)
|
||||||
class TrashKeepViewTestCase(TestCase):
|
class TrashKeepViewTestCase(TestCase):
|
||||||
@classmethod
|
@classmethod
|
||||||
def setUpClass(cls):
|
def setUpClass(cls):
|
||||||
|
|
@ -105,3 +108,23 @@ class TrashKeepViewTestCase(TestCase):
|
||||||
page = self.client.get("/?folder=keep").content.decode()
|
page = self.client.get("/?folder=keep").content.decode()
|
||||||
self.assertIn("KeptOne", page)
|
self.assertIn("KeptOne", page)
|
||||||
self.assertNotIn("NormalOne", page)
|
self.assertNotIn("NormalOne", page)
|
||||||
|
|
||||||
|
def test_purge_expired_trash_removes_old_only(self):
|
||||||
|
old = self.make()
|
||||||
|
recent = self.make()
|
||||||
|
Artifact.objects.filter(pk=old.id).update(deleted_at=timezone.now() - timedelta(days=4))
|
||||||
|
Artifact.objects.filter(pk=recent.id).update(deleted_at=timezone.now())
|
||||||
|
purged = purge_expired_trash()
|
||||||
|
self.assertEqual(purged, 1)
|
||||||
|
self.assertFalse(Artifact.objects.filter(pk=old.id).exists())
|
||||||
|
self.assertTrue(Artifact.objects.filter(pk=recent.id).exists())
|
||||||
|
|
||||||
|
def test_delete_triggers_expired_purge(self):
|
||||||
|
old = self.make()
|
||||||
|
Artifact.objects.filter(pk=old.id).update(deleted_at=timezone.now() - timedelta(days=4))
|
||||||
|
fresh = self.make()
|
||||||
|
# Deleting fresh runs the purge (inline here) -> the expired one goes,
|
||||||
|
# the freshly-trashed one stays.
|
||||||
|
self.client.post(reverse("artifact-delete", args=[fresh.id]))
|
||||||
|
self.assertFalse(Artifact.objects.filter(pk=old.id).exists())
|
||||||
|
self.assertTrue(Artifact.objects.filter(pk=fresh.id, deleted_at__isnull=False).exists())
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue