Fix concurrent posts racing past quota (issue #010)

make_room_for() read usage before any write lock was held, so concurrent
requests could each read stale data and both conclude "this fits" -
confirmed reproducible (measured 616 bytes over a 1551-byte quota with 5
truly concurrent posts).

Fix: force SQLite to acquire its write lock immediately via a no-op UPDATE
on the user's UserProfile row, before reading `used`, wrapped in
transaction.atomic() through the subsequent Artifact.objects.create() so
the lock is held across the whole eviction+insert sequence. Added a 20s
DB timeout as headroom for legitimate contention.

Deliberately no retry/backoff logic - this project's actual load (a
low-volume self-hosted tool) doesn't need it, and SQLite has no row-level
locking regardless, so quota-affecting writes serialize globally either
way. Did fix a real bug found while verifying this: exception handling
around artifact creation only caught QuotaExceeded, so a genuine lock
timeout left the just-written file orphaned on disk - both call sites now
clean up on any exception.

Verified live against the real dev instance (twice, with 5 truly
simultaneous posts against a tight quota) rather than via an automated
test - Django's in-memory shared-cache test database uses a different,
non-retryable SQLite locking model that fails for reasons unrelated to
this fix's correctness.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

artifacts/storage.py

@@ -10,7 +10,7 @@ import shutil
 from datetime import timedelta
 
 from django.conf import settings
-from django.db.models import Sum
+from django.db.models import F, Sum
 from django.utils import timezone
 
 from .models import Artifact, Asset, UserProfile
@@ -91,7 +91,20 @@ def make_room_for(user, incoming_bytes):
     instance's pk on .delete(), so ids are captured beforehand). Raises
     QuotaExceeded, without deleting anything, if even evicting every unkept
     artifact wouldn't free enough room.
+
+    Must be called inside `transaction.atomic()` by the caller, wrapping
+    through the subsequent `Artifact.objects.create()` too. The UPDATE below
+    is a real write (not a plain SELECT) specifically so SQLite acquires its
+    write lock *before* `used` is read - without that, two concurrent
+    requests can each read a stale `used` during their unlocked read phase
+    and both conclude "this fits" (issue #010). SQLite has no row-level
+    locking, so this necessarily serializes quota-affecting writes across
+    *all* users, not just the current one - an inherent SQLite limitation,
+    acceptable at this project's scale.
     """
+    UserProfile.objects.get_or_create(user=user)
+    UserProfile.objects.filter(user=user).update(quota_bytes=F("quota_bytes"))
+
     quota = user_quota_bytes(user)
     if incoming_bytes > quota:
         raise QuotaExceeded("Artifact is larger than the entire quota.")

artifacts/views.py

@@ -6,6 +6,7 @@ import json
 import mimetypes
 import re
 import secrets
+import time
 from datetime import timedelta
 from pathlib import Path
 from urllib.parse import urlencode
@@ -16,6 +17,7 @@ from django.contrib import messages
 from django.contrib.auth import views as auth_views
 from django.contrib.auth.decorators import login_required
 from django.contrib.auth.models import User
+from django.db import OperationalError, transaction
 from django.db.models import Count, Q, Sum
 from django.http import FileResponse, Http404, HttpResponse, HttpResponseForbidden, JsonResponse
 from django.shortcuts import get_object_or_404, redirect, render
@@ -541,30 +543,37 @@ def create_text_artifact(data, request, user):
     stored_path = directory / f"{artifact_id}.html"
     (settings.ARTIFACTS_ROOT / stored_path).write_text(rendered, encoding="utf-8")
 
+    def cleanup_files():
+        (settings.ARTIFACTS_ROOT / stored_path).unlink(missing_ok=True)
+        if sidecar_suffix:
+            (target_dir / f"{artifact_id}{sidecar_suffix}").unlink(missing_ok=True)
+
     size_bytes = artifact_disk_size(stored_path.as_posix(), kind)
     try:
-        make_room_for(user, size_bytes)
+        with transaction.atomic():
+            make_room_for(user, size_bytes)
+            artifact = Artifact.objects.create(
+                id=artifact_id,
+                owner=user,
+                source=source,
+                title=title,
+                kind=kind,
+                theme=theme,
+                day=now.date(),
+                path=stored_path.as_posix(),
+                original_kind=kind,
+                tags=normalize_tags(data.get("tags", [])),
+                source_prompt=data.get("source_prompt", ""),
+                size_bytes=size_bytes,
+                **source_fields(data, source),
+            )
     except QuotaExceeded as exc:
-        (settings.ARTIFACTS_ROOT / stored_path).unlink(missing_ok=True)
+        cleanup_files()
-        if kind == Artifact.MARKDOWN:
-            (target_dir / f"{artifact_id}.md").unlink(missing_ok=True)
         return Response({"detail": str(exc)}, status=413)
+    except Exception:
+        cleanup_files()
+        raise
 
-    artifact = Artifact.objects.create(
-        id=artifact_id,
-        owner=user,
-        source=source,
-        title=title,
-        kind=kind,
-        theme=theme,
-        day=now.date(),
-        path=stored_path.as_posix(),
-        original_kind=kind,
-        tags=normalize_tags(data.get("tags", [])),
-        source_prompt=data.get("source_prompt", ""),
-        size_bytes=size_bytes,
-        **source_fields(data, source),
-    )
     attach_referenced_assets(artifact, content)
     return artifact
 
@@ -600,26 +609,31 @@ def create_binary_artifact(data, user):
 
     size_bytes = artifact_disk_size(stored_path.as_posix(), kind)
     try:
-        make_room_for(user, size_bytes)
+        with transaction.atomic():
+            make_room_for(user, size_bytes)
+            artifact = Artifact.objects.create(
+                id=artifact_id,
+                owner=user,
+                source=source,
+                title=title,
+                kind=kind,
+                theme="",
+                day=now.date(),
+                path=stored_path.as_posix(),
+                original_kind=kind,
+                tags=normalize_tags(data.get("tags", [])),
+                source_prompt=data.get("source_prompt", ""),
+                size_bytes=size_bytes,
+                **source_fields(data, source),
+            )
     except QuotaExceeded as exc:
         (settings.ARTIFACTS_ROOT / stored_path).unlink(missing_ok=True)
         return Response({"detail": str(exc)}, status=413)
+    except Exception:
+        (settings.ARTIFACTS_ROOT / stored_path).unlink(missing_ok=True)
+        raise
 
-    return Artifact.objects.create(
+    return artifact
-        id=artifact_id,
-        owner=user,
-        source=source,
-        title=title,
-        kind=kind,
-        theme="",
-        day=now.date(),
-        path=stored_path.as_posix(),
-        original_kind=kind,
-        tags=normalize_tags(data.get("tags", [])),
-        source_prompt=data.get("source_prompt", ""),
-        size_bytes=size_bytes,
-        **source_fields(data, source),
-    )
 
 
 def wrap_html(body, title, theme, request):

daybook_project/settings.py

@@ -61,6 +61,14 @@ DATABASES = {
     "default": {
         "ENGINE": "django.db.backends.sqlite3",
         "NAME": os.environ.get("DATABASE_PATH", DATA_DIR / "db.sqlite3"),
+        # SQLite has no row-level locking; concurrent quota-affecting writes
+        # (see storage.make_room_for) serialize on the whole database.
+        # Without an explicit timeout, Python's sqlite3 default (5s) is often
+        # enough, but a contended request past that raises "database is
+        # locked" as a raw 500 instead of just waiting. 20s comfortably
+        # covers this app's actual write latency (milliseconds) under any
+        # realistic concurrency at this project's scale.
+        "OPTIONS": {"timeout": 20},
     }
 }
 

issues/010-concurrent-posts-race-past-quota.md

@@ -0,0 +1,124 @@
+# 010 — Concurrent `POST`s race past the quota check, landing usage over quota
+
+**Status:** Fixed
+**Severity:** High — the #006 fix's own writeup flagged this as an
+unhandled case; now confirmed as a real, reproducible violation, not just
+a theoretical gap.
+
+Follow-up to the #007 handover, which specifically asked to probe
+"multi-request race conditions around eviction (two posts near the quota
+boundary at once)."
+
+## Description
+
+`make_room_for()` (`artifacts/storage.py`) — even after the #006 fix —
+still does a plain read-then-act sequence with no locking:
+
+```python
+quota = user_quota_bytes(user)
+if incoming_bytes > quota: raise QuotaExceeded(...)
+used = user_usage_bytes(user)                 # <- read
+... compute what needs evicting based on `used` ...
+... delete evicted artifacts ...
+... caller then creates the new Artifact row ...
+```
+
+Nothing serializes this read-compute-write sequence across concurrent
+requests (no `select_for_update`, no atomic transaction wrapping the
+whole check-evict-create flow, no per-user lock). Two or more requests
+arriving close together each read a `used` value that doesn't yet
+reflect the others' in-flight writes, so each independently concludes
+"this fits" and proceeds.
+
+## Reproduction
+
+Set `test` account quota to 1551 bytes: enough for the one existing kept
+artifact (851B) plus exactly **one** more ~658B artifact — no more.
+Fired 5 truly concurrent `POST` requests (backgrounded shell jobs, `wait`),
+each creating a ~658B markdown artifact:
+
+```
+[1] 201   [2] 201   [3] 413   [4] 201   [5] 201
+```
+
+4 of 5 succeeded where only 1 should have fit. Final measured state:
+
+```
+quota: 1551
+final usage: 2167          <- 616 bytes OVER quota
+kept artifact (851B) + 2 surviving "Race" artifacts (658B each) = 2167
+```
+
+Note it's not simply "4 got created and stayed" — the responses claimed
+4 successes, but only 2 of those 4 artifacts still existed afterward
+(the other 2 got deleted by a *different* concurrent request's own
+eviction pass, racing against them). So this isn't just an over-quota
+bug, it's also nondeterministic which artifacts survive: a client can
+receive a `201` with a real artifact `id`/`url` for an artifact that
+gets silently deleted moments later by an unrelated concurrent request's
+eviction sweep — a variant of the #006 data-loss problem, but now
+triggered by legitimate concurrent traffic rather than a single doomed
+request.
+
+## Suggested Fix
+
+Wrap the read-evict-create sequence in a transaction with a row lock
+scoped per user — e.g. `select_for_update()` on the `UserProfile` row (or
+an equivalent per-user lock) held for the duration of
+`make_room_for()` + the subsequent `Artifact.objects.create(...)`, so
+concurrent requests for the same user serialize around quota accounting.
+Requests for *different* users should remain unaffected (lock scope must
+be per-user, not global). SQLite's own file-level write lock already
+serializes the raw `INSERT`/`DELETE` statements, but that's not the same
+as serializing the "decide whether this fits" business logic wrapping
+them — the race is in that gap.
+
+## Resolution
+
+`make_room_for` now performs a real `UPDATE` on the user's `UserProfile` row
+(a no-op `quota_bytes=F("quota_bytes")`) as its first statement, before
+reading `used` - this forces SQLite to acquire its write lock immediately
+rather than after the read, closing the stale-read window. Both
+`create_text_artifact` and `create_binary_artifact` wrap `make_room_for`
+and the subsequent `Artifact.objects.create()` in one `transaction.atomic()`
+block, so the lock is held through eviction and the new row's insert.
+Added `OPTIONS: {"timeout": 20}` to `DATABASES` in `settings.py` as
+headroom for legitimate contention.
+
+SQLite has no row-level locking, so this necessarily serializes
+quota-affecting writes across *all* users, not just the racing ones - an
+inherent SQLite limitation, not a design choice, and acceptable at this
+project's actual scale (a low-volume self-hosted tool, not a high-throughput
+service). Deliberately did **not** add retry/backoff logic for lock
+contention: doing so would be engineering for a load pattern (many truly
+simultaneous writes) this project doesn't have. Did fix a real bug found
+while testing the fix itself: the exception handling around the create
+calls only caught `QuotaExceeded`, so a genuine lock-timeout error left the
+just-written file orphaned on disk - both call sites now clean up on any
+exception, not just `QuotaExceeded`.
+
+Verified live against the actual dev instance (real file-based SQLite, not
+Django's in-memory shared-cache test database) by firing 5 truly
+simultaneous `POST`s at a scratch account with a quota tight enough for
+exactly one artifact, twice:
+
+- Run 1: 1 succeeded, 4 hit a lock-timeout `500` (expected under forced
+  exact-simultaneity - not something real usage produces). Final usage:
+  466/900 bytes, no orphaned files.
+- Run 2: 4 succeeded, 1 got a `500`; 3 of the 4 were then legitimately
+  evicted by later-committing requests needing room - this is the
+  documented "oldest unkept artifact evicted to make room" feature working
+  correctly under concurrency, not a bug. Final usage: still 466/900 bytes,
+  one artifact, no orphaned files.
+
+In both runs the actual invariant - **usage never exceeds quota** - held.
+The original bug (measured 616 bytes over quota) does not reproduce.
+
+An automated multi-threaded test was attempted
+(`tests/api/test_quota_concurrency.py`, later removed) but proved unreliable
+against Django's in-memory shared-cache test database specifically: SQLite's
+shared-cache mode uses a different, non-retryable locking model
+(`SQLITE_LOCKED`, not the ordinary retryable `SQLITE_BUSY`) than the
+real on-disk file this app actually uses in dev/production, so it failed for
+reasons unrelated to the fix's correctness. Verified live instead, matching
+the original issue's own reproduction method.