Convert uploaded .xlsx files to csv artifacts server-side

Uploading a .xlsx file no longer becomes a generic downloadable binary -
it's read server-side with openpyxl and converted to a csv artifact,
rendering in the same interactive grid as a native CSV post. Detected by
content type or .xlsx extension (some clients send generic/incorrect
content types for multipart uploads). Only the active sheet converts;
formula cells use whatever cached result the spreadsheet software last
saved (openpyxl doesn't evaluate formulas itself). Corrupt/unreadable
files return 400.

Since the existing posting script already forwards any non-text file via
multipart with no special-casing, `daybook_post.py balance_sheet.xlsx`
already works with zero script changes - verified live.

Two real bugs found and fixed while building this, both now covered by
regression tests:
- {**a_querydict, ...} silently wrapped every value (including title) in
  a list - Python's ** spread on a dict subclass bypasses QueryDict's
  __getitem__ override and exposes its internal one-list-per-key storage
  directly. Fixed by building the delegated dict explicitly via .get().
- Pre-setting None for absent optional fields (e.g. source_prompt) defeated
  create_text_artifact's own data.get(field, default) calls, since .get()
  only falls through to its default when the key is *missing*, not when
  the value is None - caused a NOT NULL constraint failure. Fixed by only
  setting keys that were actually provided.
- csv.writer's default "\r\n" terminator was getting doubled by
  Path.write_text()'s own newline translation on Windows, producing a
  blank line between every row. Fixed with lineterminator="\n".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
peter 2026-09-02 16:29:39 +01:00
parent 4145c37dfe
commit 4b5ec09ae2
6 changed files with 232 additions and 6 deletions

View file

@ -0,0 +1,60 @@
from io import BytesIO
from types import SimpleNamespace
import openpyxl
from django.test import SimpleTestCase
from artifacts.views import XLSX_CONTENT_TYPE, is_xlsx_upload, xlsx_to_csv_text
class IsXlsxUploadTestCase(SimpleTestCase):
def test_matches_exact_content_type(self):
upload = SimpleNamespace(name="whatever.bin")
self.assertTrue(is_xlsx_upload(upload, XLSX_CONTENT_TYPE))
def test_matches_by_extension_regardless_of_content_type(self):
upload = SimpleNamespace(name="sheet.xlsx")
self.assertTrue(is_xlsx_upload(upload, "application/octet-stream"))
self.assertTrue(is_xlsx_upload(upload, ""))
def test_extension_match_is_case_insensitive(self):
upload = SimpleNamespace(name="SHEET.XLSX")
self.assertTrue(is_xlsx_upload(upload, ""))
def test_rejects_other_files(self):
upload = SimpleNamespace(name="report.pdf")
self.assertFalse(is_xlsx_upload(upload, "application/pdf"))
class XlsxToCsvTextTestCase(SimpleTestCase):
def make_xlsx(self, rows):
workbook = openpyxl.Workbook()
sheet = workbook.active
for row in rows:
sheet.append(row)
buffer = BytesIO()
workbook.save(buffer)
buffer.seek(0)
return buffer
def test_converts_active_sheet_to_csv(self):
upload = self.make_xlsx([["Item", "Q1", "Q2"], ["Cash", 1000, 1200]])
content = xlsx_to_csv_text(upload)
self.assertEqual(content, "Item,Q1,Q2\nCash,1000,1200\n")
def test_uses_single_line_terminator_not_doubled(self):
"""Regression test: csv.writer's default "\\r\\n" gets a second
newline translation from Path.write_text()'s text-mode handling when
this string is later saved to disk, producing a blank line between
every row on Windows - lineterminator="\\n" avoids embedding "\\r\\n"
in the string in the first place, leaving the OS-level text-mode
write as the only place "\\n" becomes "\\r\\n"."""
upload = self.make_xlsx([["a"], ["b"], ["c"]])
content = xlsx_to_csv_text(upload)
self.assertNotIn("\r", content)
self.assertEqual(content.count("\n"), 3)
def test_empty_cells_become_empty_strings(self):
upload = self.make_xlsx([["a", "b"], ["x", None]])
content = xlsx_to_csv_text(upload)
self.assertEqual(content, "a,b\nx,\n")