Template catalog: versions, preview, test send, typed context

[alexeygrigorev]

Template catalog: versions, preview, test send, typed context

Status: pending
Tags: P1, backend, api, transactional, email, docs
Depends on: None
Blocks: — (downstream unification-plan adoption work depends on this; tracked in community-base, not in this repo)

Source: unification plan issue R1.3 (community-base docs/plan/phase-1.md). Relay is the unification plan's email and jobs service; this issue sits on the plan's external critical path and may proceed independently.

Scope

• Add immutable published versions to the transactional template catalog. Each client template keeps one editable draft; PUT /api/transactional/templates/{key} keeps writing the draft; publishing snapshots the draft into a new immutable version.
• Template format: markdown body with YAML frontmatter (subject required; required_context and category optional). Rendering ports the AISL pipeline: render the context into the source with the Django template engine, convert markdown to sanitized HTML, and wrap with the shared HTML wrapper and footer (port of _render_template_with_footer and its wrapper). Plain text derives from the same source.
• Direct-HTML drafts stay supported: html_body/text_body in PUT remain a first-class content form for templates provisioned through the API (CMP provisioning uses it). Markdown and direct HTML are two content forms in one catalog; the HTML form is not a legacy path.
• Sends resolve template_key plus optional template_version (default: latest published). required_context is validated against the resolved version on every transactional send path; missing keys fail the send with a clear error before any contact, message, or queue work.
• New client-Bearer endpoints under the authenticated template scope: POST /api/transactional/templates/{key}/publish, GET /api/transactional/templates/{key}/versions, POST /api/transactional/templates/{key}/preview (returns subject, HTML, text for the draft or a version), and POST /api/transactional/templates/{key}/test-send (sends a version to an allowlisted staff address).
• Management command python manage.py import_templates --dir <path> --client <slug> creates drafts from a directory of AISL-format markdown files; the plan uses it in phase 6 and DTC uses it for its templates.
• Datamailer is pre-production: no compatibility shims, aliases, or duplicated endpoints. The AISL markdown import is an explicitly scoped client-integration requirement, built as a first-class Datamailer command and format.

Codebase notes for the implementer

• EmailTemplate (mailing/models.py) already has required_context (both plain strings and {name, description} items are accepted by normalize_required_context), example_context, default_sender_id, is_transactional, is_active. It has no version, category, or markdown-source fields; add what the design needs.
• get_transactional_template in mailing/services/transactional.py currently resolves the mutable template row for all three send paths (single send, recipient-list send, transient-list send). After this change it resolves published versions; a template with no published version must fail the send instead of silently using the draft. validate_transactional_send_payload gains template_version.
• validate_template_context (mailing/services/transactional_catalog.py) already fails sends on missing required_context with context.<name>: required (400, before contact creation). Keep that error contract; make it version-aware.
• TransactionalMessage already snapshots rendered subject/html/text at enqueue time and the sender worker sends the stored bodies, so the SQS queue payload and worker stay unchanged; persist the resolved template_version on the message for auditability.
• The frontmatter category is template metadata. It is not the send-level category_tag (subscription preference category) that sends already accept; do not conflate them.
• No markdown/YAML/frontmatter libraries are in pyproject.toml yet; add what the renderer and importer need.
• The route api/transactional/templates/<slug:template_key> accepts AISL-style keys containing underscores, so imported file stems work as keys unchanged.
• New endpoints must be registered in the in-app docs (build_openapi_spec and the endpoint map in mailing/services/api_docs.py); test_api_docs_endpoint_reference_matches_openapi_paths enforces that the endpoint reference matches OpenAPI paths.
• There is no version backfill for pre-existing templates. Sandbox environments re-run seed_demo_data and scripts/upsert_cmp_templates.py; update both to publish after upsert so existing flows keep sending.
• The staff operator UI already previews drafts at /templates/ and /templates/<id>/ (render_preview in transactional_catalog.py); keep it working and reuse service functions where sensible. It is a different surface from the new API preview.

Donor references (read first)

• docs/api.md "Transactional Email API" and the email_templates table in docs/data-model.md
• ~/git/dtc-website/_docs/specs/05-events-registration-email.md, section "Relay-owned email templates"
• ~/git/ai-shipping-labs/email_app/email_templates/ (the markdown format that must import cleanly), email_app/services/email_service.py _render_template_with_footer, and email_app/services/preview_contexts.py

Verification (from the plan issue)

• Import ~/git/ai-shipping-labs/email_app/email_templates/ into the sandbox with the command -> 50 drafts created, zero errors.
• Preview of event_registration with the AISL preview context (email_app/services/preview_contexts.py) renders without missing keys.

Acceptance Criteria

• Publish: POST /api/transactional/templates/{key}/publish snapshots the current draft as immutable version N (1-based per client and key) storing subject, the canonical content source (markdown body and/or html/text), required_context, category, and the publish timestamp. Publishing a draft whose content matches the latest published version returns that version without creating a new row.
• Immutability: no API, management command, or admin action can modify a published version; PUT and every later republish touch only the draft.
• GET /api/transactional/templates/{key}/versions lists versions with version number, subject, category, published timestamp, and which is latest; GET /api/transactional/templates/{key} returns the draft plus latest_published_version.
• Send resolution: POST /api/transactional/send, the recipient-list transactional send, and the transient-list transactional send accept optional template_version; omitted means latest published; an unknown version returns 404; a template with no published version returns 409 with a no_published_version error. A failed resolution creates no contact, no message row, and no queue payload.
• Required context: every send path validates the resolved version's required_context and rejects with 400 naming each missing context.<name> before creating the contact or message.
• Auditability: TransactionalMessage stores the resolved template_version, and the send response and dry-run response include it next to template_key.
• Preview: POST /api/transactional/templates/{key}/preview accepts {context, template_version?} (draft when no version is given) and returns {subject, html, text}; missing required context keys return a 400 listing them. Preview and send share one render code path: the same version and context produce identical subject/html/text through preview and dry_run send.
• Test send: POST /api/transactional/templates/{key}/test-send sends the given version (default: latest published) to an address matched against a deny-by-default staff-address allowlist; a non-allowlisted address is rejected with a clear error and nothing is enqueued; an allowed request goes through the normal enqueue path and marks the message as a test send in metadata.
• Import: python manage.py import_templates --dir <path> --client <slug> creates or updates one draft per *.md file (key = file stem), is idempotent on re-run, treats required_context and category as optional frontmatter, and exits non-zero with a clear message on a malformed file.
• Plan verification: importing /home/alexey/git/ai-shipping-labs/email_app/email_templates/ creates 50 drafts with zero errors, and previewing event_registration with its AISL preview context renders without missing keys.
• docs/api.md documents versions, publish, preview, test send, template_version on sends, and the frontmatter format. The in-app docs at /api-docs/ stay in sync (OpenAPI paths, endpoint reference, runnable examples), and docs/data-model.md reflects the draft/version model and transactional_messages.template_version.
• Existing flows keep working after the resolution change: seed_demo_data and scripts/upsert_cmp_templates.py publish after upsert, and the existing transactional API, dry-run, CMP template script, and LocalStack integration tests pass (updated where they now need a published version).
• [HUMAN] One test-send on the sandbox to an allowlisted staff address arrives and renders correctly in a real mail client (subject, wrapped HTML with footer, readable plain text).

Test Notes

• Unit/service tests: publish and immutability; version resolution (latest default, explicit, unknown, none published); required-context validation; frontmatter parsing (with and without optional keys); import command against a fixture directory including a malformed file; preview/send render parity.
• API/view tests: PUT draft semantics and GET shape; publish, versions list, preview, test-send (allowed, denied, another client's template); send with and without template_version; dry-run parity; api_docs/OpenAPI registration.
• LocalStack/mocked AWS tests: SES payloads for a versioned send and a test-send asserted with botocore.stub.Stubber; no test sends real email.
• UI/screenshot checks: not applicable; no visible operator UI change. Tester should state screenshots are not applicable.

Blocked by: None.

[alexeygrigorev]

Groomed into an implementation-ready spec.

• Grounded in the current codebase: the transactional template API (PUT/GET), send pipeline, and required-context validation already exist; the issue now specifies what changes (immutable published versions, template_version on sends, publish/versions/preview/test-send endpoints, AISL markdown import) and what stays (PUT draft shape, direct-HTML drafts as a first-class content form, worker and SQS contract untouched since messages snapshot rendered bodies).
• Added codebase notes the implementer must read: the send resolution change breaks PUT-then-send flows until seed_demo_data and scripts/upsert_cmp_templates.py publish; new endpoints must be registered in mailing/services/api_docs.py or test_api_docs_endpoint_reference_matches_openapi_paths fails; frontmatter category is not the send-level category_tag; markdown/YAML dependencies are not in pyproject.toml yet.
• Acceptance criteria are testable and include the two plan verification bullets (50 drafts imported with zero errors; event_registration preview with the AISL preview context) plus preview/send parity via the existing dry-run path. One [HUMAN] criterion covers a real sandbox test-send delivery.
• Labels: P1, backend, api, transactional, email, docs. Depends on: none.

[alexeygrigorev]

Software Engineer Report

Work is in worktree .worktrees/issue-81-template-catalog (branch issue-81-template-catalog), uncommitted, awaiting tester and PM review. Nothing committed or pushed.

Files Changed

New:

• mailing/models.py additions: TemplateContentFormat, TemplateVersion (immutable, write-once save() guard, not admin-registered)
• mailing/migrations/0026_emailtemplate_category_emailtemplate_content_format_and_more.py
• mailing/services/transactional_markdown.py (frontmatter parsing, markdown -> sanitized HTML, markdown -> plain text; AISL pipeline port)
• mailing/services/transactional_content.py (shared TemplateContent render path + required-context validation)
• mailing/services/transactional_versions.py (publish/resolve/list/preview services)
• mailing/management/commands/import_templates.py
• templates/mailing/transactional_email_base.html (port of the AISL email chrome wrapper, multi-client header/footer)
• mailing/tests/test_template_versions.py (30 tests)

Modified:

• mailing/models.py: EmailTemplate.content_format/markdown_body/category; TransactionalMessage.template_version
• mailing/services/transactional.py: all three send paths resolve template_version (default latest published, unknown 404, none 409 no_published_version before any side effect); version-aware required-context validation; render via TemplateContent; template_version in send/dry-run responses; test_send_transactional_template_for_client with deny-by-default allowlist
• mailing/services/transactional_rendering.py: render_template_content (single render path; markdown = context first -> markdown -> sanitize -> wrapper)
• mailing/services/transactional_catalog.py: operator UI preview now routes through the shared render path (markdown drafts render converted)
• mailing/services/api.py: PUT accepts content_format/markdown_body/category (markdown requires markdown_body); GET returns latest_published_version; message status/history items include template_version
• mailing/services/api_docs.py: 4 new paths in API_DOC_PATHS, route_path_map, OpenAPI spec + schemas, endpoint reference groups, and a publish/preview runnable example
• mailing/views.py, mailing/urls.py: publish/versions/preview/test-send endpoints
• relay/settings.py: TRANSACTIONAL_TEST_SEND_ADDRESSES (deny-by-default CSV allowlist)
• mailing/management/commands/seed_demo_data.py: publishes drafts after upsert; demo messages carry template_version
• scripts/upsert_cmp_templates.py: POSTs publish after every PUT
• docs/api.md, docs/data-model.md: versions/publish/preview/test-send/template_version/frontmatter format, draft/version model, template_versions table
• mailing/tests/test_transactional_api.py, mailing/tests/test_transactional_dry_run.py, tests_integration/test_transactional_api_localstack.py: fixtures publish the template (sends require a published version)

Tests

• make test: 592 passed (was 562 before the new module; 30 new tests in mailing/tests/test_template_versions.py covering publish snapshot, identical-content republish dedupe, immutability (write-once save + not in admin), resolution (latest/explicit/404/409 no side effects), required-context against the resolved version, frontmatter parsing, markdown render/sanitize/wrapper/text, preview-vs-dry-run parity, publish/versions/preview/test-send endpoints incl. client scoping, recipient-list send with template_version, import command idempotency + malformed-file exit, OpenAPI registration)
• make lint: ruff + validate_infra pass
• make test-aws-local: 2 failed, 2 passed. The two failures are pre-existing on main and unrelated to this issue: enqueue_transactional_email now enqueues via django_tasks (DB backend; mailing/enqueue.py/mailing/tasks.py, untouched by this diff), so the stale assertions expecting a raw SQS message never see one. The send endpoints themselves returned 202 through the new published-version resolution path in that run. Needs a follow-up issue to realign the LocalStack transport tests with the django_tasks move.

Plan verifications (run in the worktree)

• uv run python manage.py import_templates --dir ~/git/ai-shipping-labs/email_app/email_templates/ --client asl-platform
  -> Imported 50 files ... created=50, updated=0, errors=0. Re-run: created=0, updated=50, errors=0. (idempotent)
• Preview of event_registration with the AISL context from email_app/services/preview_contexts.py: renders with no missing keys; subject You're registered: AI Shipping Workshop; {% if not is_host_registration %} block processed; calendar/cancel links present in text; wrapper HTML with client header/footer. Also verified preview(v1) == dry-run send(v1) byte-for-byte for subject/html/text, and the operator UI render path for the markdown draft.
• scripts/upsert_cmp_templates.py run end-to-end against a local server: 6 CMP templates PUT + published as v1; second run returns created: false for both PUT and publish (dedupe).

Dependencies

• markdown 3.10.3: markdown -> HTML conversion (same library AISL uses).
• PyYAML 6.0.3: frontmatter parsing via yaml.safe_load (minimal, no extra frontmatter framework).
• nh3 0.3.7: allowlist HTML sanitizing of rendered markdown (ammonia binding; matches AISL's sanitizer). All minimal, well-maintained, and pinned by uv.

Notes / deviations

• Markdown rendering substitutes context with autoescape=False, consistent with the existing HTML form; the nh3 sanitizer strips script/style and event handlers afterwards.
• The email chrome wrapper shows the client name in header/footer (the AISL original hardcodes "AI Shipping Labs"); unsubscribe/tracking footers stay out of transactional mail by design.
• content_format on PUT is explicit (html default). python-frontmatter was not added; frontmatter splitting is ~20 lines on top of PyYAML.
• Test-send denial is 403 with {"email": "not_allowlisted"} before any validation work that mutates state.
• No data migration backfills versions (pre-production, per issue).