Recreate an empty DataStore table when a load fails

[jguo144]

Problem

When a user sets a Data Dictionary type override that the data can't satisfy (e.g. a Vendor # column of integers set to timestamp), the next xloader upload fails and leaves the resource in a dead end:

• xloader tries a direct COPY first; on a bad value it fails, then falls back to tabulator.
• Both paths delete the existing table when field types change. The tabulator path then runs create_table + insert in a single transaction; the insert fails → the whole transaction rolls back → no table exists.
• Type overrides are stored only in the DataStore table's Postgres column comments, not on the resource. So once the table is dropped, the table, its data dictionary, and datastore_active are all gone — the Data Dictionary tab disappears and the user has no in-UI way to fix the type. Only a full re-upload recovers it.

Fix

Capture the existing field definitions (types + data dictionary, including type overrides) before the load. If all load attempts fail and the table was dropped, recreate an empty table with those fields so the Data Dictionary stays visible and editable. The user then corrects the column type and re-uploads.

Key properties:

• Best-effort — the recovery has its own try/except and never masks the real load error, which is still re-raised (job status stays error, log shows the true cause).
• No-op when the table still exists — e.g. a partial/truncated load, matching "if any data loaded, don't recreate."
• No-op for brand-new resources — nothing to preserve.
• Does not persist the file hash or mark the table as containing all records, so re-uploading the corrected file still triggers a load (rather than being skipped by the hash gate).
• Config-gated — new ckanext.xloader.recreate_empty_table_on_error option (default True) lets operators disable it.

Outcome

A failed upload leaves the resource recoverable — empty table, datastore_active True, Data Dictionary tab visible, original overrides intact — instead of a dead end.

Tests

Added TestRecreateEmptyTableOnError in test_jobs.py — unit tests that drive xloader_data_into_datastore_ with collaborators mocked (no DB/RQ worker), covering: recreate-on-drop (fields/overrides preserved, hash not persisted), no-op when table survives, no-op for new resources, config-disabled, and recovery-never-masks-original-error.

🤖 Generated with Claude Code

[orca-security-us]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca