fix(dispatcher): treat a missing tarball object as a permanent error

[VascoSch92]

Fixes the customer-visible half of OSS-9505 ([C24] HappyFox #OS00000160): automation "run now" failing with an opaque Internal error that repeated on every schedule tick.

This is fix (2) from the issue. Fixes (1) (crash-safe tarball regeneration) and (3) (MinIO durability, PLTF-3455) are not in this PR.

Warning

Do not merge as-is. Review turned up an over-trigger in this approach — see "Known defect" below. Pushing a narrowing commit.

The bug

_download_internal_tarball already raises TarballNotFoundError when the TarballUpload row is missing, but the subsequent store.read(upload.storage_path) was unguarded. When the row is live but its object is gone, s3.py maps NoSuchKey to FileNotFoundError — a plain Exception, which sails past the except PermanentDispatchError handler in dispatch step 4 and lands in _execute_run_safe. Two consequences, both of which C24 hit:

• the run is marked FAILED with the literal string "Internal error" and no detail;
• the automation is not disabled, so every schedule tick creates a sandbox, fails the same way, and tears it down (12 and 16 consecutive failures for the two affected automations).

The fix

Wrap that one read and re-raise as TarballNotFoundError, which already subclasses PermanentDispatchError. No new machinery — the existing handler disables the automation and surfaces str(exc) as error_detail. The message names the missing storage path and the remedy (recreate the automation), since the original bytes are not retained anywhere and delete-and-recreate is the only recovery.

Known defect (being fixed)

s3.py::_handle_client_error (lines 262-276) maps every ClientError to FileNotFoundError, not just NoSuchKey: NoSuchBucket, AccessDenied, and an else catch-all covering 500 InternalError, 503 ServiceUnavailable, SlowDown throttling and expired credentials all land there too.

So as committed, this PR treats a transient storage hiccup as permanent. On C24 that is not hypothetical — their MinIO is OOM-crash-looping, so a read landing during a restart returns 5xx, which would now permanently disable a healthy automation. Today those runs simply fail and recover on the next tick, so this would be a regression for the exact customer it targets.

Narrowing: raise an ObjectNotFoundError(FileNotFoundError) subclass only at the genuine-absence sites (s3.py:267-268, local.py:64, google_cloud.py:102,137) and catch that here instead of bare FileNotFoundError. Backward compatible — the seven existing except FileNotFoundError handlers keep behaving identically via the parent class, and every other ClientError stays transient.

Tests

Two tests, both verified to fail without the source change:

• test_raises_tarball_not_found_for_missing_object — unit-level: live COMPLETED row, store raises FileNotFoundError, asserts the reclassification and that the original error is chained as __cause__.
• test_disables_automation_on_missing_tarball_object — end-to-end through _execute_run: asserts the automation ends up enabled=False, that error_detail names the real cause and is not "Internal error", and that execution never entered the sandbox.

Full suite: 1148 passed. Pre-commit (ruff format/lint, pycodestyle, pyright) clean.

Notes for the issue thread

The attached reconcile_tarballs.py cannot discriminate the two candidate root causes the way step 4 of its runbook claims. It says RECORD_SOFT_DELETED implies the edit/rollback bug — but in that scenario source_upload.deleted_at is only flushed, never committed, so the rollback reverts the soft-delete and tarball_path together. The surviving state is a live, COMPLETED row with no object: verdict MISSING_OBJECT, identical to MinIO loss. RECORD_SOFT_DELETED is effectively unreachable via that path, since both writes share one transaction. Breadth (are never-edited automations affected) remains a valid signal; worth also pulling 5xx responses on PATCH /automations/{id} from C24.

Separate latent bug in the same area: preset_router.py:378 does except FileNotFoundError: return None, so a transient storage error during a prompt edit makes regeneration silently no-op. The PATCH returns 200 and the prompt column updates, but the tarball keeps the old baked prompt and nothing surfaces the divergence.

[github-actions]