Drift doctor, migration imports, database tools, security suite and hosting polish

[jhd3197]

This is the big "make the panel earn its keep" drop. It started as a marketplace cleanup and turned into a whole shelf of operator features — the kind of things you only miss the day you actually need them. The headline is a config-drift doctor that re-renders every managed resource from panel state, diffs it against what's on disk, and offers diff-confirmed repair without ever touching anything automatically. Around that sit a full migration pipeline (cPanel/DirectAdmin/Hestia archives and live-site pulls over SSH), a set of real database tools, a security suite (web-shell YARA scans, file-integrity monitoring, a cloud-metadata egress guard), and a batch of hosting quick-wins. Then two more headliners landed on top: a mail hosting extension that runs a full mail server (SMTP/IMAP/JMAP) per domain — with a preflight gate that refuses to let you send until PTR, outbound port 25 and blocklist checks actually pass — and serverkit.yaml, a declarative manifest that describes a whole multi-service project in one file the panel can scaffold, apply, re-sync on push and drift-check like everything else. Everything destructive stays opt-in and diff-confirmed, and the whole thing is backed by ~30 new test suites. The marketplace also finally collapsed into a clean two-tab store.

Highlights

• Doctor + drift detection — a daily read-only sweep flags when a managed nginx vhost or compose override no longer matches panel state; a new Doctor tab on Monitoring diagnoses drift, core services, cert expiry, disk headroom and databases, with per-item and batch repair you confirm from a diff.
• Import a whole site — a 5-step wizard at /imports restores cPanel/WHM, DirectAdmin and Hestia/Vesta backup archives (domains, databases, DB users, cron) as managed apps, plus a separate "pull a live WordPress site over SSH" flow that dumps the DB through the tunnel and rebuilds it with the URL rewritten.
• .htaccess → nginx converter — paste your old rewrite/redirect/auth rules and get nginx directives back, with anything untranslatable flagged by reason and line.
• Database tools — a live processlist with kill, a RAM-aware config tuner (suggested-not-applied, with backups and rollback), first-class managed database users, and one-click "Open in Adminer" SSO via a single-use 5-minute shadow credential.
• Security suite — job-backed malware scans with a curated web-shell YARA pass (and a pure-Python fallback), scoped file-integrity monitoring, and a default-on egress guard that blocks app containers from the 169.254.169.254 cloud-metadata endpoint.
• Mail hosting — a new built-in extension runs a managed mail server per domain (SMTP/IMAP/JMAP with mailboxes and aliases), auto-plants MX/SPF/DKIM/DMARC records through your existing DNS connections, and gates outbound sending behind a preflight that verifies PTR, port-25 reachability and blocklist status before a single message leaves.
• Describe your stack in one file — a serverkit.yaml manifest declares services, databases, domains, env and health checks for a whole project; the panel scaffolds it, shows a plan, applies it, re-syncs it on every git push and flags drift when reality wanders — with fromSecret/fromService/generate references so credentials never live in the file.
• Two new extensions — a CrowdSec integration (decisions, alerts, ban/unban via cscli) and an authoritative DNS server (managed PowerDNS with zones, records, DNSSEC and a delegation check) for homelab/air-gapped setups.
• Hosting quick-wins — per-domain bandwidth accounting with sparklines, an opt-in per-site nginx micro-cache with safe cookie/path bypasses, and first-class per-app CPU/memory limits with live usage.
• Operator conveniences — an API-backed serverkit CLI (status, doctor, repair, support-bundle, login-url…), a sanitized diagnostic support bundle, one-time login links, and a demo mode that blocks all mutations.
• Marketplace, tidied — Browse + Installed, one row per extension with a Built-in/Registry/Manual source badge and the full action set; manual install moved into a modal; registry defaults to the public serverkit-extensions index.

Technical changes

Drift / doctor

• drift_service.py re-renders the expected nginx vhost and compose override for each managed resource and diffs against disk; doctor_service.py runs the full sweep (drift, core services, cert expiry, disk, DB) and drives per-resource repair. doctor_bp at /api/v1/doctor; daily job registered via DriftService.register_jobs()/DoctorService.register_jobs(). Repairs are always diff-confirmed, never automatic.

Migration imports

• site_import_service.py + site_importers/{base,cpanel,directadmin,hestia}.py analyze an archive into a report then run a resumable, retry-from-failed-step job that maps it to an app, managed databases (preserving MySQL password hashes where the engine allows) and cron. site_imports_bp at /api/v1/imports; SiteImport model + migration 051. Upload and fetch-by-URL paths both carry SSRF and archive-traversal guards.
• WordPress SSH import (serverkit-wordpress/backend/wp_ssh_import_service.py) probes for a pinned host key and site facts, then pulls docroot + DB dump over the tunnel and rebuilds with a URL search-replace.
• htaccess_converter.py translates rewrite/redirect/auth/access rules to nginx, reporting untranslatable rules with reason + line; exposed via htaccess_tools_bp under /api/v1/apps.

Database tools

• db_process_service.py (processlist + kill, server or container), db_config_tuner_service.py (curated RAM-aware suggestions applied with backup/rollback, never auto-applied), managed_db_user_service.py + ManagedDatabaseUser model (migration 050), and db_admin_sso_service.py (single-use 5-minute shadow credential scoped to one database, Adminer SSO). Blueprints: managed_db_users_bp, db_tuner_bp.

Security

• yara_scan_service.py + yara_rules/webshells.yar (eval/base64 droppers, c99/r57/WSO markers, PHP-in-image, auto_prepend hijacks) with a pure-Python fallback when yara is absent, custom-rule upload, and quarantine/restore; malware scans moved onto the job bus and can target an app docroot.
• file_integrity_service.py baselines and diffs ServerKit-managed paths on a 6-hour job, feeding the Notifications Bus; baseline/check/accept controls on Security → Integrity.
• metadata_guard_service.py installs a default-on (opt-out) firewall rule blocking 169.254.169.254 from app containers; MetadataGuardService.register_jobs(). Additions in security_service.py / firewall.py / security.py.

Extensions

• serverkit-crowdsec — crowdsec_service.py wraps cscli for decisions/alerts/ban/unban/allowlists with graceful degradation; CrowdSecPage.jsx.
• serverkit-dns-server — dns_server_service.py runs PowerDNS in a managed container (zones, records, DNSSEC + DS records, delegation check); DnsServerPage.jsx / ZoneDetail.jsx. Both shipped in builtin-extensions/ and pre-bundled into frontend/src/plugins/.
• serverkit-mail — Stalwart-backed mail hosting on the same managed-container pattern: stalwart_service.py drives the engine over its admin API (pinned against live 0.16.11), mail_service.py + models.py own the ext_serverkit_mail_* tables (domains, mailboxes, aliases), dns_mail_service.py plants MX/SPF/DKIM/DMARC through existing DNS connections, and preflight_service.py hard-gates sending (409) until PTR, outbound port 25 and RBL checks pass. mail_jail_service.py and mail_backup.py reuse the Fail2ban and backup rails; MailPage.jsx is a tab-group UI. Two proving suites cover the extension and the preflight/DNS path.

serverkit.yaml manifests

• manifest_spec_service.py defines the v1 spec + normalizer (schema in docs/serverkit-yaml.schema.json, guide in docs/SERVERKIT_YAML.md); manifest_scaffold_service.py generates a starter manifest from an existing app.
• manifest_persistence_service.py + ApplicationManifest model (migrations 054/055) persist the parsed manifest, seed env vars and gate deploys on declared health checks.
• manifest_apply_service.py is the resolve → plan → apply engine; domain_attach_service.py gives core a first-class domain-attach path; env_reference_service.py resolves fromSecret/fromService/generate binders so secrets never sit in the YAML.
• manifest_sync_service.py re-syncs on git push and adds a manifest drift-check family + events; manifest_preview_service.py powers manifest-driven PR previews; cli.py gains manifest verbs and manifests_bp exposes it all under /api/v1.
• Frontend: wizard summary step in NewService.jsx, a Manifest section in the service SettingsTab.jsx, a project status strip in ProjectDetail.jsx, and services/api/manifests.js.

Hosting

• bandwidth_service.py aggregates nginx access logs into per-site daily transfer stats (SiteBandwidth model, migration 052); bandwidth_bp at /api/v1/bandwidth, sparklines on the Services list.
• Per-site micro-cache in nginx_service.py (10s TTL, bypass on logged-in/admin/cart cookies + paths, X-SK-Cache header, manual purge; migration 053).
• Per-app CPU/memory limits: application.py model fields (migration 048), applied in docker_service.py without editing the user's compose file; ResourceLimitsPanel.jsx.

Operator / auth

• cli.py + cli_api_client.py back the serverkit CLI verbs against the API with a break-glass 10-minute in-process token (audit-logged); support_bundle_service.py exports a secret-scrubbed diagnostic zip (support_bundle_bp).
• login_link_service.py + LoginLink model (migration 049) mint single-use, short-TTL, optionally IP-bound login URLs, hashed at rest and reaped hourly; auth.py + LoginLinksSection.jsx.
• middleware/demo.py blocks mutating calls with 403 demo_mode and seeds read-only demo credentials.
• speed_test_service.py on-demand download/upload/latency test (Ookla CLI with pure-Python fallback); speedtest_bp.

Marketplace

• Browse/Installed two-tab store; Installed lists all built-in/registry/manual extensions with source badges and full actions; registry installs stamped source_type='registry' (plugin_service.py, registry_service.py); ManualInstallModal.jsx off the topbar; registry defaults to the public serverkit-extensions index.

Misc

• DNS provider routes moved back into core (api/dns_providers.py) so provider connections don't depend on an extension; the frontend's fallback API error now names the endpoint and status (services/api/client.js).
• python-socketio bumped to 5.16.3 and python-engineio to 4.13.3 to clear three DoS advisories flagged by the dependency scan.
• New frontend services/api/* modules for every feature above, App.jsx routes, OverviewTab/SettingsTab wiring, and utility helpers (formatBytes, expiry, sanitizeSvg, clipboard). CHANGELOG + README updated; VERSION bumped to 1.7.16. ~30 new backend test suites cover imports, drift/doctor, YARA, file-integrity, metadata guard, DB tools, bandwidth, micro-cache, CLI, login links, demo mode, the manifest engine and all three new extensions.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.