Drift doctor, migration imports, database tools, security suite and hosting polish

CHANGELOG.md

@@ -28,6 +28,99 @@ awaiting a stable release:
 
 ### Added
 
+- **Mail Server extension (`serverkit-mail`)** — an opt-in built-in that runs a
+  self-hosted mail server (Stalwart, in a managed Docker container) driven
+  through its HTTP admin API. Manages mail domains, mailboxes, forwarders,
+  autoresponders and catch-all; generates DKIM keys and deploys DKIM/SPF/DMARC/MX/A
+  records through the existing DNS-provider integrations; requests a Let's Encrypt
+  cert for the mail hostname; and shows the outbound queue. A **deliverability
+  preflight** (reverse-DNS/PTR match, port-25 egress, RBL listing, listening
+  ports) runs on a daily schedule and **blocks outbound sending until it passes**
+  (an explicit force override is audit-logged), because self-hosted mail on a VPS
+  is a real commitment — many providers block port 25 and a fresh IP can be
+  pre-burned. Ships brute-force auth jails and registers the mail store for
+  scheduled backups. Not installed by default; enable it from Marketplace. It runs
+  alongside the older Postfix/Dovecot `serverkit-email` extension with entirely
+  separate identifiers.
+- **Configuration drift detection + repair ("doctor")** — a daily read-only
+  sweep re-renders the expected nginx vhost and compose override for every
+  managed resource from panel state and diffs it against disk; drift raises an
+  admin notification, and a Doctor tab on Monitoring runs the full diagnosis
+  (drift, core services, cert expiry, disk headroom, database) with
+  diff-confirmed per-item repair and batch repair. Nothing is ever changed
+  automatically.
+- **Operator CLI** — the `serverkit` CLI gains API-backed verbs for when the
+  browser isn't an option: `status`, `services list/restart`, `apps list`,
+  `doctor [--repair]`, `repair <type> <id>`, `update`, `support-bundle`, and
+  `login-url` (mints a one-time login link). Auth is a break-glass 10-minute
+  token minted in-process — root on the box already implies full control —
+  and every mint is audit-logged.
+- **Diagnostic support bundle** — one call (API or CLI) exports a sanitized
+  zip (versions, service states, setting shapes without values, recent job
+  failures, scrubbed log tail) to attach to a bug report; secrets are
+  scrubbed by pattern and by the settings secret-key list.
+- **Web-shell/YARA scanning + job-backed malware scans** — malware scans now
+  run as jobs, can target an app's docroot directly, and a curated web-shell
+  rules pass (eval/base64 droppers, c99/r57/WSO markers, PHP-in-image,
+  auto_prepend hijacks…) runs alongside ClamAV — with a pure-Python fallback
+  when yara isn't installed, custom rule upload, and one-click quarantine
+  with restore.
+- **File-integrity monitoring** — baseline-and-diff over the paths ServerKit
+  manages (nginx config, ServerKit systemd units, app docroots on opt-in)
+  every six hours, feeding the Notifications Bus; the Security → Integrity
+  tab gains baseline/check/accept controls and a what-changed view.
+- **CrowdSec extension** — a new `serverkit-crowdsec` marketplace extension
+  surfaces CrowdSec decisions and alerts, lets you ban/unban IPs and manage
+  allowlists via cscli, with graceful degradation when CrowdSec isn't
+  installed.
+- **Authoritative DNS server extension** — a new `serverkit-dns-server`
+  marketplace extension runs PowerDNS in a managed container so a ServerKit
+  box can be the nameserver for its domains: zones, records, DNSSEC with DS
+  records for the registrar, and a delegation check. For homelab and
+  air-gapped setups; complements (never replaces) the provider integrations.
+- **Per-domain bandwidth accounting** — daily nginx access-log aggregation
+  into per-site transfer stats, with 30-day sparklines on the Services list
+  and a monthly figure on the service Overview.
+- **Per-site micro-cache** — an opt-in nginx micro-cache (10s TTL) per site
+  with safe bypasses for logged-in/admin/cart cookies and paths, an
+  `X-SK-Cache` header for verification, and a manual purge button. Big cheap
+  win for WordPress/PHP sites.
+- **`.htaccess` → nginx converter** — a paste-in tool (on the Import wizard)
+  translating common rewrite/redirect/auth/access rules to nginx directives,
+  flagging anything it can't translate with the reason and line number.
+- **Import a site (migration pipeline)** — a 5-step wizard at `/imports`
+  (entry points on Services and WordPress) restores cPanel/WHM, DirectAdmin
+  and Hestia/Vesta backup archives onto ServerKit: the archive is analysed
+  into a report (domains, databases, DB users, crontab, warnings), then a
+  resumable job maps it to an app, managed databases (MySQL password hashes
+  preserved where the engine allows) and cron entries, with per-step logs and
+  retry-from-failed-step. Uploads and fetch-by-URL both supported, with SSRF
+  and archive-traversal guards throughout.
+- **WordPress: import over SSH** — point at any live WordPress site with SSH
+  credentials (`/wordpress/ssh-import`): probe shows the pinned host key and
+  site facts, then a job pulls the docroot, dumps the database through the
+  tunnel and rebuilds it as a managed site with the URL search-replaced.
+- **Database tools** — the Database Explorer gains a live processlist with
+  kill/terminate per server or container; a curated config tuner (RAM-aware
+  suggested values for vetted MySQL/PostgreSQL settings, applied with backups
+  and clean rollback, never auto-applied); managed database users tracked as
+  first-class rows; and one-click "Open in Adminer" SSO via a single-use,
+  five-minute shadow credential scoped to one database.
+- **Per-app resource limits** — CPU and memory limits are first-class fields
+  on an app (Settings → Resource Limits) showing live usage vs limit, applied
+  to the container without touching the user's own compose file.
+- **One-time login links** — admins can mint single-use, short-TTL,
+  optionally IP-bound login URLs from Settings → Users, for "log in and take
+  a look" support situations; links are hashed at rest and reaped hourly.
+- **Demo mode** — an opt-in flag that blocks every mutating API call with
+  `403 demo_mode` and offers seeded read-only credentials on the login page,
+  for running a public demo of a real panel.
+- **Cloud-metadata egress guard** — a default-on (opt-out) firewall rule
+  blocking app containers from 169.254.169.254, closing the SSRF-to-cloud-IAM
+  credential-theft class on cloud VPSes (Security → Firewall).
+- **Server speed test** — an on-demand download/upload/latency test on the
+  Monitoring page, using the Ookla/speedtest CLI when present with a
+  pure-Python fallback.
 - **Extensions platform (Phase 7 — settings slot + manifest linting)** — extensions
   can now contribute sections to the Settings page (a `settings.section` widget
   slot rendered below the active tab), and `plugin.json` manifests are shape-checked
@@ -268,6 +361,18 @@ awaiting a stable release:
 
 ### Changed
 
+- **Marketplace simplified to a two-tab store** — Browse and Installed. The
+  Installed tab now lists *every* installed extension (built-in, registry,
+  manual) with the full action set (Update / Configure / Enable–Disable /
+  Uninstall with keep-vs-purge), replacing the old duplicate "Installed" and
+  "ServerKit Plugins" tabs. Each row shows where it came from via a
+  Built-in / Registry / Manual source badge, and registry installs are now
+  stamped `source_type='registry'` in the database instead of masquerading as
+  URL installs. Manual install (URL / host folder / zip upload) moved out of
+  tab-land into an "Install manually" modal off the topbar, and Browse dropped
+  its KPI strip, sidebar panels, and duplicate category filters in favor of
+  search + one chips row (plus an All / By ServerKit / Community publisher
+  filter).
 - Overhauled the Docker UI (bulk container stats, compose listing) and migrated
   the frontend design system to SCSS `.ui-*` components.
 - Unified the local dev launcher (`dev.sh` / `dev.ps1`).

README.md

@@ -193,6 +193,13 @@ English | [Español](docs/README.es.md) | [中文版](docs/README.zh-CN.md) | [P
 
 </details>
 
+<details>
+<summary><strong>Marketplace</strong> — Browse and install extensions from the bundled catalog or the remote registry — with a unified Installed tab</summary>
+
+![Marketplace](docs/screenshots/marketplace.png)
+
+</details>
+
 <details>
 <summary><strong>Settings</strong> — Profile, security, appearance/branding, users, connections, and notification channels</summary>
 

VERSION

@@ -1 +1 @@
-1.7.11
+1.7.16

backend/app/__init__.py

@@ -76,6 +76,10 @@ def create_app(config_name=None):
     from app.middleware.security import register_security_headers
     register_security_headers(app)
 
+    # Demo mode guard — config-gated (default off), blocks mutating API calls
+    from app.middleware.demo import init_demo_mode
+    init_demo_mode(app)
+
     # Register API key authentication middleware
     from app.middleware.api_key_auth import register_api_key_auth
     register_api_key_auth(app)
@@ -163,6 +167,14 @@ def create_app(config_name=None):
     from app.api.databases import databases_bp
     app.register_blueprint(databases_bp, url_prefix='/api/v1/databases')
 
+    # Register blueprints - Managed DB users + Adminer SSO
+    from app.api.managed_db_users import managed_db_users_bp
+    app.register_blueprint(managed_db_users_bp, url_prefix='/api/v1/managed-databases')
+
+    # Register blueprints - Curated DB config tuner
+    from app.api.db_tuner import db_tuner_bp
+    app.register_blueprint(db_tuner_bp, url_prefix='/api/v1/db-tuner')
+
     # Register blueprints - Monitoring & Alerts
     from app.api.monitoring import monitoring_bp
     app.register_blueprint(monitoring_bp, url_prefix='/api/v1/monitoring')
@@ -179,6 +191,10 @@ def create_app(config_name=None):
     from app.api.snapshots import snapshots_bp
     app.register_blueprint(snapshots_bp, url_prefix='/api/v1/apps')
 
+    # Register blueprints - Declarative serverkit.yaml manifest
+    from app.api.manifests import manifests_bp
+    app.register_blueprint(manifests_bp, url_prefix='/api/v1/manifests')
+
     # Register blueprints - Projects & Environments hierarchy
     from app.api.projects import projects_bp
     app.register_blueprint(projects_bp, url_prefix='/api/v1/projects')
@@ -366,6 +382,13 @@ def create_app(config_name=None):
     # connection stay core (they back /domains); the extension borrows the single
     # core CloudflareClient, never a duplicate.
 
+    # Register blueprints - DNS provider connections. Core (they back the
+    # Settings -> Connections DNS tiles and wildcard TLS), but kept at the
+    # historical /api/v1/email/dns-providers paths from before the email
+    # extraction so existing frontends keep working.
+    from app.api.dns_providers import dns_providers_bp
+    app.register_blueprint(dns_providers_bp, url_prefix='/api/v1/email')
+
     # Register blueprints - Dynamic DNS
     from app.api.ddns import ddns_bp
     app.register_blueprint(ddns_bp, url_prefix='/api/v1/ddns')
@@ -450,6 +473,30 @@ def create_app(config_name=None):
     from app.api.ai import ai_bp
     app.register_blueprint(ai_bp, url_prefix='/api/v1/ai')
 
+    # Register blueprints - Server speed test (Monitoring card)
+    from app.api.speed_test import speedtest_bp
+    app.register_blueprint(speedtest_bp, url_prefix='/api/v1/speedtest')
+
+    # Register blueprints - Site imports (panel migration pipeline)
+    from app.api.site_imports import site_imports_bp
+    app.register_blueprint(site_imports_bp, url_prefix='/api/v1/imports')
+
+    # Register blueprints - Drift detection + doctor sweep
+    from app.api.doctor import doctor_bp
+    app.register_blueprint(doctor_bp, url_prefix='/api/v1/doctor')
+
+    # Register blueprints - Diagnostic support bundle
+    from app.api.support_bundle import support_bundle_bp
+    app.register_blueprint(support_bundle_bp, url_prefix='/api/v1/support-bundle')
+
+    # Register blueprints - Per-site bandwidth accounting
+    from app.api.bandwidth import bandwidth_bp
+    app.register_blueprint(bandwidth_bp, url_prefix='/api/v1/bandwidth')
+
+    # Register blueprints - .htaccess -> nginx converter (apps-prefixed tool)
+    from app.api.htaccess_tools import htaccess_tools_bp
+    app.register_blueprint(htaccess_tools_bp, url_prefix='/api/v1/apps')
+
     # Handle database migrations (Alembic) — must run before plugin loader
     # since the loader queries the installed_plugins table.
     with app.app_context():
@@ -566,6 +613,28 @@ def create_app(config_name=None):
         ServerOnboardingService.register_jobs()
         from app.services.preview_service import PreviewService
         PreviewService.register_jobs()
+        from app.services.metadata_guard_service import MetadataGuardService
+        MetadataGuardService.register_jobs()
+        if not app.config.get('TESTING'):
+            MetadataGuardService.ensure()  # converge the metadata egress rule (no-op when unsupported)
+        from app.services.speed_test_service import SpeedTestService
+        SpeedTestService.register_jobs()
+        from app.services import login_link_service
+        login_link_service.register_jobs()
+        from app.services.db_admin_sso_service import DbAdminSsoService
+        DbAdminSsoService.register_jobs()
+        from app.services.site_import_service import SiteImportService
+        SiteImportService.register_jobs()
+        from app.services.drift_service import DriftService
+        DriftService.register_jobs()
+        from app.services.doctor_service import DoctorService
+        DoctorService.register_jobs()
+        from app.services.file_integrity_service import FileIntegrityService
+        FileIntegrityService.register_jobs()
+        from app.services.malware_scan_service import MalwareScanService
+        MalwareScanService.register_jobs()
+        from app.services.bandwidth_service import BandwidthService
+        BandwidthService.register_jobs()
         start_job_system(app, seed=seed_builtin_schedules)
 
     # Request body size limit