diff --git a/CHANGELOG.md b/CHANGELOG.md index 002204f4..72591852 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,99 @@ awaiting a stable release: ### Added +- **Mail Server extension (`serverkit-mail`)** — an opt-in built-in that runs a + self-hosted mail server (Stalwart, in a managed Docker container) driven + through its HTTP admin API. Manages mail domains, mailboxes, forwarders, + autoresponders and catch-all; generates DKIM keys and deploys DKIM/SPF/DMARC/MX/A + records through the existing DNS-provider integrations; requests a Let's Encrypt + cert for the mail hostname; and shows the outbound queue. A **deliverability + preflight** (reverse-DNS/PTR match, port-25 egress, RBL listing, listening + ports) runs on a daily schedule and **blocks outbound sending until it passes** + (an explicit force override is audit-logged), because self-hosted mail on a VPS + is a real commitment — many providers block port 25 and a fresh IP can be + pre-burned. Ships brute-force auth jails and registers the mail store for + scheduled backups. Not installed by default; enable it from Marketplace. It runs + alongside the older Postfix/Dovecot `serverkit-email` extension with entirely + separate identifiers. +- **Configuration drift detection + repair ("doctor")** — a daily read-only + sweep re-renders the expected nginx vhost and compose override for every + managed resource from panel state and diffs it against disk; drift raises an + admin notification, and a Doctor tab on Monitoring runs the full diagnosis + (drift, core services, cert expiry, disk headroom, database) with + diff-confirmed per-item repair and batch repair. Nothing is ever changed + automatically. +- **Operator CLI** — the `serverkit` CLI gains API-backed verbs for when the + browser isn't an option: `status`, `services list/restart`, `apps list`, + `doctor [--repair]`, `repair `, `update`, `support-bundle`, and + `login-url` (mints a one-time login link). Auth is a break-glass 10-minute + token minted in-process — root on the box already implies full control — + and every mint is audit-logged. +- **Diagnostic support bundle** — one call (API or CLI) exports a sanitized + zip (versions, service states, setting shapes without values, recent job + failures, scrubbed log tail) to attach to a bug report; secrets are + scrubbed by pattern and by the settings secret-key list. +- **Web-shell/YARA scanning + job-backed malware scans** — malware scans now + run as jobs, can target an app's docroot directly, and a curated web-shell + rules pass (eval/base64 droppers, c99/r57/WSO markers, PHP-in-image, + auto_prepend hijacks…) runs alongside ClamAV — with a pure-Python fallback + when yara isn't installed, custom rule upload, and one-click quarantine + with restore. +- **File-integrity monitoring** — baseline-and-diff over the paths ServerKit + manages (nginx config, ServerKit systemd units, app docroots on opt-in) + every six hours, feeding the Notifications Bus; the Security → Integrity + tab gains baseline/check/accept controls and a what-changed view. +- **CrowdSec extension** — a new `serverkit-crowdsec` marketplace extension + surfaces CrowdSec decisions and alerts, lets you ban/unban IPs and manage + allowlists via cscli, with graceful degradation when CrowdSec isn't + installed. +- **Authoritative DNS server extension** — a new `serverkit-dns-server` + marketplace extension runs PowerDNS in a managed container so a ServerKit + box can be the nameserver for its domains: zones, records, DNSSEC with DS + records for the registrar, and a delegation check. For homelab and + air-gapped setups; complements (never replaces) the provider integrations. +- **Per-domain bandwidth accounting** — daily nginx access-log aggregation + into per-site transfer stats, with 30-day sparklines on the Services list + and a monthly figure on the service Overview. +- **Per-site micro-cache** — an opt-in nginx micro-cache (10s TTL) per site + with safe bypasses for logged-in/admin/cart cookies and paths, an + `X-SK-Cache` header for verification, and a manual purge button. Big cheap + win for WordPress/PHP sites. +- **`.htaccess` → nginx converter** — a paste-in tool (on the Import wizard) + translating common rewrite/redirect/auth/access rules to nginx directives, + flagging anything it can't translate with the reason and line number. +- **Import a site (migration pipeline)** — a 5-step wizard at `/imports` + (entry points on Services and WordPress) restores cPanel/WHM, DirectAdmin + and Hestia/Vesta backup archives onto ServerKit: the archive is analysed + into a report (domains, databases, DB users, crontab, warnings), then a + resumable job maps it to an app, managed databases (MySQL password hashes + preserved where the engine allows) and cron entries, with per-step logs and + retry-from-failed-step. Uploads and fetch-by-URL both supported, with SSRF + and archive-traversal guards throughout. +- **WordPress: import over SSH** — point at any live WordPress site with SSH + credentials (`/wordpress/ssh-import`): probe shows the pinned host key and + site facts, then a job pulls the docroot, dumps the database through the + tunnel and rebuilds it as a managed site with the URL search-replaced. +- **Database tools** — the Database Explorer gains a live processlist with + kill/terminate per server or container; a curated config tuner (RAM-aware + suggested values for vetted MySQL/PostgreSQL settings, applied with backups + and clean rollback, never auto-applied); managed database users tracked as + first-class rows; and one-click "Open in Adminer" SSO via a single-use, + five-minute shadow credential scoped to one database. +- **Per-app resource limits** — CPU and memory limits are first-class fields + on an app (Settings → Resource Limits) showing live usage vs limit, applied + to the container without touching the user's own compose file. +- **One-time login links** — admins can mint single-use, short-TTL, + optionally IP-bound login URLs from Settings → Users, for "log in and take + a look" support situations; links are hashed at rest and reaped hourly. +- **Demo mode** — an opt-in flag that blocks every mutating API call with + `403 demo_mode` and offers seeded read-only credentials on the login page, + for running a public demo of a real panel. +- **Cloud-metadata egress guard** — a default-on (opt-out) firewall rule + blocking app containers from 169.254.169.254, closing the SSRF-to-cloud-IAM + credential-theft class on cloud VPSes (Security → Firewall). +- **Server speed test** — an on-demand download/upload/latency test on the + Monitoring page, using the Ookla/speedtest CLI when present with a + pure-Python fallback. - **Extensions platform (Phase 7 — settings slot + manifest linting)** — extensions can now contribute sections to the Settings page (a `settings.section` widget slot rendered below the active tab), and `plugin.json` manifests are shape-checked @@ -268,6 +361,18 @@ awaiting a stable release: ### Changed +- **Marketplace simplified to a two-tab store** — Browse and Installed. The + Installed tab now lists *every* installed extension (built-in, registry, + manual) with the full action set (Update / Configure / Enable–Disable / + Uninstall with keep-vs-purge), replacing the old duplicate "Installed" and + "ServerKit Plugins" tabs. Each row shows where it came from via a + Built-in / Registry / Manual source badge, and registry installs are now + stamped `source_type='registry'` in the database instead of masquerading as + URL installs. Manual install (URL / host folder / zip upload) moved out of + tab-land into an "Install manually" modal off the topbar, and Browse dropped + its KPI strip, sidebar panels, and duplicate category filters in favor of + search + one chips row (plus an All / By ServerKit / Community publisher + filter). - Overhauled the Docker UI (bulk container stats, compose listing) and migrated the frontend design system to SCSS `.ui-*` components. - Unified the local dev launcher (`dev.sh` / `dev.ps1`). diff --git a/README.md b/README.md index 80f48da6..b62837e9 100644 --- a/README.md +++ b/README.md @@ -193,6 +193,13 @@ English | [Español](docs/README.es.md) | [中文版](docs/README.zh-CN.md) | [P +
+Marketplace — Browse and install extensions from the bundled catalog or the remote registry — with a unified Installed tab + +![Marketplace](docs/screenshots/marketplace.png) + +
+
Settings — Profile, security, appearance/branding, users, connections, and notification channels diff --git a/VERSION b/VERSION index 8f8b3f72..15421b30 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.7.11 +1.7.16 diff --git a/backend/app/__init__.py b/backend/app/__init__.py index 071667af..0c5416d5 100644 --- a/backend/app/__init__.py +++ b/backend/app/__init__.py @@ -76,6 +76,10 @@ def create_app(config_name=None): from app.middleware.security import register_security_headers register_security_headers(app) + # Demo mode guard — config-gated (default off), blocks mutating API calls + from app.middleware.demo import init_demo_mode + init_demo_mode(app) + # Register API key authentication middleware from app.middleware.api_key_auth import register_api_key_auth register_api_key_auth(app) @@ -163,6 +167,14 @@ def create_app(config_name=None): from app.api.databases import databases_bp app.register_blueprint(databases_bp, url_prefix='/api/v1/databases') + # Register blueprints - Managed DB users + Adminer SSO + from app.api.managed_db_users import managed_db_users_bp + app.register_blueprint(managed_db_users_bp, url_prefix='/api/v1/managed-databases') + + # Register blueprints - Curated DB config tuner + from app.api.db_tuner import db_tuner_bp + app.register_blueprint(db_tuner_bp, url_prefix='/api/v1/db-tuner') + # Register blueprints - Monitoring & Alerts from app.api.monitoring import monitoring_bp app.register_blueprint(monitoring_bp, url_prefix='/api/v1/monitoring') @@ -179,6 +191,10 @@ def create_app(config_name=None): from app.api.snapshots import snapshots_bp app.register_blueprint(snapshots_bp, url_prefix='/api/v1/apps') + # Register blueprints - Declarative serverkit.yaml manifest + from app.api.manifests import manifests_bp + app.register_blueprint(manifests_bp, url_prefix='/api/v1/manifests') + # Register blueprints - Projects & Environments hierarchy from app.api.projects import projects_bp app.register_blueprint(projects_bp, url_prefix='/api/v1/projects') @@ -366,6 +382,13 @@ def create_app(config_name=None): # connection stay core (they back /domains); the extension borrows the single # core CloudflareClient, never a duplicate. + # Register blueprints - DNS provider connections. Core (they back the + # Settings -> Connections DNS tiles and wildcard TLS), but kept at the + # historical /api/v1/email/dns-providers paths from before the email + # extraction so existing frontends keep working. + from app.api.dns_providers import dns_providers_bp + app.register_blueprint(dns_providers_bp, url_prefix='/api/v1/email') + # Register blueprints - Dynamic DNS from app.api.ddns import ddns_bp app.register_blueprint(ddns_bp, url_prefix='/api/v1/ddns') @@ -450,6 +473,30 @@ def create_app(config_name=None): from app.api.ai import ai_bp app.register_blueprint(ai_bp, url_prefix='/api/v1/ai') + # Register blueprints - Server speed test (Monitoring card) + from app.api.speed_test import speedtest_bp + app.register_blueprint(speedtest_bp, url_prefix='/api/v1/speedtest') + + # Register blueprints - Site imports (panel migration pipeline) + from app.api.site_imports import site_imports_bp + app.register_blueprint(site_imports_bp, url_prefix='/api/v1/imports') + + # Register blueprints - Drift detection + doctor sweep + from app.api.doctor import doctor_bp + app.register_blueprint(doctor_bp, url_prefix='/api/v1/doctor') + + # Register blueprints - Diagnostic support bundle + from app.api.support_bundle import support_bundle_bp + app.register_blueprint(support_bundle_bp, url_prefix='/api/v1/support-bundle') + + # Register blueprints - Per-site bandwidth accounting + from app.api.bandwidth import bandwidth_bp + app.register_blueprint(bandwidth_bp, url_prefix='/api/v1/bandwidth') + + # Register blueprints - .htaccess -> nginx converter (apps-prefixed tool) + from app.api.htaccess_tools import htaccess_tools_bp + app.register_blueprint(htaccess_tools_bp, url_prefix='/api/v1/apps') + # Handle database migrations (Alembic) — must run before plugin loader # since the loader queries the installed_plugins table. with app.app_context(): @@ -566,6 +613,28 @@ def create_app(config_name=None): ServerOnboardingService.register_jobs() from app.services.preview_service import PreviewService PreviewService.register_jobs() + from app.services.metadata_guard_service import MetadataGuardService + MetadataGuardService.register_jobs() + if not app.config.get('TESTING'): + MetadataGuardService.ensure() # converge the metadata egress rule (no-op when unsupported) + from app.services.speed_test_service import SpeedTestService + SpeedTestService.register_jobs() + from app.services import login_link_service + login_link_service.register_jobs() + from app.services.db_admin_sso_service import DbAdminSsoService + DbAdminSsoService.register_jobs() + from app.services.site_import_service import SiteImportService + SiteImportService.register_jobs() + from app.services.drift_service import DriftService + DriftService.register_jobs() + from app.services.doctor_service import DoctorService + DoctorService.register_jobs() + from app.services.file_integrity_service import FileIntegrityService + FileIntegrityService.register_jobs() + from app.services.malware_scan_service import MalwareScanService + MalwareScanService.register_jobs() + from app.services.bandwidth_service import BandwidthService + BandwidthService.register_jobs() start_job_system(app, seed=seed_builtin_schedules) # Request body size limit diff --git a/backend/app/api/apps.py b/backend/app/api/apps.py index 82157341..5e6a6967 100644 --- a/backend/app/api/apps.py +++ b/backend/app/api/apps.py @@ -837,9 +837,22 @@ def create_app_from_repository(): if not build_result.get('success'): raise RuntimeError(build_result.get('error', 'Failed to configure build')) + # Stop dropping what we detect: persist the manifest, seed non-secret + # env values, and record the health-check path (plan 17, Phase 1). + manifest_summary = None + try: + from app.services.manifest_persistence_service import ManifestPersistenceService + manifest_summary = ManifestPersistenceService.apply_import( + app, manifest, user_id=current_user_id, + source_repo=deploy_repo_url, source_ref=branch or 'main', + ) + except Exception: + manifest_summary = None + return jsonify({ 'message': 'Repository service created', 'app': _attach_deploy_config(app.to_dict(include_linked=True)), + 'manifest_import': manifest_summary, 'deploy_config': { 'repo_url': _safe_repo_url(deploy_repo_url), 'branch': branch or 'main', @@ -1257,6 +1270,9 @@ def update_app(app_id): app.python_version = data['python_version'] if 'port' in data: app.port = data['port'] + if 'healthcheck_path' in data: + hc = (data['healthcheck_path'] or '').strip() + app.healthcheck_path = hc or None if 'root_path' in data: app.root_path = data['root_path'] if 'docker_image' in data: @@ -1509,6 +1525,198 @@ def sweep_idle_apps(): return jsonify(ContainerSleepService.sweep_idle()) +# ==================== RESOURCE LIMITS (task #23) ==================== + +def _parse_docker_stats(stats): + """Normalize a `docker stats --format json` row to a small usage dict. + + Returns None when stats are unavailable (stopped app, no Docker on the + host — e.g. Windows dev). + """ + if not stats: + return None + + def _pct(value): + try: + return float(str(value).rstrip('%')) + except (TypeError, ValueError): + return None + + usage = { + 'cpu_percent': _pct(stats.get('CPUPerc')), + 'memory_percent': _pct(stats.get('MemPerc')), + 'memory_usage': None, + 'memory_limit': None, + } + mem = stats.get('MemUsage') or '' + if '/' in mem: + used, limit = mem.split('/', 1) + usage['memory_usage'] = used.strip() + usage['memory_limit'] = limit.strip() + return usage + + +@apps_bp.route('//resources', methods=['GET']) +@jwt_required() +def get_app_resources(app_id): + """The app's configured CPU/memory limits plus best-effort live usage.""" + user = User.query.get(get_jwt_identity()) + app = Application.query.get(app_id) + if not app: + return jsonify({'error': 'Application not found'}), 404 + if not _can_access_app(user, app): + return jsonify({'error': 'Access denied'}), 403 + + usage = None + if app.status == 'running' and not app.server_id: + try: + container_id = DockerService.get_app_container_id(app) + if container_id: + usage = _parse_docker_stats(DockerService.get_container_stats(container_id)) + except Exception: + usage = None # live stats are best-effort + + return jsonify({ + 'cpu_limit': app.cpu_limit, + 'memory_limit': app.memory_limit, + 'usage': usage, + }), 200 + + +@apps_bp.route('//resources', methods=['PUT']) +@jwt_required() +def update_app_resources(app_id): + """Set the app's CPU/memory limits; re-apply live when possible. + + Empty/null clears a limit. For a locally managed compose app that is + running, `compose up -d` regenerates the ServerKit override (which carries + the limits) and recreates changed containers; otherwise the limits are + saved and take effect on the next restart/redeploy. + """ + user = User.query.get(get_jwt_identity()) + app = Application.query.get(app_id) + if not app: + return jsonify({'error': 'Application not found'}), 404 + if not _can_edit_app(user, app): + return jsonify({'error': 'Access denied'}), 403 + + data = request.get_json() or {} + try: + if 'cpu_limit' in data: + app.cpu_limit = DockerService.validate_cpu_limit(data['cpu_limit']) + if 'memory_limit' in data: + app.memory_limit = DockerService.validate_memory_limit(data['memory_limit']) + except ValueError as e: + return jsonify({'error': str(e)}), 400 + + db.session.commit() + + applied = False + note = None + is_local_compose = ( + not app.server_id and app.root_path + and (app.app_type == 'docker' or app.managed_by == 'docker_compose') + ) + if app.status == 'running': + if is_local_compose: + result = DockerService.compose_up(app.root_path, compose_file=_local_compose_file(app)) + applied = bool(result.get('success')) + if not applied: + note = 'restart required' + else: + note = 'restart required' + + response = { + 'message': 'Resource limits saved', + 'cpu_limit': app.cpu_limit, + 'memory_limit': app.memory_limit, + 'applied': applied, + } + if note: + response['note'] = note + return jsonify(response), 200 + + +# ==================== MICRO-CACHE (task #21) ==================== + +@apps_bp.route('//micro-cache', methods=['PUT']) +@jwt_required() +def set_micro_cache(app_id): + """Toggle the per-site nginx micro-cache and republish the vhost. + + The flag flows through the same kwargs pipeline drift detection uses + (SiteDomainService.app_vhost_kwargs -> NginxService.render_site_config), + so an enabled cache is part of the expected vhost and never reads as + config drift. When the app has no domains yet, the flag is save-only and + takes effect on first publish. + """ + user = User.query.get(get_jwt_identity()) + app = Application.query.get(app_id) + if not app: + return jsonify({'error': 'Application not found'}), 404 + if not _can_edit_app(user, app): + return jsonify({'error': 'Access denied'}), 403 + + data = request.get_json() or {} + if 'enabled' not in data: + return jsonify({'error': "'enabled' is required"}), 400 + + app.micro_cache_enabled = bool(data['enabled']) + db.session.commit() + + applied = False + note = None + warning = None + if app.domains: + # Reuse the exact publish path (render + write + enable + reload). + from app.services.site_domain_service import SiteDomainService + result = SiteDomainService.write_app_vhost(app) + warning = result.get('warning') + applied = result.get('nginx') is not None and not warning + else: + note = ('Saved. The micro-cache takes effect when this service is ' + 'published at a domain.') + + response = { + 'message': f"Micro-cache {'enabled' if app.micro_cache_enabled else 'disabled'}", + 'micro_cache_enabled': bool(app.micro_cache_enabled), + 'applied': applied, + } + if note: + response['note'] = note + if warning: + response['warning'] = warning + return jsonify(response), 200 + + +@apps_bp.route('//micro-cache/purge', methods=['POST']) +@jwt_required() +def purge_micro_cache(app_id): + """Manually clear the micro-cache. + + The cache is ONE shared nginx zone for all opted-in sites (zones must be + declared statically, so per-site zones don't scale) — purging therefore + wipes cached entries for every opted-in site. With the 10-second TTL this + is near-harmless, and no nginx reload is needed. + """ + user = User.query.get(get_jwt_identity()) + app = Application.query.get(app_id) + if not app: + return jsonify({'error': 'Application not found'}), 404 + if not _can_edit_app(user, app): + return jsonify({'error': 'Access denied'}), 403 + + from app.services.nginx_service import NginxService + result = NginxService.purge_micro_cache() + if not result.get('success'): + return jsonify({'error': result.get('error', 'Purge failed')}), 500 + + response = {'message': result.get('message', 'Micro-cache cleared')} + if result.get('note'): + response['note'] = result['note'] + return jsonify(response), 200 + + @apps_bp.route('//scale-policy', methods=['GET']) @jwt_required() def get_scale_policy(app_id): diff --git a/backend/app/api/auth.py b/backend/app/api/auth.py index 1466f8eb..5eec30f8 100644 --- a/backend/app/api/auth.py +++ b/backend/app/api/auth.py @@ -11,8 +11,11 @@ ) from app import db, limiter from app.models import User, AuditLog, SystemSettings +# Aliased: this module already has a `get_current_user` route handler. +from app.middleware.rbac import admin_required, get_current_user as get_request_user from app.services.settings_service import SettingsService from app.services.audit_service import AuditService +from app.services import login_link_service logger = logging.getLogger(__name__) @@ -264,6 +267,150 @@ def login(): }), 200 +# ========================================== +# ONE-TIME LOGIN LINKS +# ========================================== +@auth_bp.route('/login-links', methods=['POST']) +@admin_required +def create_login_link(): + """Mint a single-use login URL. The raw token is returned exactly once.""" + current = get_request_user() + data = request.get_json() or {} + + target_id = data.get('user_id') or current.id + target = User.query.get(target_id) + if not target: + return jsonify({'error': 'User not found'}), 404 + + bound_ip = (data.get('bound_ip') or '').strip() or None + if bound_ip and len(bound_ip) > 64: + return jsonify({'error': 'Invalid IP address'}), 400 + + token, link = login_link_service.mint( + user_id=target.id, + ttl_minutes=data.get('ttl_minutes'), + bound_ip=bound_ip, + created_by=current.id, + ) + + AuditService.log( + action='login_link.create', + user_id=current.id, + target_type='user', + target_id=target.id, + details={'link_id': link.id, 'expires_at': link.expires_at.isoformat(), + 'ip_bound': bound_ip is not None}, + ) + db.session.commit() + + return jsonify({ + 'url': f'/login?link={token}', + 'token': token, + 'expires_at': link.expires_at.isoformat(), + 'link': link.to_dict(), + }), 201 + + +@auth_bp.route('/login-links', methods=['GET']) +@admin_required +def list_login_links(): + """List active (unused, unexpired) login links. Hashes are never exposed.""" + links = login_link_service.list_active() + return jsonify({'links': [l.to_dict() for l in links]}), 200 + + +@auth_bp.route('/login-links/', methods=['DELETE']) +@admin_required +def revoke_login_link(link_id): + from app.models.login_link import LoginLink + link = LoginLink.query.get(link_id) + if not link: + return jsonify({'error': 'Link not found'}), 404 + + current = get_request_user() + AuditService.log( + action='login_link.revoke', + user_id=current.id, + target_type='user', + target_id=link.user_id, + details={'link_id': link.id}, + ) + db.session.delete(link) + db.session.commit() + return jsonify({'message': 'Login link revoked'}), 200 + + +@auth_bp.route('/login-links/redeem', methods=['POST']) +@limiter.limit("30 per minute") +def redeem_login_link(): + """Redeem a one-time login link and issue normal JWT tokens.""" + data = request.get_json() or {} + token = data.get('token') + + user, reason = login_link_service.redeem(token, request.remote_addr) + if not user: + logger.info(f"Login link redeem failed ({reason}) from {request.remote_addr}") + return jsonify({'error': 'Invalid or expired link'}), 401 + + user.reset_failed_login() + user.last_login_at = datetime.utcnow() + db.session.commit() + + AuditService.log_login(user.id, success=True, details={'method': 'login_link'}) + db.session.commit() + + access_token = create_access_token(identity=user.id) + refresh_token = create_refresh_token(identity=user.id) + + return jsonify({ + 'user': user.to_dict(), + 'access_token': access_token, + 'refresh_token': refresh_token + }), 200 + + +# ========================================== +# DEMO MODE +# ========================================== +@auth_bp.route('/demo-info', methods=['GET']) +def demo_info(): + """Public demo-mode info for the login page. + + Only when demo mode is active: ensures the seeded read-only ``demo`` + user exists (stable random password stored in settings) and returns + its credentials. Otherwise reveals nothing. + """ + from app.middleware.demo import is_demo_mode_active + + if not is_demo_mode_active(): + return jsonify({'enabled': False}), 200 + + import secrets as _secrets + + password = SettingsService.get('demo_password') + user = User.query.filter_by(username='demo').first() + + if not password: + password = _secrets.token_urlsafe(12) + SettingsService.set('demo_password', password) + + if not user: + user = User( + email='demo@demo.local', + username='demo', + role=User.ROLE_VIEWER, + ) + user.set_password(password) + db.session.add(user) + db.session.commit() + elif not user.check_password(password): + # Keep the stored credential authoritative (e.g. rotated setting). + user.set_password(password) + db.session.commit() + + return jsonify({'enabled': True, 'username': 'demo', 'password': password}), 200 + + @auth_bp.route('/refresh', methods=['POST']) @jwt_required(refresh=True) def refresh(): diff --git a/backend/app/api/bandwidth.py b/backend/app/api/bandwidth.py new file mode 100644 index 00000000..e4bdab1b --- /dev/null +++ b/backend/app/api/bandwidth.py @@ -0,0 +1,54 @@ +"""REST surface for per-domain bandwidth accounting. + +Mounted at /api/v1/bandwidth (registered in app/__init__.py). +""" +from flask import Blueprint, jsonify, request + +from ..middleware.rbac import admin_required, viewer_required +from ..services.bandwidth_service import BandwidthService + +bandwidth_bp = Blueprint('bandwidth', __name__) + + +@bandwidth_bp.route('/apps', methods=['GET']) +@viewer_required +def get_apps_bandwidth(): + """Month totals + 30-day sparkline series for every app with traffic — + one call for the Services list.""" + try: + data = BandwidthService.overview(days=30) + # JSON object keys must be strings. + return jsonify({'apps': {str(k): v for k, v in data.items()}}) + except Exception as exc: # noqa: BLE001 — surface as a clean JSON error + return jsonify({'error': str(exc)}), 500 + + +@bandwidth_bp.route('/apps/', methods=['GET']) +@viewer_required +def get_app_bandwidth(app_id): + """Full daily series (default 90 days) + current-month total for one app.""" + try: + days = request.args.get('days', 90, type=int) + series = BandwidthService.series(app_id=app_id, days=days) + return jsonify({ + 'app_id': app_id, + 'days': len(series), + 'series': series, + 'month_bytes': BandwidthService.monthly_total(app_id), + }) + except Exception as exc: # noqa: BLE001 + return jsonify({'error': str(exc)}), 500 + + +@bandwidth_bp.route('/aggregate', methods=['POST']) +@admin_required +def run_aggregate(): + """Run the daily aggregation now (optionally for a specific day).""" + try: + payload = request.get_json(silent=True) or {} + result = BandwidthService.aggregate(day=payload.get('day')) + return jsonify(result) + except ValueError: + return jsonify({'error': 'day must be YYYY-MM-DD'}), 400 + except Exception as exc: # noqa: BLE001 + return jsonify({'error': str(exc)}), 500 diff --git a/backend/app/api/databases.py b/backend/app/api/databases.py index 9249acf3..dd7112ed 100644 --- a/backend/app/api/databases.py +++ b/backend/app/api/databases.py @@ -4,6 +4,7 @@ from flask_jwt_extended import jwt_required, get_jwt_identity from app.models import User, Application from app.services.database_service import DatabaseService +from app.services.db_process_service import DbProcessService from app.services.managed_database_service import ManagedDatabaseService from app.middleware.rbac import admin_required from app.services.resource_grant_service import ResourceGrantService @@ -919,6 +920,77 @@ def protect_managed_database(managed_id): return jsonify({'policy': policy.to_dict()}), 201 +# ==================== PROCESSES ==================== +# Live SHOW PROCESSLIST / pg_stat_activity per server or container, with an +# admin-only kill/terminate action. Targets mirror the explorer's connection +# shapes (host engine vs docker container). + +def _process_error_response(result): + code = 400 if result['error'] == 'unsupported engine' else 502 + return jsonify({'error': result['error']}), code + + +@databases_bp.route('/mysql/processes', defaults={'engine': 'mysql'}, methods=['GET']) +@databases_bp.route('/postgresql/processes', defaults={'engine': 'postgresql'}, methods=['GET']) +@jwt_required() +def list_host_db_processes(engine): + """List live server processes on a host engine.""" + target = {'engine': engine, 'password': request.headers.get('X-DB-Password')} + result = DbProcessService.list_processes(target) + if 'error' in result: + return _process_error_response(result) + return jsonify(result), 200 + + +@databases_bp.route('/mysql/processes//kill', defaults={'engine': 'mysql'}, methods=['POST']) +@databases_bp.route('/postgresql/processes//kill', defaults={'engine': 'postgresql'}, methods=['POST']) +@jwt_required() +@admin_required +def kill_host_db_process(engine, pid): + """Kill/terminate a process on a host engine (admin only).""" + target = {'engine': engine, 'password': request.headers.get('X-DB-Password')} + result = DbProcessService.kill_process(target, pid) + if 'error' in result: + return _process_error_response(result) + return jsonify(result), 200 + + +@databases_bp.route('/docker//processes', methods=['GET']) +@jwt_required() +def list_docker_db_processes(container): + """List live server processes inside a Docker database container.""" + target = { + 'engine': request.args.get('type', 'mysql'), + 'container': container, + 'user': request.args.get('user'), + 'password': request.headers.get('X-DB-Password'), + 'database': request.args.get('database'), + } + result = DbProcessService.list_processes(target) + if 'error' in result: + return _process_error_response(result) + return jsonify(result), 200 + + +@databases_bp.route('/docker//processes//kill', methods=['POST']) +@jwt_required() +@admin_required +def kill_docker_db_process(container, pid): + """Kill/terminate a process inside a Docker database container (admin only).""" + data = request.get_json(silent=True) or {} + target = { + 'engine': data.get('type') or request.args.get('type', 'mysql'), + 'container': container, + 'user': data.get('user'), + 'password': request.headers.get('X-DB-Password') or data.get('password'), + 'database': data.get('database'), + } + result = DbProcessService.kill_process(target, pid) + if 'error' in result: + return _process_error_response(result) + return jsonify(result), 200 + + # ==================== UTILITY ==================== @databases_bp.route('/generate-password', methods=['GET']) diff --git a/backend/app/api/db_tuner.py b/backend/app/api/db_tuner.py new file mode 100644 index 00000000..0817459e --- /dev/null +++ b/backend/app/api/db_tuner.py @@ -0,0 +1,100 @@ +"""Curated DB config tuner API. + +Target addressing (consistent with the databases API, which addresses Docker +databases as ``/databases/docker//...`` by container *name*): + +- ```` is normally the Docker **container name**, with the engine + passed as ``?engine=mysql|mariadb|postgresql`` (query param on GET, body + field on POST). +- As a convenience, an all-digits ```` is treated as a **managed + database id** (``managed_databases`` row with ``host_kind='docker'``); its + ``container_ref``/``engine``/admin credentials are used automatically. + +MySQL auth follows the existing convention: the password travels in the +``X-DB-Password`` header (never in the URL), user via ``user`` param/field. + +All endpoints are admin-only; ``apply`` requires an explicit settings payload +— suggestions are never auto-applied. +""" +from flask import Blueprint, request, jsonify +from flask_jwt_extended import jwt_required + +from app.middleware.rbac import admin_required +from app.services.db_config_tuner_service import DbConfigTunerService + +db_tuner_bp = Blueprint('db_tuner', __name__) + + +def _build_target(target, data): + """Resolve + request context into the service target dict. + → (target_dict, None) or (None, (json, status)).""" + data = data or {} + password = request.headers.get('X-DB-Password') or data.get('password') + user = data.get('user') or request.args.get('user') + + if target.isdigit(): + from app.services.managed_database_service import ManagedDatabaseService + managed = ManagedDatabaseService.get(int(target)) + if not managed: + return None, (jsonify({'error': 'Managed database not found'}), 404) + if managed.host_kind != 'docker' or not managed.container_ref: + return None, (jsonify({'error': 'Managed database is not a Docker container target'}), 400) + engine = DbConfigTunerService.normalize_engine(managed.engine) + if not engine: + return None, (jsonify({'error': f'Unsupported engine: {managed.engine}'}), 400) + if not password and managed.admin_secret_encrypted: + from app.utils.crypto import decrypt_secret_safe + password = decrypt_secret_safe(managed.admin_secret_encrypted) + return { + 'container': managed.container_ref, + 'engine': engine, + 'user': user or managed.admin_username, + 'password': password, + }, None + + engine = DbConfigTunerService.normalize_engine( + data.get('engine') or request.args.get('engine')) + if not engine: + return None, (jsonify({'error': 'engine is required (mysql|mariadb|postgresql)'}), 400) + return {'container': target, 'engine': engine, 'user': user, 'password': password}, None + + +@db_tuner_bp.route('//inspect', methods=['GET']) +@jwt_required() +@admin_required +def inspect_target(target): + """Current vs RAM-aware suggested values for the curated settings.""" + resolved, err = _build_target(target, None) + if err: + return err + dedicated = str(request.args.get('dedicated', '')).lower() in ('1', 'true', 'yes') + result = DbConfigTunerService.inspect(resolved, is_dedicated=dedicated) + return jsonify(result), 200 if 'error' not in result else 400 + + +@db_tuner_bp.route('//apply', methods=['POST']) +@jwt_required() +@admin_required +def apply_target(target): + """Apply an explicit operator-chosen settings dict (restarts the engine).""" + data = request.get_json() or {} + if not data.get('settings'): + return jsonify({'error': 'settings is required'}), 400 + resolved, err = _build_target(target, data) + if err: + return err + result = DbConfigTunerService.apply(resolved, data['settings']) + return jsonify(result), 200 if 'error' not in result else 400 + + +@db_tuner_bp.route('//rollback', methods=['POST']) +@jwt_required() +@admin_required +def rollback_target(target): + """Restore the pre-apply config and restart the engine.""" + data = request.get_json() or {} + resolved, err = _build_target(target, data) + if err: + return err + result = DbConfigTunerService.rollback(resolved) + return jsonify(result), 200 if 'error' not in result else 400 diff --git a/backend/app/api/dns_providers.py b/backend/app/api/dns_providers.py new file mode 100644 index 00000000..cdaf56c8 --- /dev/null +++ b/backend/app/api/dns_providers.py @@ -0,0 +1,64 @@ +"""DNS provider connections (Cloudflare, Route 53, DigitalOcean, GoDaddy). + +These routes back the Settings -> Connections DNS tiles and the wildcard-TLS +flows, so they are core — a panel without the serverkit-email extension must +still be able to connect a DNS provider. They keep the historical +/api/v1/email/dns-providers paths (the routes originally lived in the email +API before the extraction) so existing frontends keep working. +""" +from flask import Blueprint, request, jsonify + +from app.middleware.rbac import admin_required, viewer_required +from app.services.dns_provider_service import DNSProviderService + +dns_providers_bp = Blueprint('dns_providers', __name__) + + +@dns_providers_bp.route('/dns-providers', methods=['GET']) +@viewer_required +def list_dns_providers(): + """List configured DNS providers.""" + providers = DNSProviderService.list_providers() + return jsonify({'providers': providers}), 200 + + +@dns_providers_bp.route('/dns-providers', methods=['POST']) +@admin_required +def add_dns_provider(): + """Add a DNS provider.""" + data = request.get_json() + if not data or not data.get('name') or not data.get('provider') or not data.get('api_key'): + return jsonify({'success': False, 'error': 'Name, provider, and api_key are required'}), 400 + result = DNSProviderService.add_provider( + name=data['name'], + provider=data['provider'], + api_key=data['api_key'], + api_secret=data.get('api_secret'), + api_email=data.get('api_email'), + is_default=data.get('is_default', False), + ) + return jsonify(result), 201 if result.get('success') else 400 + + +@dns_providers_bp.route('/dns-providers/', methods=['DELETE']) +@admin_required +def remove_dns_provider(provider_id): + """Remove a DNS provider.""" + result = DNSProviderService.remove_provider(provider_id) + return jsonify(result), 200 if result.get('success') else 400 + + +@dns_providers_bp.route('/dns-providers//test', methods=['POST']) +@admin_required +def test_dns_provider(provider_id): + """Test DNS provider connection.""" + result = DNSProviderService.test_connection(provider_id) + return jsonify(result), 200 if result.get('success') else 400 + + +@dns_providers_bp.route('/dns-providers//zones', methods=['GET']) +@viewer_required +def list_dns_zones(provider_id): + """List DNS zones from a provider.""" + result = DNSProviderService.list_zones(provider_id) + return jsonify(result), 200 if result.get('success') else 400 diff --git a/backend/app/api/doctor.py b/backend/app/api/doctor.py new file mode 100644 index 00000000..a9ab7f4f --- /dev/null +++ b/backend/app/api/doctor.py @@ -0,0 +1,104 @@ +"""REST surface for the doctor sweep + configuration drift. + +Mounted at /api/v1/doctor (registered in app/__init__.py). Admin-only. + +Contract (the CLI codes against this too): + GET /drift -> {'report': |null} + POST /drift/check -> 202 {'job_id'} + POST /drift///repair -> repair result; body must carry + {'confirm': true} (400 otherwise) + GET / -> {'report': |null} + POST /run -> runs synchronously, {'report': ...} + POST /repair -> {'results': [...]} for body + {'items': [{kind, type?, id?, name?}]} +""" +from flask import Blueprint, jsonify, request + +from ..middleware.rbac import admin_required +from ..services.doctor_service import DoctorService +from ..services.drift_service import DRIFT_JOB_KIND, DriftService + +doctor_bp = Blueprint('doctor', __name__) + + +# --------------------------------------------------------------------------- # +# Drift +# --------------------------------------------------------------------------- # + +@doctor_bp.route('/drift', methods=['GET']) +@admin_required +def get_drift_report(): + """Last stored drift report (null when no sweep has run yet).""" + try: + return jsonify({'report': DriftService.get_last_report()}) + except Exception as exc: # noqa: BLE001 — surface as a clean JSON error + return jsonify({'error': str(exc)}), 500 + + +@doctor_bp.route('/drift/check', methods=['POST']) +@admin_required +def run_drift_check(): + """Enqueue a one-off drift sweep job.""" + try: + from app.jobs.service import JobService + job = JobService.enqueue(DRIFT_JOB_KIND, payload={}, max_attempts=1) + return jsonify({'job_id': job.id}), 202 + except Exception as exc: # noqa: BLE001 + return jsonify({'error': str(exc)}), 500 + + +@doctor_bp.route('/drift///repair', methods=['POST']) +@admin_required +def repair_drift(check_type, resource_id): + """Repair one drifted resource. Explicit confirmation required.""" + data = request.get_json(silent=True) or {} + if data.get('confirm') is not True: + return jsonify({'error': 'Confirmation required: pass {"confirm": true}.'}), 400 + try: + rid = int(resource_id) if resource_id.isdigit() else resource_id + result = DriftService.repair(check_type, rid) + status = 200 if result.get('success') else 400 + return jsonify(result), status + except Exception as exc: # noqa: BLE001 + return jsonify({'error': str(exc)}), 500 + + +# --------------------------------------------------------------------------- # +# Doctor +# --------------------------------------------------------------------------- # + +@doctor_bp.route('', methods=['GET']) +@doctor_bp.route('/', methods=['GET']) +@admin_required +def get_doctor_report(): + """Last stored doctor report (null when the doctor has never run).""" + try: + return jsonify({'report': DoctorService.get_last_report()}) + except Exception as exc: # noqa: BLE001 + return jsonify({'error': str(exc)}), 500 + + +@doctor_bp.route('/run', methods=['POST']) +@admin_required +def run_doctor(): + """Run the sweep synchronously (the doctor is interactive; every internal + probe is time-capped) and return the fresh report.""" + try: + report = DoctorService.run() + return jsonify({'report': report}) + except Exception as exc: # noqa: BLE001 + return jsonify({'error': str(exc)}), 500 + + +@doctor_bp.route('/repair', methods=['POST']) +@admin_required +def repair_items(): + """Batch-repair the explicit items the operator selected.""" + data = request.get_json(silent=True) or {} + items = data.get('items') + if not isinstance(items, list) or not items: + return jsonify({'error': "Body must carry a non-empty 'items' list."}), 400 + try: + return jsonify({'results': DoctorService.repair(items)}) + except Exception as exc: # noqa: BLE001 + return jsonify({'error': str(exc)}), 500 diff --git a/backend/app/api/firewall.py b/backend/app/api/firewall.py index 2f9a82fa..2087126f 100644 --- a/backend/app/api/firewall.py +++ b/backend/app/api/firewall.py @@ -4,6 +4,7 @@ from ..middleware.rbac import admin_required, viewer_required from ..services.firewall_service import FirewallService +from ..services.metadata_guard_service import MetadataGuardService, SETTING_KEY firewall_bp = Blueprint('firewall', __name__) @@ -233,6 +234,35 @@ def set_default_zone(): return jsonify(result), 400 +@firewall_bp.route('/metadata-guard', methods=['GET']) +@viewer_required +def get_metadata_guard(): + """Get cloud metadata guard status.""" + return jsonify(MetadataGuardService.status()), 200 + + +@firewall_bp.route('/metadata-guard', methods=['PUT']) +@admin_required +def set_metadata_guard(): + """Enable or disable the cloud metadata egress guard.""" + data = request.get_json() + + if not data or 'enabled' not in data: + return jsonify({'error': "Request body with 'enabled' required"}), 400 + + enabled = bool(data['enabled']) + + from ..services.settings_service import SettingsService + SettingsService.set(SETTING_KEY, enabled) + + result = MetadataGuardService.ensure() + status = MetadataGuardService.status() + if not result.get('success') and result.get('supported', True): + status['error'] = result.get('error', 'Failed to update metadata guard') + return jsonify(status), 500 + return jsonify(status), 200 + + @firewall_bp.route('/install', methods=['POST']) @admin_required def install_firewall(): diff --git a/backend/app/api/htaccess_tools.py b/backend/app/api/htaccess_tools.py new file mode 100644 index 00000000..3485c4ea --- /dev/null +++ b/backend/app/api/htaccess_tools.py @@ -0,0 +1,35 @@ +"""Small tools blueprint for .htaccess conversion. + +Kept out of apps.py on purpose (separate ownership); intended registration: + + from app.api.htaccess_tools import htaccess_tools_bp + app.register_blueprint(htaccess_tools_bp, url_prefix='/api/v1/apps') + +which exposes POST /api/v1/apps/htaccess-convert. +""" +from flask import Blueprint, jsonify, request +from flask_jwt_extended import jwt_required + +from app.services.htaccess_converter import MAX_INPUT_BYTES, convert + +htaccess_tools_bp = Blueprint('htaccess_tools', __name__) + + +@htaccess_tools_bp.route('/htaccess-convert', methods=['POST']) +@jwt_required() +def htaccess_convert(): + """Convert pasted .htaccess text to nginx directives. + + Pure text transform (no server state touched), so any authenticated + user may call it. Body: {'htaccess': ''}. + """ + data = request.get_json(silent=True) or {} + text = data.get('htaccess') + if not isinstance(text, str) or not text.strip(): + return jsonify({'error': 'htaccess text is required'}), 400 + if len(text.encode('utf-8', errors='replace')) > MAX_INPUT_BYTES: + return jsonify({'error': '.htaccess input exceeds the 256KB limit'}), 413 + try: + return jsonify(convert(text)) + except ValueError as exc: + return jsonify({'error': str(exc)}), 400 diff --git a/backend/app/api/managed_db_users.py b/backend/app/api/managed_db_users.py new file mode 100644 index 00000000..efa7af29 --- /dev/null +++ b/backend/app/api/managed_db_users.py @@ -0,0 +1,113 @@ +"""Managed database users + one-click Adminer SSO. + +Nested under a managed database id: + GET /api/v1/managed-databases//users + POST /api/v1/managed-databases//users + DELETE /api/v1/managed-databases//users/ + POST /api/v1/managed-databases//sso +""" +import logging + +from flask import Blueprint, request, jsonify + +from app.middleware.rbac import admin_required, developer_required +from app.services.managed_database_service import ManagedDatabaseService +from app.services.managed_db_user_service import ManagedDbUserService + +logger = logging.getLogger(__name__) + +managed_db_users_bp = Blueprint('managed_db_users', __name__) + + +def _get_managed_or_404(managed_id): + managed = ManagedDatabaseService.get(managed_id) + if not managed: + return None, (jsonify({'error': 'Managed database not found'}), 404) + return managed, None + + +@managed_db_users_bp.route('//users', methods=['GET']) +@developer_required +def list_managed_db_users(managed_id): + """Tracked users merged best-effort with the live engine list.""" + managed, err = _get_managed_or_404(managed_id) + if err: + return err + users = ManagedDbUserService.list_users(managed) + return jsonify({'users': users}), 200 + + +@managed_db_users_bp.route('//users', methods=['POST']) +@admin_required +def create_managed_db_user(managed_id): + """CREATE USER + GRANT scoped to this database. The password is returned + exactly once in this response and never stored.""" + managed, err = _get_managed_or_404(managed_id) + if err: + return err + data = request.get_json(silent=True) or {} + result = ManagedDbUserService.create_user( + managed, + username=data.get('username'), + password=data.get('password'), + grants=data.get('grants'), + ) + if 'error' in result: + return jsonify({'error': result['error']}), 400 + return jsonify(result), 201 + + +@managed_db_users_bp.route('//users/', methods=['DELETE']) +@admin_required +def delete_managed_db_user(managed_id, user_id): + """DROP USER in the engine and remove the tracking row.""" + from app.models.managed_database_user import ManagedDatabaseUser + managed, err = _get_managed_or_404(managed_id) + if err: + return err + row = ManagedDatabaseUser.query.filter_by( + id=user_id, managed_database_id=managed.id).first() + if not row: + return jsonify({'error': 'User not found'}), 404 + result = ManagedDbUserService.delete_user(managed, row) + if 'error' in result: + return jsonify({'error': result['error']}), 400 + return jsonify({'success': True}), 200 + + +@managed_db_users_bp.route('//sso', methods=['POST']) +@admin_required +def launch_managed_db_sso(managed_id): + """Mint a 5-minute, single-database shadow credential and return the + Adminer launch descriptor. The password crosses once, right here.""" + from app.middleware.rbac import get_current_user + from app.services.db_admin_sso_service import DbAdminSsoService + + managed, err = _get_managed_or_404(managed_id) + if err: + return err + + user = get_current_user() + descriptor = DbAdminSsoService.launch( + managed, requested_by=getattr(user, 'id', None)) + if 'error' in descriptor: + status = 400 if descriptor['error'] != 'Docker required' else 503 + return jsonify({'error': descriptor['error']}), status + + # Build the browser-reachable URL from the panel host the client used. + panel_host = (request.host or 'localhost').rsplit(':', 1)[0] + descriptor['url'] = f"http://{panel_host}:{descriptor['port']}" + + try: + from app.services.audit_service import AuditService + AuditService.log( + action='db_sso_launch', user_id=getattr(user, 'id', None), + target_type='managed_database', target_id=managed.id, + details={'engine': managed.engine, 'name': managed.name, + 'username': descriptor['username'], + 'expires_at': descriptor['expires_at']}, + ) + except Exception as e: # pragma: no cover - audit is best-effort + logger.debug('SSO audit log failed: %s', e) + + return jsonify(descriptor), 200 diff --git a/backend/app/api/manifests.py b/backend/app/api/manifests.py new file mode 100644 index 00000000..79af5d7c --- /dev/null +++ b/backend/app/api/manifests.py @@ -0,0 +1,174 @@ +"""Declarative serverkit.yaml manifest API. + +Phase 0: `scaffold` (render a v1 manifest from a live app). +Later phases add `get`, `plan` and `apply`. Admin-gated; applies are +audit-logged. +""" + +from flask import Blueprint, jsonify, request, Response +from flask_jwt_extended import jwt_required, get_jwt_identity + +from app import db +from app.models.application import Application +from app.models.application_manifest import ApplicationManifest, STATUS_PENDING +from app.models.project import Project +from app.models.user import User +from app.services.manifest_scaffold_service import ManifestScaffoldService +from app.services.manifest_spec_service import ManifestSpecService, ManifestError +from app.services.manifest_apply_service import ManifestApplyService +from app.services.manifest_persistence_service import ManifestPersistenceService + +manifests_bp = Blueprint('manifests', __name__) + + +def _load_normalized(data): + """Normalize from an inline body, or fall back to the project's stored row. + Returns (normalized, raw_text, error_response_or_None).""" + if 'content' in data: + try: + return ManifestSpecService.normalize_text(data['content']), data['content'], None + except ManifestError as exc: + return None, None, (jsonify({'error': 'Invalid manifest', 'errors': exc.errors}), 400) + if 'manifest' in data: + try: + return ManifestSpecService.normalize(data['manifest']), None, None + except ManifestError as exc: + return None, None, (jsonify({'error': 'Invalid manifest', 'errors': exc.errors}), 400) + # fall back to the stored manifest for the project + project_id = data.get('project_id') + row = ApplicationManifest.query.filter_by(project_id=project_id).first() if project_id else None + if row and row.get_normalized(): + return row.get_normalized(), row.raw_text, None + return None, None, (jsonify({'error': 'Provide `content`/`manifest`, or store one first'}), 400) + + +def _require_admin(): + user = User.query.get(get_jwt_identity()) + if not user or user.role != 'admin': + return None, (jsonify({'error': 'Admin access required'}), 403) + return user, None + + +@manifests_bp.route('/scaffold', methods=['GET']) +@jwt_required() +def scaffold_manifest(): + """GET /api/v1/manifests/scaffold?app_id=&format=json|yaml""" + user, err = _require_admin() + if err: + return err + + app_id = request.args.get('app_id', type=int) + if not app_id: + return jsonify({'error': 'app_id is required'}), 400 + + app = Application.query.get(app_id) + if not app: + return jsonify({'error': 'Application not found'}), 404 + + fmt = (request.args.get('format') or 'json').lower() + if fmt == 'yaml': + yaml_text = ManifestScaffoldService.scaffold_yaml(app) + return Response(yaml_text, mimetype='text/yaml') + + manifest = ManifestScaffoldService.scaffold_for_app(app) + return jsonify({'manifest': manifest, + 'yaml': ManifestScaffoldService.scaffold_yaml(app)}), 200 + + +@manifests_bp.route('/validate', methods=['POST']) +@jwt_required() +def validate_manifest(): + """POST /api/v1/manifests/validate { content | manifest } -> normalized summary.""" + user, err = _require_admin() + if err: + return err + + data = request.get_json(silent=True) or {} + try: + if 'content' in data: + normalized = ManifestSpecService.normalize_text(data['content']) + elif 'manifest' in data: + normalized = ManifestSpecService.normalize(data['manifest']) + else: + return jsonify({'error': 'Provide `content` or `manifest`'}), 400 + except ManifestError as exc: + return jsonify({'valid': False, 'errors': exc.errors}), 200 + + return jsonify({ + 'valid': True, + 'summary': ManifestSpecService.summarize(normalized), + 'normalized': normalized, + }), 200 + + +@manifests_bp.route('', methods=['GET']) +@jwt_required() +def get_manifest(): + """GET /api/v1/manifests?project_id= -> the stored manifest for a project.""" + user, err = _require_admin() + if err: + return err + project_id = request.args.get('project_id', type=int) + if not project_id: + return jsonify({'error': 'project_id is required'}), 400 + row = ApplicationManifest.query.filter_by(project_id=project_id).first() + if not row: + return jsonify({'manifest': None}), 200 + return jsonify({'manifest': row.to_dict(include_raw=True)}), 200 + + +@manifests_bp.route('/plan', methods=['POST']) +@jwt_required() +def plan_manifest(): + """POST /api/v1/manifests/plan {project_id, content|manifest?} -> dry-run plan.""" + user, err = _require_admin() + if err: + return err + data = request.get_json(silent=True) or {} + project = Project.query.get(data.get('project_id')) if data.get('project_id') else None + if not project: + return jsonify({'error': 'A valid project_id is required'}), 400 + + normalized, _raw, err_resp = _load_normalized(data) + if err_resp: + return err_resp + + plan = ManifestApplyService.plan(project, normalized) + return jsonify({'plan': plan}), 200 + + +@manifests_bp.route('/apply', methods=['POST']) +@jwt_required() +def apply_manifest(): + """POST /api/v1/manifests/apply {project_id, content|manifest?} -> apply.""" + user, err = _require_admin() + if err: + return err + data = request.get_json(silent=True) or {} + project = Project.query.get(data.get('project_id')) if data.get('project_id') else None + if not project: + return jsonify({'error': 'A valid project_id is required'}), 400 + + normalized, raw, err_resp = _load_normalized(data) + if err_resp: + return err_resp + + # persist/refresh the manifest row (pending) before applying + row = ManifestPersistenceService.store_manifest( + project_id=project.id, normalized=normalized, raw_text=raw, status=STATUS_PENDING) + db.session.commit() + + result = ManifestApplyService.apply(project, normalized, user_id=user.id, + manifest_row=row) + + try: + from app.services.audit_service import AuditService + AuditService.log('manifest.apply', user_id=user.id, target_type='project', + target_id=project.id, + details={'success': result['success'], 'applied': result['applied'], + 'job_id': result['job_id']}) + except Exception: + pass + + status = 200 if result['success'] else 207 + return jsonify(result), status diff --git a/backend/app/api/security.py b/backend/app/api/security.py index 612f6aa9..18116b5a 100644 --- a/backend/app/api/security.py +++ b/backend/app/api/security.py @@ -2,9 +2,16 @@ from flask_jwt_extended import jwt_required from app.middleware.rbac import admin_required from app.services.security_service import SecurityService +from app.services.malware_scan_service import MalwareScanService +from app.services.yara_scan_service import YaraScanService security_bp = Blueprint('security', __name__) +# Register the security.malware_scan job handler at boot (this module is +# imported by create_app); registration is idempotent and also re-asserted on +# every enqueue. +MalwareScanService.register_jobs() + # ========================================== # STATUS & CONFIG @@ -111,6 +118,34 @@ def scan_directory(): return jsonify(result), 200 if result['success'] else 400 +@security_bp.route('/scan/app/', methods=['POST']) +@jwt_required() +@admin_required +def scan_app(app_id): + """Enqueue a job-backed malware scan (YARA + ClamAV) of an app's docroot.""" + try: + job = MalwareScanService.enqueue_scan(app_id=app_id) + except ValueError as e: + return jsonify({'error': str(e)}), 404 if 'not found' in str(e) else 400 + return jsonify({'job_id': job.id, 'kind': job.kind, + 'path': (job.get_payload() or {}).get('path')}), 202 + + +@security_bp.route('/scan/job', methods=['POST']) +@jwt_required() +@admin_required +def scan_path_job(): + """Enqueue a job-backed malware scan of an arbitrary path.""" + data = request.get_json() + if not data or not data.get('path'): + return jsonify({'error': 'Path required'}), 400 + try: + job = MalwareScanService.enqueue_scan(path=data['path']) + except ValueError as e: + return jsonify({'error': str(e)}), 400 + return jsonify({'job_id': job.id, 'kind': job.kind, 'path': data['path']}), 202 + + @security_bp.route('/scan/status', methods=['GET']) @jwt_required() def get_scan_status(): @@ -171,6 +206,51 @@ def delete_quarantined_file(filename): return jsonify(result), 200 if result['success'] else 400 +@security_bp.route('/quarantine//restore', methods=['POST']) +@jwt_required() +@admin_required +def restore_quarantined_file(filename): + """Restore a quarantined file to its recorded original location.""" + result = SecurityService.restore_quarantined_file(filename) + return jsonify(result), 200 if result['success'] else 400 + + +# ========================================== +# YARA RULES (web-shell pass) +# ========================================== +@security_bp.route('/yara/rules', methods=['GET']) +@jwt_required() +@admin_required +def list_yara_rules(): + """List builtin (curated) and custom YARA rule files.""" + return jsonify(YaraScanService.list_rules()), 200 + + +@security_bp.route('/yara/rules', methods=['POST']) +@jwt_required() +@admin_required +def upload_yara_rule(): + """Upload a custom .yar rule file (size-capped; requires real yara to run).""" + data = request.get_json() + if not data or not data.get('filename'): + return jsonify({'error': 'filename required'}), 400 + result = YaraScanService.save_custom_rule(data['filename'], data.get('content', '')) + if not result['success']: + return jsonify({'error': result['error']}), 400 + return jsonify(result), 200 + + +@security_bp.route('/yara/rules/', methods=['DELETE']) +@jwt_required() +@admin_required +def delete_yara_rule(filename): + """Delete a custom .yar rule file.""" + result = YaraScanService.delete_custom_rule(filename) + if not result['success']: + return jsonify({'error': result['error']}), 400 + return jsonify(result), 200 + + # ========================================== # FILE INTEGRITY # ========================================== @@ -577,3 +657,89 @@ def get_sbom(sbom_id): if not sbom: return jsonify({'error': 'SBOM not found'}), 404 return jsonify(sbom.to_dict(include_sbom=True)), 200 + + +# ========================================== +# FILE INTEGRITY MONITORING (SCOPED FIM) +# ========================================== +# Baseline-and-diff over ServerKit-managed paths (nginx / systemd / +# opted-in app docroots). The legacy /integrity/* endpoints above are kept +# as-is for compatibility; this is the scoped surface the UI uses. + +@security_bp.route('/fim', methods=['GET']) +@jwt_required() +def get_fim_status(): + """Get FIM scopes, baselines and last check results.""" + from app.services.file_integrity_service import FileIntegrityService + return jsonify(FileIntegrityService.get_status()), 200 + + +@security_bp.route('/fim//baseline', methods=['POST']) +@jwt_required() +@admin_required +def fim_baseline(scope): + """Create (or recreate) the baseline for a scope.""" + from app.services.file_integrity_service import ( + FileIntegrityService, FileIntegrityScopeError, + ) + data = request.get_json(silent=True) or {} + options = data.get('options') if isinstance(data.get('options'), dict) else None + try: + result = FileIntegrityService.baseline(scope, options=options) + except FileIntegrityScopeError as exc: + return jsonify({'error': str(exc)}), 400 + except Exception as exc: + return jsonify({'error': str(exc)}), 500 + return jsonify(result), 200 + + +@security_bp.route('/fim//check', methods=['POST']) +@jwt_required() +@admin_required +def fim_check(scope): + """Diff the scope against its baseline.""" + from app.services.file_integrity_service import ( + FileIntegrityService, FileIntegrityScopeError, + ) + try: + result = FileIntegrityService.check(scope) + except FileIntegrityScopeError as exc: + return jsonify({'error': str(exc)}), 400 + except Exception as exc: + return jsonify({'error': str(exc)}), 500 + return jsonify(result), 200 + + +@security_bp.route('/fim//accept', methods=['POST']) +@jwt_required() +@admin_required +def fim_accept(scope): + """Accept current state (re-baseline the scope).""" + from app.services.file_integrity_service import ( + FileIntegrityService, FileIntegrityScopeError, + ) + try: + result = FileIntegrityService.accept(scope) + except FileIntegrityScopeError as exc: + return jsonify({'error': str(exc)}), 400 + except Exception as exc: + return jsonify({'error': str(exc)}), 500 + return jsonify(result), 200 + + +@security_bp.route('/fim/apps', methods=['PUT']) +@jwt_required() +@admin_required +def fim_set_app_optins(): + """Set the per-app FIM opt-in list: {app_ids: [1, 2, ...]}.""" + from app.services.file_integrity_service import ( + FileIntegrityService, FileIntegrityScopeError, + ) + data = request.get_json(silent=True) + if not data or 'app_ids' not in data: + return jsonify({'error': 'app_ids required'}), 400 + try: + ids = FileIntegrityService.set_app_optins(data['app_ids']) + except FileIntegrityScopeError as exc: + return jsonify({'error': str(exc)}), 400 + return jsonify({'app_optins': ids}), 200 diff --git a/backend/app/api/site_imports.py b/backend/app/api/site_imports.py new file mode 100644 index 00000000..71228796 --- /dev/null +++ b/backend/app/api/site_imports.py @@ -0,0 +1,108 @@ +"""Site imports API — migrate a control-panel backup archive into ServerKit. + +Mounted at /api/v1/imports (see app/__init__.py blueprint registration). +All routes are admin-only: an import writes files, creates apps and touches +the database engine. +""" +from flask import Blueprint, jsonify, request + +from app.middleware.rbac import admin_required, get_current_user +from app.services.site_import_service import SiteImportError, SiteImportService + +site_imports_bp = Blueprint('site_imports', __name__) + + +@site_imports_bp.route('/upload', methods=['POST']) +@admin_required +def upload_archive(): + """Accept a backup archive upload; returns the token to pass as + source.upload_path when creating the import.""" + if 'file' not in request.files: + return jsonify({'error': 'No file provided'}), 400 + uploaded = request.files['file'] + if not uploaded or not uploaded.filename: + return jsonify({'error': 'No file selected'}), 400 + try: + upload_path = SiteImportService.save_upload(uploaded) + except SiteImportError as exc: + return jsonify({'error': str(exc)}), 400 + return jsonify({'upload_path': upload_path}), 201 + + +@site_imports_bp.route('', methods=['POST']) +@admin_required +def create_import(): + data = request.get_json(silent=True) or {} + user = get_current_user() + try: + imp = SiteImportService.create( + source_type=data.get('source_type') or 'cpanel', + source=data.get('source') or {}, + options=data.get('options') or {}, + user_id=user.id if user else None, + ) + except SiteImportError as exc: + return jsonify({'error': str(exc)}), 400 + return jsonify({'import': imp.to_dict()}), 201 + + +@site_imports_bp.route('', methods=['GET']) +@admin_required +def list_imports(): + return jsonify({'imports': [i.to_dict(log_lines=0) for i in + SiteImportService.list()]}) + + +@site_imports_bp.route('/', methods=['GET']) +@admin_required +def get_import(import_id): + imp = SiteImportService.get(import_id) + if not imp: + return jsonify({'error': 'Import not found'}), 404 + return jsonify({'import': imp.to_dict(log_lines=500)}) + + +@site_imports_bp.route('//analyze', methods=['POST']) +@admin_required +def analyze_import(import_id): + imp = SiteImportService.get(import_id) + if not imp: + return jsonify({'error': 'Import not found'}), 404 + if imp.status in ('analyzing', 'running'): + return jsonify({'error': f'Import is currently {imp.status}'}), 409 + job = SiteImportService.enqueue_analyze(imp) + return jsonify({'job_id': job.id}), 202 + + +@site_imports_bp.route('//run', methods=['POST']) +@admin_required +def run_import(import_id): + imp = SiteImportService.get(import_id) + if not imp: + return jsonify({'error': 'Import not found'}), 404 + if imp.status not in ('analyzed', 'failed'): + return jsonify({'error': "Import must be analysed before it can run " + f"(status: {imp.status})"}), 409 + data = request.get_json(silent=True) or {} + from_step = data.get('from_step') or None + # Merge wizard-provided run options into the stored ones (body wins), + # e.g. {'skip_db': True, 'skip_crontab': True}. + body_options = data.get('options') or {} + if body_options: + from app import db + imp.set_options({**imp.get_options(), **body_options}) + db.session.commit() + job = SiteImportService.enqueue_run(imp, from_step=from_step) + return jsonify({'job_id': job.id}), 202 + + +@site_imports_bp.route('/', methods=['DELETE']) +@admin_required +def delete_import(import_id): + imp = SiteImportService.get(import_id) + if not imp: + return jsonify({'error': 'Import not found'}), 404 + if imp.status in ('analyzing', 'running'): + return jsonify({'error': f'Import is currently {imp.status}'}), 409 + SiteImportService.delete(imp) + return jsonify({'message': 'Import deleted'}) diff --git a/backend/app/api/speed_test.py b/backend/app/api/speed_test.py new file mode 100644 index 00000000..666aea26 --- /dev/null +++ b/backend/app/api/speed_test.py @@ -0,0 +1,35 @@ +"""REST surface for the on-demand server speed test. + +Mounted at /api/v1/speedtest (registered in app/__init__.py). +""" +from flask import Blueprint, jsonify + +from ..middleware.rbac import admin_required, viewer_required +from ..services.speed_test_service import SPEEDTEST_JOB_KIND, SpeedTestService + +speedtest_bp = Blueprint('speedtest', __name__) + + +@speedtest_bp.route('', methods=['GET']) +@speedtest_bp.route('/', methods=['GET']) +@viewer_required +def get_speed_test(): + """Last stored result + status of any in-flight speed test job.""" + try: + return jsonify(SpeedTestService.get_status()) + except Exception as exc: # noqa: BLE001 — surface as a clean JSON error + return jsonify({'error': str(exc)}), 500 + + +@speedtest_bp.route('/run', methods=['POST']) +@admin_required +def run_speed_test(): + """Enqueue a one-off speed test job. Rejects if one is already in flight.""" + try: + if SpeedTestService.is_running(): + return jsonify({'error': 'A speed test is already in progress.'}), 409 + from app.jobs.service import JobService + job = JobService.enqueue(SPEEDTEST_JOB_KIND, payload={}, max_attempts=1) + return jsonify({'job_id': job.id}), 202 + except Exception as exc: # noqa: BLE001 + return jsonify({'error': str(exc)}), 500 diff --git a/backend/app/api/support_bundle.py b/backend/app/api/support_bundle.py new file mode 100644 index 00000000..a71d9b12 --- /dev/null +++ b/backend/app/api/support_bundle.py @@ -0,0 +1,37 @@ +"""Support bundle API: build and download a scrubbed diagnostic zip. + +Registration (app/__init__.py): + from app.api.support_bundle import support_bundle_bp + app.register_blueprint(support_bundle_bp, url_prefix='/api/v1/support-bundle') +""" +import os +from datetime import datetime + +from flask import Blueprint, jsonify, send_file + +from app.middleware.rbac import admin_required +from app.services import support_bundle_service + +support_bundle_bp = Blueprint('support_bundle', __name__) + + +@support_bundle_bp.route('', methods=['POST']) +@admin_required +def build_support_bundle(): + """Build a diagnostic support bundle and return it as a zip download.""" + try: + path = support_bundle_service.build() + except Exception as exc: # noqa: BLE001 - surface as a JSON error, not a 500 page + return jsonify({'error': f'Failed to build support bundle: {exc}'}), 500 + + if not os.path.isfile(path): + return jsonify({'error': 'Support bundle was not created'}), 500 + + filename = f"serverkit-support-{datetime.utcnow().strftime('%Y-%m-%d')}.zip" + return send_file( + path, + mimetype='application/zip', + as_attachment=True, + download_name=filename, + max_age=0, + ) diff --git a/backend/app/jobs/builtin_handlers.py b/backend/app/jobs/builtin_handlers.py index 2a092523..c6e0aa71 100644 --- a/backend/app/jobs/builtin_handlers.py +++ b/backend/app/jobs/builtin_handlers.py @@ -341,3 +341,38 @@ def seed_builtin_schedules(): ScheduledJobService.ensure( name, kind, interval_seconds=interval, startup_delay_seconds=delay, ) + # One-time login-link reaper — handler registered by + # login_link_service.register_jobs() at boot. + from app.services import login_link_service + ScheduledJobService.ensure( + login_link_service.REAP_SCHEDULE_NAME, login_link_service.REAP_JOB_KIND, + interval_seconds=3600, startup_delay_seconds=120, + ) + # Adminer SSO shadow-credential reaper — handler registered by + # DbAdminSsoService.register_jobs() at boot. + from app.services import db_admin_sso_service + ScheduledJobService.ensure( + db_admin_sso_service.REAP_SCHEDULE_NAME, db_admin_sso_service.REAP_JOB_KIND, + interval_seconds=300, startup_delay_seconds=120, + ) + # Daily configuration-drift sweep — handler registered by + # DriftService.register_jobs() at boot. + from app.services.drift_service import DRIFT_JOB_KIND, DRIFT_SCHEDULE_NAME + ScheduledJobService.ensure( + DRIFT_SCHEDULE_NAME, DRIFT_JOB_KIND, + interval_seconds=86400, startup_delay_seconds=900, + ) + # File-integrity sweep — handler registered by + # FileIntegrityService.register_jobs() at boot. + from app.services import file_integrity_service + ScheduledJobService.ensure( + file_integrity_service.FIM_SCHEDULE_NAME, file_integrity_service.FIM_JOB_KIND, + interval_seconds=21600, startup_delay_seconds=1200, + ) + # Daily per-site bandwidth aggregation — handler registered by + # BandwidthService.register_jobs() at boot. + from app.services import bandwidth_service + ScheduledJobService.ensure( + bandwidth_service.BANDWIDTH_SCHEDULE_NAME, bandwidth_service.BANDWIDTH_JOB_KIND, + interval_seconds=86400, startup_delay_seconds=1800, + ) diff --git a/backend/app/middleware/demo.py b/backend/app/middleware/demo.py new file mode 100644 index 00000000..852988b6 --- /dev/null +++ b/backend/app/middleware/demo.py @@ -0,0 +1,54 @@ +"""Demo mode middleware. + +When active, every mutating /api/v1/* request is rejected with +``403 {'error': 'demo_mode'}`` so a public demo panel stays read-only. +Activation: env ``SERVERKIT_DEMO_MODE=1`` (wins) or the ``demo_mode`` +system setting. Default OFF. +""" +import os + +from flask import request, jsonify + +MUTATING_METHODS = {'POST', 'PUT', 'PATCH', 'DELETE'} + +# Auth flows must keep working so visitors can sign in as the demo user. +ALLOWLIST = { + '/api/v1/auth/login', + '/api/v1/auth/refresh', + '/api/v1/auth/logout', + '/api/v1/auth/login-links/redeem', +} + + +def is_demo_mode_active(): + """True when demo mode is on. The env var wins over the setting.""" + env = os.environ.get('SERVERKIT_DEMO_MODE') + if env is not None: + return env.strip().lower() in ('1', 'true', 'yes', 'on') + try: + from app.services.settings_service import SettingsService + return bool(SettingsService.get('demo_mode', False)) + except Exception: + return False + + +def init_demo_mode(app): + """Register the demo-mode guard. Idempotent (safe to call twice).""" + if app.extensions.get('serverkit_demo_mode'): + return + app.extensions['serverkit_demo_mode'] = True + + @app.before_request + def _demo_mode_guard(): + if request.method not in MUTATING_METHODS: + return None # GET/HEAD/OPTIONS always pass + path = request.path.rstrip('/') + if not path.startswith('/api/v1/'): + return None + if path in ALLOWLIST: + return None + # Only consult config once we know the request would be blocked — + # cheap per-request: mutating API calls only. + if not is_demo_mode_active(): + return None + return jsonify({'error': 'demo_mode'}), 403 diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.py index 5f8f6fe2..b0fce330 100644 --- a/backend/app/models/__init__.py +++ b/backend/app/models/__init__.py @@ -31,6 +31,7 @@ from app.models.registrar_connection import RegistrarConnection from app.models.container_registry import ContainerRegistry from app.models.app_volume import AppVolume +from app.models.application_manifest import ApplicationManifest from app.models.managed_database import ManagedDatabase from app.models.api_key import ApiKey from app.models.api_usage import ApiUsageLog, ApiUsageSummary @@ -64,6 +65,10 @@ from app.models.email_provider import EmailProviderConnection from app.models.system_event import SystemEvent from app.models.domain_registration import DomainRegistration +from app.models.login_link import LoginLink +from app.models.managed_database_user import ManagedDatabaseUser +from app.models.site_import import SiteImport +from app.models.site_bandwidth import SiteBandwidthDaily __all__ = [ 'User', 'Application', 'Domain', 'EnvironmentVariable', 'EnvironmentVariableHistory', @@ -104,4 +109,8 @@ 'ApplicationPreview', 'ApplicationPreviewSettings', 'ProxyStack', 'SiteBaseDomain', + 'LoginLink', + 'ManagedDatabaseUser', + 'SiteImport', + 'SiteBandwidthDaily', ] diff --git a/backend/app/models/application.py b/backend/app/models/application.py index 19d50df2..39b3de5d 100644 --- a/backend/app/models/application.py +++ b/backend/app/models/application.py @@ -26,6 +26,10 @@ class Application(db.Model): python_version = db.Column(db.String(10), nullable=True) # '3.9', '3.10', '3.11', '3.12' port = db.Column(db.Integer, nullable=True) root_path = db.Column(db.String(500), nullable=True) + # HTTP path used for health checks and the zero-downtime restart gate. + # Populated from a manifest at import, editable in Settings. NULL => no gate + # (the restart falls back to a fixed wait). + healthcheck_path = db.Column(db.String(255), nullable=True) # Docker specific docker_image = db.Column(db.String(200), nullable=True) @@ -38,6 +42,11 @@ class Application(db.Model): # Build packs (zero-Dockerfile deploys). When the build method routes through # the build-pack layer, the detected plan and any user overrides are persisted # here so the generated Dockerfile is reproducible and the UI can show it. + # Per-app resource limits (task #23). Docker-enforced caps emitted into the + # generated compose service block (`cpus` / `mem_limit`). NULL = unlimited. + cpu_limit = db.Column(db.String(16), nullable=True) # CPU cores, e.g. '1.5' + memory_limit = db.Column(db.String(16), nullable=True) # e.g. '512m', '2g' + buildpack_type = db.Column(db.String(20), nullable=True) # 'nixpacks' | 'static' | 'dockerfile-present' | 'unknown' buildpack_plan = db.Column(db.Text, nullable=True) # JSON: the detected build plan buildpack_overrides = db.Column(db.Text, nullable=True) # JSON: user overrides applied to the plan @@ -64,6 +73,11 @@ class Application(db.Model): private_slug = db.Column(db.String(50), unique=True, nullable=True, index=True) private_url_enabled = db.Column(db.Boolean, default=False) + # Opt-in nginx micro-cache (task #21): short-TTL page cache emitted into + # the site's vhost, with bypasses for auth/admin/cart traffic. NULL/False + # = off (today's behavior). + micro_cache_enabled = db.Column(db.Boolean, default=False, nullable=True) + # Environment linking environment_type = db.Column(db.String(20), default='standalone') # 'production', 'development', 'staging', 'standalone' linked_app_id = db.Column(db.Integer, db.ForeignKey('applications.id'), nullable=True) @@ -106,10 +120,13 @@ def to_dict(self, include_linked=False): 'php_version': self.php_version, 'python_version': self.python_version, 'port': self.port, + 'healthcheck_path': self.healthcheck_path, 'root_path': self.root_path, 'docker_image': self.docker_image, 'container_id': self.container_id, 'registry_id': self.registry_id, + 'cpu_limit': self.cpu_limit, + 'memory_limit': self.memory_limit, 'buildpack_type': self.buildpack_type, 'buildpack_plan': json.loads(self.buildpack_plan) if self.buildpack_plan else None, 'buildpack_overrides': json.loads(self.buildpack_overrides) if self.buildpack_overrides else None, @@ -123,6 +140,7 @@ def to_dict(self, include_linked=False): 'upload_path': self.upload_path, 'private_slug': self.private_slug, 'private_url_enabled': self.private_url_enabled, + 'micro_cache_enabled': bool(self.micro_cache_enabled), 'environment_type': self.environment_type, 'linked_app_id': self.linked_app_id, 'shared_config': json.loads(self.shared_config) if self.shared_config else None, diff --git a/backend/app/models/application_manifest.py b/backend/app/models/application_manifest.py new file mode 100644 index 00000000..2953428b --- /dev/null +++ b/backend/app/models/application_manifest.py @@ -0,0 +1,82 @@ +"""Persisted declarative manifest for a project. + +One row per project (the manifest maps to a Project). Import stores the raw +text + normalized JSON + hash + source so nothing detected is discarded and +later pushes can re-read it. History lives in the existing DeploymentSnapshot +mechanism, not a second table. +""" + +import json +from datetime import datetime + +from app import db + + +# status values +STATUS_PENDING = 'pending' # a changed manifest awaiting an explicit apply +STATUS_APPLIED = 'applied' # live state matches the declared spec +STATUS_DRIFTED = 'drifted' # live state diverged from the manifest +STATUS_ERROR = 'error' # last apply / parse failed + + +class ApplicationManifest(db.Model): + __tablename__ = 'application_manifests' + + id = db.Column(db.Integer, primary_key=True) + project_id = db.Column(db.Integer, db.ForeignKey('projects.id'), nullable=False, index=True) + + raw_text = db.Column(db.Text, nullable=True) # the manifest file as committed + normalized_json = db.Column(db.Text, nullable=True) # ManifestSpecService.normalize() output + manifest_hash = db.Column(db.String(64), nullable=True, index=True) + + # provenance + source_repo = db.Column(db.String(500), nullable=True) + source_ref = db.Column(db.String(200), nullable=True) + source_commit = db.Column(db.String(64), nullable=True) + source_path = db.Column(db.String(255), nullable=True, default='serverkit.yaml') + + status = db.Column(db.String(20), nullable=False, default=STATUS_PENDING, index=True) + last_error = db.Column(db.Text, nullable=True) + applied_at = db.Column(db.DateTime, nullable=True) + created_at = db.Column(db.DateTime, default=datetime.utcnow) + updated_at = db.Column(db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow) + + __table_args__ = ( + db.UniqueConstraint('project_id', name='uq_application_manifest_project'), + ) + + def get_normalized(self): + if not self.normalized_json: + return None + try: + return json.loads(self.normalized_json) + except Exception: + return None + + def set_normalized(self, value): + self.normalized_json = json.dumps(value) if value is not None else None + + def to_dict(self, include_raw=False): + data = { + 'id': self.id, + 'project_id': self.project_id, + 'manifest_hash': self.manifest_hash, + 'status': self.status, + 'last_error': self.last_error, + 'source': { + 'repo': self.source_repo, + 'ref': self.source_ref, + 'commit': self.source_commit, + 'path': self.source_path, + }, + 'normalized': self.get_normalized(), + 'applied_at': self.applied_at.isoformat() if self.applied_at else None, + 'created_at': self.created_at.isoformat() if self.created_at else None, + 'updated_at': self.updated_at.isoformat() if self.updated_at else None, + } + if include_raw: + data['raw_text'] = self.raw_text + return data + + def __repr__(self): + return f'' diff --git a/backend/app/models/env_variable.py b/backend/app/models/env_variable.py index b472c617..3bacdc10 100644 --- a/backend/app/models/env_variable.py +++ b/backend/app/models/env_variable.py @@ -22,6 +22,12 @@ class EnvironmentVariable(db.Model): # Compose service this var targets (NULL = all services). Lets a compose app # scope a variable to one service in the managed env overlay. target_service = db.Column(db.String(120), nullable=True) + # Reference source (manifest fromSecret/fromService/generate). When set, the + # real value is resolved at injection time and never stored in + # encrypted_value — masking stays intact and a rotated secret propagates on + # the next deploy/restart. JSON e.g. {"kind":"secret","secret":"stripe"} or + # {"kind":"service","service":"db","property":"connectionString"}. + value_from = db.Column(db.Text, nullable=True) created_at = db.Column(db.DateTime, default=datetime.utcnow) updated_at = db.Column(db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow) @@ -75,8 +81,23 @@ def value(self, plaintext): """Set the value (encrypts automatically).""" self.encrypted_value = self.encrypt_value(plaintext) + def get_reference(self): + """Parsed value_from reference, or None for a plain value.""" + if not self.value_from: + return None + import json + try: + return json.loads(self.value_from) + except Exception: + return None + + def set_reference(self, ref): + import json + self.value_from = json.dumps(ref) if ref else None + def to_dict(self, include_value=True, mask_secrets=False): """Convert to dictionary, optionally masking secret values.""" + reference = self.get_reference() result = { 'id': self.id, 'application_id': self.application_id, @@ -84,12 +105,17 @@ def to_dict(self, include_value=True, mask_secrets=False): 'is_secret': self.is_secret, 'description': self.description, 'target_service': self.target_service, + 'is_reference': bool(reference), + 'value_from': reference, 'created_at': self.created_at.isoformat() if self.created_at else None, 'updated_at': self.updated_at.isoformat() if self.updated_at else None, } if include_value: - if mask_secrets and self.is_secret: + if reference: + # a reference's real value is never serialized here + result['value'] = '••••••••' if mask_secrets else '' + elif mask_secrets and self.is_secret: result['value'] = '••••••••' else: result['value'] = self.value diff --git a/backend/app/models/login_link.py b/backend/app/models/login_link.py new file mode 100644 index 00000000..80e77c3b --- /dev/null +++ b/backend/app/models/login_link.py @@ -0,0 +1,57 @@ +from datetime import datetime + +from app import db + + +class LoginLink(db.Model): + """A single-use, short-TTL login URL minted by an admin. + + Only the SHA-256 hash of the token is stored; the raw token is returned + exactly once at mint time. A link may optionally be bound to a single + client IP. Rows are reaped on a schedule once used or expired. + """ + + __tablename__ = 'login_links' + + id = db.Column(db.Integer, primary_key=True) + token_hash = db.Column(db.String(64), nullable=False, unique=True, index=True) + user_id = db.Column( + db.Integer, db.ForeignKey('users.id', ondelete='CASCADE'), + nullable=False, index=True, + ) + created_by_id = db.Column( + db.Integer, db.ForeignKey('users.id', ondelete='SET NULL'), + nullable=True, + ) + expires_at = db.Column(db.DateTime, nullable=False) + used_at = db.Column(db.DateTime, nullable=True) + bound_ip = db.Column(db.String(64), nullable=True) + created_at = db.Column(db.DateTime, default=datetime.utcnow) + + user = db.relationship('User', foreign_keys=[user_id]) + created_by = db.relationship('User', foreign_keys=[created_by_id]) + + @property + def is_expired(self): + return self.expires_at is not None and datetime.utcnow() >= self.expires_at + + @property + def is_used(self): + return self.used_at is not None + + def to_dict(self): + # NEVER expose token_hash — the raw token is shown once at mint time. + return { + 'id': self.id, + 'user_id': self.user_id, + 'username': self.user.username if self.user else None, + 'created_by_id': self.created_by_id, + 'created_by': self.created_by.username if self.created_by else None, + 'expires_at': self.expires_at.isoformat() if self.expires_at else None, + 'used_at': self.used_at.isoformat() if self.used_at else None, + 'bound_ip': self.bound_ip, + 'created_at': self.created_at.isoformat() if self.created_at else None, + } + + def __repr__(self): + return f'' diff --git a/backend/app/models/managed_database_user.py b/backend/app/models/managed_database_user.py new file mode 100644 index 00000000..7b697829 --- /dev/null +++ b/backend/app/models/managed_database_user.py @@ -0,0 +1,77 @@ +import json +from datetime import datetime + +from app import db + + +class ManagedDatabaseUser(db.Model): + """A database user/grant ServerKit created on a managed database. + + Live engines forget nothing, but ServerKit used to: users created through + the panel left no durable trace, so restarts lost the mapping between a + managed database and the credentials provisioned for it. This row is that + trace — it powers the users panel and scopes one-click admin SSO. + + ``is_shadow`` marks short-lived, single-use credentials (e.g. the Adminer + SSO login) that a reaper drops once ``expires_at`` passes. Passwords are + NEVER stored here — they cross the API exactly once at creation time. + """ + + __tablename__ = 'managed_database_users' + + id = db.Column(db.Integer, primary_key=True) + managed_database_id = db.Column( + db.Integer, db.ForeignKey('managed_databases.id', ondelete='CASCADE'), + nullable=False, index=True, + ) + username = db.Column(db.String(120), nullable=False) + # JSON list of grant keywords, e.g. ["ALL"] or ["SELECT", "INSERT"]. + grants = db.Column(db.Text, nullable=False, default='["ALL"]') + is_shadow = db.Column(db.Boolean, nullable=False, default=False) + expires_at = db.Column(db.DateTime, nullable=True) + created_at = db.Column(db.DateTime, default=datetime.utcnow) + updated_at = db.Column(db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow) + + # Deleting the managed database removes its user rows (ORM cascade); the + # engine-side users are dropped explicitly by the service, never implicitly. + managed_database = db.relationship( + 'ManagedDatabase', + backref=db.backref('users', cascade='all, delete-orphan'), + ) + + __table_args__ = ( + db.UniqueConstraint('managed_database_id', 'username', + name='uq_managed_db_user'), + ) + + def get_grants(self): + try: + grants = json.loads(self.grants or '[]') + return grants if isinstance(grants, list) else [] + except (ValueError, TypeError): + return [] + + def set_grants(self, grants): + self.grants = json.dumps(list(grants or [])) + + @property + def is_expired(self): + return bool(self.expires_at and self.expires_at < datetime.utcnow()) + + def to_dict(self, live=None): + data = { + 'id': self.id, + 'managed_database_id': self.managed_database_id, + 'username': self.username, + 'grants': self.get_grants(), + 'is_shadow': self.is_shadow, + 'expires_at': self.expires_at.isoformat() if self.expires_at else None, + 'created_at': self.created_at.isoformat() if self.created_at else None, + 'updated_at': self.updated_at.isoformat() if self.updated_at else None, + } + if live is not None: + data['present'] = live.get('present') + return data + + def __repr__(self): + return f'' diff --git a/backend/app/models/site_bandwidth.py b/backend/app/models/site_bandwidth.py new file mode 100644 index 00000000..9347df07 --- /dev/null +++ b/backend/app/models/site_bandwidth.py @@ -0,0 +1,43 @@ +"""Per-domain daily bandwidth rollups. + +One row per (domain, day) aggregated from the nginx access logs by +``BandwidthService.aggregate`` (job kind ``bandwidth.aggregate``). ``app_id`` +is a best-effort attribution via the Domain/Application tables and is +nullable so traffic to domains the panel no longer manages is still recorded. +""" +from datetime import datetime + +from app import db + + +class SiteBandwidthDaily(db.Model): + __tablename__ = 'site_bandwidth_daily' + __table_args__ = ( + db.UniqueConstraint('domain', 'day', name='uq_site_bandwidth_domain_day'), + ) + + id = db.Column(db.Integer, primary_key=True) + app_id = db.Column( + db.Integer, + db.ForeignKey('applications.id', ondelete='CASCADE'), + nullable=True, + ) + domain = db.Column(db.String(255), nullable=False, index=True) + day = db.Column(db.Date, nullable=False, index=True) + bytes_sent = db.Column(db.BigInteger, nullable=False, default=0) + requests = db.Column(db.Integer, nullable=False, default=0) + created_at = db.Column(db.DateTime, default=datetime.utcnow) + + def to_dict(self): + return { + 'id': self.id, + 'app_id': self.app_id, + 'domain': self.domain, + 'day': self.day.isoformat() if self.day else None, + 'bytes_sent': int(self.bytes_sent or 0), + 'requests': int(self.requests or 0), + 'created_at': self.created_at.isoformat() if self.created_at else None, + } + + def __repr__(self): + return f'' diff --git a/backend/app/models/site_import.py b/backend/app/models/site_import.py new file mode 100644 index 00000000..d245959a --- /dev/null +++ b/backend/app/models/site_import.py @@ -0,0 +1,112 @@ +"""Site import runs — migrating a site from another control panel. + +A ``SiteImport`` row tracks one archive-to-app migration: where the archive +came from, what the analyse step discovered, the step-by-step run log, and +the final result. The heavy lifting happens in +``app.services.site_import_service`` as ``import.analyze`` / ``import.run`` +jobs; this row is the durable record the wizard UI polls. +""" +import json +from datetime import datetime + +from app import db + +# Formats the importer registry may grow to support. 'cpanel' ships first; +# the others plug into the same pipeline (see app/services/site_importers/). +VALID_SOURCE_TYPES = ('auto', 'cpanel', 'directadmin', 'hestia', 'wordpress_ssh') + +VALID_STATUSES = ( + 'created', 'uploading', 'analyzing', 'analyzed', + 'running', 'completed', 'failed', +) + + +class SiteImport(db.Model): + __tablename__ = 'site_imports' + + id = db.Column(db.Integer, primary_key=True) + source_type = db.Column(db.String(30), nullable=False, default='cpanel') + status = db.Column(db.String(20), nullable=False, default='created') + + # JSON: {'upload_path': ''} or {'url': 'https://...'} + source = db.Column(db.Text, nullable=True) + # JSON: caller options (app name override, which steps to skip, ...) + options = db.Column(db.Text, nullable=True) + # JSON: the analyse report produced by the format importer. + analysis = db.Column(db.Text, nullable=True) + # JSON: run outcome ({'app_id': ..., 'databases': [...], ...}) + result = db.Column(db.Text, nullable=True) + + # Appended, newline-separated human-readable log lines. + log_text = db.Column(db.Text, nullable=True) + current_step = db.Column(db.String(60), nullable=True) + error = db.Column(db.Text, nullable=True) + + created_by = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True) + created_at = db.Column(db.DateTime, default=datetime.utcnow) + updated_at = db.Column(db.DateTime, default=datetime.utcnow, + onupdate=datetime.utcnow) + + # ── JSON helpers ── + @staticmethod + def _loads(raw, default): + try: + value = json.loads(raw) if raw else default + except (ValueError, TypeError): + return default + return value if isinstance(value, type(default)) else default + + def get_source(self): + return self._loads(self.source, {}) + + def set_source(self, value): + self.source = json.dumps(value or {}) + + def get_options(self): + return self._loads(self.options, {}) + + def set_options(self, value): + self.options = json.dumps(value or {}) + + def get_analysis(self): + return self._loads(self.analysis, {}) + + def set_analysis(self, value): + self.analysis = json.dumps(value or {}) + + def get_result(self): + return self._loads(self.result, {}) + + def set_result(self, value): + self.result = json.dumps(value or {}) + + def append_log(self, line): + stamp = datetime.utcnow().strftime('%H:%M:%S') + entry = f'[{stamp}] {line}' + self.log_text = (self.log_text + '\n' + entry) if self.log_text else entry + + def log_tail(self, max_lines=500): + if not self.log_text: + return '' + lines = self.log_text.splitlines() + return '\n'.join(lines[-max_lines:]) + + def to_dict(self, log_lines=500): + return { + 'id': self.id, + 'source_type': self.source_type, + 'status': self.status, + 'source': self.get_source(), + 'options': self.get_options(), + 'analysis': self.get_analysis() or None, + 'result': self.get_result() or None, + 'log_text': self.log_tail(log_lines), + 'current_step': self.current_step, + 'error': self.error, + 'created_by': self.created_by, + 'created_at': self.created_at.isoformat() if self.created_at else None, + 'updated_at': self.updated_at.isoformat() if self.updated_at else None, + } + + def __repr__(self): + return f'' diff --git a/backend/app/notifications/catalog.py b/backend/app/notifications/catalog.py index c502a29f..c0827721 100644 --- a/backend/app/notifications/catalog.py +++ b/backend/app/notifications/catalog.py @@ -100,6 +100,14 @@ 'severity': 'warning', 'category': 'system', }, + # Daily drift sweep found managed config files that no longer match what + # the panel would write (see app/services/drift_service.py). + 'drift.detected': { + 'title': 'Configuration drift detected ({count})', + 'template': 'generic', + 'severity': 'warning', + 'category': 'system', + }, # Multi-alert monitoring digest (used by the legacy send_all path). 'monitoring.alert': { 'title': 'ServerKit alert', diff --git a/backend/app/services/bandwidth_service.py b/backend/app/services/bandwidth_service.py new file mode 100644 index 00000000..59f9e44c --- /dev/null +++ b/backend/app/services/bandwidth_service.py @@ -0,0 +1,337 @@ +"""Per-domain bandwidth accounting from nginx access logs. + +A daily job (kind ``bandwidth.aggregate``) streams each site's nginx access +log — the ``/var/log/nginx/{name}.access.log`` files the vhost templates +write (see ``NginxService.site_access_log_path``) — plus the server-wide +``access.log`` catch-all, sums ``$body_bytes_sent`` and request counts per +domain for one calendar day, and upserts :class:`SiteBandwidthDaily` rows. + +Log format handling: + * standard "combined" lines are attributed to the owning app's primary + domain (per-site logs don't record ``$host``); + * vhost-prefixed lines (``$host[:$port] `` before the combined fields, as + produced by ``vcombined``-style formats) are split by that host — the + only way lines in the shared default ``access.log`` can be attributed. + +Re-running a day replaces that day's rows (incremental-safe). On dev boxes +with no nginx logs (e.g. Windows) the job is a clean no-op. +""" +import logging +import os +import re +from datetime import date, datetime, timedelta + +from app import db + +logger = logging.getLogger(__name__) + +BANDWIDTH_JOB_KIND = 'bandwidth.aggregate' +BANDWIDTH_SCHEDULE_NAME = 'bandwidth-aggregate' + +# Keep a little over a year of daily rows. +RETENTION_DAYS = 400 + +# nginx "combined" format: +# $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent ... +_COMBINED_RE = re.compile( + r'^(?P\S+) \S+ \S+ \[(?P