sparkparse

identify spark bottlenecks without breaking your neck

what it does

sparkparse parses Apache Spark event logs and provides:

• an interactive Dash dashboard for exploring query plans, stage timelines, and task metrics
• a CLI for parsing logs into structured Polars DataFrames (CSV, Parquet, Delta, JSON output)
• a context manager / decorator for capturing logs from an active SparkSession
• an analyze command that emits a raw plan summary (numeric metrics with units) plus diagnostic findings, each with its own evidence, thresholds and coverage status

install

# base install: parsing, analysis, history and the CLI
pip install sparkparse

# optional extras
pip install "sparkparse[viz]"    # Dash dashboard
pip install "sparkparse[spark]"  # PySpark capture
pip install "sparkparse[cloud]"  # s3 / azure / gcs storage backends

usage

CLI

The log directory is a positional argument. --log-dir is accepted as an alias for the commands documented with it.

# list the event-log sources found in a directory
sparkparse logs ./logs

# parse the newest log and write output files
sparkparse get ./logs --out-format parquet

# parse one application explicitly (file name, rolling-log dir, or app id)
sparkparse get ./logs --log-file eventlog_v2_app-20260912-0001

# parse every application in the directory
sparkparse get ./logs --all-apps

# launch the dashboard
sparkparse viz ./logs

# produce LLM-friendly analysis JSON
sparkparse analyze ./logs

# compact export for long plans, with paths and expressions redacted
sparkparse analyze ./logs --compact --top-n 25 --redact

Databricks runs and experiments

Collect an existing job/run (it never launches or waits for a workload), save a portable snapshot, and compare explicit trials offline. See docs/databricks-runs.md for the tested collection contract and measure semantics.

# report only (JSON to stdout, progress to stderr)
sparkparse databricks analyze --run-id 123 --profile free

# save a snapshot, or record a trial in an explicit experiment
sparkparse databricks analyze --run-id 123 --profile free --out runs/123
sparkparse databricks analyze --run-id 124 --profile free \
  --experiment ./experiments/join-tuning --variant broadcast --revision abc123

# compare and view saved trials entirely offline
sparkparse experiments compare ./experiments/join-tuning --baseline 123 --candidate 124
sparkparse experiments viz ./experiments/join-tuning

Workflow elapsed time is kept separate from summed task durations, physical plan changes never exclude a comparison, and a Query History subtotal over only the queries that reported a metric is labelled an observed subtotal. Failed, active and warmup trials stay visible but are excluded from default aggregates.

context manager

import sparkparse

with sparkparse.capture_context(spark=spark, action="get") as cap:
    df.groupBy("id").count().show()

parsed = cap.dfs  # ParsedLogDataFrames with .dag and .combined
result = cap.result  # dataframes, metadata, capabilities, and diagnostics

Capture borrows a supplied Spark session and never stops or recreates it. Borrowed classic sessions must have event logging enabled before the workload starts. When sparkparse should own a local session, opt in explicitly with capture_context(own_session=True). Connect captures expose the same result contract, with unavailable task/stage telemetry called out in result.capabilities rather than represented as zeroes.

event logs

Rolled logs (spark.eventLog.rolling.enabled=true) are read as one logical source with ordered segments; .inprogress logs are read as far as they go and reported as incomplete. Segments are streamed line by line, never copied into memory whole. Zstd-compressed logs need sparkparse[zstd]; Spark's lz4, lzf and snappy codecs use Java-specific framing that Python cannot decode, and fail with an error naming the codec.

Neither SparkListenerApplicationStart nor adaptive execution is required. A non-AQE query keeps the plan from its SQLExecutionStart event, a query with no end event keeps a null duration, and a truncated final line is reported as truncation rather than corruption. strict=True turns those diagnostics into errors.

Use backend="classic" or backend="connect" to override detection. For post-run ingestion without a Spark session, use backend="event_log", log_file="/path/to/log". Borrowed captures select the current application's log; use an explicit log_file when its filename cannot be identified. They report ambiguous capture scope and may observe an incomplete log. Owned captures stop their session before parsing.

Capture failures raise by default; capture_errors="record" retains diagnostics and a partial result instead. This is independent of parser strict=True. Failed captures retain their temporary logs. User workload exceptions always take precedence. action="viz" produces an in-memory HTML report in cap.report, including coverage, on either backend; it does not launch a dashboard server. Capture-result JSON uses Arrow IPC tables to preserve schemas, including empty tables and nested metrics.

Spark Connect (serverless) capture

Connect capture intercepts the client's action boundaries — to_table (collect, count, show, take), to_pandas, to_table_as_iterator, and execute_command (SQL commands and writes). Each action becomes one query with its own operator metrics, logical plan, and server operation ID; actions routed through a client method that the installed version does not expose are reported as action_not_covered diagnostics rather than silently dropped. No extra action is ever executed to collect telemetry.

import sparkparse

sparkparse.probe_connect_support(spark)  # what this client version exposes

Semantics worth knowing:

• query_duration_seconds is client-observed elapsed time and includes result transfer, so result.capabilities.query_elapsed_time is at most partial. The server's cumulative operator time is kept per node (cumulTime, nanoseconds, summed over the operator subtree and tasks) and is never used as elapsed time.
• Join keys are attached only when the physical node's plan ID matches a logical join relation, when a query contains exactly one join, or when the operator name itself carries them. Otherwise keys are left unresolved; they are never matched by list position. Child nodes keep the order the server reported and joins record input_roles: unordered — build and probe sides are not asserted.
• Unrecognized operators are preserved as Unknown nodes with their raw names and graph edges intact, unless strict=True is set. Metric batches are treated as snapshots (last value per plan ID wins), never summed.
• One client supports one capture at a time; a nested or concurrent capture on the same client is rejected, and overlapping actions across threads are attributed by calling thread and flagged with a concurrent_executions diagnostic.
• Task and stage telemetry is unavailable on this source and is reported as such in result.capabilities instead of as zeroes.

Recorded executions can be replayed offline with SparkConnectCapture.from_plan_metrics(executions), where each entry carries the PlanMetrics.to_dict() payload.

decorator

import sparkparse

@sparkparse.capture(spark=spark, action="get")
def run_job(spark):
    spark.read.parquet("s3://bucket/data/").groupBy("id").count().show()

result, cap = run_job()

design goals

• simplified UI that highlights bottlenecks and their causes
• node drill-down for detailed information and metric distribution
• generation of base models and DataFrames for extensible analysis
• LLM-friendly analysis output for automated bottleneck detection

development

See the September 2026 review and implementation queue for prioritized standard/serverless parity and developer experience improvements. Databricks job/run-ID analysis and commit performance experiments are not yet supported; see the proposed run-report workflow. IMPLEMENTATION.md contains the earlier roadmap.

# install with dev dependencies
uv sync --dev

# lint + unit tests (excludes Spark-dependent integration tests)
just ci

# full tests including Spark integration tests
just ci-full

Dependencies are managed with uv and locked in uv.lock. To change a dependency, edit pyproject.toml, run uv lock, then uv sync --dev. CI installs the lockfile frozen (UV_FROZEN=1) so a stale lock fails instead of silently resolving.

capture artifacts

A capture can be written to a portable artifact directory that opens in the dashboard without the raw event logs:

with sparkparse.capture_context(spark=spark, action="viz", artifact_path="runs/job-a") as cap:
    df.groupBy("id").count().show()

sparkparse viz runs/job-a   # opens the artifact directly

Artifacts hold manifest.json (metadata, capabilities, diagnostics) plus dag.arrow and combined.arrow. The dashboard reads either a raw-log directory, an artifact, or an in-memory result, and caches parsed frames on the server so callbacks do not re-read logs.

TODOs

• structured node details like project columns and scan sources
• task box plots on hover
• metric capture via context manager / decorator
• hotspot highlighting by metrics other than duration (spill, records, etc.)
• analyze command with LLM-friendly JSON output
• reading from cloud storage
• ruff + pyrefly CI