feat(examples): add custom benchmark examples for BEAM#821
Draft
egordm wants to merge 7 commits intorelease/v4.0.0from
Draft
feat(examples): add custom benchmark examples for BEAM#821egordm wants to merge 7 commits intorelease/v4.0.0from
egordm wants to merge 7 commits intorelease/v4.0.0from
Conversation
…mments - example_baseline.py: minimal forecaster (constant median) implementing BacktestForecasterMixin - example_benchmark.py: custom target provider extending SimpleTargetProvider - run_liander2024_benchmark.py: run example baseline + GBLinear on Liander 2024 - run_benchmark.py: run example baseline + GBLinear using custom benchmark pipeline - README.md: beginner-friendly guide with setup and usage instructions
Change DnTHH:MM format to DnTHHMM (e.g. D-1T0600 instead of D-1T06:00). Colons are illegal in Windows file paths, breaking benchmark output directories. from_string() now accepts both formats for backward compatibility.
Allows users to inject their own forecast predictions and run only evaluation + analysis, skipping the backtesting step entirely. Includes format_predictions() helper and a stub forecaster factory.
…erConfig - Correct import paths for BacktestForecasterMixin and RestrictedHorizonVersionedTimeSeries - Add missing required fields to BacktestForecasterConfig in stub - Verified: pipeline correctly skips backtesting and runs eval + analysis
… data format - Rename evaluate_forecasts.py → evaluate_existing_forecasts.py - Remove format_predictions() helper and save loop (step 1) - Just point LocalBenchmarkStorage at existing parquets - Use DummyForecaster instead of custom _QuantileStub - Add README section: directory layout, parquet format table, example rows - Add LeadTime to evaluation config in example_benchmark.py
Add compare_liander2024_results.py and compare_custom_results.py that generate side-by-side comparison plots across benchmark runs using BenchmarkComparisonPipeline. Update README with comparison section.
…le-custom-baseline-benchmark
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end examples showing how to create and run custom BEAM benchmarks, plus a Windows compatibility fix.
Custom Benchmark Examples
example_baseline.pyBacktestForecasterMixin— predicts a constant medianexample_benchmark.pySimpleTargetProviderwith metrics and pipeline assemblyrun_liander2024_benchmark.pyrun_benchmark.pyREADME.mdAll files include inline comments explaining concepts for newcomers (quantiles,
RestrictedHorizonVersionedTimeSeries, config parameters, metrics, etc.).Windows Compatibility Fix
AvailableAt.__str__()now producesD-1T0600instead ofD-1T06:00. Colons are illegal in Windows file/directory names, which broke benchmark output paths likebenchmark_results/analysis/mv_feeder/OS Edam/D-1T06:00/.from_string()accepts bothD-1T06:00(legacy) andD-1T0600(new) for backward compatibility