Skip to content

Releases: sgl-project/genai-bench

v0.0.3

10 Dec 01:46
@CatherineSue CatherineSue
v0.0.3
d0780ee
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v0.0.3 Latest
Latest

What's Changed

  • [Docs] Update docs for scenario and fix markdown lint errors by @CatherineSue in #69
  • [Docs] Add traffic-scenario to github-pages by @CatherineSue in #70
  • [FIX]Make storage_bucket is_eager by @YouNeedCryDear in #73
  • [Misc] Remove tolerance in _check_discrepancy by @CatherineSue in #77
  • [Report] Add option for milliseconds by @TJ5 in #74
  • [Docs] Remove local.txt from repository by @Juno13340 in #80
  • [Docs] Add PR template and remove duplicate PR section from contributing guide by @Juno13340 in #79
  • [CI] add GHCR publish to release by @Juno13340 in #76
  • [Feature] add count-based warmup/cooldown ratio by @dbdydgur2244 in #82
  • [CI] Fix CI workflow failure by @CatherineSue in #84
  • [CI] Remove --no-reinstall as it is not supported in uv by @CatherineSue in #85
  • [UI] Fix Minimal Dashboard by @TJ5 in #89
  • [Metrics] Correct TTFT calculation for reasoning models by @Juno13340 in #87
  • [Core] Extend timeout of oci genai client and improve error handling in distributed workers by @key4ng in #88
  • [Frontend] Rename --time-unit to --metrics-time-unit by @TJ5 in #94
  • [Misc] Update PR template by @CatherineSue in #97
  • [Misc] Update PR template label sections by @CatherineSue in #98
  • [Feature] Add LlamaCPP as server engine option by @rgerganov in #93
  • [Bugfix] Fix OpenAI authentication header formatting by @WillemMomma in #99
  • [MISC] Support PR auto labeler by @key4ng in #106
  • [misc] Add issue template by @key4ng in #107
  • improve naming and format in ci by @key4ng in #111
  • [Docs] Update documentation to be more readable by @TJ5 in #96
  • [Docs] Fix github pages yml by @TJ5 in #117
  • Downsize labels to 10+ to 6 by @CatherineSue in #118
  • Add --log-dir option and improve ENABLE_UI support by @fuhengwu2021 in #115
  • [misc] Update .gitignore and remove local files after tests by @CatherineSue in #119
  • fix(openai): strip backend-unsupported params (e.g. ignore_eos) before send by @OpheliaLjh in #116
  • Revert "fix(openai): strip backend-unsupported params (e.g. ignore_eos) before send" by @CatherineSue in #123
  • [Docs] Small updates to Development Guide Section by @TJ5 in #121
  • [feature] Add TogetherAI backend support by @CatherineSue in #129
  • Fix/OpenAI ignore eos, improved code performance by @OpheliaLjh in #124
  • [Bug] Filter TPOT/inference speed by output latency by @ankrovv in #122
  • [MISC] Improve token counting with usage data in OCI GenAI user by @paxiaatucsdedu in #126
  • [release] Update version to 0.0.3 by @CatherineSue in #133

New Contributors

Full Changelog: v0.0.2...v0.0.3

What's Changed

New Contributors

Read more

Contributors

rgerganov, YouNeedCryDear, and 10 other contributors
Assets 2
Loading

v0.0.2

12 Aug 00:50
@CatherineSue CatherineSue
v0.0.2
9cfff98

Choose a tag to compare

View all tags
v0.0.2

What's Changed

  • Update docker image to docs by @CatherineSue in #10
  • Doc: Set up coversage badge by @CatherineSue in #11
  • feat: Support flexible plot configuration by @CatherineSue in #13
  • Doc: Update logo and badge alignment by @CatherineSue in #14
  • refactor: Update image-to-text to image-text-to-text by @CatherineSue in #12
  • fix: Resolve all mypy type errors by @CatherineSue in #17
  • [Documentation] Setup MkDocs for genai-bench by @key4ng in #19
  • [BUG] Fix left tokens to sample in _sample_text by @YouNeedCryDear in #20
  • [core] feature: add multi cloud support by @slin1237 in #15
  • [docs] add multi cloud docs by @slin1237 in #22
  • [docs] add mkdocs material theme by @slin1237 in #24
  • [Docs] Update broken links in README.md by @junliu-mde in #23
  • [bugfix] fix mkdocs domain name by @slin1237 in #25
  • [bugfix] mkdocs gh pages config inheritance issue by @slin1237 in #26
  • [ci] Fix GitHub Pages deployment workflow by @slin1237 in #27
  • [docs] update readme hyper links by @slin1237 in #28
  • Revert "[BUG] Fix left tokens to sample in _sample_text (#20)" by @slin1237 in #29
  • Move config example doc under usage folder by @key4ng in #21
  • fix(cli): fix typo in the help message of --traffic-scenario by @rudeigerc in #34
  • [Frontend] Make HF_TOKEN optional for tokenizer download by @junliu-mde in #33
  • [BUG] Fix wrong default threshold for check discrepancy by @YouNeedCryDear in #38
  • docs: remove non-existent entry of plot examples in mkdocs by @rudeigerc in #42
  • [Misc] Fix output_throughput metrics for single request by @CatherineSue in #40
  • docs: add description of output_latency metric in the docs by @rudeigerc in #41
  • [Bug Fix] OCIModelAuthAdapter.get_credentials() returning wrong object type by @key4ng in #45
  • [Core] Add oci genai backend support by @key4ng in #47
  • [Bugfix] Fix token-based truncation and document duplication in TextSampler by @CatherineSue in #49
  • feat: Add flexible image dataset support and make scenarios optional by @CatherineSue in #51
  • bugfix: Fix typing in ImageDatasetLoader and num_images check by @CatherineSue in #53
  • [bug] fix missing ttft record by @yiakwy-xpu-ml-framework-team in #44
  • misc: Support py3.10 by @CatherineSue in #54
  • Revert "[Misc] Fix output_throughput metrics for single request (#40)" by @CatherineSue in #55
  • [Bugfix] Allow for benchmarking using large images. by @shenoyvvarun in #52
  • bugfix: Fix TextSampler sampling logic by @CatherineSue in #58
  • Add spawn rate to prevent worker overload in distributed benchmarks by @JimmyWhitaker in #56
  • bugfix: Bring back HF dataset logging and scenario for images by @CatherineSue in #61
  • [Bugfix] Use image-text-to-text instead of image-to-text by @TJ5 in #59
  • [Misc] Improve Excel report handling when performance threshold is not met by @key4ng in #60
  • feat: remove use_scenario, add DatasetScenario, make traffic scenario optional by @CatherineSue in #62
  • [Docs] Fix errors in docs by @TJ5 in #57
  • feat(data): normalize HF dataset types and enable lazy image sampling by @CatherineSue in #63
  • [Plot] Rename plot axis label and xlimt start from 0 and update docs by @upfixer in #39
  • Revert "[Plot] Rename plot axis label and xlimt start from 0 and update docs" by @CatherineSue in #64
  • misc: Fix doc naming and use flexible plot in cli by @key4ng in #65
  • Remove hidden cap for data and set general scale start from 0 by @CatherineSue in #66
  • [Plot] Revert grid alpha and add TODOs by @CatherineSue in #67
  • [CI] Fix release.yml by @CatherineSue in #68

New Contributors

Full Changelog: v0.0.1...v0.0.2

Contributors

shenoyvvarun, YouNeedCryDear, and 9 other contributors
Loading

v0.0.1

26 Jun 21:33
@CatherineSue CatherineSue
v0.0.1
a3060d2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v0.0.1

🎉 Initial Release

This is the first public release of genai-bench - a comprehensive benchmarking tool for generative AI models and inference servers.

✨ Features

  • Support for text-to-text, image-to-text, and embedding benchmarks
  • Flexible dataset configuration with HuggingFace integration
  • Multiple API backends (OpenAI, OCI Cohere, custom)
  • Comprehensive metrics and reporting
  • Docker support for containerized benchmarking
  • Traffic scenario modeling with concurrency controls

🚀 Getting Started

pip install genai-bench
genai-bench --help
genai-bench benchmark --help

See the USER_GUIDE.md for detailed usage instructions.

📊 What's Included

  • CLI tool with comprehensive options
  • Pre-configured traffic scenarios
  • Excel and plot generation for results analysis
  • Support for distributed benchmarking

🛠️ Requirements

  • Python 3.11+
  • See pyproject.toml for full dependencies
Loading