[GPT-OSS-120B] Reference implementation #2395
Conversation
v-shobhit
force-pushed
the
gptoss-loadgen
branch
from
38f62ab to
02206f7
Compare
loadgen/mlperf.conf
Outdated
| *.Server.target_qps = 1.0 | ||
| *.Offline.target_qps = 1.0 | ||
|
|
||
| # gpt-oss placeholder values |
There was a problem hiding this comment.
Missing some config (see the file above to add the full list)
and interactive
@pgmpablo157321 @tanvi-mlcommons can we use gpt-oss-120b.Interactive now?
There was a problem hiding this comment.
I will move the gpt-oss fields to language/gpt-oss-120b/mlperf/user.conf and leave this file unchanged.
Let's continue the discussion on !2414.
|
@mrmhodak this is ready for your review. |
language/gpt-oss-120b/README.md
Outdated
| Number of parallel workers for LiveCodeBench evaluation (default: 64) | ||
| --verbose Verbose logging | ||
|
|
||
| ``` No newline at end of file |
There was a problem hiding this comment.
One last thing to put the reference score (SGLang) and reference token number of the performance dataset, and #rows in performance and accuracy dataset here (I will upstream to submission checker)
There was a problem hiding this comment.
(Not a blocker, we can do in next PR)
|
|
||
| logger.info(f"Results saved to: {output_path}") | ||
|
|
||
| logger.info("Evaluation complete!") |
There was a problem hiding this comment.
One comment (can move to next PR): we would need to print the final score to follow this regex:
"exact_match": r".*'exact_match':\s([\d.]+).*"
(as in submission checker)
add better parsing and check for harmony tokens
7 participants
Adding gpt-oss-120b reference implementation.
This uses SGLang backend to serve gpt-oss-120b model. Scripts are provided to run this in Offline/Server scenario in PerformanceOnly/AccuracyOnly modes.