[DLRMv3] Reference Implementation #2410
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
@tanvi-mlcommons @pgmpablo157321 to take a look (can ignore the whole HSTU submodules) and help resolve the CLA issues for Linjian. |
|
Hi @mrmhodak , could you help review this PR? All comments of @nvzhihanj have been addressed |
| ``` | ||
| python accuracy.py --path path/to/mlperf_log_accuracy.json | ||
| ``` | ||
| We use normalized entropy (NE), accuracy, and AUC as the metrics to evaluate the model quality. The accuracy for the reference implementation evaluated on 34,996 requests across 10 inference timestamps are listed below: |
There was a problem hiding this comment.
Are we using all 3 scores for 99%, or just the AUC? (If it's just AUC please help point it out)
Also it seems like the loadgen/mlperf.conf is missing (for latency threshold and sample count)
There was a problem hiding this comment.
Yes we need all 3 scores to be 99% close, I will clarify in the README. I will also add latency threshold to mlperf.conf.
Regarding sample count, could you explain when will we use it? I'm a bit confused as I think performance_sample_count_override
Lines 6 to 30 in 8999c4d
My understanding is that submitters are allowed to change the sample count, and as long as the minimum duration requirement for the benchmark is met, this should be acceptable.
| DLRMv3SyntheticStreamingDataset, | ||
| { | ||
| "ratings_file_prefix": os.path.join( | ||
| new_path_prefix, "data/streaming-100b/sampled_data/" |
There was a problem hiding this comment.
suggest changing it to "sampled_data/", because the
"--dataset-path-prefix", is already pointing to the location where it contains sampled_data, so this data/streaming-100b/ is no longer a valid path without user self modifying
| "--scenario-name", default="Server", choices={"Server", "Offline"}, help="inference benchmark scenario" | ||
| ) | ||
| parser.add_argument( | ||
| "--batchsize", default=10, help="batch size used in the benchmark" |
There was a problem hiding this comment.
I did a test run of this ref implementation, it throws an error because there is no type enforcement in arg parser, python will parse all of them to string instead of actual datatype. Can you add them
There was a problem hiding this comment.
besides this, accuracy successfully reproduced. Thanks!
4 participants
This PR introduces the reference implementation for DLDRMv3.
Instructions for running the benchmark are provided in the README. Please note: The dataset setup is still pending. The README will be updated with download instructions once the dataset becomes available.