This repository holds various data and various tools used by Anserini and Pyserini.
Build the included evaluation tools as follows (you might get warnings, but you can ignore):
cd eval && tar xvfz trec_eval.9.0.4.tar.gz && cd trec_eval.9.0.4 && make && cd ../..
cd eval && cd ndeval && make && cd ../..Topics for various evaluations are stored in topics/.
Metadata is stored in topics.json and aliases in topics-aliases.json.
In the aliases file, the keys represent canonical topics.
Qrels for various evaluations are stored in qrels/.
Metadata is stored in qrels.json and aliases in qrels-aliases.json.
In the aliases file, the keys represent canonical topics.
❗ At commit 43add83 (2026/08/09), topics-and-qrels/ was refactored into separate topics/ and qrels/ directories.
At the same time, this repo was renamed from anserini-tools to eval.
The associated PR is eval#118.
This breaks consumers that depend on fetching a stable topics-and-qrels/ path (on master).
Note that the most obvious solution to add symlinks won't work, as raw.githubusercontent.com URLs do not automatically redirect.
Unfortunately, there's no good solution... according to Codex, we either have to fix all downstream consumers or have separate copies of the data.
Anserini commit 9bfc04b (2026/08/11) was the first commit that pinned a specific commit (hence ensuring stability).
The associated PR is anserini#3369.
This means that any state of the repo before that commit is likely broken.