TREC 2025 RAG Baselines and validator script

TREC 2025 RAG - AG input, baseline runs, validator scripts

Participants can now begin preparing and validating their generated answers. To assist with this, we’ve also provided some strong baseline models (retrieval -> rerank and retrieval -> rerank -> generate), which has shown promising results with a modified prompt. Retrieval baselines for AG (Augmented Generation) tasks can be found here. More instructions of these can be found here.

For those working with AG (Augmented Generation) and RAG (Retrieval-Augmented Generation) tasks, a validator script has been included in the Ragnarök repository. This script will help verify that your generated answers meet the submission criteria. Make sure to carefully review the warning and error logs generated during the validation process to avoid common pitfalls such as sentence drops, duplicate citations, etc.

We encourage participants to start preparing soon as the deadline is August 17th AOE. Detailed instructions for generating and validating submissions can be found in Ragnarök.

May the force be with you, The TREC 2025 RAG Organizers