RAG

Retrieval-Augmented Generation

less than 1 minute read

TREC 2024 RAG - AG input, baseline runs, validator scripts, and submissions open

Participants can now begin submitting their generated answers. To assist with this, we’ve also provided some strong baseline models (retrieval -> rerank and retrieval -> rerank -> generate), including the latest from Cohere’s Command R Plus model, which has shown promising results with a modified prompt. More instructions of these can be found here.

For those working with AG (Answer Generation) and RAG (Retrieval-Augmented Generation) tasks, a validator script has been included in the Ragnarök repository. This script will help verify that your generated answers meet the submission criteria. Make sure to carefully review the warning and error logs generated during the validation process to avoid common pitfalls such as sentence drops, duplicate citations, etc.

Submissions are now open in evalbase, and we encourage participants to start submitting soon as the deadline is August 25th AOE. Detailed instructions for generating and validating submissions can be found in Ragnarök.

May the force be with you, The TREC 2024 RAG Organizers

You may also enjoy

TREC 2024 RAG UMBRELA Qrels Released

less than 1 minute read

We have just released relevance judgments using UMBRELA [1] for the 301 RAG test topics which we used for our evaluations [2]. You can find them at: RAG 2...