DS@GT ARC

FAQ

1. What is CLEF?

CLEF (Conference and Labs of the Evaluation Forum) is an annual independent peer-reviewed conference that focuses on advancing research in information access systems, particularly in multilingual and multimodal contexts.

CLEF aims to maintain and expand upon its tradition of community-based evaluation and discussion on evaluation issues in information retrieval. It provides an infrastructure for testing, tuning, and evaluating information access systems, creating reusable test collections, and exploring new evaluation methodologies.

Overall, CLEF strives to advance the state-of-the-art in information access technologies through its combination of academic conferences and practical evaluation labs.

2. What lab/task should I pursue?

Each lab and task addresses a problem in a particular domain. Are you more interested in natural language processing? Computer vision? Biodiversity conservation? Medical applications? Multi-modality? You may find a summary of each lab in the last question of this FAQ. Review the task overview papers on the CLEF working notes page: CEUR-WS.org/Vol-3740 - Working Notes of CLEF 2024.

3. How many tasks can I participate in?

There is no limit to tasks that someone can participate in. We strongly recommend first-time members not doing more than one task.

4. What’s the difference between a team lead and a team member?

A Lab Lead is the main person responsible for delivering the task, including:

A Lab member is responsible for the following:

Team leads and members are expected to commit time, have strong programming skills, and be legitimately curious about the lab and task.

5. How do I become a team lead for a task?

Apply in the form DS@GT CLEF 2026 Competition Signup (Form TBD) and specify your desired role (team lead or team member). If you apply to be a team lead, reach out to Anthony Miyaguchi (acmiyaguchi at gatech.edu) or Murilo Gustineli (murilogustineli at gatech.edu) with an overview of your proposed solution for the particular task.

6. What is the time commitment required to participate?

The time commitment varies depending on your role and the effort you want to put in. However, to make a meaningful contribution, you should expect to dedicate around 100–150 hours throughout the project. Think of it as the equivalent of a 2–3 unit course, requiring consistent effort. Team leads require additional time to manage their tasks and coordinate with team members. This is the type of experience where you get out what you put in. Ultimately, your level of involvement is up to you, but consistent effort is key to gaining valuable experience and making an impact.

7. Can two teams participate in the same task?

No. A person can be part of one or more teams. But a team can only do one task.

8. Why can’t I edit the meeting documents?

You must join the CLEF 2026 Google Group (TBD) to be able to edit the meeting documents.
This functionality will be granted to you after joining.

9. Is this opportunity only available for current students, or can alumni participate as well?

This opportunity is not limited to current students—GT alumni are also welcome to join our group! However, participants must be members of the Data Science @ Georgia Tech (DS@GT) club and have paid their membership dues. To join:

10. How can I earn academic credit for participating in CLEF?

If you are an OMSCS student, there are two primary ways to earn academic credit through CLEF participation:

  1. CS 8903 - Specieal Problems
  2. CS 8803 O24 - Intro to Research

CS 8903 – Special Problems

This is a supervised research course that requires special permission to enroll. To take this course, you need to:

CS 8803 O24 – Intro to Research

This course offers a general introduction to research methods and computer science research. Unlike CS 8903, you can register for this course as part of your regular course selection in the OMSCS program.

11. What are the labs available at CLEF 2025?

Below is a short overview of the labs under the CLEF 2024 conference. You may find more information on each lab by reviewing their respective overview papers on the CLEF working notes page: CEUR-WS.org/Vol-3740 - Working Notes of CLEF 2024

BioASQ

The aim of the BioASQ Lab is to push the research frontier towards systems that use the diverse and voluminous information available online to respond directly to the information needs of biomedical scientists.

CheckThat!

The main objective of the lab is to advance the development of innovative technologies combating disinformation and manipulation efforts in online communication across a multitude of languages and platforms. The 9th edition of the CheckThat! lab at CLEF targets three tasks: (i) source retrieval for scientific web claims, (ii) fact-checking numerical claims, and (iii) generating full fact-checking articles.

ELOQUENT

The ELOQUENT lab for evaluation of generative language model quality and usefulness addresses high-level quality criteria for generative language models through a set of open-ended shared tasks.

eRisk

eRisk explores the evaluation methodology, effectiveness metrics and practical applications (particularly those related to health and safety) of early risk detection on the Internet. Early detection technologies can be employed in different areas, particularly those related to health and safety. Our main goal is to pioneer a new interdisciplinary research area that would be potentially applicable to a wide variety of situations and to many different personal profiles.

EXIST

This lab focuses on the detection of sexist messages in social networks (SN), particularly in complex multimedia formats such as memes and short videos. Inequality and discrimination against women that remains embedded in society is increasingly being replicated online. In this edition, we extend the Learning with Disagreement (LwD) framework by incorporating sensor-based data from people exposed to potentially sexist content. This includes measurements such as heart rate variability, EEG, eye-tracking, and other sensor data.

FinMMEval

FinMMEval introduces the first multilingual and multimodal evaluation framework for financial large language models, assessing models’ abilities in understanding, reasoning, and decision-making across languages and modalities to promote robust, transparent, and globally inclusive financial AI systems.

HIPE

Who was where when? Accurate and Efficient Person–Place Relation Extraction from Multilingual Historical Texts.

ImageCLEF

ImageCLEF is an ongoing evaluation event that started in 2003, promoting the evaluation of technologies for annotating, indexing, retrieving, and generating multimodal data, and aiming to provide access to large collections of data across a veriety of scenarios, domains and contexts.

JOKER Lab

The JOKER Lab aims to improve AI’s understanding and handling of humor and wordplay by creating reusable test collections and tasks for humor retrieval, translation, and generation across languages.

LifeCLEF

LifeCLEF is an international research initiative in the field of biodiversity informatics that organizes yearly challenges on the automated identification and understanding of life forms, particularly using machine learning and computer vision methods. It is part of CLEF (Conference and Labs of the Evaluation Forum), which organizes benchmark challenges to advance state-of-the-art techniques in information retrieval and data analysis.

LongEval

Many components of information retrieval systems evolve over time. The LongEval Lab aims to provide a benchmark setting to the longitudinal evaluation of IR models. At its fourth edition, LongEval we focus on scholarly search and scholarly user models.

PAN

PAN is a series of scientific events and shared tasks on digital text forensics and stylometry.

QuantumCLEF

QuantumCLEF provides an evaluation infrastructure for developing Quantum Computing algorithms, particularly Quantum Annealing algorithms, for Information Retrieval and Recommender Systems.

SimpleText Lab

The SimpleText Lab seeks to make scientific knowledge more accessible by developing and evaluating AI models that simplify complex scientific texts, reduce misinformation risks, and improve public understanding through tasks on text simplification, hallucination control, and adaptive scientific content analysis.

TalentCLEF

TalentCLEF is an evaluation lab that aims to establish a public benchmark and collaborative forum for assessing NLP models in Human Capital Management, promoting fairness, multilingual capability, and cross-industry adaptability to advance skill and job intelligence technologies.

Touché

Touché focuses on developing technologies that support people in decision-making and opinion-forming, aiming to improve our understanding of these processes.

These labs collectively contribute to the CLEF tradition of community-based evaluation and discussion on various aspects of information access systems.