Call for Lab Proposals

Background

The CLEF Initiative is a self-organised body whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual information in different modalities - including text and multimedia - with various levels of structure. CLEF promotes research and development by providing an infrastructure for:

  1. Independent evaluation of information access systems
  2. Investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access
  3. Creation of reusable test collections for benchmarking
  4. Exploration of new evaluation methodologies and innovative ways of using experimental data
  5. Discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge

Scope of CLEF Labs

We invite submission of proposals for two types of labs:

  1. "Campaign-style" Evaluation Labs for specific information access problems (during the twelve months period preceding the conference), similar in nature to the traditional CLEF campaign "tracks". Topics covered by campaign-style labs can be inspired by any information access-related domain or task.
  2. Labs that follow a more classical "workshop" pattern, exploring evaluation methodology, metrics, processes, etc. in information access and closely related fields, such as natural language processing, machine translation, and human-computer interaction.

We highly recommend organisers new to the CLEF format of shared task evaluation campaigns to first consider organising a lab workshop to discuss the format of their proposed task, the problem space and practicalities of the shared task. The CLEF 2025 programme will reserve about half of the conference schedule for lab sessions.

During the conference, the lab organisers will present their overall results in overview presentations during the plenary scientific paper sessions to give non-participants insights into where the research frontiers are moving. Lab organisers are expected to organise separate sessions for their lab with ample time for general discussion and engagement with all participants - not just those presenting campaign results and papers. Organisers should plan time in their sessions for activities such as panels, demos, poster sessions, etc. as appropriate. CLEF is always interested in receiving and facilitating innovative lab proposals.

Potential task proposers unsure of the suitability of their task proposal or its format for inclusion at CLEF are encouraged to contact the CLEF 2025 Lab Organizing Committee Chairs to discuss its suitability or design at an early stage.

Proposal Submission

Lab proposals must provide sufficient information to judge the relevance, timeliness, scientific quality, benefits for the research community, and the competence of the proposers to coordinate the lab. Each lab proposal should identify one or more organisers as responsible for ensuring the timely execution of the lab. Proposals should be 3 to 4 pages long and should provide the following information:

  1. Title of the proposed lab.
  2. A brief description of the lab topic and goals, its relevance to CLEF and the significance for the field.
  3. A brief and clear statement on usage scenarios and domain to which the activity is intended to contribute, including the evaluation setup and metrics.
  4. Details on the lab organiser(s), including identifying the task chair(s) responsible for ensuring the running of the task. This should include details of any previous involvement in organising or participating in evaluation tasks at CLEF or similar campaigns.
  5. The planned format of the lab, i.e., campaign-style (“track”) or workshop.
  6. Is the lab a continuation of an activity from previous year(s) or a new activity?
    1. For activities continued from previous year(s): Statistics from previous years (number of participants/runs for each task), a clear statement on why another edition is needed, an explicit listing of the changes proposed, and a discussion of lessons to be learned or insights to be made.
    2. For new activities: A statement on why a new evaluation campaign is needed and how the community would benefit from the activity.
  7. Details of the expected target audience, i.e., who do you expect to participate in the task(s), and how do you propose to reach them.
  8. Brief details of tasks to be carried out in the lab. The proposal should clearly motivate the need for each of the proposed tasks and provide evidence of its capability of attracting enough participation. The dataset which will be adopted by the Lab needs to be described and motivated in the perspective of the goals of the Labs; also indications on how the dataset will be shared are useful. It is fine for a lab to have a single task, but labs often contain multiple closely related tasks, needing a strong motivation for more than 3 tasks, to avoid useless fragmentation.
  9. Expected length of the lab session at the conference: half-day, one day, two days. This should include high-level details of planned structure of the session, e.g. participant presentations, invited speaker(s), panels, etc., to justify the requested session length.
  10. Arrangements for the organisation of the lab campaign: who will be responsible for activities within the task; how will data be acquired or created, what tools or methods will be used, e.g., how will necessary queries be created or relevance assessment carried out; any other information which is relevant to the conduct of your lab.
  11. If the lab proposes to set up a steering committee to oversee and advise its activities, include names, addresses, and homepage links of people you propose to be involved.

Lab proposals must be submitted via EasyChair at the following address: 

https://easychair.org/conferences/?conf=clef2025

choosing the “CLEF 2025 Lab Proposals” track. 


Review Process

Each proposal submitted by 7 July 2024 will be reviewed by the CLEF 2025 Lab Organising Committee. The acceptance decision will be sent by email to the responsible organiser by 5 Aug 2024. The final length of the lab session at the conference will be determined based on the overall organisation of the conference and the number of participant submissions received by a lab.

Advertising Labs at CLEF 2024 and ECIR 2025

Organisers of accepted labs are expected to advertise their labs at both CLEF 2024 (September 9-12, 2024, Grenoble, France) and ECIR 2025 (April 6-10, Lucca, Italy). So, at least one lab representative should attend these events.

Advertising at CLEF 2024 will consist of displaying a poster describing the new lab, running a break-out session to discuss the lab with prospective participants, and advertising/announcing it during the closing session.

Advertising at ECIR 2025 will consist of submitting a lab description (abstract submission deadline October 16, 2024) to be included in ECIR 2025 proceedings and advertising the lab in a booster session during ECIR 2025.

Lab Proposals from Newcomers

If you have not organised a lab before, do not panic! The CLEF 2025 Lab Organising Committee Lab is willing to mentor you by offering help, guidance, and feedback on the writing of your draft lab proposal.

If you are a newcomer interested in receiving guidance, please send an e-mail with the following tag in the subject “[Mentorship CLEF 2025 Lab Proposals]” to: clef2025-lab-proposals@easychair.org

We also encourage newcomers to refer to Friedberg et al. (2015) for initial guidance on preparing their proposal:

Friedberg I, Wass MN, Mooney SD, Radivojac P. Ten simple rules for a community computational challenge. PLoS Comput Biol. 2015 Apr 23;11(4):e1004150.

Important Dates

  • 7 July 2024: Proposal registration (submitting a draft/abstract to Easychair)
  • 12 July 2024: Hard deadline to submit final proposal version to Easychair
  • 5 August 2024: Notification of lab acceptance
  • 9-12 Sep 2024: Advertising Accepted Labs at CLEF 2024, Grenoble, France
  • October 2024 (TBA by ECIR): Submission of short lab description for ECIR 2025
  • 6-10 April 2025: Advertising labs at ECIR 2025, Lucca, Italy
  • April-May: Lab evaluation cycle
  • May-June: Review process of participant papers
  • June 2025: Review of the condensed labs overviews
  • July 2025: CEUR-WS Working Notes Preview for Checking by Authors and Lab Organisers
  • 9-12 Sep 2025: Labs at CLEF 2025, Madrid, Spain

CLEF 2025 Lab Chairs

  • Paolo Rosso, Universitat Politècnica de València (UPV), Valencia, Spain
  • Damiano Spina, RMIT University, Melbourne, Australia

Programme Committee Members

  • Alberto Barrón-Cedeño, Università di Bologna, Italy
  • Martin Braschler, ZHAW Zurich University of Applied Sciences, Switzelrand
  • Luis Chiruzzo, Universidad de la República, Uruguay
  • Daryna Dementieva, Technical University of Munich, Germany
  • Liana Ermakova, Université de Bretagne Occidentale, France
  • Elisabetta Fersini, University of Milano-Bicocca, Italy
  • Marc Franco-Salvador, Symanto Research, Spain
  • Anastasia Giachanou, Utrecht University, The Netherlands
  • José Ángel González, Genaios, Spain
  • Salud María Jiménez-Zafra, Universidad de Jaén, Spain
  • Jaap Kamps, University of Amsterdam, The Netherlands
  • Evangelos Kanoulas, University of Amsterdam, The Netherlands
  • Jussi Karlgren, Silo AI, Finland
  • Johannes Kiesel, Bauhaus-Universität Weimar, Germany
  • Martin Krallinger, Barcelona Supercomputing Centre, Spain
  • David Losada, University of Santiago de Compostela, Spain
  • Maria Maistro, University of Copenhagen, Denmark
  • Alejandro Martín, Universidad Politécnica de Madrid, Spain
  • María-Teresa Martín-Valdivia, Universidad de Jaén, Spain
  • Arturo Montejo-Ráez, Universidad de Jaén, Spain
  • Manuel Montes-Y-Gómez, Instituto Nacional de Astrofísica, Óptica y Electrónica, Mexico
  • Roser Morante, UNED, Spain
  • Josiane Mothe, Université de Toulouse, France
  • Preslav Nakov, Mohamed bin Zayed University of Artificial Intelligence,
  • Joakim Nivre, Uppsala University and RISE, Sweden
  • Javier Parapar, University of A Coruña, Spain
  • Florina Piroi, TU Wien, Institue of Information Systems Engineering & ZFDM, Austria
  • Simone Paolo Ponzetto, University of Mannheim, Germany
  • Martin Potthast, University of Kassel, hessian.AI, and ScaDS.AI, Germany
  • Francisco Rangel, Symanto Research, Spain
  • Eric Sanjuan, Laboratoire Informatique d'Avignon- Université d'Avignon, France
  • Areg M. Sarvazyan, Symanto Research, Spain
  • Efstathios Stamatatos, University of the Aegean, Greece
  • Benno Stein, Bauhaus-Universität Weimar, Germany
  • Sara Tonelli, Fondazione Bruno Kessler, Italy
  • Theodora Tsikrika, Information Technologies Institute, CERTH, Greece
  • Rafael Valencia-Garcia, University of Murcia, Spain
  • David Vilares, University of A Coruña, Spain
  • Esaú Villatoro, Idiap Research Institute, Switzerland
  • Matti Wiegmann, Bauhaus Universität Weimar, Germany
  • Christa Womser-Hacker, University of Hildesheim, Germany
  • Eva Zangerle, University of Innsbruck, Austria
  • Arkaitz Zubiaga, Queen Mary University of London, UK


clef2025-lab-proposals@easychair.org