LongEval: Longitudinal Evaluation of Model Performance

The goal of this task is to ignite the development of Information Retrieval systems that can handle temporal data evolution. The retrieval systems evaluated in this task are expected to be persistent in their retrieval efficiency over time, as Web documents and Web queries evolve. To evaluate such features of systems, we rely on collections of documents and queries, corresponding to real data acquired from actual Web search engines.

Tasks:

  • Task 1 - WebRetrieval:

    This task uses evolving Web data to evaluate IR system longitudinally, namely, will assess wether the IR system performance is persistent over time.

  • Task 2 - SciRetrieval:

    Similar to Task 1, this task aims to examine how IR systems’ effectiveness changes over time, when the underlying document collection changes, where the documents are scientific publications.

Organizers

  • Florina Piroi - TU Wien & RSA, AT
  • Alaa El-Ebschihy - Research Studios Austria (RSA), AT
  • Philippe Mulhem - University Grenoble, FR
  • Philipp Schaer - TH-Köln, DE
  • Jüri Keller - TH-Köln, DE
  • Petra Galuscackova - University of Stavanger, NO
  • Lorraine Geuriot - University of Grenoble, FR
  • Matteo Cancellieri - The Open University
  • David Pride - The Open University
  • Tobias Fink - RSA, AT

Contact

  • florina.piroi@tuwien.ac.at
  • alaa.el-ebshihy@tuwien.ac.at
  • galuscakova@gmail.com
  • Philippe.Mulhem@imag.fr
  • jkeller@th-koeln.de
  • philipp.schaer@th-koeln.de
  • david.pride@open.ac.uk
  • matteo.cancellieri@open.ac.uk
  • lorraine.goeuriot@univ-grenoble-alpes.fr
  • tobias.fink@researchstudio.at