IMAGEVAL is all about assessing image processing technology needed for sorting, finding and describing still images contained in vast data bases. The assessment focuses on features relating to what collection holders expect in terms of how images may be used (as described by a panel of participants from the defence, industrial and cultural sectors).

CAMPAIGN AGENDA

The assessments are to be carried out in three stages:

  • Participants will receive a sample of images representing the types of images to be used during the test stage.
  • A run test using smaller databases. The aim of this run test is not to reveal the results, but to detect problems linked to organisation, databases and protocols. These problems are brought to general attention and discussed via the FORUM. This campaign was essential to the success of the real test. Participants were asked to send their results even if they though they were not totally representative of their potential.
  • The official campaign, whose results will be revealed as part of an official presentation during the end-of-campaign conference in December 2006.

GENERAL INFORMATION ABOUT THE run TESTs

IMPORTANT:The information below is taken from the reference document ImagEVAL_info_tasks_eng.pdf . Refer to the PDF file if in doubt.

Assessment bases sent on January 16 th 2006

  • Deadline for sending the results: 15 th March 2006
  • Files to be sent to TRIBVN by email
    • Subject: TaskXrunnameteamname (eg. Task4, run 2, CEA)
    • Mail : Jacques Klossa: jklossa@tribvn.com
  • For each task and sub-task, 5 runs maximum
  • 1 file per run, name=iev_t0X_run_q01_nom_bla
    • with X: task number (“ 11” and “ 12” for the sub-task 1 and 2 of task 1)
    • with the name: run name, for example cea04
  • 1 file on processing times and type of machine used + OS + anything you think useful. This file is named NbTask_RunName with:
    • NbTask: task number (“ 11” and “ 12” for the sub-task 1 and 2 of task 1)
    • Run name: name of run. Example: 4_cea03
INFORMATION ON TASK 1

There are two sub-tasks, so one answer sheet per run and per sub-task. A different metric will be used for the two sub-tasks: MAP for the first one, the average of the correct answer’s ranking for the second.

The first sub-task must be done in order to carry out the second (at least for the mock test, to be discussed for the official test).

  • Sub-task 1:

    • Number of requests = 12
    • Number of answers per request = 3

Careful when sorting images! See the ImagEVAL information document ImagEVAL_info_tasks_eng.pdf

  • Sub-task 2:

    • Number of requests = 30
    • Number of answers per request = 20

Careful when sorting images! See the ImagEVAL information document ImagEVAL_info_tasks_eng.pdf

INFORMATION ON TASK 2

    • Number of requests = 15
    • Number of answers per request = 150

Careful when sorting images! See the ImagEVAL information document ImagEVAL_info_tasks_eng.pdf

Metric : MAP

INFORMATION ON TASK 3

Two metrics will be studied. See FORUM. The last to date (following work carried out by Christian Wolf) adapts better to the task (a lot less radical) and tries to get away from the mailbox merge criterion.

See the information document for the result format.

See assessment software updates.

    • Number of answers: all images count

Important: between the mock test and the official test, certain ground truths will be provided, which will make it easier for participants to improve their techniques.

INFORMATION ON TASK 4

Run 1 must only use sample data sent to participants at the beginning of the test period.

The other runs may use other sample data provided by participants (strongly recommended!). Information concerning this data (quantity and type) must be presented in a separate document (same one as for processing time).

    • Number of requests = 4
    • Number of answers per request = 500

Careful when sorting AND listing images! See the ImagEVAL information document ImagEVAL_info_tasks_eng.pdf

INFORMATION ON TASK 5

Run 1 must only use sample data sent to participants at the beginning of the test period.

The other runs may use other sample data provided by participants (strongly recommended!). Information concerning this data (quantity and type) must be presented in a separate document (same one as for processing time).

Number of requests = 5

Number of answers per request = 1000

Careful when sorting AND listing images! See the ImagEVAL information document ImagEVAL_info_tasks_eng.pdf

NB: For those wishing to give relevant feedback, you can make runs that use sample base containing 5500 images. This must be shown in the complementary information document (same one as for processing time etc.).

Feel free to contact us with you comments, doubts, queries…and particularly via the forum.

| Site map | Home | Participants | Campaigns | Communications | Contact us |