/ Projects / Visual Object Classes Challenge

Development Kit

The development kit provided for the VOC challenge 2006 is available. You can:

Datasets

Following completion of the challenge, the training/validation sets with complete annotation have now been made available on the database page. Some additional example images are online.

The VOC2006 data includes some images provided by Microsoft Research Cambridge and “flickr”. Use of these images must respect the corresponding terms of use:

Details of the contributor of each image can be found in the file “contrib.txt” included in the dataset. Any queries about the use or ownership of the data should be addressed to the organizers.

Second Round

The second round aims to give an opportunity for groups who participated to submit additional results, and for others who might have been unable to meet the first deadline to participate. The deadline for submission of results for the second round is 11pm GMT, Friday 30th June 2006.

The second round will be run as per the first round. In evaluating the second round, submitted results will be evaluated separately from the first round to give consideration for the additional time available. Comparison of the results obtained between the first and second rounds will follow. As in the first round, ground truth annotation will not be provided for the test data.

On completion of the second round, the test set annotation will be released for researchers wishing to use the data for further evaluation.

Post-Challenge Submissions

We invite submission of results obtained after the completion of the challenge, both by participants and those who did not participate in the challenge. Results will be treated as distinct from the results obtained during the challenge due to the availability of the test data annotation. If you would like your results included in the online summary of results, please contact Mark Everingham, me@comp.leeds.ac.uk.

Results

A technical report summarizing the challenge and results is now available. The results may also be browsed online. Slides presented at the challenge workshop can also be downloaded.

If you make use of the VOC2006 data, please cite the following reference in any publications:

    @misc{pascal-voc-2006,
      author = "Everingham, M. and Zisserman, A. and Williams, C. K. I. and Van~Gool, L.",
      title = "The {PASCAL} {V}isual {O}bject {C}lasses {C}hallenge 2006 {(VOC2006)} {R}esults",
      howpublished = "http://www.pascal-network.org/challenges/VOC/voc2006/results.pdf"
    }

Timetable

  • 14 Feb 2006 : Development kit (training and validation data plus evaluation software) made available.
  • 31 March 2006: Test set made available.
  • 25 April 2006, 11pm GMT: DEADLINE for submission of results. There will be no further extensions.
  • 7 May 2006: Challenge workshop to be held in conjunction with ECCV06, Graz, Austria. The programme is now online.
  • 30 June 2006, 11pm GMT: DEADLINE for submission of second round results.
  • After 1 July 2006: Test set annotation made available.

Introduction

The goal of this challenge is to recognize objects from a number of visual object classes in realistic scenes (i.e. not pre-segmented objects). It is fundamentally a supervised learning learning problem in that a training set of labelled images is provided. The ten object classes that have been selected are:

  • bicycle, bus, car, motorbike
  • cat, cow, dog, horse, sheep
  • person

There will be two main competitions:

  1. For each of the ten classes, predicting presence/absence of an example of that class in the test image.
  2. Predicting the bounding box and label of each object from the ten target classes in the test image.

Participants may enter either (or both) of these competitions, and can choose to tackle any (or all) of the ten object classes. The challenge allows for two approaches to each of the competitions:

  1. Participants may use systems built or trained using any methods or data excluding the provided test sets.
  2. Systems are to be built or trained using only the provided training data.

The intention in the first case is to establish just what level of success can currently be achieved on these problems and by what method; in the second case the intention is to establish which method is most successful given a specified training set.

Data

The training data provided consists of a set of images; each image has an annotation file giving a bounding box and object class label for each object in one of the ten classes present in the image. Note that multiple objects from multiple classes may be present in the same image. Some example images can be viewed online.

Annotation was performed according to a set of guidelines distributed to all annotators. These guidelines can be viewed here.

The data will be made available in two stages; in the first stage, a development kit will be released consisting of training and validation data, plus evaluation software (written in MATLAB). One purpose of the validation set is to demonstrate how the evaluation software works ahead of the competition submission.

In the second stage, the test set will be made available for the actual competition. In contrast to the 2005 VOC challenge, no ground truth for the test data will be released until after the challenge is complete.

Following completion of the challenge, the training/validation sets with complete annotation have now been made available on the database page.

The table below summarizes statistics of the training/validation and test data. The data has been split into 50% for training/validation and 50% for testing. The distributions of images and objects by class are approximately equal across the training/validation and test sets. In total there are 5304 images, containing 9507 annotated objects.

train val trainval test
Images Objects Images Objects Images Objects Images Objects
Bicycle 127 161 143 162 270 323 268 326
Bus 93 118 81 117 174 235 180 233
Car 271 427 282 427 553 854 544 854
Cat 192 214 194 215 386 429 388 429
Cow 102 156 104 157 206 313 197 315
Dog 189 211 176 211 365 422 370 423
Horse 129 164 118 162 247 326 254 324
Motorbike 118 138 117 137 235 275 234 274
Person 319 577 347 579 666 1156 675 1153
Sheep 119 211 132 210 251 421 238 422
Total 1277 2377 1341 2377 2618 4754 2686 4753

Database Rights

The VOC2006 data includes some images provided by Microsoft Research Cambridge and “flickr”. Use of these images must respect the corresponding terms of use:

For the purposes of the challenge, the identity of the images in the database, e.g. source and name of owner, was obscured. Details of the contributor of each image can be found in the file “contrib.txt” in the final release of the data. Any queries about the use or ownership of the data should be addressed to the organizers.

Submission of Results

Participants are expected to submit a single set of results per method employed. Participants who have investigated several algorithms may submit one result per method. Changes in algorithm parameters do not constitute a different method – all parameter tuning must be conducted using the training and validation data alone.

Details of the required file formats for submitted results can be found in the development kit documentation. For each competition you should produce a single text file, named following the examples in the development kit. For example, your results file for competition 1 (classification trained on VOC data) on the “car” class should be named “comp1_cls_test_car.txt”; your results file for competition 4 (detection trained on your own data) on the “car” class should be named “comp4_det_test_car.txt”.

If you intend to submit results for several different methods, please place results for each in a separate, clearly named directory. For each method you submit, please include a brief description of the approach used. If you wish to withhold details of the method, e.g. for commercial reasons, please make this clear. Methods submitted without a description will be judged in a separate category of the challenge.

The results files should be collected in a single archive file (tar/zip) and placed on an FTP/HTTP server accessible from outside your institution. Email the URL and any details needed to access the file to Mark Everingham, me@comp.leeds.ac.uk. Please do not send files directly by email.

Publication Policy

The main mechanism for dissemination of the results will be the challenge webpage. It is also possible that an overview paper of the results will be produced.

If you make use of the VOC2006 data, please cite the technical report of results, see the results section.

Organizers

  • Mark Everingham (Oxford/Leeds), me@comp.leeds.ox.ac.uk
  • Luc van Gool (Zurich)
  • Chris Williams (Edinburgh)
  • Andrew Zisserman (Oxford)
Logo

Follow us:

© Knowledge 4 All Foundation Ltd.