With the exceptional increase in computing power, storage capacity and network bandwidth of the past decades, ever growing datasets are collected in fields such as bioinformatics (Splice Sites, Gene Boundaries, etc), IT-security (Network traffic) or Text-Classification (Spam vs. Non-Spam), to name but a few. While the data size growth leaves computational methods as the only viable way of dealing with data, it poses new challenges to ML methods.

This PASCAL Challenge is concerned with the scalability and efficiency of existing ML approaches with respect to computational, memory or communication resources, e.g. resulting from a high algorithmic complexity, from the size or dimensionality of the data set, and from the trade-off between distributed resolution and communication costs.

Indeed many comparisons are presented in the literature; however, these usually focus on assessing a few algorithms, or considering a few datasets; further, they most usually involve different evaluation criteria, model parameters and stopping conditions. As a result it is difficult to determine how does a method behave and compare with the other ones in terms of test error, training time and memory requirements, which are the practically relevant criteria.

We are therefore organizing a competition, that is designed to be fair and enables a direct comparison of current large scale classifiers aimed at answering the question “Which learning method is the most accurate given limited resources?”. To this end we provide a generic evaluation framework tailored to the specifics of the competing methods. Providing a wide range of datasets, each of which having specific properties we propose to evaluate the methods based on performance figures, displaying training time vs. test error, dataset size vs. test error and dataset size vs. training time.

General

The overall goal is to develop a 2-class classifier such that it achieves a low error in the shortest possible time using as few datapoints as possible.

Challenge Tasks

The overall goal is to tune your method such that achieves a low test error measured by the area over the precision recall curve in the shortest possible time and using as few datapoints as possible.

Datasets

The raw datasets can be downloaded from ftp://largescale.ml.tu-berlin.de/largescale. The provided script convert.py may be used to obtain a simple (not necessary optimal) feature representation in svm-light format.

Files are in ascii format, where each line corresponds to 1 example,
i.e.

  val1 val2 ... valdim\n
  val1 val2 ... valdim\n
  ...

  Files are binary, to obtain 1 example read 900 or 1156 bytes
  respectively (values 0..255)

Files are ascii, each line contains a string of length 200 (symbols
ACGT), i.e.

  CATCATCGGTCAGTCGATCGAGCATC...A\n
  GTGTCATCGTATCGACTGTCAGCATC...T\n
  ...

Files contains strings (webpages) separated with 0, i.e.
        html foo bar .../html\0

Submission Format

We provide an evaluation script that parses outputs and computes performance scores. We use this exact same script to do the live evaluation. It is suggested to run this script locally on a subset of the training data to test whether the submission format is correctly generated and to evaluate the results (note that the script can only be used on data where labels are available, e.g. subsets of the training data). It requires python-numpy and scipy to be installed. Additionally if matplotlib is installed, the performance figures will be drawn.

Additionally the data submission format is described below:

• Wild Competition (download example submission)

          dataset_size0 -1 traintime0 testtime0 calibration0 objective C epsilon rfb-tau output0 output1 ...
          ...
          dataset_sizeN -1 traintimeN testtimeN calibrationN objective C epsilon rfb-tau output0 output1 ...
          dataset_sizeN index0 traintime0 testtime0 calibration0 objective C epsilon rfb-tau output0 output1 ...
          ...
          dataset_sizeN index9 traintime9 testtime0 calibration9 objective C epsilon rfb-tau output0 output1 ...
  

• SVM Tracks (download example submission)A submission to the SVM track requires to take part in the wild competition, i.e. the submission file must start with the lines required for the wild competition. Then a single empty line announces the SVM-Track specific data:

  >a singly empty line here distinguishes wild from model specific track
          dataset_size0 -1 traintime0 testtime0 calibration0 objective C epsilon rfb-tau
          ...
          dataset_sizeN -1 traintimeN testtimeN calibrationN objective C epsilon rfb-tau
          dataset_sizeN index1 traintime1 testtime1 calibration1 objective C epsilon rfb-tau
          ...
          dataset_sizeN index9 traintime9 testtime0 calibration9 objective C epsilon rfb-tau
          dataset_sizeN -2 traintime0 testtime0 calibration0 objective C1 epsilon rfb-tau
          ...
          dataset_sizeN -2 traintimeK testtimeK calibration9 objective CK epsilon rfb-tau
  
  (or -3 and rbf-tau 1 ... K)
  

• Parallel Track

A submission to the parallel track consists of two parts: The first one has the exact same syntax as the wild track (with time = wall-clock-time). The second part contains the 1,2,4,8 CPU experiment run on the biggest dataset (here C denotes the number of CPUs)

>a singly empty line here distinguishes wild from model specific track
        dataset_sizeN -4 walltimeN dataloadingtimeN calibrationN objective 1 epsilon rfb-tau
        dataset_sizeN -4 walltimeN dataloadingtimeN calibrationN objective 2 epsilon rfb-tau
        dataset_sizeN -4 walltimeN dataloadingtimeN calibrationN objective 4 epsilon rfb-tau
        dataset_sizeN -4 walltimeN dataloadingtimeN calibrationN objective 8 epsilon rfb-tau

Explanation of values (Column by Column)

1. Dataset size – Values must match size of dataset or 10^[2,3,4,5,6,7]
2. Index values of 0…9 (to distinguish values obtained while optimizing) only for the biggest dataset, -1 otherwise; for the SVM track -2 for the C experiment and -3 for the rbf-tau experiment
3. Traintime – Time required for training (without data loading) in seconds / wall-clock time (including data loading) for the parallel track
4. Testtime – Time required for applying the classifier to the validation/test data (without data loading) in seconds / data loading time for the parallel track
5. Calibration – Score obtained using the provided calibration tool (values should be the same if run on the same machine)
6. Method specific value: SVM Objective
7. SVM-C / number of CPUs for the parallel track
8. SVM-rbf-tau
9. SVM epsilon
   • use 0 (for SVM Objective, SVM-C, SVM-rbf-tau, SVM epsilon) if not applicable
   • for SVM please fill in the different parameters, i.e. SVM objective, SVM-C, rbf-tau and stopping condition epsilon, as the meaning of epsilon may differ please explain in the description what epsilon stands for
   • for the linear svm track index -2 should be used when doing the experiment for C
   • for the gaussian svm track index -3 should be used when modifying rbf tau

• The following values in columns 10-… must match the size of the validation/test set.

Assessment of the submissions

Different procedures will be considered depending on the track.

Wild Competition

For the Wild track, the ideal goal would be to determine the best algorithm in terms of learning accuracy, depending on the time budget allowed. Accordingly, the score of a participant is computed as the average rank of its contribution wrt the six scalar measures:

	This figure measures training time vs. area over the precision recall curve (aoPRC). It is obtained by displaying the different time budgets and their corresponding aoPRC on the biggest dataset. We compute the following scores based on that figure: Minimum aoPRC Area under Time vs. aoPRC Curve The time t for which the aoPRC x falls below a threshold (x-overall_minimum_aoPRC)/x<0.05.
	This figure measures dataset size vs. area over the precision recall curve (aoPRC). It is obtained by displaying the different dataset sizes and their corresponding aoPRC that the methods achieve. We compute the following scores based on that figure: Area under Size vs. aoPRC Curve The size s for which the aoPRC x falls below a threshold (x-overall_minimum_aoPRC)/x<0.05.
	This figure measures dataset size vs. training time. It is obtained by displaying the different dataset sizes and the corresponding training time that the methods achieve. We compute the following scores based on that figure: Slope of the Curve b using a least squares fit to a*x^b.

SVM Tracks

For the SVM track, the point is to determine the best tradeoff between the computational effort and the learning accuracy. Accordingly, the score of a participant is computed as the average rank of its contribution wrt the five scalar measures

• Minimal objective
• Area under the Time vs. Objective Curve
• Time to reach objective within 5% tolerance, i.e. minimal t for (t,obj) with (obj-overal_min_objective)/obj<0.05
• Average Training Time for all C/Sigma
• Computational Effort (scaling with dataset size)

Validation Results

Overall Ranking

Test Results

Overall Ranking

Final Results

Final Ranking