/ Projects / BCI Competition III Challenge

Goals of the organizers

The goal of the “BCI Competition III” is to validate signal processing and classification methods for Brain-Computer Interfaces (BCIs). Compared to the past BCI Competitions, new challanging problems are addressed that are highly relevant for practical BCI systems, such as

Also this BCI Competition includes for the first time ECoG data (data set I) and one data set for which preprocessed features are provided (data set V) for competitors that like to focus on the classification task rather than to dive into the depth of EEG analysis.
The organizers are aware of the fact that by such a competition it is impossible to validate BCI systems as a whole. But nevertheless we envision interesting contributions to ultimately improve the full BCI.

Goals for the participants

For each data set specific goals are given in the respective description. Technically speaking, each data set consists of single-trials of spontaneous brain activity, one part labeled (training data) and another part unlabeled (test data), and a performance measure. The goal is to infer labels (or their probabilities) for the test set from training data that maximize the performance measure for the true (but to the competitors unknown) test labels. Results will be announced at the Third International BCI Meeting in Rensselaerville, June 14-19, and on this web site. For each data set, the competition winner gets a chance to publish the algorithm in an article devoted to the competition that will appear in IEEE Transactions on Neural Systems and Rehabilitation Engineering.

News

Results of the BCI Competition III

are available here.

BCI Competition III is closed

for submissions.

Description of Data Set I in ASCII format corrected

In the description of Data Set I in ASCII format (on the download web page) rows and columns were confused. The description is updated now.

Channel Labels in Preprocessed version of Data Set V in Matlab format corrected

In the Matlab format of Data Set V, the field clab of the variable nfo holds the channel labels. In the preprocessed version of this data set there are only 8 (spatially filtered) channels of the original 32. Erroneously, in the file with the preprocessed data nfo.clab contained all 32 channels, instead of the 8 channel subset. This is corrected now.

Restriction of the test data in data set IIIb

Please see additional information on data set IIIb.

Description of data set IVc updated

In the description of data set IVc it was said that there are 280 test trials. This information is wrong, the test set contains 420 trials. So the submission file to data set IVc must contain 420 lines of classifier output. The description is corrected now.

Submissions to Data Set IIIa and IIIb

Due to the large size of files for submissions to data set IIIa and IIIb, the files should not be sent by email, but put on the ftp server of TU-Graz: please log in by ftp to ftp.tugraz.at as user ftp with password ftp, go to the directory /incoming/bci2005/submissions/IIIa/ resp. /incoming/bci2005/submissions/IIIb/ and put you file there. Be sure to have transfer mode set to binary. If you have problems, please contact ⟨alois.schloegl@tugraz.at⟩.

Clarification of Rules for Data Set V

The description of data set V was updated in order to clarify the requirement ‘The algorithm should provide an output every 0.5 seconds using the last second of data.’, see description V.

 

Data sets

Data set I: ‹motor imagery in ECoG recordings, session-to-session transfer› (description I)
provided by Eberhard-Karls-Universität Tübingen, Germany, Dept. of Computer Engineering and Dept. of Medical Psychology and Behavioral Neurobiology (Niels Birbaumer), and
Max-Planck-Institute for Biological Cybernetics, Tübingen, Germany (Bernhard Schökopf), and
Universität Bonn, Germany, Dept. of Epileptology
cued motor imagery (left pinky, tongue) from one subject; training and test data are ECoG recordings from two different sessions with about one week in between
[2 classes, 64 ECoG channels (0.016-300Hz), 1000Hz sampling rate, 278 training and 100 test trials]

Data set II: ‹P300 speller paradigm› (description II)
provided by Wadsworth Center, NYS Department of Health (Jonathan R. Wolpaw, Gerwin Schalk, Dean Krusienski)
the goal is to estimate to which letter of a 6-by-6 matrix with successively intensified rows resp. columns the subject was paying attention to; data from 2 subjects
[36 classes, 64 EEG channels (0.1-60Hz), 240Hz sampling rate, 85 training and 100 test trials, recorded with the BCI2000 system]

Data sets IIIa: ‹motor imagery, multi-class› (description IIIa)
provided by the Laboratory of Brain-Computer Interfaces (BCI-Lab), Graz University of Technology, (Gert Pfurtscheller, Alois Schlögl)
cued motor imagery with 4 classes (left hand, right hand, foot, tongue) from 3 subjects (ranging from quite good to fair performance); performance measure: kappa-coefficient
[4 classes, 60 EEG channels (1-50Hz), 250Hz sampling rate, 60 trials per class]

Data sets IIIb: ‹motor imagery with non-stationarity problem› (description IIIb, additional information)
provided by TU-Graz (as above)
cued motor imagery with online feedback (non-stationary classifier) with 2 classes (left hand, right hand) from 3 subjects; performance measure: mutual information
[2 classes, 2 bipolar EEG channels 0.5-30Hz, 125Hz sampling rate, 60 trials per class]

Data set IVa: ‹motor imagery, small training sets› (description IVa)
provided by the Berlin BCI group: Fraunhofer FIRST, Intelligent Data Analysis Group (Klaus-Robert Müller, Benjamin Blankertz), and Campus Benjamin Franklin of the Charité – University Medicine Berlin, Department of Neurology, Neurophysics Group (Gabriel Curio)
cued motor imagery with 2 classes (right hand, foot) from 5 subjects; from 2 subjects most trials are labelled (resp. 80% and 60%), while from the other 3 less and less training data are given (resp. 30%, 20% and 10%); the challenge is to make a good classification even from little training data, thereby maybe using information from other subjects with many labelled trials.
[2 classes, 118 EEG channels (0.05-200Hz), 1000Hz sampling rate, 280 trials per subject]

Data set IVb: ‹motor imagery, uncued classifier application› (description IVb)
provided by the Berlin BCI group (see above)
training data is cued motor imagery with 2 classes (left hand, foot) from 1 subject, while test data is continuous (i.e., non-epoched) EEG; the challenge is to provide classifier outputs for each time point, although it is unknown to the competitors at what time points mental states changed; performance measure: mutal information with true labels (-1: left hand, 1: foot, 0: rest) averaged over all samples
[2 classes, 118 EEG channels (0.05-200Hz), 1000Hz sampling rate, 210 training trials, 12 minutes of continuous EEG for testing]

Data set IVc: ‹motor imagery, time-invariance problem› (description IVc)
provided by the Berlin BCI group (see above)
cued motor imagery with 2 classes (left hand, foot) from 1 subject (training data is the same as for data set IVb); test data was recorded 4 hours after the training data and contain an additional class ‘relax’; performance measure: mutal information with true labels (-1: left hand, 1: foot, 0: relax) averaged over all trials
[2 classes, 118 EEG channels (0.05-200Hz), 1000Hz sampling rate, 210 training trials, 420 test trials]

Data set V: ‹mental imagery, multi-class› (description V)
provided by IDIAP Research Institute (José del R. Millán)
cued mental imagery with 3 classes (left hand, right hand, word association) from 3 subjects; besides the raw signals also precomputed features are provided
[3 classes, 32 EEG channels (DC-256Hz), 512Hz sampling rate, continuous EEG and precomputed features]

Schedule

December 12th 2004: launching of the competition
May 22nd 2005, midnight CET to May 23rd: deadline for submissions
June 16th 2005 (approx.): announcement of the results on this web site

Submission

Submissions to a data set are to be sent to the responsible contact person as stated in the data set description. The submission has to comprise the estimated labels, names and affiliations of all involved researchers and a short note on the involved processing techniques. We send confirmations for each submission we get. If you do not receive a confirmation within 2 days please resend your email and inform other organizing committee members, e.g., ⟨alois.schloegl@tugraz.at⟩, ⟨benjamin.blankertz@first.fhg.de⟩, ⟨schalk@wadsworth.org⟩, ⟨schroedm@informatik.uni-tuebingen.de⟩, ⟨jose.millan@idiap.ch⟩
One researcher may NOT submit multiple results to one data set. She/he has to decide for her/his favorite one. However: From one research group multiple submissions to one data set are possible. The sets of involved researchers do not have to be disjoint, but (1) the ‘first author’ (contributor) should be distinct, and (2) approaches should be substantially different.
For details on how to submit your results please refer to the description of the respective data set. If questions remain unanswered send an email to the responsable contact person for the specific data set which is indicated in the description.
Submissions are evaluated for each data set separately. There is no need to submit for all data sets in order to participate.
Each participant agrees to deliver an extended description (1-2 pages) of the used algorithm for publication until July 31st 2005 in case she/he is the winner for one of the data sets.

Organizers

Albany: Gerwin Schalk, Dean Krusienski, Jonathan R. Wolpaw

Berlin: Benjamin Blankertz, Guido Dornhege, Klaus-Robert Müller

Graz: Alois Schlögl, Bernhard Graimann, Gert Pfurtscheller

Martigny: Silvia Chiappa, José del R. Millán

Tübingen: Michael Schröder, Thilo Hinterberger, Thomas Navin Lal, Guido Widman, Niels Birbaumer

References

References to papers about past BCI Competitions.

  • Benjamin Blankertz, Klaus-Robert Müller, Gabriel Curio, Theresa M. Vaughan, Gerwin Schalk, Jonathan R. Wolpaw, Alois Schlögl, Christa Neuper, Gert Pfurtscheller, Thilo Hinterberger, Michael Schröder, and Niels Birbaumer. The BCI competition 2003: Progress and perspectives in detection and discrimination of EEG single trials. IEEE Trans. Biomed. Eng., 51(6):1044-1051, 2004.
  • The issue IEEE Trans. Biomed. Eng., 51(6) contains also articles of all winning teams of the BCI Competition 2003.
  • Paul Sajda, Adam Gerson, Klaus-Robert Müller, Benjamin Blankertz, and Lucas Parra. A data analysis competition to evaluate machine learning algorithms for use in brain-computer interfaces. IEEE Trans. Neural Sys. Rehab. Eng., 11(2):184-185, 2003.

References to BCI Overview papers.

  • Eleanor A. Curran and Maria J. Stokes. Learning to control brain activity: A review of the production and control of EEG components for driving brain-computer interface (BCI) systems. Brain Cogn., 51:326-336, 2003.
  • Jonathan R. Wolpaw, Niels Birbaumer, Dennis J. McFarland, Gert Pfurtscheller, and Theresa M. Vaughan. Brain-computer interfaces for communication and control. Clin. Neurophysiol., 113:767-791, 2002.
  • José del R. Millán. Brain-computer interfaces. In M.A. Arbib (ed.), “Handbook of Brain Theory and Neural Networks, 2nd ed.” Cambridge: MIT Press, 2002.
  • Andrea Kübler, Boris Kotchoubey, Jochen Kaiser, Jonathan Wolpaw, and Niels Birbaumer. Brain-computer communication: Unlocking the locked in. Psychol. Bull., 127(3):358-375, 2001.

References to BCI Special Issues.

  • IEEE Trans. Biomed. Eng., 51(6), 2004.
  • IEEE Trans. Neural Sys. Rehab. Eng., 11(2), 2003.
  • IEEE Trans. Rehab. Eng., 8(2), 2000.

 

Logo

Follow us:

© Knowledge 4 All Foundation Ltd.