Inferring Relevance from Eye Movements Challenge

Summary

The challenge is about a new kind of task that combines a new modality, eye movements, to a user modeling task. The task is to infer the interests or state of the user based on the time series of her eye movements. Application areas will include attentive user interfaces and proactive (i.e. anticipatory) information retrieval.

The data comes from a research project which tries to complement or replace the laborious explicit relevance feedback in information retrieval by implicit feedback. The implicit feedback is derived from eye movements, which contain rich information about the attention and interest patterns of the user. The practical problem is that the signal is very noisy and the correspondence of the eye fixation patterns to user’s attention is often ambiguous.

In order to apply machine learning methods we collected a set of eye movement patterns in a controlled experimental setting where the relevance was known.  The user was asked a question and then shown a set of document titles on a display. Some of the titles were known to be relevant (class ”R”), one of them contained the answer to the question (class ”C”), and the rest were irrelevant (class ”I”). Eye movements were measured when the subjects were reading the titles.

We have so far established that relevance can be predicted to a certain degree. There is ample room for improvement, however, on the first feasibility studies. What is the best method is yet to be solved.

The Challenge

The challenge will consist of two parts: The first is directly relevant to much of current PASCAL research, and the second goes deeper into the eye movement signal.

In the first part, ”standard” feature extraction has already been carried out, and the data for each title to be classified is a time series of feature vectors. The feature set contains 21 features commonly used in psychological studies of reading, calculated for each word on the display. The task is to classify the sequence to one of the three classes, ”R”, ”C” or ”I”.

In the second, optional part, the data is the unprocessed trajectory of eye movements, that is, a sequence of two-dimensional coordinates telling where the user’s gaze was pointed. The data set will be complemented with the exact locations of the words of the titles on the screen. The task will be the same, to classify the trajectory to the three classes. This gives modelers the chance to improve on the traditional psychological feature selection.

The data set for the challenge will be measured in experiments with about 10 subjects, each reading 50 sets of titles.

The data sets will be available March 1st, 2005. The results of the competition will be evaluated by the best classification accuracy obtained using an unlabeled test dataset (a subset of data from the same subjects). The participants will submit the predicted classes of the test data titles to the organizers, who will then check the classification accuracy. An on-line validation data set will be available during the challenge.