Inferring Relevance from Eye Movements Challenge

In the Challenge we have an experimental setup, where the test subject is first shown a question, followed by ten sentences. Five of the sentences are “relevant” to the question (they are of the same topic as the question) and five of the sentences are irrelevant (they have no relation to the topic of the question). One of the relevant sentences is the correct answer to the question. The experimental setup is designed to resemble a real-life information retrieval scenario as closely possible while at the same time retaining a controlled setup where the ground truth is known. Thus, in the Challenge the meaning of “relevant” is defined in terms of this experimental setup. The objective of the Challenge is to find the best methods and features that can be used to predict the relevance from the eye movement measurements.