About the workshop

The State-based User Modelling Workshop at WSDM 2020 (SUM ’20) welcomes participation of researchers and stakeholders to explore techniques for state-based user modelling, and their applications, such as:

  • Conversational systems
  • User representation and recommender systems
  • Task understanding and supporting user tasks
  • Task-based information retrieval
  • State-aware evaluation
  • Human in the loop
  • User-aware systems
  • Personalisation ML algorithms
  • Cognitive/contextual user understanding
  • State-aware ML algorithms

This workshop aims at bringing together researchers from academia and industry that focus on different aspects of state-based user modelling to exchange ideas and build state-aware user-centric systems. We encourage submissions to this workshop in a variety of topics aiming to discuss the challenges associated with capturing and effectively utilising user state in different web search and data mining applications.

State-based User Modelling

Inferring and utilising user states and goals is becoming a timely challenge for successfully leveraging intelligent user-centric systems in the real-world. Examples of such systems are conversational agents, intelligent assistants, educational and contextual information retrieval systems, recommender/matchmaking systems and advertising systems, which all rely on identifying the user state, in order to provide the most relevant information to the user and assist them in achieving their goals.

For example, due to their interactive nature, search and recommender systems are notoriously hard to develop and evaluate, since they involve a multi-step decision-making process, where a stream of interactions occurs between the user and the system. Traditionally, to make the problem tractable, the interactions are often viewed as independent, but in order to improve such systems further, it becomes important to consider and leverage user states.

In the recent years, various interactive machine learning techniques (including bandit based and reinforcement learning based techniques) have been applied for interaction tasks as these approaches allow to learn and continuously change the strategy based on user feedback and other type of rewards.

Even though many such systems would significantly benefit from having a representation of the user’s state, there has been very limited work towards building intelligent learning mechanisms that can be used to identify, represent and update the state of the user.

Hence, devising information retrieval systems that can keep track of the user’s state and make recommendations based on this has been listed as one of the grand challenges of information retrieval that needs to be tackled during the next few years. In this context,we believe it is timely to organize a workshop that revisits the problem of designing and evaluating state-aware user-centric systems and makes sure the community, spanning academic and industrial backgrounds, is working on the right problem.

Goals and Objectives

This workshop aims to facilitate a forum for leveraging intelligent methods for modelling user’s state and its transition in an effective manner. Modelling user’s state is a challenging task with many currently open questions:

  • User representation and recommender systems: How to design accurate state-aware representations of the user and embed that information in predictive models?
  • Task understanding and supporting user tasks: How to identify the specific sub-task a user is trying to accomplish and design state-aware intelligent systems that assist users in each step of their goals?
  • State-aware evaluation: How to develop metrics and evaluation techniques that model and understand user states, so as to provide a more sensitive and enhanced evaluation of the user-centric systems?
  • Human in the loop: How to include the human in the learning "loop" in an interactive fashion, making feedback an integral part of the modelling?
  • User-aware systems: How to incorporate user state information in systems to achieve better human-machine intelligence?
  • Cognitive/contextual user understanding: How to understand from a cognitive point of view how actions taken by users and their context can influence their state?
  • State-aware ML algorithms: How do the different ML paradigms which enable a system to identify and leverage user states compare for different application scenarios?

To progress and answer these questions, it is imperative for research communities from cognitive science, intelligent tutoring systems, information retrieval, human computer interaction, psychology and other diverse fields to bring forward their inputs, which is one of the primary goals of this workshop.

We encourage sharing the ongoing work that is geared towards addressing these challenges, thus aiming to improve the research landscape and understanding of this emerging topic.

We aim to provide a platform for budding ideas from different fields to unite together giving inspiration to more powerful user modelling approaches. WSDM, with its attendance by a broad spectrum of cross-disciplinary researchers offers the ideal venue for this exchange of ideas. Topics of interest include areas concerning the above mentioned highlighted points.

Organizing Committee

  • Emine Yilmaz, University College London and Amazon
  • John Shawe-Taylor, University College London, UNESCO Chair in Artificial Intelligence
  • Sahan Bulathwela, University College London
  • Maria Perez-Ortiz, University College London
  • Rishabh Mehrotra, Spotify Research
  • Colin de la Higuera, Université de Nantes, UNESCO Chair in teacher training technologies with OER
  • Davor Orlic, COO, Knowledge 4 All Foundation

Important dates

  • December 1, 2019: deadline for submission of contributions to workshops
  • Final deadline extension Dec 15 
  • December 27, 2019: paper acceptance notification
  • February 7, 2020: WSDM-2020 workshops   


Location: Hyatt Regency Houston/Galleria | Room: Regency C (Level 2)

08:30 Welcome and setting the stage
  • Sahan Bulathwela, University College London
09:00‑10:00 Invited talk: Engagement, metrics and recommenders
  • Mounia Lalmas, Director of Research at Spotify
10:00‑10:30 Coffee Break
10:30‑11:00 Invited Talk: Measuring and improving user satisfaction with voice-based conversational systems
  • Eugene Agichtein, Winship Associate Professor at Emory University
11:00‑11:30 Contributed talk: Fatigue-Aware Ad Creative Selection by  Daisuke Moriwaki, Komei Fujita, Shota Yasui, and Takahiro Hoshino
11:30-12:00 Contributed talk: SAGE: Interactive State-aware Point-of-Interest Recommendation by Behrooz Omidvar-Tehrani, Sruthi Viswanathan, Frederic Roulland, and Jean-Michel Renders
12:00-12:30 Contributed talk: Scalable Psychological Momentum Estimation in E-sports by Alfonso White and Daniela M. Romano
12:30-14:00 Lunch
14:00‑15:00 Invited talk: Learning from User Interactions in Productivity Applications
  • Ahmed Hassan Awadallah, Principal Research Manager at Microsoft
15:00‑15:30 Coffee Break
15:30‑16:30 Invited talk: Incorporating User State in Task Based Information Retrieval and Recommendation of Educational Resources
  • Emine Yilmaz, Professor at University College London
16:30‑17:00 Closing remarks
17:00‑17:30 Optional drinks and dinner


Name Title and material
Sahan Bulathwela, University College London Welcome and setting the stage (slides)
Mounia Lalmas, Director of Research at Spotify Engagement, metrics and recommenders (slides)
Eugene Agichtein, Winship Associate Professor at Emory University Measuring and improving user satisfaction with voice-based conversational systems (slides)
Daisuke Moriwaki, Komei Fujita, Shota Yasui, and Takahiro Hoshino Fatigue-Aware Ad Creative Selection (slides)
Behrooz Omidvar-Tehrani, Sruthi Viswanathan, Frederic Roulland, and Jean-Michel Renders SAGE: Interactive State-aware Point-of-Interest Recommendation (slides)
 Alfonso White and Daniela M. Romano Scalable Psychological Momentum Estimation in E-sports (slides)
Ahmed Hassan Awadallah, Principal Research Manager at Microsoft Learning from User Interactions in Productivity Applications (slides)
Emine Yilmaz, Professor at University College London Incorporating User State in Task Based Information Retrieval and Recommendation of Educational Resources (slides)

Call for papers

SUM’20 Workshop is an event co-located within the WSDM conference on February 7th, 2020, in Houston, TX (USA).

A full-day workshop is organized for experts in Artificial Intelligence, Web Search and Data Mining to come together to address the timely topic of state-based user modelling.

The expected outcome of the workshop is to facilitate a forum for leveraging intelligent methods for modelling user's state and its transition in an effective manner. We welcome three categories of papers:

  • Research papers present novel methods and use cases of leveraging state-based user modelling. They may present experimental breakthroughs and exemplary application of user state modelling and personalisation. 
  • On-going work may cover any aspect of the development of new AI tools and methodologies related to modelling user state. In this case the project or action may not be finished, and, especially in those cases where the expected impact is long-term, the evaluation may be incomplete.
  • Position/Vision papers may concern untouched challenges or the use of technologies which up to now have not been tested on the ground.

We welcome technical contributions and position papers from a wide range of topics, including but not limited to:

  • User representation and recommender systems
  • Task understanding and supporting user tasks
  • State-aware evaluation
  • Human in the loop
  • User-aware systems
  • Cognitive/contextual user understanding
  • State-aware ML algorithms

Papers will appear in a workshop proceedings, which will be made publicly available online at the workshop website for no charge. Accepted papers will have a presentation (oral and/or poster) at the workshop. Papers must be submitted in PDF according to the new two-column ACM format published in the ACM guidelines, selecting the generic "sigconf" sample. Papers should be no more than six pages in length, including diagrams, appendices and references. SUM’20 submissions should be double-blind. 

All papers should be a maximum 6 pages in length. Position papers can be shorter (2 pages). In all cases, one additional page of cited references is allowed. All submissions and reviews will be handled electronically. Papers must be submitted through Easychair by 23:59, AoE (Anywhere on Earth) on December 1st, 2019.

For inquires about the workshop and submissions, please email to info@k4all.org 

Mounia Lalmas, Director of Research and Head of Research in Personalization, Spotify


User engagement plays a central role in companies and organisations operating online services. A main challenge is to leverage knowledge about the online interaction of users to understand what engage them short-term and more importantly long-term. Two critical steps of improving user engagement are defining the right metrics and properly optimising for them. A common way that engagement is measured and understood is through the definition and development of metrics of user satisfaction, which can act as proxy of short-term user engagement, mostly at session level. In the context of recommender systems, developing a better understanding of how users interact (implicit signals) with them during their online session is important for developing metrics of user satisfaction. Detecting and understanding implicit signals of user satisfaction are essential for enhancing the quality of the recommendations. This talk will present various works and personal thoughts on how to develop metrics of user engagement, which recommender systems can optimize for. An important message was that, for recommender systems to work both in the short and the long-term, it is important to consider the heterogeneity of both user and content to formalise the notion of engagement, and in turn design the appropriate metrics to capture these and optimize for. One way to achieve this is to follow these four steps: 1) Understanding intents; 2) Optimizing for the right metric; 3) Acting on segmentation; and 4) Thinking about diversity.

Speaker bio

Mounia Lalmas is a Director of Research at Spotify, and the Head of Tech Research in Personalization. Before that, she was a Director of Research at Yahoo, where she led a team of researchers working on advertising quality for Gemini, Yahoo native advertising platform. She also worked with various teams at Yahoo on topics related to user engagement in the context of news, search, and user generated content. Prior to this, she held a Microsoft Research/RAEng Research Chair at the School of Computing Science, University of Glasgow. Before that, she was Professor of Information Retrieval at the Department of Computer Science at Queen Mary, University of London. Her work focuses on studying user engagement in areas such as native advertising, digital media, social media, search, and now audio. She has given numerous talks and tutorials on these and related topics, including recently a WWW 2019 tutorial on "Online User Engagement: Metrics and Optimization". She is regularly a senior programme committee member at conferences such as WSDM, KDD, WWW and SIGIR. She was co-programme chair for SIGIR 2015, WWW 2018 and WSDM 2020.

Eugene Agichtein, Winship Associate Professor, Emory University, USA
Eugene Agichtein, Winship Associate Professor, Emory University, USA


As spoken conversational assistants operate in increasingly complex domains, predicting user satisfaction in conversational systems has become critical. In particular, online satisfaction prediction (i.e., predicting satisfaction of the user with the system after each turn) could be used as a new proxy for implicit user feedback, and offers promising opportunities to create more responsive and effective conversational agents, which adapt to the user’s engagement with the agent. Measuring immediate changes in user satisfaction also enables for natural user studies without explicit disruptions to user experience. I will discuss our progress on conversational satisfaction modeling, and applications to automatically evaluating the effects new agent features, e.g., conversation topic suggestion, design choices, e.g., new recommendation phrasing, and subtle VUI features, e.g., prosody modulation for agent responses.

Speaker bio

Dr. Eugene Agichtein is a Winship Associate Professor of Computer Science at Emory University in Atlanta, USA, where he leads the Intelligent Information Access Laboratory (IR Lab). Since January 2019, he has been an "Amazon Scholar” at Amazon Alexa. Eugene's research spans the areas of information retrieval, natural language processing, data mining, and human computer interaction, most recently focusing on conversational search and recommendation. Together with colleagues and students, Eugene published over 100 papers, and has been recognized by multiple awards, including A.P. Sloan Fellow and the 2013 Karen Spark Jones Award from the British Computer Society, and "test of time" and best paper awards from the SIGIR and WSDM conference. He was Program Co-Chair of the WSDM 2012 and WWW 2017 conferences. Website: http://www.cs.emory.edu/~eugene/.

Ahmed Awadallah, Principal Research Manager, Microsoft Research


Information workers spend significant amounts of their time managing personal information such as communications, documents and tasks. A recent study reported that the average information worker spends an estimated 28 percent of the workweek managing e-mail and nearly 20 percent looking for internal information or tracking down colleagues who can help with specific tasks. This highlights the potential value intelligent technologies could provide by realizing faster and more effective collaboration and information sharing. The use of artificial intelligence in online services has been steadily increasing. Over the past couple of years, we have worked on techniques for understanding user interaction in communication, information sharing and task management to create new intelligence experiences that can help us be more productive. In this talk, I will present an overview of this line of research.

Speaker bio

Ahmed Awadallah is a Principal Research Manager at Microsoft Research AI. He leads the Language & Information Technologies team which focuses on enabling machines to understand and communicate in natural language, answer complex questions, assist with task completion and understand how people interact with information.  Before joining Microsoft Research, Ahmed received his Ph.D. in Computer Science and Engineering from the University of Michigan Ann Arbor and spent some time working at IBM studying a variety of natural language processing problems. Ahmed published 80+ peer-reviewed and is an  inventor on 25 (granted and pending) patents. His past research focused on understanding, measuring and improving  Web search systems. More recently his research focuses on creating scalable machine learning models for NLP and building natural language interfaces to services and data. More details can be found at: https://aka.ms/ahmed

Emine Yilmaz, Turing Fellow and Professor at University College London
Emine Yilmaz, Turing Fellow and Professor at University College London


Accurate representation of user state plays a critical role in understanding user needs and providing them with accurate information at the correct time. In this talk, I will talk about two applications in which incorporating user state could be highly important: task based information retrieval and recommendation of educational resources. In the first part of the talk, I will describe the work we have done on incorporating representations of user state to task based search engines and show how one can use these representations to predict user needs. I will then move on to recommendation of educational resources, describe a way of representing user state (skills) and describe methods that can be used to match learners with educational resources based on their skill level.

Speaker bio

Emine Yilmaz is a Turing Fellow and Professor at University College London (UCL), Department of Computer Science, as well as an Amazon Scholar at Amazon. Between 2012 and 2019, she also worked as a research consultant for Microsoft Research Cambridge, where she used to work full as a researcher prior to joining UCL. Emine's current research interests include information retrieval, data mining and applications of machine learning. Her research until now has received several awards including a Bloomberg Data Science Research Award in 2018, the Karen Sparck Jones Award in 2015 and the Google Faculty Research Award in 2014. She has published research papers extensively at major venues such as ACM SIGIR, CIKM and WSDM, gave several tutorials as part of top conferences, and organized various workshops. She has served in various roles including co-editor-in-chief for the Information Retrieval Journal, PC Chair for ECIR 2020, ACM SIGIR 2018 and ACM ICTIR 2017 Conferences, Practice and Experience Chair for ACM WSDM 2017, and as the Doctoral Consortium Chair for ECIR 2017. She is also one of the recipients of the prestigious EPSRC Fellowship.

We would like to improve the subsequent workshops on State-based User Modelling.

Please give us your feedback here

Keep in touch

If you would like to receive more information and get involved in organizing the subsequent SUM workshops.

Please submit your email address here