Melanie Heck

I primarily conduct research in the field of Human-Computer Interaction, with a particular focus on cognitive state prediction from behavioral data. Adopting a holistic perspective, I investigate (1) methods to predict the cognitive state of a user, and (2) adaptation strategies that use the predicted state to dynamically adapt the user interface.

Adaptation strategies for multimodal user interfaces

Computing devices have pervaded almost every aspect of our lives. And while a particular communication channel may be useful in one situation, it can be rather impractical in another context. For example, text input is inconvenient when the hands are needed for another task. On the other hand, most users feel uncomfortable sending and receiving audio messages in public spaces. Multimodal interfaces allow users to choose between multiple interaction modalities, but typically give output in a default modality as specified in the application settings.

In my research, I examine strategies and rules that specify how information should be presented given a users’ situation and state. The objective is to improve the user experience by dynamically adapting either the presented content or its modality.

Cognitive state prediction with biosignal

In the Internet of Behaviors (IoB), data from sensors and other technologies is collected and analyzed to predict human behavior. Biosignals have been made readily available by ubiquitous devices (e.g., fitness trackers or smartphones). The behavioral data that they collect is nothing short of a mirror into the users’ cognitive processes. With the help of AI, biosignals can be used to decode different cognitive states. This enables applications that target not only marketing, but also healthcare or even driving assistants.

My research focus is on the connection between biosignals (e.g., eye movements, facial expressions, and speech) and dynamic cognitive states. Past projects have used machine learning, big data analytics, and other techniques to predict preferences, emotions, and comprehension. A particular interest has been the use of eye tracking to predict user preferences and emotions.

Eye tracking technology & methods

Using eye movements to predict cognitive states obviously requires that the user’s device can reliably estimate the gaze. Commercial eye trackers are both expensive and impractical for pervasive use (e.g., with mobile devices). An alternative method consists in capturing the user’s face with the devices’ integrated camera and applying sophisticated algorithms to estimate the gaze.

My research has primarily targeted gaze estimation methods for desktop computers and laptops, with a particular focus on calibration techniques that align the estimated gaze with the corresponding location on the display.