The Netherlands witnesses an annual surge in registered protests, occasionally shrouded in violence, exemplified by the Avondklokrellen, farmers’ protests, and the Extinction Rebellion’s actions. Understanding the tipping point at which these demonstrations descend into violence is paramount for societal well-being, empowering authorities to safeguard citizens and maintain order. How can we predict these tipping points? Recent qualitative research reveals the existence of an emotional energy “turning point” known as ‘collective effervescence’, akin to a symphony of applause reverberating through a stadium when a musician takes the stage for an encore. Today’s digital landscape presents an opportunity; street cameras capture protests. The question looms: Can we automatically detect these crucial turning points?

In this project, we tackle the daunting task of automatically detecting turning points in group emotional energy using state-of-the-art computer vision techniques. Our primary objective is to forge a codebook and develop an automated classification system capable of identifying collective effervescence within protests. From a social science vantage point, this breakthrough will enable us to a) discern and decode group emotions—a vital component in understanding collective behaviour, and b) uncover how complex social dynamics can be researched computationally. Meanwhile, from a legal and criminological perspective, we gain crucial insights into the conditions that propel peaceful protests toward violent upheavals. Leveraging the wealth of video material from PI Rosenkrantz Lindegaard‘s collaboration with the NCTV on a previous project, encompassing both peaceful and violent protests spanning from 2020 to 2023, we stand poised to unlock the enigma concealed within societal unrest.

Researchers:

In the context of energy transitions, the adoption of smart grid technologies and associated digital platforms are important drivers of democratization of energy production and consumption. Renewable Energy Cooperatives (RECs) play a central role in these processes, as they serve as valuable platforms for exchanging information, knowledge, and for collective agency. RECs can facilitate inclusive and participatory decision-making processes, empower citizens through local ownership over energy resources, and ensure equitable and affordable access to energy [3]. However, recent studies indicate that REC membership is predominantly comprised of middle or upper-class white males [4] . The number of RECs in the Netherlands, as well as policymakers’ interest in RECs, is rapidly increasing. Therefore, it is crucial to explore strategies for making RECs more accessible to low-income households, vulnerable groups, women, and the youth. To address this, we integrate insights from various fields (e.g., innovation studies, sustainability transitions, polycentric governance), methods and approaches (e.g., regression analysis, ethnomethodology, interaction analysis, citizen science) to answer this research question (RQ):

How can RECs and digital technologies and platforms promote social innovation and multiple value creation to foster social inclusion and citizen empowerment?

We use citizen science and ethnographic methods (e.g., focus groups, interviews, and observations) to collect data from 25 Amsterdam households (REC members and non-members). Participants will receive a Wi-Fi Energy Monitor1 that gives insight into energy consumption via an app. We study their energy consumption, behaviour, and perceptions towards RECs. This will generate insights into to what extent the knowledge of their individual and collective data enhances awareness, participation, trust, and community cohesion. Furthermore, through surveys and interviews, we also examine REC governance models (data collection, storage, and sharing) and social, political, and technical prerequisites for effective energy governance, including policies and regulations.

Researchers:

Students:

This project aims to develop a prototype for a dialogue system that personalizes recipe instructions for a dish to a user. Cooking recipes are an instructional text that many people interact with frequently, yet they pose unique challenges for automatic understanding and use in conversational systems. For automatic understanding based on language processing, recipe texts pose specific challenges such as imperative mood (“stir the batter”); implicit arguments (“beat ∅”); and complex anaphoric expressions that relate to intermediate products. For a conversational agent, recipes for the same dish often differ in which cooking actions they describe explicitly and how; it is challenging yet necessary to properly explain steps while taking into account recipe difficulty and the user’s cooking proficiency. A conversational recipe assistant would enhance user interaction with recipes for many users, assisting in recipe selection, instructing steps to the user while cooking, and addressing questions throughout.

Part I of the project will address the linguistic challenges posed by recipe text through semantic parsing. Representing recipes as graphs is a common choice for this task to capture dependencies between ingredients, tools, and actions. We will adapt an existing state-of-the-art semantic parsing pipeline to parse recipes for the same dish and then align actions between recipes for a more detailed dish-level representation. Part II of the project will investigate dialogue management to facilitate sequence expansion of recipe instructions at varying levels of detail for an adaptive conversational agent. The cooking domain is a broad knowledge space, and being able to address questions asked during cooking instruction is not trivial. A good understanding of the recipe itself is required, as is formulating the answer properly to a particular user.

Researchers:

Students:

Dutch children are reading less and less, resulting in a decline of their reading skills. Children with insufficient reading skills have trouble fully participating in school and society. Factors such as reading motivation and availability of support are important determinants of how well children read. Social robots are a promising educational tool, e.g., support offered by a robot math tutor increases children’s math performance. We aim to investigate how social robots can be used to enhance primary education children’s reading motivation and comprehension. In doing so, we strive to develop a personalized reading robot tutor, who 1) helps children to select a fitting book, which they will then start reading individually, and 2) regularly meets with the child to discuss the ongoing reading process and book-related experiences. In its tutoring role, the robot interacts with the child by asking personalized questions and giving feedback, combining the book’s contents and the child’s interests and reading level. Being tailored to the individual child’s needs, we hypothesize that the robot will boost children’s reading motivation, and by that, ultimately, their reading comprehension.

In this project, we propose a two-phase approach:

1. We chart the potential of using a reading robot tutor in school by a) organizing focus groups with different types of stakeholders – children, teachers, and librarians – to identify their wishes/needs/opinions/ideas concerning implementing robots in school to support children’s reading; b) examining the (technical) possibilities of using robots to support children’s reading, focussing on exploring the ways in which child-robot interaction can best be personalized.

2. We run an intervention study to investigate the effects of the robot tutor on children’s reading motivation and comprehension by a) using phase 1 as input for the intervention design; b) expanding an existing AI framework to implement the intervention.

Researchers:

Students:

Access to health is a fundamental human right that is enshrined in the statutes of the World Health Organisation (WHO). The right implies that everyone is entitled to health services they need, regardless of their social, economic or political standing. While the entitlement to health care is well elaborated in policy documents, the situation on the ground depicts challenges in accessing health services. The challenges are prominent with undocumented migrant communities who for long have experienced marginalisation and discrimination in accessing health care services, yet they suffer from ill health disproportionately. There is information asymmetry on the rights of undocumented people and how to handle them, particularly by the gatekeepers in hospitals. There already exist tools that aim to help undocumented people with information about their rights and possibilities, such as the Amsterdam City Rights app.[1]However, it turns out that this does not fully solve the needs of undocumented people. Their challenges typify wicked problems, which due to their complex and interconnected nature are subject to real-world constraints which hinder risk-free attempts to find a solution. In this project, we aim to develop a better understanding of the specific needs of undocumented people to get access to health care and investigate whether a personalized recommendation system will help them to fulfil those needs. The project starts by investigating how the problems faced by undocumented people in accessing health can be addressed. We will also investigate the possibilities of AI-driven chatbot interfaces to personalize and tailor information to the needs of specific persons in specific circumstances. Based on this, we will develop a digital solution that will help undocumented people to get the information and support that they need when looking for medical attention. The solution will be co-designed and evaluated by representatives of the undocumented people.

Researchers:

Students:

In this project, art historical insights on the qualitative appreciation of specific iconographic and pictorial aspects of paintings are applied to influencers’ visuals on Instagram to discover systematic patterns and insight into the role of visual elements in multimodal communication. The results of this interdisciplinary research will inform researchers of computer vision models to identify visual elements in digital media that attract viewers and explain their preferences. For influencers, artists and users of multimodal platforms in which communication through visual imagery is central, the insights from this research contribute to knowledge about the workability of visual elements and thus to their successful deployment.

In different research fields, visual elements have been found to affect viewer responses. Advertising research shows that the ways in which visual metaphors are portrayed affect viewers’ appreciation. Auction masters developed rules of thumb when valuing paintings: cats higher than dogs; a woman’s portrait higher than a man’s; creatures walking towards you higher than walking away. Analyses of art auctions have shown systematic effects of colours in paintings on their hammer prices. The visual elements that we may identify seem endless, but in Kress and Van Leeuwen (2020) a useful analytic toolbox has been developed (without assumptions about appreciation) to categorize images on the basis of explicitly defined visual features. In this project we want to explore the effects of visual features on the hammer prices of carefully curated corpora of paintings (RQ1), thus revealing specific patterns that may explain social media responses on Influencers’ Instagram posts (RQ2). In this interdisciplinary way, we will make visual contemporary communication insightful and measurable, and contribute to the knowledge about visual communication and the processes underlying global digital platforms like Instagram.

Researchers:

Students: