As large language models (LLMs) become central to political communication, news generation, and decision support, their biases carry real-world consequences. While bias studies in AI have addressed race and gender, political bias remains underexplored, despite its critical role in democratic societies.

In our initial work on the PoliBias benchmark, developed through 2 AI MSc and 6 AI BSc theses supervised by Dr. Jieying Chen, we found that popular LLMs consistently display left-leaning tendencies and exhibit negative bias towards right-wing or conservative parties. These findings, based on over 13,000 parliamentary motions from the Netherlands, Norway, Spain, and the US, raise important concerns. Yet, we currently lack the political science expertise to interpret what such patterns mean for the real-world use of LLMs, for instance, when they are deployed in text summarization, policy analysis, or political Q&A.

This project seeks to bridge that gap. Dr. Chendi Wang, the only member from the Department of Political Science within the Network Institute, will join the project to help interpret the results in light of party systems, ideological polarization, and democratic discourse. Her expertise is crucial to assess which model behaviors constitute problematic bias, which reflect political structures, and what ethical or policy implications arise.

To publish this work at top AI venues, it is essential to enrich the current computational framework with deeper, theory-driven insights. This requires a close collaboration with a political science expert, enabling us to develop more context-sensitive, comparative, and socially meaningful analyses that go beyond technical evaluation.

Together, we will extend the benchmark, analyze ideological and sentiment patterns in model outputs, and explore how LLMs may reinforce or challenge political power dynamics. The project will train two Academy Assistants, one from each field, who will collaborate on data analysis and interpretation, producing a benchmark and publication that combine computational insight with theoretical depth.

This collaboration directly advances the Digital Society mission by addressing fairness, transparency, and the democratic impact of AI systems.

Supervisors:

Academy Assistants: tba