As part of the THEMIS 5.0 project, we are thrilled to share the outcomes of our first co-creation series through a series of blog posts. These workshops, conducted across eight European countries, were designed to engage stakeholders, particularly AI end-users, to better understand their concerns, hopes, and expectations regarding the integration of AI in their respective sectors. In this post, we delve into the methodology adopted for these workshops, which were instrumental in gathering crucial insights from diverse participants across Europe.
The Two-Pronged Approach: Attitudes and Trustworthiness
The co-creation methodology employed in these workshops centred around two main themes:
AI End-Users’ Attitudes Towards AI at Work
Contextual Understanding of Trustworthiness Parameters
Each of these themes was addressed in separate sessions during the workshops, allowing for focused and in-depth discussions. This structured approach enabled participants to explore and articulate their perspectives on AI, both in terms of its practical implications at work and the broader ethical considerations associated with its deployment.
Session 1: Understanding AI End-Users' Attitudes
The first session, often referred to as the "scenario game," was specifically designed to probe participants' attitudes toward AI. This interactive session utilised scenarios tailored to the specific use case sectors represented in the workshops. Participants were presented with three distinct scenarios, each offering three possible responses, to gauge their views on:
Their general attitudes towards the integration of AI tools in their professional environments
The types of flaws in AI tools that they would find least acceptable
Their likely course of action when their professional judgment contradicts an AI tool’s suggestions
To facilitate dynamic discussions, participants physically moved to tables corresponding to their chosen responses. Each table, moderated by a facilitator, provided a space for deeper exploration of the motivations behind their choices. This physical and interactive element of the workshop was designed not just to gather data but to encourage participants to engage fully with the topic, leading to more nuanced insights into their attitudes towards AI.
The aim of this session was to gain a comprehensive understanding of how AI end-users perceive the implementation of AI in their work environments. The diverse opinions gathered offered valuable insights into the general attitudes, potential barriers, and hopes that these end-users harbour regarding AI. Notably, the session was structured to capture a wide array of opinions rather than to determine the most popular views, ensuring that even minority perspectives were recognised and valued.
Session 2: Exploring Trustworthiness in AI
The second session focused on trustworthiness—an essential factor in the adoption and success of AI systems. Given that the THEMIS 5.0 ecosystem is designed to optimise three out of seven identified trustworthiness parameters—fairness, accuracy, and robustness—this session was critical in understanding how these parameters are perceived by end-users.
Participants were divided into groups and tasked with creating user requirements for hypothetical AI tools, guided by templates that prompted them to consider each of the trustworthiness parameters in turn. Over three rounds, participants discussed and defined what fairness, accuracy, and robustness meant within the context of their specific sector. Although they were provided with formal definitions from the EU’s High-Level Expert Group on AI, participants were encouraged to interpret these concepts based on their professional experiences and sector-specific needs.
At the conclusion of these discussions, participants voted on which trustworthiness parameter they considered most crucial for their sector. This exercise not only generated a wealth of data on user requirements but also fostered a sense of ownership among participants, as their input directly influenced the development of the THEMIS 5.0 tools.
Results and Analysis: A Pan-European Effort
The co-creation workshops were conducted across eight European countries: Austria, Bulgaria, Denmark, Greece, Lithuania, Poland, Spain, and The Netherlands. Each country was assigned a specific use case sector—healthcare, port management, or disinformation in media—based on regional relevance and the likelihood of gathering meaningful insights. A total of 188 AI end-users participated in these workshops, with the following distribution across sectors:
Port Management: 75 participants across three workshops (average 25 participants per workshop).
Disinformation in Media: 72 participants across three workshops (average 24 participants per workshop).
Healthcare: 41 participants across two workshops (average 20.5 participants per workshop).
The recruitment of participants varied by country, with seven countries handling their recruitment strategies internally and one outsourcing the task to a research agency. A wide range of communication methods was employed, including social media, messaging services, physical posters, direct invitations, and word-of-mouth. The goal was not to achieve demographic representativeness but to ensure diversity in professional backgrounds, age, gender, and experience with AI, enriching the discussions and outcomes of the workshops.
The results from these workshops were meticulously translated into English and analysed by DBT (The Danish Board of Technology) now known as Democracy X, which has taken the lead in synthesising these findings. The first session's scenario game results were analysed to identify themes across the different scenarios, which will inform the creation of user preference groups by our partner Trustilio. These themes provide crucial insights into the general attitudes, hopes, and barriers experienced by AI end-users in the respective sectors.
For the second session, the user requirements were analysed within each trustworthiness parameter. This analysis led to the identification of 200 unique user requirements, which were further clustered within the parameters of fairness, accuracy, and robustness. These clusters serve as the foundation for developing indicators of trustworthiness that will guide the future development of AI systems under the THEMIS 5.0 framework.
The success of these initial workshops marks just the beginning of THEMIS 5.0’s engagement activities. The insights gathered will inform the next phases of the project, which include online stakeholder workshops and living labs in Denmark, Bulgaria, Greece, and Spain. These workshops will involve AI developers, service providers, and other key stakeholders in evaluating the first mock-ups of the THEMIS 5.0 ecosystem.
Stay tuned for our next blog post, where we will explore the results of the workshops including the attitudes of participants towards AI and their hopes for its integration into their professional lives.
Comments