In late 2024 Open Inclusion, on behalf of the Institute of People Centred AI at the University of Surrey (as part of the UKRI Centre for Doctoral Training in AI for Digital Media Inclusion), conducted research to understand the implications of AI enabled technologies on people with a diverse range of disabilities.
Our research included:
- Quantitative consumer research (UK, n=500)
A survey of a nationally representative sample (age, gender and region), including 200 disabled and 300 non-disabled individuals, to understand current attitudes towards and usage of AI.
- Qualitative expert research (global, n=17)
In-depth interviews and a roundtable with experts from Europe/UK, North America, Middle East, Africa and Australasia working at the intersection of disability and AI. With this remarkable group, we documented key opportunities and concerns with practical examples illustrating each, considering both what is happening now and looking to the future.
Access to more detailed research findings
The research was first released in person at Google’s Accessible Design Centre in London in January this year. The recording of the full session is available here and includes Christopher Patnoe from Google, Prof. Adrian Hilton from the University of Surrey, Anna Landre and Breandan Ward from Open Inclusion sharing their personal and professional perspectives, and Michelle Parfitt and Christine Hemphill sharing the research findings. You can also access the research slides from that presentation.
There is also a recording of a 15 min fireside chat from the Zero Project Conference at the UN in Vienna between Christine Hemphill and Christopher Patnoe discussing the research findings.
Why disability-inclusive AI matters?
AI influences many aspects of our lives. We use it every day in many ways, whether we realise it or not, and it is used on us in every sector we engage with. From government services we can/can’t access, what job we may or may not know about or get, how we learn, work and engage with others, to what we watch on TV, AI increasingly impacts us all.
This increasing influence raises critical questions for disability inclusion. Is AI empowering disabled people and fostering equity, or is it inadvertently creating new barriers and exacerbating existing inequalities?
This stage of AI proliferation, while exciting, can inadvertently introduce technical and social ‘debt’ in the form of embedded biases, current and future harms. Harms that we have unintentionally allowed in, during our excited but underinformed stage of deployment. ‘Debt’ that will need to be identified, unpicked, addressed and reconciled in the future incurring interest along the way.
Addressing this debt will require conscious effort to identify, dismantle, and rectify these issues,
As AI advances both in power and pervasiveness its failures will increasingly be our society’s failures, ever deeper and more impactful if we don’t proactively address them. Conversely well-conceived and deployed inclusive AI solutions can address currently unmet needs in new ways, reducing today’s inequities and increasing the capability of everyone in society to experience more equitably and contribute more fully.
Consumer perceptions: A Growing Unease
Our UK consumer research revealed a general trend of greater positivity towards the current impact of AI compared to their future expectations. This indicates a widespread concern about the trajectory of AI deployment.
Notably, disabled respondents were less optimistic about AI’s impact on society, both now and in the future, compared to their non-disabled counterparts.
Examining three specific AI use cases; biometrics, voice interfaces, and adaptive digital media we found some significant differences in perceived benefits, concerns, and desired mitigations between disabled and non-disabled respondents.
For example,
- Benefits: Disabled individuals were less likely to perceive the benefits of these AI applications compared to non-disabled individuals. For instance, only 42% of disabled respondents felt biometrics would speed up tasks, compared to 51% of non-disabled respondents.
- Risks: Concerns about potential risks were more pronounced among disabled respondents. For voice technologies, 32% of disabled individuals worried about unreliability making tasks harder, compared to 22% of non-disabled individuals.
- Mitigations: A significantly higher proportion of disabled respondents desired effective mitigations to manage AI outcomes and reduce risks across all three use cases. For adaptive digital media, nearly half of disabled respondents (48%) wanted increased personal data security and transparency about AI usage (46%), over 10% higher than non-disabled respondents (34% for both).
In summary, disabled people in the UK are consistently more concerned about increasing application of AI both generally and more specifically across a range of use cases relative to their non-disabled counterparts. They are less positive about the benefits, more concerned about the risks and wish for more interventions such as laws and regulations, transparency of where and how AI is being used, human involvement, especially in monitoring and explainability of outcomes relative to people without disabilities.
The implications of AI bias (as it relates to disability) is perceptible across the broader, non-expert community of people who will be impacted by solutions.
Expert perceptions: The Time to Act is Now
The experts who participated work in a range of roles and organisations from technologists at AI giants like Google, to AI ethicists in academia, disability-oriented AI creatives developing new assistive technologies, those who have co-created adaptive government services with disabled people and those who are helping develop national and international AI policies and guidelines that consider disabled people.
Some specialise in one area of disability (such as the Director of Research for the American Federation of the Blind), one area of AI (such as Voice UI) or another characteristic (such as individuals with age-related access needs). Many work across a wide range of disabilities. Over half have personal as well as professional experience of disability.
Shared concerns
There were a number of concerns about AI impacting disabled people that were widely shared across this group.
These included,
- Privacy and security – Increased challenges and risks for disabled people
- Amplified social biases – AI amplifying the reality of ableism
- Data biases –
- Data gaps: disabled people insufficiently represented in the underlying datasets
- Correlation gaps: disability status not known to understand differential implications
- Data application: thinning datasets and use of averages to apply data for training or fuelling an AI solution removes or deprioritises outliers – including disability-informed experiences
- Distorted perspectives – The power of the creator. Specifically, who is / isn’t involved in creating AI solutions influences knowledge, perspectives and blind spots
- Asymmetric implications – Benefits go to one group and the costs or risks to another group
- Left behind or explicit harm – Ongoing social biases and discrimination can influence how seriously individuals, companies and governments treat the risk of harm to disabled people from the development and deployment of AI solutions.
One major privacy concern relating to disability is using AI to identify whether someone is disabled or not.
There are many reasons this may do significant harm, from discriminating against someone on the basis of their disability such as for insurance or work opportunities to using this illegally or at least unethically gained, unconsented and unverified information to assess social benefit rights.
“In some countries government agencies were accused to use some pieces of this data without consent to confirm the status of a disability. Just tracking your social media, your post to identify if you really have a disability or not use this information to pay pension and financial support to you or not.”
Yonah Welker EU AI and Disability Policy Expert
The atypical nature of many disability experiences mismatching with AI practices and usage was noted by many of our experts.
“Disability is a non-average characteristic. AI is trained on averages, so when it encounters someone who is not average… if protective measures aren’t taken, it’s likely to reject or be biased against the person who’s not average”
Arielle Silverman, AFB Director Research
In amplifying biases, there were a range of specific examples provided where AI-enabled solutions harmed disabled individuals, including in their ability to access work, gain effective education or health outcomes. Datasets may be incomplete, distorted relative to a more diverse reality and inappropriately applied. A key to managing these risks is first understanding the potential for harm then protecting individuals with atypical characteristics such as disability from the harm.
“I think we’ve talked about all the potential areas of bias. It’s the data, it’s the model development, it’s the evaluation.
So just be aware of all of those potential areas of bias, and then also identify the level of risk. What is the risk? Is it financial? Is it social? Is it life threatening for somebody with a disability? Keeping that in mind, what’s the cost of an error here?”
Sara Basson, Accessibility Evangelist, Google
In summary, people involved right across the AI ecosystem (those defining AI rules and regulations, funders, designers, developers, procurement, deployment and management) need greater awareness of these risks. They also need stronger proactive drivers, both carrot (such as education) and stick (such as regulation), to offset them.
Shared excitements
There were a many solutions that our experts shared where AI is enabling new and exciting capabilities for people across the spectrum of disability
These included,
- New or improved assistive technologies – Many disabled people rely on assistive tech (AT). AI can help personalise digital AT solutions to an individual’s needs. It can create new, efficient, lower cost and higher capability solutions that were not previously possible, improving independent capability and personal agency. Three particularly interesting areas for assistive tech solutions progressing included,
- AI + robotics
- AI + alternate and adaptive interfaces
- AI + connected ‘smart’ homes or spaces
- Efficient accessibility – whether designing or testing a built environment or digital solution, AI can help identify access barriers disabled people may experience, more efficiently offer solutions or directly remediate it. It doesn’t replace skilled experts or user insight, but can augment human input and improve efficiency.
- Micro-personalisation – mainstream technologies increasingly offer adaptive capability which can help disabled people equitably and efficiently use the product. Beyond accessibility, (ensuring there is an option available for people with different disabilities) micro personalisation supports preferences and ensures solutions are efficient not just accessible. AI can lower the cost and increase the potential for understanding personal needs and preferences and supporting them with low user effort.
Some exciting examples of AI being used to enable and empower disabled people in new ways or at significantly lower costs than was possible before were shared.
“Aaron’s avatar, the first one cost $25,000 to make. We can now give an avatar to someone basically at almost no cost.”
Lavonne Roberts, Scott-Mogan Foundation (who is creating an avatar with personalized voice technology allowing people to use an adaptive communication that maintains their own voice after they no longer can speak due to ALS or other speech degenerative conditions)
AI can greatly enhance the efficiency of disabled people at work or in education also removing barriers that exist today. Tools from Grammerly to LLM based code assistants can reduce the impact of someone’s disability on their productivity.
“It [AI] will do so much of the first pass coding and recoding and rewriting… People who don’t have 10 flying fingers and who can’t compete in the workplace today, well it now no longer matters how fast your fingers fly, it’s how you think.”
Gregg Vanderheiden, Prof Emeritus Trace Centre (Assistive Tech and Accessibility) Uni of Maryland
Some AI-powered assistive tech creates new challenges that will need to be understood and addressed, such as privacy concerns around apps or wearables such as glasses that help people who are blind or low vision understand an environment they are in.
“There are some new image description apps that have come out recently, but it requires a person to take a photo of someone or something and upload it into the cloud. People might not want to be photographed without their consent, and have their photos described by AI. There has been a lot of controversy lately about whether the AI should be able to describe people estimating a person’s age, race or gender because sometimes it might not get that right”
Arielle Silverman, Research Director American Foundation for the Blind
The same technology (computer vision and facial recognition) is used in potentially harmful ways also, such as monitoring people in repressive states.
Is there a risk of disabled people being used to enable creators to justify building things that otherwise may not be acceptable, or governments or commercial organisations deploying solutions using a ‘Trojan horse’ justification to get it through?
We need to define ways that help reduce the risk of misuse or even unintentional harm and maintain the benefits to the disability community of these powerful solutions.
Key takeaways and practical actions
We can all play a part to ensure this new set of technologies does less harm and provides more equitable, ethical and sustainable value in the many different solutions and services where it may be applied.
Across our spaces of influence, professionally and personally, we can help create, inform, progress, accept, critique and reject solutions to ensure AI delivers to its promise, more fully to all.
There were 7 consistent recommendations to ensure more disability-inclusive, ethical and useful AI-solutions.
- Inclusive involvement throughout
- Transparency
- Explainability
- Human oversight
- Offer alternatives (and describe trade offs)
- Manage risks aligned to consequences
- Define and work within guardrails and monitor outcomes
Depending on your professional and personal points of influence, these principals may emerge in very different ways.
Here are a few ideas for people in procurement, research or AI design/development roles;
Conclusion: Our collective responsibility
As Jeremy Dunning and Richard Roger noted way back in 2004, “Technology is too important to leave to the technologists.”
This sentiment is true for the development and deployment of AI today. If the perspectives and experiences of marginalized communities, specifically including disabled people, are not deeply and meaningfully involved throughout the creation, deployment and monitoring AI solutions, we will fail the current and future generations. We all need access considerations sometimes and some people need additional consideration in the design of solutions more consistently due to their disabilities. Let’s make sure we design for humans and let’s recognise and respect the full range of natural human differences.
AI inclusion is everyone’s business. We can all play a part in making this new suite of technologies work better for us all.
Be curious, be critical, learn what others know, share what you know, challenge, and proactively support the creation of an AI ecosystem that works equitably and effectively for all, including those with disabilities.
This is the moment where the path may make our world more inclusive and empower capability or less so and further disabled and marginalise a valuable capable community. The difference is what we do now. So please, lean in, be bold, and let’s work collaboratively to shape a future where AI truly empowers and includes us all.