Interview: AR app ‘Dorothy’ helps people living with dementia

Infotec.news speaks exclusively to Dr Samir Shah and Ben Williams from the team behind the innovative, intuitive ‘Dorothy’ app that is already transforming lives, enabling those living with dementia to be more independent and supporting those who care for them.

The project has just been awarded an £80,000 Discovery Award grant as part of the £4m Longitude Prize on Dementia — which aims to tackle one of the biggest health challenges of our time. We wanted to know exactly how the app helps people and where the technology will go next. 

Dorothy App prototype being trialled at Abbcross Nursing Home, Romford

Where did the idea for the ‘Dorothy’ app come from?

Dr Samir Shah: From my patients. I’m a neuropsychiatrist and run a memory clinic. I’m also the Chief Clinical Information Officer at North East London NHS Foundation Trust. You’ll know that we’ve got an ageing society and an increasing amount of people living with dementia.

One of the first things dementia can affect is the prefrontal cortex, which means a person loses language. That impairs their ability to express themselves and their needs. In care homes, you’ll often find people living with dementia can get quite agitated because they can’t communicate their needs and so those needs aren’t met. You can understand the frustration. It must be difficult for somebody in their 90s, who has lived a full life, to have to ask for help to go to the bathroom. If your struggle to communicate, it’s all the more frustrating. That’s difficult for the person living with dementia — and puts a strain on those around them, too.

It’s important for these people to be able to interact with the world. Less interactivity affects your mood, activity levels and well-being.

Then I met Ilya Rubin. His grandfather had moved in with him, having started to get a little confused. But he also couldn’t say what he wanted, which made him quite agitated. Ilya had a lot of experience working on autonomous navigation systems for car companies and thought something similar might help his grandfather to be more independent. We started to look into this together, just before Covid. We’ve been working on it for a few years.

What does the ‘Dorothy’ app do?

Dr Samir Shah: First, we map the area. We use head and object recognition to map fixed points. That gives us much more accuracy than GPS, which is good to four or five metres. Ilya tells me Dorothy is good to within 30 centimetres. Having mapped the space, we tag areas of importance. With an icon-based interface, the user can say, ‘Okay, I want to go here.’ In yellow and black, which is best for vision, the app then draws a route guiding them to wherever they need to go. We call it the ‘yellow brick road’, after The Wizard of Oz. That’s why the app is called ‘Dorothy’.

We don’t use any text on screen to make the app simpler and more intuitive. We’re also based in east London, where for many people English is not their first language.

The app then uses learning algorithms to prompt the user: ‘It’s been a few hours, so you probably need to get up and go to the bathroom.’ We can tailor that to whatever the person needs.

Dorothy App prototype being trialled at Abbcross Nursing Home, Romford

Given that people using the app struggle to communicate and can get agitated, it must have been tricky to scope and test this.

Dr Samir Shah: We had to be sensitive, yes. I do a lot of clinical work with care homes and explained what we were trying to do. They were happy to test the app while I was there. With that support, we tested it for about a year, on people actually living with dementia.

What was the result?

Dr Samir Shah: Very quickly, we could all see the benefits. One of our biggest success stories involved somebody I’d diagnosed about a year before he took part in the test. He’d been admitted to the care home and could never really orientate himself. We started using Dorothy with him and within a week he was saying, ‘Oh! My bedroom is this way!’

It can’t all have been that simple.

Dr Samir Shah: Yes, there were times I didn’t think it would work. We had a feature with a telephone ring sound alert. Some people using the app didn’t know what to press. Several people heard the ring and start dancing. That was a bit unexpected.

But we learned from these things and amended the icons and features. The thing we found very quickly is that even helping people in a relatively small way can really impact their day-to-day life.

You also said that this could help those around the person living with dementia.

Dr Samir Shah: One of the biggest issues with dementia is the system round the person. If you care for someone living with dementia, you have a 33% chance of depression. That is very high. We also wanted to support staff in care homes.

We use both cameras on the iPad so that relatives and carers — if they’re authorised to have access — can see what the person is doing, where they’re going and whether they need help. They can easily check that they’ve eaten meals and been to the bathroom.

The care home can also make sure they contact the appropriate person and at the right time. Let’s say the person living with dementia has three grown-up children who all help out but are each available at different hours or days of the week. We can put that into the app and automatically switch between them. As a relative, you no longer get 20 missed calls. Instead, you get one call when you’re available. You might finish work, get home and then the care home calls to tell you what’s being going on that day. You’re in a better place to take that in and then offer help or make decisions.

We also found that those visiting a care home might not speak the same language as staff — English might not be the first language of either. But you also might arrive at a care home in the evening to find a different shift on duty, with staff you don’t know. That can be a barrier to communication. The app makes it easier to see what is needed, helping visitors and staff to support the person living with dementia.

The app is working now — and well — so what is the new funding for?

Dr Samir Shah: We’re developing Dorothy into something bigger. Currently, it works within care homes. We’ve now started to try and make as many environments dementia friendly as we can. But this is where I hand over to Ben.

Ben Williams, you’re Senior Project Lead at innovation centre Care City. How did you get involved in the ‘Dorothy’ app?

Ben Williams: At Care City, we’re involved in a whole number of different health care projects. I’ve worked a lot in advocacy in social care: helping people who don’t have the mental capacity but face life-changing decisions. I worked with Dr Shah on some other things and then heard about this. I could immediately see how it would solve so many problems and give people back their independence.

As Dr Shah says, we wanted to extend what the app is already doing. We’ve just completed a consultation with the London Clinical Network on Dementia, looking into the barriers and problems experienced after diagnosis. A number of issues came out of that but a lot of people said things like, ‘I would love just to be able to go to a cafe with friends.’ There were those who missed going to the shops.

Or there was the daycare centre staff who told us they used to organise trips to museums but can’t do that any more because there isn’t time and resource to be sure they know where everything is — the accessible entrance, the cloakroom or the toilets. When you’re caring for people where incontinence is an issue, you can’t spent 20 minutes wandering around trying to find things. It can be stressful for everyone.

That’s the kind of area we now want to look into with this funding for the ‘Dorothy’ community. A museum or supermarket, cafe or pub, can let you download something to the app. Augmented reality will then show you where everything is. It’s quick and simple, and you don’t need to ask. People told us they want to get out and do things, and engage with the world, but they’re scared to do so. That’s what we’re hoping to do with the app — take away that fear.

How far can this technology go?

Ben Williams: Last week, we were at the HQ of the Alzheimer’s Society to show them this technology. They’ve just worked on a project to make Wembley Stadium dementia friendly and they asked us if we’d be able to map individual seats. You could then be sat there with your friends enjoying a game, but when you want to get a drink or find the nearest toilet, the Dorothy app would be able to guide you there and then back to your individual seat. That would be useful for me, let alone everyone else.

And that’s the sort of thing we’re going to explore: testing the system to its limits to see how many different environments we can open up.

Dorothy, Care City and TPXimpact shared the £80,000 Discovery Award grant to develop the ‘Dorothy’ community. Based on the wayfinding ‘Dorothy’ app, the platform means any organisation can offer indoor navigation, making them more accessible and dementia-friendly while enabling independence for the user.