Intelligent Supportive Environments for Older Adults (COACH project)
Keywords: Cognitive device, cognitive orthosis, smart homes, assisted cognition, context-aware design, ADL prompting, ADL guidance.
Overview of Research
Dementia reduces a person's ability to perform activities of daily living (ADL) because of related difficulties in remembering the proper sequence of events that must occur and how to use the required tools. The current solution is to have a caregiver supervise activities and continually provide prompts as to the next step in the activity. Family caregivers can find assisting their loved ones to be particularly upsetting and embarrassing as it necessitates invasion of privacy and often role reversal. Research suggests that dependence on a caregiver might be improved using a supportive environment, or cognitive orthosis, that provides needed reminders and monitors progress.
The COACH (Cognitive Orthosis for Assisting with aCtivites in the Home) is a prototype of an intelligent supportive environment being developed to assist people with dementia complete ADLs with less dependence on a caregiver, representing one of the first clinically tested supportive devices to use artificial intelligence techniques. To date, three successive prototypes of the COACH system have completed clinical trials based around the ADL of handwashing with subjects who had moderate-to-severe dementia. In the first two cases, these trials showed that the number of handwashing steps subjects were able to complete without assistance from the caregiver increased noticeably when the device was present. Statistically significant individual changes ranged from approximately 10 to 45 percent. The third trial looked to explore the capabilities of the COACH in a real-world setting and to identify any changes required to further advance the system in future settings within the homes of older adults with dementia. The COACH was installed in a long-term deployment at the Toronto Memory Program – a multidisciplinary, community based, medical facility specializing in the diagnosis and treatment of Alzheimer’s disease and related disorders. This trial was unique to the first two in that the system was not individually calibrated for each specific user, revealing weaknesses in the generalizability of the system to multiple unknown users.
System Overview
COACH uses artificial intelligence (AI) algorithms and a single video camera to monitor progress, determine context, and provide pre-recorded prompts when necessary. For example, if the person forgets to turn off the water, the device prompts him/her to do so. The device has the capability of adjusting its parameters and cueing strategies to meet the changing needs and preferences of each user.
The only hardware visible to the user can be as little as a single video camera mounted over the sink, or may even include a video monitor to provide audiovisual cues (Figure 1). The COACH system can be thought of as three systems working together: 1) the tracking system, 2) the planning system, and 3) the prompting system, as depicted in Figure 2. These three systems, along with several other software modules, interact with the users (Figure 3) who are typically an informal caregiver and with an older adult with dementia.
- The tracking system uses a video camera to identify the position of the user's hands, as well as interactions with task objects (i.e. the soap, towel, tap, water, and sink). The latest version of COACH uses edge skin colour segmentation and flocking routines to track hand and object positions Figure 4 . There are several acceptable pathways to complete the handwashing task, as represented in Figure 5.
- The planning system is responsible for determining which step in the handwashing the user is completing and what the appropriate response (i.e. action) should be. The planning system is currently based on a partially observable Markov decision process (POMDP) model and is able to estimate unobservable states, such as the user's responsiveness and level of dementia. The planning system calculates the most probable state of the world, which is used with a pre-computed policy to determine what action COACH should take. Actions available to COACH are to do nothing (i.e. continue observing the user), select a prompt, or call the caregiver to intervene.
- The prompting system initiates the selected action. If the user is prompted, the amount of detail that is provided in the prompt varies in specificity to match the user's abilities. For example, if the user is being asked to turn on the tap, a low specificity prompt would be audio only (e.g. "[Subject name], you are washing your hands. Turn on the tap.") while a high specificity would have a video demonstrating the step plus more detailed audio (e.g. "[Subject name], use the silver lever in front of you to turn the tap on"). If necessary, the device will repeat a cue or increase the level of specificity of cuing to promote the correct response from the user. If the user does not respond to any of the cues, the device calls for a caregiver to provide assistance.
Figure 1: Photo of the washroom setup at the Toronto Memory Program
Figure 2: Outline of the systems of the COACH system
Figure 3: Conceptual system overview [click to enlarge Fig. 3]
Figure 4: A captured video frame with raw image (top left), hand tracking and region (top right), belief state (bottom left) and skin image (bottom right) [Click to enlarge Fig. 4].
Figure 5: Flow diagram of steps involved in handwashing.
Future Work
Past clinical trials have guided our development of the prototype for the next set of clinical trials, which are to be home-based (i.e. within-home trials) set to start in 2013.
- Tracking: The current colour segmentation/ flocking method will be combined with a depth camera to track the position of both hands, as well as more contextual information about what the hands are actually doing (e.g., scrubbing).
- Planning: The POMDP for handwashing will be generated using a SyNdetic Assistance Process (SNAP). SNAP is a method for automatically generating POMDP policies for complex tasks requiring much less time than previous methods requiring POMDP expert knowledge.
- Prompting: The system will continue to play verbal or video (i.e. verbal and visual prompt over a LCD screen) prompts to the user when an error is made.
Funding Sources
Research Team
Alex Mihailidis, University of Toronto
Jesse Hoey, University of Waterloo
Marek Grzes, University of Waterloo
Babak Taati, University of Toronto
Jennifer Boger, University of Toronto
Craig Boutilier, University of Toronto
Geoff Fernie, Toronto Rehabilitation Institute
Pascal Poupart, University of Waterloo
Related Publications
- Czarnuch, S. and Mihailidis, A. (in review). An efficacy study of the COACH in a real-world deployment. Journal of Ambient Intelligence and Smart Environments (Thematic Issue on Designing and Deploying Intelligent Environments).
- Grzes, M., Hoey, J., Khan, S., Mihailidis, A., Czarnuch, S., Jackson, D. and Monk, A. (in review). Relational approach to knowledge engineering for POMDP-based assistance systems as a translation of a psychological model.International Journal of Approximate Reasoning.
- Czarnuch, S. and Mihailidis, A. (2011). The design of intelligent in-home assistive technologies: Assessing the needs of older adults with dementia and their caregivers. Gerontechnology, 10(3), 165-178. doi:10.4017/gt.2011.10.3.005.00
- Hoey, J., Von Bertoldi, A., Craig, T., Poupart, P. and Mihailidis, A. (2010). Automated Handwashing Assistance For Persons With Dementia Using Video and A Partially Observable Markov Decision Process. Computer Vision and Image Understanding, 114(5), 503-519. (Special Issue on Computer Vision Systems). doi: 10.1016/j.cviu.2009.06.008
- Mihailidis, A., Boger, J., Craig, T. and Hoey, J. (2008). The COACH prompting system to assist older adults with dementia through handwashing: An efficacy study. BMC Geriatrics, 8(28). doi:10.1186/1471-2318-8-28
- Mihailidis, A., Boger, J., Canido, M. and Hoey, J. (2007). The use of an intelligent prompting system for people with dementia: A case study.ACM Interactions (Special issue on Designing for seniors: innovations for graying times) , 14(4), 4‑37. doi:10.1145/1273961.1273982
- Boger, J., Hoey, J., Poupart, P., Boutilier, C., Fernie, G. and Mihailidis, A. (2006). A Planning System Based on Markov Decision Processes to Guide People with Dementia Through Activities of Daily Living. IEEE Transactions on Information Technology in BioMedicine, 10(2), 323-333. doi:10.1109/TITB.2006.864480
- Mihailidis, A., Carmichael, B. and Boger, J. (2004). The use of computer vision in an intelligent environment to support aging-in-place, safety, and independence in the home. IEEE Transaction on Information Technology in Biomedicine (Special Issue on Pervasive Healthcare), 8(3), 238-247. doi:10.1109/TITB.2004.834386
- Mihailidis, A., Barbenel, J.C. and Fernie, G. (2004). The efficacy of an intelligent cognitive orthosis to facilitate handwashing by persons with moderate-to-severe dementia. Neuropsychological Rehabilitation, 14(1/2), 135-171. doi:10.1080/09602010343000156
- Mihailidis, A. and Fernie, G. (2002). Context-aware assistive devices for older adults with dementia. Gerontechnology, 2(2), 173-189. doi:10.4017/gt.2002.02.02.002.00
- Mihailidis, A., Fernie, G. and Barbenel, J.C. (2001). The use of artificial intelligence in the design of an intelligent cognitive orthosis for people with dementia, Assistive Technology, 13(1), 23-39. doi:10.1080/10400435.2001.10132031
