An Intelligent Supportive Environment for Completing a Factory Assembly Task: The NCAPS project (COACH@Work)
Keywords: Artificial intelligence (AI), prompting system, assistive technology, animated agent, supportive workplace environment.
This project in in collaboration with the University of Colorado and is part of the RERC for the Advancement of Cognitive Technologies (ACT).
Overview of Research
One of the most common and empowering activities in our culture is the ability to work. Work provides a validation of an individuals’ ability to earn money and support themselves, to add value to society, and be part of one of the most universal activities in the world.
Housed out of the University of Colorado, the RERC-ACT project takes five projects and integrates them together to create a solution intended to help people with cognitive disabilities to succeed in the workplace. The specific aim of this project is to substantially increase the number of trained individuals with cognitive disabilities who are placed in the workforce and who are successful in their jobs through the development and implementation of computer-based systems to support the knowledge and skills required by people with cognitive disabilities to enter and remain in a workplace.
IATSL is working in collaboration with the University of Colorado on the development of NCAPS, a sub-project on the RERC-ACT to create the COACH@Work system. This research involves adapting IATSL's existing COACH system for use in a factory assembly task, specifically putting together a novelty first aid kit filled with boxes and bottles of chocolates, shown in Figure 1. A breakdown of the task sub-steps is shown in Figure 2.
The system employs two non-recording video cameras to view the workspace and utilizes various vision algorithms to unobtrusively track the worker's progress through the assembly task. Some of these algorithms include detecting optical flow to determine when an item has been picked up, or isolating for particular colours to see if a coloured area of the workspace is exposed meaning that an item is or is not in place. A few of the vision algorithms used by the system are demonstrated in Figure 3. The hand tracker algorithm developed for the original COACH project is also used to observe the worker. This hand tracker algorithm, shown in Figure 4, is used to recognise the position of a user’s hands by tracking groups of pixels that have been calibrated to the user's skin tone. Using artificially intelligent algorithms, the system determines the worker's progress and provides prompts - if and only if he or she requires them - to guide the user through the task. This allows the worker to engage in the task to the maximum extent of his or her abilities. If the user pauses or has a significant timeout, the AI will provide a prompt to complete the next step.
Prompts are played on a monitor that is mounted at eye-level in front of the user. These prompts come in the form of an animated agent, shown in Figure 5, and can be made to reflect the abilities of the user. If necessary, the device will repeat a cue or increase the level of specificity of the prompt to promote the correct user-response. If the worker continues to struggle with a step, an eventual prompt will involve calling a job supervisor to provide assistance. A text message or email will then automatically be sent to the job supervisor, asking for them to provide assistance to ensure the step is competed correctly.
This project is engaging in preliminary testing and trials in the near future. These trials will help determine the efficacy of the system in determining the state of the task as well as the efficacy of the prompts in aiding the users. A comparison between the AI and a heuristic decision making model will also be made. A newer version of the prompting agent is going to be implemented in the near future.
Figure 1: Overhead view of the workspace and the assembly task (click to enlarge Fig. 1)
Figure 2: Flow diagram of steps involved in the assembly task (click to enlarge Fig. 2)
Figure 3: Examples of output from the various vision algorithms used in the system (click to enlarge Fig. 3)
Figure 4: The hand-tracker and regions of interest in the workspace (click to enlarge Fig. 4 )
Figure 5: The Animated Agent prompting avatar (click to enlarge Fig. 5)
Research Team
Ryan Keyfitz, University of Toronto
Alex Mihailidis, University of Toronto
Michael Melonis, University of Colorado
