Real-time Sensory Substitution

Published by: One Switch on 2nd Dec 2010 | View all blogs by One Switch

Have you seen the video above (or had it described to you)? What a grin! And what a fantastic idea for making a mainstream game more accessible. Many thanks to Eelke Folmer for sending us this info (read on for more detail), and to Tony for creating it. Exciting and ground-breaking stuff! We can't wait to get our hands on this to take to some roadshows.

"Video games rely upon being able to see, as visual cues typically indicate what input to provide and when. For example, in a first person shooter when an enemy appears this tells the player what to do (shoot or run away) and when (now). If you are blind you may be unable to see these stimuli and although video games provide audio or even vibrotactile feedback, this doesn't contain enough information to decide what input to provide.

Several accessible video games have been developed that provide compensatory feedback that allows players who are blind to play these games using audio or vibrotactile feedback, such the blindhero guitar game we developed. To adapt an existing video game to provide alternative cues we must have access to the source code, which for most commercial video games is not possible as these are closed source.

In this project we explore whether alternative cues can be provided without modifying the game, using a technique we've dubbed "real time sensory substitution". Using real time video analysis we detect those visual cues that require a response from the player and these are substituted into a non visual modality.

The video at the top of this page shows how real time sensory substitution is implemented for Kinect Sports so the hurdles game can be played without visual cues but with vibrotactile feedback provided by Wii remotes. Real time sensory substitution is developed by Tony Morelli, and is planned for public release soon."