StorySign app converts text in children's books to sign language

December 5, 2018 by Bob Yirka, Phys.org
Credit: Huawei

A team of researchers and engineers at Huawei AI has come up with a unique way to help deaf children learn to read sign language—by using AI and augmented reality to interpret printed books. They have also hit on a way to help deaf children develop enjoyment in reading.

The idea for the app was a response to the frustration many parents feel when trying to read a book to their deaf child—in most such situations, a hearing parent is trying to include their deaf child in an activity that is an automatic rite of passage for those without a hearing impairment. To solve this problem, researchers at Huawei struck on the idea of using to help a child read a book. The AI/AR system is embedded in an app called StorySign that can run on a phone or tablet—the parent or child holds the phone over the page and an avatar is superimposed next to the text and interprets the words via signing.

As the story progresses, the app highlights the words being signed, helping the child understand that the words are connected to the sign used to represent them. The researchers report that they also did a significant amount of work to develop an avatar that children could relate to. They worked with Aardman Animations to create an avatar named Star, which does more than just display —she also uses body and to convey emotions, helping the to better understand the contextual nature of the material in the book. One caveat is that the app only works with certain children's books, all published by Penguin Random House.

The app was developed with support from the European Union of the Deaf and the British Deaf Association. It is free, and available for Android and Apple products in 10 languages in 10 European countries. The developers say that if the app proves popular, it will be made available in more regions and in more languages.

More information: consumer.huawei.com/uk/campaign/storysign/app/

© 2018 Science X Network