It is not easy for most of us to discover our creative potential, reach inner harmony and express our feelings with sounds or images. And the task gets even harder for those who can’t see. Luckily, in this age technology can come to the rescue. 

The issues of using neurotechnologies and human-computer interaction to help children with any impairments experience the most of life were discussed back at the 2019 St. Petersburg International Cultural Forum. Over the next months, a team of experts was formed bringing together specialists from different organizations: ITMO University, the Konstantine Grot boarding school for the visually impaired, the Peterfond fund of educational projects, the Digital Youth NGO, and the Diana Gurtskaya Center for Social and Cultural Rehabilitation.

The team decided to create a research laboratory that would aim at exploring ways to adapt digital interfaces to work with children who need special adaptation. That was the start of the Neuroculture project. 

“This is one of the first truly interdisciplinary projects in Russia, where every step – setting the task, discussing it, implementing the ideas – is the work of experts in completely different fields. Why did ITMO join the project? The reason is in the goal that we set in our strategy for 2027: first of all, we are focusing on working directly with the individual,” explained Rector of ITMO University Vladimir Vasilyev during the presentation.

The miracle table 

In the summer of 2020 the project was supported by the Presidential Grant Foundation. The financing was aimed at developing a special device to help visually impaired children in their creative endeavors. As a result, the team came up with Neurotafl. It was developed by experts from ITMO’s Center of Usability and Mixed Reality and media artists. The team was headed by Mikhail Klimovskiy, advisor to ITMO University Rector.

The device itself resembles a sci-fi spaceship control panel – a special holder has an interactive haptic surface. It was created by students of St. Petersburg Stieglitz State Academy of Art and Design. 

“Our students had a challenging task, because it is through design that we bring technology into user experience,” says Sergey Helmyanov, head of Industrial Design Department at the Academy. “The professors also assisted the students in choosing the best option for the eventual solution. We can only evaluate their choices by testing them.”

By manipulating special cubes placed on the device’s surface, users can create various auditory content. And you don’t even need to know musical notation or have absolute pitch – the neural network at the core adapts to each user and sets the sound range and the device’s operation individually for each user. 

“From the very beginning, children came up with another name for the device, which I count as an award. They dubbed it “The miracle table,” remembers Yuri Didevich, a media artist and architect of the project. “The cubes used by children are active computer elements or computer avatars. They trace user interaction. And, truth be told, this is the main catch – that the system can from the very beginning watch and analyze all user actions. In the future, we are hoping that the system will learn from the experience and help users interact with it.”

But the device doesn’t only serve a creative purpose. It is also meant to help children develop abstract thinking. How can we explain what is a square or a circle to someone who’s never seen them? Children can touch the shapes and read about them using Braille, but that doesn’t give them the whole picture. 

Neurotafl can assign special sounds or haptic sensations (using vibrating motors) to figures drawn by children. This should promote the work of those brain regions that are associated with imagination. 

The creators of the “miracle table” also hope that the device can be used for communication between children and parents who don’t have the same impairments. Using the same “language” thanks to Neurotafl, they can better understand each other.      

New opportunities 

The children at Konstantine Grot boarding school are now only beginning to explore Neurotafl. This stage will last until spring 2021. By then, the developers will have gathered enough data to make an updated, more user-friendly modification of the device. All in all, it is planned to create three prototypes. 

After that, the developers hope, it will be possible to discuss mass production of such interfaces to deliver them to various boarding schools all over Russia. To achieve this, the device will have to be smaller, lighter, cheaper, and easier to use. 

“Apart from the project team, students of our program in Multimedia Technologies, Design and Usability took part in the project,” comments Artem Smolin, head of the Center for Usability and Mixed Reality at the Faculty of Software Engineering and Computer Systems. “Together with these Master’s students, we analyze the features of the device and its field of application, as well as apply our competencies in human-computer interaction.”

Teachers will also join in the work to figure out the best ways to apply the device in education and potentially stimulate the activity of different brain regions. In this work, they will be assisted by physiologists from the  Pavlov Institute of Physiology of the Russian Academy of Sciences. All of these steps will help create the methodological recommendations that will be shared with other boarding schools in Russia. 

“The innovations born in our school are broadcast to other regions of the country. We serve as a federal resource center and a place for pilot projects of the Ministry of Education,” says Alexey Mukhin, head of the Konstantine Grot boarding school for the visually impaired. “This project is aimed at the social and cultural adaptation of people with health limitations, it helps them cross the borders between technologies, art, and science. The social importance of the project is highlighted by the need to change the way such people are looked at in our society.”

It is planned to bring such devices to other categories of people. For instance, the developers are looking to test one of the prototypes at the neuropsychiatric boarding school in Peterhof. They are also considering applying the device in treating depression and other disorders in adults.  

Finally, the developers want to design other devices with a completely different technology at the core that will also help children with health limitations. Right now, though, the device is only used by students of the Konstantine Grot boarding school, who demonstrated their first performance using the device at the presentation.