ITMO Student Artur Shaikhatarov on Helping People with Special Needs

Artur, in 2018, you talked about an app for people with special needs called LifeFeel. Could you tell us more about its further fate?

The project has already come a long way. As you rightly said, I’ve been working on this app since the ninth grade. Back then, I wanted to create something interesting and helpful while I’m still full of energy and have free time. The idea came from my personal experience. In 2015, I met a person who used a tactile writing system like Braille to communicate with other people. At that moment, I realized that I could make their lives easier by developing a special messenger that would be useful for those who don't know the tactile alphabet, as well. 

I drew inspiration from Interstellar. There was a scene where the main character communicated with his daughter from a black hole using his watch and Morse code. It made me think about creating a smartwatch that would transmit information through vibrations: a short vibration is for a dot, and a long one – a dash.

In 2018, I got into ITMO University and moved to St. Petersburg where I continued to work on the project. By that time, I had developed a prototype of the smartphone and smartwatch allowing users to communicate unilaterally with the deaf-blind. The project became part of R&D for the Innovation Promotion Fund. However, then I realized that I needed a team to follow it through. 

Where did you find your team members?

Lucky for me, there are lots of IT specialists at ITMO University, and many of them wanted to join. That’s how I found Ivan Mosyagin and Arthur Kupriyanov.

How did you want to scale up your product?

From the very beginning, I knew that this project is not about earning money but about helping people and making the world a better place. Yet some organizations can potentially get interested in our work. These are, for instance, companies hiring deaf-blind people. By doing this, they fulfill their social responsibility, provide employment for people with special needs, and help to reduce taxes. I thought that they might want an app for communicating with their employees. It can be interesting to public authorities, too.

“Write a letter for the deaf-blind”

How did you develop the prototype?

We had to figure out how to let the deaf-blind not only receive but also send messages. At first, I thought about using Morse code but it would be too slow. Then, I decided to split the watch screen in half, with dashes and dots being on different sides. However, it was inconvenient for users who couldn’t see the watch and thus made mistakes. 

How did you solve this problem?

We turned to machine learning. I thought of Google’s feature for changing handwritten text into typed text and asked a deaf-blind if they can draw a letter on the screen with their finger. It turned out that they are used to this practice when they meet people who don’t know the tactile alphabet. They take their hand and draw letters in their palm. It all came together! But there was another problem.

What kind of problem?

It’s not hard to build your own neural network for recognizing handwritten letters but it takes training for it to perform well. The problem is that we don't have access to the data banks used by Google and other IT giants. Therefore, we had to collect a set of examples of all the letters of the Russian alphabet, so that the neural network could recognize letters despite their size, handwriting style, and slant. So, I announced the Write a letter for the deaf-blind social action. Anyone could go to our website and write letters with their fingers. Although it was 2020, the year of the pandemic, we managed to collect around 200,000 samples and as a result, we achieved an accuracy rate of over 90%. 

SOS

How did it happen that the Ministry of Emergency Situations (MChS) got interested in your work? 

In 2019, I participated in the Innopolis accelerator and was invited to deliver a TED talk in Kazan. The theme got people's attention. Radiy Khabirov, the head of the Republic of Bashkortostan, made a post where he called me “our Bashkir Zuckerberg”. After a while, the MChS contacted us, too. 

At that moment, we hadn’t adapted our neural network but continued to experiment with Morse code. We met with the representatives of the MChS and told them that people with special needs could use our smartwatch and app to send SOS messages. By pressing a special button, they would notify 112 services about the emergency and provide their geolocation and personal data: address, blood type, and related diseases. Yet it was possible that operators would only receive a notification but didn’t know about what really happened.

Then, it was after that we introduced the neural network in 2020. For the presentation, we invited a deaf-blind person who learned to use the app and sent the message “Fire”.

So, it was a success, right?

Yes, people from the MChS were a bit stunned when they saw an untrained person sending an SOS message with all the necessary details. 

What happened next?

We agreed to gradually integrate our system into 112 services. After they finish their part of the app, we will sign the necessary contracts and get right to work. 

How many regions will it cover?

It depends on 112 services. The first will be the Republic of Bashkortostan. And although we don't have any more contracts, we plan to add two more regions into the system. 

And then?

The major challenge is to attract officials but we want to increase the number to up to 45 regions by the end of 2025. 

Do you take into account that not all people can afford smartwatches?

We discussed it with the MChS people, and they consider providing all people with special needs with smartphones and smartwatches with the built-in app. Yet there is no official information about the number of the deaf-blind in Russia. It is somewhere around 15-20,000 people. Some of them are considered deaf and some – blind. As a result, we have situations when a deaf-blind person gets a radio as a gift. 

The goal is Beijing

As far as I know, the International Paralympic Committee expressed keen interest, too. Is that right?

Yes, it is. in Bashkortoston, where I’m from, the Paralympic movement is actively promoted, and some of the Paralympic medalists are from this region. Many athletes already use smartwatches to track their heartbeat, blood pressure, and so on. So the Committee became interested in using our messenger so that athletes and coaches could communicate.

I was in karate for about 10 years, participated in numerous competitions, and I remember well that my coach and I had special code words for certain moves. When my opponent was vulnerable to an attack, my coach shouted the code word. The same could be applied at the Paralympic competition but instead of the voice, there will be our messenger. 

How is the situation with athletes?

In the summer of 2020, I had the chance to demonstrate the possibilities of the application to coaches. This spring, I went to a training base in my homeland where I could test the app with the help of swimmers, kayakers, and other athletes. Then, we learned that the device performs poorly in water, and while it’s of no use for ping-pong players, it’s suitable for athletes and kayakers.

Is there a chance that we will see your watch on the hands of Russian athletes in Japan?

Sadly, we’re not sure that we will be able to implement the system before the Tokyo Olympics as now the athletes are too busy preparing for the games. Probably, we’ll do it by the Winter Olympics in China. Plus, there will be more sports.

“When will you come?”

Initially, you wanted to create a messenger for the deaf-blind, not only a watch for emergencies or a communicator for athletes. How is your entire project coming along?

We do have a patent but we haven’t uploaded the app to Google Play or App Store yet. We’re waiting for all deaf-blind people to get an opportunity to receive a smartwatch for free. As I said, we discussed it with the representatives of the MChS.

Do you have competitors? 

Our main competitor is a special braille keyboard for computers, laptops, and smartphones. This is a large device that allows you to read books and write articles. 

While such a keyboard isn’t really affordable (it costs around 200,000 rubles), our app is both affordable and allows you to quickly communicate with people without using bulky and expensive equipment. 

People don’t use messengers for reading books but send simple messages like “How are you doing?” or “When will you come?”. We want to create a messenger for communication as an analog to the well-known Telegram and WhatsApp.

How do you want to do it?

We’ve thought about it and decided to create our own app as there are some difficulties with bots and channels. Plus, we want to introduce a broader functionality including communication with the MChS.