Speech Up is a virtual speech coach that can analyze speech, facial expressions, emotions, and behavior, identify fillers, and even detect cognitive distortions such as prejudices or stereotypes.
As shared by Yulia Khlyupina, the idea came from personal experience. As a student and a researcher, she had to face her public speaking fear when participating in conferences, seminars, and project defenses, and after talking to her colleagues, she realized that almost everyone experiences some difficulties during public talks.
The app works as follows: a user records their presentation (or loads a ready-made video of it into the program) for the assistant to evaluate it against five criteria, namely coherence, clarity, dynamics, persuasiveness, and adherence to communicative norms. The coach reviews not only specific parameters but also all the factors as a whole.
Once the system analyzes the video, the user obtains detailed interactive analytics and recommendations on what can be improved, which stem from common methods used by psychologists and speech experts. When used regularly, the system can offer data on a user's progress over time.
“There’s no need for neural networks if you want to determine the volume or tempo of a speaker’s speech, but when it comes to such complex characteristics as confidence, persuasiveness, and congruence (the alignment of a person’s emotions with their speech), it’s the job for AI. To cover those parameters, we turned to neural network architectures that work with multimodal data from several sources, including video, audio, and text,” notes Yulia Khlyupina.
Yulia Khlyupina. Credit: vk.com/generationtech
The neural network is built on a dataset of over 64 hours of recorded interviews and 2,500 texts, markups of which involved a large team of linguists and psychologists.
Yulia worked on Speech Up when she was doing her Master’s at ITMO’s Faculty of Digital Transformation and defended it within the Startup as Thesis program. Last summer, the team joined a six-month SberStudent program and this fall, presented a full-fledged version of their assistant, which was tested on employees of various companies and participants of the accelerators by the National Technological Initiative and Higher School of Economics.
She also participated with this project in Гонка стартапов (Startup Race – a Russian TV show for young entrepreneurs – Ed.) where she had to complete tasks from Russian star experts designer and businessman Artemy Lebedev, TV presenter and sports commentator Dmitry Guberniev, and others. As a result, she got to present her technology at the Demo Day; her project received high marks and will soon be tested at Sberbank.
