AR/AI Headset by ITMO Startup Lands Top-25 Spot at Sber500 IT Accelerator

We discussed the cutting-edge AR technology, the role of cognitive sciences in its development, and the current demands of the oil and gas industry with the startup’s cofounders, leading researchers at ITMO’s Research and Educational Center for Photonics and Optoinformatics and ITMO Fellows – Arseny Alexeev and Evgeny Alexeev.

How did you come up with the idea for the headset? Was your intention to improve on an existing technology or to respond to the needs of the industry?

Arseny: First, we identified the needs of the Russian oil and gas industry. Our country has a vast territory, and deposits are mainly located in hard-to-reach areas. Specialists at these locations often lack the technical expertise needed to effectively carry out engineering tasks. Similar devices already exist in the world, but they are not as necessary in other countries. Our goal is to ensure that the headset operates across a wide temperature range, in harsh climatic conditions, and transmits information using a local network.

Could you explain how the headset works and which problems it targets?

Evgeny: The product most similar to ours is a Canadian device by RealWear. It’s an augmented reality device designed to display instructions and enable remote communication with experts, but it features an opaque display at the periphery of the user's vision. In contrast, we aim to optimize information output as much as possible, which is why a key element of our headset is a transparent augmented-reality display in the form of goggles. Instructions are projected onto real objects.

Arseny: This display is the innovative component at the core of the entire device. Compared to the devices made by other companies, with our technology, it’s possible to produce flat, thin, bright, and sharp displays while making them widely accessible.

Evgeny: Another important element that we want to introduce into the headset is artificial intelligence. Engineers have to work in harsh conditions – and sometimes under stress, in the case of malfunctions or accidents. AI will adapt the headset to the user’s behavior. For instance, if the user makes a mistake, you need to stop them, but without scaring them or hindering the necessary actions.

Arseny: Different people perceive information differently. Some favor audio, some – visuals, and some – kinesthetics. In order to optimize our device for every employee, we need the AI to collect information about their behavior and then set up the headset based on this information.

Evgeny: In the case of AR devices, such research is in its infancy worldwide. There are numerous challenges that developers face here. The most well-known issue is the vergence-accommodation conflict, which refers to the contradiction between how your eyes look at an object and where they focus. The effect of complete immersion in a three-dimensional digital world can lead to fatigue, as this is an unnatural way for the brain to perceive 3D images. Reducing this fatigue when using the goggles is another challenge we are addressing with the help of cognitive science.

How do users communicate with experts?

Arseny: We have two assistants. One of them is an AI, similar to GigaChat, Alice, and any other AI that can be interacted with using voice or gestures. The headset’s camera recognizes your movements and understands when you need to call the virtual assistant. It is then visualized, allowing users to interact with AI in the way that is familiar to anyone these days. The difference is that AI is typically trained on an open knowledge base from publicly available sources, while our virtual assistant will be trained on closed corporate data to provide users with specific and relevant information.

The second assistant is a real person. If you find that the data from the virtual assistant is insufficient, you can connect via video with a remote expert. This person has extensive experience and is sitting in an office, watching the images from the headset’s camera. They can display instructions on real objects. For example, if there is a valve in front of you, it is necessary to highlight a specific area to show where the problem lies. The expert sees the image on their screen, outlines it with a tool or symbol, and the engineer sees not just an image, but a digital hint pinned to the object. Even when moving around the valve, you will see the exact hint in the right place.

Why did you focus your research on the oil and gas industry?

Arseny: A very important aspect of developing such devices is having an application strategy. In general, the industries that could utilize such a headset are diverse, ranging from railways and construction to medicine. However, the oil and gas sector is the most promising because, in Russia, where it is one of the key industries, there are major companies that need our technology.

Evgeny: Another important point is the cost of downtime. In other industries, the time spent waiting for a specialist to arrive is not as critical. However, the cost of downtime for oil rigs is so enormous that our device can truly provide significant benefits. For comparison, the cost of developing our headset is comparable to just a couple of days of inactivity at a drilling station.

You have an interdisciplinary team that even includes experts in cognitive science. How did you find each other and start working together?

Arseny: Evgeny and I came to ITMO through the ITMO Fellowship program fairly recently. I am the CEO, while Evgeny is the CTO. We both have backgrounds in nanophotonics and nanoelectronics, and in recent years we’ve been focusing on optics. I specialize in the manufacturing of commercial products and integration of new technologies into production.

Evgeny: My work deals with the optical part of the device and the technology for its quality testing and assessment. Almost always, it is the case that at early stages the results don’t match the expectations. My task is to understand why.

Arseny: When we joined ITMO, we started expanding our network. Thus, our team welcomed Sergey Stafeev, DSc, a professor of applied optics at ITMO. He introduced us to Tatiana Chernigovskaya, a renowned Russian scientist and a professor of neurosciences of the Institute of Cognitive Sciences at St. Petersburg State University. We were developing the headset as engineering experts, but we were lacking the expertise to evaluate the interaction of users with our device. Dr. Chernigovskaya grew interested in AR as a new way of perceiving information and how it can affect human behavior and cognitive functions. Another member of our team is Dmitry Moskalyov, a specialist in AI and machine learning and an IT entrepreneur. Thanks to him, our device will get an AI assistant and learn to detect a user’s cognitive profile. 

How did you learn about Sber500?

Arseny: From publicly available sources. In my view, this is the largest and most intensive accelerator in our country. We submitted an application and passed the first stage. This year, there were over 1,500 teams, and only 25 made it to the finals. Our device was selected due to its significant DeepTech component, which we are very proud of.

Evgeny: We were even told that we were playing at an “extra hard” level. DeepTech projects have a tendency to require large investments and longer realization timeframes.

Arseny: In the second stage, we entered the acceleration program. Sberbank organized a comprehensive series of training sessions, lectures, and workshops, which determined the finalists – the top 25 – and we made it onto the list. The third stage was similar in format, but even more intensive. During this stage, we were preparing for the final Demo Day; we received assistance in preparing a business model for our project presentation, developing an investment search strategy, and partner outreach. This provided a significant incentive for the teams to work on important aspects for the commercial implementation of the product, in addition to the technology itself.

How did your presentation go during Demo Day?

Evgeny: The final stretch of our participation in the accelerator was very intense: we worked very hard to bring our technology to the state that we liked. One of the bright positive experiences was meeting other finalists. It’s very inspiring to see how many ambitious talents there are in our country; and we were happy to find ourselves among them.

Arseny: I also want to note the work of mentors and the accelerator’s team. They truly helped us present our project in the best light. The jury featured representatives of venture funds, major Russian companies, and government organizations. We really value their feedback and comments.

Who was interested in your startup?

Evgeny: Three venture funds that specialize in DeepTech projects. We are glad to be able to promote such a project and receive feedback from investors.

Arseny: Today, we are just happy that our presentation went well, but tomorrow we will get back to work and start working out our next steps. 

How are you planning to improve the headset? 

Arseny: Currently, the headset is an MVP. After showcasing all its current capabilities, we will work on developing the project into a fully commercial product. We already have the key technology in place. Now, we need to implement the AI- and cognitive behavior-related components and build the optimal architecture of the device around them. Focusing on a specific industry will help us use resources efficiently and create the best product specifically for workers in the oil and gas industry.

Evgeny: It turns out that it’s rather challenging to make a universally applicable device due to different requirements in different sectors. Naturally, we hope that companies from other fields will grow interested in our headset – and then we will adapt the device for their needs.

Does that mean that it will be adaptable for use in specific fields?

Arseny: In the future, yes. But we believe it crucial to first make the best possible device for one field, and only then expand into others.