We sat down with the head of the lab, Dr. Sergey Kolyubin, to talk about the research done here and how aspiring roboticists can join in.

Path to robotics

I enrolled at ITMO in 2004 to study automation and control. In six years, I wrote and defended two graduation papers. At that time, Dr. Alexey Bobtsov, then a young researcher himself, was assembling his team and invited me in. I wrote my first papers on control systems. But at that time, it was hard to make a living as a scientist – the salaries were low and we did most of our work outside university.

Right after graduating from my Master’s program in 2010, I decided to work in science full time and got into a joint project by ITMO and General Motors (GM) on adaptive fuel injection control in automotive engines. A part of the work was done at a GM research center in the US, where I traveled as an invited researcher. After my first year of work there, I brought over to ITMO a new project on the growingly popular topic of robotics. In 2011, I joined the company’s joint research project with NASA on sensor information fusion and computer vision for the humanoid robot Robonaut 2.

In 2012, I was invited to the Norwegian University of Science and Technology, which was launching a brand new industrial robotics laboratory. There, I spent three and a half years doing research in movement planning and control, as well as calibration and force/torque control of manipulators. When I was faced with the choice of either staying on another short-term contract or coming back home to launch my own lab at ITMO’s School of Computer Technologies and Control, the choice was obvious. So, in 2016 I returned to St. Petersburg and in early 2017, with the team of the Center for Nonlinear and Adaptive Systems, defended an application for the Project 5-100 program and launched a new international laboratory. It will turn 10 years old in 2027.

Read also:

PhD in Robotics: Alexey Vedyakov Talks Robot Control and Industrial Path for PhD Students

How the team works

The Laboratory of Embodied Intelligence in Robotics consists of several research teams that share the same space, professional equipment, and robots. We have industrial manipulators, mobile platforms, quadcopters, test ranges, servers, and access to a pilot production facility at the faculty, where we can quickly manufacture mechatronic module prototypes. The laboratory employs algorithm developers, applied programmers, computer vision engineers and ML engineers, designers, and electronics specialists. Each group works on its own area of expertise. The staff focuses on tasks that require a combination of computer modeling, mechanism design, vision, motion planning, and control. We teach robots of different types, from manipulators to underwater robots, to understand what they see, plan actions, make decisions, and carry them out, operating reliably and autonomously in dynamic environments without special onboard sensors.

When businesses fail to find solutions on the market, they come to us for advanced development; we assemble representatives of different research groups for tasks that combine their complimenting competencies. Thus, we can take on complex projects and ensure results of higher technological readiness. Often, we discover new useful interdisciplinary applications for ourselves, as well.

These days, the main demand is for applied engineering solutions – they ensure our financial stability. But we are, of course, primarily a research laboratory: we need to solve not just today’s problems, but also think of tomorrow’s ones so as to stay relevant and never miss a breakthrough. We combine applied engineering work with frontier research and this is reflected in the structure of the lab, which has ties to two departments: the Faculty of Control Systems and Robotics and the recently opened Institute of Mathematics.

The main focus of our frontier research is embodied intelligence. We are developing algorithms that are integrated into hardware and take into account how a robot functions in real time and the real world, not in ideal environments. That’s why we cover the entire chain: from designing a “smart body” to the cognitive layer, its “brain.”

The keywords defining our research field are robustness, physical coherence, multimodality, semantics, adaptability, and energy efficiency. In robotics, it’s important for neural networks to account for a robot’s limitations and avoid contradictions to the global physical context, the laws of nature; it’s not enough to produce statistically viable results. This year, Sergey Zagoruyko, a world-renowned researcher, has joined our lab to help bring such projects to life. 

Among the results of such studies are publications at major robotics conferences ICRA and IROS. Recently, we have also set ourselves the task of expanding our horizons and publishing at other A* conferences, including CVPR, AAAI, ICML, and ECCV. ITMO supports these ambitions by funding projects in the field of AI. I believe this isn’t just “working for your reputation” – it’s a smart investment. By publishing articles at this level, we demonstrate the high-class competency of our research teams. We’re showing how an idea has led to an experiment, with clear results and extensive testing and drawing comparisons with the best counterparts. Such papers help us find industrial clients. We need to demonstrate our level with something more than beautiful presentations.

We also contribute to open-source projects. ITMO is an active participant of such initiatives via the ITMO OpenSource community. Nearly all of our findings that aren’t reserved for our clients are instead published on our GitHub.

As for the topics of our projects, they emerge in different ways: some are suggested by our students, some come from the industry, and some are aimed at trying to solve the “perpetual” problems on a new mathematical basis. The main requirement is that the idea is connected to relevant problems and has a clear expected effect.

Teaching a robot to see and understand

One of the problems we are trying to solve is teaching a robot to understand what they see, navigate its environment, and complete commands issued in natural language. Several of our projects are tackling different sides of this problem.

For instance, one of our latest frameworks, AgentGrounder, helps robots find objects based on a text query. Imagine a robot entering a room for the first time and receiving a command: “Find a blue mug on the desk by the window.” To complete the task, the robot looks around and makes a list of all the objects it sees. When it receives the command, the system separates it into semantic parts, compares it with the list, and identifies the needed object. The main advantage here is that everything happens with no additional tweaking for a specific room or object. We use large visual language models that connect words to pictures, and add geometric thinking so that the robot can understand where to look and how far the object is. The entire system is integrated through agent pipelines that make it possible to flexibly configure a chain of information transformations, call upon different tools, and perform intermediate validation of result quality. Such algorithms will be useful in robotics, augmented reality, virtual reality, and 3D graphics software.

Another recent solution is R5DGS, a 4D dynamic environment reconstruction system. The system augments the 3D image captured by a regular camera with a time axis: the algorithm calculates where moving objects will end up in the next moment. At the same time, the algorithm doesn’t split the image into thousands of small pieces; instead, it works with an object as a whole by tracking its movements and predicting where it will be next. The approach speeds up computations by about 11 frames per second – for systems like these, that is a major result that makes it possible to work in real time without lag or stuttering. The motion remains natural and error-free. In addition, the system can identify the required object from a text query and display it from the specified viewpoint. These algorithms are useful for precise computer simulations – for example, in autonomous vehicles or augmented reality.

Projects for businesses

There are two types of work that we do for industrial partners.

The first one is initiative projects.

Some of them are implemented with Sberbank; the company supports primarily the projects developed by students of our joint Master’s program Robotics and Artificial Intelligence. Sberbank doesn’t reserve the rights to the results: we publish them in open-access journals and these projects often make it to the world’s leading A* conferences.

The second type are projects where a client sets a business task for researchers. Here are some recent examples:

In 2025, we developed a robot chemist for Sberbank. It’s a prototype of a cell for an autonomous laboratory, made for a target scenario of automated assembly for CR2032 lithium batteries. These are flat, round batteries used in motherboards, remote controls, and watches. The station’s task is to ensure that the robot, without human involvement, picks up miniature parts, measures out the required amount of electrolyte (the conductive liquid inside the battery), and hermetically seals the casing. It is important to achieve high precision from the robot, because even the slightest inaccuracy in picking up the parts or dispensing the electrolyte will render the battery unusable. But the goal was not just to solve one specific case; it was to create a system that could be scaled to other scenarios and handle a wide range of similar tasks. We worked on this project with AIRI and presented it at the AI Journey conference in 2025.

Together with Sberbank’s Robotics Center, we are also developing a system that teaches robots to manage objects using sight and touch at the same time. This is a continuation of their widely-known model that was created by a team featuring ITMO graduates and PhD students who are now working at the company. Apart from its practical value, the project is also important for science: it helps answer fundamental questions about how models link signals from different types of sensors to actions, reproducing fairly complex manipulation skills.

Another project that we are working on with other ITMO teams is a roboticized kitchen for VkusVill: it’s a system that replaces cooks on the “last mile” of working with prepacked and ready meals before delivering them to clients. Our responsibility is a computer vision module – the eyes and spatial memory of the robot. We are also developing an integrated robotic cell that autonomously assembles orders of fruit and vegetables of a given weight, while at the same time gently handling delicate objects. This requires a combination of design solutions, a system for processing sensory information and computer vision, and a system for movement planning and control.

How much you can make as a scientist

Anyone who joins the team is assigned a specific task and immediately joins the development. All lab members are officially employed for the term of a specific project and later the contracts are renewed. The core of the team remains the same and, in time, many combine their roles in several projects. As junior researchers, Master’s students receive around 40-50 thousand rubles a month doing exploratory research. When a researcher advances to working on tech projects, this sum triples; when they lead a team or are responsible for a key module, their salary grows to 200-250 thousand rubles a month.

Working at the lab is an opportunity to conduct truly frontier research and grow professionally in a good environment. As a bonus, Master’s and PhD students use their work at the lab when writing their theses; this way, they don’t have to struggle to make time for studies, as is often the case when they work elsewhere. What’s more, this helps them complete mandatory internships, provides source material for term papers, and opens the possibility to intern at our partners, such as Sberbank’s Robotics Center.

If upon graduation, our students decide to enter the market, we help with that, too. For instance, the aforementioned Sberbank makes special offers for our graduates, while a portfolio of successful cases and top publications will give you a boost at any interview.

How to join the team

For those looking to join the lab, twice a year (in September-October and January-February) we open applications and announce it on our channel and VK page (there, you can also find updates on the lab’s projects). If you’d like to join the team, email me at s.kolyubin@itmo.ru, attach your resume and a small cover letter. Project experience will be a bonus, but your willingness to learn and grow is more important.

The next step is a test assignment that helps us check the necessary skills and knowledge and align our expectations with those of the candidates. For example, the last time we deliberately designed the test assignments so that they could only be solved by efficiently using generative AI tools. What matters to us is seeing how the applicant thinks, what methods they use to solve a task, and how consciously they use technology. In fact, with a properly designed assignment, using a chat bot will not mask a lack of knowledge or experience, but will reveal it instead. The same applies to a lack of systematic and critical thinking.

After they’ve completed the test assignment, we invite candidates in for an interview. There, we make the final decision on which team within the lab a successful candidate will join and what tasks they will perform.

We welcome PhD and Master’s students and even those getting their Bachelor’s degree – for them, it’s a great chance to work with lab equipment and join serious research projects. Master’s students can conduct their own research and published papers, including in collaboration with experienced scientists, while our PhD students prepare A* publications and write their theses alongside their work. After graduation, many stay on to lead our research groups.