Participants of It's Your Call! Forum Develop Algorithm for Assessing Robot Performance

This year’s educational forum featured a case championship in which 120 students (15 teams) attempted to solve a robotics case created by the staff of Sberbank’s robotics center in one day. At the end of the event, the developers of the best solutions received gifts from the forum’s general partner. 

As part of the task, the participants tried on the roles of developers by designing software that can evaluate how well a robot is carrying out a test task. To that end, each team was given a text description of their robot's task (e.g., customer service, product assembly, or research activities) along with details about its environment and surroundings. The participants also received outlines of the best possible outcomes to each task, as well as a model-predicted action plan for their robot. The developed projects were measured against one key criteria, accuracy value – that is, the model's ability to distinguish between successful and unsuccessful plans. 

As a result, the teams produced models that can predict whether the task will be completed successfully based on the suggested course of action. These models can aid in training robots to comprehend commands in natural language, which will improve communication between humans and robots, enabling the latter to be more independent and adapt to different scenarios. If so, people will be able to send a task, be it “iron a shirt” or “receive a delivery,” to their robot assistant, instead of creating a new algorithm each time.

“These days, text models, which are likewise used for action planning, are becoming increasingly common. If we want these models to produce efficient plans for robots, we need to learn how to evaluate them, and here, our task helps assess the accuracy of forecasting the robot's action plan. When reviewing the students’ projects, we took into account the quality of their solutions, as well as their presentation and design. Though not all students were adept at machine learning, they were able to catch up and produce worthy solutions,” said Georgiy Mkrtchyan, the chief development engineer at Sberbank’s robotics center.

The best solution was presented by Daniil Uskov’s team, which included seven students from the National Research Nuclear University MEPhI, the National University of Science and Technology (MISiS), St. Petersburg State University, Lipetsk State Technical University, and others. Their algorithm demonstrated an accuracy value of 70%. 

“We combined multiple methods in our solution, and I believe this is what helped us to win. We looked into an algorithmic and empirical approach to machine learning, then tested several machine learning strategies, and, still, decided to turn to language models like ChatGPT. Few of us were well familiar with machine learning, so we had to learn it on the go. Our charm and presentation skills also were a big help,” shared Daniil Uskov, the captain of the winning team and a student at the National Research Nuclear University MEPhI.