Nowadays, ChatGPT is undoubtedly one of the strongest examples of artificial intelligence systems and we are willing to ignore alternatives, but is it fair? The goal of this article is to analyze the YandexGPT’s potential and decide if it is competitive or not yet.

Image generated by YandexGPT

What is YandexGPT?

YandexGPT is a neural network by Yandex that is designed to create chatbots and smart assistants, and to generate, structure, and summarize textual information. For the first time, the model appeared in February 2023, when a team was created to train a generative system using the GPT-3-based language model called YaLM 2.0. Currently, at the beginning of 2024, the model is only used in the virtual assistant Alice by Yandex, but in the future, the company plans to use YandexGPT to develop search technologies, email, and other company products.

Is YandexGPT good at programming?

There are many different areas in which GPT models can be helpful. I have chosen to write code in Python to analyze the potential of YandexGPT. To assess the neural network's knowledge in this field, I tested it using a basic Python knowledge test (https://kursy.guru/test/python). It includes 37 questions covering various aspects of the language. The neural network answered 26 out of 37 questions correctly, earning it an average level of proficiency. As for ChatGPT, it scored 30 out of 37 points and got an advanced level, but didn’t go far. YandexGPT’s results are pretty good, so let’s test its skills in practice.

To see the neural network's coding abilities, I created tasks of various difficulty levels. Having analyzed the code generated by YandexGPT, I can conclude that the model handled all test cases of different difficulty levels quite well, with minor shortcomings. A significant advantage of the model was the ability to find optimal solution paths using built-in features of the Python language. It is important to note that the neural network cannot identify errors in the code. Unlike it is with ChatGPT, when I pointed out the inaccuracies, YandexGPT simply rewrote the code, often leaving the previous inaccuracies of the same nature unchanged. It also struggled with identifying errors in existing code provided by the user, failing to understand the question.

Is YandexGPT good at communicating?

Now, it is obvious that the YandexGPT model can write good code, but it is reasonable to ask if the model is convenient to communicate with. I doubt it. For me, interaction with the model was not easy because it cannot engage in a conversation. In contrast to ChatGPT, each response does not take into account the previous content and the overall context of the conversation. This is a serious disadvantage in cases where clarification of the question is required. When I asked long questions, the neural network had difficulties understanding the information. This can become a significant limitation in cases where clarification of the question or discussion of complex topics is needed.

Final thoughts about YandexGPT

In my opinion, Yandex’s model has great potential in writing code. If you need some examples of programs to work with, YandexGPT is a good source to ask. However, the model requires a lot of conversational improvements because now it is hard to talk to. Would I choose YandexGPT as a replacement for ChatGPT? Not now. This Russian alternative still needs further development, but I am optimistic about it. It's worth revisiting YandexGPT in a year and assessing the progress. We’ll see!