For IT students, graduation projects contain two parts: the program code and a lengthy theoretical description. However, due to errors, hurry, or changes in project architecture, the description of a program’s functions can sometimes be at odds with its implementation. At the defense, the committee receives an unfinished paper and the student may hear questions they didn’t prepare for.

This is a common problem not only for universities, but for science as well: researchers publish papers, but the attached code can be outdated or non-functional. This is where the new educational project had started.

A team from ITMO University adapted their earlier solution, OSA (Open Source Advisor), to meet student needs. Originally, the tool was used to verify that scientific articles match the attached repositories. The researchers realized that scientific papers are not unlike graduation theses – and that’s how the version for educational purposes, OSA.Edu, came about.

This isn’t an antiplagiarism or assessment system. The tool functions like an assistant, comparing the text description and the project’s code and identifying any discrepancies between them. According to the developers, the main stand-out feature of the tool is the depth of analysis: the majority of its counterparts check only code formatting or licensing, while OSA.Edu performs a complex evaluation.

First of all, the system verifies the quality of the repository and checks whether the code is prepared for use by others: whether it includes key project information, tests, and other good practices for open repositories.

Second, it evaluates the correspondence between the text and the code as a percentage. OSA.Edu extracts concrete statements from the text (e.g., “the system implements an XYZ algorithm”) and looks for their implementation in the code. The index of correspondence shows how many of the described functions are actually working. According to the team, a number higher than 70% can be considered very good, taking into account the model’s possible mistakes. 

The tool is based on LLMs and agent architecture, which enables it to simultaneously maintain in its “memory” dozens of text pages and the directory structure with files.

OSA.Edu functionality

For students, the tool offers a quick pre-defense verification of their theses. Students need to upload the text part of their diploma and add a repository link. OSA then analyzes the materials and produces a report with the percentage of correspondence (similar to antiplagiarism tools). This report can then be used to fix any identified mistakes.

For lecturers, a big mismatch between the code and its description is a red flag. At the defense, the committee physically cannot read a full thesis and verify the attached code in the allotted 15 minutes. OSA.Edu can help lecturers see if there are any considerable internal contradictions and then ask students targeted questions. The final decision is still made by humans; the AI only points out the potential weak points.

Pilot run results and student feedback

Around 80 Master’s graduation papers from ITMO’s Artificial Intelligence Technologies Faculty were tested during the service’s pilot run. The results were revealing: the correspondence index was 0-10% for unfinished projects and 80-90% – for those with neatly structured repositories. Each student received the report produced by the app that they used to polish their theses.

“We could, of course, use the service as a strict inspector, but we believe in a different approach. It could be that a student was thinking about an older architecture or just forgot to update their text. Our task is to automatically detect such discrepancies and give students the chance to fix them. It’s an assistant, not a judge,” says Andrey Getmanov, the head of the project, a PhD student, and a junior researcher at ITMO’s Institute of Artificial Intelligence.

The developer team themselves benefitted from the tool. OSA.Edu was developed in the AIST (Artificial Intelligence Science & Technology) laboratory of the Artificial Intelligence Technologies Faculty. Its team includes both experienced faculty and students, including first-years. For some students, the project counted towards their research activities, while for others, it was an actual job with a salary and the experience of developing a product with commercial potential.

What’s next

OSA.Edu is currently in the approbation stage. In the near future, it will be tested on PhD dissertations, where there are stricter requirements for logic and correspondence. The developers expect to get more detailed comments on how to improve their system’s functionality.

Next, the solution can become a standard tool for those working with complex technical documentation: from students to researchers. The project’s main ambition is to make the description match the code and thus increase the quality of research and development.