ITMO and Friends of St. Petersburg Project Restore Lost Paintings with AI

Designed by architect Yury Felten, the Zubov Wing was added to the Catherine Palace in 1779-1785. During WWII, its mid-19th century interiors were damaged in a fire; out of the 103 paintings that decorated the room of Empress Catherine II, only ten remain.

Neural networks will be used to help restore the damaged panels, all as part of a workshop founded by the social project Friends of St. Petersburg (Gazprom) in collaboration with Tsarskoe Selo Museum and ITMO University.

A total of 37 students of all levels joined the project; they represented several local universities: ITMO, St. Petersburg State Forestry University, Peter the Great St. Petersburg Polytechnic University, St. Petersburg Stieglitz State Academy of Art and Design, St. Petersburg State University, St. Petersburg State Institute of Culture, St. Petersburg Mining University, St. Petersburg State University of Aerospace Instrumentation, University 2035, and the NTI Center of Educational Competencies. Students got to apply their expertise in various fields, including applied computer science, innovation studies, restoration, product design, information systems and technologies, architecture, preservation of cultural heritage objects in landscape architecture, AI engineering, oil painting restoration, organization system management, and communication design.

ITMO was also represented by the project’s curators: Anastasia Laushkina, an assistant at the Artificial Intelligence Technologies Faculty, and Mikhail Sinko and Valeria Volokha, both engineers at the National Center for Cognitive Research. Together, the experts organized seminars for students where they talked about using AI in art, their experience of image restoration with neural networks, various approaches to image generation and the resulting defects, as well as the ethics and evolution of AI.

Grouped into five teams, the participants restored 63 images based on short descriptions of their subjects found in historical documents. In their work, they used Stable Diffusion, DALL-E, Flux, Sora, as well as DataSphere, the ML service from Yandex.

Experts from the Tsarskoe Selo Museum have selected 10 artworks that were the most accurate in style and historic aspects. They will be improved upon and used by artists to create new panels for the Zubov Wing. Additionally, the organizers will create a digital 3D copy of the room.

“The importance of our project is two-fold: first, it helps unravel the fears that AI will replace human professionals and thus demonstrates that AI can offer new ideas as a creative assistant. Second, the workshop showed students from different fields how important each of them is, as image restoration involves a multitude of stages, including going through historical records and similar works, selecting materials, and securing expert approval at each stage,” shared Anastasia Laushkina.