European Games in Minsk
The European Games is an international multi-sport event contested by athletes from European nations. Organized once in four years by the European Olympic Committees (EOC), this time the games took place in the capital of Belarus, Minsk. As is the custom with the Olympic Games, which inspired the European Games format, the program of the latter included festive opening and closing ceremonies. These featured a variety of elements from sportspeople march, musical performances and performances bringing to life the nation’s folklore. What made the show special, though, is that it was enhanced by augmented reality representations of bison, fireflies, the dragon Zmok, which is one of the key figures in Belarusian mythology, and some other objects. All in all, the opening and closing ceremonies program incorporate some ten augmented reality episodes, some of which were included in the live TV broadcast, while others were an exclusive feature for spectators watching the games on the stadium.
You can see the AR in action in the recordings of the ceremonies (below is a video from the opening one) and on the official app which will be discussed below. Short videos shot by the spectators attending the shows are available here, here and here.
The idea of using augmented reality objects in shows is not a new one. The technology has been used in live broadcasts of cybersports competitions on Dota2 and League of Legends, while just last year, Eminem launched an app that offers visitors of his concerts a new immersion effect. The defining feature of the AR technology used at the European Games is in its comprehensive approach: the AR elements could be seen by all spectators, whether they watched a live TV broadcast or were present at the stadium. The latter category could experience the objects either through the information screens set up at the stadium or download the app Opening Ceremony AR Experience before the show began. Some episodes, for example, one with the dragon Zmok and bison, are still available for viewing. All you need to access those is to download the app from App Store or Google Play.
ITMO University’s Laboratory of Interactive Visualization role in the organization of the event
All these spectacular AR effects were developed by the specialists at ITMO University’s Laboratory of Interactive Visualization. The staff created both the mobile app and the whole complex of apps for introducing augmented reality in the TV broadcast. As noted by Andrey Karsakov, head of the laboratory and the Master’s program “Game Development Technologies”, it was the company Sechenov.com, which organized and implemented the opening and closing ceremony, that first became interested in introducing augmented reality to the show. A meeting with the laboratory’s staff paved the way for the collaboration on the program, which was decided to revolve around comprehensive immersion into augmented reality.
“Never before had we done a show that would include elements of augmented reality, and we love to explore new solutions and take on new challenges,” shares Evgeny Timashchuk, executive producer at Sechenov.com. “We try to constantly be on the look-out for new trends and are always searching for ways to make the show even more interesting and exciting. The technological progress and the widespread implementation of technologies into our daily lives are setting the agenda. We have nothing but positive things to say about our collaboration with ITMO University’s Laboratory of Interactive Visualization, which we plan to continue going forward. As for AR, I believe that the next step of the technology’s development would be in the direction of the interaction of spectators with the augmented reality objects via the screens of their devices, which will give them an opportunity to influence what is happening in the show.”
Concept of the ceremonies
The opening ceremony of the 2019 European Games was dedicated to the history of Belarus, its traditions and prominent figures. For example, the part when the country’s flag was carried out to the stadium was enhanced with an AR flock of storks. Two of the birds, which constitute an important symbol of Belarus, even landed into the nest set up above the central stage, meeting the children who carried the flag. During the Legend episode, which served as a representation of the country’s folklore, the narration of the story of the dragon Zmok included the creature flying over to the stadium and breathing fire on the spectators. In the story about Belarus craftspeople, replicas of drawings by the artist Léon Bakst, who was born in the Belarusian city of Grodno, ascended from the stadium to the skies. In its turn, the episode titled Białowieża Forest had a herd of bison, another of Belarusian symbols, racing through the sky under the rhythmic beating of drums.
The closing ceremony, which was dedicated to modern Belarus, included only one episode using augmented reality technologies, but it was as impressive and important as ever. A couple of seconds before the flame of the games was to die out, a fern flower bloomed in the flame cauldron, discharging a flow of seeds that spread all over the stadium as if bidding the spectators goodbye. Every smartphone user could reach out to the magical, lucent seeds floating around them. The episode finished with the magical composition soaring to the skies under the accompaniment of the song Time To Say Goodbye by Oxana Volkova and Nikolay Baskov.
The script, production and aspects of graphical content fell into the remit of Sechenov.com, while ITMO University specialists were responsible for the entire software implementation of the former’s ideas and technical support during the shows. All in all, the preparations for the ceremonies took the team seven months. For the most part, the collaboration was a remote one: ITMO specialists carried out the development process in St. Petersburg and flew to Minsk two weeks before the start of the game to participate in the setting up of the stage and rehearsals on the Dynamo stadium in Minsk.
Challenges of the implementation process
In the course of their work, the staff of the Laboratory of Interactive Visualization faced a number of difficulties. One of the most serious ones was connected to ensuring precise time synchronization of AR episodes with live action on the stadium. While the broadcast version allowed for pre-setting a control time code, this wouldn’t have worked in the infrastructure of the mobile app: the stadium was attended by over 20,000 spectators, and the number of app users bumped up to 30,000 with volunteers, athletes and organizers taken into consideration. Implementing the solution via the internet would have been made problematic by the potentially huge load on the mobile network and steep costs of renting the equipment that would have been powerful enough to withstand it.
The staff of the laboratory tackled the problem in an original way by enabling the app to operate with no internet connection. This was made possible as a result of a collaboration with the company Cifrasoft from Izhevsk, which develops solutions on the synchronization of mobile devices based on an audiotrack, so that a phone would hear an audiotrack and understand its timing. Though the members of the team initially were apprehensive that loud sounds on the stadium could lead to possible glitches occurring, the results of first tests alleviated these fears: the technology Cifrasoft SDK demonstrated stable operation even in the case of loud ambient sounds.
Another issue was connected to light, which always requires special attention when working with augmented reality objects because of its influence on the perception of the picture at large. As the opening ceremony was set to begin even before sunset, the lighting had to undergo dynamic and significant changes throughout the first quarter of the event. Moreover, all main activities and performances were accompanied by an extremely nuanced dynamic lighting from more than 100 light sources set up over the whole perimeter of the stadium. Because of all this, organically merging the images with the real light was a big challenge for the developers.
AR for emotions
Andrey Karsakov believes that in the future, we will witness many other interesting projects with comprehensive use of augmented reality, including those by ITMO University’s Laboratory of Interactive Visualization. The specialist admitted that the talks on the next big project are already ongoing, but he can’t divulge any details.
“In many ways, applying AR in large-scale open-air shows is still very much an experiment. The real cases that have used it are few and far between. The work process now is still very raw, which is even more the case with integrating AR into mobile apps and live broadcasting. That is why, with some of the occurring issues, we had to solve them with a make-do-and-mend approach. On the other hand, we are very glad that we had the chance of working on such a large-scale project and as part of such a massive team of professionals. Our work on the European Games provided us with the understanding of which directions we should be developing towards, what aspects could be refined and how to improve the process of developing this type of content. I think that AR will become more and more widespread on live events because it really packs a punch effect-wise and certainly makes the show more memorable. I hope that in the future, the technology will also become more interactive, to ensure that the spectators get as positive and exciting experience as possible,” says Andrey Karsakov.
