Contents

  1. What is AR?
  2. AR devices: from camera lucida to Spectacles
  3. How AR works
  4. Components of AR devices
  5. AR applications
  6. AR challenges on the agenda

What is AR?

That’s a very flexible term that can mean different things. First, it’s important to distinguish AR from other similarly named technologies.

For example, in contrast to VR (virtual reality), the aim of AR is not to immerse users into a different world, but to “transfer” digital objects into the real world. There is also MR (mixed reality), which is either a type of or the next stage of AR: such technologies don’t only “bring” digital objects into the real world, but also enable us to interact with them.

AR has three distinct properties:

  • Unrestricted transfer of digital objects into the real world. Projectors also transfer digital objects (presentations, for example) into the real world, but there are limitations – for instance, the projection can only be displayed on a flat, even surface. With AR, digital objects are fully visible in any conditions.
  • Links to the real-life location of objects. If we move the projector, the image moves with it. However, with an AR device we can attach the image to a specific point in space and it will remain there even if the observer changes their position.
  • Interaction with digital objects. With AR, such objects can be touched or moved.

AR devices: from camera lucida to Spectacles

The history of AR began much earlier than we tend to think. The first device that performed the function of transferring an object from the “virtual” world into the real one was devised by the German mathematician and astronomer Johannes Kepler in 1601. However, there is no confirmation that the scientist managed to create a physical prototype. It was only in 1807 that the English physicist William Wollaston patented an invention that can be considered a physical embodiment of the idea of augmented reality. The camera lucida is an optical device for transferring images onto paper, similar to a camera obscura. It was most commonly used to train artists.

Devices that were somewhat closer to modern ones appeared in the mid-1950s, when engineers had developed aircraft windshields that provided various metrics for pilots. However, such windshields were impractical due to their large size and complex interface. Additionally, pilots could not simultaneously focus on the data on the screen and the runway, which was not only inconvenient but also dangerous.

The age of modern augmented reality devices began in 2012. At that time, the industry experienced its first wave of miniaturization: developers aimed to create not only understandable interfaces, but also compact devices similar in size to regular sunglasses. The most popular examples here were Google Glass and R-7 by ODG. However, they were inconvenient to use; developers had to choose between a large display on the glasses’ lenses and their compact size. This stage concluded in 2023 when the industry finally accepted that traditional engineering approaches were ineffective for AR devices – instead, AR called for innovative optical technologies.

Alongside the first wave of miniaturization, a second one began in 2015 and continues today. This period saw groundbreaking discoveries in photonics and electronics, such as image waveguides and special nanostructures that could “capture” light. In other words, the very technologies required by those designs from the past. Among the latest products developed in this field are Orion glasses and Spectacles. These devices are flat, miniature, and have a large screen with sufficiently high resolution.

HoloLens 2. Credit: microsoft.com

HoloLens 2. Credit: microsoft.com

AR devices: from camera lucida to Spectacles

Take your phone and bring it close to your face — just a couple centimetres away from the eyes — and try to read the text from the screen. It’s impossible, right? Our eyes just can’t focus on the image. This is the main challenge for AR devices. We don’t see the objects located this close and only optical “tricks” can make our eyes perceive digital objects from several centimetres away as if they are located at an arm’s length or further. At the same time, we have to be able to see other objects within our field of vision: digital and real objects have to seem to be in the same dimension, otherwise we won’t be able to focus on them simultaneously.

The scientific approach. For our eyes, any image, including digital objects, is a set of light rays emitted by or reflected from an object. The human eye determines how far an object is, among other things, based on the degree of divergence of the light beam. The larger the angle, the closer the object appears to the eye. In contrast, distant objects emit rays that are almost parallel to each other.

That’s why, in order to make digital objects visible, we have to make the rays they emit parallel to each other to trick our eyes. A lens can turn a divergent set of rays into parallel ones in a process called collimation; that’s why the displays capable of this trick are called collimating. 

The real-life application. In the first AR glasses, the lens was combined with a semi-transparent mirror and positioned directly in front of the eye. The image from a small digital screen was projected from above, hit the collimator, where the rays turned parallel, and then reached the eye. However, devices with this design tend to be bulky and complicated to use.

Later, during the first wave of miniaturization, developers began to use a more complex approach. In this setup, the image is also projected from above (for example, from a smartphone screen), then reflected off a curved mirror lying parallel to it and into a semi-transparent mirror positioned at a 45-degree angle in front of the eye, finally reflecting directly into the eye. This means that rays are reflected several times off different surfaces. As a result, it is possible to reduce the size of the mirror and the lens, and thus the device itself.

However, a true breakthrough in the development of collimating displays came with the invention of image waveguides. These are optical components that make it possible to collect, hold, and deliver rays of an image “to the user's eye” while remaining transparent to the surrounding world. In appearance, they resemble the lenses of ordinary glasses. The image from a miniature projector “falls” onto the waveguide glass and gets “trapped” there. Simultaneously, an image of the real world passes through the same glass. It combines with the already “captured” digital image, and then the merged picture reaches the user’s eye. With this technology, it is possible to create AR displays that are virtually indistinguishable from the lenses of regular glasses. Therefore, leading smart device manufacturers now use exclusively these technologies in their AR glasses.

Components of AR devices

Apart from optical elements, modern AR devices can boast complex electronics. For instance, they process information from the environment with various sensors that can detect: 

  • Your location – with a gyroscope, accelerometer, compass, GPS, and microphones;
  • Your actions – with biometric sensors, microphones, and gesture recognition;
  • Direction of your gaze – with eye-movement and lighting sensors;
  • What you see – with stereo sensors, ToF and SL cameras, LIDARs, as well as infrared and proximity sensors.

AR devices also have a reality recognition system, which is based on two data sources:

  • Object markers. For this, a 3D model of the space is created, and a special marker resembling a QR code is “applied” to some objects. The device reads information from these markers and understands how the world around it currently looks. The main drawback of this system is the need to create a 3D model of the space in advance. Therefore, this approach is mainly used in manufacturing.
  • Structured light. Here, the device itself creates a 3D model of the space in real-time. The device projects a grid of infrared points onto the surrounding environment and analyzes changes in their size and relative positions. This method of reality recognition is more suitable for personal devices. However, this approach requires greater computational power, which complicates, adds weight to, and increases the cost of the final product.

AR applications

Industry. With an AR headset, specialists located anywhere in the world can fix any given piece of technology by sending real-time instructions to a maintenance engineer on-site. It’s also possible to monitor the onsite worker’s actions and guide them.

Construction. Here, AR glasses allow engineers to see their future creations in real-size and onsite, thus noticing any errors in the building under construction and fixing them.

Medicine. One of the most popular trends here is the development of “X-ray vision” AR glasses for medical professionals – meaning that surgeons would be able to see MRI scans of patients overlaid on their bodies during operations.

Education. One of the many plausible scenarios for use of AR glasses are personalized museum tours, where every visitor has their own guide – an avatar that can be anyone, from Einstein to Aristotle.

Personal devices. With AR headsets, you can play chess “in the air” or throw a virtual basketball. Headsets can also display a planned route right onto your actual surroundings. They can also be beneficial for people with disabilities – by alerting them about nearby objects (traffic lights, people, cars) or describing the space with a voice assistant.

Credit: tinx / photogenica.ru

Credit: tinx / photogenica.ru

AR challenges on the agenda

There are still plenty of issues that remained unresolved in the field:

  • Short device lifespan. Currently, a compact device operates for only a couple of hours without additional charging, making it unsuitable for continuous use like a smartphone.
  • User myopia. People with poor vision require additional lenses, as it is not possible to combine AR glasses with glasses for nearsightedness.
  • Occlusion. Adding a digital object alongside real-world objects is relatively easy. But what if you need to remove a part of this digital object from the real world? Or at least modify it – such as making it match the color of a building? There are currently no solutions for how to achieve this with transparent AR glasses.
  • Privacy. AR glasses can continuously record and save all user actions, and third parties, including criminals, may gain access to this data. Therefore, reliable methods for data protection in this area still need to be developed.

The article is based on the lecture given by Arseny Alexeev and Evgeny Alexeev, ITMO Fellows at the Research and Educational Center for Photonics and Optical IT, at Manege Central Exhibition Hall.