1. What is Augmented Reality?
Augmented Reality (AR) is an interactive 3D experience that combines your vision of our real world with computer-generated graphics of various kinds in real time.
Virtual reality (VR) and AR accomplish two very different things in two very different ways, despite their devices’ often similar designs. VR replaces your view of reality, taking you somewhere else into a computer-generated immersive 3D world. AR adds to your view of reality, projecting information and graphics on top of what you’re already seeing. They’re both powerful technologies that show a lot of promise and are only now starting to catch on in mainstream entertainment, work, and society at large.
AR has similarities to VR, but it has differences that make it its own category of experience.
Virtual reality (VR) creates a completely synthetic virtual world within a headset. As the user, you are placed inside a 3D environment and can then move around and interact with completely computer-generated elements. VR is a completely self-contained computer environment while AR layers computer elements over and amongst real worlds objects in your natural vision in real time.
This article is a brief overview introduction to AR, an introduction to some of the hardware, and some of the common applications for this emerging technology.
Unlike virtual reality, augmented reality is not limited to a wearable device and is being developed and implemented on phones and tablets in addition to AR glasses or AR headsets. Although this technology has huge potential for mass adoption, it is often less understood that virtual reality.
With new products such as Lenovo’s AR Concept Glasses and NReal Light AR smartphone glasses, users will be able to use PCs and smartphones in AR, on the road, and with visual privacy, because the small portable and wearable smart glasses also allow PC and smartphone users to access their PC-based work, social media, and games remotely.
Augmented reality can be seen in a number of ways, such as through AR glasses that combine the view of your immediate surroundings with computer graphics, or on a smartphone display that does the same thing using the phone’s camera to see and manipulate the real world in on the phones screen and overlay computer graphics on top of the camera’s view of the world.
Immersive AR systems may combine the computer elements with real-world elements with convincing depth, perspective, and other rendering characteristics, and the AR system will understand the real environment well enough to be able to position relevant synthetic elements “in front of” and “behind” real objects and otherwise be able to interact with the real world in a meaningful way.
Immersive AR systems are some of the most advanced implementations of AR, but are often very expensive and bulky, often designed for specialist use cases and marketed towards companies rather than consumers, although recently AR headsets are starting to make their way into the hands of enthusiast consumers and gamers. This technology is designed for free movement, while projecting images over whatever you look at.
AR heads up displays can offer something as simple as a data overlay that shows the time, to something as complicated as holograms floating in the middle of a room.
While many would say that the head mounted display implementation of augmented reality, such as the expensive enterprise-oriented Microsoft Hololens is at the leading edge of this technology, Smartphone camera-based AR is by far the most accessible, as billions of people around the world have a smartphone, AR has been made accessible to them through the Apple app store and google play store.
Smartphone based AR gaming really started to enter the public consciousness when in 2016 the Game Pokemon GO, from the world famous Pokemon gaming franchise, became extremely popular all over the world, and was downloaded onto many smartphones.
In Pokemon Go, the AR software projects an interactive Pokemon creature on your smartphone screen, overlayed on the background on top of whatever the camera is looking at. This marked the popularisation of a type of gaming where people would go around the real world, such as the woods or the streets with their smartphone camera pointed at the natural environment, interacting with the creatures they would see on their screen “discovered” at particular locations in the real world. It is a different idea to almost all other computer gaming where players sit in their home and play the games.
AR technology, in particular this kind of smartphone based low-cost AR experience has some distinct disadvantages compared with virtual reality: visual immersion. While VR completely covers and replaces your entire field of vision, replacing it with a fully immersive 3D computer generated world, mobile AR apps only show up on your smartphone or tablet screen, and even the advanced AR headsets such as Microsofts HoloLens can only project images in a limited area in front of your eyes. It isn’t very immersive when a hologram disappears once it moves out of a rectangle in the middle of your vision, or when you must stare at a small screen while pretending that the object on that screen is in front of you.
Basic AR that overlays simple information over what you’re looking at can function perfectly fine with 3 degrees of freedom. However, most AR applications utilise 6 degrees of freedom in some form, tracking your physical position so the software can maintain consistent positions for the images it projects in 3D space. Therefore, the HoloLens uses a stereoscopic camera and pattern recognition algorithms to determine where it is at all times, and why more advanced, AR-centric smartphones use multiple rear-facing cameras to track depth.
Augmented reality has nearly limitless possibilities for the creative imagination, and has many applications separate from just gaming and entertainment. Popular Phone-based AR software apps have been recognizing their surroundings and providing additional information about what it sees for years now, offering live translation of text in foreign languages, even pop-up reviews of restaurants in the real-world location as you point the camera at them. Dedicated AR headsets, such as the HoloLens, can do even more, letting you virtually place different apps as floating windows around you. In this sense there is the possibility of creating a large multi monitor computer set up right within the head mounted display.
Currently, AR is only widely available on smartphones, and doesn’t have the vision-augmenting aspect of enterprise-level AR displays. This means that phone AR is limited, and the industry is waiting for more high-level consumer-oriented headsets to catch on before the true power of AR can be developed for a large audience.
2. Different types of approaches to AR hardware
Heads Up Displays or HUDs
This is a transparent Head mounted display that presents data to the user in front of their eyes, hence the user does not need to look away from their desired viewpoints and look down at a screen. Additional data displayed could include routes, locations, diagrams, or chats with other device users, and even 3D images and video feeds.
Holographic Displays
Augmented Reality glasses based on this technology display 3D holograms overlaid on the real-world user’s vision, to render a mixed reality experience to the user. The hologram image is generated using light diffraction techniques. Perhaps the most notable example of this approach is the Microsoft HoloLens.
Smart Glasses
AR smart glasses are wearable computerised glasses that add extra information, ideally 3D images and information such as animations and videos, to the user’s real-world scenes by overlaying the computer-generated or digital information on the user’s real-world. It can retrieve information from computers, smartphones, or other devices and can support WiFi, Bluetooth, and GPS. This is basically just a less elaborate and more compact variation of a HUD device described above.
An example of this is Vuzix M100. This kind of AR is mainly used in an industrial setting where workers need to be given live information as they work with their hands and cannot be holding a smartphone to be constantly looking at and distracted by.
Mobile Handheld
Handheld AR is using handheld devices such as smartphones tablets or phablets (large phones) on which AR apps are installed. This kind of AR is widespread and accessible, and can be downloaded, often for free on iOS and Android devices, so long as they have a camera. compared with the AR headsets that are worn on the head, which are usually aimed at a more specialist and professional users, they are accessible, are easy to use and cheap, and have many gaming and entertainment use cases; this may have chances in the future as developers attempt to reach a broader retain market with Headsets.
Examples include using your smartphone to try out virtual models of furniture on your house floor, on the IKEA app, or playing Pokemon Go on an AR app on your smartphone.
VR/AR merger headsets
An approach to AR that is being developed is the idea that AR can be integrated into a VR headset, so the headset can have both AR and VR features at the same time. This effectively means it’s a VR headset with camera/s on the front that can show the real world within the VR headset, and then render AR content over the camera image of the real world. This is a very promising best of both worlds approach which has the potential to deliver both an AR experience and a VR headset in one convenient product. Probably the highest end version of this at the time of writing is the Varjo XR-3. This idea is sometimes referred to by Microsoft as an integrated example of MR, mixed reality, which is an umbrella term to include both AR and VR hardware and software.
These products can with be stand-alone with on board processing power and batteries within the headset or can be tether by a cable to a PC with a graphics card a power supply to render the content and supply power.
3. Applications and use cases
AR’s value is in the way it can interpret, manipulate, and enhance a view of the real world in real time; below are some ways in which AR can be used and is actively being used right now.
Retail
Perhaps one of the most practically useful applications of AR for most users will be in retail applications, such as sniffing out deals and promotions in the real world, and “try before you buy” experiences. With the pandemic having changed the world, more people than ever are shopping online from home and there is less public enthusiasm for public retail spaces. AR will allow consumers to see exactly how products will look in their own homes before purchasing them. Seeing a 2D image of a product on a website from one angle can often be a very poor representation of what the actual product is like when it arrives and is in front of your eyes. A 3D AR representation of the product can take a lot of the uncertainty and guesswork out of the buying process. There are already popular apps, such as Wayfair, Houzz, and Ikea which let you place furniture and other products in your own real-world environments. Tesla released an AR App that lets you see the tesla vehicle in AR, how it would look in your garage, before commuting to the stress of a purchase. This makes the buying process more engaging, interesting, and removed uncertainty
Mapping and navigation
The business directory app Yelp was an early adopter of AR with its now-discontinued Monocle feature. Monocle overlayed information about local businesses when viewing the immediate surrounding on your phone’s display. Pop up messages were shown where nearby restaurants were located along with basic information such as how far away, they were. This kind of service isn’t just limited to Smartphones, Mercedes-Benz, for example, is introducing a car that overlays navigation information on the wind-shield, helping you drive without looking away from the road.
Education
AR is finding its way into curriculum to enhance traditional learning methods and make them more engaging and interesting for students, giving them a more memorable and enjoyable learning experience. For example, textbooks can be marked with codes that, when scanned by a smartphone, can display additional content or 3D visualizations.
Hardware maintenance, assembly and construction
AR has great potential in the coming years to significantly reduce reliance on physical technical manuals and improve productivity at the same time by overlaying relevant information in a worker’s field of view while performing maintenance or other industrial tasks. Advanced AR systems can not only understand context to show the right information but help identify components and workflows using highlights and overlays. BMW, for example, is doing exactly that with a pilot program using AR on the assembly line. The most advanced AR systems are aimed at the enterprise market where the companies selling the AR can take a low volume, high cost, high profit margin, approach. This is typical of the start of an emerging technology as the economies of scale and production are not yet there to reach a broad consumer market, so businesses who are willing to spend large amounts of money are being approached first. As is usually the case, what large companies will be able to get their hands on first will make their way to a broader audience including gamers and consumers, where the technology and production capacity matures. Some of the most advanced products in the space such as the Varjo XR-3 headset combine super high-fidelity VR and VR technology into one headset and is priced at about $7000 and is aimed at large companies who have that kind of money to spend on a super advanced and realistic experience.
Medical applications and medical training
AR technology, like VR, holds a lot of potential in the medical space; there are many potential use cases for AR in medicine which are emerging, such as an AR headset that overlays crucial information to a surgeon who has hands full of surgical instruments. The headset can overlay important information about the patient, or even instructions from a more experienced surgeon elsewhere in the world who is observing and guiding the procedure. This is just one of an enormous potential number of use cases for AR in medicine.
4. Summary
To summarise, AR is an emerging technology and content creation platform with huge potential to enhance art and creativity, entertainment, education, medicine, training and industrial work by overlaying computer graphics over a vision of the real world. There are many hardware and software approaches to delivering this experience, which is just now starting to enter the public consciousness, and is anticipated to grow quickly as the technology becomes more user friendly, affordable and well known.
This overview covers the most important information to bring you up to speed on AR technology in the modern day. To follow up, see a separate overview of the role of AR in the world of education and training, specifically.