# Augmented reality

Augmented reality overlays digital content on the physical world through phones and headsets. Learn tracking, scene understanding and industrial and consumer uses.

## Overview

Augmented reality, or AR, places digital information, such as labels, arrows, instructions or 3D models, on top of the real world as seen through a phone camera, a tablet or a headset. Unlike virtual reality, which replaces what you see with a computer-generated world, AR keeps the real world in view and adds to it. A technician sees the next bolt to remove highlighted on the actual engine. A shopper sees a sofa standing in their own living room before buying it.

For this to work, the device must understand where it is and what it is looking at, many times per second. Computer vision tracks the device's position, finds floors, walls and tables, recognises objects and estimates lighting, so that virtual content stays fixed in place as you move. Much of this is AI: models that detect, segment and understand scenes, and increasingly multimodal models that can describe what the camera sees and answer questions about it.

An everyday picture is having an expert's sticky notes floating exactly over the machine you are repairing, showing which part to check, which way to turn it and what the reading should be, and staying in place as you walk around the machine.

## What it is

| Component | Role |
|---|---|
| Display | Shows virtual content over the real view: a phone screen, video pass-through headset or optical see-through glasses |
| Cameras and sensors | Capture the scene and motion: cameras, inertial measurement units, depth sensors |
| Tracking | Estimates the device's position and orientation in real time |
| Mapping | Builds a model of the surroundings, such as planes and meshes |
| Scene understanding | Recognises objects, surfaces, people and text |
| Anchors | Fix virtual content to real positions so it stays in place |
| Rendering | Draws virtual content with correct perspective, occlusion and lighting |
| Interaction | Lets users act through touch, gaze, hand gestures or voice |
| Content and data | The 3D models, instructions and live data shown to the user |

AR sits on a spectrum that researchers call the reality-virtuality continuum. At one end is the real world, at the other a fully virtual environment. AR lies close to the real end; "mixed reality" is often used for experiences where virtual objects interact more fully with the real scene. Many companies now use "spatial computing" or "extended reality (XR)" as umbrella terms.

## Where it came from

In 1968 Ivan Sutherland described a head-mounted three-dimensional display, built with his student Bob Sproull, which overlaid simple computer graphics on the user's view; it is often cited as the first AR system. The term "augmented reality" was introduced in the early 1990s by Thomas Caudell and David Mizell, researchers at Boeing, who built a head-mounted display to guide workers assembling aircraft wiring.

In 1994 Paul Milgram and Fumio Kishino described the reality-virtuality continuum, placing AR between the real world and virtual reality. In 1997 Ronald Azuma published a widely used definition: AR combines real and virtual, is interactive in real time and is registered in three dimensions. In 1999 Hirokazu Kato created ARToolKit, an open-source library that tracked printed markers with a camera and made AR experiments accessible to many developers.

Smartphones brought AR to consumers in the 2010s. Mobile games and camera filters reached very large audiences, and in 2017 the major mobile platforms released AR frameworks, ARKit and ARCore, that used visual-inertial tracking without markers. Headsets for enterprise and consumer use followed, and the OpenXR standard from the Khronos Group provided a common interface across devices. More recently, advances in [3D reconstruction](/tools/gaussian-splatting) and multimodal AI have made scene capture and understanding much richer.

## How it works

### Step 1: track the device

Visual-inertial odometry combines camera images with motion sensors to estimate how the device moves. It detects distinctive features in each frame, follows them across frames and fuses this with accelerometer and gyroscope data. Simultaneous localisation and mapping (SLAM) extends this by building a map while tracking, so the device can recognise places it has seen.

### Step 2: understand the scene

The system detects horizontal and vertical planes, estimates depth, and builds meshes of surfaces. AI models detect and segment objects, using techniques like those in [YOLO](/tools/yolo), [Mask R-CNN](/tools/mask-r-cnn) and [SAM](/tools/sam), read text, and estimate the pose of known equipment so that instructions can be attached to exact parts.

### Step 3: anchor content

Virtual objects are attached to anchors: a point on a plane, a recognised image, a known object or a geographic location. Shared or persistent anchors let several users see the same content in the same place, or let content reappear on a later visit.

### Step 4: render convincingly

The renderer draws content from the device's current viewpoint, many times per second. Occlusion hides virtual objects behind real ones, and light estimation matches shading to the room. Latency between movement and display must be very low, or content appears to swim and users may feel discomfort; this makes AR a [real-time system](/tools/real-time-systems).

### Step 5: connect data and interaction

Enterprise AR connects to back-end systems: work instructions, sensor readings from a [digital twin](/tools/digital-twins), maintenance records or remote experts on a video call. Users interact by tapping, gesturing, looking or speaking, and [multimodal models](/tools/multimodal-models) can answer questions about what the camera sees.

## Key concepts and terms

| Term | Meaning |
|---|---|
| SLAM | Simultaneous localisation and mapping |
| Visual-inertial odometry | Tracking motion by fusing camera and inertial sensor data |
| Six degrees of freedom (6DoF) | Tracking both position and rotation in three dimensions |
| Anchor | A fixed reference point for virtual content |
| Registration | Accurate alignment of virtual content with the real world |
| Occlusion | Real objects correctly hiding virtual ones |
| Optical see-through | Glasses where the real world is seen directly through lenses |
| Video pass-through | Headsets that show the real world via cameras on a screen |
| Field of view | How much of the user's vision can display virtual content |
| Motion-to-photon latency | Delay between movement and the display updating |
| Spatial anchor | A persistent anchor shared across sessions or users |

## A worked example

This scenario is illustrative. A company that services industrial air compressors wants to reduce errors and training time for a complex filter replacement with 22 steps.

The team creates a 3D model of the compressor and trains an object-pose model on photos so that a tablet can recognise the machine and align the model to it within about a centimetre. Each step's instruction is attached to the relevant part: an arrow shows which panel to remove, a highlight marks the correct valve, and a live pressure reading from the machine's sensors confirms the system is depressurised before the next step unlocks.

In a trial, newly trained technicians use the AR guide on 40 jobs and a paper manual on 40 comparable jobs. With AR, average job time is about 25 per cent shorter and recorded errors fall from 7 to 2. Technicians report that the guide is most helpful for rare jobs and less useful for routine ones, so the company focuses AR content on infrequent, high-risk procedures and adds a remote-expert call option for unusual faults.

## Strengths and limitations

**Strengths:**

- Puts information exactly where the work happens.
- Reduces errors and training time for complex, infrequent tasks.
- Lets customers visualise products in their own space.
- Enables remote experts to guide on-site staff visually.
- Connects live data, such as sensor readings, to physical equipment.

**Limitations:**

- Headsets can be heavy, have limited battery life and narrow fields of view.
- Tracking struggles in poor lighting, featureless spaces or with reflective surfaces.
- Creating and maintaining 3D content is costly.
- Cameras in workplaces and homes raise privacy and security issues.
- Safety risks arise if virtual content distracts from real hazards.

## How it compares with alternatives

| Approach | Immersion | Real-world context | Typical use |
|---|---|---|---|
| Augmented reality | Medium | Fully kept | Guidance, visualisation, remote assistance |
| Virtual reality | High | Replaced | Training simulations, design reviews |
| Mixed reality | Medium to high | Kept, with interaction | Collaborative design, complex guidance |
| Video and paper manuals | Low | Separate from the task | Routine instructions |
| [Digital twin](/tools/digital-twins) dashboards | Low | Remote view | Monitoring and planning |

## Practical guidance and best practices

1. Start with tasks that are complex, infrequent, error-prone or costly to train.
2. Choose phones or tablets first where hands-free use is not essential.
3. Test tracking in the real environment, including lighting and reflective surfaces.
4. Keep overlays minimal; show only what the current step needs.
5. Link content to authoritative data, such as current procedures and live sensor values.
6. Design for safety: never block the view of hazards.
7. Handle camera data according to privacy rules and site security policies.
8. Measure outcomes such as task time, errors and training hours against a baseline.

## Common mistakes

- Building impressive demonstrations without a measurable job to improve.
- Overloading the view with information, which distracts users.
- Ignoring how content will be updated when equipment or procedures change.
- Testing only in an office, then failing in a bright, noisy or cluttered workplace.
- Choosing a headset before confirming that workers can wear it for real shifts.
- Recording video from sites or homes without clear consent and retention rules.

## Frequently asked questions

### What is the difference between AR and VR?

AR adds digital content to the real world, which remains visible. VR replaces the real world with a fully virtual one, usually in a closed headset. Mixed reality headsets can do both, switching between pass-through and fully virtual views.

### Do I need a headset for AR?

No. Many successful AR applications run on phones and tablets, which people already own. Headsets matter when workers need both hands free or must see guidance continuously while working.

### How does AR know where things are?

It tracks visual features in camera images and fuses them with motion-sensor data to estimate the device's position, while building a map of surfaces. Recognition of images, objects or locations then lets it anchor content to specific real-world places.

### Where does AI fit into AR?

AI powers tracking and depth estimation, recognises and segments objects, estimates the pose of equipment, reads text, translates signs, and with multimodal models answers questions about what the camera sees. It also helps create 3D content from photos and video.

### Is AR used outside industry?

Yes. It is used in retail for trying products at home, in navigation, education, museums, games, sports broadcasting, healthcare training and social media filters. Industrial uses are often where measurable returns are clearest.

## Glossary

- AR: augmented reality, digital content overlaid on the real world.
- VR: virtual reality, a fully computer-generated environment.
- XR: extended reality, an umbrella term for AR, VR and mixed reality.
- SLAM: building a map while tracking position within it.
- Anchor: a real-world reference for placing virtual content.
- Pass-through: showing the real world through device cameras.
- Pose estimation: determining an object's position and orientation.
- OpenXR: an open standard interface for XR devices and applications.

## Summary

Augmented reality overlays digital information on the real world through phones, tablets and headsets, using tracking, mapping, scene understanding and low-latency rendering to keep content in place. It grew from Sutherland's 1968 head-mounted display and Boeing's early 1990s work, through ARToolKit and the 2017 mobile AR frameworks, to today's AI-driven scene understanding. It delivers most value on complex, infrequent tasks and product visualisation, provided content is accurate, overlays are minimal, tracking is tested on site, and safety and privacy are designed in.