A Short History of Augmented Reality

1862 · THE OPTICAL ANCESTOR

A ghost made of plate glass

Not augmented reality, but the visual premise arrived more than a century before anything digital could deliver it.

An engineer named Henry Dircks worked out the illusion and could not persuade a theatre to rebuild itself around it. John Henry Pepper, who ran the Royal Polytechnic Institution, saw the design, found a way to fit it into a stage that already existed, and presented it publicly on 24 December 1862 during a performance of Dickens's The Haunted Man and the Ghost's Bargain.¹ History gave the effect Pepper's name rather than Dircks's, which is its own lesson about who gets remembered.

The method was almost insultingly simple. A large pane of glass sat between the audience and the stage, angled. An actor stood in a hidden pit below, lit hard with limelight. The audience saw a transparent figure apparently sharing the stage with solid ones.

This was not augmented reality by the technical definition the field later settled on. Nothing tracked the viewer. The apparition was the live reflection of a separately lit human being, not a computer-generated object registered in three dimensions, and its illumination came from its own lamps rather than from the stage it appeared to occupy. What Pepper established was the premise: a figure that is not present appearing to occupy the same scene as things that are.

The limitation is the interesting part. The illusion held because the geometry was fixed. The glass, the pit, the lamps and the seating were all built around one another, and the effect worked within a designed viewing zone rather than everywhere in the house. Nobody could walk around the ghost. Every step that follows in this story is an attempt to let them.

The trick never really left, incidentally. It is the principle behind the apparitions in Disneyland's Haunted Mansion, and a modern descendant of it put Tupac on stage at Coachella in 2012.

1901 · THE IDEA IN PRINT

Spectacles that label people

One of the earliest fictional descriptions commonly cited as an AR precursor, and it went straight to the social problem.

In The Master Key, published in 1901, L. Frank Baum, the author of The Wizard of Oz, gave his hero a pair of spectacles called the Character Marker. Worn in company, they showed the wearer a letter on the forehead of each person in view: G for good, E for evil, and so on, visible only to the person wearing them.²

Baum noticed the awkward part immediately. Rob declines to look at his own family, because he is afraid of learning something about them he will not be able to un-learn. He goes on using the glasses elsewhere. That hesitation, rather than any optical detail, is what makes the passage worth citing: it is an early sketch of the privacy and social-acceptance questions that would eventually surround head-worn displays. The devices and the outcomes turned out to be very different, but the question Baum put his finger on was the right one.

1961 TO 1968 · THE COMPUTER-GRAPHICS HEADSET

So heavy it hung from the ceiling

Head-mounted displays predate Sutherland. What his group added was computer-generated imagery locked to a real room.

Philco engineers Charles Comeau and James Bryan demonstrated Headsight in 1961: a head-mounted display with a television screen and magnetic head tracking, wired to a remote camera that turned when the wearer's head turned.³ Morton Heilig had filed a patent in 1957 for a head-mounted stereoscopic television apparatus, granted in 1960 and later commonly known as the Telesphere Mask.⁴ His separate Sensorama simulator, patented in 1962, surrounded filmed experiences with stereo imagery, sound, wind, vibration and smells, including a motorcycle ride through New York City.⁵ Wearable displays were not waiting to be invented.

What those earlier displays lacked was computer-generated imagery coupled to the wearer's tracked head position. Ivan Sutherland's 1965 essay The Ultimate Display runs a page and a half and is still worth reading. He imagined a room in which a computer controlled the existence of matter, where a displayed chair would be good enough to sit in and displayed handcuffs would confine. He was wrong about how it would arrive and largely right about why anyone would want it.

In 1968, working at Harvard with collaborators including Bob Sproull, Ted Lee and Dan Cohen, and with the ultrasonic head-position tracker designed and built by Charles Seitz and Stylianos Pezaris at MIT Lincoln Laboratory, he completed a partly see-through, stereoscopic, head-tracked display driven by computer graphics.⁶ It drew simple wireframe shapes that held their position in the room as the wearer moved around them. Its mechanical support hung from the ceiling, which is where the later joke nickname, the Sword of Damocles, came from. The nickname has outlived most of the research.

The graphics were sparse, transparent wireframe objects: a cube, a surrounding room, even a molecular structure. That was enough. The point had been made: a computer-generated object could be tied to a place in a real room while a person moved freely around it. The audience was finally unbolted.

Nor was the head-mounted route the only one. In the 1970s Myron Krueger built Videoplace, which put participants' projected silhouettes into a shared responsive scene with no headset at all. He called it artificial reality, and it remains one of the more elegant things anyone has done in the field.⁷

1990 TO 1997 · GETTING A NAME

Boeing had a formboard problem

The phrase "augmented reality" comes out of a factory process question, not a vision of the future.

A Boeing aircraft contains a great deal of wiring, and each harness was assembled on a full-size plywood formboard. Long computer-generated paper plots were attached to plywood formboards. Pegs mounted on each board defined the required routing, and workers laid individual wires across them, which meant preparing, storing and swapping physical layouts for every variation.⁸ Around 1990 Tom Caudell and David Mizell proposed replacing the plots with a head-mounted display that would draw the correct routing onto a blank, reusable board. Caudell is conventionally credited with coining the phrase "augmented reality" for this work; the documented paper he and Mizell published on it appeared in 1992.

The early 1990s got serious in more than one place. Louis Rosenberg, at the US Air Force Armstrong Laboratory, built Virtual Fixtures in 1992: virtual guides overlaid onto a real manual task, with measurements showing that operators performed better with them.⁹ During the same period Steven Feiner's group at Columbia developed KARMA, documented in a 1993 publication, which used a head-mounted display to guide someone through servicing a laser printer.¹⁰ Neither was glamorous. Both worked. Feiner's group then took AR outside: its first outdoor mobile prototype was built in 1996, and the 1997 Touring Machine combined a backpack computer, a head-worn display and a handheld interface to annotate buildings and locations around Columbia's campus.¹¹

In 1997 Ronald Azuma published a survey that gave the field a widely cited working definition, and its three conditions are still the ones most people reach for.

Combines

It combines real and virtual content.

Interactive

It responds in real time, not in playback.

Registered

Virtual content is anchored in three dimensions.

The third condition is the one that costs money. Accurate registration was one of the major reasons AR spent decades in laboratories and specialised industrial systems while ordinary computer graphics went to the movies. Displays, processors, cameras, calibration, batteries, field of view and interaction design were all substantial limitations too.¹²

1998 · THE ONE MOST PEOPLE HAVE SEEN

Broadcast AR arrives as a yellow line

Among the most widely viewed examples of augmented reality ever deployed, and almost nobody files it under that heading.

Sportvision's virtual first-down line debuted on ESPN on 27 September 1998, during Bengals at Ravens. A yellow stripe lies on the grass at the line to gain. It sits flat on the turf, follows the camera as it pans, and disappears correctly underneath any player who runs across it.

Getting that right in 1998 took encoders on the participating cameras reporting pan, tilt and zoom many times a second, a survey of the stadium accurate enough to model the crown of the field, and a per-frame colour decision separating grass from everything standing on it so that players occluded the line rather than the other way around. Real and virtual content, responding live, anchored in three dimensions. It is a genuine, working instance of the thing, running on national television while the public image of AR still centred on headsets.¹³

Fox had tried something adjacent in 1996 with FoxTrax, the glowing hockey puck, and the reception was rough. It is tempting to file that as a simple failure, but the truth is more useful. FoxTrax was aimed at a real viewer problem, since a puck genuinely is hard to follow on television, and audience reaction was mixed rather than uniformly hostile: committed hockey fans disliked it strongly, while more casual viewers were less troubled. Among its difficulties were expense, calibration, later changes in broadcast rights, and the visual design itself, including a comet trail that struck existing fans as an intrusion into the look of the sport.¹⁴ The lesson is not that usefulness does not matter. It is that usefulness was not sufficient on its own: the augmentation also had to be visually restrained and acceptable to the audience the sport already had. Technology that succeeds has a tendency to stop looking like technology.

1999 TO 2009 · MARKERS AND MAGAZINES

The decade of the black-and-white square

A cheap webcam and a freely available library put augmented reality in front of anyone with a printer.

Hirokazu Kato and collaborators developed and demonstrated ARToolKit in 1999, including at SIGGRAPH that year, with the work associated with Kato, Mark Billinghurst and the University of Washington HIT Lab. It was released as an open-source project in 2001.¹⁵ It tracked printed black-and-white markers: point a webcam at the square, and a model appeared standing on it, correctly oriented. A great many AR demos through the following decade were built on ARToolKit or on something imitating it.

The markers were a compromise, and the people building them said so at the time. A high-contrast square is easy to find in a camera frame, which is precisely why it was needed. Natural-feature tracking already existed, but on the low-cost hardware of the period, and without a previously prepared map of the environment, little else in an ordinary scene could be detected and tracked as cheaply or as consistently. The square was a confession.

The era produced some glorious excess. Continuing that outdoor line, in 2000 Bruce Thomas and colleagues at the University of South Australia built ARQuake and played it on the campus lawn wearing a backpack computer, a GPS unit and a headset.¹⁶ By 2008 phones were just capable enough to try: Wikitude launched on Android that year and Layar followed in 2009, both hanging labels on the horizon using GPS and a compass, both wobbling like a compass in a thunderstorm. In December 2009 Esquire put a marker on its cover so that readers could hold the magazine up to a webcam and watch Robert Downey Jr. appear. It sold magazines, and it taught a great many people the phrase.¹⁷

2007 TO 2016 · THE RESEARCH THREAD

Losing the square, slowly

Markerless tracking was demonstrated in research laboratories years before it arrived in a phone API.

Running alongside the marker era was the work that eventually replaced it. In 2007 Georg Klein and David Murray at Oxford presented PTAM, which tracked a camera's position and built a map of an unprepared scene at the same time, with no printed target anywhere. By 2009 they had a version running on a camera phone. The capability that later felt like it appeared from nowhere had in fact been demonstrated on consumer hardware nearly a decade before most people saw it.¹⁸

Google took a run at it in hardware. Project Tango was publicly announced in 2014, with work preceding the announcement, and reached a consumer handset in 2016, adding depth sensing and motion tracking to a phone.¹⁹ Only two commercial Tango phones reached market, the Lenovo Phab 2 Pro in 2016 and the Asus ZenFone AR in 2017, and the programme was wound down, but Google has been explicit that ARCore was built on what Tango produced. When the mainstream APIs arrived, they were standing on a decade of published work.

2013 TO 2016 · AR MEETS THE PUBLIC

Glass, HoloLens, and a monster in the park

One product failed for a tangle of reasons at once. Two others succeeded by attaching themselves to something people already did.

Google Glass reached Explorer Edition buyers in 2013 at US$1,500. Strictly it was a small heads-up display rather than registered AR, but it is the product much of the public learned the category from, which made its reception expensive for everyone else. The reasons it stalled were plural: privacy and social-acceptance concerns, a prototype price attached to something sold into consumer expectations, short battery life, heat and display limitations, and no clear consumer use case underneath any of it. On the privacy question specifically, the device had no clear outward recording indicator, though its display did illuminate while recording. The insult "Glasshole" entered general use before the Explorer programme ended in 2015. Glass never became the mainstream consumer platform its publicity had suggested, although the technology continued in enterprise editions.²⁰ Baum had sketched the shape of that problem in 1901.

Microsoft's HoloLens, announced in 2015, was a fuller implementation: self-contained mapping, gesture input, virtual objects that stayed where you left them in a room, and no need for fiducial markers. The Development Edition opened for pre-order in February 2016 at US$3,000 and began shipping that March,²¹ and the field of view felt like watching a hologram through a letterbox. Microsoft never released HoloLens as a mainstream consumer product. Its deployments stayed concentrated in development, research, enterprise and military programmes, and related Microsoft technology became the basis of the US Army's IVAS programme.²²

AR's quieter consumer breakthrough arrived on the front-facing camera. Snapchat introduced Lenses on 15 September 2015, attaching animated effects to a tracked face in real time.²³ Unlike Glass, they required no new device and created no uncertainty about whether someone was recording: the camera was already open, pointed at its owner, inside a social ritual that already existed. Face tracking became one of the first forms of AR that hundreds of millions of people used without thinking of it as AR at all. By 2020 Snap reported that more than 200 million people engaged with its AR features on an average day.²⁴

Then on 6 July 2016 Pokémon GO launched and, for one strange summer, a very large number of people went outside.²⁵ Niantic had been rehearsing since Ingress in 2012, and John Hanke's team had come out of Keyhole, the company whose mapping work Google acquired and developed into Google Earth and Maps.²⁶ Here is the wrinkle worth keeping: surveys and later player studies found that many regular players rarely or never used the camera AR mode, because catching was faster and easier without it.²⁷ The camera mode still mattered, just not to repeated play: images of Pokémon apparently standing in real streets and real living rooms did a great deal of the game's promotional work. The game's location layer, the walking, the collecting, the social play and the strength of the Pokémon brand all mattered more to its success than stable visual registration did. That is a result the industry has never entirely wanted to look at.

2017 TO THE PRESENT · THE PART THAT BECAME ORDINARY

Registration becomes a line of someone else's code

The change was not a breakthrough in tracking. It was distribution.

Apple announced ARKit in June 2017 and shipped it publicly with iOS 11 that September.²⁸ Google released an ARCore preview in August 2017²⁹ and shipped version 1.0 in February 2018. Neither company invented markerless tracking, as PTAM and Tango had already shown. What they did was put it behind a standard API on hundreds of millions of devices people already owned, which turned out to matter more than any individual technical advance.

The underlying technique is visual-inertial odometry: the camera feed and the phone's accelerometer and gyroscope are used together, each constraining the other, because they fail in different ways. The camera struggles with blank walls and motion blur while the inertial sensors drift over time. Combining them produces something far steadier than either alone.³⁰ It does not produce something perfect, and drift and outright tracking loss remain ordinary experiences.

Two related myths are worth retiring. Early ARKit did not work out the shape of a room; it tracked device motion and initially detected horizontal planes, with richer scene understanding arriving later. And markers did not disappear. Fiducial and image targets are still the right tool when content needs to be anchored to a known object, or when initialization has to be reliable and immediate.

Dedicated depth sensing later reached Apple's mainstream Pro devices. LiDAR arrived on the iPad Pro in March 2020 and the iPhone 12 Pro that October.³¹ It did not remove the guesswork so much as constrain it, supplying direct depth measurements that software still has to assemble into a usable model of the scene. On headsets, passthrough was not new, but high-quality consumer colour passthrough improved sharply with Quest 3 in 2023,³² and Apple's Vision Pro launched in the United States on 2 February 2024 from US$3,499.³³

The commercial surprise has been elsewhere. The display-free Ray-Ban Meta generation launched in 2023 with a camera, speakers and an assistant, and nothing attempting to register anything against the world, and it sold: EssilorLuxottica reported growth above 200% in the first half of 2025.³⁴ Meta followed in 2025 with Meta Ray-Ban Display, but even Meta classifies that as display AI glasses, a category it keeps separate from the augmented-reality glasses represented by its Orion prototype.³⁵ A display in front of an eye is not, by itself, augmented reality. Registration remains the dividing line. Still, the pattern is hard to miss. Sixty years of research into holding a virtual object in place, and the breakout consumer product in the smart-glasses category politely declined to try.