Augmenting Reality and the Customer Experience


Only a complete nerd (greetings, fellow travelers) would bounce out of bed at 7:37 a.m. on Saturday, point her iPhone at the Harvard Business Review ("HBR"), and watch pistons, compressors and repair instructions spring to life in 3D, right off the printed page. And I wonder why my kids refuse to get up before noon on weekends. 


Smart businesses, especially in the automotive, retail, and manufacturing sectors, are quickly joining the AR nerd brigade, to the tune of $60 billion in annual spending by 2020. For example, today, 10% of retail executives surveyed by HBR say they are currently making substantial investments in AR. By 2020, that number is expected to rise to 32%. The leading applications are in service and maintenance, followed by manufacturing (quality), sales and marketing. In manufacturing, AR reduces the time to perform typical quality control tasks by 25% or more. In warehouse operations, Intel has used AR to reduce picking time by 29%. And marketers know that in this always-on, hyper-connected era, the best marketing is an outstanding customer experience, one that makes customers a) want to come back and b) tell all their friends.

AR works when a camera-enabled device (such as an iPhone) identifies a physical object, either from its shape (those oh-so-cute but annoying dog ear filters, which detect a face) or a marker placed on it (the HBR article, which had a marker in the large circular knob on the upper right). AR software streams data from sensors on the physical product to a digital model of that physical product, often combining that information with other business or environmental data. The model is known as the "digital twin." The software then superimposes information from the digital twin onto the user's view in 3D (in this case, the iPhone, although it could be any screen). In this way, the user's view is part real, and part digital. In some cases, the user can interact with the object via voice, gesture, or using a digital button or slider. 

The printed page.

The printed page.

Brought to life w/ AR markers and the author's iPhone.

Brought to life w/ AR markers and the author's iPhone.


The best AR applications include context. Context is relevant information, based on knowledge of the user, the last interactions with the user, or environmental or calendar cues (such as the impact of extreme heat or cold on a machine, or the arrival of holiday seasons). They are personalized to the user, saving them time. And finally, they are in the moment - delivered exactly when the customer is ready to buy the product. And that's why AR is a marketer's dream. 

AR applications in manufacturing and field service are readily apparent, as shown in the HBR "demo." But how can AR help the consumer where she shops - at retail? I thought about this as I trailed my many children into Shopper's Drug Mart in search of last-minute Halloween costumes. Wouldn't it be great if I could fire up my Shopper's Drug Mart iPhone app in our living room, point the phone at one of the kids, and see multiple in-stock, age-relevant costumes appear on her body so we can test the fit? We could then pay for it at home in the app using a digital wallet, wave my iPhone at the self-checkout scanner at the store, and walk out the door.

Sound like science fiction? No, today, retailers from Sephora (makeup) to IKEA (furniture) to Amazon (well, everything, but specifically fashion) are experimenting with augmented reality to save their customers time, shrink the sales cycle, and save themselves rework and returns. For example, IKEA's Place app accurately sizes a physical room and drops digital furniture into it. The app allows users to point their phone at a room in their home and see how a piece of furniture from IKEA's catalogue looks in it. And the dimensions are accurate. Leveraging Apple's ARKit, data from the phone's camera sensors is used create a digital model of your room (including measurements, lighting, and shadows).  The app then drops a digital piece of content (in this case, a chair), into your view, sized according to the chair's dimensions relative to the room in your view.

Will it fit? And look good with the shag carpet?

Will it fit? And look good with the shag carpet?


Amazon recently filed for a patent that could allow you to preview how a wristwatch looks on your hand, again from the comfort of your living room. It uses a three-dimensional sensor like a Microsoft Kinect to create a rash of data points (there's that digital model again) of your human hand, then captures data from the camera in your phone to predict how light would reflect off the watch as you moved your hand around. Amazon's Echo Look includes a depth-sensing camera that will, on command from its sister product Alexa, take a photo of you in your latest high-fashion gear. Its companion StyleCheck platform will then use machine learning to compare two different looks and tell you which style is better suited for you. Meanwhile, Amazon is capturing all those images in the cloud. It's not a leap to think that Amazon could soon create a digital model of your precise measurements, then overlay digital clothes on that model. Not surprisingly, Amazon Fashion recently launched 7 new private label fashion brands, on which it makes healthy 40 percent margins.

Many big brands are experimenting with AR. They may not get it absolutely right out of the gate, but they are absolutely experimenting.

Why is AR going to be so important, in a world where many predict AI, or machine intelligence, will soon surpass human intelligence? Because we're better together. Computers are better at predicting the best course of action given a known, predictable pattern of facts. People are simply better at reasoning in context in novel situations, as ABB's chief digital officer, Guido Jouret, says in the HBR article. Humans also have more creativity and imagination than machines. Augmented reality enables human potential and judgment. It does not replace it.

...we believe that combining the capabilities of machines with humans’ distinctive strengths will lead to far greater productivity and more value creation than either could generate alone. What’s needed to realize this opportunity is a powerful human interface that bridges the gap between the digital and physical worlds. We see AR as a historic innovation that provides this.
— Michael Porter and James Heppelmann, Harvard Business Review, Nov 2017

That's good news for all of us. And one hell of an opportunity for marketers, safety managers, and service techs, to name a few, in industries as diverse as automotive, retail, and heavy manufacturing.

Oh, hey kids! So glad you're up. Did you know Pokemon Go and those Snapchat filters are AR applications? Mmm-hmm. Mom rules. 

sheldon tech support.gif