We’ve all been there. You’re shopping at your favorite store, perhaps running an errand for someone else. But in order to make the purchase, you need to know if it’s right for you. The sales associate is busy helping someone else, or they may be missing in action. You just need the details about the product so you can get it right the first time without having to return it later or have buyer’s remorse. You want to be satisfied.

When shopping online, of course, it’s all laid out in front of you. The specs. The sizing chart. All the features and options. The ingredients. If you’ve got a browsing history or are otherwise familiar to the online retailer, recommendations will no doubt appear on your monitor or via email that are based on what may also interest you. Deals arrive in your inbox. You’re making an informed decision with just a few clicks, and there’s no line at the checkout.

Retailers strive to make the shopping experience at physical stores more personal, yet they find themselves at a disadvantage compared to the ease and convenience of online shopping. They’re starting to turn to augmented reality (AR) technology to stem that tide, and thanks to smart phones – and the components inside those phones – consumers can have the instant information they need as they walk the aisles (sales clerk optional).

Smart phones unite the real and virtual worlds
Smart phones are becoming indispensable in today’s world, and in most smart phones you’ll find MEMS-based accelerometers, gyroscopes and magnetometers that make these devices intuitive to motion and position. Augmented reality leverages the phone’s physical movements and interacts with the user as it provides contextual information. In a mall or store, smart phones have the potential of learning a shopper’s exact location and would have the ability to send targeted ads or coupons to the shopper as they navigate the aisles. Location-based services, of course, would also ensure the shopper quickly finds the product they want. The sensors inside the phone must provide precise position and compass direction to make contextual awareness possible. The cell phone camera and a robust user interface ties it all together to make augmented reality a two-way experience. Here’s a summary of the basic sensors in a smart phone and their relation to augmented reality.

Motion sensors, as the name implies, can measure a device’s movement. When used with other basic sensors, they can help reflect the device’s physical environment. MEMS accelerometers use a flexible layer of silicon that mimics the three axes of a traditional accelerometer. The changes in motion exhibited by the silicon can then be used to calculate change in orientation. When patterns can be developed from this data, the nature of the user’s movements can be ascertained, such as walking, etc.

Gyroscopes measure the angular rate or rotation around a device’s x, y, and z axes. Gyroscopes rely on the Coriolis effect and resulting capacitance changes to obtain the measurements.1 This enables a device’s precise response to orientation and movement. Gyroscopes are necessary to recognize gesture quickly.

Magnetometers measure the variations in a magnetic field. In theory, they should account for the Earth’s magnetic field, acting as the smart phone’s compass. However, according to one vendor, magnetometers are susceptible to EMI and are sometimes unpredictable.2 When free of EMI, these three-axes devices can be used to help determine navigation headings and dead-reckoning for location-based services.

Optical sensors take the images captured by the phone’s camera and help overlay contextual information to the user via the cell phone screen.

Sensor fusion simplifies AR implementation
Information gleaned from individual sensors on a smart phone is subject to the limitations of that particular sensor. To make augmented reality possible, motion sensing data must be aggregated and interpreted. That’s why intuitive software is needed to process the data from the disparate sensors to create a “sum that’s bigger than the parts” scenario. According to Mike Stanley of Freescale, the techniques involved with sensor fusion “should trade off strengths and weaknesses” of the various sensors. They must also “improve the quality and noise level of computed results by taking advantage of known data redundancies between sensors and knowledge of system transfer functions, dynamics and/or kinematics.”3 

A software-based sensor fusion environment or software development kit (SDK) optimizes hardware to help create a better user experience by providing location data faster and with more accuracy. Combined with indoor mapping, the location of specific products and deals, can then be relayed to a smart phone and ultimately matched to the user’s environment and overlayed to their screen. Leading vendors that offer sensor fusion software include STMicroelectronics, Invensense and Freescale Semiconductor whose eCompass SDK can be used with a new a six-axis sensor that combines an accelerometer and magnetometer and ASIC in a single chip.

Augmented reality is already gaining widespread use in military and aerospace environments. In retail, applications are emerging. This past spring, Walmart customers were able to use their smart phones in a scavenger hunt based on the Disney film “The Avengers” as part of a co-promotion between the studio and retailer. Another location-based phone app helps beer drinkers who are loyal to Stella Artois find nearby bars that serve the brand. Bar names and locations, along with their distances are overlaid right on the user’s screen. Taxi info is also provided. IBM Research, meanwhile, is working on an app that will really make shopping personal. Users of the app will specify what they’re looking for in a certain product, and while in the store, capture images of products on store shelves via their cell phone video cameras. Shoppers will receive information on the product and rankings based on the user-provided criteria in order to make quick, informed purchasing decisions. It will also tie in loyalty rewards or incentives and offer shoppers suggestions for similar items.
AR still faces some technological and user adoption hurdles to get beyond novelty-type applications in retail. There is still room for improvement for image recognition technology, and data must be organized and linked in a way that ensures product information is organized and easy to access in mobile platforms. Stephane Gervais-Ducouret, global marketing manager for sensors at Freescale stresses that the full technology must be ready from end-to-end “Even if each piece of this technology exists in separate pieces, it still requests some work to put them together in an efficient system open to consumers.” This also includes a bigger commitment from retailers who are not yet convinced that consumers want and need more information as they are browsing the aisles. 

A glimpse into the future
Google’s Glass project may offer a glimpse into the future of location-based augmented reality, and it may help give AR a bigger push into the retail space in the next few years. And beyond the smart phone, interactive digital platforms are being developed in which users can virtually “try on” clothes and jewelry overlaid right on their computer screens.

According to Juniper Research, over 2.5 billion mobile AR apps will be downloaded to smart phones and tablets per year by 2017.4 More people around the world are becoming smart phone users, so it’s up to the technology and retailers to keep pace. From a hardware standpoint, this will continue to be possible as user interfaces improve and sensors and sensor fusion provide better gesture recognition. Gervais-Ducouret adds that “The coming generation of application processors with powerful GPU (Graphic Processing Units) and supporting standardized software for image recognition will greatly improve image recognition efficiency.” And for retailers, that means an openness to use the technology to better engage the consumer where the purchasing decision is to be made.

1. Esfandyari, Jay, Roberto De Nuccio and Xu, Gang, “Introduction to MEMS gyroscopes,” Solid State Technology, November 15, 2010.
2. Understanding smart phone sensor performance: magnetometer.
3. Stanley, Michael, “What is sensor fusion?”, August 3, 2011,
4. Juniper Research,