Amazon is using artificial intelligence (AI) to increase the accuracy of its Just Walk Out technology.

A multi-modal foundation model is being used to increase accuracy by using the same transformer-based machine-learning models underlying many generative AI applications. It applies them to physical stores.

Amazon has accomplished this by analysing data from cameras and sensors throughout the store simultaneously, instead of looking at which items shoppers pick up and put back in a linear sequence. For retailers, the AI system makes Just Walk Out faster, easier to deploy and more efficient. For shoppers, this should mean worry-free shopping at even more third-party checkout-free stores worldwide.

Just Walk Out uses cameras, weight sensors and a combination of AI technologies so shoppers in physical stores can buy food, beverages, merchandise and more without having to wait in a checkout queue, or stop at a cashier.

Just Walk Out technology, which launched in 2018, was built using generative AI and machine learning available at the time to figure out who took what. Previously, the AI system analysed shopper behaviour sequentially – their movement and location in the store, what they picked up, and the quantity of each item – each action processed one after another. However, in unusual or novel shopping scenarios such as if a camera view was obscured due to bad lighting or a nearby shopper, the sequential approach could take time to determine purchases with confidence, and sometimes required manual retraining of the model.

The multi-modal foundation model for physical stores increases the accuracy of Just Walk Out technology even in complex shopping scenarios with variables such as camera obstructions, lighting conditions and the behaviour of other shoppers, while allowing Amazon to simplify the system.

The AI system can achieve higher levels of accuracy by analysing all sensor data simultaneously, rather than sequentially. It looks at multiple inputs – cameras, weight sensors, and other data – and prioritises what’s most important to determine accurately the variety and quantity of items selected. It also uses continuous self-learning and transformer technology, a type of neural network architecture that transforms sensor data into outputs or receipts for checkout-free shopping.

For example, a shopper might pick up and put down multiple varieties of yogurt, in different combinations, and as they are doing so, another customer might reach for the same item, or the freezer door could fog up, obscuring the cameras’ view. In complex situations such as these, the new model can quickly and accurately determine the actual items taken by each shopper. This is thanks to its ability to process inputs simultaneously from various sources, including in this scenario weight sensors on the fridge shelves, continuously learn from these inputs, and decide which are most important to sort out accurately who took what. This reduces receipt delays and increases the ease of deployment of the technology for retailers.

The AI model (www.aboutamazon.com/artificial-intelligence-ai-news) reduces the rare need for model retraining in unfamiliar shopping scenarios, as the self-learning system continues to teach itself. To support Just Walk Out’s continuous learning, it is trained on a 3D map of the store to understand placement of fixtures – such as shelves, fridges, freezers and coffee machines – and products, as well as an image catalogue of the store’s merchandise so it can visually recognise every item accurately. This enables the AI system to recognise shopper behaviour, even if the store is remerchandised and items are rearranged or misplaced.

“The improvements to our AI system are so seamless that you will continue to enjoy the same contactless checkout-free shopping experience you’ve come to expect at Just Walk Out stores, all while protecting your privacy,” said Jon Jenkins, vice president at AWS. “Upon entering a Just Walk Out store you present a form of payment at the entry gate. The system immediately associates you with that payment method and begins mapping your journey around the store, adding or subtracting items from your virtual cart as you pick them up or put them down.”

The technology does not collect any biometric information; it only tracks how shoppers interact with the products and fixtures such as shelves or fridges, correctly identifying the products and quantities with which the shopper leaves the store.

Just Walk Out (justwalkout.com) is available in more than 170 third-party locations at airports, stadiums, universities, hospitals and more in the USA, UK, Australia and Canada. More are planned for this year.