Project by: Sarah Krumulis, Matthieu Bornet, Olena Levchun
Advertima, a Computer Vision and Machine Learning company, visually interprets human behavior in the physical world, enabling smart spaces of the future to interact with people in seamless and meaningful ways. Their real-time and easily accessible data stream, called the ‘Human Data Layer’, powers their smart retail solutions, including Autonomous Store and Smart Signage. These next-generation solutions empower companies to create the customer experience of tomorrow: frictionless, relevant and highly-targeted. Advertima is fully compliant with the EU General Data Privacy Regulation (GDPR).
Tools and technologies used in this project:
Models: Linear Regression, Random Forest, Basic Neural Network
Sarah Kurmulis, Olena Levchun, and Matthieu Bornet were tasked with processing, cleaning, and engineering the data from multiple stores and building models using Machine Learning and Deep Learning to understand consumers’ walking paths around the store monitors. Knowing who is currently present in the display area of the screen and also paying attention is a prerequisite for showing relevant ads. More precisely, the team predicted whether a given consumer will be present in the display area over the next few seconds. A subsequent improvement allowed the team to predict the exact position of the consumer within less than a meter deviation. By analysing the head orientation, they were also able to predict whether someone will be looking at the monitor, a proxy for paying attention.
The project’s goal was to improve the precision of Advertima’s personalised advertising. Presently, Advertima extracts only a few basic features of the customers walking in front of their screens - age and gender. These are inferred by facial recognition software. The actual photo is never stored; only the two features are kept and properly anonymized. Knowing the age and gender allows for limited personalization (ex:showing shaving products to adult men or cosmetic products for young women). Since loading and displaying a concrete advertisement can take on average 3 seconds, it is important to know who is going to be in front of the monitor in advance.
To this end the team analyzed the moving patterns of a large collection of customers. The second by second trajectories of people walking in the viewing areas of monitors from different installations were made available by Advertima. The data also contained other movement-relevant information, such as head position with respect to the screen. From the positional data the team built a Machine Learning model to predict the position of a customer given his or her past trajectory in the immediate future (1, 2 and 5 seconds). The model performed very well, predicting a customer’s position with an average precision of less than 1 meter.
A subsequent iteration improved the model to also predict the head orientation. In this way, targeted advertisements can be shown not only to people close to the screen, but also to those looking in its direction.
As part of the project the students performed extensive data exploration and feature engineering, which allowed them to also report some insightful visualizations. In particular, they produced maps of hotspots where people tend to walk predominantly. These so-called attention zones can be used to optimize the store layout and the specific position of the screen.
By completing this project, Advertima was supported in creating more accurate analyses for their personalized advertising. Potential next steps for the team of the three bootcamp graduates were defined as follows:
- Implementation & fine-tuning of the Machine Learning model
- Further exploration of Deep Learning techniques
- Adding product position information for improved targeting
Their models reached high levels of accuracy for predicting presence (86% mean accuracy at 5 seconds in the future) and position in stores (8 cm mean position accuracy for a 1s forecast, 112 cm for a 5s forecast), and valuable insights into zones of attention to the screen. These results will enable the company to further improve its targeted ad system.
Matthieu Bornet says:
The team at Advertima was very helpful and working together on improving their smart signage product using Machine Learning was an exciting learning experience.
Interested in reading more about the Final Student Projects? Then check out some other interesting Full-Stack and Data Science projects.