[Expert Talks] Machine Learning role in the Frame Removal experience

About Xavier Naturel, Machine Learning Researcher at Fittingbox

After joining Fittingbox in 2019 as a Machine Learning Researcher, Xavier and his team have been working on several Machine Learning methods that detect landmarks on faces, and image segmentation. This research is the basis of the functioning of the Diminished Reality experience "Frame Removal", which, unlike Augmented Reality, virtually removes an element from the real world.

Frame Removal, a truly immersive experience!

Launched in 2021, Frame Removal is a world first. It is an exclusive add-on of the Fittingbox Virtual Try-On, which makes life easier for glasses wearers. As soon as the user launches a Virtual Try-On session, this innovative technology detects in real time the marks corresponding to the physical pairs of glasses worn on users’ faces, to erase them on the screen and replace it with the 3D digitized frame, previously chosen by the users in question. The goal: to be able to see clearly when choosing the next pair of glasses, and to be able to keep the corrective lenses on your nose while trying new frames on.

Learn more about Frame Removal

How does Frame Removal work?

To ensure that Frame Removal works, Xavier and his team exploit Artificial Intelligence and its sub-fields, in particular Machine Learning and Deep Learning. The latter is essential for the detection of each pixel belonging to the glasses worn by the user. This detection process is better known as “semantic segmentation”.

In short, each pixel is detected by a Deep Learning algorithm (the DeepLabv3+) and then classified into 3 categories: background, lenses and frame. Once the pixels are labelled in one of these categories, the fitting stage can be carried out as if the user was not wearing glasses.

Try this new experience on your mobile, computer or tablet

In a nutshell, how does Machine Learning play a fundamental role in how Frame Removal works?

Machine Learning, as explained at the beginning of this article, is a sub-field of Artificial Intelligence. It is an algorithm that learns to interpret images content from a set of known images.

The data set from which the Frame Removal algorithm is taken, draws information from our Virtual Try-On, a semi-synthetic data set. Fittingbox Virtual Try-On technology is used to generate digitized glasses on a real face, as well as masks, landmarks and visibility information, the so-called “ground truth.”

It is precisely by having learned to discern and analyse these 3 elements - background, lenses and frame - separately, that our Frame Removal is able to recognise the physical frame when the user starts a Virtual Try-On experience with real glasses on the face. At that point, the deep neural network provides the location of the glasses so that they can be erased and replaced by the 3D digitised pair to be tried on.