SexyFace and Recognizing Refugees – Dr. Vivienne Ming’s Journey of Using AI for Good
Dr. Ming trained an AI model to recognize humans a player found attractive – but rather than apply it to dating, used it for lost refugee reconnection & education.
Dr. Vivienne Ming is one of the most inspiring scientists I have had the privilege of listening to in the Bay Area when I was a student at UC Berkeley. A theoretical neuroscientist – or as she would say in her talks, a “mad scientist” by profession, she has worked on a range of devices, applications, programs, and pieces of content in a remarkable journey that has intersected her personal journey with the limits of technology as well as its implications around her.
One of her programs – SexyFace, started off as a program where people would select faces that they find attractive – within a few clicks, the program could identify the kinds of people that someone found attractive; it was free to play, but priced to get a recommendation of someone you found pretty. The underlying mechanism was a deep neural network that learnt a language of faces, understood players’ preferences and more. However, the underlying ‘deep density component model’ that was developed for this program could go beyond just identifying sexy (a category) and be applied to recognizing facial expressions and more. Dr. Ming decided that her technology could be applied to something much more important.
The UN (as of when Dr. Ming worked on this project in the late 2000s / early 2010s) had a book with over a million photographs of orphaned refugees worldwide. Dr. Ming’s model learnt from people playing SexyFace, and once trained, was applied to try to reunite orphans with their families by having people selecting between pictures on what their lost, loved ones looked like. A few iterations of playing the game later, the AI would be able to point out the lost kids to families and a process to reunite them could be started. The company had not even set out to do this – while Dr. Ming always intended to use SexyFace for better purposes than dating applications, she was more focused on helping students learn better by creating a new form of interactive learning and testing that replaced the SATs and GREs. The refugee application came from a connection at Ericsson, who connected Dr. Ming to Technology for Good and the Refugees United campaigns run by the UN.
It is worth noting how monumental the effort put in by Dr. Ming was in changing refugee relocation – before the application was put forth on tablets to people, recognition would usually be done by searching physical books, page after page, hoping not to miss the crucial picture of your loved one as you scrolled past millions.
The tragedy of this story is in its clash with capitalism if I may say. Dr. Ming was running an unfunded startup trying to prove to VCs that the app could be used for education related purposes, earning fixed revenue, and was pushed to say no to working with the Ericsson team. By the time other ways of monetizing the technology had failed and she returned, the moment had passed and while Ericsson saved many lives, the impact was lesser than it could have been. She had been doing this work under a company named Augniscient, which according to Pitchbook was sold to a company named Rabbit. When clicking through, I discovered that Rabbit, a video chat application to help share experiences, had gone out of business in 2019, so I assume that the company failed to make a dent into other applications of Dr. Ming’s technology. If one was to diagnose what could have been done better from a business perspective, the technology could have been monetized by being licensed to companies lacking technical talent for various use cases (beyond the realm of dating) while also continuing to use the core trained model for refugee recognition applications.
To me, Dr. Ming’s journey is inspirational – she had the courage to push an AI application for good, utilize deep neural networks in one way externally (by having people play SexyFace) to train her models and deploy them for other refugee / humanitarian use cases that pose the most challenging issues of our times. The GTM here was despite not being entirely monetizable, a genius approach, and opens the realm of questioning: “Are there other ways of training models for our application by gathering data outside of the particular application we are trying to aim for” – something that was extremely unique ten years ago.
Dr. Ming’s advocacy also extends to another problem with AI – “do the benefits of AI flow disproportionately to the rich?” Dr. Ming argues in her work that AI should be thought of as a human right, in the way we view vaccines, rather than to enhance SAT scores by 2 points or makes a wristband buzz when an Amazon employee packages something wrong. Dr. Ming has responsibly scrapped several projects where her AI has shown signs of bias.
Sources:
https://www.socos.org/sexyface-part-1/
Thank you for sharing, Saad. I agree with you that Dr. Ming’s intentions here feel very admirable. It could be that Dr. Ming is simply before her time – I do feel that as the costs of computation continue to decline, more ML applications that don’t necessarily have to generate lots of profit could become viable… Until then though, I could see why Dr. Ming would have difficulty sustaining these applications. As long as the computation required for training and maintaining ML models remains costly, I feel that commercial applications will need to be the primary focus for innovation in the field…
interesting application! I am not sure I fully get how it works, but wouldn’t it be similar to the systems police use to recreate faces of people who disappeared / were kidnapped? I am always a bit terrified by these facial recognition softwares as I can easily imagine them being used in a “big-brother” type of world where people on streets would be easily recognized by the government / system, or where AI could analyze your facial reactions and translate them into emotions for other people.. I am curious to hear your thoughts on such applications. I get from the tone of the article that you are a fan of this technology – maybe you will be able to ease my fear a bit 🙂
Thanks for the exciting post! I can imagine that her presentation must have been really exciting! She had a great idea and found an easy way to train her models – sex always sells,… Too bad the software didn’t make it to a breakthrough. I don’t know the exact competitive landscape, but I hope that there are other companies that have taken on this problem, especially with such an important issue as refugees who have lost their families….
Great post and great technology! I smiled when reading about reuniting refuge families. Technology commercialization has historically been difficult. This is just another example of how big AI can be used for the good; we need more stories like this.
While reading, I imagined a possible use case in which families can find missing relatives. Kidnapping is still present in Latin-America and all too often cases are unresolved. Having an AI that constantly scraps the web for image matches of lost relatives would be extremely valuable for hurting families.
Super interesting post, Saad. I love the idea that while the original use case was frivolous, it was intended to prove a concept that could eventually serve a higher purpose. I wonder if there was some way to maintain a profit producing arm while furthering the refugee work?
Thank you so much for sharing the story Saad and I love the question Dr. Ming poses at the end. I feel like there’s a broader question here as well about the potentially negative impact of venture capital on AI development. It closes off development in areas that may not be easily monetizable but could still be tremendously beneficial to society.
Very interesting post Saad, you are tying together so many important threads all in one (reuniting families, AI use as a public good, failures of capitalism, VC biases). I am curious about how Dr. Ming optimized to have a low false negative rate, as I imagine she would want to minimize to near zero the instances where the model does not connect a searching family with their loved one erroneously.
This was such a fascinating post — thank you for writing, Saad! Getting a glimpse into how Dr. Ming viewed AI, its potential for use as a tool for empowerment and good, is really inspiring. Applying AI to humanitarian crises struck me as such a novel idea as most people tend to try to find commercial applications for AI to improve existing workflows and processes. And, though ingenious, it was personally heartbreaking to read about the refugee application… I can’t imagine the hope and desperation people felt as they scrolled through countless faces of people, trying to identify those that looked like their lost loved ones. Gave me so much to think about!
Super interesting application of face recognition and a wonderful read, Saad! Thanks for writing about this! She had a great idea, it was actually impacting human lives yet it seems she had difficulty getting securing funding or convincing VCs of the value. For such similar AI for good applications and technologies in general, where it is difficult to show immediate monetary value, but clearly has tremendous value added to lives of underprivileged/underserved, I’m wondering what the solution could be to circumvent such funding challenges?
Saad I’ve never heard of this it’s genius!!! It’s unfortunate it didn’t work out – I agree with some of the other comments that the founder is ahead of her time – or rather just in the wrong time when sex and online dating for sex are still somewhat taboo as instances of cat fishing and other manipulations and mischaracterizations remain a part of content moderation challenges.
Thank you so much for writing about SexyFace, Saad! I loved learning about how the founder used a commercial use case (dating) to train the model and then repurpose the AI for such a meaningful use case as saving refugee lives. I wonder what other use cases the technology could address – maybe for crimes and investigations, the technology could be used to scan through police databases for suspects that might match facial composites?