Hello, Barbie? NLP and the first conversational doll
Hello, Barbie? It's the stuff of horror movies but dolls are now coming to live thanks to NLP and AI technologies.
Introduction
Since its release in 1959, the barbie doll has been a focal point in US culture as the most sold doll of all time. Due to backlash and increasing public outcry, Barbie sales declined from over 1.3 billion in 2011 to less than 1 billion in 2013. Mattel, the billion dollar parent company of Barbie, was certainly going to make some changes. To stay culturally relevant, many toy companies have released product lines with different ethnicities, body shapes, etc. Mattel went a different route.
At the time, the wave of conversational technologies was taking off as Apple’s Siri and Amazon Alexa were released in 2011 and 2013, respectively. Mattel’s Hello Barbie FAQ states that the most common request from consumers is that they “want to have a conversation with Barbie.” To meet this request and capitalize off the growing market for conversational tech, Mattel partnered with SF-based startup ToyTalk to bring the Barbie doll to life.
Digital Innovation of the Quintessential Doll
Through hardware hidden in the doll’s physique, ToyTalk and Mattel have created a doll that can carry a conversation decently well through artificial intelligence and natural language processing. First, parents have to download the Hello Barbie app to ensure that the doll is connected to wifi and for parents to listen in on conversations between Barbie and their child. When the doll is spoken to, the following steps take place:
- Button pressed: The doll is idle when unactivated. To begin a conversation with the doll, users must click the button that duals as a belt buckle. A phrase is then requested, selected, and sent, such as “hello [name],” to start the conversation
- Microphone detection: Barbie then waits for the response. A recorder and microphone embedded into the necklace of the doll detects and collects all words, questions, conversations, etc.
- Transmission: After a response has been given, its analyzed. The recorded speech is then sent to a control center via cloud based technology. Barbie’s necklace, belt, and legs are wifi-enabled to allow for this step to take place.
- Processing: once received, ToyTalk’s speech recognition software uses natural language processing (NLP) to parse down the recorded input. Moreover, ToyTalk’s algorithms utilize “natural language rules” so that the recorded language can be translated into data that a response can be generated to. The ToyTalk NLP algorithms likely use semantic analysis to understand identifiable parts of the recorded speech, rather than syntactic analysis. If the response is not clear, the child will be asked to repeat it.
- Response: Based on the parsed recording, Barbie will respond with a response compiled from 8,000 recordings of voice actors. These responses are primarily based off of programmed dialogue branches. Once Barbie has responded, the recordings are then stored on ToyTalk servers to remember things about the child for the next conversation. “For example, if a child tells Hello Barbie that she is an only child, Barbie will no longer ask her about siblings.” Moreover, Barbie constructs a strong digital persona for the child including their family, their likes, etc. Recordings are also used to test and improve the process.
The entire process, speech from child to response from Barbie, takes about one second. Similar to other conversational technologies, like Siri, Barbie does have a default response for unexpected questions or statements. While Siri might say “Hmm… I seem to be having trouble hearing you. Sorry about that,” Barbie says “Really? No way!” or “‘That’s sounds like something you should talk to a grown-up about.’’ which are said to work in most conversations with young children covering unexpected topics.
To ensure that Barbie is not responding to everything being said around the doll, the doll’s belt buckle functions as a button to initiate conversation.
Benefits to Customers and Users
There are many use cases to both the customer (parents) and users (kids).
- Personalize usage experience: thanks to ToyTalk’s flag model, Barbie stores and uses information that the child has already divulged to promote an appearance of memory. This is a significant improvement in personalization of dolls as each child’s doll will truly be unique to them.
- Child therapy: it is already customary to use dolls and other recreational children’s items to help kids talk about therapy-related issues that likely do not come up in conversation.
- Monitoring: thanks to the Parent Portal, parents can hear every conversation the child has with their Barbie.
Challenges
Unsurprisingly, Hello Barbie was not received warmly by all consumers. Though Hello Barbie was born within the wider context of growing conversation technology, it was one of the first sophisticated consumer products in that space specifically for young children. Though the histrionics of parents regarding technology in the lives of their children is common, there are some Hello Barbie specific concerns worth addressing. Firstly, it was determined that the cloud-based technology was not very secure given the sensitivity of the data. Hackers could feasibly tap into microphone and listen to recorded conversations. Additionally, there were concerns that a talking Barbie would stifle creativity amongst children as they will not longer have to imagine their doll conversing with them.
Horizons
Hello Barbie is still in circulation but is not actively produced by Mattel at the moment. The backlash that met Hello Barbie certainly does not preclude Mattel from going down that route again. Mattel has explored a $300 hologram version of Hello Barbie with minimal success.
Though Mattel has taken a notable pause in their integration of burgeoning technologies, the product is an interesting study of one of the first forays into putting the power of NLP technology into the hands of young children. The result was to be expected but, as software based applications continue to move the needle, the next Hello Barbie can’t be far away.
Excellent article!
With IoT devices constantly coming under attack by hackers, I think it is understandable that parents are skeptical about exposing their children to an IoT device with recording facilities. However, with better IoT security solutions being invented every day, Hello Barbie (and other similar devices) may well be accepted by parents in the near future.
I also wonder if the branding of the doll may have contributed to its failure. As parents are increasingly keen on introducing their children to educational toys early on, could ‘Hello Barbie’ have instead be designed and branded as an educational doll instead? While child therapy is indeed a great use case, the market segment may be significantly smaller than that of educational toys.
Another risk that should also be considered is the likelihood of bias. Since ‘Hello Barbie’ is a trailblazer in speech processing for children, some of the bias concerns around AI are likely not completely addressed and understood. Thus, the exposure to negative PR and regulatory threat is significant.
Great article, thank you for sharing! While I totally understand why it was not well received by all consumers I still think that Mattel should use their feedbacks and go forward with this kind of technology in general. I think the value proposition has some potential as a support for language development, especially in the context of more and more kids growing up exposed to screens at a young age – TV, tablets and smartphones – all of which are one way communication entertainment that doesn’t develop communication skills and may be harmfull to children’s psychological development.
I am a bit surprised that the doll was as controversial as it seems to be. While data privacy is a more sensitive topic when it comes to children, electronic devices such as games are likely capturing children’s data and may also be susceptible to hacks. I understand the argument of stifling children’s creativity. However, one could argue that iphones and ipads are also stifling children’s creativity and social connections by capturing their attention instead of motivating to play outside. Where should we draw the line with children and technology?