How Zoom leverages AI to provide the best videoconferencing experience

After 9 years of sustained growth, Zoom was well positioned when the covid pandemic hit. With 2,500 employees and a profitable business, the company had just done its IPO less than one year before. Thanks to a UX-centric videoconferencing experience, and a cloud-native architecture which proved to be easy to scale, Zoom was able to scale from 10 million daily meeting participants to 200 million daily meeting participants in just one month.

Zoom was definitely one of the biggest winners of the pandemic and the quarantine. Its stock price more than tripled, and by June 2020 the company had capitalized a 169% year-over-year increase in their revenues. Their mobile app was downloaded 600,000 times in a single day. Zoom was able to position itself as the standard software for videoconferencing, even against the biggest and most renowned tech companies. From traditional enterprise solutions like Cisco WebEx, to suite-integrated products like Microsoft Teams and Google Meet, competition was fierce. The reason why Zoom was able to win over Cisco’s flawless track record and reputation, and Microsoft’s and Google’s existing synergies, was most probably its superior video technology.

Since its inception, Zoom decided to take a video-centric approach and, instead of using legacy solutions, decided to engineer and patent its own technology architecture. Zoom allows hundreds of simultaneous participants in their meetings, making it necessary for the service to handle connections of dissimilar qualities across different geographies. Therefore, Zoom was designed as a cloud-based technology, with servers distributed around the world, and mediating the transfer of audio and video, instead of using peer-to-peer (P2P) connectivity.

Additionally, Zoom adopted artificial intelligence at the core of its product.  By optimizing connections, video compression, and audio transfer, Zoom guaranteed an improved video quality. AI was also used to develop additional features to augment the experience and the interaction with users.

Almost every aspect of the product uses some sort of Artificial Intelligence:

Pathways to a Just Digital Future

Watch this tech inequality series featuring scholars, practitioners, & activists

• Virtual Backgrounds: Being one of its flagship features, Zoom allows users to blur its background or to replace it with an image or a video in order to increase their privacy during meetings. This feature uses a machine vision technique known as Image Segmentation which can identify the subject and subtract the background. Additionally, the app includes a feature called “touch up my appearance” which applies a filter to smooth the user’s skin and reduce blemishes.
• Video compression: Similar to the technology used for virtual backgrounds, the video algorithm identifies the most relevant sections of the image and maximizes the resolution of those areas. As people tend to focus more on faces than in the background, this feature prioritizes the subject when the bandwidth is limited.
• Audio processing: Zoom uses Deep Learning algorithms to process audio signals and improve audio quality in meetings. Using toolkits like Keras, Tensorflow or PyTorch, the system implements machine learning models like Noise Suppression, Voice Activity Detection, Speaker Recognition, Speech Enhancement and Music Detection.
• Live transcription: Using several NLP (Natural Language Processing) technologies like Speech-to-Text, translation, summarization, and sentiment analysis, Zoom now offers meeting hosts to include a live transcription of the conversation, which results specially useful for participants with a bad audio quality or in a noisy environment.
• Anti-Fraud: Zoom uses machine learning techniques to develop fraud detection models in order to identify and prevent unwanted abuse of Zoom services, like pornography, telephone fraud, and spam.
• Integration with AI assistants: Since February 2021, Zoom integrates with Google’s AI Assistant. Users can start a meeting from their Nest Hub devices by just saying “Hey Google, Start a Zoom Call”. Additionally, Google WiFi and Nest routers can be set up to prioritize Zoom traffic.
• Zoom Rooms: As the covid pandemic comes to an end, and the world transitions to the “New Normal”, Zoom’s next challenge is to win the office space. Zoom offers recommended hardware for setting up Zoom Rooms with special features and an improved meeting experience. Again, AI is at the center of the experience, enabling features like auto-tracking cameras, speaker focus, and people counting.

Sources:

Agrawal, S. (2020, November 12). Virtual Background for Video Conferencing using Machine Learning. Medium. https://towardsdatascience.com/virtual-background-for-video-conferencing-using-machine-learning-dfba17d90aa9

Barolo, P. (2020, June 29). A Glimpse Into How AI Will Make Meetings Better. Zoom Blog. https://blog.zoom.us/glimpse-into-how-ai-will-make-meetings-better/

Carter, R. (2021, February 9). Zoom Integrates with Google’s AI Assistant. UC Today. https://www.uctoday.com/collaboration/video-conferencing/zoom-integrates-with-googles-ai-assistant/

Hill, J. (2021, March 8). Update on Live Transcription For Free Accounts. Zoom Blog. https://blog.zoom.us/update-on-live-transcription-for-free-accounts/

Sloane, K. (2021, February 10). Zoom launches live transcription service. Learning Tech Blog. https://blog.learningtech.virginia.edu/2021/02/09/zoom-launches-live-transcription-service/

Thomas, A. (2020, April 21). Zoom Turns To Artificial Intelligence To Block Nudity On Its Platform. Analytics India Magazine. https://analyticsindiamag.com/zoom-turns-to-artificial-intelligence-to-block-nudity-on-its-platform/

Zoom Video Communications: Eric Yuan’s Leadership During COVID-19. (2020). Harvard Business School, 2–11.