Hospital Sírio Libanês: AI during the covid crisis

The whole world was severely impacted by the covid-19 pandemic. It overwhelmed healthcare systems that were unprepared for the capacity requirements during the surge of cases. Scarce resources like ICU beds, medical professionals, and equipment were spread thin. Brazil, top in many economic rankings in the world, found itself atop a ranking it didn’t want: covid cases and deaths. Sírio-Libanês, a top-tier hospital in Brazil, decided to find a way to use AI to help better allocate resources between hospitals for covid cases and faced three important challenges.

Challenge 1) Input data

The hospital defined their objective as predicting if a patient would require an ICU bed with individual clinical data and not just epidemiological and demographic data. Having state-of-the-art EHR systems, the hospital was able to gather a decent amount of clinical data from patients, albeit having to deal with a large amount of missing data for certain variables they wanted to track. The EHR system was only as good as the manual input required to feed it.

Challenge 2) Sourcing talent to create a prediction model

Even though the hospital has a data science team, it is still small and spread thin with several projects. Given the urgency of the problem, Sirio-Libanês decided to crowdsource a solution. It submitted anonymized patient data to Kaggle, a platform known for its data science competitions. As with crowdsourcing initiatives such as those performed by NASA, there could have been clashes with the internal team. However, the urgency of the mission was stronger than any internal conflicts.

Challenge 3) Evaluating models and applicability

Another important issue with crowdsourcing solutions is how to evaluate them. While simple analyses with a confusion matrix could do the trick at a superficial level, an important aspect of the models was their applicability.

Even though the competition had already concluded, a team of Brazilian HBS students decided to test their data science skills in this Kaggle competition for a project in the HBS Advanced Analytics course. When they were able to treat missing data and achieve extraordinary results in their models, they decided to show some doctors what the students believed was the ultimate model. Despite having great results, they received the same reaction from each doctor they spoke with: this model can’t be used by the team. When digging deeper on the reasons, the unanimous conclusion from physicians was that the model was impractical. While there was a richness of data and different inputs to choose from in the dataset, some relevant inputs for the model were being measured infrequently. Others, such as some blood tests, had high lead times and were costly to take. The physicians needed a model that could quickly prioritize who needed additional attention, in order to conduct more extensive tests and blood work. The team of HBS students eventually built a triage model that was tailored to their needs and learned a valuable lesson.

Pathways to a Just Digital Future

Watch this tech inequality series featuring scholars, practitioners, & activists

Integrating AI to organizations is not just about building the ultimate model. It’s about ensuring AI can embed itself in an organization’s processes and add value, even if that means that more precise / accurate models can exist. AI is not helpful in the end if it’s not used by the stakeholders that need to interact with it the most.