Mass General Hospital Radiology

Properly assessing a diagnostic image in radiology is imperative to a positive patient outcome. Unfortunately, there are major challenges that confront most radiologists, but AI and ML can help.

Mass General Hospital (MGH) has some of the leading radiologist experts in the world. Radiologists’ role is to diagnose diseases and injuries through examining medical images that they’ve taken through x-rays, CT scans, MRIs and other high-quality diagnostic imaging. In addition, they have also become the drivers of a range of radiation-based therapies that treat a wide range of the most serious disease. Their role consists of the following steps:

  1. Consulting with your referring physician about the proper examination (which diagnostic imaging to use).
  2. Directing the radiology technicians on which diagnostic imaging to use.
  3. Assessing the images to recommend any other examinations you may need.
  4. Correlating findings from all diagnostic images.
  5. Recommending treatment, or directly treating diseases using radiation or interventional radiology, based on the images.[i]

Properly assessing a diagnostic image is imperative to a positive patient outcome. Unfortunately, there are two major challenges that confront most radiologists. First imaging technology has advanced rapidly. Much of the advance is not just better and safer “picture taking” hardware, but in the software that takes the raw data coming off the scanner and creates readable images that the radiologist can evaluate.

Putting these images together is not a simple task, as it takes a “chain of handcrafted signal processing modules that require expert manual parameter tuning and often are unable to handle imperfections of the raw data, such as noise.”[ii] What these systems see as noise may actually contain relevant data that is lost to the radiologist. It’s estimated that the human eye picks up less than 10% of the data available on the image. Knowing what data is potentially relevant is therefore a function of the learning the system designers build into the algorithms.[iii] Second, there is a plethora of image types and an enormous number of pathological circumstances that can make images hard to read. Tumors, for example, can be entangled in many physical structures, or not. The machine can present them in color or black and white, 3D or flat. The problems of false readings are further compounded by the infrequency of complexity that characterizes the practice of most radiologists.

ML, and AI have the power to drastically improve these challenges. AI methods are beginning to recognize complicated patterns in imaging data that the human eye could not catch. Further, the ability of machines to incorporate massive amounts of clinical data and marry it with all the visual data available through advanced imaging, allows the machines to learn in ways that provide both better images, but also more accurate diagnoses. In this process, AI could provide quantitative assessments that have generally been out of the reach of most radiologists.[iv]

AUTOMAP is a new AI approach used by MGH Radiology. It’s is able to create higher quality images using less data than a radiologist would need when creating the imaging himself/herself, and takes less time, which is optimal for a doctors’ efficiency and the health of the patient (less radiation).[v]

MGH is also working to use AI in the next two years to understand the origins of many diseases and how they are linked to one another. This could help MGH provide preventative treatments to patients. They are working to use AI to find patterns in these images. Radiologists can then take the data from the thousands of images taken using AUTOMAP, see what the diagnostic and treatments were, figure out what the outcome from those images were, and then use that information to better understand what data is most relevant in creating new images and in turn, what those images most accurately reflect clinically. In other words, the more high-quality images and accurate diagnoses MGH has, the better they are to diagnose and treat diseases that they see in images in the future.

Lastly, MGH is working to establish a common language and network to share these images. They must build reference datasets and standardize the criteria for imaging protocols. Additionally, they will need to recruit and train qualified doctors and scientists this work. These professionals will play a huge role in creating the AI tools, and in optimizing them as the networks and protocols develop and grow.[vi]

There are two important questions that I am left with. The first is: to what extent does a doctor have final say about what the data from AI is telling them? For instance, if AI results are telling the doctor one thing, but the doctor’s own analysis and gut is telling them that the results are incorrect, what are the next steps that the doctor should take? Related, my second question is if there is a danger of doctors relying too heavily on AI, and having those results override their own judgement. Whose fault is it then if something goes wrong: the hospital’s or the doctor’s? (800)


[i] Radiological Society of North America, et al. “What Does a Radiologist Do?”, 1 Apr. 2017,

[ii] “Learning to See: New Artificial Intelligence Technique Dramatically Improves the Quality of Medical Imaging.” Massachusetts General Hospital, Mass General Imaging, 21 Mar. 2018,

[iii] American Journal of Roentgenology. 2017;208: 754-760. 10.2214/AJR.16.17224

[iv] Hosny, A, et al. “Artificial Intelligence in Radiology.” Current Neurology and Neuroscience Reports., U.S. National Library of Medicine, Aug. 2018,

[v] “Learning to See: New Artificial Intelligence Technique Dramatically Improves the Quality of Medical Imaging.” Massachusetts General Hospital, Mass General Imaging, 21 Mar. 2018,

[vi] Radiology Leaders on Artificial Intelligence: Get in Front.” Mass General Advances in Motion, 7 June 2018,



When HHS becomes iOS: Harnessing Open Innovation to Combat the US Opioid Crisis


Detroit and the Future of the Automobile: Will Additive Manufacturing Help GM Beat Silicon Valley at its Own Game?

Student comments on Mass General Hospital Radiology

  1. Melanie —

    I thought your article on how ML and AI are assisting with higher quality imaging and more accurate diagnoses for radiologists was extremely interesting. You posed challenging questions on the extent in which doctors may rely on ML and AI, and at what point, should a medical error occur, should the blame be placed on the machine versus the human. Hopefully, at the end of the medical analysis, given the higher quality imaging and predictive diagnosis, if a radiologist has conflicting thoughts, then second opinions could be acquired to assist with final recommendations. Ultimately, while ML and AI can be extremely reliable and an efficient method for analysis, I believe that the human is still responsible for recommending a certain procedure. Further, there may be other emotional or external factors that could impact the recommendation, and I believe that the doctor will be able to more appropriately assess such complex elements. Thanks!

  2. This is amazing work and I’m very excited for the future of medicine reading about the advancements you’ve described. In some ways I hope that we can generalize the principles of AUTOMAP, effectively using AI to extend our human capabilities and achieving better patient outcomes by combining unique insights provided by physicians and algorithms. This relates to your questions about disagreements between physicians and AI and here I think advancements in AI that can provide more explanation will be crucial. One solution may be systems that will provide a diagnosis but simultaneously highlight in green the particular parts of the image that the algorithm used to support one diagnosis versus highlighting in red the parts of the image that the algorithm used to support a different diagnosis.

Leave a comment