Take Step 1 back to square 1

US physician trainees have to take multiple extremely high-stakes exams that cause significant stress. As of 2022, the most critical exam, USMLE Step 1, has become fully pass/fail, meaning program directors lose one of their most important data points about residency applicants. Will this help medical student wellness as intended or will there be more unintended consequences?

As described in Moon’s Forbes article, in February 2020, the FSMB and NBME announced that the Step 1 exam will become pass/fail in 2022. The diagram below summarizes the steps and key exams in the majority of allopathic (M.D.) physician training pathways in the US. Each of the exams (MCAT, Steps 1-3, board exams) cause an extreme amount of pressure on high-achieving trainees in a competitive environment roughly every 1-4 years.

Summary of exams taken in training pathways of most allopathic (M.D.) physicians in America. Colored blocks indicate steps in training, and arrows below indicate when exams are usually taken. The first 2 years of medical school are usually spent in classrooms studying the science and basic concepts of medicine (biochemistry, statistics, public health, anatomy, how healthy bodies work, how diseases can alter these systems, etc). Step 1 is supposed to test for minimum competency of these foundations before transitioning into hospitals. Of note, some medical schools do have significant variations in their curricula, e.g., some medical schools have had their students take Steps 1 and 2 both after clinical core rotations. It is impossible to predict what most schools will do after 2022 because it takes time to make major changes in medical school curricula since schedules of each class of students affects other classes of students and limited resources (e.g., there is a limit on how many students can be doing clinical rotations at the same time). You may also notice that Step 2 CS disappears after 2022. Step 2 CS (clinical skills) was an 8 hour exam with standardized patients to assess abilities in History & Physical exam (verbal and physical interactions with patients). It required students to travel to 1 of 5 testing centers in the entire US. It was put on hold during the COVID-19 pandemic, and then permanently canceled in 2021.

Step 1 was designed as a minimum competency exam. It has always been “pass/fail”, but a numeric score and percentile were also reported. Competitive residency programs, lacking other measures to compare applicants, relied on Step 1 scores to help rank applicants for the Match. Less competitive residency programs relied on scores to try to predict trainees who may fail specialty board exams in the future. A chart summarizing results from NRMP’s annual report on Match results are shown below; see also complete Excel files of AAMC’s Tables B1 and B2.

Median and interquartile range of Step 1 scores of matched and unmatched residency applicants by specialty. Note the significant difference between matched applicants in very competitive specialties like dermatology (~250) and less competitive specialties like family medicine (~220).

If the problem was student stress/wellness and residency programs inappropriately using Step 1 scores, they could still allow students and schools to see their score details and only include the pass/fail status in residency applications. It can be helpful to students and schools to see how they perform compared to their peers nationally and help improve pre-clinical education.

They should revise the content of Step 1. As a result of better resources in the highly profitable test prep industry, students’ scores and the minimum competency “pass” threshold have both risen over the years. In order to obtain a high score, students must understand core concepts but also memorize an absurd volume of obscure factoids that are irrelevant to 99.99% of physicians (e.g., which amino acid has 3 different titration points [histidine]). Step 1 questions should focus on the most fundamental concepts and clinically relevant content.

Residency programs need some objective measures that help compare students from around the world (international applicants). Without Step 1 scores, programs will have to rely more on other measures (e.g., school prestige, publications, leadership/volunteering, Dean’s Letters, Step 2 scores). Extracurricular opportunities can vary significantly among schools and among students; students who have family responsibilities, financial limitations, or work part-time during school would be disproportionately disadvantaged due to the significant time and resources required to rack up those activities. Dean’s Letter closings can be extremely subjective, and explicit or implicit in coding students’ rank within the school, and doesn’t help compare students from different schools. Before 2022, students who were disappointed in their Step 1 score had a chance to significantly improve their Step 2 score; students satisfied with their Step 1 score could delay taking Step 2 until after the Match. After 2022, all residency programs will probably require a Step 2 score. Students and schools will feel pressure to take Step 2 sooner with more weeks of dedicated study time to have better scores that are available in time for residency applications. All of these consequences could hurt students, schools, and residency programs.


Data

  • NRMP’s annual report, “Charting outcomes in the Match: senior students of US MD medical schools” PDF
  • NRMP’s data reports by year (e.g., Advance data tables, Match by the numbers, Press release) link to website with PDF links
  • AAMC’s annual report, “Table B1. Test scores and experiences of first-year residents, by specialty” link to Excel file
  • AAMC’s annual report, “Table B2. USMLE Step 1 and Step 2 CK scores of first-year residents by specialty” link to Excel file

Additional reading

  1. “USMLE Step 1 is now pass/fail – who benefits from this big change?” Moon K. Forbes, 4/7/20, https://www.forbes.com/sites/kristenmoon/2020/04/07/usmle-step-1-is-now-passfailwho-benefits-from-this-big-change/?sh=2b3ef0cf4873
  2. “Objective measures needed – program directors’ perspectives on a pass/fail USMLE Step 1” Makhoul AT, et al. NEJM, 6/18/20, https://www.nejm.org/doi/full/10.1056/NEJMp2006148
  3. “The step 1 exam is going pass-fail. Now what?” Chretien K. AAMC News, 1/11/22, https://www.aamc.org/news-insights/step-1-exam-going-pass-fail-now-what
  4. “How demands on residency applicants are growing” Murphy B. AMA blog, 5/12/21, https://www.ama-assn.org/residents-students/residency/how-demands-residency-applicants-are-growing
    1. “How will students, medical schools, and residency programs adapt to step 1 scoring changes?” Grady F, Jewell D. Academic Medicine, 2/2021, https://journals.lww.com/academicmedicine/Fulltext/2021/02000/How_Will_Students,_Medical_Schools,_and_Residency.5.aspx

Ellen Kim, MD, MPH, is a Clinical Informatics Fellow at Mass General Brigham, a Radiation Oncologist at Brigham & Women’s Hospital (BWH)/Dana-Farber Cancer Institute, an NLM BIRT fellow, and a student in HMS DBMI’s Master of Biomedical Informatics program. This summer, she will join the Department of Radiation Oncology at BWH as clinical faculty and as the Director of Clinical Informatics. Ellen can be reached at ellen_kim@dfci.harvard.edu.

Previous:

Improving Uber User Experience Through Causal Inference Analysis

Next:

The Gamification of Wellness at Work

Student comments on Take Step 1 back to square 1

  1. Hi Ellen,

    Thank you for your post. This is something that I am all too familiar with. We don’t have USMLE in Australia but use other surrogates to try and differentiate candidates into competitive training programs. There has been a reliance on obtaining extra master’s programs (MPH, MSc, MMed, MSurg, etc.) to add points to a candidate’s CV. What has resulted is that there has been degree inflation amongst candidates and a lot of people are getting ‘too many’ degrees. I agree that the Pass/Fail system doesn’t really solve the issue at hand, because institutions will look for or make their own alternative tests/exams to differentiate people. On the surface, candidates may think that it will help them, but in reality, the institutions may be making very biased decisions or ones without good evidence-based rigor. Would love to chat about this more to see what other suggestions you think may work!

  2. Hi Ellen,

    Thank you for writing this eye-opening piece on how the alteration in the USMLE Step 1 feedback that students and residency programs receive might have potential, unintended adverse effects on students and residency programs alike. In consideration of the purported goals of Step 1 being (1) to assess the competency level of medical students early within their education (be it post-completion of the pre-clinical coursework or the clinical core) and (2) to inform the development and refinement of medical education curricula to promote the competency of medical students, and the purported goal of FSMB and NBME’s 2022 feedback change being to improve student wellness by reducing student stress via the elimination of their Step 1 scores relative to their peers, I find your selective score disclosure proposal quite appealing and promising. This entire scenario calls to mind the feedback administration dilemma in the AMD case that we reviewed earlier in the semester. As with the change to a ratings-free program in the AMD case, I wonder how the FSMB and NBME went about contemplating and weighing the tradeoffs of the potential increase in bias and loss of the equitable function that the standardized Step 1 scores fulfilled with the potential gains in student wellness that may be brought about by eliminating Step 1 scores, and to what extent student feedback (like employee feedback in AMD) was involved in this making this change.

Leave a comment