By BRYAN CARMODY, MD
Well, it happened.
Beginning as soon as 2022, USMLE Step 1 scores will be reported pass/fail.
I’m shocked. Starting around two weeks ago, I began hearing rumors from some well-connected people that this might happen… but I still didn’t believe it.
I was wrong.
The response thus far has been enormous – I haven’t been able to clear my Twitter mentions since the news broke. And unsurprisingly, the reaction has been mixed.
In the future, I’ll post more detailed responses on where we go from here – but for now, I’d like to emphasize these five things.
1. By itself, making USMLE Step 1 pass/fail doesn’t fix much.
Simply getting rid of three digit scores doesn’t improve medical education. And it doesn’t make residency selection any better, either.
It does give us the opportunity to make changes. And the importance of that should not be understated.
Put simply, this is the greatest opportunity for medical education reform since Flexner, and the greatest opportunity to re-design residency selection since… ever.
2. The path of least resistance does not take us where we want to go.
Most of the commentary on social media thus far has taken the following form:
- Now only Ivy League medical students will get into competitive specialties.
- Don’t worry! USMLE Step 2 CK will just be the new Step 1.
- I guess DOs and IMGs can forget about getting into top residency programs.
And you know what? Most of these concerns are justified.
If we do nothing, Step 2 Mania is the natural result of a pass/fail Step 1. And unless we give program directors more useful information – and the time to thoughtfully review all the applications they receive – they’ll gravitate to another convenience metric out of necessity.
But that does not mean it’s inevitable.
3. We all have a say in what comes next.
If the rise and fall of a scored Step 1 teaches us nothing else, it should clearly demonstrate these two things.
First, advocacy works. There is no way that the NBME’s entrenched, out-of-touch bureaucracy would have ever voluntarily changed Step 1 to pass/fail – unless their feet had been repeatedly been held to the fire by the public over the past year. Change is possible – but you have to speak up.
Students, I’m looking at you – especially those from “non-elite” or international medical schools. You have a say in what comes next. How should residency programs evaluate you? Surely, answering multiple choice questions with peripheral relevance to real patient care is neither the best nor the only way to identify your talent. It’s time to dream bigger – and demand more meaningful evaluation.
Second, medical students will rise to whatever bar we set for them.
Ask them to memorize an 800 page review book so they can excel on a multiple choice question test of basic science esoterica? Done and done. Students knew most of what they were learning was pointless – but they worked tirelessly because we told them it mattered.
So program directors, now I’m looking at you. Whatever you tell students to do, they’ll do. You want high Step scores? Students will give them to you. But what really matters to you? Do you want residents who will serve the underserved? Contribute to research? Arrive on July 1 with specialty-specific knowledge? Whatever you say, goes. If you choose to grump about the loss of the bad old days, you’re abdicating your authority to set the standards you want.
4. No one should mourn the loss of a scored Step 1.
Look around my Twitter mentions, and you can find touching testimonials mourning the demise of Step 1 scores. There are tales of useful factoids brought to bear on patient care decades later; of students inspired to consume as much (mostly useless) scientific knowledge as possible; of doors opened and careers launched because of one shining moment on test day years ago.
Give me a break.
The Step 1 score was the biggest false god in medical education, and no one should be too sad that it’s going away.
Was it objective? Sure. But the ability of three-digit scores to discriminate between applicants was actually fairly imprecise.
Did it test some useful concepts? Of course. But scoring highly required memorizing esoteric information that would never benefit an actual human patient.
Did some students benefit from scoring highly? Absolutely. But did Step 1 scores really “level the playing field” for students from international or non-prestigious medical schools overall? I doubt it.
Did it give program directors an easy way to screen applications? Yup. But was that way meaningful? Probably not. The main value of Step 1 was that it gave you a number. But if the content of the test doesn’t matter, why not just have medical students throw darts, or race go karts, or memorize digits of pi?
And of course, our idol worship of Step 1 came at a significant opportunity cost. The fact that most of us turned out okay doesn’t prove that Step 1 was the best way, or that that we wouldn’t have been better off under another system.
Here’s the thing:
Whether you liked a scored Step 1 or not, the current system is not what any of us would have designed if we were building something from the ground up.
In the scored Step 1 era, we sat back and watched as preclinical medical education turned into a glorified USMLE prep course with a five-figure/year tuition bill attached. We did nothing as residency applications spiraled out of control and program directors’ outsourced their most important decision-making to the NBME.
We shouldn’t be proud of any of this. And we shouldn’t mourn the loss of the three-digit score that enabled us to neglect the need for reform for so long.
More importantly, there is no turning back now.
If you opposed #USMLEPassFail, do your grieving for the scored USMLE – and then get to work. Saying “I told you not to make Step 1 pass/fail!” is a hollow victory. There is an opportunity here to engage our students in tasks that could actually lead them to be better physicians – but we have to seize it.
And if you supported #USMLEPassFail, now is not the time to dance on the three-digit score’s grave. We need to get to work, too – or this hard-won victory will go for naught.
5. The way forward… my opinion
Expect more from me on these topics in the future, but for now, let me put these three thoughts in your head.
WE NEED MEANINGFUL EVALUATIONS.
My 5th grader’s report card provides more useful information than any medical school transcript I’ve ever seen. He doesn’t get an “A” in math – I get to see a lengthy list of math skills and an assessment of where he’s at with all of them. Maybe he’s comfortable dividing fractions, but still developing proficiency with using variables to set up algebraic equations.
Letter grades are nice – but they don’t tell me what he knows and what he doesn’t. (I don’t get a histogram of other students’ performance and where he falls – because how other students performed is kind of irrelevant if my goal is to help him become better in math.)
The fact that elementary schools provide more information-rich evaluations than medical schools is shameful. In the past, we’ve justified this because of Step 1. Starting in 2022, that excuse will ring hollow.
Let’s be honest: there should never be a medical school evaluation in which a students is exceptional in every area. (If there is, it tells you more about the school than the student.)
Our goal should be to make medical school formative, not performative; to help learners maximize their potential without fearing failure; to provide and communicate honest feedback that leads to improvement and growth along the spectrum of medical education. (For their part, program directors need to focus less on relative accomplishment and more on potential to grow and thrive in that program/field.)
We don’t need another arms race. We do need more thoughtful evaluation. We should require students to engage in tasks that leave them – and their future patients – better off. And schools that refuse to participate in this process should be penalized by program directors and medical school applicants alike.
WE NEED HOLISTIC REVIEW.
I don’t think we need a one-size-fits-all metric to tell us who are the “best” applicants. Honestly, we don’t.
Certain talents and traits get you a lot farther in certain disciplines than others. Even within the same discipline, different programs have different needs and serve different missions. What makes a good doctor? It’s complicated. And trying to reduce that complexity down to a single three digit number is silly.
Holistic review is possible. It’s what we do at the residency program at which I’m an associate program director. But it’s labor intensive – and it requires human judgment. That last part makes some people uncomfortable, and I understand why. Holistic review has the potential to devolve into bias and cronyism. But that’s not a reason to not do it – it’s a reason to strive to do it well, to make decisions deliberately, and to build some checks and balances to make sure you stay aligned with your values.
WE ALSO NEED APPLICATION CAPS.
The only way to get away from screening metrics and treat every applicant as an individual is to limit the pile of applications that program directors receive.
Look, I’ve preached this sermon before. But you’d better believe I’m gonna be carrying this particular gospel to the people again in the near future.
Afraid of application caps? Worried that they’ll disadvantage students from non-elite schools? Stay tuned for a data-driven expose of the misinformation surrounding caps coming soon.
Dr. Carmody is a pediatric nephrologist and medical educator at Eastern Virginia Medical School. This article originally appeared on The Sheriff of Sodium here.
The post #USMLEPassFail: A Brave New Day appeared first on The Health Care Blog.