Educators consider medical students' performance on the United States Medical Licensing Examination, or USMLE, Step 1(www.usmle.org) to be a measure of how well students learned basic science and medical concepts. So, the University of Texas Medical Branch, or UTMB, School of Medicine in Galveston was pleased when failure rates on this critical examination plunged by more than two-thirds after the school implemented comprehensive curriculum changes. An even greater drop in failure rates was seen among underrepresented minorities.
In a paper presented at the recent annual meeting of the American Educational Research Association, UTMB faculty members said the failure rates of students the first time they took the exam declined from 7.5 percent in the classes matriculating in 1995-97 to 2.3 percent in the classes starting in 2003-05 -- a drop of 69 percent.
In addition, mean scores on Step 1 increased by 14 points, with improvements across the entire range of student scores on the Medical College Admission Test, or MCAT(www.aamc.org), according to faculty members.
The difference, said the educators, was that the 2003-05 classes were exposed to an integrated medical curriculum that emphasized problem-based learning(www.studygs.net) as opposed to traditional rote memorization and lectures.
Lead author Steven Lieberman, M.D., vice dean for academic affairs and a professor of internal medicine at UTMB School of Medicine, told AAFP News Now that both administrators and faculty members were surprised at the enormous improvement after the comprehensive curriculum changes. The paper describes Step 1 scores as "critical" outcomes that are used for residency selection.
Although the Medical College Admission Test, or MCAT, has been revised several times since it was first administered in 1928, students' scores on it do not necessarily predict how well they will do on the United States Medical Licensing Examination, or USMLE, Step 2(www.usmle.org), which students take during their fourth year of medical school.
In fact, according to a study in the June issue of Academic Medicine(journals.lww.com), a decline has been seen in the predictive validity of more recent versions of the MCAT on USMLE Step 2 scores. USMLE 2 tests students' ability to apply medical knowledge and skills to patient care.
The study examined the USMLE scores and other performance predictors of 7,859 students in 36 classes entering Jefferson Medical College in Philadelphia between 1970 and 2005. The students were divided into three groups according to which MCAT version they took: the pre-1978 version; the version used between 1978 and 1991; and the latest, post-1991, version.
Overall, the authors noted, "the MCAT predictive validity coefficients obtained in this study are encouraging and support the use of the MCAT scores in the screening of a large applicant pool."
However, they added, "there was an unexpected pattern of systematic decline in the validity of the MCAT in predicting Part II/Step 2 performance."
"This downward trend is important, given the ongoing consideration by the National Board of Medical Examiners to have only one licensing examination during undergraduate medical education, most likely to be used for evaluation at the time that the Step 2 examination is now taken."
"Although the findings support the short- and long-term predictive validity of the MCAT, opportunities to strengthen it remain," the study's authors concluded. "Subsequent revisions should increase the test's ability to predict performance on United States Medical Licensing Examination Step 2 and must minimize the differential validity for gender."
The study comes at a time when the Association of American Medical Colleges is reviewing the MCAT(www.aamc.org) and expects to come out with a revamped exam no earlier than 2013.
"Based on previous literature, we didn't expect improvement in Step 1 scores or failure rates," Lieberman said. "But most of the previous reports addressed isolated changes in parts of educational programs, whereas, we coordinated changes in multiple aspects.
"This gets at the major point of our findings: There is no single modification that explains the changes in performance. It's the fact that there were coordinated changes among multiple subcomponents of our system that led to improvements that have not been seen in isolated interventions."
According to Lieberman, medical educators are keenly interested in making comprehensive changes in medical education, but many end up focusing on a single innovation because of institutional factors such as departmental autonomy, overburdened faculty, limited resources and institutional inertia.
Lieberman said that in 1998, the medical school started implementing changes to the curriculum during the program's first two years -- the basic science/preclinical years -- to modernize the program. The changes were implemented gradually and continue to be fine-tuned, he said.
One central change was a hybrid set of teaching methods that emphasized problem-based learning and active, student-centered learning and inquiry. The methods also highlighted self-direction, applicability of material to students' careers and solving authentic problems.
Tests stressed knowledge integration and the application of concepts learned to clinical scenarios. Examinations were rewritten in the style of the Step 1 exam so students could gain deliberate practice in taking Step 1.
The medical school made other changes, as well. To help students who were having difficulties with the new teaching and testing styles, the school launched a support system that included peer tutoring and professional academic counseling. Students at high risk for academic difficulty were identified early and offered additional preparation and support.
The results were striking:
- Only 2.3 percent of students in UTMB's integrated medical curriculum failed on their initial Step 1 attempt, while the national failure rate on the Step 1 exam during the same period was 6 percent to 7 percent.
- Compared with 9 percent of female students in a traditional curriculum, only 1.9 percent of female students in the integrated curriculum failed on their first Step 1 attempt, a decrease of almost 79 percent. Failure rates among males declined from 6.6 percent for those in a traditional curriculum to 2.8 percent among those in the integrated program, a decrease of nearly 58 percent.
- Among underrepresented minorities (e.g., black, Hispanic, Puerto Rican and American Indian/Alaskan native students), the Step 1 failure rate decreased by about 77 percent, from 16.6 percent to 3.9 percent. Black students showed the greatest improvement, with the failure rate decreasing by 94 percent, from 25 percent to 1.6 percent.
- Mean scores on Step 1 among students in the integrated medical curriculum increased 14 points compared with students in a traditional curriculum. The increase was even larger, at 14.6 points, among underrepresented minorities. Black students had the highest gains at 20.8 points. In fact, students across the entire range of MCAT scores benefited from the integrated curriculum.
The increases in mean Step 1 scores are "the largest reported in the literature and are seen across the entire range of MCATs, indicating that our approach benefits students independent of their previous academic performance," Lieberman said.