Program Improves Scores, Knowledge Retention for Third-Year Residents

CINCINNATI—An analysis by University of Cincinnati faculty members shows that a multiple-choice testing program coupled with a novel year-long clinical experience helps internal medicine residents improve their scores on the Internal Medicine In-Training Exam (IM-ITE).

These findings, published in the advance online edition of the Journal of General Internal Medicine scheduled for print in November, validate the benefit of this and similar programs in improving internal medicine resident education at UC and possibly at other institutions. This may lead to a higher certification rate for new internal medicine physicians and better clinical care for patients. 

"The IM-ITE, given annually in October, is a validated examination that assesses internal medicine resident knowledge acquisition during training,” says Bradley Mathis, MD, associate professor in the division of general internal medicine, UC Health primary care physician and lead investigator on the study. "Many programs use the IM-ITE to counsel residents, to create individual remediation plans for residents and to make fundamental programmatic and curricular modifications.”

Mathis and colleagues in the department designed a rotation-specific multiple-choice testing program and a separate board review testing program administered during 12 months of ambulatory and inpatient care elective experience with the intent of improving residents’ knowledge.

"The purpose of this study was to determine the association between these educational interventions and IM-ITE percentile scores in third post-graduate year categorical residents who were exposed to the educational program,” he says.

Faculty in the division of general internal medicine developed this program, which involved administering pre- and post-rotation multiple-choice tests during the long-block (12 consecutive months of residency from November of Year 2 to October of Year 3 consisting of ambulatory care, inpatient and outpatient electives and research experiences with minimal overnight call) using examinations of 30 to 50 questions each.

They created the exams using the American College of Physicians’ Medical Knowledge Self Assessment Program and chose questions based on the previously defined set of goals and objectives for each rotation. Topics included cardiology, gastroenterology, hepatology, nephrology, hematology oncology, endocrinology, pulmonary medicine and rheumatology.

Residents took the pre-test during the first week of their rotation and received confidential feedback regarding their performance and were encouraged to discuss areas of weakness with their elective rotation attending physician to create a personalized learning plan for the month.

During the last week of the rotation, each resident received and completed a post-test that used different questions than the pre-test but covered similar areas. Then, residents were asked to reflect on progress made during the elective rotation and create further study plans for the areas in which they were having trouble.

In addition, working collaboratively with the American Board of Internal Medicine (ABIM), the faculty created six additional 60-question multiple-choice examinations based upon the self-assessment modules in the ABIM Maintenance of Certification program.

"If students fell short in their medical knowledge, they were required to meet with the administration to develop a personalized learning plan, which varied on learning styles but including scheduled text reading and review, review of exams and test taking skill, peer-guided study groups and introduction to alternative learning methods such as video and audio taped reviews,” Mathis says, adding that residents met with administration at least quarterly but had the opportunity to meet more often.

Participants included 104 residents. Forty-five participated in the study group during 2008-2009, and 59 residents in the three classes that preceded the use of the testing program, 2005–2007, served as controls.

"We analyzed the change in median individual percent correct and percentile scores between the first and second year and between the second and third year in both control and study groups,” Mathis says. "For our main outcome, we compared the change in median individual percentile rank between the control and study groups between the second- and third-year testing opportunities.”

Mathis says that after undergoing the testing and the patient care experience, the study group showed a significant increase (8.5 percentile points) in their score between the second- and third-year examinations.

"This was better than the increase of 1 percentile point in the control group,” he says. "These results mean that exposure to a broad range of medical information in a structured learning environment provided the necessary feedback and tools for individual improvement in medical knowledge, hopefully leading to higher ABIM certification scores and better long-term clinical performance for these physicians as well as better experiences for patients.”

Funding for this study was provided by a UC College of Medicine medical education research grant.