A data-informed approach to bar passage
A collaboration among UC researchers may be changing the conversation around bar passage
When Joel Chanvisanuruk, JD ’06, was hired as director of Academic Success and Bar Programs at Cincinnati Law in 2009, conventional wisdom pointed to incoming student credentials—notably, LSAT scores and undergraduate GPAs—as the best indicators of bar passage.
Now, thanks to a collaboration with Amy Farley and Christopher Swoboda, both of who are professors in UC’s School of Education, that conventional wisdom is being called into question. And they have the data to back it up.
In 2019, Farley, Swoboda, and Chanvisanuruk published A Deeper Look at Bar Success: The Relationship Between Law Student Success, Academic Performance, and Student Characteristics in the Journal of Empirical Legal Studies. The article examines five years of Cincinnati Law alumni data, including course performance and detailed bar score reports collected by Chanvisanuruk, in an attempt to understand and quantify predictors of bar success.
Their results suggest that bar passage can be predicted by a wide set of variables, while LSAT and undergraduate GPA are in fact weakly predictive. They found information from the first year of law school – even a student’s performance in just one first semester course – explains significantly more variation in bar passage.
“When we were embarking on this, we heard a lot from other schools that the way to increase bar passage was to essentially get better students at your law school. We heard that over and over again,” said Farley. “What we found in our data was that, at least among admitted students, LSAT, incoming GPA, the rigor of undergraduate institutions, a student’s race, gender, age, everything we know about them, tells us very little about who is going to be successful on the bar.”
Collecting the data
Farley, Swoboda, and Chanvisanuruk’s analysis was possible in part because of a decision Chanvisanuruk had made years prior.
While law schools generally get a report on who passed and who failed the bar exam, Chanvisanuruk knew, from having taken the Ohio bar, that test-takers were sent a more detailed score report breaking down their performance on each section of the exam. In order to get a more complete sense of graduate performance, Chanvisanuruk sought approval from the Supreme Court of Ohio, which administers the Ohio Bar Exam, to create a student waiver to gain access to the detailed score reports. When that was approved, Chanvisanuruk began collecting the data that would eventually inform his work with Farley and Swoboda.
“We pushed the analysis past that binary of pass/fail into where they are on the continuum,” said Chanvisanuruk. “Then we’re also able to look at subscales. How did they perform specifically on the multiple choice, on the essays?”
“Something that I think sets our research apart that we've learned as we've been doing this work is that most law schools don't have the kind of incredible data access that the University of Cincinnati has,” said Farley. “Joel [Chanvisanuruk] was really smart and innovative in thinking about partnering with the bar and getting student permissions to be able to access their bar passage data.”
We pushed the analysis past that binary of pass/fail into where they are on the continuum.
Questions lead to collaboration
As part of his portfolio at Cincinnati Law, Chanvisanuruk spearheaded a number of programs designed to increase students’ chances at success in law school and, ultimately, on the bar exam.
A few years after being hired, he’d introduced Structured Study Groups (SSGs), a sort of peer-mentorship program pairing groups of 1Ls with upper-level students. The groups meet twice a week and focus on developing and applying study skills specific to law school as well as strategies to improve exam preparation and performance.
“The idea is to take all that stuff that’s unwritten (which you’re just expected to know), and make it explicit,” said Chanvisanuruk.
Chanvisanuruk had also created a bridge program available for incoming 1Ls prior to orientation that was designed to introduce fundamental concepts and skills necessary for success in law school. In addition, he was teaching an upper-level bar writing course available to third-year law students.
In order to better understand whether these programs were having the intended effect, Chanvisanuruk, who, aside from having earned his JD from Cincinnati Law, also has a master’s degree in Public Affairs from the O’Neill School of Public and Environmental Affairs at Indiana University Bloomington, began enrolling in graduate statistics courses at the University of Cincinnati’s School of Education.
“I knew I wanted to take statistics, specifically in educational evaluation, so that I could better understand whether the programs were having the affect I wanted them to have,” he said.
He came to the statistics courses with two questions. He wanted to understand whether the programs he was managing were having the intended effect. But also, he said, there was the question of when, and to what degree of certainty, you could predict the ultimate bar passage of a student.
It was in one of those classes that for the first time he ran a multivariate regression analysis—a method of statistical analysis used to measure the degree at which more than one independent variable (e.g. predictors) and more than one dependent variable (e.g. outcomes), are linearly related—comparing students’ performance in their first-year courses and their ultimate bar passage. By that point, he’d collected three years of detailed score reports from Cincinnati Law graduates who’d taken the Ohio Bar Exam.
That initial analysis led to the collaboration with Farley and Swoboda, who collectively have expertise in applied statistics, quantitative methods, higher education, student success, and equity and opportunity.
We had a reviewer say, basically, ‘this is really different than what we know.' Like, you guys need to talk about this a lot more because this goes against what everything else says.’
Getting to work
With the support of grants from the University of Cincinnati and the AccessLex Institute (the President and CEO of which is Cincinnati Law alumnus Christopher Chapman, JD ’93), Farley, Swoboda, and Chanvisanuruk set out to better understand the various factors that lead to a student’s success on the bar exam.
Farley says she was motivated by the larger context around law schools and the bar exam.
“It's this very interesting little microcosm of much of the things we know about higher education,” she said. “We have this competitive, challenging and rigorous small community of learners. We have the ultimate high-stakes assessment at the end—the bar exam. A high pressure, consequential assessment that has huge consequences for not only the students themselves, but also for the field.”
Farley said both she and Swoboda were also excited at the amount of data available, including the five years of detailed score reports Chanvisanuruk had collected by that time, as well as course performance data.
“Having access to not only who passed the bar, but subscale data about their performance on the bar is almost unprecedented in legal education,” said Farley. “That allows us to do a lot of really complex analysis around looking at what are the things that contribute to bar success.”
Their work together led to the publication of A Deeper Look at Bar Success: The Relationship Between Law Student Success, Academic Performance, and Student Characteristics. They found that, while a student’s final law school GPA is most predictive of bar success, their first semester GPA was significantly predictive. And, in contrast to previous studies, their data indicated that the LSAT and undergraduate GPA were only weakly predictive of first‐time bar passage among admitted students.
“We had a reviewer say, basically, ‘this is really different than what we know,’” said Farley. “Like, ‘you guys need to talk about this a lot more because this goes against what everything else says.’”
“All the information that is collected at the point of admission pales in comparison to a student’s performance in Contracts in terms of predictability,” said Chanvisanuruk. “The fact that we can intervene so very early makes it really exciting.”
The data suggested that a student who was predicted not to pass based on their first semester GPA could remedy that by taking more upper level bar courses. At Cincinnati Law, their findings resulted in a policy change and modification of Rule 9 of the JD Academic Rules. Under the rule, students who perform under a 2.8 after their first semester or their first year of law school are required to take five upper-level bar courses, as well as a newly created, skills-based ethics course in their second year of study, and an upper-level bar writing course in their final semester of study. The rule went into effect for the graduating class of 2021.
In qualitative follow-up and interviews with students, Farley said they found students were less likely to enroll in upper-level bar courses if they hadn’t performed as well as they would’ve liked in their first year.
“We found that there's sort of these perverse incentives around taking upper level bar courses,” said Farley. “Students who don't do well in their first year have a strong desire to try to improve their overall GPA. And some of the calculus that they do around that involves taking less rigorous coursework to increase their GPA. But what that does, we found, is students who take fewer upper level bar courses are less likely to pass the bar. So, there's potentially this unintended consequence where underperforming students are less likely to take these upper level courses.”
Looking to the future
What’s novel in Chanvisanuruk’s approach to academic success and bar passage is its interdisciplinary quality. When Chanvisanuruk started, he said, it was uncommon for law schools to partner with researchers in education to better understand factors that contribute to bar passage.
Now, with five other law schools in the process of replicating the same study that Farley, Swoboda, and Chanvisanuruk have done at UC, their collaborative, data-informed approach may mark a sea change in how law schools go about preparing students for the bar exam.
“It can be hard for institutions of higher education to share their data and to be open to letting other people into their world,” said Farley. “I think that's not necessarily the norm, although it is changing in legal education. I really applaud [Cincinnati Law] for their commitment to serving all students. And I think UC is really seen across the country as a leader in this space.”