We’d have thought this was blindingly obvious.. maybe not?
Low-performing law students scored higher on final exams when given access to artificial intelligence, while their high-performing classmates performed worse when using the technology, a new study at the University of Minnesota found.
Researchers compared the final exam scores of 48 students in two courses—Introduction to American Law and Legal Reasoning and Insurance Law. The students first took the final without AI, then took a second, different final using GPT-4, the latest large language model from Open AI.
They found that GPT-4, which produces human-like text based on user prompts, vastly improved student performance on multiple-choice questions. Students saw a 29 percentage-point improvement over their actual exam scores, which were earned without GPT-4. Those gains were especially pronounced among low-performing students, who saw a 45 percentage-point increase in their exam scores with AI.
But GPT-4 didn’t help students score higher on the essay portion of the exams, and top students didn’t see any score improvement when using AI. In fact, their exam scores were about 20 percentage-points lower when using GPT-4.
“This suggests that AI may have an equalizing effect on the legal profession, mitigating inequalities between elite and nonelite lawyers,” reads the study released this month by Minnesota law professor Daniel Schwarcz and University of Southern California law professor Jonathan Choi.
It’s the latest entry in a growing body of research looking at AI within legal education. Earlier studies found that the exam scores of the previous version of GPT-4 matched those of mediocre law students, while a newer one found that GPT-4 can pass the bar exam.
Schwarcz told Reuters on Tuesday that AI access in his new study could have made high-performing law students somewhat lazier on their exams, or that the technology may have made them less likely to tap their legal reasoning skills.
“Once someone frames an issue for you, you sort of lose the cognitive mindset of independently assessing it,” he said.
