Law Sites rerports
The legal technology company Filevine has launched an AI-native citator and a companion brief-checking tool inside LOIS, its Legal Operating Intelligence System AI platform, that its CEO says perform as well as or better than the leading legal citators from LexisNexis and Thomson Reuters.
The citator has two core features. One checks a filed or draft brief for hallucinated citations, altered quotations, and mischaracterized authority. The other checks whether a proposition is still good law by letting a lawyer highlight a specific passage in an opinion and find out whether later courts have undercut the holding expressed in the passage.
Although Filevine had previously announced the citator in June as part of its new LOIS Legal Research offering, which provides access to millions of opinions from U.S. federal and state courts, CEO Ryan Anderson, in a briefing this week, told me the company had not heavily promoted the citator until it believed the product was fully ready for customers to use.
The company had not previously announced the anti-hallucination tool, which is being released for the first time this week.
As Filevine described the citator in its June release: “While legal verification tools rely on explicit citation relationships and apply treatment signals to entire opinions, LOIS Legal Research operates at the user-selected passage level.” This enables legal professionals to verify the exact rule they plan to rely on in their argument.
The Challenge Of Citators
I have previously written that developing a strong citator tool is the Holy Grail of a legal research platform. Among companies to have succeeded in developing their own in recent years were vLex and Paxton, Descrybe, and Midpage, and if you read those linked articles, you will see that each of them took distinct approaches to the problem. (My Midpage article only teased its forthcoming citator, which it has since released.)
But it is not an easy task. In fact, in my briefing with Filevine this week, John Rizner, the product manager who led the development of the citator, pointed out something that most law librarians will know but that may be news to almost everyone else, which is that the two citators lawyers have long considered to be the gold standard routinely disagree with each other about whether a case is still good law.
He pointed to a 2018 study in the Law Library Journal by Paul Hellyer, a librarian at William & Mary Law School, who reviewed 357 citing relationships that at least one of Shepard’s, KeyCite or BCite had labeled negative.
All three citators agreed that the treatment was negative only 53 times, meaning that in 85% of the relationships in Hellyer’s sample, the three citators did not agree that a negative treatment had occurred.
Hellyer also examined whether the flags that did appear were correct. Counting relationships the citators omitted or mislabeled as positive or neutral, together with those labeled negative but incorrectly described, he found overall failure rates of 33% for Shepard’s, 38% for KeyCite and 72% for BCite.
Rizner also cited a second line of research, from Susan Nevelow Mart, the former director of the law library at the University of Colorado, whose 2013 examinations of digest and citator results found that the overlap between the Westlaw and Lexis citator systems on relevant results was small – only about a third.
As the abstract of Mart’s article put it: “Neither algorithm is doing a very good job of finding all the relevant results; the overlap between the two citator systems is not that large.”
Ahead of our briefing, Rizner ran his own test. He pulled Freedom Mortgage Corp. v. Engel on both platforms and filtered each down to negative treatment in reported opinions. Westlaw found 28 and Lexis found 13. The overlap between the two sets, by his count, was about 41%.
“If my law firm happens to have Lexis, I might be missing opinions because I don’t have Westlaw, or vice versa,” Rizner said.
What that means, Anderson said, is that even though the legacy citators have been best in class up until now, that does not mean that they do not make mistakes.
Citators As An Essential Tool
At this point, you may be thinking: “Sure, but what does this all have to do with Filevine, which is not a legal research platform?”
Read More
