Ambrogi reports….
I have made no secret of the fact that I consider litigation analytics to be one of the most important technologies to have gained traction in recent years. Writing about analytics a year ago on Above the Law, I titled the post, This Tech Can Turn the Tables in Litigation. In my year-end summary of the most important legal tech developments of 2018, my top item was “analytics become essential,” noting, “We could be nearing a point where it would be malpractice for a lawyer not to use analytics.”
While all of that still holds true, and while the technology has continued to evolve and the array of products to expand in the year since then, it is important for legal professionals to understand that litigation analytics is still a developing technology, and that there are weaknesses in both the technology and the data on which they rely.
That means that these products differ not only in the kinds of analytics they offer, but also in the results they deliver.
The differences among these products were dramatically highlighted by a study conducted earlier this year by a group of law librarians who compared federal court results across seven legal analytics products. The products they compared were: Bloomberg Law, Docket Alarm Analytics Workbench (from Fastcase), Docket Navigator, Lex Machina, Lexis Context, Thomson Reuters Monitor Suite and Westlaw Edge.
Consider this seemingly simple research query: “In how many cases has Irell & Manella appeared in front of Judge Richard Andrews in the District of Delaware?” When the librarians tested that query across the seven products, they got widely different answers.
The librarians tested 16 “real world” research questions across the seven products, with similarly varying results. Although they have not yet published their findings, four of the participants — Diana Koppang, director of research & competitive intelligence at Neal, Gerber & Eisenberg; Tanya Livshits, director of research services at Irell & Manella; Kevin Miles, manager of library services at Norton Rose Fulbright; and Jeremy Sullivan, manager of competitive intelligence & analytics at DLA Piper — presented their findings during an analytics “deep dive” session in July at the annual meeting of the American Association of Libraries.
(I was a participant in that panel, as was Jean O’Grady, who wrote about the analytics study in a post at her Dewey B Strategic blog: What Do Law Firms Need to Know About Buying Litigation Analytics Products?)
The librarians limited their study to federal district courts, even though some of these platforms also provide analytics for state courts. They also limited their focus to docket analytics, meaning analytics based on federal docket data, not content analytics that focus on language and citation patterns in judges opinions, such as Precedent Analytics in Westlaw Edge and Context from LexisNexis.
One reason for the divergent results across platforms, the librarians found, was what they called “the PACER problem.” In a nutshell, the problem with PACER data is that it is replete with problems. PACER data is full of typos. The Nature of Suit codes used to label types of matters often do not reflect the true focus of a matter. Names of parties, firms and attorneys are not normalized for consistency. Attorney moves sometimes result in matters being associated with the wrong firm.
(The problems with PACER data were discussed in greater depth in a guest post here earlier this year by Josh Becker, chairman of Lex Machina and head of legal analytics at LexisNexis: Guest Post: Not All Legal Analytics Tools Are Created Equal.)
There are also major differences among the products in the search options and types of analytics they provide. For example, here are how the products compared in their ability to show analytics regarding case and motion outcomes
Red more results and rest of Bob’s article at
Legal Analytics Products Deliver Widely Divergent Results, Study Shows
