In a world overflowing with information, knowledge has never been so available. At the same time, wading through it all requires ever-increasing effort, with Google, Microsoft (via Bing), Apple (via Siri), and Amazon (via Alexa) investing enormous resources into tools that organize and understand the world.
While these companies have made great progress on general purpose search, there haven’t been the same advances in specialized domains like law. The approaches needed to access legal information are subtly different, and require distinct techniques and technologies. For example, the average legal query is significantly more complex and more likely to be looking for a needle in a haystack than the typical Google query. This puts an extra onus on legal search providers to research and invent techniques better suited for the law.
Today Judicata is publishing data and information that we hope will seed a benchmark for legal search and drive qualitative improvements across the industry. Just as the ImageNet dataset helped drive dramatic improvements in computer vision and deep learning, we’d like to see a growing collection of legal data be used to fuel advances in legal research tools.
Motivating this decision are the following beliefs:
- The quality of legal search query results can be objectively measured.
- The performance of legal search engines on those queries can be better or worse.
- The best legal search engines can and should perform well on all queries.
We encourage other legal search providers to contribute their own data, as a collective effort will make us all better in the long run. An open benchmark moves everyone forward because it allows for objective evaluation which makes improvement possible.
Search Matters
Legal precedent is voluminous. Lawyer time is limited and expensive. Discovering the right information can be difficult and frustrating. It is the job of a legal search engine to separate the wheat from the chaff.
And yet, a recent paper evaluating legal search engine behavior demonstrates that there is wide variation in the results that legal search engines return. University of Colorado Professor Susan Nevelow Mart tested several dozen queries across a handful of search engines, and found that those engines returned wildly different results, many of them irrelevant.
These differences and the quality of results returned are meaningful and lawyers understand how impactful they can be:…
More at https://blog.judicata.com/legal-search-sharing-judicatas-data-to-drive-progress-811eed64f04b
