Raghu Ramanathan, who in February was named president of the Legal Professionals segment within Thomson Reuters, overseeing all its products for the legal profession, believes there should be open benchmarking on legal AI products, he told me during an interview earlier this week.
“I do believe the industry needs open benchmarks on AI performance in legal assistants,” he said. “I welcome it. Personally, I think the more well-funded, well-conducted research there is – even multiple ones so we can compare and contrast – it’s better for the industry, it’s better for the customers.”
His remarks come on the heels of a study by researchers at Stanford University that found that Thomson Reuters’ AI legal research product, AI Assisted Research for Westlaw Precision, delivered hallucinated results nearly a third of the time, and at twice the rate of the comparable Lexis Nexis product, Lexis+ AI.
The study’s authors said the results point to “the need for rigorous, transparent benchmarking and public evaluations of AI tools in law.”
But Ramanathan, who was formerly a vice president at SAP, said he was surprised by the results of the study, in that they do not comport with the company’s own internal testing or with what he is hearing from customers.
In our conversation earlier this week, we talked about Ramanathan’s background, his priorities as president, and his views on the market.
Read the interview
