Instructive that this article is penned by “Anonymous”.

By Anonymous.

Lawyers make mistakes. That’s why they take out professional liability insurance. This raises the question in the wake of the Stanford University study into the accuracy of Thomson Reuters’ and LexisNexis’ generative AI tools: what are we comparing these systems against when it comes to hallucinations and inaccuracy?

Or put it another way: yes, generative AI systems create errors, but are lawyers without such technology more accurate?

The regulation around this topic makes it clear that legal perfection is not expected. For example, in England & Wales the Solicitors Regulation Authority (SRA) requires firms to hold professional indemnity insurance. And it’s this ongoing need for insurance coverage that is a key reason why the SRA will step in when a law firm collapses. In short, regulators expect lawyers to make errors.

So when we’re comparing the accuracy rates of generative AI or another overhyped and under-explained technology with what lawyers do, then we’re comparing two inherently flawed processes. Humans are imperfect and so is generative AI. The question then is: how flawed are they and what level is acceptable?

What is not properly quantified at present is how many mistakes ‘a’ lawyer makes, or how many mistakes a legal ‘team’ providing advice make. Clearly it’s not zero. So how good are human lawyers?

Can we get some hints from looking at how law firms are charged for insurance?

Going through the renewal process for professional indemnity cover (in England & Wales) you’re evaluated on your ‘gross fee income’ and this is then adjusted by the kind of work done and, obviously, your claims history. This rounds out at about 1.5% of turnover for larger firms in England & Wales. Even assuming actuarially fair insurance, this does not imply a 98.5% accuracy rate for legal work overall.

A lot of the mistakes of the sort that are being spotted in the Stanford study are unlikely to contribute to this figure as the structure of the work mitigates them: supervisors, managing associates and partners will catch their juniors’ errors; opposing counsel will query language that doesn’t make sense to them; and in the most embarrassing case, clients will spot things that seem off.

Also, many mistakes are never spotted. Or, if they are, they get smoothed over. Looking through old work bibles usually leads to comments like ‘I’m not sure I would have drafted it that way’ and then a resolution not to use those documents in the future.

I.e. Insurance failure rates and Professional Indemnity Insurance claims may dramatically understate the frequency of mistakes.

This means that although regulators expect lawyers to make significant errors, many of those errors are 1) picked up internally, 2) highlighted by opposing counsel, 3) picked up by the client, or 4) manage to slip through the net and no-one notices….until someone eventually does.

This in turn raises the question of what is ‘good enough’ for a client, given that they must be accepting work product that is often imperfect?

Article – Artificial Lawyer: GenAI Hallucinations? Lawyers Aren’t Perfect Either

Instructive that this article is penned by “Anonymous”.