Roshi Sharma of l Law Beam posted the following to linked in late last week.
The AI tells the truth but not the ghumans trying to flog it to other humans.
I broke Harvey in 3 days.
Yesterday, the world’s leading AI legal platform achieved a $15.5 billion valuation.
I trialled Harvey last week at LawBEAM. For hardcore legal analysis, it could not keep up.
I asked Harvey to advise on the UK financial promotions regime for cryptoassets. It came back with a 58 page document, perfectly polished and formatted.
We ran the output through LawBEAM’s engine. Harvey made several fatal errors of law and fact that would be dangerous in any client facing product.
I asked Harvey to explain what had happened and the response was startling:
“I Don’t Read Statues the Way a Lawyer Does”
“I Confuse Plausibility with Accuracy”
“I Cannot Distinguish Current Law from Superseded Law Unless Told”
“I Fabricated Where I Should Have Stopped”
“I Had the Right Sources and Still Got It Wrong”
“I am a text-generation system, not a legal research system…”
Credit to Harvey, it did produce a brutally honest account of its performance.
Read the full Transcript
