AI in the Courtroom: A Landmark Warning for the Legal Profession
The NSW Court of Appeal has issued a landmark judgment that every lawyer needs to read.
In May v Costaras [2025] NSWCA 178, the court dismissed a property appeal but delivered a powerful warning on the reckless use of generative AI in legal proceedings.
A self-represented litigant used AI to prepare submissions, resulting in:
??Nonsensical legal arguments.
??Citations of wholly irrelevant cases.
??A hallucinated, non-existent case with pinpoint paragraphs.
The Court’s obiter is a masterclass on the ethical perils of this technology. It highlights the non-negotiable duty for practitioners to verify every AI-generated output against authoritative sources. Blind reliance is professional recklessness.
This is not an anti-technology stance; it is a critical call for professionalism, oversight and verification.
I have analysed the key takeaways and their profound implications for practice in a new article.
Read the full case note here: https://www.linkedin.com/pulse/perils-generative-ai-courtroom-judicial-warning-from-may-david-lim-s855c/
He writes at the end of his piece
Questions for Discussion
- Beyond Verification: The judgment mandates verification, but what does that process actually look like in practice? Does verifying every AI-generated citation negate the efficiency gains that justify its use in the first place?
- Specialised v. General AI: Would the outcome have been different if the litigant had used a specialised, legal-specific AI tool trained on verified legal databases, as opposed a general-purpose LLM like ChatGPT? Does the profession need to draw a distinction?
- Judicial Training: How should judicial officers be trained to identify AI-generated submissions? Should there be a formal requirement for parties to disclose the use of AI in the preparation of their court documents?
- Sanctions: What are the appropriate sanctions for a legal practitioner who, after this clear warning, fails to verify an AI-generated submission and presents hallucinated authorities to the court? Should it be treated as a mere error or as a more serious ethical breach?
- Access to Justice: The Court acknowledges that AI might improve access to justice, but this case shows it can also be a hindrance. How can the legal system harness the potential of AI to genuinely help SRLs without overwhelming the courts with AI-generated “noise” and misinformation?
