AI-WESTLAW ADVANTAGE | Deep Research Shortcomings
In preparing a new set of exercises for a Transactional Skills course I am creating illustrations about how to use AI tools and some of the dangers.
1. I have given prompts to learn differences in New York and Florida law about the requirements for enforceability of settlement agreements and preliminary financing documents like term sheets. For students, this focuses on basic documents transaction lawyers will encounter. Importantly, litigators need to know about settlement agreements too. (So, the course adds value regardless of career paths students might choose.)
2. I have run the queries multiple times and received inadequate responses each time. One round of prompts mentioned the Tribune case as if there had been no subsequent developments. Another failed to mention the Tribune case at all, but in its “modern” analysis did not mention AmCan.
3. The “verify” function did not help in these cases because it was just more AI generated material. It can only “verify” what it has stated or quoted. It cannot help with the “dog that did not bark” it seems.
4. As ethics rules in both Florida and New York explicitly require lawyers to verify work performed by AI, the verify function in Westlaw Advantage sounds helpful. But it actually could prove dangerous. I doubt a court would find that use of this feature satisfies the lawyer’s obligation to verify.
5. While it is interesting to look at this output, and it perhaps could help enhance a deeper analysis with human tweaking, it is basically not helpful if you do not already know what the law is so you can spot omissions.
6. What the lawyer needs to do for this collection of research is to actually read the cases that AI does cite and then look for cases that seem important but were omitted. And then analyze the old-fashioned way by human thought.
7. I am very concerned that features like “verify” create a trap for well-intentioned lawyers. AI simply cannot “verify” in the sense required by canons of ethics. See below when Deep Research admitted it was not very deep at all. Given that this tool is purpose built for law, this should be an eye opener for students. Seth Chandler‘s posts motivated this post. Thanks to an HLS classmate!
