Here’s the introduction to the piece and yes like any SLAW piece it is detailed and well worth a read.
Prof Sean Rehaag recently published, “Luck of the Draw III: Using AI to Examine Decision-Making in Federal Court Stays of Removal”. This research entered my feed as it pertains to immigration and refugee law. Indeed, the research demonstrates interesting trends related to Federal Court decisions and Stay Motions. For example, Winnipeg has the lowest grant rates across Canada at only 16.2%. For immigration practitioners, I will briefly discuss the conclusions of this paper and my own analysis. Prof Rehaag focused this paper on statistics and his methodology. The paper offers scant analysis of the underlying numbers. The paper is invaluable for any legal researchers who are thinking of using ChatGPT or any other AI tool in their work. From my perspective, this paper is a must-read.
Disclaimer: my spouse is an academic and she has become mildly obsessed with ChatGPT since we first heard about it in Dec 2022. Rarely a day goes by when we are not talking about its uses, both good and bad. She has already had students submit papers that were written with the use of AI. I have not (yet) found a use for ChatGPT in my practice; however, I would be surprised if that day is not in the near future.
Methodology of Legal Research
Prof Rehaag devotes a significant portion of his paper on methodology, going into significant detail on exactly how the AI was used. I would encourage you to read the paper itself. His writing is both clear and concise. For example:
The specific methodology used in this study involved several steps. First, data was collected from all Federal Court online dockets from the past ten years. Next, machine learning language models were created and applied to classify and extract information from docket entries. Additional logic was then applied to infer case-level data using the classifications and extracted information. Data verification was undertaken to ensure the accuracy of the resulting dataset. Finally, statistical analysis was undertaken on the dataset.
Prof Rehaag is famous (this is not an overstatement) among immigration lawyers for his statistics and mathematical comparisons of Federal Court judges. His research has been used as the basis for many “inherent bias” arguments against judges. Personally, I have successfully used his research to have a judge recuse himself.
Getting to the meat, Prof Rehaag explains how GPTs function for the user:
GPTs are machine learning models using neural networks – specifically transformers – that are pre-trained on large quantities of text from the Internet. The initial training is unsupervised, meaning that the system does not use data labelled by human beings and then tries to match that labelling. Instead, the task that model is trained on is to predict (or calculate the probability) of the next word or sequence of text after any given sequence of text in the massive dataset of text it uses. This form of training makes GPTs particularly well-suited to generating predicted sequences of words based on an inputted prompt.
As stated above, GPTs are not useful for all types of legal research. In the context of comparing large datasets of Federal Court decisions, however, Prof Rehaag and his team at the Refugee Law Lab have explained exactly how a GPT may be used to “fine-tune” the system. Here is an example from his article:
Read the full article
