Lawyering in the Age of Artificial Intelligence

We conduct the first randomized controlled trial of AI assistance's effect on human legal analysis. We randomly assigned sixty students at the University of Minnesota Law School each to complete four separate legal tasks (drafting a complaint, a contract, a section of an employee handbook, and a client memo), either with or without the assistance of GPT-4, after receiving training on how to use GPT-4 effectively. We then blind-graded the results and tracked how long the students took on each task. We found that access to GPT-4 slightly and inconsistently improved the quality of participants' legal analysis but induced large and consistent increases in speed. The benefits of AI assistance were not evenly distributed: in the tasks on which AI was the most useful, it was significantly more useful to lower-skilled participants. On the other hand, AI assistance reduced the amount of time that participants took to complete the tasks roughly uniformly regardless of their baseline speed. In follow up surveys, we found that participants reported increased satisfaction from using AI to complete legal tasks and that they correctly predicted the tasks for which GPT-4 would be most helpful. These results—which will likely serve as a lower-bound estimate on AI's capacity to improve the efficiency of legal services—have important normative implications across the future of lawyering. For law schools, they suggest the importance of deliberately and holistically assessing when and how law students are trained to use AI. For lawyers and judges, they suggest that the time to embrace AI is now, though the contours of what that will mean can and should vary significantly by practice area, task, and the stakes of the underlying matters. And for purchasers of legal services, our results suggest that it is time to reconsider what types of legal matters should be sent to outside counsel rather than handled in-house, and how matters that are handled externally are managed and billed.