Law Commons^™

Legalbench: A Collaboratively Built Benchmark For Measuring Legal Reasoning In Large Language Models, Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego A. Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael A. Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li

All Papers

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers …

A New Metaphor: How Artificial Intelligence Links Legal Reasoning And Mathematical Thinking, Melissa E. Love Koenig, Colleen Mandell

Marquette Law Review

Artificial intelligence’s (AI’s) impact on the legal community expands exponentially each year. As AI advances, lawyers have more powerful tools to enhance their ability to research and analyze the law, as well as to draft contracts and other legal documents. Lawyers are already using tools powered by AI and are learning to shift their methodologies to take advantage of these enhancements. To continue to grow into their shifting role, lawyers should understand the relationship between AI, mathematics, and legal reasoning.

The Limits Of Law And Ai, Ryan Mccarl

University of Cincinnati Law Review

For thirty years, scholars in the field of law and artificial intelligence (AI) have explored the extent to which lawyers and judges can be assisted by computers. This Article describes the medium-term outlook for AI technologies and explains the obstacles to making legal work computable. I argue that while AI-based software is likely to improve legal research and support human decision making, it is unlikely to replace traditional legal work or otherwise transform the practice of law.

Natural Language Processing For Lawyers And Judges, Frank Fagan

Michigan Law Review

A Review of Law as Data: Computation, Text, & the Future of Legal Analysis. Edited by Michael A. Livermore and Daniel N. Rockmore.

Ai Report: Humanity Is Doomed. Send Lawyers, Guns, And Money!, Ashley M. London

Law Faculty Publications

AI systems are powerful technologies being built and implemented by private corporations motivated by profit, not altruism. Change makers, such as attorneys and law students, must therefore be educated on the benefits, detriments, and pitfalls of the rapid spread, and often secret implementation of this technology. The implementation is secret because private corporations place proprietary AI systems inside of black boxes to conceal what is inside. If they did not, the popular myth that AI systems are unbiased machines crunching inherently objective data would be revealed as a falsehood. Algorithms created to run AI systems reflect the inherent human categorization …

Ok, Google, Will Artificial Intelligence Replace Human Lawyering?, Amy Vorenberg, Julie A. Oseid, Melissa Love Koenig

Law Faculty Scholarship

Will Artificial Intelligence (AI) replace human lawyering? The answer is no. Despite worries that AI is getting so sophisticated that it could take over the profession, there is little cause for concern. Indeed, the surge of AI in the legal field has crystalized the real essence of effective lawyering. The lawyer’s craft goes beyond what AI can do because we listen with empathy to clients’ stories, strategize to find that story that might not be obvious, thoughtfully use our imagination and judgment to decide which story will appeal to an audience, and creatively tell those winning stories.

This article reviews …

Rise Of The Robot Lawyers?, Milan Markovic

Milan Markovic

The advent of artificial intelligence has provoked considerable speculation about the future of the American workforce, including highly educated professionals such as lawyers and doctors. Although most commentators are alarmed by the prospect of intelligent machines displacing millions of workers, this is not so with respect to the legal sector. Media accounts and some legal scholars envision a future where intelligent machines perform the bulk of legal work, and legal services are less expensive and more accessible. This future is purportedly at hand as lawyers struggle to compete with technologically savvy alternative legal service providers.

This Article challenges the notion …

Rise Of The Robot Lawyers?, Milan Markovic

Faculty Scholarship

The advent of artificial intelligence has provoked considerable speculation about the future of the American workforce, including highly educated professionals such as lawyers and doctors. Although most commentators are alarmed by the prospect of intelligent machines displacing millions of workers, this is not so with respect to the legal sector. Media accounts and some legal scholars envision a future where intelligent machines perform the bulk of legal work, and legal services are less expensive and more accessible. This future is purportedly at hand as lawyers struggle to compete with technologically savvy alternative legal service providers.

This Article challenges the notion …

Professions And Expertise: How Machine Learning And Blockchain Are Redesigning The Landscape Of Professional Knowledge And Organization, John Flood, Lachlan Robb

University of Miami Law Review

Machine learning has entered the world of the professions with differential impacts. Automation will have huge impacts on the nature of work and society. Engineering, architecture, and medicine are early and enthusiastic adopters of automation. Other professions, especially law, are late and, in some cases, reluctant adopters. This Article examines the effects of artificial intelligence (“AI”) and Blockchain on professions and their knowledge bases. We start by examining the nature of expertise in general and the function of expertise in law. Using examples from law, such as Gulati and Scott’s analysis of how lawyers create (or don’t create) legal agreements, …

Automatically Extracting Meaning From Legal Texts: Opportunities And Challenges, Kevin D. Ashley

Articles

This paper examines impressive new applications of legal text analytics in automated contract review, litigation support, conceptual legal information retrieval, and legal question answering against the backdrop of some pressing technological constraints. First, artificial intelligence (Al) programs cannot read legal texts like lawyers can. Using statistical methods, Al can only extract some semantic information from legal texts. For example, it can use the extracted meanings to improve retrieval and ranking, but it cannot yet extract legal rules in logical form from statutory texts. Second, machine learning (ML) may yield answers, but it cannot explain its answers to legal questions or …

Using Ai To Analyze Patent Claim Indefiniteness, Dean Alderucci, Kevin D. Ashley

Articles

In this Article, we describe how to use artificial intelligence (AI) techniques to partially automate a type of legal analysis, determining whether a patent claim satisfies the definiteness requirement. Although fully automating such a high-level cognitive task is well beyond state-of-the-art AI, we show that AI can nevertheless assist the decision maker in making this determination. Specifically, the use of custom AI technology can aid the decision maker by (1) mining patent text to rapidly bring relevant information to the decision maker attention, and (2) suggesting simple inferences that can be drawn from that information.

We begin by summarizing the …

The Rise Of Artificial Intelligence In The Legal Field: Where We Are And Where We Are Going, Sergio David Becerra

The Journal of Business, Entrepreneurship & the Law

The twenty-first century has brought significant technological advancement that permeates all aspects of our lives. The legal field, though slow in the adaption of this technology, is beginning to pick up the pace. Artificial Intelligence (AI) technology is used now to perform legal work once completed solely by legal practitioners. This Comment outlines what AI is and reviews the current use of AI in the legal field. It also identifies AI products and developments that are in place. Finally, it argues that lawyers will always be needed in the practice of law, despite the continued growth of AI.

A Rule Of Persons, Not Machines: The Limits Of Legal Automation, Frank A. Pasquale

Faculty Scholarship

No abstract provided.

Bridges Ii: The Law--Stem Alliance & Next Generation Innovation, Harry Surden

Publications

Technological change recently has altered business models in the legal field, and these changes will continue to affect the practice of law itself. How can we, as educators, prepare law students to meet the challenges of new technology throughout their careers?

Teaching Law And Digital Age Legal Practice With An Ai And Law Seminar, Kevin D. Ashley

Chicago-Kent Law Review

This article provides a guide and examples for using a seminar on Artificial Intelligence (AI) and Law to teach lessons about legal reasoning and about legal practice in the digital age. Artificial Intelligence and Law is a subfield of AI/ computer science research that focuses on computationally modeling legal reasoning. In at least a few law schools, the AI and Law seminar has regularly taught students fundamental issues about law and legal reasoning by focusing them on the problems these issues pose for scientists attempting to computationally model legal reasoning. AI and Law researchers have designed programs to reason with …

Computer Models For Legal Prediction, Kevin D. Ashley, Stephanie Bruninghaus

Articles

Computerized algorithms for predicting the outcomes of legal problems can extract and present information from particular databases of cases to guide the legal analysis of new problems. They can have practical value despite the limitations that make reliance on predictions risky for other real-world purposes such as estimating settlement values. An algorithm's ability to generate reasonable legal arguments also is important. In this article, computerized prediction algorithms are compared not only in terms of accuracy, but also in terms of their ability to explain predictions and to integrate predictions and arguments. Our approach, the Issue-Based Prediction algorithm, is a program …

Capturing The Dialectic Between Principles And Cases, Kevin D. Ashley

Articles

Theorists in ethics and law posit a dialectical relationship between principles and cases; abstract principles both inform and are informed by the decisions of specific cases. Until recently, however, it has not been possible to investigate or confirm this relationship empirically. This work involves a systematic study of a set of ethics cases written by a professional association's board of ethical review. Like judges, the board explains its decisions in opinions. It applies normative standards, namely principles from a code of ethics, and cites past cases. We hypothesized that the board's explanations of its decisions elaborated upon the meaning and …

Designing Electronic Casebooks That Talk Back: The Cato Program, Kevin D. Ashley

Articles

Electronic casebooks offer important benefits of flexibility in control of presentation, connectivity, and interactivity. These additional degrees of freedom, however, also threaten to overwhelm students. If casebook authors and instructors are to achieve their pedagogical goals, they will need new methods for guiding students. This paper presents three such methods developed in an intelligent tutoring environment for engaging students in legal role-playing, making abstract concepts explicit and manipulable, and supporting pedagogical dialogues. This environment is built around a program known as CATO, which employs artificial intelligence techniques to teach first-year law students how to make basic legal arguments with cases. …