Law Commons^™

Legalbench: A Collaboratively Built Benchmark For Measuring Legal Reasoning In Large Language Models, Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego A. Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael A. Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li

All Papers

The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers …

Using Artificial Intelligence In The Law Review Submissions Process, Brenda M. Simon

Faculty Scholarship

The use of artificial intelligence to help editors examine law review submissions may provide a way to improve an overburdened system. This Article is the first to explore the promise and pitfalls of using artificial intelligence in the law review submissions process. Technology-assisted review of submissions offers many possible benefits. It can simplify preemption checks, prevent plagiarism, detect failure to comply with formatting requirements, and identify missing citations. These efficiencies may allow editors to address serious flaws in the current selection process, including the use of heuristics that may result in discriminatory outcomes and dependence on lower-ranked journals to conduct …

Does Ai Hold The Keys? Bloomberg Law’S Docket Key Unlocks Federal District Courts, Rachel S. Evans

Articles, Chapters and Online Publications

Evans shares a review of Bloomberg Law's newly expanded docket search "Docket Key" by providing a brief intro to docket searching and explaining the type of AI-machine learning at work in the product.

The CS-SIS Blog Committee is charged with providing CS-SIS members with timely and useful information through an official yet informal medium about relevant subjects for the membership, including the activities of the members, committees, and Executive Board.

Ok, Google, Will Artificial Intelligence Replace Human Lawyering?, Amy Vorenberg, Julie A. Oseid, Melissa Love Koenig

Law Faculty Scholarship

Will Artificial Intelligence (AI) replace human lawyering? The answer is no. Despite worries that AI is getting so sophisticated that it could take over the profession, there is little cause for concern. Indeed, the surge of AI in the legal field has crystalized the real essence of effective lawyering. The lawyer’s craft goes beyond what AI can do because we listen with empathy to clients’ stories, strategize to find that story that might not be obvious, thoughtfully use our imagination and judgment to decide which story will appeal to an audience, and creatively tell those winning stories.

This article reviews …

Automatically Extracting Meaning From Legal Texts: Opportunities And Challenges, Kevin D. Ashley

Articles

This paper examines impressive new applications of legal text analytics in automated contract review, litigation support, conceptual legal information retrieval, and legal question answering against the backdrop of some pressing technological constraints. First, artificial intelligence (Al) programs cannot read legal texts like lawyers can. Using statistical methods, Al can only extract some semantic information from legal texts. For example, it can use the extracted meanings to improve retrieval and ranking, but it cannot yet extract legal rules in logical form from statutory texts. Second, machine learning (ML) may yield answers, but it cannot explain its answers to legal questions or …

Using Ai To Analyze Patent Claim Indefiniteness, Dean Alderucci, Kevin D. Ashley

Articles

In this Article, we describe how to use artificial intelligence (AI) techniques to partially automate a type of legal analysis, determining whether a patent claim satisfies the definiteness requirement. Although fully automating such a high-level cognitive task is well beyond state-of-the-art AI, we show that AI can nevertheless assist the decision maker in making this determination. Specifically, the use of custom AI technology can aid the decision maker by (1) mining patent text to rapidly bring relevant information to the decision maker attention, and (2) suggesting simple inferences that can be drawn from that information.

We begin by summarizing the …

Ethics Of Using Artificial Intelligence To Augment Drafting Legal Documents, David Hricik

Articles

Skynet is not and may never be self-aware, but machines are al-ready doing legal research, drafting legal documents, negotiating disputes such as traffic tickets and divorce schedules, and even drafting patent applications. Machines learn from us, and each other, to augment the ability of lawyers to represent clients—and even to replace lawyers completely. While it also threatens lawyers’ jobs, the exponential increase in the capacity of machines to transmit, store, and process data presents the opportunity for lawyers to use these services to provide better, cheaper, or faster legal representation to clients. By way of familiar example, instead of determining …

Topic Modeling The President: Conventional And Computational Methods, J.B. Ruhl, John Nay, Jonathan Gilligan

Vanderbilt Law School Faculty Publications

Legal and policy scholars modeling direct actions into substantive topic classifications thus far have not employed computational methods. To compare the results of their conventional modeling methods with the computational method, we generated computational topic models of all direct actions over time periods other scholars have studied using conventional methods, and did the same for a case study of environmental-policy direct actions. Our computational model of all direct actions closely matched one of the two comprehensive empirical models developed using conventional methods. By contrast, our environmental-case-study model differed markedly from the only empirical topic model of environmental-policy direct actions using …

Computer Models For Legal Prediction, Kevin D. Ashley, Stephanie Bruninghaus

Articles

Computerized algorithms for predicting the outcomes of legal problems can extract and present information from particular databases of cases to guide the legal analysis of new problems. They can have practical value despite the limitations that make reliance on predictions risky for other real-world purposes such as estimating settlement values. An algorithm's ability to generate reasonable legal arguments also is important. In this article, computerized prediction algorithms are compared not only in terms of accuracy, but also in terms of their ability to explain predictions and to integrate predictions and arguments. Our approach, the Issue-Based Prediction algorithm, is a program …

Capturing The Dialectic Between Principles And Cases, Kevin D. Ashley

Articles

Theorists in ethics and law posit a dialectical relationship between principles and cases; abstract principles both inform and are informed by the decisions of specific cases. Until recently, however, it has not been possible to investigate or confirm this relationship empirically. This work involves a systematic study of a set of ethics cases written by a professional association's board of ethical review. Like judges, the board explains its decisions in opinions. It applies normative standards, namely principles from a code of ethics, and cites past cases. We hypothesized that the board's explanations of its decisions elaborated upon the meaning and …

Designing Electronic Casebooks That Talk Back: The Cato Program, Kevin D. Ashley

Articles

Electronic casebooks offer important benefits of flexibility in control of presentation, connectivity, and interactivity. These additional degrees of freedom, however, also threaten to overwhelm students. If casebook authors and instructors are to achieve their pedagogical goals, they will need new methods for guiding students. This paper presents three such methods developed in an intelligent tutoring environment for engaging students in legal role-playing, making abstract concepts explicit and manipulable, and supporting pedagogical dialogues. This environment is built around a program known as CATO, which employs artificial intelligence techniques to teach first-year law students how to make basic legal arguments with cases. …