Federal Reserve Bank
of San Francisco Federal Reserve Bank
of San Francisco

Board of Governors

  • Teaching to the Test: Evaluating the Performance of Generative AI Models for Economic Analysis is Harder than You Think

    Wendy Dunn, Ellen Meade, Nitish Sinha, Raakin Kabir

    This study leverages the unique information dissemination structure of Federal Open Market Committee (FOMC) meeting minutes to investigate the data leakage hypothesis in economic text analysis. We employ hand-labeled data as our ground-truth benchmark and assess whether the past performance of these LLMs in a setting with potential leakage holds up when they are instead […]

  • Financial Stability Implications of Generative AI: Taming the Animal Spirits

    Anne Lundgaard Hansen, Seung Jung Lee

    This paper investigates the impact of the adoption of generative AI on financial stability. We conduct laboratory-style experiments using large language models to replicate classic studies on herd behavior in investment decisions. Our results show that AI agents make more rational decisions than humans, relying predominantly on private information over market trends. Increased reliance on […]

  • Can LLMs Improve Sanctions Screening in the Financial System? Evidence from a Fuzzy Matching Assessment

    Jeffrey S. Allen, Max S. S. Hatfield

    We examined the performance of four families of large language models (LLMs) and a variety of common fuzzy matching algorithms in assessing the similarity of names and addresses in a sanctions screening context. On average, across a range of realistic matching thresholds, the LLMs in our study reduced sanctions screening false positives by 92 percent […]

  • Educational Exposure to Generative Artificial Intelligence

    Jean Xiao Timmerman

    Given the fast and pervasive adoption of generative AI, it is important to explore how generative AI’s effect on labor markets may influence postsecondary institutions and their students, considering the technology’s potential to reshape workplace dynamics and the types of skills that are valued. Postsecondary institutions play a significant role in preparing students for the […]

  • Total Recall? Evaluating the Macroeconomic Knowledge of Large Language Models

    Leland D. Crane, Akhil Karra, Paul E. Soto

    We evaluate the ability of large language models (LLMs) to estimate historical macroeconomic variables and data release dates. We find that LLMs have precise knowledge of some recent statistics, but performance degrades as we go farther back in history. We highlight two particularly important kinds of recall errors: mixing together first print data with subsequent […]

  • Generative AI at the Crossroads: Light Bulb, Dynamo, or Microscope?

    Martin Neil Baily, David M. Byrne, Aidan T. Kane, Paul E. Soto

    With the advent of generative AI (genAI), the potential scope of artificial intelligence has increased dramatically, but the future effect of genAI on productivity remains uncertain. The effect of the technology on the innovation process is a crucial open question. Some inventions, such as the light bulb, temporarily raise productivity growth as adoption spreads, but […]

  • Artificial Intelligence Methods for Evaluating Global Trade Flows

    Feras A. Batarseh, Munisamy Gopinath, Anderson Monken

    International trade policies remain in the spotlight given the recent rethink on the benefits of globalization by major economies. Since trade critically affects employment, production, prices and wages, understanding and predicting future patterns of trade is a high-priority for decision making within and across countries. While traditional economic models aim to be reliable predictors, we […]

  • Research in Commotion: Measuring AI Research and Development through Conference Call Transcripts

    Paul E. Soto

    This paper introduces a novel measure of firm-level Artificial Intelligence (AI) Research & Development—the AIR Index—derived from the semantic similarity between earnings conference call transcripts and leading AI research papers. The AIR Index varies widely across industries, with sustained strength in computer and electronic manufacturing, and accelerating growth in computing infrastructure and educational services seen […]

  • CardSim: A Bayesian Simulator for Payment Card Fraud Detection Research

    Jeffrey S. Allen

    Payment fraud has been high in recent years, and as criminals gain access to capability-enhancing generative AI tools, there is a growing need for innovative fraud detection research. However, the pace, diversity, and reproducibility of such research are inhibited by the dearth of publicly available payment transaction data. A few payment simulation methodologies have been […]

  • Measuring AI Uptake in the Workplace

    Leland Crane, Michael Green, Paul Soto

    Artificial Intelligence (AI) may be poised to raise productivity across various domains, including writing (Noy and Zhang 2023), programming (Peng et al. 2023), and research and development (Toner-Rodgers 2024; Korinek 2023). However, understanding the extent to which AI—and generative AI in particular—has been adopted as part of the production process remains an open question. This […]