Federal Reserve Bank
of San Francisco Federal Reserve Bank
of San Francisco

Board of Governors

  • Total Recall? Evaluating the Macroeconomic Knowledge of Large Language Models

    Leland D. Crane, Akhil Karra, Paul E. Soto

    We evaluate the ability of large language models (LLMs) to estimate historical macroeconomic variables and data release dates. We find that LLMs have precise knowledge of some recent statistics, but performance degrades as we go farther back in history. We highlight two particularly important kinds of recall errors: mixing together first print data with subsequent […]

  • Generative AI at the Crossroads: Light Bulb, Dynamo, or Microscope?

    Martin Neil Baily, David M. Byrne, Aidan T. Kane, Paul E. Soto

    With the advent of generative AI (genAI), the potential scope of artificial intelligence has increased dramatically, but the future effect of genAI on productivity remains uncertain. The effect of the technology on the innovation process is a crucial open question. Some inventions, such as the light bulb, temporarily raise productivity growth as adoption spreads, but […]

  • Artificial Intelligence Methods for Evaluating Global Trade Flows

    Feras A. Batarseh, Munisamy Gopinath, Anderson Monken

    International trade policies remain in the spotlight given the recent rethink on the benefits of globalization by major economies. Since trade critically affects employment, production, prices and wages, understanding and predicting future patterns of trade is a high-priority for decision making within and across countries. While traditional economic models aim to be reliable predictors, we […]

  • Research in Commotion: Measuring AI Research and Development through Conference Call Transcripts

    Paul E. Soto

    This paper introduces a novel measure of firm-level Artificial Intelligence (AI) Research & Development—the AIR Index—derived from the semantic similarity between earnings conference call transcripts and leading AI research papers. The AIR Index varies widely across industries, with sustained strength in computer and electronic manufacturing, and accelerating growth in computing infrastructure and educational services seen […]

  • CardSim: A Bayesian Simulator for Payment Card Fraud Detection Research

    Jeffrey S. Allen

    Payment fraud has been high in recent years, and as criminals gain access to capability-enhancing generative AI tools, there is a growing need for innovative fraud detection research. However, the pace, diversity, and reproducibility of such research are inhibited by the dearth of publicly available payment transaction data. A few payment simulation methodologies have been […]

  • Measuring AI Uptake in the Workplace

    Leland Crane, Michael Green, Paul Soto

    Artificial Intelligence (AI) may be poised to raise productivity across various domains, including writing (Noy and Zhang 2023), programming (Peng et al. 2023), and research and development (Toner-Rodgers 2024; Korinek 2023). However, understanding the extent to which AI—and generative AI in particular—has been adopted as part of the production process remains an open question. This […]

  • Using Generative AI Models to Understand FOMC Monetary Policy Discussions

    Wendy Dunn, Ellen E. Meade, Nitish Ranjan Sinha, Raakin Kabir

    Research conducted using AI/ML tools

    In an era increasingly shaped by artificial intelligence (AI), the public’s understanding of economic policy may be filtered through the lens of generative AI models (also called large language models or LLMs). Generative AI models offer the promise of quickly ingesting and interpreting large amounts of textual information. Thus far, however, little is known about […]

  • Reasons Behind Words: OPEC Narratives and the Oil Market

    Celso Brunetti, Marc Joets, Valerie Mignon

    Research conducted using AI/ML tools

    We analyze the content of the Organization of the Petroleum Exporting Countries (OPEC) communications and whether it provides information to the crude oil market. To this end, we derive an empirical strategy which allows us to measure OPEC’s public signal and test whether market participants find it credible. Using Structural Topic Models, we analyze OPEC […]

  • Manufacturing Sentiment: Forecasting Industrial Production with Text Analysis

    Tomaz Cajner, Leland D. Crane, Christopher Kurz, Norman Morin, Paul E. Soto, Betsy Vrankovich

    Research conducted using AI/ML tools

    This paper examines the link between industrial production and the sentiment expressed in natural language survey responses from U.S. manufacturing firms. We compare several natural language processing (NLP) techniques for classifying sentiment, ranging from dictionary-based methods to modern deep learning methods. Using a manually labeled sample as ground truth, we find that deep learning models—partially […]

  • Tracking Real Time Layoffs with SEC Filings: A Preliminary Investigation

    Leland D. Crane, Emily Green, Molly Harnish, Will McClennan, Paul E. Soto, Betsy Vrankovich, Jacob Williams

    Research conducted using AI/ML tools

    We explore a new source of data on layoffs: timely 8-K filings with the Securities and Exchange Commission. We develop measures of both the number of reported layoff events and the number of affected workers. These series are highly correlated with the business cycle and other layoff indicators. Linking firm-level reported layoff events with WARN […]