Benjamin F. Jones | AI in Research & Development

Mongkha Pavlick:

Welcome everyone. I’m Mongkha Pavlick, executive vice president of the Supervision Team here at the San Francisco Reserve Bank. I’m excited to kick off our latest EmergingTech Economic Research Network (EERN) event virtually from our Los Angeles branch office. Understanding how technologies like artificial intelligence (AI) influence productivity, innovation, and economic growth is a key focus of the SF Fed’s research and business outreach functions that in turn helps to inform monetary policy decisions. With the San Francisco Fed’s district covering the nine western states, we are uniquely situated to observe how advancements in AI from Silicon Valley to the Silicon Slopes in Utah and the countless other hotbeds of innovation will power changes in our economy. One of the most impactful ways in which we are seeing AI alter existing processes is in research and development, which of course is foundational to the creation of new products and goods. Fittingly, today, we are honored to welcome Benjamin Jones, professor of entrepreneurship at Kellogg School of Management, Northwestern University.

Professor Jones is a leading expert on the relationships between innovation, technological change, and economic growth. Today, he will present research on AI in research and development, sharing a framework for analyzing how AI accelerates the ideas production function and transforms productivity gains. Following his presentation, Professor Jones will be joined for a Q&A session by Huiyu Li, live from our San Francisco office. Huiyu is a research advisor here at the San Francisco Fed and serves as a co-head of the Emerging Tech Economic Research Network. As a reminder, this event is being livestreamed and recorded and can be accessed on our website following the discussion. Finally, please note that the views you will hear today are those of our speakers and do not necessarily represent the views of the Federal Reserve Bank of San Francisco or the Federal Reserve System. So with that, let’s begin.

Huiyu Li:

Thank you, Mongkha. Welcome to another EERN event. It’s my pleasure to welcome Professor Ben Jones to give us a seminar on AI and research and development. Professor Jones is an expert in this area. He has been working on innovation, research, and development, and the impact on economic growth for a long time, both from the macroeconomic perspective, but also really down to the technology level, very microeconomic level. So I think it’s great that we can have someone with such a broad range of perspective to share his insights with us. So over to you, Professor Jones.

Benjamin Jones:

Thank you, Huiyu. It’s great to have a chance to join you virtually and talk about the role of artificial intelligence and research and development. And I look forward to having some dialogue after this presentation as well. So just to give some clear motivation before we dive into the process of research and development, when you think about the long run trajectory of the standards of living, both in terms of our real incomes, what we can consume, but also in terms of our health and longevity. If you look over the long run, if you go back to 1870, from 1870 to today, US income per capita is about 18 to 20 times higher per person in real terms. So we’re just far better off in terms of our incomes, but also we live about 35 years longer from birth. So there’s been enormous progress in both the economic side in terms of productivity and output per worker, but also in terms of our health and wellbeing.

The graph here shows you income per capita and real terms from 1870 towards the end of the 21st century. And you can see there’s been very steady progress of about 2% per year. And when you dig into that, like, “Well, why are we doing so much better in terms of our standard of living?” The main thing that just strikes you, if you look at this carefully, it’s that we’re coming up with new and better ways of doing things. It’s not that we just do more of what we knew how to do in 1870 or the 1900. If you think about transportation, just to take one example, in 1970, you think maybe about, or 1850, you think about a wagon, taking… I’m sitting in Illinois right now, so if I wanted to go to California, I’d go in a covered wagon, and it would take five or six months, and it was pretty dangerous.

Now you can make that trip in a jet airplane in less than five hours. So there’s enormous progress, and it’s really about coming up with new ways of doing things, not by doing more of what we knew how to do already. And so when we think about AI, kind of a fundamental question is going to be, can AI further accelerate that rate of progress? Maybe it just sustains what we’ve already achieved in terms of the rates we experienced. Maybe we go faster now because of AI. And that’s what I want to dig in today. If new ideas or innovation, as we might say broadly in this research and development process are what is driving rising standards of living and income or health, then you can quickly see that how AI influences research and development, this discovery, this innovation process is going to be really first order to understanding its ultimate economic effects and its effects on human welfare.

And arguably, it becomes its most important economic effect. While AI may have enormous effects in the real economy and how we make shoes or cars or do all sorts of things in our ordinary production, that’s very important. But if it actually changes the rate of growth through the research and development process, that will become incredibly important. Now, to get at that, what I want to do in this presentation and the following conversation is really dig into two related perspectives. And the first, I want to talk about creativity and the idea of whether can AI be creative? Can it really discover new things? And then secondly, having done that, I want to talk more broadly about tasks, about the kind of research tasks that we do in research and development, and to the extent to which AI can really meaningfully change our rate of progress from that perspective.

Okay. So to dive in first on creativity, I’m going to talk about the nature of creativity briefly, kind of in a very micro way. I’m then going to talk very briefly about humans. We’re not perfectly creative, but what are our strengths and weaknesses? And then I’m going to come and look at AI in comparison to humans, given these ideas about the nature of creativity. So to start with, I find this metaphor of a hallway with doors on it, closed doors, very useful because you can think of what’s going on in a research process in R&D and innovation and coming up with new ideas. It’s you’re trying to find something that we didn’t know before, didn’t know how to do before. And I think of that as kind of walking down a hallway and there are all these doors and you can look in a door with some effort and some cost and try to see if there’s something really important and new in that door.

And so we’re kind of searching along this infinite corridor, if you will, for new ideas. And the question we ask as researchers is often to start with, “Well, what door should I try to open?” And what you’ll find when you open a door is that a lot of innovation doesn’t work. It’s really hard, right? We’re kind of stumbling around. We’re not really sure how to do things. And so often you open a door and you’re like, “Oh, it’s a broom closet. This is not a very valuable avenue to anything.” But sometimes you open a door and it might be like CRISPR in biotech or maybe AI itself and you say, “Wow, this is like a doorway to a palace. There’s going to be a lot of value that we could find of new ways of doing things inside this door.”

So the way that Academics have talked about this a lot, is about what we call the exploit versus explore trade-off, where you can think of a person or a company, they’re on one stretch of the hallway where they have expertise and they’re trying to look down the hallway to the next thing they might discover or improve. So exploit is kind of looking closely around your areas of expertise, whereas explore would be to walk quite a bit of a ways down that hallway where you’d be quite uncertain about what you might find. On the other hand, there might be something really new and exciting if you were to travel further along that hallway. So one version of how we think about creativity is this hallway and whether it’s you stay locally or you kind of go further along from what you know.

There’s another way of thinking about creativity, the second way I want to emphasize, which is that we think of the creative process as fundamentally combinatoric. It’s really interesting, if you read about creativity in very different fields, people talk about it in the arts or in music. They talk about it in science. They talk about it in invention, people kind of coming up with new mechanical devices and many other contexts. You’ll see that independently, people constantly come back to this idea that what people are doing creatively is they’re making combinations. They’re taking things that exist and then building a new combination that hadn’t existed before. And by building this new combination, it might turn out to be not very valuable, but it might turn out to be very valuable. And if that new combination is valuable, then other people can begin to use it and they can imitate it or put it as an ingredient into their own further combinatoric creativity process.

So you see this very generally. And one thing you’ve got to realize though when you start talking about combinatorics is that the space that you can search combinatorically is basically infinite. You don’t need a whole lot of ingredients before you can make a very, very large number of different combinations. And so while creativity is combinatoric, it’s also sort of puzzling because the space is too large to search. So it’s clear that what humans are doing, and what we’ve done for a long time with creativity, is on the one hand things seem very combinatoric. On the other hand, we don’t just try really bad combinations. We have some expertise or judgment that allows us to say, “Those particular ingredients are going to be really useful, say to building a rocket ship, and these ingredients might be useful for making dinner,” like if you’re a chef making combinations, but we probably don’t use the spice rack for making dinner to make the rocket ship. We kind of have a lot of judgment about which kinds of ingredients are useful for which kinds of questions or problems that we’re trying to solve.

So we can think of humans as sort of searching along a hallway. We can think of us as making combinations, but a key element here is that we’re always relying on our expertise. We’re making bets and we want to make bets where we kind of understand the ingredients and have some sense that there’s a high probability when we make a sort of combination that it might at least have some value.

And this gets us quickly to a problem, which is that humans ultimately are pretty limited in their creativity in the sense that humans can only know so many things. We can only be expert in so many things. This is an idea that I call the burden of knowledge and features in a lot of my research, but maybe Einstein said it the best back in 1932, and I’ll read this quote, which has to do with scientific discovery. He said, “Knowledge has become vastly more profound in every department of science, but the assimilative power of the human intellect is and remains strictly limited.” In other words, an individual can only know and be expert in so many different things.

Hence, Einstein says, “It was inevitable that the activity of the individual investigator should be confined to a smaller and smaller section.” So in other words, humans have increasingly narrow views of all of science and technology because we keep coming up with more and more discoveries, new things, new topics, new tools, but a given human can only know so many and only can only master so many of those things. And so their ability to kind of have a wide impact becomes increasingly limited. And in fact, as you can see in Einstein’s quote, there’s kind of a lament here, kind of a sadness that the researcher becomes kind of increasingly narrow and sort of stuck in smaller and smaller areas of research.

Now, as a recent empirical example of this kind of phenomenon, this is from a paper from last summer in nature from Hill et al, it’s called, “The Pivot Penalty in Science and Technology.” And basically what you’re measuring here, if you look kind of horizontally on the X-axis, this is how far a researcher or research team is moving when they write a new paper or it could be a new invention, if it’s a patent. These are papers I’m showing you, but the results are the same for patents. The X-axis is saying, the horizontal axis is saying, “How far are we moving from what we did before?” So you’ve done a body of work and then a low pivot, a pivot of zero in this measure is… Your next thing you’re doing, your next invention, your next research paper is very, very closely related in terms of the underlying ingredients from what you normally do. While a pivot of one going further out would be that you’re jumping into completely new territory for you. Maybe it’s not new to science, but it’s new to you.

The vertical axis is measuring how good the invention turns out to be based on some sort of measure of impact, whether it’s patenting or papers, you can use various measures. What you see here is that the more you move, the further you move from your expertise, the worse it goes. That you’re on a down slope, you’re getting a smaller probability of having kind of a big insight. The second thing you see is if you look at the dates, the blue dots here are the 1970s. This is research in the 1970s. The green dots are the 1990s, the red dots are the 2010s. What you see is that it’s getting steeper. In other words, there’s an increasing penalty. There’s always been a penalty for trying to move into new areas, but it’s getting worse with time. That’s that Einstein point, the burden of knowledge that we end up being narrower. We’re stuck in narrower areas.

And so when you think about creativity, you want to think about people being able to draw on ingredients in some expert way, but the reality is humans are limited in our capacity to do that. And we’re increasingly limited because we know a smaller share of all the tools and ingredients that are out there, and so we tend to have narrower ideas and narrower impact.

Okay, so where does that leave us? Well, that sets up a comparison with AI. And so when you think about a micro level, humans are kind of stuck on a narrow section of that hallway, a given human. Different humans are in different parts of the hallway, some people are doing biochemistry, some are doing rocket science, some are doing economics, but given wherever you are, you’re kind of stuck where you are a little, but you can’t just suddenly do rocket science if you’re an economist and vice versa.

So while humans are experts, but they’re searching locally around their own expertise and their creative insights, well, what does AI do? Well, AI is of course trained, these world models anyway, they’re trained on reading all text, for example, that’s available. They read everything. So in that sense, they kind of know what is known and they have a much broader set of ingredients, which they can combine. So in that sense, an AI can seem especially potentially creative because it has a much broader set of ingredients to work with. It’s also because it’s a computer, and while in some sense, these hyperscale data centers are expensive and costly. Per question you ask, they’re very, very cheap. This is why people use them now for coding, for example, like Claude Code instead of coding yourself, because if you can get a computer to do the task, it’s going to be able to do it at a much lower cost than you can for that task, and that’s going to be another big advantage.

So what are we seeing? We’re seeing actually that at least in certain domains, AI is showing rather remarkable creativity. It’s not necessarily some strange human feature, creativity. If it’s just search or it’s combinations, well, maybe AI can do that too. And so if you look on the right of this slide, the upper figure is showing a tool GNoME from DeepMind, which does material discovery. So basically materials, you’re combining, it’s very literally combinations, you’re combining elements into materials that then may have properties. And what you’re seeing here is that GNoME was able to train itself on the pale blue data, which is the number of known stable materials that are used in everything from solar panels to carbon fiber, tennis rackets, airframes of airplanes.

All these materials with known properties, it was able to sort of train itself on all those known materials. And then it was able to suggest – these are these dark blue bars – suddenly 10 times as many materials that it thinks are stable at room temperature are new and may have interesting properties. So, it was basically, if you think about the hallway, it’s like taking every door on that hallway, and we had no idea what was behind some of these doors. And it’s putting a label on a lot of those doors saying, there’s a material in here with interesting properties that’s stable. That’s very exciting. It’s directing humans out of their narrow search and it’s doing it at very low cost to sort of suddenly label a lot of doors on that hallway.

The lower picture is one about chemical synthesis. This is a tool where chemical synthesis is about combining molecules in some sequence to make some other chemical compound. And AI is also very good at suggesting those kinds of combinations because they can be trained in specialized models on chemical synthesis knowledge.

Okay. So, this is a pretty optimistic picture. It’s suggesting that AI might be actually more creative in a sense and overcome certain human limits in creativity and do things at high scale and low cost. And now I’m showing you a picture of AlphaFold, which is the other for a prominent science algorithm from DeepMind, which of course won the Nobel Prize, well, the Nobel Prize in 2024. This determines protein folding. Again, able to sort of solve a problem that’s much harder for humans at high scale and at much lower cost. So, this all suggests that, will we see a huge acceleration in progress? Will we see big advances in productivity and growth in the economy? Will we see big improvements in health through the discovery and AI driven discovery of all sorts of new ideas and solutions to problems? Maybe.

But if you think about it more deeply, you’d realize that opening new doors on this hallway, which is we can think of as sort of setting the creative direction, “Hey, look at this protein or try this material.” That’s often the beginning and not the end of a research process. And there are many other things that have to happen to decide if a material or a protein is meaningful, is something you can use or solve for in some meaningful way. And so to understand AI’s capabilities and its limits, its constraints in R&D, we need to generalize a bit and think about it beyond just a creative spark, a direction of travel for a research project, and more broadly about the set of research tasks that you actually have to undertake in order to complete a research project to bring some kind of new and valuable product, for example, into the society, into the market.

So, now I’m going to switch over to the task perspective. And while the creativity perspective makes one optimistic, I think this perspective is going to cause us to be a little bit more cautionary in terms of what our expectations might realistically be in terms of AI’s capacity to radically, at least, accelerate research and development processes.

So, what I want you to think about is there’s some outcome we care about. I’m calling it Y. And it could be health, it could be a productivity, it could be a new material, something that has some properties that you want to improve. Thinking of a health context, bio context, biology context, the goal might be very narrow. It might be like, “Determine the shape of a given protein.” That’s what that AlphaFold tool does. It might be broader than that. It might be, say, “I don’t just want to know about a protein shape. I want to do something with it. I want to maybe help you survive from a certain type of cancer.” And it might be really broad. We want to just improve people’s longevity as well. So, you have some kind of idea of an output that you care about, an outcome you care about, and you want to improve its rate of progress. Now, to do that, we need to do a bunch of research tasks. So, it depends what the outcome is. If it’s the Fed doing monetary policy research, there’s a certain set of research tasks you do. And if you’re doing drug discovery for cancer, that’s a very different set of research tasks, but there’s a set of research tasks that you would undertake in order to push forward on some measure of progress.

And what I want you to think about is that all those tasks, some, most maybe currently, can be done by labor, like workers, researchers, but some of them might be done by machines. And AI is one type of machine and may be a very exciting machine in the research process, partly because it might do a lot of tasks and partly because it might get really good at those tasks. So, it might radically improve.

All right. So, how do we think about that? I want you to think about some research area, biochemistry, rockets, science, making CubeSats to low earth orbit, whatever it is. And there is a whole set of tasks, okay? Some are done by labor, some are automated, meaning they’re done by a machine, which doesn’t have to be an AI. It could be an AI, but it could be a centrifuge in a biolab or it could be something else like telescope.

And then at each of those tasks, we have a productivity. We’re so good at that. We’re not very good at this. We’re great at that. At certain tasks we’re better at than others, and labor has got a certain capacity to do certain tasks, and then machines have a certain capacity to do the tasks that the machines do.

Okay. So, when you think about it that way, what you’ll quickly see is that if you want to make a claim about what AI is going to do in some research area, so pick an area, you have to take a stand – may have to take a stand on a bunch of other things – but you definitely have to take a stand on three key questions: One is, among all the research tasks, what share of those can AI actually do? If it’s a small set of tasks, it’s going to have a smaller effect. If it’s a large set of tasks, it’ll have a larger effect.

Second, how good is it at the tasks it does? Let’s say we give that task to AI. Is it sort of as good as a human? Is it a lot better than a human? That’s obviously going to matter. The third thing, which is less obvious, is what we’re going to call the strength of bottlenecks, which is to say, how interdependent are these tasks? Can you make progress at, say, solving a certain type of cancer just from understanding a protein? Or do you need a lot more than that to make any progress? So, the role of bottlenecks is going to be very important. I’m going to detail that as we go.

All right, so just some intuition. So, the first thing is that if there’s a set of tasks and some is done by labor and some is done by a machine, when machines are able to take over more of those tasks, so let’s say AI can now do these things that humans used to do, then what’s going to happen? Well, that’s going to sort of accelerate progress. Right. Why? Because the labor is now focusing on a smaller set of tasks, and so labor can concentrate on these tasks and then get more of them done, and then machines are doing the rest. And so automation, if you will, leverages scarce labor. It allows labor to go further. And so you’re going to get more output of research, more output of ideas per person or per dollar, if you will.

All right, the second thing is that AI can get good at the tasks. So, conditional AI may or may not take over a certain task, but if AI does take over a task and it’s really good at it, that’s also of course going to make that task, it’s going to relieve that as a constraint, and we’re going to have a lot of that task done and that will also tend to accelerate progress. So, these are the two features of AI. It’s going to leverage scarce labor by taking over more tasks and then the tasks it does, it might be really, really good at. Good at because it’s very smart, but also maybe good at just because it’s really cheap because it can do a lot fast or it doesn’t cost very much to ask an AI to make some code. And so it’s not maybe better code than the human would write, but it’s done a lot faster and a lot cheaper and that’s a huge advantage.

Okay. However, so that sounds optimistic too, but here’s where the fly is in the ointment. It’s that bottlenecks in research or many production processes seem very, very powerful. So to sort of say that, let me just back up for a second and talk about computers in general and the real economy, not just research and development, but just anything we do with computers.

Because of Moore’s law over the last five decades, computers have become extraordinarily productive. The kind of amount of calculations or so called FLOPS, how many calculations they can make per dollar is grown by like 10 to the 17th, a number that’s so big it’s very hard to get your mind around it. And computers, they’re not just fast at compute and cheap. They’re everywhere. They’ve gone from sort of nowhere. They’re very niche applications on a few early mainframes to something that everyone is using at work, at home. Think about how much time you spend staring at a screen of a computer in a given day. It’s like everywhere, all the software, all the applications. We use it in so many different ways. And so the computers have become incredibly efficient, and they’ve spread in their use, partly because of that, to so many different kinds of applications.

And yet, if you look over the last 50 years, while computers have gone from like one to 10 to the 17 times more productive, if you look at the total average productivity in the economy, what we call total factor productivity, it’s gone from like one to three. So we’ve been growing, but it’s far, far more modest. And so you might wonder, how is that possible? How can we have this ubiquitous technology called the computer that is so amazing and just incredibly more efficient than it used to be and yet not have huge gains in productivity overall?

And one answer to that, the most obvious answer, is something economists often refer to as Baumol’s cost disease, but we can just think about as bottlenecks. It’s because the things we get really good at actually sometimes turn out not to make that much of a difference. So driving growth, that improvement, but then we become held back by the things that we are struggling to improve, that are still important, but are struggling to improve.

So for example, leaving computers aside, agriculture is another area where we’ve had incredible technological progress. We’ve gone from hand labor, say harvesting crops to the combine harvester as I’m showing you in the upper part of this slide, which is an unbelievably efficient machine. I think a combine harvester can bring in like millions of pounds of corn in a day, one combine harvester. It’s unbelievable. And so corn at the farm gate is now much, much cheaper in part for that reason.

But there’s a lot of other things we do in society that are huge shares of our expenditure and GDP or our value that magically, not like combine harvesters, not like computers, there’s been no amazing machine, there’s been no massive improvement or really much of a change at all in how we do things. So, I’m showing you just a picture here of a family having a meal in a restaurant from 50 years ago versus today. It looks kind of the same.

I mean, it’s the same technology. There’s still wait staff, there’s a chef, a cook, there’s certain equipment in the kitchen. There’s no magical radical improvement like we see in transportation or farming or in computing. And yet, what we spend on meals out of the home is much larger than what we spend on computers because computers are so efficient, they’re so cheap, we don’t spend that much on them. And we end up being dragged down by the things that are important but hard to improve. So, that’s also going to be education services. It’s going to be government services, construction. A lot of things that are big in parts of the economy, but have not seen these radical improvements.

Now, coming back to R&D, I think we see a similar kind of picture. So, we have already developed through history really amazing research machines. Just to show you some pictures, obviously the telescope on the upper right or the microscope on the lower left, these are improving human powers of observation basically by an infinite amount. You can see the heavens in a way that no one could ever see with their eye, and we can see things that are very small in a way that no one could ever see with their eye. So, we have just these incredible empirical tools. And telescopes are not an incidental niche machine. They’re like the center of astronomy and cosmology. Microscopes are not an incidental niche machine. They’re the center of much in biology, for example.

For the economists here in the room, on the upper left, of course I’m showing something we know, which is Stata, which is a standard statistical package. And again, analyzing data, that’s a large share of what we do professionally. We might do some theory, we might do some empirics, but we’re using statistical software that’s a high share of our activity. If you think of a machine like Stata, the compute and the software, this is so much more efficient than we are. It’s unbelievable. So, you can run a regression with a million observations in a few seconds. If you wanted to do that by hand, it would take you your entire life to do the matrix calculations, and it would take me my entire life, and I would also mess up. I’d get it wrong, because at some point I’d make an algebraic error.

So statistical software, modern statistical software is just infinitely more productive basically than humans, and it’s a huge share of what we do. And here’s the question, “Has the rate of progress in economics radically accelerated with the rise of statistical software?” Do we accelerate understanding at 10X what we used to do? It’s hard to know. It’s hard to know what the measure is, but I would suggest to you that it doesn’t seem that way.

We don’t think of the post-statistical software era as this radical structural break in the rate of knowledge of the economy. We still struggle with understanding things. Similarly, telescopes are incredible devices at the center of cosmology, but now we have dark matter and dark energy, and people are still struggling. We have microscopes, but we still have all these diseases we can’t make progress on.

And so what I want to point out is that making progress on research problems, it often requires many tasks. So, we don’t necessarily have the right conceptual models in economics or maybe the right empirical strategies, even though we can do the analysis with the software that are really going to decisively answer really important questions. Similarly, we don’t necessarily have that syncing between theory and empirical evidence or all the empirical evidence we need with these other tools and other disciplines.

Even AlphaFold, which has won the Nobel Prize, and it basically just tells you the shape of these proteins across a wide waterfront of the human proteins and gene and other species besides, so is that going to suddenly radically accelerate our life expectancy? No, because if you talk to bio people, they’re going to tell you that that’s really important and it’s a huge achievement, but actually to do drug design, there’s so many other steps that are beyond that and are experimental beyond the knowledge of the protein’s shape in a basic way from the fold, the protein fold, that we still have to do all this other stuff, and this is going to help, but it’s not going to radically shift or accelerate. It’ll help, but it’s not going to be a 10X improvement in our rate of, say, discovering cancer meds. And we can maybe talk more about that in the Q&A.

So what does this mean? I’m going to maybe skip this in the interest of time, plus I hesitate to put an equation. I was told this is for the general audience. I’ve been very good about not putting any equations. But what I want to point out is that bottlenecks are a very, very strong phenomenon.

This is an equation which is called a generalized mean, which is just a way of taking an average of numbers. And when everyone thinks about the mean, they think of the arithmetic mean. They think, “Oh, I’ll just take up a bunch of numbers and I’ll add them up and then divide by how many numbers I have.” That’s the arithmetic mean. But you kind of know there’s a lot of other ways to take means. There’s the geometric mean, there’s a harmonic mean. And you might say, “Okay, well, when we combine stuff,” like if I were to combine people’s income, you could say, “Well, this person has $1,000,000, that person has $1. So together they on average have $500,000.” And that’s an arithmetic mean. But a lot of production processes don’t average together that way.

So for example, if I built a rocket ship which has like 1,000 parts, but I don’t put in the engine, it is not going to fly. The success of the rocket ship is not the average of the components. It’s held back by weak links, by bottlenecks. I need all the parts to work. Software doesn’t run if some of the parts fail. Your human body, unfortunately, our health is not the average of how healthy our organs are. If any one of our critical organs isn’t functioning, no matter how healthy the other ones are, we will not survive.

So a lot of systems in economics and in health don’t depend on an arithmetic average of things. They depend on weak links. They depend on bottlenecks. If something is failing, you make much less progress or you have much less success than you think.

And evidence that I’ve done with a former postdoc, Mohammad Ahmadpoor, suggests that actually the way science and invention works is taking a harmonic average. That’s this equation with rho with equal to negative one, but don’t worry about that. The point is that when you get to a harmonic average, it’s not what you’re good at that matters, it’s what you’re bad at that matters.

I’m taking the harmonic average here just to remind you of what harmonic averages are. If rho is negative one, that’s a harmonic average. So let’s say that we had a productivity of one across all the research tasks in some cancer discovery problem. And then I took half of the research tasks and I made us infinitely productive at those, they do it perfectly, instantaneously, for free. You might think, “Wow, we’d be infinitely good at cancer discovery.” But when the world is the harmonic average, yeah, you’re really, really good at this one thing and maybe even half the things. But if the rest of the things are still like you’re only as good as a one, not an infinity, when you take a harmonic average of one and infinity, the number you get is two. It’s a lot closer to one than infinity.

So when you have harmonic averages, it’s what you’re bad at that really kind of holds you back. A way to think about this in R&D is maybe AI will be very good at conceptualization, it’ll give us lots of creative ideas to try, but still we have to do a lot of experimentation, and experimentation is going to be the bottleneck. And so our rate of progress is going to be severely muted by that problem. That’s not to say we won’t accelerate, but it just means that some of the more radical ideas about how AI could radically transform science and R&D are less likely.

So just to summarize: innovation is central to progress, I started there. R&D is really, really an important process in the economy. The implications of AI for the rate of progress are going to depend on three key features about AI: It’ll be the share of research tasks the AI performs in some area. It’ll be AI’s average productivity at the tasks it performs. And then it will be the strength of bottlenecks. And my last comments were trying to point out to you just how powerful bottlenecks are.

So AI can be very good at certain research tasks, for example, even the creative tasks that we started with. They might be really creative in a sense, because of its ability to make combinations across a broader set of expert knowledge. But executing on research typically requires many other tasks. And then bottlenecks are powerful. In fact, a bottleneck, as I just tried to show you, is more powerful even than the number infinity. Bottlenecks kill infinities.

So even a super-intelligence of AI, if it’s limited to certain tasks, it can easily be overcome by the bottlenecks, and so it will make less of a difference than you think. And empirical evidence does suggest, both in the real economy, which we didn’t talk about, but we could if you’d like, and in the research and development process, that in many areas we seem to see strong bottlenecks. So I think that has to temper, in some sense, our expectations. So we can see acceleration, and we’ll see some really cool stuff, but it may not be the radical transformation that some people have been suggesting.

Okay. So that was my presentation. Thanks very much, and I look forward to some conversation and dialogue.

Huiyu Li:

All right. Thank you very much. Thank you very much, Ben, for a great presentation. Not all of us here are researchers, so I found that your presentation really brought to life what is research and development for everyone.

I also like how your framework with tasks brings some of what we hear about AI into context, that you might hear that there are amazing advances in AI by narrow benchmarks, but then the question is, is that really going to change a lot of things? And your framework raises all the relevant questions. So to just bring the framework to a little bit more concrete, for you as a researcher, suppose your “yt,” your output, is like a research paper. What would be examples of like tasks, and how AI could expand the task and improve productivity and the bottlenecks? Can you just give us some examples?

Benjamin Jones:

Sure. So I’m an economist, so I tend to use social science methods. I think we probably start with a question, right? I have a question I’m thinking about, and then I’m trying to assemble ingredients that are methods and data. So let’s say it’s an empirical question. I’d be thinking, “Is there a dataset that has relevant information about that question, measurement?” And if the question is a causal question, I might say, “Well, do I have a causal method, or am I looking for a more descriptive method?” I’d be thinking about the tool sets I could use depending on what the question is.

Maybe I’d say there’s facts, but the really interesting thing here is how the facts intersect conceptually. And so maybe I want a theory, even a formal theory, and the empirics and the theory are going to stick together. We think about a question and we think about an avenue, and that’s the door, and then we go in the door and we try and we fail.

So then we say, “Okay, well, I had this dataset and I had this idea.” And then I look at the data and I’m like, it’s just very mushy, or I realize there’s a problem and the data is missing something, or it’s got some issues in how it was constructed, or maybe I don’t have a good measurement. So I have to think about a new measurement, and I have to get some other data.

So we’re going to do some trial and error, and there’s a lot of tasks. And then of course, once you develop the story of the research and you have findings, you might have some tables and figures, you might have some theory with some assumptions and some simple proofs, then you’re going to move towards a lot of other tasks and trying to road-test it out. Let me attack these facts, let me attack the model. Can I make it more robust? Is it robust to other tests?

And then eventually, of course, you’re going to move, if it’s surviving all of these skepticisms and all these checks, you’re going to move to writing. You’re going to write it up, and there’s going to be literature reviews that you’ve been doing probably along the way that you’re going to coalesce.

So if you think about it, there’s actually a very large range of tasks that you’re doing. And AI might help in a way with a lot of those tasks, but there may be some tasks where it is not as well-equipped, at least now yet, to take over.

Huiyu Li:

So the recent advances in AI, like GenAI, compared to previous AI technologies, do you see any difference in terms of how it might affect research development? Maybe it could expand the range of tasks or do other tasks better. What is your view on that?

Benjamin Jones:

Well, I think there’s a big difference, and there’s a big question for all of us here. So if you think about the most successful discovery tools that are AI, they’re not these large-language world models. It’s not ChatGPT or Claude. AlphaFold is a very particular algorithm. GNoME is a very particular algorithm. They’re designed to solve specialized questions. AlphaFold determines protein shapes, it doesn’t do anything else. And in some sense, it’s been given a limited set of ingredients that are relevant to its problem. It’s been given a very particular AI-based algorithm that’s very distinctive, and then it’s been deployed on high-scale compute at low cost to basically tell you the protein shapes of all these things. So some of the most powerful examples we’ve seen so far are specialized.

When you go to the large-language models, now they’re the ones that are interesting from a creativity point of view, in the sense that they… I’m an economist, but I can chat about quantum mechanics with ChatGPT and I seem like I’m learning something. Maybe it’s not telling me the truth, but this is where you can start having very disparate ingredients, maybe in your creative process.

On the other hand, it makes errors, right? It makes mistakes. It’s very hard to check, especially when it’s saying something outside your expertise. It sounds good, but is it right? And so that could be very misleading.

I’ll point out to you, by the way, when you think about bottlenecks, mistakes are costly. So a lot of the benchmarks that people get very excited about with AI is, “Oh, it can take a test,” like a math test or the Math Olympiad, and it gets a 98%. That’s true, if the task was take a test, answering questions, it could be very, very good. But a lot of tasks, it’s that 2% that you got wrong that killed the whole project, right?

And this goes back to: when you take a test, we all get graded on the arithmetic average of our answers. We just add them up. But in a lot of… Like a rocket ship, it’s not okay to be 98% right. If 2% of the system will fail, the whole thing’s going to fail. So that’s tricky.

Now, AI might get to the point where it can do that, but I think this is where the LLMs are very interesting creatively, but they’re not necessarily reliable enough to just let them do their thing without humans in the loop or double-checking and being absolutely sure about key components.

Huiyu Li:

Yes. This certainly resonates with my own experience experimenting with Codex or Claude Code in my own research process. I find that now I can do a lot of tasks with AI that I couldn’t do before. I can give them my paper and my code for them to check for consistency, that I couldn’t do before.

And then also, because it’s close-to-home research, I also think a bit about bottlenecks, where I am the bottleneck in lots of the cases. Usually it’s not good to be a bottleneck.

Benjamin Jones:

Me too. Me too.

Huiyu Li:

Usually it’s not good to be a bottleneck, but I think if I take your framework, it’s actually a job-security, in some sense. Because your framework, if machine productivity goes up, the payment of share of income that goes to bottlenecks actually increases, because that pins down the total outcome. So in that sense, bottleneck is not always a bad thing.

Benjamin Jones:

Bottlenecks save labor.

Huiyu Li:

Yes.

Benjamin Jones:

On the other hand, they also can slow growth, right? But that’s exactly Baumol’s cost disease. The things that we’ve gotten incredibly good at, like agriculture or, say, automation and manufacturing – agriculture becomes a much smaller share of overall output, GDP. So does manufacturing. And then we’re left with services taking over everything because they’re the things we’re not very good at.

And the reason for that is that when you get really good at something, you can flood the market with it and its price plummets. So the price is falling fast enough that even though you’re providing more quantity of manufactured goods and more farming output, the price is falling faster, and so it takes up a smaller share.

When you get to labor, it’s good for labor if there are bottleneck tasks that we still need labor to do, because the money, the payments, the factor payments, go to the thing that is hard. And so you’re right. So we will survive longer as researchers and other workers, and the economy will survive longer and be pretty well-paid insofar as bottlenecks, some of them, remain the province of labor.

Huiyu Li:

Yes, you bring up that there’s a difference between different industries in terms of progress so far. Also, R&D as a share of sales spent in industry is also very different, you’ve shown that in your previous work as well. Do you see these advancements in AI changing that landscape, that maybe industries that have been slow-growing so far will have a chance to pick up in terms of growth?

Benjamin Jones:

Yes and no. Just to build what we were just talking about, in some sense we’ve gotten stuck with services as a huge share of the economy that we struggle to know how to improve. What does it really mean to improve financial services or insurance services or government services, for example?

And we’ve made huge progress in transportation in the past, from the horse to the jet plane, but since we’ve had the jet plane, we don’t really get faster across town or across the country. We’re sort of stuck, in some ways.

So I think, in a way, AI, because it is a cognitive technology, it may actually apply well to services. GPS tells us how to get places. If it can drive a car, it can take over a labor component of transportation services. It may be really good at a lot of service tasks in medical diagnosis, legal briefs, writing contracts, right? Things that take up a large share of the modern economy, we might finally have a technology that can actually make those things much less expensive. And so that actually could unlock growth and targeting it in areas where we struggled to improve productivity in the past.

So I think that is one reason to be optimistic. And research as well. Research is always… we underinvest in research as a society. There’s not enough people doing enough different discovery projects given how valuable it is to society. As I’ve tried to show you today, the evidence is that the social returns to research and development investment is just absolutely enormous, both in the private sector and in the public sector. So that’s a hard political-economy problem: to invest more in, say, science. But if you can leverage scientists’ time with AI and a given amount of spend or a given amount of scientists can produce more research, better research, that would be, as I tried to argue, it’s exceptionally exciting. And so that’s also been a hard problem. So I think there’s reasons to think that AI may be well-targeted to some of the places that we’ve struggled to improve in the past.

Huiyu Li:

Thank you. So let me move to the question portions. We received a lot of questions for Ben. So first, let me start with one that actually came from our live audience. Question is, have you seen any indication that AI might be able to help think about how to overcome bottlenecks beyond what previous technologies could do?

Benjamin Jones:

Yes. I think anything … In some sense, in bottleneck situations, everything’s the bottleneck until you get really good at it and then the other things become the bottleneck. And just take coding. The places where AI seems most likely to do very well are things that we already do in a computing environment, in a virtual environment. So in biology and cancer discovery, there’s so much that has to happen in a wet lab that AlphaFold is exciting, but it’s still going to be a lot of bottlenecks. But if it’s a technology where we do it on a computer anyway … So if you think like computer science, if you’re developing algorithms or you’re developing machine learning algorithms, coding, where the whole work is on the computer, that’s kind of in the silicon environment where AI doesn’t have the bottleneck of having to go out in the real world and get a pipette or do some wet lab experiment. So I think those are areas where we can see AI maybe having its biggest effects.

Huiyu Li:

Thank you. You also have done research on collaborations or team researchers. So we had a question about, in terms of AI’s impact on collaboration among researchers, how do you think that would change the team setting for researchers?

Benjamin Jones:

Yeah. My expectation is that, other things equal, team sizes will become smaller because you can think of the AI becomes a collaborator. And it’s a pretty good collaborator. Going back to the burden of knowledge perspective I suggested earlier in the talk, everyone’s gotten narrower and narrower and narrower. So one thing I’ve explored a lot over the years in research is how do we respond to that as researchers? And one obvious answer is we work in bigger and bigger teams because we’re trying to pull together complimentary forms of expertise to tackle harder problems and bigger and wider problems. A classic example would be like the first airframe was the Wright Brothers, say, two people, they were like leading aeronauts of their time. Take a modern Airbus or Boeing airframe, it’s 30 different PhD disciplines just to design and manufacture the jet engines. So there’s just an enormous amount of depth of knowledge that has to come together.

So one of the reasons we look for teammates is in fact to fill in our own holes and kind of allocate according to comparative advantage in a team and sort of push things forward. AI is very good at some tasks. And so maybe we’ll use the AI instead. It also can stay up all night. It doesn’t want credit. You don’t have to worry about credit. There’s all sorts of things that it can do. So I think it’s an interesting type of teammate. And so what I expect is that we’ll see, and maybe you’re seeing this yourself. Huiyu, you were talking about you’re using AI, I’m using AI. Certain things I might have asked the research assistant before, I’m like, “Well, it’s just a lot easier to ask the AI that’s right here.” And I’ll just do that, and it gives me an answer that’s kind of like what a research assistant would’ve given me. And so that’s, for me, is, in some sense, a smaller human team as I start to team with the AI instead.

Huiyu Li:

Yeah, certainly. I’ve already seen econ papers with AI being the co-author. The question is how will that get published?

Benjamin Jones:

Someone told me a result that is not in the paper yet, but they said that team sizes are actually now falling. Where they’ve been rising forever. I haven’t seen that. That’s like hearsay. So I’m saying it in a livestream. So anyway, don’t quote me. I’m not sure that’s entirely true, but with some time, we’ll see if that’s actually a measurable effect.

Huiyu Li:

Thank you. Okay. So another question is about how AI may affect R&D investment strategies for governments and from firms.

Benjamin Jones:

Yeah. No, that’s a great question and it relates to something I was alluding to before. So there’s a lot of evidence that the social value and the private value of research and development is truly huge. I have a paper, for example, that says that for every dollar we put into R&D, society gets at least $5 back in terms of social value. And so it’s one of the great underinvestments. There’s no set date at which we will solve Alzheimer’s or certain cancers. It’s a matter of how hard we try, how diverse the set of approaches we take, et cetera. And so we, as a society, seem to greatly underinvest in solving these challenges through the research process.

And part of it is that science, when you discover something new, it doesn’t often have any direct commercial application. It’s a new discovery, but it’s going to unlock. It’s like you’re opening doors in the hallway, you’re opening doors so that then the private sector can come through and find new value. But the science part is kind of seeding that whole discovery and opening up new areas of that hallway, that’s so essential. So there’s a real kind of nice complementarity and symbiotic relationship between science and the market in this sense. But you’re going to rely on the public sector, philanthropy to support a lot of that science because it itself directly there’s no patent. There’s a paper, it doesn’t remunerate itself. So we need grants and we need other ways to fund it.

Given how hard that is for most countries, it’s a difficult political economy problem, I think it’s partly that the public as a whole doesn’t fully appreciate how important science is to unlocking their standard of living and improving it and solving problems. We end up in a situation where I think research dollars are scarce. They’re too scarce. And so to the extent that AI can take a given dollar and make it go further. So let’s say that researchers can now focus on certain tasks and the AI can do the software coding, write your STATA or whatever much more cheaply, then you can kind of move faster in your research. A given amount of dollars, a given amount of people, can go faster. And so that’s quite exciting to me because I sort of think we greatly underinvest in it. And so to the extent that AI does allow us to leverage those dollars, we should get a lot of social benefit.

Huiyu Li:

I guess there’s also a question about what is the value that we spend this dollar on in terms of like, are we going to help solve cancer or will we try to send the next man to the space? Yes, that’s a difficult question.

Benjamin Jones:

Yes. Or military applications. AI is tricky, right? So I’ve talked today about the economic implications and from the lens of history, our advances in science and research and invention, some of them are problematic, but on average, they have greatly improved standards of living and health. But AI is going to do a lot of different things because it’s usable in so many different ways. Is it helping decentralized actors build bioweapons? That’s a problem. Is it helping people … Now, it’s the recent news item, can we just hack the financial system because we can find bugs in all the software? That’s a problem.

And then of course we are in an international game theoretic competition between great powers, and as globalization is sort of fracturing and people are moving back towards great power competition, that probably weakens our capacity to regulate AI in a meaningful sense internationally. And we’re seeing a lot of autonomous weapons systems. And so as countries compete, one does worry that some of the application sets that are going to be coming from AI create various, possibly quite large risks for society. So while I think it could be great on the economic side, I think from a security side, it is, in some other context as well, it is potentially quite worrisome.

Huiyu Li:

Thank you. We also had a question about the competitive environment across firms. So questions whether smaller businesses versus larger businesses will better leverage AI. And I guess now you see one person start off, now that you have AI, you can start a business much easier. So the question is about how do you think AI will affect the competitive environment?

Benjamin Jones:

Yeah. It’s a bit of an open question. Because training these models, the world models, the big large language models is extraordinarily expensive, and then delivering the inference at scale. We have a small number of giant companies investing unbelievably large amounts of money, so they’re being very competitive with each other, but it has looked for a while like somehow they’re going to capture all the value. They seem to be in a race where they think that if they can get ahead of their competitors, they’re going to get a huge return on all this capital they put in the ground and they’re going to kind of win.

But I think actually there’s a kind of a counter narrative which is pretty strong, which is that the open source models are always just a little bit behind these lead models. And it may just be that the world models, these large language models are really commoditized. Token prices seem to be plummeting in terms of queries. They’re very competitive. So that may be not where the value is. That might be the kind of the background foundation. So where would the value be? It may be very cheap input from upstream that allows you to do a lot of things. And so then it may well flow downstream to small companies.

I think the question will be, as always with invention and who gets, like there’s the value of the invention for society, and then there’s who gets the value personally. Is there appropriability? So what would give any player in this space some market power where they can charge prices above their costs and then sort of capture value? And it may well be in proprietary data. So it may be that there’s world models that are very useful foundations, but then they’re fine-tuned to very particular applications and they’re much better at that application for having been fine-tuned. That’s one version of the world. And then the people who have that proprietary data to which they fine-tune it might be capturing a lot of value. It might turn out that these world models are just world models. They’re so good that they’re better at every application than something fine-tuned and trained. And so then maybe the value will go back upstream as well. So it’s a little bit of a question whether we need specialized models like AlphaFold’s versus large language models and where the ultimate value is going to be. Surely it’s going to be a mix.

But I think to the theme of your question, it also is seemingly lowering the cost of starting a company in the sense that you might have fewer founders, employees, because you can use the AI as an input into your team. And so that’s another type of benefit beyond who captures the value. It’s, “how costly is it?” “What’s your cost side look like?” And is it lower cost to be inventive and try to do something new?

Huiyu Li:

Yeah, smaller team sizes. So one final question from our audience was, can you share insights on how AI’s improvement for R&D could actually translate into lower cost of goods for consumers?

Benjamin Jones:

Sure. Well, so sometimes we think about a product being improved in terms of its quality, and that’s a little harder to think about cost, but we get more for the cost we provide. But often we think about a production process being improved, for example, a process improvement and a production process, a better material. Firms are going to, on average, choose the lower cost way of making a thing, and then they’re going to compete with each other. So if they’re competing with each other, they’re going to be pressed and they’re looking for lower cost ways of doing things, and they’re going to say, “Now choose an AI to do that task as opposed to a human,” because it’s going to give them a cost advantage. If they’re also competing with each other, that’s going to pass on to consumers. So then the cost advantage of the technology passes through into lower real prices for consumers, and so people benefit in general. And that’s sort of the broad process of innovation on average going over a long period of time.

Another world would be one where there’s a monopolist who sort of has a lower cost way of doing something, but they can just mark up for, say a pharmaceutical under a patent, they can mark up the price really, really high, and so that can make it hard to access and then trying to capture more of the value for themselves. So you could see that sometimes too.

But by and large, in the history of innovation with competitive markets, we’re going to see exactly … That’s why the standard of living is so much higher today than it was in 1870 because the value is effectively going on to the consumer.

Huiyu Li:

Thank you very much. Okay. So thank you so much, Professor Jones, for a wonderful seminar. Thank you.

Benjamin Jones:

Thank you.