Pricing and Productivity: The Economics of AI

Louise Willard:

Welcome everyone. I am thrilled to join you today for our next emerging Tech Economic Research Network event. I’m Louise Willard, and I serve as the Executive Vice President and Chief Information Officer here at the Federal Reserve Bank of San Francisco. EERN Events are opportunities for academics, researchers, technologists who are studying the economic impacts of emerging technologies to exchange ideas, learn about research from different sectors, and share insights more broadly with those who are interested in these topics.

For those of us at the Fed, they also provide important insights on how developments from emerging technologies can potentially affect productivity, price pressures, and the labor market, all of which are critical to the Fed’s dual mandate of stable prices and full employment. Today’s installment of EERN Event Series will explore pricing and productivity in the age of AI through the lens of Amazon Web Services.

We are so glad to welcome Jacob LaRiviere, chief economist at Amazon Web Services. He’ll be joined in conversation by my colleague, Sylvain Leduc, executive vice president and director of economic research at the San Francisco Federal Reserve. As head of information technology services here at the San Francisco Fed. I’m personally seeing how rapid advances in AI are changing the way we approach and leverage technology to optimize business outcomes and change the way we work. So, this event promises to be timely and insightful.

As a reminder, this event is being recorded and can be accessed on our EERN website following the discussion. Finally, please note that the views you will hear today are those of our speakers, and do not necessarily represent the views of the Federal Reserve Bank of San Francisco or the Federal Reserve System. Over to you, Sylvain.

Sylvain Leduc:

Okay, thank you, Louise. Jacob, welcome to the San Francisco Fed.

Jacob LaRiviere:

Yeah, thanks for having me. It’s good to see you.

Sylvain Leduc:

Turns out you have links to the area and California more broadly. Let me guess, you have an office that’s not very far from here to start with, and then you did your undergrad at UC Berkeley, grad school at UC San Diego. Trained as an environmental economist to start with, taught at the University of Tennessee for a few years, then went to Microsoft and then Amazon.

Jacob LaRiviere:

That’s right.

Sylvain Leduc:

I’m curious to start with what attracted you to the tech sector?

Jacob LaRiviere:

So first off, thanks for having me. It’s been great to hear from a bunch of people that work at the Fed since this was announced, and I hope it’s useful. So yeah, so I’ve always viewed myself as an economist first, and then the topics that I researched initially were environmental, but my slant has always been heavily IO, and so in particular that took me into electricity markets. And my training is in structural IO my advisor was a structural IO demand estimation person, and so that’s my academic background.

And then in terms of the technology sector, I’ve always been pretty fascinated with it. So when I graduated in 2010, that’s right when the very first economists were going into sort of Yahoo Research and eBay, and these were my contemporaries, folks that I was on the market with. And so I’ve always kept very, very in close contact with them, and then got the R1 job. Research was going great.

And when there was a new group that was being formed at Microsoft called the Office of the Chief Economist, I was fortunate enough to have the opportunity to apply, and I was the first external hire in that group.

Sylvain Leduc:

Sounds pretty formal also.

Jacob LaRiviere:

It was. It sounds formal, but Microsoft has this long tradition of doing academic research. And so for me it was a nice transition into the technology sector, did that for six and a half years, and then wound up at Amazon to see a different way of doing it and to work on some of the fascinating problems there.

Sylvain Leduc:

You’re at AWS, Amazon Web Services. It’s a little bit different from what the Amazon we know and maybe the econ group that is there. Maybe before we start, what are you doing at AWS and how would that differ from, I don’t know, if you had the same job at Amazon, for instance?

Jacob LaRiviere:

Yeah, yeah, yeah. So AWS is Amazon Web Services. So, it’s a cloud service provider that effectively provides internet-connected servers, chip storage, and things like that. And there’s a bunch of services that sit on top of it. And so it is not B2C, it’s B2B. So, that’s the key difference.

If you think about the history of economists working in the tech sector, you might think of people that are either optimizing sponsored search auctions or folks that are trying to figure out what type of content to show in what type of situations. This is somewhat different. This is really about the efficient allocation of productive resources in a B2B setting. So, some differences about B2B is that it’s mediated by a sales cycle.

So, Sylvain, question, what share of the US economy, what share of the labor force?

Sylvain Leduc:

I just ask the questions.

Jacob LaRiviere:

So what share of the labor force are salespeople? Guess, order magnitude.

Sylvain Leduc:

Oh God, salespeople…

Jacob LaRiviere:

Is it 0.1%, 1%, 10%?

Sylvain Leduc:

Say it again.

Jacob LaRiviere:

0.1%, 1% or 10%?

Sylvain Leduc:

I’d say 10%.

Jacob LaRiviere:

It is.

Sylvain Leduc:

Here we go.

Jacob LaRiviere:

There you go.

Sylvain Leduc:

We didn’t practice before.

Jacob LaRiviere:

There you go. All right, round of applause. So the interesting thing here, if you think about it from an economics perspective, what’s the fundamental problem that salespeople are solving? It’s information frictions. And so ultimately, AWS has a ton of really powerful and amazing services that it’s… I think, customers that are just starting out, the builders that were some of the first customers are at this point very sophisticated. And AWS has tried hard to build services always for our customers at all different strata, whether it’s like the hardcore builders or more novices that are just getting their feet wet. And if you think about that, solving that information problem can take some time, and so the sales cycles tend to be longer. And so that’s the first key difference.

A second difference, and it shares a similarity with electricity sectors is that you got to buy a bunch of stuff, build a data center, and then amortize that down. And so that’s another, it’s more similar to airlines or the electricity sector, even a hotel or something like that where you’re building this big CapEx expenditure and then you’ve got to efficiently allocate it. So, those are two of the type of core differences, and we can talk about more, but it is different.

And so what this means is that it is structural IO in nature because the data, you’re not getting a trillion observations necessarily every day or whatever it is, and don’t quote me on that, but the rate at which your customers are interacting with your product, it’s high. But in terms of the points at which we can help them launch new services and build apps for their customers, it’s just different than how it is on dot coms and app. Does that make sense?

Sylvain Leduc:

Yeah, absolutely. And you’re leading a group of about, what, 60 economists, 50 economists?

Jacob LaRiviere:

60 economists, so it’s a combination of engineers and economists and scientists. And so a key difference between how economists are organized in the tech sector versus say an institution like the Fed or in an academic department, is that there’s a mix of engineers and scientists and economists. And so what that means is that economists, for example, aren’t necessarily running their own models on their client device.

And so if I ask this room, how many people are running models every month to provide information to policymakers, chances are you are all running models. But on a team like ours, there’d be like an ML dev and an ML ops environment where those models would be run very quickly so that you’re not doing that by hand every single month.

Sylvain Leduc:

So, it’s interesting you mentioned your knowledge of the electricity market because one thing we’re seeing right now is businesses are investing a fair amount in AI. Where you see it, I guess, most strikingly is investment in data center. And so you see it here, compare for instance, structures have been pretty flat, and you could think about the office market in particular having gone down around the country, but data center has been really, really powerful. You see this about tripling in terms of real investment over two and a half years.

I mean, the demand is so much now that even Google has put out this project, Suncatcher, about two weeks ago where they’re exploring the idea of putting data center in space to get closer, to have more solar power, because there’s just so much demand for energy and for data processing. And you see this also in terms of just the amount of investment we’re seeing in terms of real investment in information processing equipment. So think about computers, think about servers, all the parts that are related to that.

You see just recently the growth rate has been just outstanding, and some of it is diverted by trade because of the tariff and a lot of the equipment we’re buying are imported. But nonetheless, a big chunk of this is just the enthusiasm and the need to invest in AI. And this has been so strong, stronger than what we’ve seen during the tech boom of ’95, 2005.

And so firms are really enthusiastic about this. They’re investing, they’re positioning themselves. I’m sure Amazon is somewhere in that data. So I guess the question is what’s AWS doing right now? How’s your team exploring this new technology on a day-to-day basis?

Jacob LaRiviere:

So, a couple things here. First is that, yeah, there is a lot of demand for this stuff. So, AWS Q3 earnings, our growth was accelerating. So, it’s not like it’s decelerating, it’s accelerating. And so the demand is real and different companies and users are using this technology for a variety of things.

And it’s still very early. It’s still very early to where this is just the… It’s very new. I’ll just say that this is still very, very new technology and we don’t know what it’s going to look like yet.

So, in terms of how AWS is investing, absolutely we’re investing in this infrastructure, because we want to make sure that we have the capacity that our customers want and to meet them where they are. The second thing is that we’re developing new services. And so we’ll probably talk a little bit about this later. There’s going to be a demo later, and there are a bunch of different services to help people leverage these technologies, like Q, which is like a coding assistant you can think of that can sit in your terminal that you use or your IDE. There’s Q CLI. There’s other products called Quick Suite that have the capacity to fundamentally change how people work together. It’s pretty remarkable.

And so the company’s building the infrastructure, developing the services that sit on top to try and meet customers where they are, and then in terms of how we’re using it all the time, and there are some real material gains from this. So, a sharp example from my team is that there is… So, we have a forecast that we put out on my team. Now, that forecast, we want to expose it so that different people in the business can look at it.

And my guess is that there’s plenty of… Raise your hand if you put out a forecast in this room, and chances are there’s more than one person who puts out a forecast in this room. So, how do you expose that? And so the build to expose this in the way that we need it would’ve taken three months using pre-gen AI stuff. With our gen AI stuff, it was two weeks. Two weeks.

And so if you think about this, and I think we’ll probably talk about this in a little bit, the cost of software development broadly is probably coming down in some dramatic ways. And you can think about how that impacts the macro economy. And for me, it’s really exciting, like goosebumps exciting.

Sylvain Leduc:

So typically, you think there are fixed costs of adopting new technology. You might have to, I mean, you have to learn about it, you might have to… You may need the skills. Or I don’t know, in this case, the data environment, the compute environment might be different than what you had before.

So, I’m wondering if this is true or not because there’s a sense that this adoption rate of the new technology is faster than what we’ve seen in the past. Maybe we’re all playing on it, with it, we have it on the phone, there’s a familiarity. But to rip the productivity gains, I wonder if firms have to do just more than that, and if there are fixed costs that you’re encountering. I’m just curious about that. What have you seen in your experience at AWS?

Jacob LaRiviere:

Yeah, so there’s a couple of different things. One, we can talk about the within org diffusion of this technology. And so there’s going to be, in any organization, there’s going to be the leaders and the laggards for any new technological innovation. And that’s certainly the case.

And so there’s that. And there’s tactics that we can use to increase experimentation rates, but ultimately it’s exciting stuff. And when you have the aha moment with gen AI, I don’t know if anybody in this room has had it. I certainly have. I’m sure Sylvain has had if he’s playing with this stuff on his phone. But it’s pretty powerful stuff.

In terms of how I think about it, the cloud in general, even pre-gen AI, has changed the fixed cost of investment in fundamental ways because as an entrepreneur, I don’t have to purchase the technical infrastructure. I can rent it. And so it’s turned a CapEx model into an op-ex model. And that has been huge, at least I would argue.

At least, look, I don’t know that there is the study that has tried to isolate with the effect of that turning a CapEx into an op-exes on the macro economy yet. I would be curious though, just from an organizational perspective. And then if we think about the gen AI component, the thing that’s exciting for me is if we think about… And it’s not just the cost of software development, it’s the cost of economics research and having smart models and having the right models and things like that to inform policy and new technology creation.

But ultimately, there is a fixed cost to develop an app, say, and if the costs of development go way down, the amount of growth that that’s going to generate is going to be a function of the shape of that marginal fixed cost distribution that’s right there. And we don’t know what that looks like yet, but holy moly, I think that there’s going to be some exciting innovation to come out of this.

Sylvain Leduc:

Clearly, these businesses are investing in AI to become more productive, more profitable. And just to put a little bit of context here, this is just growth of average growth across a few periods where we’ve seen recently post-pandemic in red, compared to what we had the decade preceding the pandemic where productivity growth was really slow. But before that, we had this boom in productivity between ’95 and 2005.

And so we’re not at that level yet. And this may not be completely sustained or AI driven, but I’m wondering, at AWS, you have so much data… We get lots of people who registered asked questions. A lot of questions were about measurement of productivity, as a matter of fact.

And you think about maybe in manufacturing it’s easier to measure productivity. If you’re in the services sector, in the knowledge economy, that’s not always easy to measure that. So maybe to start with, I’m wondering if AWS has put out studies, if you’ve done randomized control trial to try to tease out what does AI do in terms of productivity enhancement, if anything? So, anything of the sort that your team has been putting out.

Jacob LaRiviere:

So, there are teams that have been studying this stuff internally, and what we’ve been finding is it’s similar to what the academic literature has been finding, is that it increases velocity and all the good stuff that you would expect just from looking narrowly at software developers where you can look at these outcome metrics that are pretty well-defined, like lines of code committed-

Sylvain Leduc:

More quantitative.

Jacob LaRiviere:

Exactly, exactly. And so yeah, we’ve certainly been doing that, and we’re finding, like I said, similar stuff to what the academic literature’s finding. The-

Sylvain Leduc:

Do you find it increases productivity for the least performing worker? Do you get that? That’s a finding you often get in the literature on this.

Jacob LaRiviere:

So, I’m not the right person to answer exactly in the minutia what happens there. What I’ve been focused more on is understanding how to improve adoption for my team, and then thinking about how to look at how it can change cost to serve our customers and how we can benefit our customers directly by having more innovations for them. And so if you think about the number of product releases and stuff like that, that hit the mark for our customers, that’s how I’ve been thinking about it.

And then if we think more broadly of what is the impact and how to measure this stuff, I think it’s a really challenging problem. And I think that there’s the two schools, one is the partial equilibrium school where it’s like, oh, we’re going to look at the lines of code and we’re just going to say the production process is exactly what it is right now and for that exact same production process, what’s the change? And that’s sort of the partial equilibrium view.

And there’s some good internal validity if you can do some randomized control trials and things like that. The challenge is that if we think of the general equilibrium model, I mean, this technology, if there’s ever a candidate for a technology to really drive dramatic labor productivity improvements, this has all the attributes of that, it’s pretty remarkable. And so if I ask you 15 years ago, how many people are going to be working on decentralized digital currencies that run on blockchain in the Bay Area, and what’s the total market cap of that going to be?

Sylvain Leduc:

I wouldn’t have made a forecast.

Jacob LaRiviere:

If I asked you 15 years ago, you’d be like, “What are you talking about…” And so it’s really hard to get those general equilibrium estimates. And so I think respecting that, if we think of how do we actually estimate what this stuff is, and just understanding what we can know and what we can’t is I think useful and having that-

Sylvain Leduc:

Right. You want to be guided by the experiments and the science maybe, but not blinded by it either.

Jacob LaRiviere:

Not blinded, not blinded by it at all. Because a lot of this stuff is going to be lower bounds because you’re really focused on the partial equilibrium.

Sylvain Leduc:

Where do you see the biggest improvement right now with your team?

Jacob LaRiviere:

Yeah, so there’s a couple different dimensions. One is the example that I just gave you, where it’s like the stuff that we were already doing, we can go faster. And then the other thing is that we’re actually developing new technologies to use this. And so the development of agents and so data mining agents to where we can look at how we can better serve our customers and build better products for them and meet them where they are in ways that are fast and interrogable.

And so not the type of stuff where we’re just getting, say, an insight using a natural language interface, but we’re also looking at SQL code that is generated from the agent and then run on… It’s actually executed on databases and you can investigate that it has been executed on the databases, so having the interrogable layer as well. It’s again this partial equilibrium story and then the general equilibrium story of how we’re able to move faster through these new technologies that our team’s developing and then also using the products that AWS is actually developing.

And so, there’s this product called Quick Suite that’s pretty remarkable. And so I don’t know if anybody’s… Has anybody even used Quick Suite yet? So what you can do is you can create a knowledge base. So think of my team. My team’s been around… That I’m fortunate to lead. I mean, these are really smart people. I have a really good, really awesome job, because just like here, my guess is that you have a bunch of smart cookies.

So, what is the sum of all the research that this group has put out in the past, say, five years? You’ve probably written a lot of papers and written a lot of code and stuff like that. Put all those documents into a knowledge base and then be able to hit that knowledge base with natural language querying and hypothesis generation, where you used to have to wait.

And if I wanted to find out about something about risk premia, I have to call up Bauer, and then he has to tell me who to talk to and then on and so forth. And it’s this game of telephone. You can just go to your knowledge base, query it directly, and then get to the answer so soon. And that lets you get to the value add that you are going to create as a specialist, as a researcher, in the economy very soon. And so a lot of these… It’s exciting.

Sylvain Leduc:

At the same time, the models have improved tremendously in two years, but they’re not perfect. So, what guardrails are you putting to make sure that what you get is really right, that when you call Michael Bauer, it gives you the right answer?

Jacob LaRiviere:

Yeah, exactly. So, that’s the interrogable component. And so having the citations and as a researcher, making sure that what the… In particular as an economics researcher, making sure that what the output is, is interrogable is, I think, really, really important.

Sylvain Leduc:

Right, right. So, many people that are online now are students and they have lots of questions about what it means for entry-level jobs, what kind of skills they have to develop to be productive in the new workforce. And I think some of it might be coming… You sense a little bit of anxiety of course. And some of it I think is coming from this, this idea that the unemployment rate of recent college graduates have been more elevated than the average unemployment rate in the US, which was not the case before.

There are different reasons for this, but it captures the attention a little bit that maybe AI is eliminating some of these positions. And so, it’s creating a little bit of anxiety for people who are still in college right now. So, I’m just wondering in your experience, how has that played out at AWS? Are you demanding less entry-level skills? Are you posting less or not?

Jacob LaRiviere:

Yeah, so to start with, if you look at this curve, I think that you can cut the data to say, “Okay, look at this, the green line relative to the blue starting in 2022 or whatever. Oh, no.”

But if you go back to 2010, I mean, you can see that green and blue… This is a longer-term thing. And I think that if similar to this partial equilibrium, general equilibrium story, there’s just a lot going on in the economy. I think that the demographic changes with the Baby Boomers and figuring out how all that stock of highly experienced folks relates to new hires, I think, that’s really tricky. And so it’s again, this partial equilibrium, general equilibrium story from a research perspective.

In terms of our group personally, we’re actually hiring more younger people into our group.

Sylvain Leduc:

Interesting.

Jacob LaRiviere:

Just anecdotally what happened is a lot of tech companies hired a lot during COVID and then hiring stalled out. So, now what happens is the organizations, or at least my organization that I ran, was a little bit thick in the middle and a little top-heavy. And so getting back to that nice shape that we would traditionally have has meant to actually more junior hiring. And so that’s-

Sylvain Leduc:

That’s reassuring.

Jacob LaRiviere:

It is. And then furthermore, I think that it’s, again, this partial equilibrium, general equilibrium story where I teach at University of Washington still, I teach one class a year, and the students that I’m teaching that are juniors and seniors are going to be native to using coding assistants.

And nobody that has, at least in my vintage, I am native to a coding assistant. Like I was telling Sylvain earlier, I learned C++ in the ’90s, and there was no coding assistants back then. And so these younger folks are going to be doing some amazing things with this technology, because it’s going to be baked in to their workflows and they’re going to be going so fast.

And fortunately, they’re going to need some experienced folks to make sure that… to guide them. And so I think from my perspective, it’s like should people like Sylvain and me be the folks that are worried here so that we don’t get lapped by this new generation.

Sylvain Leduc:

So, do you think that they have to have a different set of skills? Are you looking for a different set of skills when you’re on the market for new economists?

Jacob LaRiviere:

I think for me and for my group, the types of folks that tend to be successful in technology companies, I think that’s going to be fixed. It’s a bit more of an entrepreneurial component to it. That doesn’t mean that it’s not the other things.

But I think that making sure that people are invigorated by creating technologies and policies and products that help people and that help our customers at AWS, that’s the bedrock of things. And really wanting to change the real economy in ways that make people’s lives better.

Sylvain Leduc:

Just on demand and meeting the customers a little bit, if you back up a bit from Amazon, we hear a lot about dynamic pricing, for instance, and you could think that AI is just ripe for that, because it can analyze so much data. Are firms using this for that? Are they using it for pricing in general? What are you seeing?

Jacob LaRiviere:

The short version is right now, our team is not working on any generative AI real-time pricing. Where we are using it is to understand our customers better.

Sylvain Leduc:

Like preferences and what they…

Jacob LaRiviere:

Like what does good look like for that customer, and understanding what good looks like and what makes customers happy. And then doing more of that stuff and still experimenting on things, but quickly identifying what is not creating value for them and then what is.

And so, if I think about the Fed and its mandate, I think one exciting thing could be using novel types of data. So, if you think of 10 years ago there was the Texas Data paper, or I don’t, how old is Texas Data now? I’m not sure but… Pretty old. Yeah, exactly, pretty old.

So, I think ultimately what large language models can be good at is putting structure onto unstructured data. They’re really good at creating hierarchies and complex data structures. And so, I think that using aggregated data, like weekly summary statistics, that’s a way of doing it.

But I’m curious, and I’m excited about the idea of using novel types of data and then letting the LLM figure out where signal is from an economics research perspective, and then creating good policy as a result of that. And so, some more agent-based modeling that could come out of that I think could be exciting.

Sylvain Leduc:

30 minutes on this topic just flies by. And so we’re already about time now, but maybe some parting thoughts. What surprised you the most about this AI journey so far?

Jacob LaRiviere:

So, it has been fast. The pace of development of these services and these tools is just really fast, and part of that is due to the speed with which the tools enable development. Another surprising thing has been the diffusion within companies.

And so if you think of as an economist researcher and you say, “When does this company adopt AI?” And you’re defining that as the first user in that company first adopts some product, there is a within-company diffusion curve. And so, you’re going to have a lot of measurement error in trying to figure out what the downstream impact of that is for that company.

And so, I think some things like the pace, the impact, honestly, the impact and the ability to change modalities with Quick Suite and the knowledge base, where you take all your documents and how you can sort of query them. I think that can really change how people work and how people interact in the workplace. And like I said, we are early, early days.

We’re still in the early days for cloud service providers in general. It’s still early. And then gen AI is, what is it, a couple years old. This is so early, and so it’s really exciting.

Sylvain Leduc:

Well, this has been a fantastic discussion. Thank you so much for your time. You’ve given us a lot of information to consider as we continue on our journey here and thinking about adopting gen AI.

For those of you who are following us, our next event will be on December 15 where my colleague Huiyu Li is going to have a discussion with Raffaella Sadun, who’s an expert on management and productivity, and so please join us then. But for the time being, Jacob, it’s been a pleasure. Thanks for joining us.

Jacob LaRiviere:

Yeah, thanks for having me.