AMGFC24 - Betazone: Keeping AI on Track

Zeina Soufan: Good evening, ladies and gentlemen. I am truly delighted to welcome you all to this important session on keeping an eye on Iraq. As a television journalist who recently embarked on a thrilling, yet demanding journey of launching a new TV show. I initially felt a wave of hesitation when the World Economic Forum extended their invitation for me to moderate this session. The prospect of balancing my own program with the responsibility of exploring this significant topic was a bit worrying, particularly given my status as an enthusiastic learner rather than an expert in the field. Nonetheless, the reason I could embrace this opportunity rested largely in the reliable support I knew I would receive from my air assistant. I affectionately regard this assistance as she a reflection of the remarkable partnership we share. While I was busy watching previous lectures of my guest on YouTube, she was efficiently summarising numerous academic papers that have helped me shape today's talk. This collaboration not only enabled my presence here today, but also deepened my understanding of the capabilities and future of AI. Seems I am eager to incorporate creatively in my television work later on. However, the rapid progression in air development also presents all of us with a compelling question as I continue to advance. What are the implications if such capabilities could ultimately substitute our roles altogether? This represents a profound and thought provoking dilemma facing organisations, economies and individuals alike. At this juncture, we are merely beginning to chart these possibilities. With critical decisions looming on the horizon at various societal and institutional levels. I am thrilled to share the stage with Professor Erik Brynjolfsson, who is the Jerry Yang, and Akiko Yamazaki, professor and senior fellow at the Stanford Institute for Humans Centre and director of the Stanford Digital Economy Lab. Thank you all for being here and I look forward to an engaging and insightful discussion. Welcome, everyone. And Professor, this stage is yours.

Erik Brynjolfsson: Thank you so much. It is such a pleasure to be here and have a chance to share some of the research we've been doing at Stanford at the Stanford Digital Economy Lab and my company work Helix. And let me start with a little bit of a discussion starter that Andy McAfee and I like to use at dinner. Sometimes we get you can use this when you go home and with your friends, we ask what were the most important things that ever happened in the world. Now that's a big question. And you get people giving all sorts of answers. Some people will talk about empires and conquests, natural resource discoveries or discoveries of whole new continents, kings and queens. And there's no real way to resolve what was the most important thing. But I'm an economist, and one thing you can do is actually measure how much the average person changed. And if you look at, say, GDP per capita, you what you see is that, nothing much happened for century after century. Millennia after millennia. The average person lived very close to subsistence level. Their families did. Their grandparents did. Their great, great grandparents did. So all these other things hardly made a difference in their daily lives until. At one point, boom, things started really taking off. What was that? Well. The what steam engine was invented in the late 1700s. And after that, things really changed for most people. Our lives are so different than those 50 or 100 or 200 years ago. We have about 50 times greater GDP per capita on average than our ancestors did a few couple hundred years ago. So I would argue that the most important thing was a technology, the Watts steam engine that ignited the industrial revolution and started this upward arc that's still growing exponentially. The Watts steam engine was the first really important general purpose technology. We would call it a T, and I would argue that AEI is a g p t. I know you're thinking, wait, GPT for no, that's generative pre-trained transformer. I'm very annoyed that we economists, we used to just say GPT and have it be general purpose technology, but the A.I. folks have stolen the anachronism. But I'm still going to use it in both ways. AI is both kinds. It is a general purpose technology like the steam engine, because it has these three important characteristics. It's pervasive. It can be improved over time and it spans complementary innovations. There are other GPT electricity is a GPT, the internal combustion engine. But I think air is a particularly important GPT GPT of our era. And you could say that it's the most g the most general of all general purpose technologies. And the reason for that is, well, this guy Demis Hassabis, just got the Nobel Prize last week. He's the co-founder of DeepMind. You go visit his offices in King's Cross in London. They have a little motto there. Our goal is to solve intelligence and then use that to solve all the other problems in the world. So that's their little mission statement. I'm sure your company and your country have a mission as well. Maybe it's an equally modest mission as this one, but it does sound ambitious. But in a way it's true, isn't it? If you could solve intelligence, you could do so much for the environment. You could do so much to address poverty, health care, so many other problems in the world could be helped if we could do better on intelligence. So in that sense, it's not just one of the general purpose technologies. It may be the most important, the final and ultimate general purpose technology. And that's what we're addressing right now, is these general purpose technologies that AI is transforming the world. These are the ones that drive that arc of improvement. And what McAfee and I think is that just as the steam engine helped power our muscles, we now have technologies that power our brains, our minds. And they'll be even more important in terms of accelerating economic growth and productivity. Now, part of the reason I'm excited is not just that this technology is available, but what we see is that it's rapidly improving. So here's some data from the A.I. Index, a report we put out at Stanford every year, about 520 pages. This is the latest one. And it's filled with charts like this. And what you see is a lot of lines going up into the right. They kind of stretch down the screen so they don't look quite as steep as they did on my original slide, but they're still pretty steep. And what they're going for is from, you know, 20, 30, 60% up to much higher levels of performance. 100% is actually the level of humans at doing those same tasks, whether it's image recognition or competition level mathematics. Not only are they going up, but this seems like it's happening faster and faster. So we are rapidly getting machines in more and more specific tasks that are able to do as well as humans. That dotted line is really important because it's a threshold. It's almost like a phase transition when like water turns into steam. When you cross that threshold. It's a fundamental change because when a machine can do things better than a human, then if you are an entrepreneur or a manager or a policymaker, you're going to start switching things over from human to machine. And that transition can have huge effects on the economy, and it's happening in more and more countries. There's still don't worry, there's still a lot of things that we humans are better at, but there are many where machines are actually more more successful and that number is increasing really rapidly. So the obvious question or the one that I get most asked most often is does this lead to the end of work? Are the machines going to replace us all? And that's an important question. Even other folks like this Nobel Prize winner, a different Nobel Prize winner last week, also got his Nobel Prize. Jeff Hinton. He's kind of the father of AI. He developed some of the technologies that led to all those upward sloping lines like deep learning. He had some thoughts about this. He was looking at how they could read medical images. And back in 2016, he said, hey, you know, it's obvious that machines are better than radiologists, so we don't need radiology anymore. We should stop training them. Maybe in five years, maybe in ten years. But it's clear. That's obvious. So he said that, what, eight years ago? So what's been happening to radiologists? Has the number been going down? Has it been going flat? What's been going on? Well, if you look at the data from a data driven kind of guy, whoa. It's going up. The demand for radiologists has been going up since that time. In fact, it's tripled over this short period from a thousand job postings to 3000 job postings and many other data sets give similar messages. So that's not what Jeff Hinton was expecting. It's not what a lot of people are expecting. Yes, it's true that I can read medical images, but when you talk to hospitals, they actually can't get enough radiologists. They need more of them. So what's going on? What's driving that? Well, the key thing is that air is an amazing technology, but technology is just the beginning. You need to do a lot more transformation of work to really get the benefits. And in particular, you need to think about all the tasks that are being done. In my view, it's not the job that is the atomic unit of work, it's the task. So a job is a bundle of different tasks like radiologists, as many different task economists as different tasks. Bus driver. They all do different tasks, not just one thing. You can take all those tasks and then you can score each of the individual tasks. I can help with some of them, and there's other ones that can't help at all. I can help with reading medical images. It doesn't help with lifting boxes. Once you've scored them, you can then roll them back up to the job level, but maybe weight them by how much time the person spends on it or how much wages are associated with it. And you can even roll them up to a whole company. Or I should add, you could roll it up to a whole city or country as well. And then you get a sense of where the tasks are that are going to be affected by a A.I.. This is the work that I've been doing with Daniel Rock and with Tom Mitchell, and we've been trying to understand which tasks will be most affected by AI and then how that's affecting jobs and work. It's not the. And what we found was there is not a single job where A.I. just runs the table and is able to do everything in each case. I was able to help with some tasks and not others, and there's not a single job where I had no effect either. Let's go back to our radiology example. So according to Owen, that from the Bureau of Labour Statistics in United States, there are about 27 distinct tasks that radiologists do. One of them is interpreting images, and that's what Jeff Hinton was talking about, reading the images and interpreting. Is this cancer? Is this emphysema? But they do other things. For instance, sometimes they have to administer conscious sedation. Now, I don't know about you, but I would not want an AI machine to come in and give me sedation. No, I want a human to be involved in that. And if you go through the 27 different tasks, what you find is that AI helps with some, but others you really need a human involved. And that's one of the reasons why we had more demand for radiologists, is that yes, air is helping, but you still need to keep a human in the loop. In fact, a loop. In fact, you still need to keep a human in charge. And in general, not for every occupation, but for most of them. What we see is this pattern where you want to have humans involved, even as parts of their tasks are being done by A.I. more and more. If you go through the whole economy. There's a study done by Daniel Rock and a team at OpenAI. And what they found was that about 80% of the jobs have at least 10% of their tasks that will be affected by AI. So almost every job has some parts that are some significant parts are affected. And 19%, almost a fifth of them, had 50% or more of their affected. Those tend to be the more high paying jobs. There are a lot of jobs like doctors, like radiologists, like lawyers. Those are ones that often required a or had the opportunity for a lot of AI to be helping them. So we're seeing this big transformation happening in work. So what should you do tomorrow morning? Well, whether you're a CEO or manager or CTO or policy maker, there's two pieces of advice. First off, you need to develop a plan. You have to figure out where is I going to be affecting you and do that not at the occupation or the whole company level. Go down to the task level. What individual tasks can I do? And then add them all up. And that's what we did in our study. You can go to our papers and see that once you've identified where those opportunities are, then you need to be ruthless about tracking them. I think that the time for experimentation is over. This is not just a time to be playing around with the technologies. You need to be looking at ways to deliver real business value, real economic value, real value for your countries. And that is doable. There's a lot of examples of that being done very successfully. So go ahead and track the progress and make sure you're doing it. Both those goals can be achieved if you pursue the task based approach, breaking things down to these individual tasks. And that's going to be a strategy going forward where we can shorten that time from invention of amazing technologies towards delivering better productivity. Shortening the gap between from technology to productivity. So ultimately, I'm optimistic that we will be able to use these technologies to have dramatic changes in the world. Boost GDP, boost unmeasured health and environmental benefits as well. And the key to that is not just great technologies, but transforming work with the task based approach. Thanks very much. So I think we're going to have a discussion. A little bit of time will become some some chairs, a little time for for discussion, maybe some Q&A. Thank you.

Zeina Soufan: Thank you, Professor, for a very insightful presentation. I'd like also to welcome our audiences and ask them to please feel free to raise their hands if they feel they have a follow up question to a question I ask the professor, or if they have any comment at any point in time. We're all partnering in this session. Let me start, Professor, by by asking you if you believe we have enough research that is being done by economists to understand really the economic consequences of AI.

Erik Brynjolfsson: Well, that's a great question. And I have to say that there's this amazing gap that's emerging right now that's growing right now. My friends in computer science are improving the technology. There's more hardware, there's more funding. There's some great, very smart people like Dennis and Jeff. And here we have Stuart Russell and others that are dramatically improving our capabilities. At the same time, our organisations, our institutions, our laws, and most fundamentally our economic understanding is not improving very rapidly. In fact, in some ways it's not improving at all. And that gap between the technological capabilities and what we actually understand is getting bigger and bigger as the technologies get better. So we need to close that gap. And what this specifically what we need to do is invest more in economics and sociology and policy and public policy research to understand better what are the implications for society. Because it is dangerous to have that gap. That gap is where most of our big problems and challenges of the next ten years are going to be. Also, where most of our big opportunities to make the world better are going to be.

Zeina Soufan: Professor, you mentioned radiology and radiologists. Are all jobs augmented, but.

Erik Brynjolfsson: Not all jobs are augment able, but more than most people think. Too many CEOs, when I talk to them or policymakers, the first place they go is, hey, I should automate this job or I'm going to automate that. And it's true that sometimes I can automate instead of augment, but usually the bigger opportunities are for augmenting, for using A.I. to make the job better and to increase what people can do. Sometimes there are things that we can largely replace what's happening. But I think the big wins are in keeping humans in the loop and using AI to add to creativity, to add to quality, allow people to do their jobs better. And that's where people should be focusing more of their attention.

Zeina Soufan: Do we have any real life studies on other jobs other than radiologists that can that have presented or produced actual numbers on how much productivity efficiency?

Erik Brynjolfsson: Yeah. Well, I've been involved in a number of these shows as part of my research. Let me tell you about one of them. In call centres, we were lucky that there was a kind of a natural experiment where a company rolled out a new call centre technology based on crest technology actually, and some people got access to it. They were used for call centres and some operators got access to it and some didn't. So you could kind of compare them. One against each other is almost like an experiment. And what we found was that the people who had access to the technology very rapidly improved in their performance, up to about 35% improvement for many of them. Interestingly, they also the customers were happier if you looked at got the the customer satisfaction surveys or if you looked at the transcripts, you could see the customer sentiment, how many happy words and how many angry words there were. And the sentiment improved dramatically. And last but not least, the the employees also seemed to be happier. There was less turnover in the employees, so they seem to like helping the customers more. And all this happened within just about 4 to 5 months. So it's a very rapid improvement. There have been other studies of coding writing software that are similar management consulting sales, so there's more and more evidence about the potential performance effects.

Zeina Soufan: Professor, you said that every manager, every CEO would wake up and do some mapping of his organisation, understand where to introduce a. Can this not be a bit overwhelming with all these advances?

Erik Brynjolfsson: It is overwhelming. And so when we did our first paper, I mean, we did we have a research team and we went through and we analysed them by hand. Now there's software available. I've got a Start-Up called Work Helix that does that in an automated way, and there's ways you can do it a lot faster. But I really think you need to be data driven. I mean, the alternative is to kind of do it with gut and instinct. I listen to anecdotes and stories and, and I think the time for that is passed. If you do this data driven, task based approach, you're able to identify the opportunities and there's evidence for which ones work and which ones don't. And then you can make decisions when all the different people in your organisation saying, you know, we need AI here. We need to either. And the vendors are coming in and their stories in the newspaper. You can't make sense of all that without this kind of data driven approach and going with the task level.

Zeina Soufan: Professor Will we need to be ready for serious organisational and managerial new models?

Erik Brynjolfsson: Yes. I think that the next 5 to 10 years are going to be a period of tremendous business transformation, the biggest business transformation we've ever seen. And there are going to be new trillion dollar companies created. There are going to be other some other big companies that are wiped out through this turbulence and this transformation. The technology is just the tip of the iceberg of a much bigger transition. New business processes, new ways of reaching customers, new core technologies, new goods and services. And right now, a lot of people, entrepreneurs, managers and companies are inventing them. Policymakers need to be watching this and preparing their workforce for this massive transformation going forward.

Zeina Soufan: Professor, if we are talking about growth and productivity regained due to this powered business with AI, are we going to witness some slowdown before we see some growth? Are we going to lose jobs before we gain jobs?

Erik Brynjolfsson: So so I think we'll will. Technology always been destroying jobs, has always been creating jobs at the same time. So we have this transition. One thing that definitely doesn't work is try to freeze everything in place and just say we're going to leave all the same jobs the way they always are. The successful companies, the successful countries have always had a dynamism to them and a transition. But there can also be a sometimes paradoxically, a bit of a productivity slowdown before it takes off. This is something I call the productivity curve. Can look a little bit like a J, a little down and then up. And the reason looks like a J is that initial period companies are doing that transformation. They're developing new business processes. They're reskilling the workforce. These are creating intangible assets, but they're not measured anywhere. And so all that effort doesn't show up as greater output. But once they have new processes and figured out how to use the technology, then you have a takeoff. But that early period can be very discouraging while you're waiting to see if you can get things working correctly. These J curves have appeared with other technologies and we're seeing them appearing with AI as well.

Zeina Soufan: Since you're mentioning measuring. Yes measurement, how do you measure productivity that is the result of cognitive work?

Erik Brynjolfsson: Yeah, that's one of the hardest things. Physical work is often easier to measure. You know, if a person is laying bricks, you can count them. If a person's you coming up with new ideas, how do you know if they're really coming up with new ideas working at it? Some people try to do things like measuring lines of code, and I think that's a very bad proxy. But you do need to have these KPIs, these key performance indicators. In the case of the call centre, for instance, I gave you the examples of productivity call centre volume, customer satisfaction, employee turnover. Every company has a lot of data about how their operations are going and you can tap into that data and with APIs and then turn those into KPIs. At my company workplace, we call it APIs for KPIs. It's a way of knowing what's happening in company and then getting these key performance indicators. You can even do it the whole economy level. We have a new metric called GDP, B GDP that measures the benefits, not just the cost. And that is also helping to understand some of the intangibles and the cognitive labour. But as the economy transitions through this business transformation, we can't just rely on the old metrics. Traditional GDP is wonderful, but it's missing many parts of the economy that are very important. And so the new KPIs, the new metrics are needed. And part of the CEOs or managers or policymakers job is to think hard about these new metrics and implement them.

Zeina Soufan: Professor, how important is it to have even small and medium businesses on board?

Erik Brynjolfsson: Yeah, these technologies help create new ecosystems for big companies and smaller companies. Often the smaller companies are the ones that are first to embrace some of these technologies, and there's a transition that goes on. I like to see the big companies also adopt them, but sometimes it's slower. It's harder for a big company to transition. CC Start-ups adopting them, but it's ultimately a whole ecosystem of different sized companies.

Zeina Soufan: Do you fear do you fear industrial concentration in the hands of the big corporations?

Erik Brynjolfsson: I do. I think that one of my concerns is that these technologies could be used to centralise decision making and centralise power, and that might be more efficient in some ways, but it also could be dangerous in terms of economic concentration and ultimately concentration of wealth and power. And I think most of us would prefer to see a world of widely distributed decision making. But we have to work towards them.

Zeina Soufan: Professor, you showed us that. A chart with 100%. The boiling level. The human level. Yeah. This. This, if I may call it obsession of having a AI that is perfectly imitating humans, right? Is this too ambitious on our part or. Yeah, not ambitious enough on our part. Yeah.

Erik Brynjolfsson: So that's a great question. And while I is getting very good at individual tasks, I think it's a mistake to think of there just being one dimension of intelligence and say that like once A.I. gets to human level, you know that that's it. The reality is, is it intelligence is multidimensional. That chart had many different lines. And humans are better at some already. You know, a pocket calculator is better than me at doing arithmetic. And, you know, bats can do echolocation in chimpanzees, believe it or not, have better short term memory. So there's many different kinds of humans. There are many different kinds of of intelligence. And if we use a machine, if we think of a machine trying to imitate a human perfectly, you know, which is sometimes called the Turing test of intelligence, I think that's much too narrow a focus, for it will make A.I. that doesn't exceed humans, and it may the ways that it could be exceeding it and also tends to replace humans. A better approach is to have A.I. that does new things, in particular does things that humans are not good at. And then we each have our strengths. We can have humans do the things that we're good at, you know, taking care of kids and talking to each other and interacting and machines do what they're good at, you know, optimising a route or doing some other things. And they each play to their strengths instead of working so hard to have A.I. imitate humans. I call this the Turing Trap, this idea of having a machine that tries to perfectly imitate humans. Instead, we should have A.I. that does new and different things.

Zeina Soufan: Professor, let me ask. I'm looking always for four hands.

Erik Brynjolfsson: You just raise your hand.

Zeina Soufan: Any questions? Okay. If you see anyone raising their hands, just.

Erik Brynjolfsson: Don't be shy.

Zeina Soufan: Yes, please.

Erik Brynjolfsson: Right in the front.

Zeina Soufan: Thank you. So, Professor, robotic process automation is over a decade old, and even that hasn't really been adopted widely to replace a lot of tasks that could be easily replaced. Why do we believe that AI is going to do better than RPA?

Erik Brynjolfsson: Yeah, so I think there's so much hype about AI and there's a lot of nonsense out there where A.I. is not appropriate. There's also real opportunities for that is are paying off. And the real job for all of us is to distinguish the hype from the reality. It makes me so sad when companies spend billions of dollars on failed projects and industries are wasting their money. But whether it's RPA or other kinds of software, there are use cases where it does work well. Maybe some of the proponents have overhyped it, and I'm sure some of the proponents of AI have overhyped it. But, you know, in the call centre, we saw up to 35% productivity improvements in software. In some cases, we've seen 50% or more predictive improvements in sales management consulting. There are studies that you can hear on my website. You can read about them where particular applications there's documented improvement. And I think there's more yet to come. But I don't think companies should just throw AI up against the wall and say, hey, let's you know, more. I just automatically works. That's that's a bad strategy. What you need to do is, as I said, do the task based analysis and identify which are the real projects, the payoff and which are the ones that are a waste of money. They should be doing that with RPA as well. They should be doing that with traditional robotics. They should be doing that with machine learning for predictive analytics. All of these categories, you've got to be data driven and not just assume it always works or assume it always fails.

Zeina Soufan: On the level of policymakers. Professor, do you feel that there is enough awareness around the globe among policymakers that I can really take us in totally different directions?

Erik Brynjolfsson: No, I think that this is the big challenge. And one of the things the message that I like to leave with policymakers, which is, is that there's no single inevitable future, that AI is either going to always pay off or the AI is going to destroy jobs, or that A.I. is going to lead to concentration. All of these are possible futures. But A.I. is a very powerful tool. And if you think about it, if you have a powerful tool, almost by definition that means you have more power to change the world. That's what tools do. That means your choices make a bigger difference than before. So we have a lot of different paths ahead of us, one with greater productivity, more widely shared prosperity, one with many people involved, or the opposite of all those things. Policymakers need to realise that we have the initiative. It's not what A.I. is going to do to us. It's what choices we are going to do. Using A.I. to transform the world. And ultimately, that means we have to think hard about what are our values, what are our goals, What is it that we hope to achieve with A.I. and make sure that we are steering A.I. in those directions rather than just blindly hoping for the best?

Speaker 3: Yeah. Well, thanks for sharing the early example of a radiologist.

Erik Brynjolfsson: Yeah.

Speaker 3: I think one of the big things the medical field is the problems was a purpose because in us, typically we have almost the data as true 20% medical as in most diagnostics.

Erik Brynjolfsson: That's right.

Speaker 3: So if that's a problems, then the purpose will be which task cost that? Then I should solve that problem. It's a process, right? So I believe is. But sometimes we sell. I can do many tasks. Yeah, but you have a critical problem first. So there's a priority?

Erik Brynjolfsson: Absolutely. Yeah, exactly. Once you have the task based analysis, it's not enough just to know the task. You need to see which ones I can do well and which ones are important too. That's why, as I said, you need to wait them by, you know, either the time or the money or the value and then focus. So it's when you you as a task based analysis, you can start seeing where those opportunities are ahead of you. And I think that we're moving from an era of sort of gut instinct and stories to an era of data driven decision making. And that's going to unlock a huge amount of value. I mean, literally, there's going to be trillions of dollars of value created. And some of it's even on measure like in health care of long, greater longevity and greater health span.

Zeina Soufan: It's a question from the gentleman in the back.

Erik Brynjolfsson: Right. Right there in the back next to you.

Speaker 4: Thank you. And one question. You mentioned productivity. And, you know, I think we all have high hopes. There's a colleague of yours, Erin Archibald, Darren Ajibola, who he came up with a paper which is quite interesting, which kind of comes up with, you know, adding up these examples of where one sees already, you know, progress in productivity, editorial up. And he says it's actually quite underwhelming. Right. I would like to hear your reaction to it. Sure. And related to this, you know, which you just mentioned to the question of the gentleman, you know, there's already a big high. If you say there are going to be trillions of gained in these industries and the markets seem to anticipate all of this. That's right. And this is obviously not a session to make a call on markets. But nevertheless, are you concerned and I know your research, if you know how long these gaps are timing wise, that, you know, the hype is in particular in the market is already discounting all of this in prices?

Erik Brynjolfsson: Well, I'm not going to make stock market predictions. But first, let me say that their own estimable is a dear friend of mine. I agree. But he also I mentioned two Nobel prizes. Their own also got the Nobel Prizes last week. So everyone's getting. Except for me. No, no, but but I'm I'm a big fan of his work and we've had many discussions about AI and other things. I do disagree with him on this particular study about the impacts. I won't get too much into the technical details, but he goes through the ways that I could affect and he kind of assumes no additional progress, that there's no additional implementation. And he and he also there's a number of like judgements you have to make. And in my view, each time he chooses the most pessimistic possible interpretation of these different areas. We're already beginning to see the glimmers of some productivity. But but, you know, I put my money where my mouth is, so I put a bet. There's a site called Long bets.com where you can place any bet. And Bob Gordon, another economist who's a pessimist, thought the activity would be slow over the next ten years. I said it's going to be higher. So we have a better long Bet sitcom about what productivity is going to be in the 2020s. And I guess we're going to find out in a few years who's right about that. But my bet is backed up by a lot of the research I did with Anton Cornick and with Martin Baily and with Daniel Rock and others. And part of it is because I think not only the technologies we already have are in place, but we're beginning to see the business innovation needed to harvest them. And I'm trying to shorten that time by showing people that task based approach. So maybe I'm cheating by trying to boost the productivity and teach companies how they can harvest it a lot faster. Two to the last part of your question. Sometimes these lags can be very long electricity and the steam engine had decades between the introduction of the technology and when it actually paid off with a I think it can be much shorter. The markets are anticipating that. And part of the reason is that I can be use all of you can use. I hope you all using it already like chat T or Gemini or or clod because it doesn't require special training. You can just talk to it in natural language like English. And that's why many of these sites in the call centre, which is 3 to 4 months before we started seeing some productivity gains. But I anticipate that the productivity gains are going to be significant. And I've talked to people, the Congressional Budget Office and elsewhere, that these payoffs will be quite large.

Zeina Soufan: I'm told that we've gone way over our time. Thank you so much. I encourage you all to continue the conversation.

Unidentified: Outside of the professor. Thank you, Professor. Thank you to all. My pleasure. Thank you. Thank you.