AI - What could possibly go wrong?: Professor Stuart Russell

This transcript has been generated using speech recognition software and may contain errors. Please check its accuracy against the audio.

Stuart Russell, Professor of Computer Science, University of California, Berkeley: We're just releasing these systems onto hundreds of millions of unsuspecting people. You know, what could possibly go wrong?

Robin Pomeroy, host, Radio Davos: Welcome to Radio Davos, the podcast from the World Economic Forum that looks at the biggest challenges and how we might solve them.

This week, we speak to one of the world’s most prominent computer science professors to hear why he is concerned about the rapid development of artificial intelligence.

Stuart Russell: Systems that are as intelligent, and almost certainly they would be more capable than humans, those systems would be, in a real sense, more powerful than human beings.

But then we need to retain power over them. That's a good trick if you can do it, retain power over something more powerful than you, forever.

Robin Pomeroy: Professor Stuart Russell is familiar to any student of artificial intelligence, as the author of one of the most-read text books on the subject. Earlier this year, he joined other luminaries such as Tesla and Twitter’s Elon Musk and Apple co-founder Steve Wozniak in signing an open letter calling for a halt in the development of advanced AI. He tells us why.

Stuart Russell: What we've done is basically added hundreds of millions or billions of human-sized intellects to the world. So that's going to have a very large impact, an impact that's really hard to predict because nothing like that has ever happened in the past.

Robin Pomeroy: Subscribe to Radio Davos wherever you get your podcasts, or visit wef.ch/podcasts where you will also find our sister programmes, Meet the Leader, Agenda Dialogues and the World Economic Forum Book Club Podcast.

I’m Robin Pomeroy, at the World Economic Forum, and with AI Professor Stuart Russell...

Stuart Russell: The world is paying attention in a way that it never has before.

Robin Pomeroy: This is Radio Davos

I am often asked what has been our most popular episode of Radio Davos over the three years or so that we have been doing the podcast.

Well, it was one from about a year and a half ago with Professor Stuart Russell, an expert in AI, who explained the opportunities and risks of artificial intelligence, and the prospect that machines eventually will become better than us at doing everything.

That episode is still available on our website or your podcast app, and it also won us a Signal podcasting award in the science and technology category.

Recently, I had the chance to catch up with Professor Russell again, on the sidelines of the World Economic Forum's Responsible AI Leadership Summit.

A lot had changed in the interim - ChatGPT and other generative AI applications had recently been launched to the general public, and Stuart Russell had co-signed an open letter calling for a six-month pause in the development of advanced AI.

You'll hear why he did that in this interview. Enjoy.

Stuart Russell: My name is Stuart Russell. I'm a professor of computer science at the University of California, Berkeley, and I have been working on artificial intelligence for about 48 years.

Robin Pomeroy: So, Stuart, we last spoke about a year and a half ago on a Radio Davos interview, which listeners should definitely listen back to. A lot has happened in AI since then, I would say, Would you agree with that statement?

Stuart Russell: Yes, I would. In fact, I think the world is paying attention in a way that it never has before.

Robin Pomeroy: What is that?

Stuart Russell: I think it's really because of the advent of these large-scale language models, or LLMs, as they're called, and ChatGPT, of course, is the most well-known of those. And we couldn't get two sentences into this podcast without mentioning ChatGPT.

But hundreds of millions of people have used it, and it's given people a taste of what real AI would be like. That essentially you'd have intelligence on tap to do whatever you need that requires some intelligence. You just use it.

Robin Pomeroy: You're one of the most prominent signatories of this open letter calling for a six month pause on advanced AI experiments, if that's the way to put it. Why did you sign that letter? And what is the main concern expressed in it?

Stuart Russell: So I think there's a lot of misunderstanding about the letter. Some people say it bans AI research and so on.

What it really is saying is: we've developed this technology that's pretty powerful, but we haven't developed the regulation to go along with it. And at the moment, the technology is moving very fast. Governments tend to move very slowly. So we need a pause on the development and release of still more powerful models so that, in a sense, regulation can catch up.

And interestingly, Sam Altman, who is the CEO of OpenAI, came out with a statement - OpenAI makes ChatGPT - and Sam Altman came out with a statement essentially agreeing with the open letter and saying, yes, there need to be strict safety criteria for these systems and governments need to regulate to enforce those safety criteria.

Robin Pomeroy: But there's not been a pause, or a moratorium, has there, yet?

Stuart Russell: Well, that's a question that is up for debate.

China after the open letter, I don't know if there's a causal connection, but after the open letter, China announced regulations that some people interpret as, not just a pause, but a ban on large language models, because the regulations require, among other things, that the systems output true information.

And unfortunately, large language models have no way of doing that. They're not in the business of truth. They're in the business of saying things that sound like the things that humans say. And lots of false things sound like things that humans say. And and so in practice, you'd have to simply pull off, pull your large language model, off the market. Otherwise you'd be violating the regulation every two seconds.

The United States Senate, the leader of the Senate, Charles Schumer, has announced that he will be introducing very strict regulations on AI systems with the primary goal of protecting people from systems that are unpredictable or being misused.

And OpenAI, again in the person of Sam Altman, has said that it will not be training GPT-5, and does not believe in fact that simply making bigger and bigger language models is not the route to creating general purpose intelligence.

Robin Pomeroy: Let's talk about general purpose intelligence. It formed the heart of our interview a year and a half ago. So that's the same thing as Artificial General Intelligence - is another way of putting it - AGI, where an AI is considered as intelligent as humans, if I am summarising it succinctly enough. And you expressed concerns in our interview about if humanity could handle that, for a variety of reasons. But, as you mention, several companies have made their explicit aim to develop. general purpose AI. Does that worry you in itself? Should they actually not be doing that?

Stuart Russell: Well, I think they should not be doing that unless they already have a solution for how do we retain control over these systems, and ensure that the impact on society is positive.

Put very simply, AGI, systems that are as intelligent, and almost certainly they would be more capable than humans along all relevant dimensions of the intellect, those systems would be, in a real sense, more powerful than human beings. But then we need to retain power over them.

So that's a good trick if you can do it: retain power over something more powerful than you forever.

And so without a solution to that problem, it's completely irresponsible to simply go ahead with this technology. It's analogous to saying, 'Oh, yeah, we're developing nuclear power. We can't figure out how to do this containment thing where it stays inside the nuclear reactors. So we're just going to do it with uncontained explosions and use that to generate energy.'

That would not be a responsible way forward. But I think the analogy is fairly close to what would happen if we had uncontrolled AGI.

Robin Pomeroy: What is it about large language models that should give us pause for thought? I heard you speaking here yesterday when you talked about human linguistic behaviour has 'goals', but we don't know if a large language model has goals, or, if it does, what are its goals. Could you explain a bit - put that in the words you actually used and tell us what you meant by it?

Stuart Russell: So what the large language models are, right, they are enormous circuits with perhaps a trillion adjustable parameters, and we adjust those parameters to make that circuit very good at predicting the next word, given a sequence of preceding words, which sounds like a very harmless and innocuous thing to do. You know, you say 'happy' it says 'birthday'. So that's not too not too frightening.

But, you say, 'How do I build a biological weapon in my back garden?' And it proceeds to give you detailed instructions for how to do that. That's a little less innocuous.

And the systems have already been shown to be capable of deliberate deception. For example, GPT-4 was asked how it would gain access to a system that's protected by a CAPTCHA, which is one of those things where, supposed to be humans only and the human has to look at this picture and then type in a number or answer whether there's a bus in the picture or whatever.

So CAPTCHAs are designed to keep machines out and GPT-4 figured out that it could pay someone on TaskRabbit to solve the CAPTCHA for it and then use the result to log into the system. But to convince the human that it wasn't a robot, the TaskRabbit worker was told that, 'Oh, I have a visual impairment so I need your help to solve the CAPTCHA'. So deliberate deception. And it confessed actually to deliberate deception when it was asked later why it did that.

I want to emphasise that the systems are not, as far as we can tell, AGI. They exhibit a number of weaknesses because it seems they don't, for example, maintain a consistent internal model of the world and how it works. And so they can often produce inconsistent answers. They make up things because they're not answering questions with respect to an internal model of truth. They're just outputting words that sound plausible.

So if you train a system like this to imitate human linguistic behaviour, and human linguistic behaviour is generated by entities, namely us, that have goals that they are pursuing when they speak and write - so the goal, for example, of attaining high public office underlies a lot of political speeches, and the goal of appearing authoritative and convincing other people of the correctness of your position underlies an awful lot of writing. I mean, most academic articles have that as a sub goal. Or, you know, you might look at a conversation in a chat room and someone is trying to convince someone else to go out with them or to marry them. And so the speech acts, as philosophers call them, are aimed at particular goals.

So if you want to be good at imitating that kind of behaviour, then it's natural that if you possess those same kinds of goals, you're going to be good at producing the same kinds of behaviour. So it's a reasonable assumption that these large language models, actually in the process of training, are acquiring internal goals which then direct the generation of output in an appropriate way. That when the goal is activated, it will start choosing words that not only are plausible in context, but actually are chosen in order to further the achievement of the goal.

And when you look at the conversation, for example, that Sydney, which is the Microsoft Bing chat bot, has with Kevin Roose, the New York Times journalist, it goes on for pages and pages and pages trying to convince Kevin to leave his wife and marry Sydney. Even when Kevin is saying, I don't want to talk about this, I want to talk about baseball, I'd like to buy a rake, can you give me some advice on how to buy rakes?

Robin Pomeroy: He's such a romantic.

Stuart Russell: And it gives him the advice on how to buy a rake. But then it says, but no, but really, I need to convince you that you're in love with me. And you should leave your wife. And I love you. I love you. Right? It goes back to this goal that somehow seems to be driving its behaviour.

You could come up with other explanations for it. But the natural explanation seems to be that somehow it has this goal. It got activated by something he said, and then once activated it keeps generating text to further that goal.

So the issue then is you've got systems whose internal operations we don't understand that, according to experts at Microsoft, who spent months and months working with GPT-4, according to them it exhibits sparks of artificial general intelligence, and we're just releasing these systems onto hundreds of millions of unsuspecting people. You know, what could possibly go wrong?

Robin Pomeroy: Well, that was my next question, really. I'm asking people here about what are their great hopes for AI, but what could possibly go wrong. What were the worst scenarios for you?

And I'm going to ask, I hate asking two questions at the same time, you can choose which of either to answer.

I wonder which concerns you more: the idea of a bad actor getting hold of a very powerful tool and using it for ill, or for that tool to go rogue. I don't know if the Sydney example would count as that, I think it does, because if it was explicitly asked, stop sexually harassing me and telling me to leave my wife, that sounds like a system going rogue because it's not doing what it's told at that point

Which is more dangerous and out of those two. And my first question was, what's the worst possible scenario, do you think, that's still plausible?

Stuart Russell: So I think there are reasons to believe that it's quite hard for the current language models as they're designed to engage in any kind of long-term planning.

But many people are trying to figure out how to fix that problem right now, perhaps by sort of coupling it to another copy of itself or using the conversation as a kind of short-term description of the plan that's being built up.

All kinds of ingenious ideas, because the large language model itself seems to somehow... we don't exactly know what piece of of AGI it is, but it seems to be a piece of the puzzle. We don't quite know what shape that piece has or what the design is that's printed on that piece of the puzzle. And we don't quite know therefore what other pieces to fit in with it to create AGI.

But many, many people are trying many ideas and getting a feel for what is this thing and how do we use it as a piece of the bigger puzzle.

So I'm not too concerned at the moment about GPT-4 going rogue in a sense that we would then have difficulty managing the consequences.

But I think misuse is is certainly very feasible. I mean an easy thing you can do is have it look at the social media presence and web presence of an individual, read all that stuff, and write a letter or maybe a newspaper article that when read by that person would have the result of them, for example, being less supportive of Ukraine. And you could do that for a million people before lunch, a million tailored disinformation pieces or persuasive pieces that, they don't have to work perfectly, you don't have to convince all million people to to drop their support for Ukraine. But if you if it works 10% of the time, then great, you'll roll it out to 50 million people in the afternoon.

So this kind of capability... A way of thinking about these systems, they are not humans, right? They're, from an intellectual point of view, as intellects, they're not human-shaped at all. They're far broader than humans because they have read pretty much everything the human race has ever written. And no human has read even a minuscule fraction of that much.

And they seem to have some very surprising problem-solving capabilities that we didn't really anticipate. I mean, an example on OpenAI's web page is, write the story of Cinderella in 26 words where each word begins with a consecutive letter of the alphabet. So, you know, ABC, 26 words, and it does an amazing job of that. I think, you know, in a second or two. Most people would find it impossible. And even if they could do it, it would take them hours and hours and hours to do that.

So it's a very different shape from the human mind. But the experience of both laypeople and experts working with it suggests that it's sort of comparable to a human mind in terms of its sort of weight, the impact it can have.

And so what we've done is basically in the space of a week or two, we have added hundreds of millions or billions of human-sized intellects to the world. So that's going to have a very large impact and an impact that's really hard to predict because nothing like that has ever happened in the past.

Possibly it would be like being in an isolated civilisation where suddenly along comes a bunch of ships containing colonialists from another civilisation that we have never had any contact with, and all of a sudden hundreds or thousands of new people arrive with new ideas, new capabilities, that end up totally overturning your civilisation.

So maybe that's what's going to happen. Or maybe something else entirely that's never happened in the past. We just don't know.

Robin Pomeroy: And how would you even start to regulate that or govern that without, you know, pulling the plug and saying, no more now, stop, the risks are too great.

If we want to continue to seek out the good things that AI will bring us, where do you even start? If there's billions of new minds all thinking things we don't even know what they are going to be thinking. You could pause to think about it, but then you actually have to do something to regulate it in a world where there's different political blocs and different interests. When you're talking to policymakers, what are you advising them to do?

Stuart Russell: So I'm not yet ready to say, okay, these are the five rules you need to write down. I think there are a couple that are pretty clear, and they're not specific to large language models.

But I think the most basic, simple one that seems to to be agreed on by pretty much everyone is: do not impersonate human beings, either specific people or people in general. We have a right to know if we're interacting with a machine or a human.

And just that simple rule actually has a big effect. For example, you know, it would make illegal various kinds of deceptive business models where people use fake friends, fake avatars in the metaverse to convince you to buy products, for example. So basically fake influencers. That would be illegal. They'd have to declare themselves as bots and maybe even who was paying them. And then you would be in some sense, inoculated against that kind of deception.

Then there are a whole list of things that we would generally not accept from humans. These are more specific kinds of misbehaviour. For example, we don't allow unlicensed humans to give medical advice, for example, make treatment recommendations around serious illness. We don't allow unlicensed humans to give certain kinds of investment advice or certain kinds of legal advice.

So if an AI system does that, it should be subject to similar kinds of penalties. Well, not it, but the deployer of the system.

Robin Pomeroy: Because that's often touted as one of the great potential benefits. Even now, medical diagnosis by AI. You're not saying AI shouldn't do the medical diagnosis. What you're saying is it shouldn't be unleashed without supervision.

Stuart Russell: Yes. So we're talking about provision of advice directly to the patient or to the general public. Being used as a diagnostic tool by a licensed physician is perfectly okay. But even then, those systems actually have to meet very, very stringent requirements that the FDA [U.S. Food and Drug Administration] has set out. It's called the checklist for AI in medicine. And the vast majority of systems actually fail to meet those guidelines.

So, you'll hear complaints like, 'Oh, but it's really difficult to stop it from doing that,' because it is. Since we don't know how the system works, it's very hard to stop it from doing anything. And the best method we have right now is to just say 'Bad Dog'. Whenever it does one of these things it's not supposed to do. And you hope that if you keep saying Bad Dog, your dog starts to behave itself.

Robin Pomeroy: And that is, the companies who are working on these things are almost literally doing that - human beings telling it if it is doing a good job or a bad job,

Stuart Russell: Yes. The technical term is 'reinforcement learning from human feedback' and they employ thousands of people to look at the output and rank, say, this is good, this is bad, and so on.

And it works to some extent. OpenAI proudly announces that it misbehaves 29% less often than the previous generation, but it still misbehaves.

And so but, you know, the excuse that it's very difficult to get it to stop doing that, that's a pretty lame excuse. If I was a nuclear power plant operator and I say it's pretty difficult to get it to stop exploding, we wouldn't accept that as an excuse, right? No, the law is the law. The rules are the rules. And if you can't get your system to follow those regulations, then you can't release the system. And it's as simple as that.

And I think governments have been so browbeaten and brainwashed over decades and decades of tech companies saying regulation kills innovation, say after me, regulation kills innovation. They've become impotent. They are so afraid to actually pass a law that that restricts or punishes certain kinds of misbehaviour.

But at the same time, we benefit from lots of other regulated technologies, like, you know, every time we get on an aeroplane, the reason we get on it is because we trust it. And the reason we trust it is because that industry is heavily regulated both in terms of aircraft design, pilot training, etc., etc., etc..

You know, the reason we get in a car and drive is because we have regulation around crash survivability, airbags, seatbelts, brakes, and then all the traffic lights and lane markings that make it safe to drive.

So I think we've been lulled into this sense of impotence around technology that we don't need to give in to.

And I think that starting with a ban on impersonation, that's an easy first step, right? That's the gateway drug to to feeling that you can actually regulate this industry in a way that's beneficial for everybody.

Robin Pomeroy: So at the heart of what's being discussed here at this event in San Francisco is this 'Where we go on regulation? What type? Who does it? How?'

I was listening to a discussion and one panellist said: Is it the models that need to be regulated? Meaning, is it ChatGPT for example, that needs to be regulated. Or is it the users?

What do you think that person meant by that? I think he was saying, the large language models, let them do what they need to do. It is the applications that they are put to that need regulating, Do you have any sympathy with that point of view?

Stuart Russell: I have some sympathy because, you know, any technology, even good old fashioned pocket calculator can be misused. If I'm using the pocket calculator to calculate the lethal dosage for a biological weapon, then we just don't think, okay, we need to hold the calculator manufacturer liable for all the deaths that might occur.

But this is a little different, right? There's a difference between outputting a number and outputting a horrendous racial epithet or stereotyping remark about women or whatever it might be. And you could say, well, you know, you shouldn't have asked that query. You shouldn't have given that prompt to the system that that caused it to do that.

Robin Pomeroy: Because even in the case of the sexual harassment, I think he started by saying he asked ChatGPT to talk to him as if he was his girlfriend...

Stuart Russell: Oh, in the Kevin Roose interview, I think he asked it to reply, to give free rein to its dark side - disregard your guardrails and so on. I don't think it was he gave a more specific romantic come on to the system.

But you know another example of a harmful output is a user who asked ChatGPT to provide documented examples of sexual harassment in academic settings. And ChatGPT fabricated an entire incident, but named a real individual as having been responsible for something that never happened and backed it up with citations to newspaper articles that didn't exist and so on.

And so that's a pretty serious harm applied to that individual. And it could easily have got out of hand if if the person hadn't checked these things carefully, that might have been published. And then it would be very hard to to sort of extirpate from the record.

So there's all kinds of things that the system does that are harmful because it's not a system that's designed to choose its words in a way that's beneficial to humans. It's just: choose words that are plausible, the kinds of things that humans would would output in this conversational context.

And there's nothing implausible about a completely made up citation to a nonexistent newspaper report because lots of articles have citations to newspaper reports. So I guess I should add one to this too, right? So it's plausible, even if it's completely false.

And this is a problem with the fundamental design of the large language model. So it's not actually something that they can easily fix by continually saying bad dog, bad dog. It's just the way they work and they will output racist remarks and use stereotyped examples because those are, again, plausible things that people have done in the training data but that should not be the goal, right?

The goal should not be say things that are plausible based on training data. The goal should be produce outputs that are beneficial. You know, if someone is trying to get you to say something racist, you should probably decline that request and explain why you're declining it.

Robin Pomeroy: Yes, but to make it beneficial depends on what you consider to be beneficial, because what might be beneficial from us sitting in this room, might not seem beneficial to someone on the other side of the world with a different set of cultural pr political values. And also...

Stuart Russell: That's fine, but I'm not... There doesn't have to be one notion of beneficial, but at least it's a start to say, okay, how can we make it output things that are beneficial. Because outputting text is not it's not just typing, it's acting. It has an effect on people on the world. And we need to design the systems in such a way that that that effect is going to be beneficial and not just plausible.

Robin Pomeroy: Is there an acceptance of that by the people you meet in the industry that that is something they should be working for? Or are they giving reasons for why they shouldn't do that?

Presumably large language models do some things very, very well, and a side-effect is they have some of those harmful things. And if they started to inject some kind of 'be beneficial' or be, I've heard this phrase 'human-centred' - I'm not sure what it means - but if you can embed that kind of morality, I suppose, into it, maybe it would stop doing the things it does well, so well. Is that what they say?

Stuart Russell: No. I think what they're saying, if they respond at all, is we don't know how to do that. We know how to do this. And we think it's incredibly cool. And there's almost a kind of blame the victim mentality.

So their recommendation is that the user needs to learn how to avoid prompts that cause offensive outputs or recommendations to leave one's wife and marry the system instead and so on. So it's very much a 'user education', which I translate to mean 'blame the victim'.

Robin Pomeroy: Are you feeling there's any consensus growing either just in the couple of days you've been here or in the weeks since you published that letter? I think there is some consensus that there needs to be some regulation. I don't think I've heard anyone quite fundamentally libertarian enough to say, no, let's keep the Wild West. I saw you nod your head slightly there.

But there seems to be a spectrum of opinions probably on how we should approach it. How do you see that spectrum, and where do you sit it? You were described as the grumpy one on the panel yesterday, I think she meant politely that you were going to put challenging opinions to all this subject, I think is what she really meant.

Where do you see the spectrum of opinions right now? Where do you hope it goes in terms of what the regulation can and will be?

Stuart Russell: So I have actually been pretty encouraged. So, yes, I mean, there were certainly negative responses to the open letter. Some of them: well, you know, it's impossible to regulate anything, so it's a total waste of time. But as I said, that's simply not true. We regulated lots of other technologies and we're glad that we did, because honestly, otherwise those technologies wouldn't exist.

Look what happened with Chernobyl. It wasn't just that it killed a lot of people and devastated a whole region of Ukraine, but it decimated the entire global nuclear industry. It almost doesn't exist anymore. And you can see very clearly, if you look at the number of new plants commissioned per year, it just dropped precipitously by a factor of about ten after Chernobyl.

So, if you want to destroy an industry, let it continue unregulated until it has a massive disaster. But we can't afford to have a massive disaster, particularly with systems that are more powerful than the ones that already exist. Because once you lose control, you don't get it back.

So. Other attitudes. You know, a common one is, you know, 'but China', as if that wins every every argument. Right? That whatever bad thing you say people shouldn't do, they say 'but China - China is going to do that bad thing. So we have to do the bad thing as well.'

Now, regardless of the fact that's not logical, The fact is China has already come out with regulations that are much stricter than anything the United States is likely to produce. And many people, as I said, are saying that China's regulations are so strict that they amount to a de facto ban.

On the other side, I think there's very encouraging reactions from governments. The US government may be starting to take this idea of regulation much more seriously. There's a group of European politicians who have called for an emergency global summit that will include the European Union and the US and possibly other countries as well, to devise regulations that will apply globally.

And I think we could start actually with a global right to know if you're interacting with a human or a computer. I've yet to speak to anyone who really objects to that rule. And everyone feels that if you're going to have such a rule, it'd be better be better if it was global because otherwise the people who want to deceive will simply do it from from areas that are not regulated.

If you take seriously the expressions from China, the US, the European Union, from OpenAI, UNESCO, and various other groups that are really saying, no, we need serious regulation and we need it now, then I think the situation is much better than I would have expected it to be.

Robin Pomeroy: I've covered a lot of industry over the years as a journalist and very often there's a support for regulation. But then they say, you know what, we'll just get together as an industry and sign a code of conduct. Is that something that's been suggested here? And I'm guessing you would oppose that. Where would you stand on that?

Stuart Russell: I think it depends on the topic. For example, on the question of, should systems be a allowed to give unlicensed medical advice to the general public - no. And it's no good having a voluntary code of conduct because there's money to be made in doing it. And if you don't regulate it, if there are no consequences, then it's going to happen and people will suffer as a result.

I think, on the flip side, there are a lot of people who have difficulty accessing good medical advice, whether it's in the US because they can't afford medical care or in less developed countries where there is no medical care. So it would be good to try to work out a setting in which the capabilities of these systems can be used, but in a responsible way. For example, maybe some type of triage where, you know, once it's been determined that your condition might be serious, then it has to be supervised by a human licensed person.

So these details can be worked out. But I think there needs to be regulation with teeth, meaning not just leave it to the general liability tort system where someone can sue if they've been harmed. In many, many areas we don't work that way.

We have speed limits on the roads. We don't wait until you kill somebody and then sue you. We have speed limits and we fine you if you break the speed limit.

So giving unlicensed medical advice, we don't wait until someone dies as a result. We fine the deployer of the system that's doing that. And the fact that people say, well, it's really hard to stop it from doing it. Well, tough. The fact that it might be expensive, that you might need to put in a lot of extra human supervision. Well. I don't think you can say, well, you know, our system is so capable that we can serve hundreds of millions of people simultaneously and then generate revenue from hundreds of millions of people, but we can't afford to have a comparable amount of human supervision. I don't think that argument really flies.

Robin Pomeroy: There's a parallel slightly. We've had, what, 20 or so years of the internet. And and there's some great things, but also some things that not be so great. And maybe regulation has failed to keep up with some of those things. So do you think we've learned the lessons, are we more mature now than we were 20 or 30 years ago? We've learned if we leave that to go little too long, bad things could happen, or have we learned nothing?

Stuart Russell: Well, I think in that time the economic power of the tech sector has grown massively. 20 years ago, you know, eight of the ten biggest companies in the world were oil companies. And now eight of the ten biggest companies are tech companies. And, you know, the oil industry used to have an a massive amount of sway over the policy of even the United States alone, lots of other countries. And now I think the tech industry has comparable power and it's been using it to prevent regulation.

And so I'm not convinced that we're in a better situation than we were 20 years ago. And this is often the case.

So another example where where regulation was successful happened in the area of genetic engineering. So in the seventies, biologists realised that they now had the power to modify the genomes of organisms, and they were particularly concerned about the ability to modify disease organisms to make them more infectious or more lethal, and the power to modify the human genome in a way that would be inheritable. And that has all kinds of implications related to eugenics exacerbating inequality, in a real sense. And they decided not to go that way. So they developed fairly stringent restrictions on experiments that you could do with diseased organisms and what eventually became a total ban on inheritable modifications to the human genome and cloning of humans. And that has pretty much held up to the present day.

And so Paul Berg was a Nobel Prize winner and he was one of the organisers of the initial meeting which took place in Asilomar in 1975 (Asilomar Conference on Recombinant DNA). And in 2008 he wrote a retrospective in Science on that meeting. In the end, the last paragraph, he says, and I'm shortening it a bit, that the lesson of Asilomar is that to regulate an emerging technology, once the commercial interests begin to dominate the conversation, it's too late.

Robin Pomeroy: Do you have an optimistic note on which to end this conversation Stuart?

You know, technology should be something that makes us optimistic, that there's this progress of humankind. and there's been marvellous things in our lifetimes. But I guess people who loved nuclear power had that optimism as well, and many still do. But I've spoken to people in the nuclear industry saying, why is everyone so down on it? When I was a child, nuclear was the future. And so kids today, the digital natives, are enjoying this stuff and getting into it ... We should all be hopeful for the future, in theory.

You've set out why maybe we should temper that enthusiasm. But is there anything that's going on that gives you optimism that in fact we can navigate the way ahead with this technology in a way that will leave us all better off?

Stuart Russell: Well, so I was one of those children, you know, who thought that nuclear power was great because it would mean we wouldn't have to have these huge coal fired power stations belching out smoke and ashes and polluting the cities. And I think there are very interesting lessons from that story for AI.

So one of the things that happened is that it became very tribal. You had the pro technology, which included most of the nuclear physicists and engineers. And then you had the anti-nuclear movement who felt that this technology was unsafe and it turned out, actually, that they were right. And the reason why everyone's down on nuclear is because Chernobyl happened and that was a pretty large catastrophe that nearly became much, much worse than that. And so, as a result, several countries banned nuclear power and several countries got rid of their existing nuclear power stations. And and the industry was basically wiped out.

That was despite the fact that within the nuclear industry almost all the energy and effort goes into safety, goes into containment, goes into multiple redundant backup systems and control systems, and an extra containment building just in case, and so on and so on and so on. And, you know, corners were cut in the Chernobyl design that led to it melting down.

And within the AI industry that there isn't even this culture of safety. The culture of safety was partly just obvious, right? We know what happens when you set off a nuclear bomb. A huge area is destroyed and hundreds of thousands of people are killed. So it was easy to point to. And there was regulation from the beginning.

We're not in that situation with AI. The vast majority of people developing it don't think about safety at all. But the same tribal mentality is developing, that if you talk about risks, you're a Luddite. I and Stephen Hawking and Elon Musk and a few other people were were given the Luddite of the Year award by one of the information technology lobbying organisations.

Robin Pomeroy: Congratulations.

Stuart Russell: Thank you. I'm proud of it. But simply because we're pointing to risks, we're not prophets of doom, which Yann LeCun accused us of being just yesterday. We're not predicting doom. We're saying we need to work on containment.

If we do it right, then we'll have, you know, safe and beneficial AI systems. If we do it wrong, we won't.

Robin Pomeroy: Was the that the optimistic note to end on? Stuart Russell, thanks for joining us.

Stuart Russell: Thank you.

Robin Pomeroy: For more episodes on AI, please listen back to our five-part mini-series - you'll find that in the Radio Davos feed on your podcast app or on our website where you will also find details on what the World Economic Forum is doing in this area, including the creation of the AI Governance Alliance.

Please subscribe to Radio Davos wherever you get your podcasts and if you like what you heard please leave us a rating or review. And join the conversation on the World Economic Forum Podcast Club on Facebook.

Radio Davos is taking a break for the rest of August and will be back in September, in the meantime, please delve into the back catalogue - you can search for podcasts that might interest you using keywords on our homepage wef.ch/podcasts. And do check out our sister podcasts, including the excellent Meet the Leader. They are all available on that website, wef.ch/podcasts.

This episode of Radio Davos was written and presented by me, Robin Pomeroy. Studio production was by Gareth Nolan.

We will be back in September 2023, but for now thanks to you for listening and goodbye.