Latest topics
» Old Film Trailers (to 2009)
by Sary Today at 0:18

» Marble head of Dionysus discovered under Rome
by Sary Yesterday at 23:49

» Beautiful Feminine Art
by OsricPearl Yesterday at 4:02

» Song Cover-Versions & Originals
by Sary Tue 18 Jun - 1:15

» UK Migration Issues
by Neon Knight Mon 17 Jun - 20:19

» Now listening to . . .
by Neon Knight Sat 15 Jun - 0:48

» Opinions on National Identity by Country
by OsricPearl Thu 13 Jun - 5:00

» Denmark's New Government
by Sary Thu 13 Jun - 0:27

» Favourite Film & TV Scenes
by Neon Knight Wed 12 Jun - 23:03

» The genetic argument against race - how good is it?
by Neon Knight Tue 11 Jun - 0:15

» Rise of Nationalist Parties in Europe
by Neon Knight Fri 7 Jun - 23:24

» WWII bomb in London
by OsricPearl Wed 5 Jun - 17:37

» Tommy Robinson
by Neon Knight Mon 27 May - 23:10

» I'm moving
by Sary Mon 27 May - 22:39

» Personality traits linked to political orientation
by Neon Knight Mon 27 May - 8:56

» The Englishmen with African Y-DNA
by Neon Knight Sun 26 May - 19:03

» Vampires
by OsricPearl Sun 26 May - 3:26

» Social Perception of Skin Colour in Japan
by OsricPearl Sun 26 May - 3:24

» Classic TV Theme Music
by Neon Knight Wed 22 May - 23:11

» Screening prisoners for brain injuries could cut re-offending
by Sary Sat 18 May - 12:46

» Animals - News & General
by Neon Knight Thu 16 May - 22:42

» Rainbows in European Mythology
by Sary Mon 22 Apr - 17:48

» What is your religion or phylosophical moral code?
by Sary Thu 18 Apr - 23:14

» Brexit Saga Update
by Neon Knight Fri 12 Apr - 22:53

» What do the Irish think of the English?
by Neon Knight Tue 9 Apr - 23:40

» UK: 'feral kids overrun drug-ridden jails'
by OsricPearl Mon 8 Apr - 3:46

» Science Knowledge Quiz
by Sary Wed 3 Apr - 1:01

» Germany's AfD Party
by Neon Knight Tue 2 Apr - 17:13

» 24 genes linked to wealth
by OsricPearl Fri 29 Mar - 16:14

» Mexican President Demands Reparations
by OsricPearl Thu 28 Mar - 2:22

» New Genetic Study of Iberians
by OsricPearl Tue 26 Mar - 15:31

» The Marrakesh Migration Declaration / Pact
by Neon Knight Mon 18 Mar - 21:28

» Christchurch Massacre - Right Wing Extremism
by OsricPearl Sat 16 Mar - 18:41

» Political Systems
by OsricPearl Mon 11 Mar - 2:57

» Monty Python Scenes & Sketches
by Neon Knight Sat 9 Mar - 21:15

» Why the paranormal is probably real
by Sary Sat 9 Mar - 15:42

» Classic TV Adverts
by Neon Knight Fri 1 Mar - 22:38

» My strange Phenotype
by Neon Knight Fri 1 Mar - 0:50

» Finland, Luxembourg & Ireland worst for anti-black harassment
by OsricPearl Thu 28 Feb - 14:45

» Language Test for Political Type
by OsricPearl Mon 25 Feb - 4:30

An AI apocalypse is possible

Reply to topic

View previous topic View next topic Go down

An AI apocalypse is possible Empty An AI apocalypse is possible

Post Neon Knight on Mon 19 Nov - 21:04  Quoting:

I want to convince you of something: that an ‘AI apocalypse’ is not a ridiculous thing to worry about. Sure, there are other, more near-future things to worry about involving artificial intelligence (AI) – including privacy and surveillance, and the use of AI-controlled weapons on the battlefield. But we can worry about more than one thing at a time. And while the idea of AI destroying humanity is, I think, not likely, nor is it so improbable that we can dismiss it, as some people do, as quasi-religious mumbo-jumbo, or bad sci-fi.

. . . The risk is not that AI might become ‘self-aware’, or that it might turn against its creators, or that it will ‘go rogue’ and break its programming. The risk is that, instead, it will become competent. The risk is that it will do exactly what it is asked to do, but it will do it too well: that completing what sounds like a simple task to a human could have devastating unforeseen consequences. Here’s roughly how that could go. One group that worries about ‘AI safety’, as it’s known, is the Machine Intelligence Research Institute (MIRI) in Berkeley, California. Their executive director, Nate Soares, once gave a talk at Google in which he suggested that, instead of The Terminator, a better fictional analogy would be Disney’s Fantasia.

An AI apocalypse is possible 191da8-20141201-fantasia

Mickey, the Sorcerer’s Apprentice, is asked to fill a cauldron with water. When the Sorcerer leaves, Mickey enchants a broom to do it for him, and goes to sleep. Inevitably enough, the broom obeys him perfectly, eventually flooding the entire room and tipping Mickey into the water.

Of course, if Mickey simply told the broom to keep bringing water and never stop, then he’d only have himself to blame. But even if he’d told the broom to bring the water until the cauldron was full, it would probably still have gone terribly wrong. Imagine the broom filled it until the water was four inches from the top. Is that ‘full’? How about one inch? The broom isn’t sure. Well, surely when it’s right at the top, and water is splashing on the floor, the broom is sure? Well, probably 99.99% sure. But, crucially, not completely sure. It can’t do any harm to add more water, in case, say, its eyes are deceiving it, or the cauldron has a leak. You haven’t told the broom to “fill the cauldron until you’re pretty sure it’s full”, you’ve just said “fill it until it’s full”.

A human would know that other things – not flooding the room, for instance – are more important than ever-more-tiny increments of certainty about how full the cauldron is. But when you ‘programmed’ your broom ‘AI’, you didn’t mention that. The broom cares about nothing else but the fullness of the cauldron. What we humans think of as simple goals are actually surrounded by other, much more complex, considerations, and unless you tell the AI, it won’t know that.

There are other problems. For instance, the goal of ‘fill the cauldron’ is most easily completed if you, the broom ‘AI’, are not destroyed, or switched off, or given another new goal. So almost any AI would be incentivised to stop you from switching it off or destroying it – either by fighting back, or perhaps by copying itself elsewhere. And almost any goal you are given, you could probably do better with more resources and more brainpower, so it makes sense to accumulate more of both. Eliezer Yudkowsky, also of MIRI, has a saying: “The AI does not hate you, nor does it love you, but you are made out of atoms
 which it can use for something else.”

Steve Omohundro, an AI researcher, suggests that even something as harmless-sounding as a chess-playing AI, simply ordered to become as good at chess as possible, could be very dangerous, if precautions weren’t taken. It would, for instance, be in its interests to acquire unlimited amounts of matter to build more computers out of, to enable it to think ever more deeply about chess. That may not strike you as inherently dangerous, but if you consider that you are made of matter, and so is the Earth, you may see the potential problem. The fear is that a powerful, “superintelligent” AI could literally end human life, while obeying its innocuous-seeming instructions to the letter.

. . . Shane Legg and Demis Hassabis, the founders of Google’s DeepMind AI firm, are on record saying it’s a serious risk, and DeepMind has collaborated on research into ways to prevent it. Surveys of AI researchers find that a majority of them think that superintelligent AI will arrive in the lifetimes of people alive now, and that there is a strong possibility – roughly a 1 in 5 chance – that it will lead to something “extremely bad (existential catastrophe)”, i.e. human extinction.

I’m not saying that this is inevitable. But I do worry that people discount it utterly, because it sounds weird, and because the people who talk about it are easy to dismiss as weird . . . Just because the people saying something are weird, doesn’t mean they’re wrong.


Between the velvet lies, there's a truth that's hard as steel
The vision never dies, life's a never ending wheel
- R.J.Dio
Neon Knight
Neon Knight
The Castellan

Male Posts : 1423
Join date : 2017-03-05

Back to top Go down

View previous topic View next topic Back to top

Permissions in this forum:
You can reply to topics in this forum