Talk:Instrumental convergence

Merge proposal
I think that Paperclip maximizer should be folded into this article, as the other article is stub-length, and (AFAICT) the paperclip maximizer is a thought experiment intended to illustrate the idea of instrumental convergence. Spectra239 (talk) 10:18, 30 January 2016 (UTC)
 * Sounds good to me. Rolf H Nelson (talk) 21:39, 30 January 2016 (UTC)
 * I support. Gap9551 (talk) 02:13, 31 January 2016 (UTC)
 * Support - I originally created the Paperclip maximizer page, but agree that it fits better with the context provided here. Paperclip Maximiser (talk) 08:25, 6 February 2016 (UTC)
 * Done. Rolf H Nelson (talk) 04:05, 26 February 2016 (UTC)

Counteracting
I am missing the section on how to protect humanity from the paperclip problem. Zezen (talk) 09:01, 4 November 2017 (UTC)


 * Well, if you find it, you'll probably get statues put up to you for the next trillion years as the one who delivered the cosmic endowment to mankind. So good luck! 103.20.170.81 (talk) 10:01, 21 November 2017 (UTC)
 * @Zezen Added a pointer in a new 'impacts' section. Rolf H Nelson (talk) 19:06, 25 November 2017 (UTC)

Convergence kind of a good thing?
I deleted the second paragraph:

"Instrumental convergence suggests that an intelligent agent with apparently harmless goals can act in surprisingly harmful ways. For example, a computer with the sole goal of solving the Riemann hypothesis could attempt to turn the entire Earth into computronium in an effort to increase its computing power so that it can succeed in its calculations."

IMHO, the paragraph confuses association with causation. The supposed tendency of strong AIs for instrumental convergence does not make them more harmful. If anything, it allows us to better predict their behavior. Should they diverge in every which way depending on goals, they would be far more dangerous. — Preceding unsigned comment added by 176.11.124.251 (talk) 14:45, 2 February 2018 (UTC)


 * Instrumental convergence, the hypothesized tendency for intelligent agents to want to seek various generically-useful instrumental capabilities, is currently framed in the literature as a bad thing. If you have a strong source indicating the opposite, that should be added. Rolf H Nelson (talk) 20:32, 3 February 2018 (UTC)


 * I think the way YouTube's recommender system drives viewers to more and more extreme content is a good example of a rudimentary paperclip maximizer. The aim is to maximize money by showing ads with content people want to see, Google's "don't be evil" is not incorporated into the programming and it is difficult to do so. Dmcq (talk) 11:11, 12 September 2018 (UTC)

Paperclip maximizer author
The article mentions Nick Bostrom as the author of the idea of the paperclip maximizer, however Eliezer Yudkowsky seems to consider himself the author of that idea (tweet). The article does not seem to have a citation for the source. A citation would not solve the problem unless one of the two has cited the other one at some point, to establish the direction. What do we know about the origin of this idea?

trylks (talk) 23:19, 16 November 2021 (UTC)


 * Eliezer Yudkowsky said a bit more on Sam Harris's podcast (transcript): "As far as I know, it’s me. It could still be Bostrom. I asked somebody, “Do you remember who it was?” and they searched through the archives of the mailing list where this idea plausibly originated and if it originated there, then I was the first one to say “paperclips.”"


 * I haven't come across any quotes by Nick Bostrom on this topic. --Steve (talk) 02:08, 18 November 2021 (UTC)

some more tweets by eliezer yudkowsky claiming to have originated the meme. this matches my memory; unfortunately, i do not have a copy of whatever mailing list it might have been to verify the claim. 135.180.194.177 (talk) 22:47, 9 November 2022 (UTC)
 * https://twitter.com/ESYudkowsky/status/1578459400389345280
 * https://twitter.com/ESYudkowsky/status/1582862689218404354


 * Tweets by Eliezer and youtube videos in which Eliezer says that he is the originator of the paperclip maximizer idea are not WP:RS, WP:NPOV references.--FeralOink (talk) 08:34, 5 June 2023 (UTC)