The LessWrong/“rationality” crowd gwern is part of like to pat themselves on the back for identifying the trend and resulting harms from super intelligent AI going back decades.
And yet, the main tangible contribution emerging from this early recognition (and obsession) was to inspire a lot of brilliant CS/math types and billionaires like Peter Thiel or Jaan Tallinn to build and fund companies like OpenAI and Anthropic which went on to literally create the very thing they were convinced will destroy the world, leading to an ever escalating arms race to “AGI”. Also lots of crypto and a murky web of non-profits doing unclear stuff.
The proximity to money in the Silicon Valley/VC world seems to limit the imagination to founding yet another new tech startup that really, definitely won’t compromise their values and will be different this time.
The LessWrong/"rationality" crowd aren't compromising their values; they just have bad values that lead to negative outcomes, masked by high-minded rhetoric that they themselves only half believe.
It's a popular movement that has wound its way through out society, and the fruits are clear: surveillance capitalism, eugenics, fraud. It's honestly quite impressive how consistently you can look at [recent negative trend] and find someone at the bottom calling themselves a rationalist and claiming the same set of influences.
The fundamental tenet of that set isn't rationalism but the belief that they are superior to the exploitable masses. That does not lead anywhere good.
Even setting that crowd aside, almost every time I see somebody puffed up about how "rational" they are it turns out to involve a vigorous avoidance of the values they're working from.
I know somebody who says that doing debate in high school and college was terrible for him because it trained him to always argue his point but never to question it. For example, at the time he thought of himself as supremely rational even as he'd make a strong case for why he shouldn't quit smoking. It was only much later that he could admit that was his addiction and his ego talking.
Oh wow, that's amazing. The need to be verbally right and perfect no matter the cost is such a curse, so blinding. I hope they all get free of it someday.
And yet, the main tangible contribution emerging from this early recognition (and obsession) was to inspire a lot of brilliant CS/math types and billionaires like Peter Thiel or Jaan Tallinn to build and fund companies like OpenAI and Anthropic which went on to literally create the very thing they were convinced will destroy the world, leading to an ever escalating arms race to “AGI”. Also lots of crypto and a murky web of non-profits doing unclear stuff.
The proximity to money in the Silicon Valley/VC world seems to limit the imagination to founding yet another new tech startup that really, definitely won’t compromise their values and will be different this time.