I like how Zitron does a good job of distinguishing firm overall predictions from specific scenarios (his chaos bets) which are plausible but far from certain. AI 2027 specifically conflated and confused those things in a way that gave it’s proponents more rhetorical room to hide and dodge.
I like how he doesn’t even bother debunking it point by point, he just slams the very premise of it and moves on.
system memory
System memory is just the marketing label for “having an LLM summarize a bunch of old conversations and shoving it into a hidden prompt”. I agree that using that term is sneer-worthy.
I have three more examples of sapient marine mammals!
I was thinking this also, like it’s the perfect parody of several lesswrong and EA memes: overly concerned with animal suffering/sapience, overly concerned with IQ stats, openly admitting to no expertise or even relevant domain knowledge but driven to pontificate anyway, and inspired by existing science fiction… I think the last one explains it and it isn’t a parody. As cinnasverses points out, Cetacean intelligence shows up occasionally in sci-fi. to add to the examples… sapient whales warning the team of an impending solar flare in Stargate Atlantis via echolocation induced hallucinations, the dolphins in hitchhiker’s guide to the galaxy, and the whales showing up to help in one book of Animorphs.
I was trying to figure out why he hadn’t turned this into an opportunity to lecture (or write a mini-fanfic) about giving more attack surface to the AGI to manipulate you… I was stumped until I saw your comment. I think that is it, expressing his childhood distrust of authority trumps lecturing us on the AI-God’s manipulations.
I have context that makes this even more cringe! “Lawfulness concerns” refers to like, Dungeons and Dragons lawfulness. Specifically the concept of lawfulness developed in the Pathfinder fanfiction we’ve previously discussed (the one with deliberately bad BDSM and eugenics). Like a proper Lawful Good Paladin of Iomedae wouldn’t put you in a position where you had to trust they hadn’t rigged the background prompt if you went to them for spiritual counseling. (Although a Lawful Evil cleric of Asmodeus totally would rig the prompt… Lawfulness as a measuring stick of ethics/morality is a terrible idea even accepting the premise of using Pathfinder fanfic to develop your sense of ethics.)
you can’t have an early version that you’ll lie about being a “big step towards General Quantum Computing” or whatever
So you might think that… but I recall some years ago an analog computer was labeled as quantum annealing or something like that… oh wait, found the wikipedia article: https://en.wikipedia.org/wiki/Quantum_annealing and https://en.wikipedia.org/wiki/D-Wave_Systems . So it sounds to a naive listener like the same sort of thing as the quantum computers that are supposed to break cryptography and even less plausible things, but actually it can only do one very specific algorithm.
I bet you could squeeze the “quantum” label onto a variety of analog computers well short of general quantum computing and have it technically not be fraud and still fool lots of idiot VCs!
It’s a nice master post that gets all his responses and many useful articles linked into one place. It’s all familiar if you’ve kept up with techtakes and Zitron’s other posts and pivot-to-ai, but I found a few articles I had previously missed reading.
Related trend to all the but achskhually’s AI booster’s like to throw out. Has everyone else noticed the trend where someone makes a claim of a rumor they heard about an LLM making a genuine discovery in some science, except it’s always repeated second hand so you can’t really evaluate it, and in the rare cases they do have a link to the source, it’s always much less impressive than they made it sound at first…
Even for the people that do get email notifications of Zitron’s excellent content (like myself), I appreciate having a place here to discuss it.
Apparently Eliezer is actually against throwing around P(doom) numbers: https://www.lesswrong.com/posts/4mBaixwf4k8jk7fG4/yudkowsky-on-don-t-use-p-doom ?
The objections to using P(doom) are relatively reasonable by lesswrong standards… but this is in fact once again all Eliezer’s fault. He started a community centered around 1) putting overconfident probability “estimates” on subjective uncertain things 2) need to make a friendly AI-God, he really shouldn’t be surprised that people combine the two. Also, he has regularly expressed his certainty that we are all going to die to Skynet in terms of ridiculously overconfident probabilities, he shouldn’t be surprised that other people followed suit.
Guns don’t kill people, people kill people.
I missed that it’s also explicitly meant as rationalist esoterica.
It turns in that direction about 20ish pages in… and spends hundreds of pages on it, greatly inflating the length from what could be a much more readable length. It then gets back to actual plot events after that.
I hadn’t heard of MAPLE before, is it tied to lesswrong? From the focus on AI it’s at least adjacent to it… so I’ll add that to the list of cults lesswrong is responsible for. So all in all, we’ve got the Zizians, Leverage Research, and now Maple for proper cults, and stuff like Dragon Army and Michael Vassar’s groupies for “high demand” groups. It really is a cult incubator.
I actually think “Project Lawful” started as Eliezer having fun with glowfic (he has a few other attempts at glowfics that aren’t nearly as wordy… one of them actually almost kind of pokes fun at himself and lesswrong), and then as it took off and the plot took the direction of “his author insert gives lectures to an audience of adoring slaves” he realized he could use it as an opportunity to squeeze out all the Sequence content he hadn’t bothered writing up in the past decade^ . And that’s why his next attempt at a HPMOR-level masterpiece is an awkward to read rp featuring tons of adult content in a DnD spinoff, and not more fanfiction suitable for optimal reception to the masses.
^(I think Eliezer’s writing output dropped a lot in the 2010s compared to when he was writing the sequences and the stuff he has written over the past decade is a lot worse. Like the sequences are all in bite-size chunks, and readable in chunks in sequence, and often rephrase legitimate science in a popular way, and have a transhumanist optimism to them. Whereas his recent writings are tiny little hot takes on twitter and long, winding, rants about why we are all doomed on lesswrong.)
Weird rp wouldn’t be sneer worthy on it’s own (although it would still be at least a little cringe), it’s contributing factors like…
the constant IQ fetishism (Int is superior to Charisma but tied with Wis and obviously a true IQ score would be both Int and Wis)
the fact that Eliezer cites it like serious academic writing (he’s literally mentioned it to Yann LeCunn in twitter arguments)
the fact that in-character lectures are the only place Eliezer has written up many of his decision theory takes he developed after the sequences (afaik, maybe he has some obscure content that never made it to lesswrong)
the fact that Eliezer think it’s another HPMOR-level masterpiece (despite how wordy it is, HPMOR is much more readable, even authors and fans of glowfic usually acknowledge the format can be awkward to read and most glowfics require huge amounts of context to follow)
the fact that the story doubles down on the HPMOR flaw of confusion of which characters are supposed to be author mouthpieces (putting your polemics into the mouths of character’s working for literal Hell… is certainly an authorial choice)
and the continued worldbuilding development of dath ilan, the rationalist utopia built on eugenics and censorship of all history (even the Hell state was impressed!)
…At least lintamande has the commonsense understanding of why you avoid actively linking your bdsm dnd roleplay to your irl name and work.
And it shouldn’t be news to people that KP supports eugenics given her defense of Scott Alexander or comments about super babies, but possibly it is and headliner of weird roleplay will draw attention to it.
To be fair to DnD, it is actually more sophisticated than the IQ fetishists, it has 3 stats for mental traits instead of 1!
It’s always the people you most expect.
It’s pretty screwed up that humble bragging about putting their own mother out of a job is a useful opening to selling a scam-service. At least the people that buy into it will get what they have coming?
Lots of woo and mysticism already has a veneer of stolen Quantum terminology. It’s too far from respectable to get the quasi-expert endorsement or easy VC money that LLM hype has gotten, but quantum hucksters fusing quantum computing nonsense with quantum mysticism can probably still con lots of people out of their money.