Continuation of the lesswrong drama I posted about recently:

https://www.lesswrong.com/posts/HbkNAyAoa4gCnuzwa/wei-dai-s-shortform?commentId=nMaWdu727wh8ukGms

Did you know that post authors can moderate their own comments section? Someone disagreeing with you too much but getting upvoted? You can ban them from your responding to your post (but not block them entirely???)! And, the cherry on top of this questionable moderation “feature”, guess why it was implemented? Eliezer Yudkowsky was mad about highly upvoted comments responding to his post that he felt didn’t get him or didn’t deserve that, so instead of asking moderators to block on a case-by-case basis (or, acasual God forbid, consider maybe if the communication problem was on his end), he asked for a modification to the lesswrong forums to enable authors to ban people (and delete the offending replies!!!) from their posts! It’s such a bizarre forum moderation choice, but I guess habryka knew who the real leader is and had it implemented.

Eliezer himself is called to weigh in:

It’s indeed the case that I haven’t been attracted back to LW by the moderation options that I hoped might accomplish that. Even dealing with Twitter feels better than dealing with LW comments, where people are putting more effort into more complicated misinterpretations and getting more visibly upvoted in a way that feels worse. The last time I wanted to post something that felt like it belonged on LW, I would have only done that if it’d had Twitter’s options for turning off commenting entirely.

So yes, I suppose that people could go ahead and make this decision without me. I haven’t been using my moderation powers to delete the elaborate-misinterpretation comments because it does not feel like the system is set up to make that seem like a sympathetic decision to the audience, and does waste the effort of the people who perhaps imagine themselves to be dutiful commentators.

Uh, considering his recent twitter post… this sure is something. Also" “it does not feel like the system is set up to make that seem like a sympathetic decision to the audience” no shit sherlock, deleting a highly upvoted reply because it feels like too much effort to respond to is in fact going to make people unsympathetic (at the least).

I’m not sure I even want to give Elon that much? Like the lesswrong website is less annoying than twitter!

Elon is widely known to be a strong engineer, as well as a strong designer

This is just so idiotic I don’t know what made up world Habryka lives in. In between blowing up a launch pad, the numerous insane design and engineering choices of the cybertruck, all the animals slaughtered by neuralink, and the outages and technical problems of twitter, you might be tempted to hope that the idea of Elon Musk as a strong engineer or designer would be firmly relegated to the dustbins of early 2010 where out-of-the-loop people could manage to buy the image of his PR firms. I guess Musk-cultist and lesswrong have more overlap than I realized (I knew there was some, but I didn’t realize it was that common).

Even taking their story at face value:

• It seems like they are hyping up LLM agents operating a bunch of scripts?

• It indicates that their safety measures don’t work

• Anthropic will read your logs, so you don’t have any privacy or confidentiality or security using their LLM, but, they will only find any problems months after the fact (this happened in June according to Anthropic but they didn’t catch it until September),

If it’s a Chinese state actor … why are they using Claude Code? Why not Chinese chatbots like DeepSeek or Qwen? Those chatbots code just about as well as Claude. Anthropic do not address this really obvious question.

• Exactly. There are also a bunch of open source models hackers could use for a marginal (if any) tradeoff in performance, with the benefit that they could run locally, so that their entire effort isn’t dependent on hardware outsider of their control in the hands of someone that will shut them down if they check the logs.

You are not going to get a chatbot to reliably automate a long attack chain.

• I don’t actually find it that implausible someone managed to direct a bunch of scripts with an LLM? It won’t be reliable, but if you can do a much greater volume of attacks maybe that makes up for the unreliability?

But yeah, the whole thing might be BS or at least bad exaggeration from Anthropic, they don’t really precisely list what their sources and evidence are vs. what is inference (guesses) from that evidence. For instance, if a hacker tried to setup hacking LLM bots, and they mostly failed and wasted API calls and hallucinated a bunch of shit, if Anthropic just read the logs from their end and didn’t do the legwork contacting people who had allegedly been hacked, they might "mistakenly’ (a mistake that just so happens to hype up their product) think the logs represent successful hacks.

Another ironic point… Lesswronger’s actually do care about ML interpretability (to the extent they care about real ML at all; and as a solution to making their God AI serve their whims not for anything practical). A lack of interpretability is a major problem (like irl problem, not just scifi skynet problem) in ML, you can models with racism or other bias buried in them and not be able to tell except by manually experimenting with your model with data from outside the training set. But Sam Altman has turned it from a problem into a humble brag intended to imply their LLM is so powerful and mysterious and bordering on AGI.

A lesswronger wrote an blog post about avoiding being overly deferential, using Eliezer as an example of someone that gets overly deferred to. Of course, they can’t resist glazing him, even in the context of an blog post on not being too deferential:

Yudkowsky, being the best strategic thinker on the topic of existential risk from AGI

Another lesswronger pushes back on that and is highly upvoted (even among the doomers that think Eliezer is a genius, most of them still think he screwed up in inadvertently helping LLM companies get to where they are): https://www.lesswrong.com/posts/jzy5qqRuqA9iY7Jxu/the-problem-of-graceful-deference-1?commentId=MSAkbpgWLsXAiRN6w

The OP gets mad because this is off topic from what they wanted to talk about (they still don’t acknowledge the irony).

A few days later they write an entire post, ostensibly about communication norms, but actually aimed at slamming the person that went off topic: https://www.lesswrong.com/posts/uJ89ffXrKfDyuHBzg/the-charge-of-the-hobby-horse

And of course the person they are slamming comes back in for another round of drama: https://www.lesswrong.com/posts/uJ89ffXrKfDyuHBzg/the-charge-of-the-hobby-horse?commentId=s4GPm9tNmG6AvAAjo

No big point to this, just a microcosm of lesswrongers being blind to irony, sucking up to Eliezer, and using long winded posts about meta-norms and communication as a means of fighting out their petty forum drama. (At least us sneerclubers are direct and come out and say what we mean on the rare occasions we have beef among ourselves.)

Thanks for the information. I won’t speculate further.

Thanks!

So it wasn’t even their random hot takes, it was reporting someone? (My guess would be reporting froztbyte’s criticism, which I agree have been valid if a bit harsh in tone)

Some legitimate academic papers and essays have served as fuel for the AI hype and less legitimate follow-up research, but the clearest examples that comes to mind would be either “The Bitter Lesson” essay or one of the “scaling law” papers (I guess Chinchilla scaling in particular?), not “Attention is All You Need”. (Hyperscaling LLMs and the bubble fueling it is motivated by the idea that they can just throw more and more training data at bigger and bigger model). And I wouldn’t blame the author(s) for that alone.

BlueMonday has had a tendency to go off with a half-assed understanding of actual facts and details. Each individual instance wasn’t ban worthy, but collectively I can see why it merited a temp ban. (I hope/assume it’s not a permanent ban, is there a way to see?)

I was wondering why Eliezer picked chess of all things in his latest “parable”. Even among the lesswrong community, chess playing as a useful analogy for general intelligence has been picked apart. But seeing that this is recent half-assed lesswrong research, that would explain the renewed interest in it.

Yud: “Woe is me, a child who was lied to!”

He really can’t let down that one go, it keeps coming up. It was at least vaguely relevant to a Harry Potter self-insert, but his frustrated gifted child vibes keep leaking into other weird places. (Like Project Lawful, among it’s many digressions, had an aside about how dath ilan raises it’s children to avoid this. It almost made me sympathetic towards the child-abusing devil worshipers who had to put up with these asides to get to the main character’s chemistry and math lectures.)

Of course this a meandering plug to his book!

Yup, now that he has a book out he’s going to keep referencing back to it and it’s being added to the canon that must be read before anyone is allowed to dare disagree with him. (At least the sequences were free and all online)

Is that… an incel shape-rotator reference?

I think shape-rotator has generally permeated the rationalist lingo for a certain kind of math aptitude, I wasn’t aware the term had ties to the incel community. (But it wouldn’t surprise me that much.)

I couldn’t even make it through this one, he just kept repeating himself with the most absurd parody strawman he could manage.

This isn’t the only obnoxiously heavy handed “parable” he’s written recently: https://www.lesswrong.com/posts/dHLdf8SB8oW5L27gg/on-fleshling-safety-a-debate-by-klurl-and-trapaucius

Even the lesswronger’s are kind of questioning the point:

https://www.lesswrong.com/posts/dHLdf8SB8oW5L27gg/on-fleshling-safety-a-debate-by-klurl-and-trapaucius?commentId=BhePfCvbGaNauDqfz

I enjoyed this, but don’t think there are many people left who can be convinced by Ayn-Rand length explanatory dialogues in a science-fiction guise who aren’t already on board with the argument.

A dialogue that references Stanislaw Lem’s Cyberiad, no less. But honestly Lem was a lot more terse and concise in making his points. I agree this is probably not very relevant to any discourse at this point (especially here on LW, where everyone would be familiar with the arguments anyway).

And: https://www.lesswrong.com/posts/3q8uu2k6AfaLAupvL/the-tale-of-the-top-tier-intellect?commentId=oHdfZkiKKffqSbTya

Reading this felt like watching someone kick a dead horse for 30 straight minutes, except at the 21st minute the guy forgets for a second that he needs to kick the horse, turns to the camera and makes a couple really good jokes. (The bit where they try and fail to change the topic reminded me of the “who reads this stuff” bit in HPMOR, one of the finest bits you ever wrote in my opinion.) Then the guy remembers himself, resumes kicking the horse and it continues in that manner until the end.

Who does he think he’s convincing? Numerous skeptical lesswrong posts have described why general intelligence is not like chess-playing and world-conquering/optimizing is not like a chess game. Even among his core audience this parable isn’t convincing. But instead he’s stuck on repeating poor analogies (and getting details wrong about the thing he is using for analogies, he messed up some details about chess playing!).

Eh, cuck is kind of the right-winger’s word, it’s tied to their inceldom and their mix of moral-panic and fetishization of minorities’ sexualities.

Remember when a bunch of people poured their life savings into GameStop and started a financial doomsday cult once they lost everything? That will happen again if OpenAI goes public.

I’ve seen redditors on /r/singularity planning on buying OpenAI stock if it goes public. And judging by Tesla, cultists buying meme stock can keep up their fanaticism through quite a lot.

It seems like a complicated but repeatable formula: Start a non-profit dedicated to some technology, leverage the charity status for influence and tax avoidance and PR and recruiting true believers in the initial stages, and then make a bunch of financial deals conditional on your non-profit changing to for profit, then claim you need to change to for-profit or your organization will collapse!

Although I’m not sure how repeatable it is without the “too big to fail” threat of loss of business to state AGs. OTOH, states often bend the rules to gain (or even just avoid losing) embarrassingly few jobs, so IDK.

i’ve listened to his podcast, i’ve read his articles, he is pretty up front about what his day job is and that he is a disappointed fanboy for tech. the dots are 1/1000th of an inch apart.

For comparison I’ve only read Ed’s articles, not listened to his podcasts, and I was unaware of his PR business. This doesn’t make me think his criticisms are wrong, but it does make me concerned he’s overlooked critiquing and analyzing some aspects of the GenAI industry because of these connections to those aspects.

This week’s southpark makes fun of prediction markets! Hanson and the rationalists can be proud their idea has gone mainstream enough to be made fun of. The episode actually does a good job highlighting some of the issues with the whole concept: the twisted incentives and insider trading and the way it fails to actually create good predictions (as opposed to just getting vibes and degenerate gambling).

and the person who made up the “math pets” allegation claimed no such source

I was about to point out that I think this is the second time he claimed math pets had absolutely no basis in reality (and someone countered with a source that forced him to) but I double checked the posting date and this is the example I was already thinking of. Also, we have supporting sources that didn’t say as much directly but implied it heavily: https://www.reddit.com/r/SneerClub/comments/42iv09/a_yudkowsky_blast_from_the_past_his_okcupid/ or like, the entire first two thirds of the plot of Planecrash!

So if I understood NVIDIA’s “strategy” right, their usage of companies like Coreweave is drawing in money from other investors and private equity? Does this mean, that unlike many of the other companies in the current bubble, they aren’t going to lose money on net, because they are actually luring in investment from other sources in companies like Coreweave (which is used to buy GPU and thus goes to them), whileleaving the debt/obligations in the hands of companies like Coreweave? If I’m following right this is still a long term losing strategy (assuming some form of AI bubble pop or deflation we are all at least reasonably sure of), but the expected result for NVIDIA is more of a massive drop in revenue as opposed to a total collapse of their company under a mountain of debt?