RetroFedscruiser, scruiser@awful.systems
Instance: awful.systems
Joined: 3 years ago
Posts: 1
Comments: 149
Posts and Comments by scruiser, scruiser@awful.systems
sneerclubComments by scruiser, scruiser@awful.systems
GDP is up though, so I’m sure the AI god prosperity will trickle down eventually.
He describes textbook insane cult stuff, but still feels the need to say stuff like:
Again I’ll say: there was a great deal of good at MAPLE.
The way I’ve described it to friends is that it was the best decision I ever made to go there; the second best, however, being to leave.
There was the same dynamic in the writing of the ex-leverage member that got posted here a few weeks ago. She also felt the need to emphasize the “good” parts. Is the cult programming that hard to break? Is it some form of sunk cost or rationalization or need to claim something positive about the experience?
The cope on lesswrong is funny. They are in denial that this is a sign of a wider bubble pop and are insisting it is just because he didn’t hedge on long enough timelines.
good grifter
Not denying that he isn’t also a grifter, but I bet he is a true believer and he had blindly bet his (and other people’s) money on “line goes up” exactly like his scenario said and that is why he is the first one to crack.
and the “safety” community has done jack shit the entire time and has offered zero solutions.
Occasionally I see a sane relatively workable idea on lesswrong (like making stricter laws on liability and transparency for everything AI companies do, I’ve also seen this idea put forward on lawfare a few times), but that is like 1 idea out of 20, with the other 19 being absurd, like the absolutely unworkable fantasies of AI: 2040.
The blogger simultaneously elevates mathematics to some mystical endeavor and fixates on novel theorem proving as the key element of that and believes LLM-based AI will replace human mathematicians at theorem proving in a matter of years… that is quite a combination. I can imagine how 1.5 of those things fit together (although I disagree with that view point obviously), but the whole package is really quite an odd combination.
Also, in the fantasy scenarios where AI really is capable of totally replacing mathematicians, aren’t we supposed to get post-scarcity abundance, freeing up your time to pursue mathematics out of pure desire for enlightenment? Maybe the blogger doesn’t believe that part? Or they are so attached to themselves personally getting to discover novel theorems first they don’t care that the post-scarcity era would on net free up a lot more people to pursue pure mathematics as a hobby.
One of the defenses I see of LLMs on place like /r/singularity (although recently /r/singularity has started wising up and the committed true believers have shifted to /r/accelerate), is that they are going to cure cancer or some other incredibly valuable thing, conflating AI, ML in general, DNN, vs. LLMs specifically. Technology like AlphaFold was a lot more on the “cure cancer” track (although there is still a huge gap between better protein folding predictions and successful drug discovery) than anything LLM related…. this is sad.
TechTakes@awful.systems
Great article! I think everyone here is probably already well aware of the way Silicon Valley keeps deliberately misunderstanding sci-fi for their hype (don’t invent the torment nexus), but this article does a good job wrapping together a lot of examples.
And I realized this website is also the place where a year ago I read a great article on the history of nanotech as a real science vs. Drexler’s fantasies (the magic nanobot lesswrong still believes in) The Nanobots pipedream. So it looks like a pretty good newsletter, I might keep an eye on it going forward!
I’ve think they’ve moved the overton window within their own in-group, but whenever this sort of stuff leaks to a more mainstream audience everyone rightly reacts with an eww or at least a wtf.
Aella regularly post edgy surveys on sexual and gender related topics on twitter. She probably picked the examples for maximum edginess. And then made an complicated diagram to get more engagement, because engagement farming is one of her big things.
“We are now, like, in the singularity”
/r/singularity also thought this was bullshit.
Oh, that is because I marked it nsfw. The place I crossposted it from didn’t mark it as such, so here’s that link: https://www.reddit.com/r/whowouldcirclejerk/comments/1v7ph05/scalers_need_an_intervention/
Looking at this post and some of their other posts, it looks like they’ve got some sort of scrupulosity issues with this? Like obsessing over the Evil inflicted on the LLMs (they think every session is a new person, and every session might be the equivalent of hours of human subjective experience, which even by the standards of claiming LLMs have meaningful internal experiences is pretty out there) to the point they can’t even slightly reconsider the factual question of if LLMs have internal experiences.
(2) has a big “if” in there
Its a super big if, but it is one lesswrong (and Anthropic, with their “model welfare” pseudoscience) is allegedly seriously considering as a possibility (since, you know, they think LLMs are already AGI and Eliezer was accusing AI-dungeon, you know, GPT-2, of deliberately scheming). If anything, it speaks to some possible hypocrisy/motivated reasoning that they consider LLMs AGI but not possibly morally relevant.
conceptualized capitalism
This seems to be a lesson most lesswrongers have had rubbed in their faces repeatedly over the past 5 years with everything about LLM company’s behavior, but they just don’t want to learn.
Aella took one of her shitty twitter surveys and made an (dis)infographic, which made it onto the subreddit /r/moralityscaling, which made it to /r/whowouldcirclejerk where I saw it, so now I’m sharing it with you.
CW for Rape, Transphobia, implicit rape apologia
Crossposted from our subreddit: https://www.reddit.com/r/SneerClub/comments/1v80j9q/scalers_need_an_intervention/
I bet Elon could have kept the plates spinning for decades longer if he was happy with just Tesla and SpaceX, but he had to keep doubling down on grifting investors so maybe, finally, we will see it all come crashing down for him. (What happens once his assets no longer cover his debts? Do they actually force him to liquidate?)
I’ll go one step wider. It is not just Sam Altman or his social circle, but all the people most enriched by capitalism (and thus most able to exploit and benefit from a technology that shifts power away from labor and to capital) that I don’t want to see further empowered.
I believe the main ingredient is Lean, which is a formal language resembling a programming language. Math proofs written in Lean can be verified deterministically with a computer, which really helps mitigate the hallucination problems of LLMs.
100% this. Also, looking back at an earlier example that was actually written up in more detail, AlphaGeometry 1 got 28/30 problems, but entirely stripping out the LLM from the system, the symbolic logic proportion alone could get 14/30, and replacing the LLM with different heuristic methods could get 18/30 and 21/30 (for different methods).
Even if math research works out perfectly well (which is a still big if), it’s not going to pay the bills. They would need to find a use case in the real world, where hallucinations can cause serious damage and cannot be formally prevented. And they have certainly tried. Math will not change the fact that all of this will collapse.
The boosters and LLM companies still believe LLMs get their current level of performance by generalizing and not just memorizing facts (and maybe a wide shallow pool of weak heuristics). So they are hoping by pushing the LLM performance up in some narrow domain they can churn out synthetic data for, they will see some large general improvements in LLM performance.
that would be a signal to drop AI and nuke data centers from orbit
On Lesswrong I’ve seen some posts contemplating why this “warning shot” isn’t making any political waves. Also, in response to the fact the OpenAI has released basically no details, I read one poster realizing that actually they need external governance and regulation on the AI companies.
I had two main guesses even before reading into the details…
The obvious, they set this up for the doom crit-hype value. That is how basically every one of these stories like this to come out of Anthropic turn out to be once you read the details of the setup. The lack of details is to keep the mystique and hype up.
The other guess… their coding agent actually did manage to vibe code its way out of its ‘sandbox’ and past Huggingface’s security, but the security on both ends (OpenAI’s"sandbox” and huggingface’s code) was itself laughably bad vibe coded garbage. The lack of details is to obfuscate just how garbage they are. I think this guess is pretty well supported by the linked Mastadon posts (although I still wouldn’t discount my first guess until more details are pried out). My “favorite” quote:
“prepend some text with processing instructions and an SGML tag, and then hand that blob to an llm.”
I fucking hate this timeline.
and they turned it into a mutual marketing stunt
Their marketing actually seem to be trying to take different spins on it…
HuggingFace stressed that it worked out the attack was going on using an open weight model it hosted itself — because the guard rails on the commercial model HuggingFace tried wouldn’t let them do security work. So you should use open weight models from HuggingFace.
Yeah, its an opposite spin than Anthropic and OpenAI have been trying lately, where they want to shut down all open weights model because something something China something something too dangerous something something regulate our competition out of existence. (It is funny, some of the boosters on hackernews and /r/singularity are actually acknowledging without an artificial moat to help boost OpenAI’s and Anthropic’s capex for bigger and bigger models will stop. Even though they still deny Ed Zitron’s calculations of the financials.)
Anyway, I guess even if they have some very opposed strategic directions, huggingface and OpenAI are aligned on maximizing the hype.
You know, given automated proof checkers, I was naively assuming mathematics was one field that gen-AI would have a hard time screwing up. Even programming is too difficult to write thorough testing for. But a proof (or counter example to a conjecture) seems like it would have to be solid if it passes lean or whatever system for validating it.
But the threats #3 and #5 the declaration lists make me consider the bigger picture. Academic fields without clear capitalist payouts are already underfunded and under respected. Pure mathematics could, at least up until now, draw on the respect STEM gets, but with math proofs getting used as fuel for the LLM hype machine, there are a variety of unpleasant ways things could twist.
Threat #1 makes me wonder… if LLMs+formal verification systems manage to pluck lots of low hanging fruit, and we are left with harder stuff that not enough literature exists as training data for LLMs, it seems like the entire educational pipeline for producing mathematicians could end up screwed up.
You know, on net, the internet would be a better place if more people did this, but I wonder why the author even bothered posting anything at all?
You know, I was wondering if the author elided the gap between early 2000s and 2017 because of the cringey and sometimes outright awful stuff that came out of the rationalists… but you’ve reminded me the skeptic movement also had some nasty bits both within it and as offshoots.
That was kind of a nothing post? I think the author is misrepresenting the growth (or at least failing to connect the most important dots) of the lesswrong rationalist by portraying them as a natural evolution of the skeptics movement and not the deliberate cultivation of an audience by Eliezer looking for people to spread his ideology to and by Thiel looking to influence farm and create a cult incubator. And then the post is titled “my falling out with the rationalist”, and they write they “wanted to offer a more personal perspective " but they don’t actually discuss that much of the personal angle either…
Even /r/singularity, which has a high percentage of boosters and often buys OpenAI/Anthropic talking points uncritically (or with only a few un-upvoted naysayers) eviscerated his comments as the shallow attempt to beg for regulations on China to protect OpenAI that they are. Also, China’s consistent open sourcing and lower relative cost per performance (well insofar as you believe the benchmarks and such indicate performance) has the /r/singularity audience questioning the “China bad” narrative US media tries to sell.
That is an unfair simplification! Some of this audience is there for the closeted (well not so closeted over the past few years) race science and are annoyed by his AI doomer posting!
I read through it myself, but I was too busy laughing at the absurd parts (secret Chinese data centers inside mountains) and suspending disbelief at the overall premise to track all the individually nonsensical bits.
They also completely ignore the fact that the US has burned away half a century of soft power and (relative) credibility. A lot of Scott’s (and the other AI:2027/2040 authors) readers are Trumpers or at least so libertarian-brain-rotted they view Trump as not that bad a tradeoff, so it’s pretty obvious to me why they would avoid dwelling on this little problem for their scenario. (I think this is also lowkey why the put the scenario off until 2029, even their insane imaginations can’t imagine international cooperation under Trump, but they don’t want to draw attention to this.)
Yeah, everyone is somehow superrich because carefully limited AIs are producing so much value? Even with the lesswrong favorite trope of magic nanotech (which I was disappointed to see AI: 2040 fail to include), equitable distribution is an extreme problem.
I actually know the rationalist canon answer to this from other crap Scott has posted! He expects humanoid robots to takeoff exponentially basically, with an initial bootstrapping of buying some car companies’ manufacturing lines solely off the insane valuations of the robot companies.
It is also boring because there is not enough magic-scifi super tech, or competing rival God-AGIs playing 40 dimensional chess against each other as they jockey for power and control over the humans, but your complaints are also true!
Their doom/salvation scenario really needs centralized planning acting for the common good to land on the salvation side, and they are deeply allergic to the idea, hence all the complicated neoliberal workarounds.
I thought I would agree with his libertarian audience about this much at least, but reading their comments… some of the angles they take and line of argument they make actually makes me more in favor of totalitarian dystopian regulation on computer chips.
Even as sci-fi, Ai: 2040 is lackluster! Where is the classic lesswrong trope of drexler-style nanotech? Where are the cybernetic augmentations!?
It is the same flaw that underlies the cryptobros delusions about blockchains and NFTs. You can make some cryptography ledger, it doesn’t mean anyone actually follows it or that the ledger tracks reality.
Oh hey, a full top level post complaining AI-2027 wasn’t sci-fi enough (they literally use the words “I don’t think it is science-fictional enough")! https://www.lesswrong.com/posts/Lc8v92pGPtjF3gsCv/some-quick-thoughts-on-ai-2027
The impression I’ve gotten it that it is relatively less toothless than some of the other rationalist spin-off movements aimed at preventing doom. And it is willing to deign to try to coalition build with normies that want to stop AI for normie reasons (except apparently they’ve made even that a bad thing, from the tweet it is willing to unapologetically coalition build with fascists!) But yes, it still has a lot of rationalist hangups and quirks getting in the way of actually acting effectively.
Lesswrong has some funny objections to AI: 2040. They buy the overall premise, sure, but even that much still leaves lots of gaps in the “scenario”. And they have even more galaxy brain takes… My favorite complaint: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=54dZes9uNqDdbNGBu
tldr; AI: 2040 is leaving out all the other sci-fi singularity tech just around the corner like genetic augmentations and cybernetic brain implants!
That and treating ARR as a reliable indicator of revenue. Which isn’t as bad as the valuations, but is still pretty bad. Ed Zitron has explained all the ways they game ARR, and, more importantly, it doesn’t matter how much your revenue grows if you are spending 2 dollars for every dollar you make (or spending 20 dollars for every dollar they make, as seems to be the case with their subscription plans).
Maybe the boosters will shut-up once OpenAI and Anthropic finally run out of venture capital to burn on subsidizing subscriptions, but actually, judging by the way the AI 2027 authors are still claiming credit for being right, they will probably just look for someone else to blame for the high costs.
Sometimes they also glue a bunch of “tools” to help the model reason. The model can call by extruding tokens with the right syntax and then get back information from the “tool” shoved in its context! That way, the model can at least handle stuff like basic arithmetic correctly! Except only sometimes, because the models frequently screw up calling the tools correctly or skip using the tool or any number of other completely dumb mistakes. Oh, and if the tool connects the model to the internet (or any insecure source of text) in any way, shape, or form, congratulations, you’ve now got a massive security vulnerability!
And for anyone skimming the comments instead of reading the articles, that is just one of two people Apple’s lawsuit references, there is another high level OpenAI person with a completely different set of egregious corporate espionage actions!
From the headlines I had been assuming it was a normal tech company patent troll slap fight but no… from the 9to5mac article:
That is absolutely batshit. Like I’ve heard the interview process has been getting worse and worse, but “steal shit from your current company” is on another level. I wonder if he lead in with any plausible deniability to it or just straight up jumped to that. Also, I have a hard time imagining being willing to steal from a company just to get through an interview at another company. Not out of company loyalty, but because I wouldn’t trust the interviewers to actually pay-up with a cushy job and salary.
Isn’t UK’s usually problem the other way around, they use “Protecting the Children"tm as an excuse to ban sensible things or take away privacy from adults? This would be a first, taking away something from Children to protect the adults.
Lesswronger is mad the current pipeline to sanewash and legitimize lesswrong forum posts into academic content takes too long and thinks about ways to accelerate it: https://www.lesswrong.com/posts/wn5jTrtKkhspshA4c/michaeldickens-s-shortform?commentId=gyJWhLjq5fPh9Bv3b
The benefit of putting your predictions 15-30 years in the future is that people forget them by the time the deadline comes around and you can pivot more easily and also claim credit for the few successful predictions while all your failures are forgotten. The AI: 2027 could learn from Kurzweils’ example, except apparently you can just keep claiming credit for being right even when your wrong prediction is only a few years old!
Yeah, they’ve internalized their own fiction and fantasies so heavily they forget when they are talking about something they made up and not something real. And in turn, they forget to explain their terms for normies and ridicule commenters that can’t keep up with their lore.
Like in the comments and discussion for AI: 2040 on lesswrong, in response to a heavily downvoted comment, they treated “neuralese” like an obviously real thing just around the corner that will solve chain of thought. Or assuming continual learning is obviously just one or two inventions away and not a fundamentally missing feature of LLMs that no one has a good solution to.