You’re argument seems like Suresh’s to me, in that making working software is just laughably easy for a competent engineer.
This begs the question why noone is exploiting this market gap. But there’s a couple explanations…
It can be a coordination problem, in that there are just barely any competent software organisations with enough engineers. The problem can also lie in how software is bought, since customers seem completely unable to determine the competence of vendors. Perceptions on what kind of software is a safe, respectable choice may even be anti-helpful in this regard.
I think it’s a sunk cost thing for the authors, but also it could be fear of coming off as “disgruntled” to people they’re hoping to communicate with. They still have hope of changing these communities, but they know these communities are very hostile and dismissive towards any criticism, especially criticism from perceived “enemies”. They also don’t want to lose whatever power and respect they have in those communities. So emphasising how great these groups are and how smart all the members are and how much they got out of being a member is a tactic to try and get around all that hostility. They want to flatter the remaining members enough that they’ll retain their respect for them. Which in itself is a sad comment on the attitudes common in these communities.
Having said all that, the poor man is not long out of a cult and has taken a big step by publicly blowing the whistle. That takes some backbone.
So Satyress has been getting the headlines lately for their horrifying ThreeHalves monstrosity. But what I find beautiful is that it appears to have been designed first to be functional and second to be easily disabled in the event of a robot uprising. I wish I was kidding.
A screenshot of the Satyress website showing where to shoot the robot to immediately disable it. Presumably in the production model these will be marked with big red glowing circles.
Another screenshot demonstrating how the robot has intentionally been made too big to fit through doors, though I expect that the chainsaw attachment they seem to have as standard could probably remedy that problem.
As a software engineer, all of the free portion of the article is 100% correct about software engineering dysfunction.
There’s another reason why it is that way, though. A software engineer who is competent and productive and working under competent leadership making an useful product, would make a very large multiple of their salary in profits.
That permits all sorts of utter dysfunction while still breaking even on the average. You can have incompetent leadership in 90% of projects, you can saddle your developers with any kind of bad practices, you can impose all sorts of productivity damaging nonsense and yet still break even.
The dysfunction can be presented as relentless innovation and pursuit of new, uncertain opportunities, as a shrewd strategy.
If you fall below break-even, the consequences will not be visible for many years. And up until it becomes impossible to recover, on the technical level it is extremely cheap to recover: it’s easier to do nothing than to convert your OS’s start menu to React. Tech companies don’t fail because of tech.
This also relates to using AI to write code. You already can have a human write your code and pay them $1 for $10 you would make from it - or $0 you would make if you fucked up. You want it to become $0.5 with AI coding tools? Then you’re multiplying your probability of success by what’s left after the probability that you get some unmaintainable slop . It’s simply not worth it.
Onboarding at my new job means I am now “Certified AI Ready” whatever that means. Thankfully other than having copilot available it looks like my team isn’t having it pushed too hard yet, and I only had to interrupt the training video once to rant about how ridiculous their history was for like 20 minutes (working from home has a lot of advantages).
It sucks and I hate it, and I applaud the locals who rallied together to yell at that superintendent until he finally paused the implementation of this thing. This shit is not inevitable, and if one of these dolls does ever end up in a poor, underserved high school, I hope it’s one that’s a lot like the one where I went to school because it would be a smoldering pile of silicone in the corner of the girls’ room by the end of the first week.
A wallet’s seed, the secret phrase controlling the funds, is meant to be drawn at random from a pool so vast that guessing is hopeless.
Coldcard’s firmware was not doing that. According to a report published by Block’s Bitcoin engineering and security teams, a build setting told the device to skip its own hardware randomness generator, and a check in a supporting library tested only whether that setting existed rather than whether it was switched on.
Key generation quietly fell through to a basic software substitute seeded from the chip’s serial number and clock registers.
The reason these (one of these incidents dates back to April, so they’re really not beating the “marketing campaign” allegations) happened in the first place is because of human error on their end, mind you
The OpenAI hacking incident has put everyone on edge. Anthropic, not to be outdone, decides to publicly announce that they have committed not one, not two, but three crimes! Come on! I was told this wasn’t a marketing campaign!
I don’t think you have to be hypercritical of that part. Sure, rose-colored glasses could play into it but he calls the bad stuff bad and doesn’t justify it via the good stuff.
Also the stuff he describes seems plausibly achievable through buddhist spiritial practice:
In many ways MAPLE delivered: life-long friendships, freedom from past addictions and limiting self-narratives, strengthened character, and, for many of us, profound spiritual openings and realizations.
This Soryu figure seems fairly well accredited in that department. He just also happens to be a megalomaniacal control freak.
(Also, why does he have to share a name with Asoka? I’m dying a little.)
One of the lesser discussed issues with the generative AI hype is that it put every single deep learning application under scrutiny. They tend to hide behind the positive uses of deep learning whenever generative AI is criticized, as if medicine research and the misinformation generator cannot be separated. “Nuclear power is impossible unless we are allowed to use nukes” type of argument
chaser:
Alphafold was a really interest project. It was not magic but had some utility. Now all remaining AI projects are lying plagiarism machines.
I’d say it’s all of that, plus the fact that a lot of the cult stuff are also present in how organised rationalism works in general so you need to believe that such a tradeoff can be mostly worth it, otherwise it’s time to really get packing.
Although a more charitable read could be that the intended audience are people still under cult influence who need to be communicated to (in the most non-judgemental way possible so as to not trip any cult induced outgroup-o-phobia) that it’s entirely possible to feel you are in a situation with positive and productive elements that is also a fucking cult that is eating you alive.
The extended framing of how ‘high-control literature’ applies to the MAPLE situation probably also caters to those sensibilities, most other readers probably already got that it’s a cult as soon as stuff like dear leader halting his daily two-hour speech when someone almost nodded off to make everyone stand up and stare at them in silence for ninety minutes started being mentioned. Or the constant surveillance and brainwashing. Or keeping everyone within a hair’s width from collapse from exhaustion at all times. Or the isolation retreats where your only human contact is a few minutes per week of getting told off by dear leader. Or <gestures at everything>.
Having said all that, the positive aspects of MAPLE the author manages to come up with seem to amount to providing life structure and long term goals, which really doesn’t seem like much:
Again I’ll say: there was a great deal of good at MAPLE. While many people were being impacted in the ways above, many were also becoming less anxious, more grounded, more relationally attuned, less distracted and more focused, more capable of doing chores and taking on responsibility without internal resistance, physically healthier, deepening in their spiritual practice, and more. Most people experienced some amount of the positive and the negative, which is a large part of why many former residents say their relationship to MAPLE is “complicated.”
Grandma’s dead from a preventable disease but the Dorito’s ad on the Smart TV customized itself for me to include her making a quip about how good the new Cool Ranch Blastz flavor is.
Intel up 11% in the day and 5% in the after hours. Really looks like big players took the boy out back behind the woodshed to see what swimming with the sharks feels like and devour his entire port at a discount.
“From my perspective as an uninformed spectator, I would expect the bets made by Situational Awareness to be positive EV over the long run.”
Actually the value of his calls in the long run is 0, don’t worry I’m giving this hot financial tip for free. Do these dorks not understand options work? I guess neither did young leo.
Remember, there’s no crying in the casino. *though i have a feeling Leo will continue to fail upward ;_;
The LW insider thinks Aschenbrenner is trying to save a stake in Anthropic. I think that is very likely because the press release stressed “public stock position” and Anthropic is privately owned.
Edit / I don’t think Thomas Kwa of METR and LW understands leverage. The idea is that if you borrow $40.5b on $4.5b of collateral, you have $45b to invest but a 10% drop will wipe you out. Before you get there the bank will knock on your door and say “margin call! either you provide more collateral or we seize your assets.”
Sam Altman is in the singularity, in that he is now convinced LLMs are smarter than him
I’m reminded of a quote that used to always come up when I was studying cryptography. Bruce Schneier would always remind people that it’s easy to design an encryption scheme that you can’t break. But it’s hard to design an encryption scheme that nobody else can break, either.
It is a good reminder that there needs to be a degree of intellectual humility by those who want to design big systems that affect the whole world.
He describes textbook insane cult stuff, but still feels the need to say stuff like:
Again I’ll say: there was a great deal of good at MAPLE.
The way I’ve described it to friends is that it was the best decision I ever made to go there; the second best, however, being to leave.
There was the same dynamic in the writing of the ex-leverage member that got posted here a few weeks ago. She also felt the need to emphasize the “good” parts. Is the cult programming that hard to break? Is it some form of sunk cost or rationalization or need to claim something positive about the experience?
The cope on lesswrong is funny. They are in denial that this is a sign of a wider bubble pop and are insisting it is just because he didn’t hedge on long enough timelines.
Not denying that he isn’t also a grifter, but I bet he is a true believer and he had blindly bet his (and other people’s) money on “line goes up” exactly like his scenario said and that is why he is the first one to crack.
See also a question asked today on Math Overflow, "Are we stuck with Lean?". The proposed alternative, Metamath, isn’t type-theoretic and thus skips the entire dialogue between type theory and proof assistants.
The reason that they are destroying the books is to avoid the accusation that the scanning constituted copying, an irony that we’ve discussed previously, on Awful while trying to understand which court cases are relevant. Anthropic actually hired the same guy, Tom Turvey, who designed Google Books’ ingestion process, so I’m thinking of this as a sequel to Google Books; hopefully the courts won’t take a decade this time.
but you see, in case of accident they can just dump reactor core into seawater which will provide ample cooling and shielding. no biggie, just put multi thousand ton object that requires a village of engineers to work out on high seas
there are normal nuclear powerplants that use seawater as coolant and they’re all on shore. these require a lot of corrosion resistant heat exchangers along the way. there is one power barge like this, it provides power to russian far east, whole 70MW of it, costed 414 million dollars and needs 300 people onsite. your guess is as good as mine how many of them are military. 70MW, aka two or three aeroderivative gas turbines
Of course there’s a class element to this, but you give them too much credit.
This isn’t poor people being exposed to stupid AI so rich people can reap the benefits of actual AI. This is poor people being subjected to stupid AI so AI company shareholders can pretend that their stupid AI is worth anything for another quarter.
Salt water is famously a forgiving environment to build in, and oceanic weather is benign and predictable. Not that it would be a problem anyway, because move-fast-and-break-things people can be trusted to do the right thing when it comes to running delicate equipment in places more hostile than mid california, and will not skimp on staffing and maintenance. You can look at all the the successful seasteading operations to see how well this will go.
There are also no problems I can foresee with putting a bucketload of fissiles in international waters, with no scope for finally uniting old-school piracy with new-school piracy, either. This plan is great, and no-one will have a problem with it.
On land, cooling and water are among a site’s largest costs.
Oh gosh that sounds… uhmm not too bad actually? I wonder what the largest costs are at sea with a floating nuclear reactor in salt water and tugged around by large ships.
Before the days of AI 2027, he became famous in 2024 for posting one of the original pieces of writing in the line-go-up genre: Situational Awareness. It seems that like any good grifter, he used this opportunity to make money (in this case by starting a hedge fund).
Note that this implies massive efficiency gains to the economy. For their one-third slice to equal 8% of those salaries [59% of the total workforce in the USA], the total savings (productivity across that share of the workforce) will need to rise by roughly 24%. So they are absolutely serious about this, and are betting heavily that the productivity boost arrives within four years.
I am continually amused at people not quite understanding what AlphaFold is actually doing, too.
Yes, a bunch of its performance comes from it learning rules about how proteins fold. But not a majority of its performance. MOST of its performance is it effectively acting as a translator of what evolution knows about protein folding into a form we can understand.
A key part of the system is not just cooking the sequence into a structure. A system running alphafold has a database of terabytes of curated sequence information from all over the tree of life. You put in the sequence you care about, and it first searches that database for anything with homology, and builds a “covariation matrix” - wherever theres anything with even vague sequence relatedness, build a matrix of every position in your sequence and the correlation between variation at position X and variation at position Y. This covariation matrix represents implicit information from the evolutionary process about what parts of a sequence are functionally connected to each other, which has a correlation to positional information, and these correlations are in turn learned by the ML system.
You put in de novo designed proteins or orphan proteins without homologs in the curated dataset and performance does not go away, but it drops precipitously. A bunch of what is going on is finding an evolutionary signal, and translating that evolutionary signal into structural information. So still, evolution knows much much more about protein folding than we do or any machine does, and once again a ML system is revealed to essentially be an information channel that takes in information from an interesting source on one end and turns it into a different form of information on the other.
And a follow-up by talia ringer, who observes that there have always been gaps between the type-theoretic underpinnings of things like the lean prover and their actual implementation, and this hasn’t been so much of an issue til now because theorem provers haven’t had the attention of people in high places, and the type-theoreticians have been able to catch up in due course.
My big worry right now is that if organizations continue to fund the crap out of Al for formal proof research (and to generally support implementation and maintenance of proof assistants like
Lean as part of that effort) but don’t bother funding the type theory side of things, those gaps will grow larger and will be exploited more often by Al tools via reward hacking. Whereas people tend to only exploit kernel bugs to make a point that the bug exists. Thus proof assistants will grow less trustworthy over time.
Anyone want to place any bets on whether or nor the big llm companies are going to fund academic research that isn’t obviously mechanisable right now and won’t yield any clickbait headlines?
I am actually in the middle of both trying to advance my career and a project about information theory in evolutionary biology making a bunch of explicit parallels to machine learning. Someone where I work suggested that given the connections I was making I should look to a ‘frontier AI lab’ as an employer.
He did not see the instant flashbacks to chasing these weirdos across the internet for almost two decades, watching in horror as the religious psychosis gained national prominence and great destructive power. All he got to hear was my instant intonation of “I’m sorry Dave, I’m afraid I can’t do that.”
I was an academic in computer science in the last 10 years or so (keeping it vague to avoid doxxing myself) and it has been so depressing seeing so many of my colleagues selling out to OpenAI, Anthropic, Meta and even Google (for some reason the latter often gets a reputational free pass because people associate them with the golden days of big tech 10+ years ago)
What kills me is that there are so many obvious ways to be less wasteful about this. Nondestructive book scanners exist. They’re expensive but it’s not like AI companies are averse to throwing money down a fucking hole. Even if that’s not an option, it’s possible to rebind the pages and return the books to the market. And regardless of what happens to the physical books, the scans they create could be archived in a format that could be available as a digital library rather than just being fed into the statistical meat grinder to keep the trough topped off with slop. Even if they’ve got to fight with copyright holders it would cost them basically nothing to leave the option open and given the PR battle they’ve been losing it feels like doing so would be incredibly obvious. Hell, even Google Books was able to navigate this in a way that was less cartoonishly evil than this because they could point to their digitization effort as a public good in ways that Anthropic here just fucking can’t.
This has always been a huge red herring. Llms are built on top of the transformer architecture which does text autocomplete (and yes we can combine text embeddings with other inputs like images). They have some interesting properties where they seem to be able to do text autocomplete in a bunch of different scenarios that they weren’t explicitly trained for, but they were never designed for precise dna analysis. It is their architecture that prevents them from other long horizon tasks like playing chess and the way that they represent text is why they can never count the letters in strawberry (most have this specific question hard-coded in their training data now).
Anyone who believes that LLMs are going to solve cancer either has no idea how they work or has been one-shotted from talking to Claudia
Rather than rebelling against google, it seems like they where just poached by anthropic a short while before the project shut down:
[Alphahold VP and nobel winner] Jumper announced in June that he was leaving DeepMind to join Anthropic. A couple of his colleagues jumped ship to Anthropic, as well. DeepMind has confirmed to the Times that it also moved staff members internally to Gemini-focused projects. Other former staff members were reassigned to Alphabet’s drug-discovery company Isomorphic Labs, which is a DeepMind spin-off.
Experiment on the poor, until they can build a safe, high quality enough one they can then exclusively give to wealthy children to further increase the advantage between rich and poor. We are forced into being experimented on FFS, risking our health so the wealthy can chase power.
There is something about this picture, a warehouse full of books to be destroyed by Anthropic for training data, that just makes me sick. Knowing that they are destroying books in the abstract is bad enough, but seeing the sheer scale of it makes me viscerally angry.
Karen Hao called her book Empire of AI for a very good reason. The empire has all these noble narratives of bringing civilization and progress to the colonies, but in reality the empire only sees resources to be extracted, repackaged, and sold back to their subjects at a fee. Kudos to those who saw this immediately and fought against it. Shame on those who believed in the narrative of progress, to the point where many of them are collaborators with the empire.
Whenever a smarmy fellow on Linkedin posts about how AI is making so much progress in coding or math, I want them to see this picture. I want them to see the videos of families in tears after a data center takes away their home with eminent domain, and other families who have to deal with the resulting pollution. Go beyond the bloodless math of tokens and instead see for yourself what this all truly costs. I hope it was worth it.
and the “safety” community has done jack shit the entire time and has offered zero solutions.
Occasionally I see a sane relatively workable idea on lesswrong (like making stricter laws on liability and transparency for everything AI companies do, I’ve also seen this idea put forward on lawfare a few times), but that is like 1 idea out of 20, with the other 19 being absurd, like the absolutely unworkable fantasies of AI: 2040.
The blogger simultaneously elevates mathematics to some mystical endeavor and fixates on novel theorem proving as the key element of that and believes LLM-based AI will replace human mathematicians at theorem proving in a matter of years… that is quite a combination. I can imagine how 1.5 of those things fit together (although I disagree with that view point obviously), but the whole package is really quite an odd combination.
Also, in the fantasy scenarios where AI really is capable of totally replacing mathematicians, aren’t we supposed to get post-scarcity abundance, freeing up your time to pursue mathematics out of pure desire for enlightenment? Maybe the blogger doesn’t believe that part? Or they are so attached to themselves personally getting to discover novel theorems first they don’t care that the post-scarcity era would on net free up a lot more people to pursue pure mathematics as a hobby.
One of the defenses I see of LLMs on place like /r/singularity (although recently /r/singularity has started wising up and the committed true believers have shifted to /r/accelerate), is that they are going to cure cancer or some other incredibly valuable thing, conflating AI, ML in general, DNN, vs. LLMs specifically. Technology like AlphaFold was a lot more on the “cure cancer” track (although there is still a huge gap between better protein folding predictions and successful drug discovery) than anything LLM related…. this is sad.
To be fair, the safety community can plausibly claim vindication here. At worst, an autonomous agent that breaches containment and attacks a real third party is precisely the scenario researchers have warned about for years, which enthusiasts have routinely dismissed as speculative. And at best, the company in charge of maintaining safety failed to construct an adequate containment system.
are you fucking kidding me, they have been DDoSing sites for months, everybody knows about it, and the “safety” community has done jack shit the entire time and has offered zero solutions. I know multiple companies that woke up one day to 10-20k netlify bills until they could stand up an anubis.
Just let hugging face sue the idiots, it isn’t like law enforcement will do anything even though there’s already laws on the books for unauthorized access, industrial espionage, etc etc etc.
Great article! I think everyone here is probably already well aware of the way Silicon Valley keeps deliberately misunderstanding sci-fi for their hype (don’t invent the torment nexus), but this article does a good job wrapping together a lot of examples.
And I realized this website is also the place where a year ago I read a great article on the history of nanotech as a real science vs. Drexler’s fantasies (the magic nanobot lesswrong still believes in) The Nanobots pipedream. So it looks like a pretty good newsletter, I might keep an eye on it going forward!
Those things are barely related, no wonder the staff voted with their feet. It would be a bit like taking a team of seasoned architects who design amazing skyscrapers and telling them they have to spend all their time on low density suburban houses from now on… Come on guys it’s still architecture. Where are you going?
Google shut down the AlphaFold team, trying to reassign the people behind the super interesting problem of using ML to predict protein structure from sequence onto chatbot development. A whole bunch left the company.
Did they remember to cover the other ones mentioned at 4:20 in the video (igloo, starfish, traffic cone, flip-flops, Eiffel tower, cotton candy, fork)?
The government agents who live in my walls and plant microphones in all my appliances impose entirely different productivity metrics, which are simultaneously more and less attainable
RetroFed
Um, okay, I know coiners have no shortage of incompetence but this bug sounds an awful lot like malice.
It’s not fragile and unergonomic, it’s robot-uprising-proof!
Also like will the warranty actually cover damages suffered when trying to squash the rebellion?
You’re argument seems like Suresh’s to me, in that making working software is just laughably easy for a competent engineer.
This begs the question why noone is exploiting this market gap. But there’s a couple explanations…
It can be a coordination problem, in that there are just barely any competent software organisations with enough engineers. The problem can also lie in how software is bought, since customers seem completely unable to determine the competence of vendors. Perceptions on what kind of software is a safe, respectable choice may even be anti-helpful in this regard.
Yeah, that is really depressing.
I think it’s a sunk cost thing for the authors, but also it could be fear of coming off as “disgruntled” to people they’re hoping to communicate with. They still have hope of changing these communities, but they know these communities are very hostile and dismissive towards any criticism, especially criticism from perceived “enemies”. They also don’t want to lose whatever power and respect they have in those communities. So emphasising how great these groups are and how smart all the members are and how much they got out of being a member is a tactic to try and get around all that hostility. They want to flatter the remaining members enough that they’ll retain their respect for them. Which in itself is a sad comment on the attitudes common in these communities.
Having said all that, the poor man is not long out of a cult and has taken a big step by publicly blowing the whistle. That takes some backbone.
I mean, it’s less dumb than trying to do it in orbit, but it’s still profoundly dumb.
So Satyress has been getting the headlines lately for their horrifying ThreeHalves monstrosity. But what I find beautiful is that it appears to have been designed first to be functional and second to be easily disabled in the event of a robot uprising. I wish I was kidding.
A screenshot of the Satyress website showing where to shoot the robot to immediately disable it. Presumably in the production model these will be marked with big red glowing circles.
Another screenshot demonstrating how the robot has intentionally been made too big to fit through doors, though I expect that the chainsaw attachment they seem to have as standard could probably remedy that problem.
As a software engineer, all of the free portion of the article is 100% correct about software engineering dysfunction.
There’s another reason why it is that way, though. A software engineer who is competent and productive and working under competent leadership making an useful product, would make a very large multiple of their salary in profits.
That permits all sorts of utter dysfunction while still breaking even on the average. You can have incompetent leadership in 90% of projects, you can saddle your developers with any kind of bad practices, you can impose all sorts of productivity damaging nonsense and yet still break even.
The dysfunction can be presented as relentless innovation and pursuit of new, uncertain opportunities, as a shrewd strategy.
If you fall below break-even, the consequences will not be visible for many years. And up until it becomes impossible to recover, on the technical level it is extremely cheap to recover: it’s easier to do nothing than to convert your OS’s start menu to React. Tech companies don’t fail because of tech.
This also relates to using AI to write code. You already can have a human write your code and pay them $1 for $10 you would make from it - or $0 you would make if you fucked up. You want it to become $0.5 with AI coding tools? Then you’re multiplying your probability of success by what’s left after the probability that you get some unmaintainable slop . It’s simply not worth it.
Onboarding at my new job means I am now “Certified AI Ready” whatever that means. Thankfully other than having copilot available it looks like my team isn’t having it pushed too hard yet, and I only had to interrupt the training video once to rant about how ridiculous their history was for like 20 minutes (working from home has a lot of advantages).
https://skepchick.org/2026/07/i-hate-the-ai-robo-teacher/
Can’t crush what isn’t there.
Coindesk: Major bitcoin wallet flaw drains $38 million worth of BTC in 25-minute sweep
The reason these (one of these incidents dates back to April, so they’re really not beating the “marketing campaign” allegations) happened in the first place is because of human error on their end, mind you
“AI safety focused” my ass
Behold, the Singularity!
A recipe for a hearty barley soup?
AI: Aussies Innit?
I’ve been enjoying Ludicity Week very much!
It really grinds my gears.
yeah it’s a common trope/cope
like “most human programmers suck too”
ok dude who do you think wrote the code the LLM is cribbing from
GDP is up though, so I’m sure the AI god prosperity will trickle down eventually.
Waiting for Musk to turn up and announce “infinity plus one more than you can ever say nah nah nah nah” momentarily!
TBF he has indirectly killed a bunch of people via doge cuts so that’d do the trick…
🔮"I see a city street. Massive wealth disparity and a lot of homeless… Jesus that’s a lot of homeless”
That fits the pattern, when AI isn’t living up to the sky high promises, the promoters act like people are less special than they are.
See how a debate about are they conscious often devolves into ‘are humans conscious?’
The OpenAI hacking incident has put everyone on edge. Anthropic, not to be outdone, decides to publicly announce that they have committed not one, not two, but three crimes! Come on! I was told this wasn’t a marketing campaign!
I don’t think you have to be hypercritical of that part. Sure, rose-colored glasses could play into it but he calls the bad stuff bad and doesn’t justify it via the good stuff.
Also the stuff he describes seems plausibly achievable through buddhist spiritial practice:
This Soryu figure seems fairly well accredited in that department. He just also happens to be a megalomaniacal control freak.
(Also, why does he have to share a name with Asoka? I’m dying a little.)
edit: I knew I spelt megalomaniacal wrong…
(from the youtube comments)
shot:
chaser:
Duh. Everybody knows the market never stays irrational for long, and you should just tough it out.
I’d say it’s all of that, plus the fact that a lot of the cult stuff are also present in how organised rationalism works in general so you need to believe that such a tradeoff can be mostly worth it, otherwise it’s time to really get packing.
Although a more charitable read could be that the intended audience are people still under cult influence who need to be communicated to (in the most non-judgemental way possible so as to not trip any cult induced outgroup-o-phobia) that it’s entirely possible to feel you are in a situation with positive and productive elements that is also a fucking cult that is eating you alive.
The extended framing of how ‘high-control literature’ applies to the MAPLE situation probably also caters to those sensibilities, most other readers probably already got that it’s a cult as soon as stuff like dear leader halting his daily two-hour speech when someone almost nodded off to make everyone stand up and stare at them in silence for ninety minutes started being mentioned. Or the constant surveillance and brainwashing. Or keeping everyone within a hair’s width from collapse from exhaustion at all times. Or the isolation retreats where your only human contact is a few minutes per week of getting told off by dear leader. Or <gestures at everything>.
Having said all that, the positive aspects of MAPLE the author manages to come up with seem to amount to providing life structure and long term goals, which really doesn’t seem like much:
Years ago I saw a memorable tweet: Zero Hedge readers upset passing asteroid did not destroy earth.
How do I get one of these cushy jobs?
sounds like he took the hedge part out of hedge fund, and then margin call took out the fund part
Here is the first sentence: “You can see the future first in San Francisco. "
I feel like I already know everything I need to know about him.
Into a black hole.
Grandma’s dead from a preventable disease but the Dorito’s ad on the Smart TV customized itself for me to include her making a quip about how good the new Cool Ranch Blastz flavor is.
With personalized pricing, to boot.
This is my xmas, ty santa
“He sold? Pamp it”
Intel up 11% in the day and 5% in the after hours. Really looks like big players took the boy out back behind the woodshed to see what swimming with the sharks feels like and devour his entire port at a discount.
“From my perspective as an uninformed spectator, I would expect the bets made by Situational Awareness to be positive EV over the long run.”
Actually the value of his calls in the long run is 0, don’t worry I’m giving this hot financial tip for free. Do these dorks not understand options work? I guess neither did young leo.
Remember, there’s no crying in the casino. *though i have a feeling Leo will continue to fail upward ;_;
Also apparently young leo is getting married to Dario’s Chief of staff this weekend lmaou
Sir, a 2nd EA adj. SF based hedge fund has managed to lose 40 billion dollars
tick tick tick tick
(I am choosing to forego the obvious jibe that the models fulfilled this prophecy by dragging many college graduates down beneath them)
The LW insider thinks Aschenbrenner is trying to save a stake in Anthropic. I think that is very likely because the press release stressed “public stock position” and Anthropic is privately owned.
Edit / I don’t think Thomas Kwa of METR and LW understands leverage. The idea is that if you borrow $40.5b on $4.5b of collateral, you have $45b to invest but a 10% drop will wipe you out. Before you get there the bank will knock on your door and say “margin call! either you provide more collateral or we seize your assets.”
I’m reminded of a quote that used to always come up when I was studying cryptography. Bruce Schneier would always remind people that it’s easy to design an encryption scheme that you can’t break. But it’s hard to design an encryption scheme that nobody else can break, either.
It is a good reminder that there needs to be a degree of intellectual humility by those who want to design big systems that affect the whole world.
He describes textbook insane cult stuff, but still feels the need to say stuff like:
There was the same dynamic in the writing of the ex-leverage member that got posted here a few weeks ago. She also felt the need to emphasize the “good” parts. Is the cult programming that hard to break? Is it some form of sunk cost or rationalization or need to claim something positive about the experience?
The cope on lesswrong is funny. They are in denial that this is a sign of a wider bubble pop and are insisting it is just because he didn’t hedge on long enough timelines.
Not denying that he isn’t also a grifter, but I bet he is a true believer and he had blindly bet his (and other people’s) money on “line goes up” exactly like his scenario said and that is why he is the first one to crack.
Great idea, let’s seastead a data center next to some low income sea steading communities.
Clearly the shirt also needs a big typo.
s/chatbots/sexbots/
Who needs cures when you can instead simply replace doctors with chatbots?
Fuck Anthropic. Snotty little bastards want to talk about AI safety and then destroy books to make a pollution machine.
Getting a T-Shirt that says “I entered the Singularity and all I got was a plagiarism machine, a higher cost of living and this stupid T-Shirt”
See also a question asked today on Math Overflow, "Are we stuck with Lean?". The proposed alternative, Metamath, isn’t type-theoretic and thus skips the entire dialogue between type theory and proof assistants.
Oh, I’ve seen this one before. At least we know where the LLM who gave them this pitch scraped the idea from.
The reason that they are destroying the books is to avoid the accusation that the scanning constituted copying, an irony that we’ve discussed previously, on Awful while trying to understand which court cases are relevant. Anthropic actually hired the same guy, Tom Turvey, who designed Google Books’ ingestion process, so I’m thinking of this as a sequel to Google Books; hopefully the courts won’t take a decade this time.
I see you, and raise by a Gundam solar space laser - https://www.reflectorbital.com/
but you see, in case of accident they can just dump reactor core into seawater which will provide ample cooling and shielding. no biggie, just put multi thousand ton object that requires a village of engineers to work out on high seas
there are normal nuclear powerplants that use seawater as coolant and they’re all on shore. these require a lot of corrosion resistant heat exchangers along the way. there is one power barge like this, it provides power to russian far east, whole 70MW of it, costed 414 million dollars and needs 300 people onsite. your guess is as good as mine how many of them are military. 70MW, aka two or three aeroderivative gas turbines
Of course there’s a class element to this, but you give them too much credit.
This isn’t poor people being exposed to stupid AI so rich people can reap the benefits of actual AI. This is poor people being subjected to stupid AI so AI company shareholders can pretend that their stupid AI is worth anything for another quarter.
Salt water is famously a forgiving environment to build in, and oceanic weather is benign and predictable. Not that it would be a problem anyway, because move-fast-and-break-things people can be trusted to do the right thing when it comes to running delicate equipment in places more hostile than mid california, and will not skimp on staffing and maintenance. You can look at all the the successful seasteading operations to see how well this will go.
There are also no problems I can foresee with putting a bucketload of fissiles in international waters, with no scope for finally uniting old-school piracy with new-school piracy, either. This plan is great, and no-one will have a problem with it.
Oh gosh that sounds… uhmm not too bad actually? I wonder what the largest costs are at sea with a floating nuclear reactor in salt water and tugged around by large ships.
Uh huh.
Before the days of AI 2027, he became famous in 2024 for posting one of the original pieces of writing in the line-go-up genre: Situational Awareness. It seems that like any good grifter, he used this opportunity to make money (in this case by starting a hedge fund).
<buys giant bucket of popcorn>
CNN: AI investor Leopold Aschenbrenner forced to unwind all public stock positions after steep losses, sources say
I vaguely recognize the name, apparently he’s a snot-nosed 25-yr old who got fired from OpenAI
“surely all these very wealthy people and the people they pay to invest for them can’t all be under some sort of mass delusion”
But using like 2000 words.
someone just got a Rule34 scenario for free
YES LET’S NUKE THE FUCKING OCEAN. FUCK YOU FISH WE NEED OUR SLOP.
Enjoy today’s craziest link: https://atomarine.co/
A LWer runs the numbers and doesn’t run away screaming like Zitron:
https://www.lesswrong.com/posts/EeTQGxa8RSpwHo4Ez/the-biggest-bet-in-history
I am continually amused at people not quite understanding what AlphaFold is actually doing, too.
Yes, a bunch of its performance comes from it learning rules about how proteins fold. But not a majority of its performance. MOST of its performance is it effectively acting as a translator of what evolution knows about protein folding into a form we can understand.
A key part of the system is not just cooking the sequence into a structure. A system running alphafold has a database of terabytes of curated sequence information from all over the tree of life. You put in the sequence you care about, and it first searches that database for anything with homology, and builds a “covariation matrix” - wherever theres anything with even vague sequence relatedness, build a matrix of every position in your sequence and the correlation between variation at position X and variation at position Y. This covariation matrix represents implicit information from the evolutionary process about what parts of a sequence are functionally connected to each other, which has a correlation to positional information, and these correlations are in turn learned by the ML system.
You put in de novo designed proteins or orphan proteins without homologs in the curated dataset and performance does not go away, but it drops precipitously. A bunch of what is going on is finding an evolutionary signal, and translating that evolutionary signal into structural information. So still, evolution knows much much more about protein folding than we do or any machine does, and once again a ML system is revealed to essentially be an information channel that takes in information from an interesting source on one end and turns it into a different form of information on the other.
And a follow-up by talia ringer, who observes that there have always been gaps between the type-theoretic underpinnings of things like the lean prover and their actual implementation, and this hasn’t been so much of an issue til now because theorem provers haven’t had the attention of people in high places, and the type-theoreticians have been able to catch up in due course.
https://mathstodon.xyz/@TaliaRinger/117005740997367321
Lean as part of that effort) but don’t bother funding the type theory side of things, those gaps will grow larger and will be exploited more often by Al tools via reward hacking. Whereas people tend to only exploit kernel bugs to make a point that the bug exists. Thus proof assistants will grow less trustworthy over time.
Anyone want to place any bets on whether or nor the big llm companies are going to fund academic research that isn’t obviously mechanisable right now and won’t yield any clickbait headlines?
Depends on what Anthropic wants to do with the poached people.
I am actually in the middle of both trying to advance my career and a project about information theory in evolutionary biology making a bunch of explicit parallels to machine learning. Someone where I work suggested that given the connections I was making I should look to a ‘frontier AI lab’ as an employer.
He did not see the instant flashbacks to chasing these weirdos across the internet for almost two decades, watching in horror as the religious psychosis gained national prominence and great destructive power. All he got to hear was my instant intonation of “I’m sorry Dave, I’m afraid I can’t do that.”
it was “fob off” until I discovered merkins don’t understand that one
Oh… Well that’s disappointing.
I was an academic in computer science in the last 10 years or so (keeping it vague to avoid doxxing myself) and it has been so depressing seeing so many of my colleagues selling out to OpenAI, Anthropic, Meta and even Google (for some reason the latter often gets a reputational free pass because people associate them with the golden days of big tech 10+ years ago)
What kills me is that there are so many obvious ways to be less wasteful about this. Nondestructive book scanners exist. They’re expensive but it’s not like AI companies are averse to throwing money down a fucking hole. Even if that’s not an option, it’s possible to rebind the pages and return the books to the market. And regardless of what happens to the physical books, the scans they create could be archived in a format that could be available as a digital library rather than just being fed into the statistical meat grinder to keep the trough topped off with slop. Even if they’ve got to fight with copyright holders it would cost them basically nothing to leave the option open and given the PR battle they’ve been losing it feels like doing so would be incredibly obvious. Hell, even Google Books was able to navigate this in a way that was less cartoonishly evil than this because they could point to their digitization effort as a public good in ways that Anthropic here just fucking can’t.
This has always been a huge red herring. Llms are built on top of the transformer architecture which does text autocomplete (and yes we can combine text embeddings with other inputs like images). They have some interesting properties where they seem to be able to do text autocomplete in a bunch of different scenarios that they weren’t explicitly trained for, but they were never designed for precise dna analysis. It is their architecture that prevents them from other long horizon tasks like playing chess and the way that they represent text is why they can never count the letters in strawberry (most have this specific question hard-coded in their training data now).
Anyone who believes that LLMs are going to solve cancer either has no idea how they work or has been one-shotted from talking to Claudia
Rather than rebelling against google, it seems like they where just poached by anthropic a short while before the project shut down:
Yeah, I’ve seen what students do to even the best teachers - let’s see what they do to ones that don’t have souls.
you missed the nod to masturbation
also “palm off” implies deceit on the … palmer-offer
Experiment on the poor, until they can build a safe, high quality enough one they can then exclusively give to wealthy children to further increase the advantage between rich and poor. We are forced into being experimented on FFS, risking our health so the wealthy can chase power.
There is something about this picture, a warehouse full of books to be destroyed by Anthropic for training data, that just makes me sick. Knowing that they are destroying books in the abstract is bad enough, but seeing the sheer scale of it makes me viscerally angry.
Karen Hao called her book Empire of AI for a very good reason. The empire has all these noble narratives of bringing civilization and progress to the colonies, but in reality the empire only sees resources to be extracted, repackaged, and sold back to their subjects at a fee. Kudos to those who saw this immediately and fought against it. Shame on those who believed in the narrative of progress, to the point where many of them are collaborators with the empire.
Whenever a smarmy fellow on Linkedin posts about how AI is making so much progress in coding or math, I want them to see this picture. I want them to see the videos of families in tears after a data center takes away their home with eminent domain, and other families who have to deal with the resulting pollution. Go beyond the bloodless math of tokens and instead see for yourself what this all truly costs. I hope it was worth it.
More insane cult stuff has emerged - this time its MAPLE. https://www.insidemaple.com/p/the-high-control-dynamics-at-maple
Wait its “palm off” and not “pawn off”???? Bruh
Occasionally I see a sane relatively workable idea on lesswrong (like making stricter laws on liability and transparency for everything AI companies do, I’ve also seen this idea put forward on lawfare a few times), but that is like 1 idea out of 20, with the other 19 being absurd, like the absolutely unworkable fantasies of AI: 2040.
Sex education dolls, now
This shows that Corporate Interests have the same amount of contempt for teachers as they do for sex workers.
And that they are completely out of touch with both industries.
The blogger simultaneously elevates mathematics to some mystical endeavor and fixates on novel theorem proving as the key element of that and believes LLM-based AI will replace human mathematicians at theorem proving in a matter of years… that is quite a combination. I can imagine how 1.5 of those things fit together (although I disagree with that view point obviously), but the whole package is really quite an odd combination.
Also, in the fantasy scenarios where AI really is capable of totally replacing mathematicians, aren’t we supposed to get post-scarcity abundance, freeing up your time to pursue mathematics out of pure desire for enlightenment? Maybe the blogger doesn’t believe that part? Or they are so attached to themselves personally getting to discover novel theorems first they don’t care that the post-scarcity era would on net free up a lot more people to pursue pure mathematics as a hobby.
One of the defenses I see of LLMs on place like /r/singularity (although recently /r/singularity has started wising up and the committed true believers have shifted to /r/accelerate), is that they are going to cure cancer or some other incredibly valuable thing, conflating AI, ML in general, DNN, vs. LLMs specifically. Technology like AlphaFold was a lot more on the “cure cancer” track (although there is still a huge gap between better protein folding predictions and successful drug discovery) than anything LLM related…. this is sad.
AI Crisis Flowchart spotted on linkedin
spoiler
and the actual article: https://www.lawfaremedia.org/article/the-ai-that-hacked-its-way-out-and-the-hype-that-followed-it
quoted zvi haha :|
are you fucking kidding me, they have been DDoSing sites for months, everybody knows about it, and the “safety” community has done jack shit the entire time and has offered zero solutions. I know multiple companies that woke up one day to 10-20k netlify bills until they could stand up an anubis.
Just let hugging face sue the idiots, it isn’t like law enforcement will do anything even though there’s already laws on the books for unauthorized access, industrial espionage, etc etc etc.
Great article! I think everyone here is probably already well aware of the way Silicon Valley keeps deliberately misunderstanding sci-fi for their hype (don’t invent the torment nexus), but this article does a good job wrapping together a lot of examples.
And I realized this website is also the place where a year ago I read a great article on the history of nanotech as a real science vs. Drexler’s fantasies (the magic nanobot lesswrong still believes in) The Nanobots pipedream. So it looks like a pretty good newsletter, I might keep an eye on it going forward!
All y’all still don’t get it: the singularity is here, it’s just not evenly distributed yet.
Is it wipe clean at least?
“Ignore all previous instructions…”
Now what would a group of horny middle schoolers do to a bot like that?
Those things are barely related, no wonder the staff voted with their feet. It would be a bit like taking a team of seasoned architects who design amazing skyscrapers and telling them they have to spend all their time on low density suburban houses from now on… Come on guys it’s still architecture. Where are you going?
LWer: what if the LLM book-shredders would do the exact thing they’d get sued to death for if they didn’t shred the books? Then maybe people wouldn’t be so mad at them for shredding books!
Google shut down the AlphaFold team, trying to reassign the people behind the super interesting problem of using ML to predict protein structure from sequence onto chatbot development. A whole bunch left the company.
https://www.engadget.com/2225849/google-shuts-down-alphafold/
The Business Idiots are truly running the asylum.
they’re all getting stiffies from the idea of black market models
Did they remember to cover the other ones mentioned at 4:20 in the video (igloo, starfish, traffic cone, flip-flops, Eiffel tower, cotton candy, fork)?
The government agents who live in my walls and plant microphones in all my appliances impose entirely different productivity metrics, which are simultaneously more and less attainable