important instance shit

We are defederating awful.systems from:

3 days ago on the 12th I pushed a new version of the awful.systems infrastructure flake and our production ruggedized Lemmy fork to try and fix the problem where AI scrapers would come in and push over the instance, resulting in a bunch of 503s and general slowness. I’m happy to announce we’re now running with iocaine, which seems to have fixed things nicely – though as always, let me know if anything seems off and ping me on Mastodon if the instance goes down. in addition to iocaine, I’ve added a suite of live metrics available to instance admins and moderators. if I haven’t sent you credentials for this yet and you think you should have them, shoot me a message. I apologize that these changes took me so long to get working, but I’m very happy they seem to be effective. I’ve also got some notes on running iocaine in production I can share with any interested parties, and I’ll be reaching out to iocaine’s developer with some suggestions around making it a little bit easier to work with, since at this point it’s a crucial part of running an independent web site.

the instance is going down for a bit at 8 AM GMT on November 12th to finally put us behind iocaine, which should hopefully reduce or eliminate the amount of 503 errors we’re seeing (since all of them are scrapers pushing over the instance) and expose some metrics for admins trying to keep track of how our instance is performing

the good news: we’re now on the newest stable lemmy!

our version of lemmy is old enough that clients like mlem are starting to break due to API drift, so I’m finally upgrading us to the latest stable version of lemmy. this will involve a bit of downtime and potentially a number of breakages; keep an eye out for anything that doesn’t look right after the upgrade and let us know!

after some extended downtime, I rolled out the following changes to our instance:

since we’ve been experiencing a few image cache breakages, I’m scheduling some maintenance for January 24th at 8AM GMT to upgrade our pict-rs version, increase the total amount of storage available to our production instance, and do a handful of other maintenance tasks. this won’t include a lemmy upgrade, but I plan to do one soon after this maintenance round. I anticipate the maintenance should take around 2-4 hours, but will post updates on the instance downtime page and Mastodon if anything changes.

this is somewhat of a bigger update, and it’s the product of a few things that have been in progress for a while:

we’ve exceeded the usage tier for our email sending API today (and they kindly didn’t email me to tell me that was the case until we were 300% over), so email notifications might be a bit spotty/non-working for a little bit. I’m working on figuring out what we should migrate to — I’m leaning towards AWS SES as by far the cheapest option, though I’m no Amazon fan and I’m open to other options as long as they’ve got an option to send with SMTP

On Monday, June 26th, 2023 at 07:44:07 UTC, our esteemed host @self posted:

we’re seeing a bit of spam come in from lemmy.world. if you happen to see any (and a lot of it seems to be in DMs), make sure to flag it. that’ll let both us and the originating instance’s mods know. if we get a bunch of reports and it seems like lemmy.world isn’t cleaning things up properly, we’ll take further steps to limit the amount of spam we get

this one should hopefully fix the remaining token issues folks have been having, though I’m not seeing anything in the commit log about fixes for the other session and pagination issues we’ve noticed. as always, let me know if anything looks broken. I’m still working on getting Photon deployed, which might be a good workaround for the frontend breakages we’ve been seeing.

today’s (later than planned) upgrade to lemmy 0.19.2 provisionally appears to have gone alright. if you see excessive amounts of jank (and your page footer can’t decide what version of lemmy it’s running on, IE it shows separate FE and BE versions), clear your browser cache and cookies since lemmy doesn’t seem to do that cleanly on its own

I’m taking awful.systems down for a bit tomorrow (January 13) around 11 PM GMT because after 16 release candidates and 2 hotfixes, lemmy 0.19.x finally seems like a safe enough upgrade. this is going to be a major one, so I’ll be taking our instance down temporarily to get a database backup before I apply the upgrade. expect exceptional levels of jank!

Awful Systems practically invented the term “abyss diving”. We will enable the aptitude of eyeballs to unleash. If all of this sounds marvelous to you, that’s because it is! We will deliver the capacity of federated communities to enhance. Think intra-virtual. Think micro-co-branded. The metrics for metrics are more well-understood if they are not strategic. The global, wireless ActivityPub communities factor can be summed up in one word: 1000/60/60/24/7/365. What does the commonly-used term “re-sizing” really mean? Without compelling, cross-media abyss diving, you will lack interfaces. We apply the proverb “You cannot have your cake and eat it too” not only to our relationships but our ability to maximize. Do you have a plan of action to become end-to-end? We do.

now that threads is starting to federate, they sure as fuck aren’t with us

a couple of our regulars have expressed interest in having an anti-cryptocurrency sub here. so interest check: reply to this thread if you want us to have buttcoin

update: the fix for this was stupid, please let me know if anything still looks broken

given the absolute fucking state of the open source community in general, and the fact that hacker news of all places is where the majority of new open source projects get discovered, is there any interest in starting a community here where folks can announce and solicit for help with their open source projects?

whoa, lemmygrad got a vaporwave logo and a much stupider name! too bad their posts are still fucking terrible

some quick awful.systems infrastructure updates:

I added rammy to the instance blocklist because it’s apparently unmoderated and has been invaded by anime nazis

I defederated us from two lemmy instances: - exploding-heads: transphobia - basedcount: finally I get to ban most of r/PoliticalCompassMemes in one go

we suffered some extremely unexpected downtime while I deployed a trivial change (a reverse proxy from http://awful.systems/archives to http://these.awful.systems/archives) to prod

see this lemmy-ansible github issue for the fix; basically, our web server now knows how to handle activitypub traffic in a more conforming way

https://sub.rehab

big update, awful.systems is now a federated lemmy instance. let me know if anything looks broken! here’s what to expect: - to pull up an awful.systems community on another instance, just paste that community’s URL into the other instance’s search bar - federation with other lemmy instances should work, and probably kbin too? there’s no way I can find to pull in pre-federation posts on remote instances though, so send your friends here to read the backlogs - we can’t federate with most of mastodon right now because lemmy doesn’t implement authorized_fetch, which is a best practice setting for mastodon instances. if your instance doesn’t use it, try entering something like @sneerclub@awful.systems into your mastodon search; lemmy communities are represented to mastodon as users - this is pretty much an experimental thing so if we have to turn it off, I’ll send out another post - reply to this post with ideas for moderation tools and instances you’d like to see blocked (and a reason why) and we’ll take action on anything that sounds like a good idea

I was gonna do this quietly since I was doing it mostly for security fixes, but now I guess I gotta announce that I deployed lemmy 0.18.1 to the awful.systems cluster. changes include

I rolled out some minor but important updates to the deployment cluster just now: - email notifications are now enabled in production. let me know if this tanks performance. also we’re on a fairly limited email plan so I’ll post an update if we exceed its limits (which’ll break notifications again) - we now have a staging deployment and a development deployment, which let me make these changes without taking production down - lemmy-ui in production is now running in production mode, which should improve performance slightly - late update: awful.systems now reports a correct lemmy backend version, so lemmy mobile apps should work. I confirmed that mlem on ios works, but let me know if jerboa or anything else is broken

I just finished a migration that doubled the resources awful.systems has available. let me know if I fucked anything up and didn’t notice

this instance runs on open infrastructure. the code that deploys awful.systems is available here.