RetroFedImprove very slow library scans on Jellyfin 10.11 / 12 on spinning media
Avid Amoeba
I’ve been trying to upgrade from 10.10 to 10.11 for a while now, as the Android TV app keeps nagging me, and every attempt ended with impossibly long library scan times.
After some thorough investigation, it appeared that a Home Videos type collection causes unending scan (yet to be solved), but also that Jellyfin does a lot of writes to the config directory (either database or metadata or both). Mine’s on spinning media part of a ZFS pool. I tried a few performance tuning options, such as testing the config dir with recordsize (similar to block size) of 4K, 8K, 64K, 128K and library scans fell from 30-40 minutes down to 8-13min with 4K-64K. The ZFS tuning wiki suggest 64K recordsize with LZ4 compression for SQLite workloads such as Jellyfin. That seems to work as well as 4K and 8K but likely is faster when reading thumbnails and such.
Note that upgrading to 12-rc3, which is supposed to speed up library scans did not improve scan times for me. Optimizing config/database write speed did. I cross-checked the culprit by experimenting with moving the config dir to NVMe and RAM. Both of those got the scan times down to 8-9 minutes compared to the optimized spinning media’s 12-13.
So if you had upgraded (or about to) to 10.11 your library scans are (about to get) dog slow and your Jellyfin’s config dir resides on spinning media, optimize its write performance for SQLite.
Share on Mastodon
Share on Reddit
Share on WhatsApp
They won’t do that because it’s expensive and free software generally doesn’t have the budget to do this
I don’t know why you think the “just works” is any different from tunneling with a service. Your server is still opened to the internet through upnp with direct connections and through a tunnel to plex.tv when a direct connection is not possible. In fact plex is inherently less secure because their infra is both the encryption endpoint (as opposed to your client with Jellyfin and Tailscale or whatever) and their infra has been vulnerable in the past (like the massive breach in 2022).
Jellyfin with something like nginx and a vpn is open to the internet, yes, but a service that tunnels (like wireguard, Tailscale) bypasses this issue and it’s up to you to set it up as to your level of comfort
Plex is just easier but as with all things tech (especially those infected with VC dollars) “ease” translates to less secure and far more likely to exploit your data
I came up with a funny strategy I use to lock it down a bit. What’s exposed to the internet for me is Apache2 reverse proxy. The proxy is locked down to reject all connections EXCEPT for the ones coming from a special subdomain which is something like a 64-character long random string. This prevents pretty any unwanted connections. Obviously the special subdomain must remain as secret as a shared password among the Jellyfin users. It works for trusted users.
What I want ideally is an “authenticated firewall.” OpenWrt rejecting all connections on the open port except for an allowlist of IPs. Then there must be a system where users can authenticate and their IP is added to the allowlist. I haven’t found an off-the-shelf solution like this but I’ll make it some day. Too bad I figured this random string subdomain trick cause it seems good enough for now. :D
A semi-automated whitelist solution would be nice. I’ve settled for Crowdsec with very strict automatic banning behavior.
How is your DNS set up for that subdomain? Is it on a wildcard DNS record?
Do you mean the SSL cert? Yes, that’s wildcard on *.mydomain.com. Then the subdomain is kvtn4ftxfreurdcw7qtr21mcywxaqqm.mydomain.com.
In this case I mean the DNS entry for the random string. The thing pointing that subdomain at your IP/proxy.
It’s just an A record pointing to my IP. IP’s updated from my router via the DNS provider API.
If that A record isn’t a wildcard, anyone can see it, is what I’m getting at.
Hm. As far as I know that can only be done if AXFR is enabled (it’s not), if the domain has entered some search engine that has remembered it, through SSL cert that’s not wildcard, or through brute force lookup. Am I missing something?
Give it a look on https://dnsdumpster.com/ or similar. Your dns records are inherently public, so anyone that works out the domain can work out the subdomain. In fact there are plenty of tools that just scan every domain/subdomain they can find.
Thanks for the pointer. Checked, tried a couple others - they don’t know about it. They have some other non-random DNS records. AFAIK they can absolutely find it if they scan for all domains, but it’ll take forever if the name is randomly-generated and sufficiently long. Someone has to be determined to spend the resources. This doesn’t guard against that but against bots trying to fuck with the service at the port. I could move it to a wildcard though. There’s an overlap with another subdomain (they’re actually sub-subdomains and the first sub is common) but I could move that.