Hey everyone,
We've started to hit the upper limit of what our current server is able to handle in terms of performance, and we've noticed its having a performance impact on the site itself.
For example, this is the CPU usage as I type this:
We're going to make the move to a much more powerful server, as we've done all we can on this one to control performance.
If all goes smoothly, I predict a couple of hours downtime while we migrate the database between servers and then move the surrounding services over.
The window is going to be 5 hours, so any posts made in this time will most likely be lost, especially if we have to roll back to the old server.
Once the update is done, Cloudflare should handle most of the DNS issues but you might need to hard refresh the site.
You can keep an eye on status.lemmy.zip for more information.
Please let me know if you have questions in the comments, but I will be doing a full breakdown in the next server update in a couple of days.
Thanks
Demigodrick
Oh dang, what's caused the uptick on server usage? More people joining?
Yeah pretty much, we're still on an upwards trend of active users so thats taking some of the resource.
Part of the issue is all of the federation traffic as the rest of the fediverse grows, and processing all of that. Some sites have struggled with lemmy.world for example due to their size.
The backend services (such as pictrs to serve images) are getting more advanced but that also requires more resources.
Interesting. One advantage I could see to federation would be the ability to process some content on a delay at times of high load. I.e. prioritize local content then catch-up on federated content as possible. I assume that isn't how it works rn?
That could help alleviate federation forcing smaller instances to have to keep up with at least similarly big hardware requirements.
I believe there are discussions around "batching" updates together rather than sending each update as it happens. Pros and cons both ways (slower federation updates, but less resource) so the very clever people and devs will hopefully be able to come up with a good solution.