51
Home
457 readers
1 users here now
Lemmy.zip instance discussion.
For all things relating to Lemmy.zip.
Main instance rules apply, with the additional rules below:
- This community is intended for Lemmy.zip users only. Comments or posts from external instance users will be removed at the admin's discretion. Repeat offenders will be banned.
founded 1 year ago
MODERATORS
52
53
54
55
57
Now that Steamboat willy is public domain someone should edit it to make it out new Mascot.
58
Welcome to 2024! I hope everyone had a nice and enjoyable festive period, and I'm excited to see what the next 12 months holds for us all!
Here is the monthly update on everything happening with Lemmy.zip.
New Communities
Server Updates
So quite a lot has happened since the December Update including the release of Lemmy 0.19 and 0.19.1, troubles with Lemmy federation across the network, a change to our hardware, and some moderator-friendly additions to Zippybot.
- Lemmy 0.19 Release
On the 16th December we took the server briefly offline in order to apply the 0.19 update, which took about 20 minutes to do some database migrations and then all came back online quite smoothly. We also took the opportunity to update the server OS and all of the supporting services to their latest versions too, which all in all took about 2 hours, most of which was waiting for various backups to complete. We have updated Pictrs (the software which shows you the images!) to its latest non-beta version, but there is a new update coming for Pictrs which requires some downtime to allow for prep work for this update to take place (although core Lemmy will still work). We'll post more on this when it happens.
It was quickly discovered we had quite a big issue (well, technically two but we worked through the second one which was caused by docker-specific settings which I'm happy to expand on if anyone is interested) which was that suddenly our CPU was maxing out, our RAM was maxing out, and our site was grinding to a halt. Something has changed, most likely due to the new persistent federation queue, which is causing the database to continuously eat server resources. Here's a lovely screenshot showing how the server was maxing out during my "wtf is happening" phase that I posted to our matrix chat:
Due to this, I've finally had to admit defeat and move us off a VPS to a Dedicated Server (technically dedicated vCPUs) which has kind of sorted our performance issues, by delaying the restart of the database from every hour and half to once a day.
This graph shows the before and after of our RAM usage. The left side shows the RAM being maxed out by the database and then the system killing and restarting the database in a vicious cycle (with our measly 8gb of RAM and the right hand side shows our new RAM usage - its still going up, but with 16GBs the cycle only occurs once a day, and for a second or two at most. Unfortunately this is a bug I can't fix, and isn't replicated by all Instances apparently (although I have seen a fair few other admins mention it) so hopefully it can be fixed at some point.
There is also a bug in relation to federation which due to all the DB restarts we weren't actually facing until we had a bit more power, when suddenly the server stopped federating with lots of other instances. There is a really handy public tool where you can see all this info live - It's important to note that the failing instances numbers are fine, if you try to visit them you'll probably get an error. The dead instances are instances our server hasn't been able to communicate with in 3 days, and so has stopped trying. Feel free to check that list and if you think any shouldn't be on there, let me know and I can manually reset them.
The issue lies with the "lagging" instances - we suddenly saw a thousand instances on that list, including some of the biggest instances, which indicated an issue. Its normal to see one or two on there, but not a thousand. Thankfully, a database restart does fix this issue so I am manually checking this website once or twice a day and will restart it if necessary, but if you do notice any issues ping me a message and I can take a look. In the future I pan to automate this but have not yet fully recovered from the festivities!
Lemmy 0.19.1 was also released, fixing some bugs such as the "show read posts" option not working, which was quietly applied.
- ZippyBot Moderator updates
Work continues to build some form of serviceable "automod" for moderators, so Zippy now includes the functionality for community mods to schedule posts to their communities, either on a one-off or ongoing basis.
The command for this is #autopost followed by some qualifiers - you can get the full info by sending Zippy a message with the command #autoposthelp, which will send you back something like this:
Updates will continue to happen to this for added functionality and I'm always open to ideas.
Donations
This is probably a more important topic than it previously has been - due to the extreme generosity of some of our users, we've been in a financial position that has allowed us to not need to worry about funding too much. However, with the move from a VPS to a dedicated server, our costs have almost doubled from £16.36ish a month to £30 a month. This means, assuming nobody donated another penny, we'd run out of funds in less than 8 months. Our current level of donations is just short of this new monthly figure too, which does mean we will eventually run out of funds at some point too even if nothing changed with donations.
Therefore, my ask is that if you’ve been enjoying your time here, please consider a donation - it will help keep Lemmy.zip online and all donations go to paying for the server and for the surrounding services like image hosting and email.
Interaction reminder (help support the instance!)
If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)
I wanted to take a moment to encourage everyone to actively engage with our platform. Your participation can take many forms: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.
It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.
If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads.
Server Performance
Graph time!
CPU usage over last 30 days:
Disk usage over 30 days:
Network usage:
Image hosting stats: (over 200gb now!)
Cloudflare stats:
Deeper dive into those metrics:
And finally traffic stats from around the world:
As always, if there are any other metrics you'd like to see let me know!
Otherwise thats it for this month, thank you all for being here :)
Demigodrick
59
60
61
62
63
64
65
66
67
68
Hi All,
Hope everyone had a good November.
Here's the monthly update on what is happening at the moment on Lemmy.zip!
Here's some communities you might be interested in!
New Communities
Server Updates
At the moment we're in a bit of holding pattern pending the release the of Lemmy 0.19 - The latest dev post indicates that the release of 0.19 should be quite soon once they're able to fix the remaining bugs. Following this, we'll schedule some downtime so we can update the server.
Some of the main points of 0.19 are:
- per user Instance Blocking, i.e. the ability for a user to defederate a whole instance themselves.
- 2FA fix and reset. This will stop you being able to lock yourself out of your account when setting up 2FA, but if you do have it enabled currently it will reset and you'll need to redo the setup.
- user settings import/export between servers
You can see more information in this dev post on the changes coming.
In case you've missed it, we also shared the public "Federation Stats" dashboard which is linked in the side bar, or you can click here to go straight there - this is a live(ish) view of Lemmy.zip's federation status.
We've also added some small functions to ZippyBot around reporting posts/comments (working on messages) to acknowledge that the report has been reviewed. We're looking at expanding this (after 0.19 is released) to provide some feedback on the action taken/not taken etc, with the ethos of trying to be as transparent as possible.
We're always on the lookout for new ideas or things we can do better - let us know if there is anything you can think of!
Finally, next week Lemmy.zip turns 6 months old!
Interaction reminder (help support the instance!)
If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)
I wanted to take a moment to encourage everyone to actively engage with our platform. Your participation can take many forms: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.
Your involvement is vital in shaping and nurturing our Lemmy.zip instance. It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.
Together, we can make Lemmy.zip an even more vibrant and welcoming space for everyone. Thank you for being part of our community!
Donations
As always, we're extremely thankful to people who are kind enough and in a position to donate. You can always see the instance's financial position on OpenCollective, and what we're aiming towards. If you do donate, you'll get your name in the Thank You Thread.
If you've been enjoying your time here, please consider a donation - it will help keep Lemmy.zip online and all donations go to paying for the server and for the surrounding services like image hosting and email.
Want to help but not in a position to do so financially? Then please remember to interact with the instance! The more users who interact = the better Lemmy.zip becomes!
Server Performance
Time for some graphs, numbers, and stats!
CPU usage over the last 30 days:
Network stats over the last 30 days:
Some disk stats over the last 30 days:
Here's our current image hosting storage:
And here is the headline graph from Cloudflare (please note, all these graphs are still affected from Cloudflare's API downtime last month, so traffic is slightly higher in reality)
This is a more detailed look at the requests through Cloudflare:
This is a more detailed look at the bandwidth:
And this is a more detailed look at the unique visitors:
Finally, this is the traffic breakdown for the top countries over the last 30 days:
Finally, I'd just like to thank everyone for being here - without you, this project wouldn't be as amazing as it is!
As we head in to December I'm mindful that my next update will in 2024! So I would like to take this opportunity to wish everyone a joyful and peaceful festive period, and I hope you get to spend it with your family, friends, and the people you love.
Thank you,
Demigodrick
69
70
71
72
73
74
Hi All,
Hope everyone had a good October and enjoyed the spooky season!
Just dropping in again to give an overview of what is happening with the server and what we've been working on, along with the usual graphs and stuff 😁
Here's some communities you might be interested in!
- All Things Tom Scott - A community dedicated to the YouTuber, presenter, and media personality Tom Scott!
- Data Breaches - Information about data breaches, data leaks, ransomware attacks, and other related stories.
- Starfield - Discussion of the Bethesda game Starfield
- Game Deals - A comprehensive overview of all the games on sale (or free!) across the PC world.
New Communities
Since my last post, there have been a couple of new communities you might be interested in checking out:
Server Updates
So we've been working on a few things over the last couple of weeks, including revamping some of the "admin-run" communities (Gaming, Home and Announcements) with new icons (and a banner!).
We've also comprehensively updated our Legal documentation. Previously our Terms of Service was on the Legal tab, but we've found this hasn't been a good way to clearly display information or to add further information to. So we've introduced legal.lemmy.zip with all the legal documentation (i.e. cookie policy, terms of service, code of conduct) and this allows us to easily add more as we need, or update quickly and clearly. Please have a look and let us know what you think!
We've also been working on a public dashboard for server performance so we can continue to be transparent about how the server is run. A preview of what we've got working so far:
We're not quite ready to have this accessed publicly, but hopefully it won't be too long!
ZippyBot has also had some fairly substantial updates, including the ability to broadcast messages to all users, adding news articles to the gaming community, and the biggest of all is that Zippy has replaced the slur filter we used to have in place. Now, rather than remove a word on the slur list regardless of context (which stifled discussion and prevented users here from taking part in discussions on external instances) Zippy will report any comments or posts containing a word on the slur list, and the moderator of the community can make a decision on how to resolve that. This should be a much more hands off and flexible approach to handling this.
Interaction reminder (help support the instance!)
If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)
I wanted to take a moment to encourage everyone to actively engage with our platform. Your participation can take many forms: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.
Your involvement is vital in shaping and nurturing our Lemmy.zip instance. It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.
Together, we can make Lemmy.zip an even more vibrant and welcoming space for everyone. Thank you for being part of our community!
Donations
As always, we're extremely thankful to people who are kind enough and in a position to donate. You can always see the instance's financial position on OpenCollective, and what we're aiming towards. If you do donate, you'll get your name in the Thank You Thread.
We're currently in a stable financial position, with around 10 months of funding in the pot. This is a great position to be in, but our ask is that if you've been enjoying your time here, please consider a donation. We'd love to be able to save enough to switch to a dedicated server in the future which would improve performance and make the site snappier and more responsive.
Want to help but not in a position to do so financially? Then please remember to interact with the instance! The more users who interact = the better Lemmy.zip becomes!
Server Performance
Time for graphs and some stats!
First is the usual Hetzner CPU usage over the last 30 days:
And second is the Hetzner network stats for the last 30 days:
I'm also including a little sneak peak of the other dashboard we'll soon be sharing with real time performance stats:
Here's our image hosting stats:
And here are some Cloudflare stats. For those that aren't aware, Cloudflare had a bit of a whoopsie the past couple of days, hence why the graphs drop to 0 for a bit! Thankfully Lemmy.zip was fine and continued to work during this.
Overall stats over 30 days:
More detailed overview of traffic through Cloudflare over the last 30 days:
Traffic from regions over the last 30 days:
And finally unique visitor stats over the last 30 days:
Finally, I'd like to thank everyone for being here and helping to build Lemmy.zip into what it is. As always, if you have any questions please put them below or feel free to drop me a message.
Demigodrick
75