26
Home
467 readers
1 users here now
Lemmy.zip instance discussion.
For all things relating to Lemmy.zip.
Main instance rules apply, with the additional rules below:
- This community is intended for Lemmy.zip users only. Comments or posts from external instance users will be removed at the admin's discretion. Repeat offenders will be banned.
founded 1 year ago
MODERATORS
27
28
29
30
31
Hello all, welcome to the May server update!
Lets have a look back at what has happened over the last month, and what could be happening into the future.
New Communities
We've had a few new communities pop up, please check them out, interact with them, and give them a subscribe!
There are also some existing communities that are looking for mods. Fancy trying out being a mod with a pre-existing community? Now is your chance!
- Tesla (142 Subscribers)
- Playstation (547 Subscribers)
- Xbox (185 Subscribers)
- Apple (391 Subscribers)
- Star Trek (159 subscribers)
Server Updates
We've been on our new server over a month now, with amazing performance, and no downtime (outside of planned maintenance!).
This has allowed me more time to look at the secondary services and less time intervening in restarting a broken server.
To that end, I have started a collection of Mod Tools that are built into Lemmy.zip in this thread. Lemmy doesn't really come with any tools outside of the report button, so I've been trying to build the tools that I think help support mods but also make our communities friendly places to be. I think very early on there was a general consensus that moderation shouldn't be automated and I've tried to stick to that ethos wherever possible. There are no tools that take moderation actions without a community mod or an Admin stepping in to review first.
I will continue to add more tools as I am able to get them tested and running on the site - very soon I'll have the next tool up on there which allows community mods to set their own "words" for the bot to look out for - for example, you can set it to scan for the word "bread" and every time it finds that word in a new post or comment in that community, it will generate a report for the mod. This allows mods to set their own dictionary of words they don't want to appear in their communities. (We still have an instance-wide bad words list but that works differently now, and instead lets the admins know on Matrix when a bad word has been detected - these are the sorts of words that we absolutely won't allow and/or break the law.)
Luckily our application process helps to weed out most of the spammers and bad actors. We don't generally allow spam/throwaway email addresses (Although I do fully understand why people want to use them, so we do allow email forwarding addresses - I myself use Proton's email forwarding thing all the time!).
We read each and every application message and do reject those that we feel aren't genuine (the amount of applications that just say "lemmy in" is wild) - that said, we very much do appreciate those who take the time to write something funny or personal in them, and they do brighten our day!
Ps - thanks to those who have helped to test these tools out and iron out the bugs!
Over the next month I think we'll see Lemmy 0.19.4 released, bringing with it a bunch of new updates and fixes.
Interaction reminder (help support the instance!)
If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)
I just want to let everyone know that the easiest way you can support Lemmy.zip is to actively engage with content here: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.
It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.
If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads.
Please don't be a lurker! :)
Donations
Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip has now tipped over 50 euros a month ($54, £43) and is mostly funded by the community!
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
I am also looking at future fundraiser ideas (fundraiser committee anyone!) - if you have any ideas please let me know either in the comments, via PM, or on Matrix.
Graph time!
Due to a (now fixed) issue with Prometheus, I can't show you the full 30 days of the server performance. Instead, the CPU/RAM are over the last 7 days.
Instead, have a bonus image of the current server details as I type this:
Now to the usual stats:
CPU - Yes, we're not using very much at all.
RAM - I'm pretty happy with this, the performance is exactly where I want it to be.
Image hosting:
Cloudflare overview for the last 30 days:
Requests over 30 days:
Bandwidth over 30 days:
Unique visitors over 30 days:
Traffic: (I don't know whats happened in Singapore to drive up traffic - popular fediverse server maybe?) Finland is where Lemmy.world is hosted (along with Lemm.ee and Lemmy.zip!) which is why we see so much traffic from there.
So that's it for a relatively stable April - We'll see what happens in May!
As always if there is anything else you'd like to see I am happy to add it in, just let me know.
PS - next time I do one of these we'll be only a few days short of 1 year old!!!
Thanks
Demigodrick
32
33
34
35
36
Going to pin this for 24 hours so all mods hopefully get a chance to see it! Sorry!
As we continue to expand on the mod tools for Lemmy.zip community mods, I thought I'd better start keeping a collection of them in a post, what the tools do, and how they can be used.
All commands are invoked by sending a PM to ZippyBot with the relevant text commands.
Welcome Message
The Welcome Message feature allows a community mod to set a custom "Welcome message" for a community that gets sent to a user when they subscribe to a community.
Welcome Messages can be set up by sending the command #welcome -c community_name -m "welcome message" to ZippyBot. You'll receive a reply back confirming this has been successful.
Anyone that subscribes to your community after this will receive the PM you set up. The PM will be sent by a bot account called WelcomeBot specifically set up for this task.
Important notes
- The text of your welcome message must start and end with a speech/quote mark, i.e.
"welcome text". - You can use any punctuation/markdown you want, including emojis and links.
- You can change the message by sending the same command with updated text.
- You can delete the message entirely by using the command
#welcomedelete community_name.
Example
The full code for this looks like:
#welcome -c gaming -m “Welcome to the Lemmy.zip Gaming Community!
All content related to gaming on any platform is welcome here including links to news articles, discussions about the latest games, and reviews of games from across all genres.
Please remember to follow the rules in the sidebar 😊
***
*This is an automated message. Beep Boop.* "
Autopost/Scheduled Posts
The Autopost/Scheduled Post features allows mods to set either a one-off or recurring post in their community, which is then pinned. If the post is a recurring post, the old post is unpinned and the new post is pinned when it is triggered.
A typical command would look like #autopost -c community_name -t post_title -b post_body -d day -h time -f frequency
-c- This defines the name of the community you are setting this up for. This is the original name of the community when you created it.-t- This defines the title of the post. You can use the modifiers listed below here.-b- This is the body of your post. This field is optional, so you don't need to include it if you don't want a body to your post. You can use the modifiers listed below here too.-u- This defines a URL you can add to your post. This field is optional, so you don't need to include it.-d- This defines a date you want the first post to occur in YYYYMMDD format, i.e.20230612which would be 12th June 2023, or the day of the week you want your thread to be posted on, i.e.monday.-h- This defines the time of the day you want this thread to be posted, i.e.12:00. All times are UTC!-f- This defines how often your thread will be posted. The options that currently exist areonce,weekly,fortnightly,4weekly, ormonthly.
There are some modifiers you can use either in the title field (-t) or the body field (-b).
%d- This will be replaced by the day of the month, i.e.12%m- This will be replaced by the name of the month, i.e.June.%y- This will be replaced by the current year, i.e.2024%w- This will be replaced by the day of the week, i.e.Monday. For example, having-t Weekly Thread %d %mmight be created asWeekly Thread 12 Junedepending on the day it is posted.
Finally, if you want to delete a scheduled autopost, use the command #autopostdelete with the ID number of the autopost, i.e. #autopostdelete 1. You can also delete the latest pinned thread if you include y at the end, i.e #autopostdelete 1 y.
Important notes
- Using the
oncefrequency means your post will never be unpinned, you'll need to do this manually. - The post will be created and pinned by ZippyBot, so you won't receive replies to your inbox.
- Because ZippyBot is an Admin, you won't be able to delete its post - you can however make a reply to a post you want removed with
#deleteand ZippyBot will pick this up and remove its post.
Example
#autopost -c gaming -t What are you playing this week? %m %d %y Edition -b Hey there everybody! Weekly check in time once again. So… What are you playing this week? -d Monday -h 07:00 -f weekly
This creates the weekly thread in the Gaming community.
RSS Feeds
This allows a mod to add an RSS feed to their community, which ZippyBot will regularly check and create a new post if a new RSS post is detected.
A typical command would look like #rss -url rss_url -c community name - but there are a few modifiers you can use.
url- This is the URL of the rss feed, and is mandatory.c- This is the community name and is mandatory.t- This will tag your post titles with a preceding tag in square brackets, i.e.-t "RSS POST"will result in each post being tagged with[RSS POST]title_inc- Adding this will mean that only posts that INCLUDE the string you define will be posted, i.e.-title_inc "title must be included". The speech marks are mandatory if you use this option, and you can have multiple filters by using a commma between them.title_exc- Adding this will EXCLUDE any posts that match this string, i.e.-title_exc "dont include this". The speech marks are mandatory if you use this option.url_inc- Adding this will filter the post based on the link to the content in the RSS feed. You can use it in a way such as-url_inc "goodlink.com"to ensure that only posts where the link to the content is forgoodlink.com. Speech marks are mandatory.url_exc- Adding this will exclude content based on the link to the content RSS feed, such as-url_exc "badlink.com". Speech marks are mandatory.new_only- Adding this will mean that on the creation of this RSS feed, ZippyBot won't scan for existing posts and only start looking at posts after starting this feed.
Finally, if you want to delete an RSS feed from your community, use the command #rssdelete with the ID number of the RSS feed, i.e. #rssdelete 1.
Important notes
- ZippyBot checks for new RSS posts every 10 minutes
- ZippyBot makes the RSS post, so you won't receive a notification of a reply to a post.
- If you don't use the
-new_onlyflag, you'll get the last 3 posts on the first time this command is used.
Example
#rss -url https://rss-url.xml -c gaming
As I develop new tools or work on existing ones, I'll continue to add them here or update the commands.
We're always on the lookout for what we can add too, so if you have any ideas for new tools we're really open to hearing them and seeing if we can implement them.
Final important note - while all these tools work to some degree, I can't guarantee they're bug free. I may need to reset these at some point if a bug is discovered. Please let me know if anything doesn't work as expected!___
37
38
Hello all, and welcome to the April 2024 Server update.
What a busy month we've had! I hope you enjoyed the April fools IPO announcement. Obviously, we're not going public.
New Communities
We've had a few new communities pop up, please check them out and subscribe if you like them!
- Weird West
- Squared Circle - Welcome to all the Kbin refugees!
Server Updates
What. A. Month.
Following our previous server upgrade a little while back, we'd seen that performance was steadily decreasing and the site performance was slowing down too, sometimes taking seconds to load a page.
This is what I was seeing on the 27th March:
As you can see, all red = not good.
I first tried offloading all secondary services to a second server (like the mobile/old UIs and some of the admin tools we run) which unfortunately didn't have much of an impact overall - the server just wasn't able to handle all the traffic and database operations anymore.
For reference, the server was a 4 core (vcpu's - so 4 threads) Virtual dedicated server with 16gb of RAM (Hetzner CCX23). In some form this was still the original server we started Lemmy.zip on, just rescaled through Hetzner's platform. At this point, rescaling it up again just wasn't financially viable, as the performance to price benefit stopped being in favour of cloud servers, and now tipped over to proper dedicated servers.
So, after researching the best long-term solutions I settled on Hetzner's EX44 dedicated server. It has 12 cores (20 threads!) and 64GB of RAM.
It is a beast.
This is the comparison performance as I type this:
This should give us plenty of growing room as the fediverse continues to expand.
Other things that happened in March include ZippyBot properly breaking, and dealing with a large spam wave hitting the Lemmy network.
ZippyBot decided it just didn't want to work anymore, at which point I realised that due to my own poor programming I hadn't put enough detail in Zippy's logs, which look a bit like this:
We use Zippy a lot, like sending welcome messages to new users, background moderation stuff, stopping spam users etc. So having Zippy not working was not ideal.
The problem was that Zippy wasn't throwing any errors, it was just stopping and not restarting, seemingly randomly. During the initial debug phase, I implemented a docker healthcheck into Zippy (another new thing I had to learn how to do!) and used another container to keep restarting Zippy if the the healthcheck came back unhealthy.
Finally, it turned out that the RSS feature built into Zippy for community mods was failing when the URL it reached out to for the feed didn't exist anymore. One of the RSS feeds had changed their URL and not put a redirect in, so Zippy was hitting a 404 error and failing silently. There is a check when the RSS feed is set up to make sure the feed exists, but not one after that. Lesson learned.
The Lemmy network saw a large spam wave attack from the EvilToast instance, which was resulting in multiple accounts sending thousands of comments/posts a minute and overwhelming some instances. To get around this, I temporarily defederated from EvilToast for the period of the attack, until their admin confirmed they had implemented a captcha and stopped completely open signups. Instances that allow completely open signups are usually targeted by spammers due to the easy way they can create so much traffic. Hopefully new instances being created into the future will learn from this and enforce at least some method of validating their users.
As promised, here is a breakdown of the server move process. If you don't care about sysadmin stuff, you can safely skip this next bit.
Making the move to the new server was a new experience for me (If you weren't aware, I am not a sysadmin by trade and am learning as I go!) so the day of the move was a bit nerve wrecking with visions of accidentally losing the database forever. This is the rough order of steps I took during the maintenance window.
First, I backed up the old server using Hetzner's very handy cloud backup tool. That took about 10 minutes, after which I changed the nginx config to point away from lemmy itself. It was meant to point to the status site but i misconfigured the nginx file and instead it sent traffic to Authentik (our admin SSO solution to lock down our backend services). Oh well, this was the only thing that went wrong thankfully so I'll take that.
I stopped all the docker containers on both servers (main lemmy server and secondary UI server) and then compressed the postgres database files. While this was happening, I recreated the folder structure on the new server, and spun up Portainer, and then the Lemmy services stack so that the files/configs needed were generated and all in the right place, before shutting lemmy back down until everything was ready.
I then rsynced the database between both servers, and decompressed it in the correct place on the new server. All in all this took about an hour, mostly waiting for the files to compress/decompress. At this point I needed to change the DNS records to point to the new server, so I could request all the SSL certs for the required domains. This is when I took the old server offline and changed the DNS records in Cloudflare to point to the new IP.
Once I had requested the SSL certs, I then created the nginx config files for lemmy and some of the important subdomains (i.e. the mobile site) at which point everything was in place for lemmy to be turned back on. After a very tense couple of minutes (and a few container restarts!) lemmy.zip finally came back to life just as it had been left, and pretty quickly I could see federation was working as we started to pull new posts through.
Next up was fixing pictrs as images weren't loading - that service has a real issue with memory requirements and was stuck in a boot loop, so giving it access to 10 gigs of memory (a bit overkill admittedly) brought it back to life. And we were back!
The next hour or two was spent re-implementing the backend services, checking everything was working correctly, testing emails and images, and then finally deciding on and testing a backup solution.
As Backblaze has served us so well for images, our solution will back everything on the server up during the night (UTC) and then securely store it in a private Backblaze B2 bucket, with a full versioning solution for suitable daily, weekly, monthly and yearly backups. Good stuff. I do need to test this soon though to make sure it works on a restore. Will probably spin up a Hetzner cloud server and try it out.
I am happy to expand on this process if anyone is interested, just ask away in the comments.
Interaction reminder (help support the instance!)
If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)
I just want to let everyone know that the easiest way you can support Lemmy.zip is to actively engage with content here: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.
It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.
If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads.
Donations
Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip has now tipped over 50 euros a month ($54, £43) and that's not including the surrounding services (email, URL, hosting the seperate status server - which I am covering myself).
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
I am also looking at future fundraiser ideas (fundraiser committee anyone!) - if you have any ideas please let me know either in the comments, via PM, or on Matrix.
Graph time!
This was the monthly performance of the Old server before it was turned off:
You can see the demand really start to ramp up towards the end of the month.
This was the 24 hour performance on the 28th:
The hourly peaks were where the performance really dropped out for about 20 minutes, followed by 40 minutes of still pretty poor performance.
This is the performance of the new server of the last 24 hours:
Just a little bit of a difference :)
Our current image hosting stats:
This is our Cloudflare overview for the last month:
Cloudflare requests:
Cloudflare bandwidth:
Cloudflare unique visitors:
Cloudflare traffic:
And that's pretty much it. As always, if there's anything you'd like more information on, or you'd like to see included in these regularly, let me know.
Cheers,
Demigodrick
39
40
41
42
43
44
Hey all - hope everyone has enjoyed their February and the extra day!
Here's a roundup of everything that has happened on Lemmy.zip over February and into March.
New Communities
We've had even more communities pop up this month, please check them out and subscribe if you like them!
- Star Trek
- The Amazing Digital Circus (Personal recommendation - I had no idea what to expect watching the pilot and enjoyed it very much!)
MaddieTian has also kindly brought the Xbox community back to life - please give it a sub and support it!
(PS - if you're a community mod and would like feature your community here, let me know and I'll happily pop it up here in April)
We still have one modless community - Apple - if you're interested in taking this over let me know.
Server Updates
It has been a bit of quiet month since last time. There have been no Lemmy releases but (at least from my perspective) most of the federation issues we had previously have been fixed and things seem a lot smoother.
The performance changes we'd also made to the server have appeared to worked wonders (again, thanks to locuester for their support with the database) and we're sitting quite happy performance wise, with no major unplanned downtime.
Last time I discussed that we had doubled the image size from 1mb to 2mb per image, and throughout February we've doubled that again to 4mb. This hasn't really had any impact on performance or storage, and I'm happy to keep it at 4mb for now, which should allow for quite good images. They still get compressed when they're uploaded but it shouldn't have too much of an impact. Here's a 3mb image i did some testing with.
Our feedback survey closed and we've taken a look at the results/comments and we're really thankful for all your responses. Many of you were very kind about the site which always makes me smile. The feedback isn't something we can report back as %s of answers, but we have taken on board comments and suggestions. One of my key things is to rewrite the instance rules to be clearer and flow better, and also to separate out the moderator community conduct bit as its own section and expand a bit on that.
What was interesting to see though was that many people have gone "all in" on the fediverse and replaced the large tech company apps with fediverse alternatives.
Finally, ZippyBot has had a very small update to make the report confirmations less spammy by limiting who they go to and in what circumstances. I've also changed the way we deal with urgent reports by keyword filtering in the report, so if you include something like "CSAM" in the report the admins will get pinged on matrix so we're at least aware faster of an issue.
In Zippy's next update i'm hoping to include the ability for mods to add RSS feeds to their community (including filtering for certain words) but it is quite a large task so taking me and my limited python knowledge a bit longer than I'd like to get it done - but I'm almost there! If there are any other tools mods feel they'd benefit from then let me know - happy to build them into Zippy.
Interaction reminder (help support the instance!)
If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)
I just want to let everyone know that the easiest way you can support Lemmy.zip is to actively engage with content here: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.
It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.
If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads.
Donations
I hate banging on about money and I'm not very good at it, but here is the general reminder that Lemmy.zip only exists because of the generous donations of its users.
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
Therefore, my ask is that if you’ve been enjoying your time here, please consider a donation - it will help keep Lemmy.zip online and all donations go to paying for the server and for the surrounding services like image hosting and email. We have options for one-off donations of any amount, or recurring donations at set amounts.
You can see all the kind donators in the Thank You thread - you could get your name in there too!
Graph time
We've basically fully moved over to Grafana now for stats tracking, so here are some lovely graphs for you!, covering all of February.
Cpu:
Memory: (the dips are either server or database restarts)
Disk/storage:
Image storage:
Cloudflare overview:
Cloudflare requests:
Cloudflare bandwidth:
Cloudflare visitors:
Cloudflare traffic:
If there are any stats/graphs/insights i've not posted above but you'd like to see, let me know!
That's it for now - but as always if you have any questions pop them below or pm me.
Hope everyone has a lovely March and I'll be back again in April!
Cheers,
Demigodrick
45
46
47
48
49
50