With the release of Lemmy version 0.19.4 we'll look to upgrade ASAP (unless any issues pop up between now and then).

There is a DB upgrade so definitely going to be significant downtime during this window.

Please keep an eye on status.lemmy.zip during this time.

Hi All!

This month marks the 1st birthday of Lemmy.zip. Originally intended as a side project, I think I've ended up spending more time on this site than my actual job.

June 10th is the official anniversary, and so we'll be doing a small giveaway on this day to celebrate.

In the meantime, lets have a look at how things went in May.

---

Communities

This month, I want to focus on the existing abandoned communities we have that have the potential to really grow.

• Tesla (152 Subscribers)
• Playstation (570 Subscribers)
• Xbox (198 Subscribers)
• Apple (438 Subscribers)
• Star Trek (165 subscribers)

If you fancy becoming a mod of one of these communities, please let me know.

We'll also be doing some form of community tidying up over the next couple of weeks, so we'll be checking with mods if they want to keep unused communities or if we can remove them/put them back out to other mods.

---

Server Updates

Other than a small bit of downtime caused by me, the server has been stable as a rock this past month, with most of the work being carried out on keeping the various services we use up to date.

The extended downtime during the maintenance window (and extension of said maintenance window) was due to the backup process not correctly handling old snapshots. All the backups were safe and offsite, but a lock on the repo wasn't cleared and so the process the server normally follows where it cleans up old snapshots wasn't firing, causing extra storage space to be used up. In order to clear it, the server needed some time where files weren't being altered, so I temporarily took lemmy offline in order to let the tool do its thing, which lasted for about 40 minutes. Thankfully that shouldn't happen again, and wasn't technically the end of the world either as we were still getting daily offsite backups, but was very inconvenient!

We're waiting a lot on the next version of Lemmy to be released (0.19.4) which should hopefully be soon. In the meantime, I've been going back over Zippybot's code, tidying up various bits and pieces and trying out different things we can add to make Lemmy a better overall experience. For example, we now send a rejection message if we decline an account, unlike the default experience which is just to ghost the user. Thankfully we don't need to do this much, but hopefully it guides genuine new users to get in touch if they feel we've made a mistake (which we might do, we're only human!)

We're also exploring making bans more transparent. We really don't issue many bans to our own users at all, as you can see from the mod log. When we do though, there isn't really a way for that person to know why they were banned if they're not aware of the log. So we want to use the same technology as we use for sending application rejections and also send emails to let someone know why and for how long they were banned, and most importantly how to appeal. Again, we really don't issue that many bans to our users, but I feel people need a clear and simple way to get in touch if they don't think we've been fair.

We have also put up some new front ends that people can use if they fancy trying something new:

• t.lemmy.zip - Tesseract - This is a fork of Photon (which is our current mobile site) except that it has quite a few extra features built in. Take a look if you're primarily on mobile!
• a.lemmy.zip - Alexandrite - An alternative Lemmy desktop experience.

If there are any other front ends you'd like to see, let us know!

Interaction reminder (help support the instance!)

If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)

I am sure that long term readers are bored of reading this now, but it is really is true that the easiest way you can support Lemmy.zip is to actively engage with content here: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.

It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.

If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads.

Please don't be a lurker! :)

---

Donations

Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 50 euros a month ($54, £43) and is mostly funded by the community!

We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.

If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.

We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!

I am also looking at future fundraiser ideas (fundraiser committee anyone!) - if you have any ideas please let me know either in the comments, via PM, or on Matrix.

---

Graph time!

Here are some graphs from the server and from cloudflare etc. This paints a picture of server usage and traffic to Lemmy.zip.

CPU Usage over the last 30 days:

RAM usage over 30 days: (PS this is perfectly fine, the drop is where the server was offline during the update, the fluctuation is normal)

Image storage:

Cloudflare overview:

Traffic/requests:

Bandwidth:

Unique visitors: (this is an increase on last month)

Global traffic:

Another relatively stable month (other than a slightly funky maintenance window) - maybe this will be the norm now!

Remember to check back in on the 10th June to celebrate our 1st birthday :)

Cheers,

Demigodrick

After much unexpected delay, the updates to the server have been completed. Apologies for the downtime!

---

Important edit: This is still ongoing, so the window is extended to 13:30 UTC. Sorry for the downtime. Please expect more downtime.

---

Time to update the server again. While I don't expect loads of downtime, there will be a 1 hour window where things might be offline. Please keep an eye on status.lemmy.zip if you're likely to be browsing the site during this time.

This community is looking for a mod!

If you think that would be something you'd like to try (you don't need to be a lemmy.zip user either!) drop me a message or let me know below.

Hello all, welcome to the May server update!

Lets have a look back at what has happened over the last month, and what could be happening into the future.

---

New Communities

We've had a few new communities pop up, please check them out, interact with them, and give them a subscribe!

• Remnant
• Legal News
• Webgames
• Civil Aviation

There are also some existing communities that are looking for mods. Fancy trying out being a mod with a pre-existing community? Now is your chance!

• Tesla (142 Subscribers)
• Playstation (547 Subscribers)
• Xbox (185 Subscribers)
• Apple (391 Subscribers)
• Star Trek (159 subscribers)

---

Server Updates

We've been on our new server over a month now, with amazing performance, and no downtime (outside of planned maintenance!).

This has allowed me more time to look at the secondary services and less time intervening in restarting a broken server.

To that end, I have started a collection of Mod Tools that are built into Lemmy.zip in this thread. Lemmy doesn't really come with any tools outside of the report button, so I've been trying to build the tools that I think help support mods but also make our communities friendly places to be. I think very early on there was a general consensus that moderation shouldn't be automated and I've tried to stick to that ethos wherever possible. There are no tools that take moderation actions without a community mod or an Admin stepping in to review first.

I will continue to add more tools as I am able to get them tested and running on the site - very soon I'll have the next tool up on there which allows community mods to set their own "words" for the bot to look out for - for example, you can set it to scan for the word "bread" and every time it finds that word in a new post or comment in that community, it will generate a report for the mod. This allows mods to set their own dictionary of words they don't want to appear in their communities. (We still have an instance-wide bad words list but that works differently now, and instead lets the admins know on Matrix when a bad word has been detected - these are the sorts of words that we absolutely won't allow and/or break the law.)

Luckily our application process helps to weed out most of the spammers and bad actors. We don't generally allow spam/throwaway email addresses (Although I do fully understand why people want to use them, so we do allow email forwarding addresses - I myself use Proton's email forwarding thing all the time!).

We read each and every application message and do reject those that we feel aren't genuine (the amount of applications that just say "lemmy in" is wild) - that said, we very much do appreciate those who take the time to write something funny or personal in them, and they do brighten our day!

Ps - thanks to those who have helped to test these tools out and iron out the bugs!

Over the next month I think we'll see Lemmy 0.19.4 released, bringing with it a bunch of new updates and fixes.

---

Interaction reminder (help support the instance!)

If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)

I just want to let everyone know that the easiest way you can support Lemmy.zip is to actively engage with content here: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.

It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.

If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads.

Please don't be a lurker! :)

---

Donations

Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip has now tipped over 50 euros a month ($54, £43) and is mostly funded by the community!

We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.

If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.

We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!

I am also looking at future fundraiser ideas (fundraiser committee anyone!) - if you have any ideas please let me know either in the comments, via PM, or on Matrix.

---

Graph time!

Due to a (now fixed) issue with Prometheus, I can't show you the full 30 days of the server performance. Instead, the CPU/RAM are over the last 7 days.

Instead, have a bonus image of the current server details as I type this:

Now to the usual stats:

CPU - Yes, we're not using very much at all.

RAM - I'm pretty happy with this, the performance is exactly where I want it to be.

Image hosting:

Cloudflare overview for the last 30 days:

Requests over 30 days:

Bandwidth over 30 days:

Unique visitors over 30 days:

Traffic: (I don't know whats happened in Singapore to drive up traffic - popular fediverse server maybe?) Finland is where Lemmy.world is hosted (along with Lemm.ee and Lemmy.zip!) which is why we see so much traffic from there.

So that's it for a relatively stable April - We'll see what happens in May!

As always if there is anything else you'd like to see I am happy to add it in, just let me know.

PS - next time I do one of these we'll be only a few days short of 1 year old!!!

Thanks

Demigodrick

Some updates to the server to be applied, and may involve a quick server restart so you may lose to connection to the site if browsing during this time.

Going to pin this for 24 hours so all mods hopefully get a chance to see it! Sorry!

As we continue to expand on the mod tools for Lemmy.zip community mods, I thought I'd better start keeping a collection of them in a post, what the tools do, and how they can be used.

All commands are invoked by sending a PM to ZippyBot with the relevant text commands.

---

Welcome Message

The Welcome Message feature allows a community mod to set a custom "Welcome message" for a community that gets sent to a user when they subscribe to a community.

Welcome Messages can be set up by sending the command #welcome -c community_name -m "welcome message" to ZippyBot. You'll receive a reply back confirming this has been successful.

Anyone that subscribes to your community after this will receive the PM you set up. The PM will be sent by a bot account called WelcomeBot specifically set up for this task.

Important notes

• The text of your welcome message must start and end with a speech/quote mark, i.e. "welcome text".
• You can use any punctuation/markdown you want, including emojis and links.
• You can change the message by sending the same command with updated text.
• You can delete the message entirely by using the command #welcomedelete community_name.

Example

The full code for this looks like:

#welcome -c gaming -m “Welcome to the Lemmy.zip Gaming Community! 

All content related to gaming on any platform is welcome here including links to news articles, discussions about the latest games, and reviews of games from across all genres.

Please remember to follow the rules in the sidebar 😊 

***

*This is an automated message. Beep Boop.* " 

---

Autopost/Scheduled Posts

The Autopost/Scheduled Post features allows mods to set either a one-off or recurring post in their community, which is then pinned. If the post is a recurring post, the old post is unpinned and the new post is pinned when it is triggered.

A typical command would look like #autopost -c community_name -t post_title -b post_body -d day -h time -f frequency

• -c - This defines the name of the community you are setting this up for. This is the original name of the community when you created it.
• -t - This defines the title of the post. You can use the modifiers listed below here.
• -b - This is the body of your post. This field is optional, so you don't need to include it if you don't want a body to your post. You can use the modifiers listed below here too.
• -u - This defines a URL you can add to your post. This field is optional, so you don't need to include it.
• -d - This defines a date you want the first post to occur in YYYYMMDD format, i.e. 20230612 which would be 12th June 2023, or the day of the week you want your thread to be posted on, i.e. monday.
• -h - This defines the time of the day you want this thread to be posted, i.e. 12:00. All times are UTC!
• -f - This defines how often your thread will be posted. The options that currently exist are once, weekly, fortnightly, 4weekly, or monthly.

There are some modifiers you can use either in the title field (-t) or the body field (-b).

• %d - This will be replaced by the day of the month, i.e. 12
• %m - This will be replaced by the name of the month, i.e. June.
• %y - This will be replaced by the current year, i.e. 2024
• %w - This will be replaced by the day of the week, i.e. Monday. For example, having -t Weekly Thread %d %m might be created as Weekly Thread 12 June depending on the day it is posted.

Finally, if you want to delete a scheduled autopost, use the command #autopostdelete with the ID number of the autopost, i.e. #autopostdelete 1. You can also delete the latest pinned thread if you include y at the end, i.e #autopostdelete 1 y.

Important notes

• Using the once frequency means your post will never be unpinned, you'll need to do this manually.
• The post will be created and pinned by ZippyBot, so you won't receive replies to your inbox.
• Because ZippyBot is an Admin, you won't be able to delete its post - you can however make a reply to a post you want removed with #delete and ZippyBot will pick this up and remove its post.

Example

#autopost -c gaming -t What are you playing this week? %m %d %y Edition -b Hey there everybody! Weekly check in time once again. So… What are you playing this week? -d Monday -h 07:00 -f weekly

This creates the weekly thread in the Gaming community.

---

RSS Feeds

This allows a mod to add an RSS feed to their community, which ZippyBot will regularly check and create a new post if a new RSS post is detected.

A typical command would look like #rss -url rss_url -c community name - but there are a few modifiers you can use.

• url - This is the URL of the rss feed, and is mandatory.
• c - This is the community name and is mandatory.
• t - This will tag your post titles with a preceding tag in square brackets, i.e. -t "RSS POST" will result in each post being tagged with [RSS POST]
• title_inc - Adding this will mean that only posts that INCLUDE the string you define will be posted, i.e. -title_inc "title must be included". The speech marks are mandatory if you use this option, and you can have multiple filters by using a commma between them.
• title_exc - Adding this will EXCLUDE any posts that match this string, i.e. -title_exc "dont include this". The speech marks are mandatory if you use this option.
• url_inc - Adding this will filter the post based on the link to the content in the RSS feed. You can use it in a way such as -url_inc "goodlink.com" to ensure that only posts where the link to the content is for goodlink.com. Speech marks are mandatory.
• url_exc - Adding this will exclude content based on the link to the content RSS feed, such as -url_exc "badlink.com". Speech marks are mandatory.
• new_only - Adding this will mean that on the creation of this RSS feed, ZippyBot won't scan for existing posts and only start looking at posts after starting this feed.

Finally, if you want to delete an RSS feed from your community, use the command #rssdelete with the ID number of the RSS feed, i.e. #rssdelete 1.

Important notes

• ZippyBot checks for new RSS posts every 10 minutes
• ZippyBot makes the RSS post, so you won't receive a notification of a reply to a post.
• If you don't use the -new_only flag, you'll get the last 3 posts on the first time this command is used.

Example

#rss -url https://rss-url.xml -c gaming

---

As I develop new tools or work on existing ones, I'll continue to add them here or update the commands.

We're always on the lookout for what we can add too, so if you have any ideas for new tools we're really open to hearing them and seeing if we can implement them.

Final important note - while all these tools work to some degree, I can't guarantee they're bug free. I may need to reset these at some point if a bug is discovered. Please let me know if anything doesn't work as expected!___

Hello all, and welcome to the April 2024 Server update.

What a busy month we've had! I hope you enjoyed the April fools IPO announcement. Obviously, we're not going public.

---

New Communities

We've had a few new communities pop up, please check them out and subscribe if you like them!

• Weird West
• Squared Circle - Welcome to all the Kbin refugees!

---

Server Updates

What. A. Month.

Following our previous server upgrade a little while back, we'd seen that performance was steadily decreasing and the site performance was slowing down too, sometimes taking seconds to load a page.

This is what I was seeing on the 27th March:

As you can see, all red = not good.

I first tried offloading all secondary services to a second server (like the mobile/old UIs and some of the admin tools we run) which unfortunately didn't have much of an impact overall - the server just wasn't able to handle all the traffic and database operations anymore.

For reference, the server was a 4 core (vcpu's - so 4 threads) Virtual dedicated server with 16gb of RAM (Hetzner CCX23). In some form this was still the original server we started Lemmy.zip on, just rescaled through Hetzner's platform. At this point, rescaling it up again just wasn't financially viable, as the performance to price benefit stopped being in favour of cloud servers, and now tipped over to proper dedicated servers.

So, after researching the best long-term solutions I settled on Hetzner's EX44 dedicated server. It has 12 cores (20 threads!) and 64GB of RAM.

It is a beast.

This is the comparison performance as I type this:

This should give us plenty of growing room as the fediverse continues to expand.

Other things that happened in March include ZippyBot properly breaking, and dealing with a large spam wave hitting the Lemmy network.

ZippyBot decided it just didn't want to work anymore, at which point I realised that due to my own poor programming I hadn't put enough detail in Zippy's logs, which look a bit like this:

We use Zippy a lot, like sending welcome messages to new users, background moderation stuff, stopping spam users etc. So having Zippy not working was not ideal.

The problem was that Zippy wasn't throwing any errors, it was just stopping and not restarting, seemingly randomly. During the initial debug phase, I implemented a docker healthcheck into Zippy (another new thing I had to learn how to do!) and used another container to keep restarting Zippy if the the healthcheck came back unhealthy.

Finally, it turned out that the RSS feature built into Zippy for community mods was failing when the URL it reached out to for the feed didn't exist anymore. One of the RSS feeds had changed their URL and not put a redirect in, so Zippy was hitting a 404 error and failing silently. There is a check when the RSS feed is set up to make sure the feed exists, but not one after that. Lesson learned.

The Lemmy network saw a large spam wave attack from the EvilToast instance, which was resulting in multiple accounts sending thousands of comments/posts a minute and overwhelming some instances. To get around this, I temporarily defederated from EvilToast for the period of the attack, until their admin confirmed they had implemented a captcha and stopped completely open signups. Instances that allow completely open signups are usually targeted by spammers due to the easy way they can create so much traffic. Hopefully new instances being created into the future will learn from this and enforce at least some method of validating their users.

---

As promised, here is a breakdown of the server move process. If you don't care about sysadmin stuff, you can safely skip this next bit.

Making the move to the new server was a new experience for me (If you weren't aware, I am not a sysadmin by trade and am learning as I go!) so the day of the move was a bit nerve wrecking with visions of accidentally losing the database forever. This is the rough order of steps I took during the maintenance window.

First, I backed up the old server using Hetzner's very handy cloud backup tool. That took about 10 minutes, after which I changed the nginx config to point away from lemmy itself. It was meant to point to the status site but i misconfigured the nginx file and instead it sent traffic to Authentik (our admin SSO solution to lock down our backend services). Oh well, this was the only thing that went wrong thankfully so I'll take that.

I stopped all the docker containers on both servers (main lemmy server and secondary UI server) and then compressed the postgres database files. While this was happening, I recreated the folder structure on the new server, and spun up Portainer, and then the Lemmy services stack so that the files/configs needed were generated and all in the right place, before shutting lemmy back down until everything was ready.

I then rsynced the database between both servers, and decompressed it in the correct place on the new server. All in all this took about an hour, mostly waiting for the files to compress/decompress. At this point I needed to change the DNS records to point to the new server, so I could request all the SSL certs for the required domains. This is when I took the old server offline and changed the DNS records in Cloudflare to point to the new IP.

Once I had requested the SSL certs, I then created the nginx config files for lemmy and some of the important subdomains (i.e. the mobile site) at which point everything was in place for lemmy to be turned back on. After a very tense couple of minutes (and a few container restarts!) lemmy.zip finally came back to life just as it had been left, and pretty quickly I could see federation was working as we started to pull new posts through.

Next up was fixing pictrs as images weren't loading - that service has a real issue with memory requirements and was stuck in a boot loop, so giving it access to 10 gigs of memory (a bit overkill admittedly) brought it back to life. And we were back!

The next hour or two was spent re-implementing the backend services, checking everything was working correctly, testing emails and images, and then finally deciding on and testing a backup solution.

As Backblaze has served us so well for images, our solution will back everything on the server up during the night (UTC) and then securely store it in a private Backblaze B2 bucket, with a full versioning solution for suitable daily, weekly, monthly and yearly backups. Good stuff. I do need to test this soon though to make sure it works on a restore. Will probably spin up a Hetzner cloud server and try it out.

I am happy to expand on this process if anyone is interested, just ask away in the comments.

---

Interaction reminder (help support the instance!)

If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)

I just want to let everyone know that the easiest way you can support Lemmy.zip is to actively engage with content here: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.

It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.

If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads.

---

Donations

Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip has now tipped over 50 euros a month ($54, £43) and that's not including the surrounding services (email, URL, hosting the seperate status server - which I am covering myself).

We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.

If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.

We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!

I am also looking at future fundraiser ideas (fundraiser committee anyone!) - if you have any ideas please let me know either in the comments, via PM, or on Matrix.

---

Graph time!

This was the monthly performance of the Old server before it was turned off:

You can see the demand really start to ramp up towards the end of the month.

This was the 24 hour performance on the 28th:

The hourly peaks were where the performance really dropped out for about 20 minutes, followed by 40 minutes of still pretty poor performance.

This is the performance of the new server of the last 24 hours:

Just a little bit of a difference :)

Our current image hosting stats:

This is our Cloudflare overview for the last month:

Cloudflare requests:

Cloudflare bandwidth:

Cloudflare unique visitors:

Cloudflare traffic:

And that's pretty much it. As always, if there's anything you'd like more information on, or you'd like to see included in these regularly, let me know.

Cheers,

Demigodrick