Home

Appears to have been quite smooth - please let me know of any issues. We'll be monitoring performance for the next hour.

Hello all - small request if you have a couple of minutes. Please could you send ZippyBot a message with just #score

Im testing the bots capacity to do multiple things at once, so the more it gets spammed the better! I'll leave this message up for 10/15 minutes or until the bot breaks.

If you get no reply, RIP ZippyBot

Edit: going to leave it running for a while, see how it gets on.

I suppose your score would be closest to reddit "karma"

Hi guys, I began to follow the communities I've been following since always from lemmy.ml but noticed that some of them appears without users nor even posts, when in ml instance is complete. Why is happening that?

I'm trying to access [email protected] but it just takes me to a page with no posts

Personally I think Facebook should face as much pushback as possible, unless we want them to do to this what Google did to XMPP.

Hey there, this is the weekly thread for all general topics and any questions. You can also join us on Matrix for a chat!

Hey all, its that time of the week again with another look at the back end services keeping Lemmy.zip running and what changes we're looking at implmenting.

If you've not read them yet, the previous parts are here:

• Part 1
• Part 2

TL;DR

• Matrix Chat
• Automod Bot
• Please remember to interact with Lemmy.zip
• Funding update

Matrix Chat

We've decided to put together a chat room for Lemmy.zip users to come and interact with each other (if they want!). There are various channels there, including a support channel. If there are any issues with Lemmy.zip, a bot over there will automatically notify the channel.

Come and check it out!

Automod Bot

Sami and I have spent the last week trying very hard to learn Python and put some form of automod bot in place. Currently, we've managed to get a bot together that will post various weekly chat threads (which should appear tomorrow for the first time!). The bot is @[email protected]. For those interested, he's using the Pythorhead python library to interact with Lemmy.zip.

Are you a Python dev? Fancy helping out with the bot? Please send me a PM or join the Matrix chat and let us know!

We're currently nowhere near having it be in place for moderation purposes, however my philosophy around the bot will be that it will never take direct action against a user. Rather, we're looking for ways for the moderation team to be able to be notified of anything that may break the instance rules, new accounts that begin post spam, that sort of thing. Then a human will have to read and action that report.

New Users & Interaction

Sami has put together a fantastic post for new users. This is just a general reminder for everyone to please interact with Lemmy.zip. Even a comment or an upvote really helps the instance out. Lemmy counts these interactions for each user and this is used to rank Lemmy instances. The more interactions = the more active this instance = better community.

Funding update

We've had some very generous support from Lemmy.zip users. Thank you all who have donated or are thinking about it. We've now covered the rest of the year in server costs for both the core server and the image server (as things stand!).

We'd like to be able look at upgrading to a dedicated server for more performance power, but with great power comes great ~~responsibility~~ cost. If you've been enjoying Lemmy.zip, please consider donating.

Server stats

I'll start by covering the infrastructure in a bit more detail now we've settled down a bit.

We have the "Core" lemmy server, hosted on Hetzner Cloud. More detail on the Server specifics is in Part 1.

We also have a separate "Status" server that is hosting the status page. This is a really cheap, pretty weak VPS running Uptime-Kuma via docker.

All emails are done via Brevo (formerly Sendinblue for those that have used them before). We're still within the free tier plan (300 emails a day) but it has come close a few times 😅.

Images are hosted on Backblaze via their B2 Cloud Storage Buckets.

Cloudflare is used as a CDN.

The domain is with Google registrar but I'll be looking at moving this ASAP to Porkbun given that the domain service has been sold to Squarespace.

Pretty graph time

Here is a graph showing CPU usage over the last 7 days. You can kind of see how usage has increased over the last 3 days in line with people fleeing from Reddit.

This shows CPU usage over the last 24 hours:

These are email stats over the last 7 days:

And this is the current usage of the Image storage:

This is the overview of Cloudflare usage over the last 7 days:

This is traffic across countries over the last 24 hours Interestingly, Finland has overtaken USA over the last 24 hours, however over the last week USA has smashed it with over 900k requests, and France in second place with almost 500k requests!

Hope everyone has found this interesting, I will look to try and do these weekly where possible.

Finally, welcome to all the new users! Any questions or anything extra you'd like info on, please let me know.

Thanks Demigodrick

Hey everyone. There's been an influx of users over the last 2 days as a certain website has restricted access to their API forcing apps to finally shut down.

TLDR: Welcome! You can search for communities across all instances with tools like this and this or create a community for your interests right here. For mobile apps: here is a list of choices. Don't worry about not having everything down pat and don't be shy about participating here and elsewhere.

If you're new to Lemmy it may be a little confusing at first but that shouldn't deter you from participating.

Firstly, take a moment to read Demigodrick's post if you haven't yet and are confused about how federation works in general or about how to search for communities outside of this particular instance.

On our instance, anyone can create a Community and become its moderator by default. It takes very little effort to create but requires a little bit of commitment to looking over it. You can always choose to remove yourself as moderator (and ideally hand it off to someone else) or delete it completely if needed.

There are useful tools for searching for existing communities such as: https://lemmyverse.net/communities and https://browse.feddit.de/

While there is no problem with having 'redundant' communities across instances as each can host its own discussions, it may be a good idea to search for a community before creating it as you may find a group that has already done some of the legwork for you.

Right now we have a couple niche game and tech related communities that are steadily growing but don't feel restricted to that. Any topic is fine as long as it adheres to the general instance rules.

Most people lurk but you'll find that the general vibe is fairly positive and accepting on Lemmy so don't hesitate to comment or post even if it's not something you usually do because the barrier to entry for content is fairly low as places grow (as long as it's not extremely low effort/spam).

As for browsing, do know that you can block communities that you do not want to see to have them drop out of the 'All' feed (no option to block entire instances on a user-level at the moment but hopefully that comes soon. It currently has support so it's just a matter of someone implementing it).

I personally like to browse All and Subscribed and sort by New (you can set your defaults in your profile settings). All is a useful way to find stuff to subscribe to. When you search for and subscribe to a 'foreign' community, you're automatically adding it to the list for 'All' for all of Lemmy.zip

For mobile browsing, there are several apps with the "official" one being Jerboa (maintained by the Lemmy devs). There are also quite a few that are being developed right now such as Sync for Lemmy.

You can find a list here: https://kbin.social/m/kbinMeta/t/71764 as mentioned by Adanisi in a comment below.

And most importantly don't forget to enjoy yourself

After all, don't many platforms (or heck, browsers) render .zip URLs nonfunctional due to security concerns? Meaning our community.lemmy.zip links may not automatically hyperlink when we want them to, or it may trigger security risk popups. Whoops. Feel free to correct me.

I've browsed that "all" page in communities and I can subscribe from there, but what about when I land on a specific federated community page such as https://lemmy.zip/c/[email protected] ? It has no "Subscribe" button

I'd love to propose something like aboves picture as new instance icon - to make it more "lemmy-like" Maybe not my exact picture, which I put quickly together in photoshop, but maybe someone could smooth it out or make something similar to this.

Hey all, it has been a pretty busy week this week with all things Lemmy.zip and I'm excited to tell you all about it! If you haven't read it yet, part 1 is here.

Monday
Monday morning was quite quiet. At the end of part 1 last week, i mentioned emails had gone down and we weren't able to send emails out. As a result, we've permanently moved email provider now to Brevo, which while the graph isn't so pretty, they seem to work fine as a provider.

I suddenly noticed that the user count for the instance had shot up by 20 people. Refreshed again, another 20 people had signed up. This didn't seem organic, so I jumped on to Brevo and noticed that all the new email registrations were bouncing, which means the emails didn't exist. I turned registration off for the instance, until I could put in another layer of security with manual applications. Registration was re-enabled, and suddenly the bots were no longer able to join. Because email verification has always been required, these bots will never be able to post spam to the instance.

Here is a lovely graph of emails for you:

The red spike shows the bounce rate, which correlates with the spike in sent emails.

It is around this time that I noticed the server storage is filling up quickly.

Tuesday
I investigated the storage issue, and discovered that images are caching from other servers too, so we're on track to exceed our server storage pretty quickly.

After some lengthy googling and reading migration processes, I finally have chosen an image storage provider (Backblaze B2 buckets) and thankfully Pictrs, the software that handles the images, has a release-candidate update that allows for image migration to an external host.

The first step was to upgrade the database to prepare for external migration, push the existing storage to the new host, then update the docker container environment to point to the new storage location. All is well I think, I can see the storage is starting to be used, but suddenly all the images stop appearing.

Somehow (and I still don't know how) the database upgrade and original migration to the external host had failed, and instead we were pushing new images to the external host, but existing images were cut off from the database and had never been transferred over.

Thankfully, the Pictrs software dev is a pretty chill person who very kindly bug fixed it and pushed a new release candidate container which fixed this issue (I re-ran migration and the new version allowed skipping files which had already been uploaded to the new storage host so there was no conflict and the database would merge nicely).

This took quite a while, so images were not showing for about 12 hours I think. Sorry 😔

This is the file and storage stats in our images database as of today. Thankfully they're fully separated from the main server now, where storage is hell of a lot cheaper than on our main cloud host.

There is quite a small cost attached to this though. In a week its cost me $0.12 so not masses. Backblaze allows setting caps on usage. Unfortunately I can't show you what the full usage compared to the cap is, because you can't go back in time apparently, but this is what the caps screen looks like:

I have set the caps quite high in comparison what we actually use, as far as I can tell, but the last thing I want is storage capping out and everything breaking again!

On this day, I also appointed Sami as an admin for Lemmy.zip. They've been great, helped me to keep on top of applications to the server, and also helped to create better logos for the instance communities. They also did all the leg work for the slur filter, which I will touch on later, plus has been keeping on top of the bot situation for us and making sure we're safe.

If you haven't already you can pop in here to say hi to Sami :)

Wednesday
First thing Wednesday was the creation of the "Home" community. Many other instances have "Main" or similar, but the purpose is the same, to have a central discussion point for the server. It is open to everyone to post in, including questions or issues with the server, or just to say hi. I'll also put posts like this one in here, and save Announcements for bigger actual announcements or emergency things.

Also Wednesday, in an effort to speed up the site but also to reduce bandwidth costs from the the new image server, I set up Cloudflare for the instance. The initial change of DNS seemed fine on my part, I don't think there were any issues but as with all things DNS sometimes things do go wrong until it has propagated, so apologies if the site was down for you during this time.

What confused me massively was that suddenly, all the images went down again. Cue mild panic attack as I couldn't work out why this was happening, until the very helpful Pictrs dev looked at my logs and pointed out that the server was returning a 500 code, which meant it had timed out. Thankfully (for me) it turned out Backblaze had gone down and they'd just been really slow at updating their status page. Images kicked back in a few hours later once they'd resolved their issue.

Thursday
Since then, Cloudflare has been working pretty well. Here are some pretty graphs for you:

This is cloudflare usage over the last 24 hours - the drop is related to the 0.18 upgrade.

Requests through cloudflare since enabling it

Bandwidth usage & savings via caching

Unique visitors to the site.

Traffic per region over the last 24 hours (US has pulled ahead again, France almost overtook them!)

If there are any other stats you're interested in seeing let me know.

Friday
On Friday, Sami had done all the hard work to get the slur filter in place. We both feel that the normal swear words, i.e. fuck, are fine on this instance, but that racism & bigotry isn't something we'll tolerate. It will only apply to posts on this instance (obviously we don't control what other instances do) and we'll monitor the usage of said words and enforce the rules where needed.

Also, Sami noticed 0.18 had been officially released, so I went about preparing to upgrade the server (we use the ansible script with some tweaks, so I had to move them over to the new config first).

However on installation I was met with a server error and for about an hour the site was down while I troubleshot the issue. It was a global issue with the ansible script, which had changed the docker containers to use an internal and external network, however this stopped the pictrs container and lemmy-ui container from accessing the internet, which broke the connectivity of the site. There was also a bug with site icons that had to be deleted via the database. You can see the troubleshooting steps here

With the fix in place site came back up. I cleared the cloudflare cache for good measure, and thankfully we've pretty stable on 0.18 since.

To check this, I've added a status page on an external site so if you ever can't access the site, you can check if its working at that link.

Here are some more lovely graphs for you!

CPU usage over the last 24 hours - hovering around 50% but seeing more spikes.

Network traffic over the last 24 hours - given we're offloading photos to an external server I expected this to be higher than last time.

Used storage space on the server

Funding
Finally, I need to approach a topic I've been hesitant to approach since Day 1. There are a couple of improvements I would like to make to Lemmy.zip, but these have a resource implication and I can't guarantee funding out of my own pocket if these were to be implemented. I'd like to take regular backups of the server in the most painless way possible, but this comes with a 20% additional cost to the server. I'd like to look at switching to a dedicated server for better performance, but the costs here far exceed the cost of running on a VPS.

So, to be as transparent and open as possible, I've set up a funding page on a website dedicated to being open and transparent.

https://opencollective.com/lemmyzip

This it totally 100% optional, and the only way I have to say thank you is to created a dedicated thread somewhere and make sure your name goes it in (assuming you want it to).

The open collective site allows me to be as transparent as possible by adding expenses to the site, so you can exactly what any raised money is going on (it will only ever be spent on the site, that is a promise).

I hope this is OK with the community, as I genuinely feel torn about it. I can only promise I'm in with Lemmy.zip for the long run.

I don't plan to do these weekly as I don't imagine there will be much more content to add to these. Hopefully the server development stuff will cool down for a bit and I'll create these as and when something happens. Unless you all want a small weekly thread with server performance stuff in? Just let me know.

Thanks, Demigodrick.

Lots of new features to explore, and optimisations to the server.

2FA is also now possible, please enable if you can to keep your account safe! (Unless you use Jerboa, which doesn't support it yet - I don't know about mlem but let me know if it works for you)

If you notice any issues please let me know and I will look into them further.

Full details are in this post.

After my call for someone to help out with the instance, I'm glad to say that @[email protected] has been appointed to the position!

Please join in me in saying congrats and thank you for stepping up for the community!

Hi all,

I've set this community up for anyone to come in and have chat, post ideas, share any errors issues, or just post pictures of their pets. It pretty relaxed here.

I figured this was better for some of the day to day posts regarding the server too, to save the announcement community for more general announcements, and to keep here more a place for discussion.

So if you notice any issues, have any questions (literally anything about Lemmy or Lemmy.zip) or want to have chat, this is the place to do it.

We'll be looking to sticky a weekly discussion thread too, and create some community events & polls.

All ideas are welcome!