Hi All,

Small amount of site maintenance to take place, which will include a server restart so there will be some downtime during this 1 hour window.

Site status can be seen on the status page.

Thanks,

Demigodrick

Edit 1: Maintenance is now underway. I'll update this post when complete.

Edit 2: Site upgrade is complete, but some extra work is taking place so the window will be extended for another hour, until 13:00 UTC.

Edit 3: Done!

Hello All,

Heads up that maintenance on the site will take place at 19:00 UTC on Saturday 30th.

This will include rebooting the server, so there will be definite downtime although we hope this will be to a minimum.

We'll also be applying the latest lemmy update 0.18.5 at the same time.

Thanks

Demigodrick

Looks to have gone smoothly - please check your favourite communities and let me know if there are any issues.

Hi All,

Just a heads up that at 12 noon tomorrow (about 16 hours from now) the site will temporarily be offline for a short period while we perform some maintenance and upgrade the performance of the server.

We'll keep the downtime as brief as possible, and you can keep an eye on status.lemmy.zip to see the progress.

We'll make another post when the updates are completed.

Thanks!

Hello all,

Over the last 48 hours the community has had the opportunity to vote on defederating with Exploding Heads and rammy.site (as an extension of Exploding Heads following its abandonment).

Before we move on to the result, I just want to add that there are no winners or losers here, and rather this is a vote we had to call because the moderation tools for Lemmy users and for Lemmy admins are not yet robust enough, and that in the future we hope that the real best outcome will be to allow users to choose to defederate/block instances entirely.

The results of the poll are as follows:
76% of votes were to defederate, with 24% of the vote to not defederate.

Based on the majority of votes, we've now implemented the block on those instances.

If/when the robust moderation tools are in place in the future, we will revisit this block and look instead to block problematic communities.

I appreciate everyone taking the time to vote, and all the discussion that has taken place around this topic. It has also helped the admin team to better be aware of our responsibilities to yourselves as users of this instance, and our instance's footprint in the wider fediverse.

I also need to mention we've been made aware of illegal content on the burggit.moe instance involving depictions of underage children in sexual acts. Even in cartoon style, these are illegal in many countries, including the UK, and so we've implemented a block on that instance too under the "illegal content" clause in our Defederation Policy. Upon review it appears we had very few (i.e. 1) subscribers to most of these communities on that instance, and so we don't believe this will have any impact on the current userbase.

Hello all,

This is a heads up that this weekend we will be asking the community to vote on defederating with Exploding Heads and Rammy.site, following an increase in their content making its way into our instance and moderation queue.

This content borders on illegal in my/the servers locality (if it was to be clearly iilegal we would have defederated with them immediately to protect the instance) and is usually hate-filled and targeted towards minorities.

So far we've had very little to do with them, but have come across some posts wishing (although not directly stating) harm on trans people and other minorities.

My personal take is the content being generated in those communities is not the content that this community needs to see or be a part of. Given the increasing visibility of this content coming over to us, I feel now is the time for the community to vote.

However, as I have said from the start, these decisions are not mine to take alone, and are for the community to come together to agree on. I never take defederation lightly, and have not defederated with any other instance (except threads) yet. Therefore this is a decision for the community to weigh in on.

In order to protect users here the vote will be held anonymously (we'll release details on Friday), and we encourage discussion to be held on our Matrix space so as to prevent users here being targeted for any views.

The vote will be held at some point on Friday and run for 48 hours over the weekend to give everyone time to vote.

For transparency, I have not reached out to EH mods (and the rammy.site admin is no longer active or moderating their instance) as I don't think this would produce any valuable discussion, they're unlikely to change their instance's purpose for us.

https://canvas.toast.ooo

The Canvas event has started. You can go there with the above link and sign in by entering your instance name (lemmy.zip if you're a user on this instance) and your username (no password entry is required).

After that you will receive a DM on Lemmy with a code that you can enter on the canvas site to confirm your identity. You might need to press 'Sign Up' afterwards on the main Canvas and it should show your username meaning you're good to go.

Quick rundown of the rules:

1 pixel per minute. Up to six stacked pixels. Notification and timer for upcoming pixels. No hateful/illegal/NSFL content is permitted.

Event runtime:

This event will last 72 hours, starting August 4th, 12:00AM EST (4:00AM UTC) and ending on August 6th, 11:59PM EST (3:59AM UTC)

• Matrix Space
• Lemmy Community

Hey everybody,

If you're interested in a Fediverse equivalent of a collaborative online canvas where each user gets to place 1 pixel every minute @[email protected] is running an event this coming weekend using Pxls.

This event will last 72 hours, starting August 4th, 12:00AM EST (4:00AM UTC) and ending on August 6th, 11:59PM EST (3:59AM UTC)

Here are the instances that are joining:

• toast.ooo
• lemmy.zip
• lemmy.ca

Join the Lemmy Community [email protected]

Join the Matrix Space #lemmy-canvas:matrix.org

Link to original post: https://lemmy.zip/post/888011

More info will follow before the start of the event!

Hi All,

Following diagnosis of issues we think the site has now succesfully upgraded to 0.18.3.

PLEASE check all of your favourite communities ASAP and let us know if there are any issues, so if we do need to roll back again we don't lose too much data.

I really appreciate everyone's patience with us with this, we hit an issue with the postgres database and have learned many lessons from this experience!

Please note, mobile/beta/front end sites might not be updated yet. I've pulled all the latest updates now but if there are any errors hopefully they'll be fixed soon.

Due to a corrupted database following the upgrade we've needed to roll back to a point before the upgrade happened.

I'm really sorry about this, but we've had to take this step so we can continue to properly run the site.

Anything added to the site in the last 12 hours will have been wiped, i.e. posts, communities. We're looking at ways to make sure this doesn't happen in future.

Happy to answer any questions below.

Hey everyone, we've made a new theme for our instance. It's set as the default instance theme at the moment so if the theme setting in your profile is set to 'Browser Default' then you should already be seeing it.

Folded is a theme based on another custom theme, Lemonberry, which made things a lot easier to figure out so shout out to [email protected] for the base theme.

It's currently only a 'dark mode' and hasn't had much testing so please let us know if you see any glaring issues that we missed. Also, constructive criticism is more than welcome with this being a first attempt.

Also, if you wish to opt out of this theme all you need to do is click on your username on the top right -> Settings -> Theme and select a different one. The old instance default should be 'darkly' if you just want to revert to that.

Otherwise, you can set it to 'folded-v4' or 'Browser Default' to check out the new theme.

Here are some pics of the theme:

Hope you like it

Edit: RC version of lemmy-ui has been released to fix this issue for emojis, which has been applied to lemmy.zip.

Hello all,

Around 5 hours ago Lemmy.world and lemmy.blahaj.zone experienced a "hack" targeting admin accounts, which then altered the sites and spread spam etc.

Thankfully the attack vector was figured out quite quickly and mitigations were put in place. Sami was very quick to act and defederated us from those instances to prevent their exploit spilling over into our site.

The attack vector is custom emojis, which allowed attackers to exploit weaknesses via cross site scripting. More info is available here for those interested: https://github.com/LemmyNet/lemmy-ui/issues/1895

This attack gives attackers access to your "session". They won't know your password as they did not have access to the database or the server. Attackers would have had access to your user settings page, so they could potentially see your email address.

Lemmy.zip had a custom emoji in place from testing a few weeks ago, so as an extra precaution we've reset the secrets table in the database which should have logged everyone out (sorry!). This would prevent attackers still having access to any accounts on our site.

Its important to add that at this stage I don't believe any of our users have been compromised, due to Sami's quick action to defederate and remove the custom emoji once this was known as the attack vector. No accounts on this instance were involved in the posting of spam and none of our admin accounts were compromised either.

If we find out any more information we'll add it here. We'll continue to implement all security fixes as they become available.

Defederating from Lemmy.world and lemmy.blahaj.zone until they have their shit back together as they seem to have been compromised (extent is not fully known at the moment). Beehaw took their servers down preemptively too.

Rogue admin accounts can do harm to other instances too this is a precautionary measure until we have a better idea of what we're dealing with.

Reminder not to click on sketchy links!

Should be down for a short time while the update is applied.

You may need to do a hard refresh on your browser to refresh your cache - Shift + F5 on chromium browsers, Ctrl + F5 on Firefox.

As always you can check status.lemmy.zip for updates if you can't access the site.

Just a heads up that the latest version of the Lemmy server has been released. I'll be applying the update shortly. I don't expect too many issues but you might lose connection temporarily.

Edit: Honest to god pictures are broken again. FFS. I know what the issue is though, it lies with the change in the internal nginx proxy method - attempting a fix now> BE

Edit 2: images are back, more info here: https://github.com/LemmyNet/lemmy-ansible/issues/96

Originally I was trying to put Lemmy.zip behind a CDN (cloudflare) to reduce costs, however suddenly images stopped loading. Cue mild panic attack.

You may notice that there are no images on the local instances.

Turns out this is because Backblaze has gone down, who host all our storage.

https://status.backblaze.com/

When they come back up, pictures should automatically load again.

(I'm using us-east for image storage and this is the server thats gone down while the others still work, talk about bad luck!)

Hello All,

So today a few Lemmy instances have been hit by a spam bot. We had 80 new accounts created in a matter of minutes, but thankfully due to email verification being required, none have been able to actually complete the sign up process.

Because of this, I've had to enable further measures to ensure sign ups are real people. Anyone signing up will now need to explain that they're human AND verify their email address. This seems to have stopped the bots for now.

Because manual approval is required for each account, I'm looking to appoint a community member to help moderate and admin the instance.

If you're interested, please send me a message (click on my profile and click send message) and give me a quick.overview of yourself and why you want to admin the instance.

To be eligible you must:

• be a member of Lemmy.zip
• account must be suitably old (minimum 3 days)
• must be active on Lemmy

If you think this is something you'd be interested in then please send me a message.

On the eve on Lemmy.zip turning one week old, and following requests from members of the community, I'm putting together a post to show how Lemmy.zip is being run, the resources taken up in the back-end and the general day to day issues I've come across and resolved.

Hardware

Firstly, the setting up of Lemmy.zip.
Lemmy.zip (the domain) was purchased on Google Domains (apparently now sold off to squarespace?), and I have the domain for two years.

Secondly, the cloud platform is Hetzner Cloud. The server is based in Falkenstein, Germany. The server "name" is CX21 - Its an intel CPU, with 2x vcpus, 4GB of RAM, a 40GB SSD and 20TB of outgoing traffic.

After setting up SSH, I followed the Ansible guide to set up Lemmy, which got the instance set up pretty much instantly. Great success.. or so I thought!

Setup

The first thing I did was try federating with some other instances, which appeared to work straight away. Great. I tried some of the settings, added a logo, turned on user registration and email validation, added a site bio etc. Then I tried logging in and out. Oh no. I couldn't log back in with the Admin account, because email verification was turned on and I hadn't verified my email! Now, I know how to fix this via SQL commands, but at the time I had no clue. I also wasn't receiving any emails from Lemmy.zip. After some googling, it turns out Hetzner blocks port 25, which is the default postfix port (postfix is the piece of software that handles emails). So now I was locked out with seemingly no way to get back in. Whoops. So I went and deleted the server and started from scratch. I reinstalled Lemmy, set up the instance again and made sure email verification was off this time. I posted in a few places about the server, and suddenly people were joining!

The next step was to get email working. I did try SendGrid for emails first, who immediately refused to verify me and still, a week later, have not replied to my support request.

Next I tried Mailersend, who had quite a complex validation but did finally approve me. Then I had to figure out how to get the server to send emails using the Mailersend details.

After a couple of hours of testing, it turns out you only need to add the details to the lemmy config.hjson file, and you don't need to alter anything in the postfix container. I re-uploaded Lemmy, set email validation to true, and created a test account - boom! Email received.

For the most part, that was the first 24 hours done. I think by the end of the first day we had ~20 users, which I was really excited about.

Stats

As I'm typing this, we have 369 (nice) users, which is incredible. At one point, we even made it onto the recommended servers list however we've since been knocked off the list due to the uptime dropping after I've had to restart the server a few times. Still, another moment I was really proud of.

Most of the week has been quite smooth in all honesty, until today when I had a few reports that users couldn't register or log in to their accounts. It took quite a while to track down the issue, as the logs were not really giving anything away. Finally, I saw a line in the log which said api_routes_websocket: email_send_failed: Connection error: timed out (this literally took me 3 hours to find this one line - tip, set logging level to warn and not the default info if you want any chance of finding errors in a busy instance!)

With a bit of googling, I was able to check my connection to the Mailersend smtp server and noticed that the telnet connection was timing out, which indicates that port 587 is being blocked somewhere in the chain. I have emailed Mailersend support but being a Saturday i'm not expecting a response for a while.

Before Mailersend stopped working, here is what the email stats look like:

You can see the expected tail off as Reddit opened back up, but then you can see emails drop to 0.

So with some frantic googling, I've switched (hopefully temporarily) to another provider, and emails are working again. Phew.

In terms of server performance, have some fancy graphs:

Server performance over the last week. The three big spikes relate to me doing something intensive on the server, rather than anything Lemmy is doing. Usage is mostly around 50% which is great, and gives us lots of growing room still.

Here are some other graphs to look at:

Live server usage as I'm writing this

Live network usage

Storage space used:

Storage used out of the 40gb - 36% used. There was an issue with the pictrs container (the software that uploads/manages photos) writing a log that was about 7gb in size before I noticed. That has been fixed with a rotating log file rather than one big file.

Other stuff

Thought I'd add some stuff I've liked from the last week. Firstly, if you haven't already subscribed to the Starfield community and have an interest in the game, I urge you to do so.

A community I've quite enjoyed too is Dad jokes - definitely worth checking out if you want a laugh.

End

Hopefully thats been interesting for people - if there is anything you'd like to see more on, let me know. Happy to add more detail (where I can!)

Anyone is welcome to send me a message on here, I have no issues talking more privately. I have also set up an email address ([email protected]) for anyone who is locked out of an account or needs to share personal details and isn't comfortable using this platform to do so.

Edit: appears to be fixed.

Lemmy.zip may be up and down all day - emails have stopped working (no discernible reason why) and so registrations are closed, but you should be able to login without having a registered email to stop current users from being locked out. I'll update when I know further information.

Edit: for anyone that is interested, I can't telnet port 587 on the mail sending platform. I can do it fine for other smtp providers so I assume it is a current provider issue. I'm looking at switching providers to see if this helps.

Edit 2: I've switched email providers and it appears we're back up and running again - email resets appear to work. I'll switch email verifcation back on and allow new users.

If anyone is having any issues, please email me at [email protected] if you can't access your account, and I will look into it.