3317
submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

Another day, another update.

More troubleshooting was done today. What did we do:

  • Yesterday evening @phiresky@[email protected] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
  • @[email protected] created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
  • We started using this image, and saw a big drop in CPU usage and disk load.
  • We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
  • We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
  • We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @[email protected] , @[email protected] , @[email protected], @[email protected] , @[email protected] , @[email protected] for their help!

And not to forget, thanks to @[email protected] and @[email protected] for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

top 50 comments
sorted by: hot top controversial new old
[-] [email protected] 863 points 1 year ago* (last edited 1 year ago)

server load is too low, everyone upvote more stuff so i can optimize more

edit: guess there is some more work to be done 😁

[-] [email protected] 141 points 1 year ago

Upvote causes an endless spinner on Liftoff. 😁

[-] [email protected] 43 points 1 year ago

I'm getting 504 gateway time outs when I try to upvote

load more comments (2 replies)
load more comments (1 replies)
[-] [email protected] 115 points 1 year ago

I don't understand your graph. It says you are measuring gigabit/sec but shouldn't the true performance rating be gigabeans/sec for a Lemmy instance?

load more comments (7 replies)
[-] [email protected] 73 points 1 year ago

aye aye sir, to the upvote machine!

[-] [email protected] 26 points 1 year ago

Double the image upload size and you will see more shitposts

load more comments (1 replies)
load more comments (5 replies)
[-] [email protected] 437 points 1 year ago

Test:

Upvote if you can see this comment. πŸ‘

load more comments (10 replies)
[-] [email protected] 252 points 1 year ago

The change is noticeable. Good job guys.

Thanks for the updates.

[-] [email protected] 94 points 1 year ago

I agree. Felt it immediately when I started browsing. Everything is faster and more responsive, on top of the error messages disappearing

load more comments (1 replies)
load more comments (1 replies)
[-] [email protected] 124 points 1 year ago

Wow. So much smoother today.

Great work.

You dropped this πŸ‘‘

load more comments (2 replies)
[-] [email protected] 123 points 1 year ago

This is why having a big popular instance isn't all bad. It helps detect and fix the scaling problems and inefficiencies for all the other 1000s of instances out there!

[-] [email protected] 56 points 1 year ago

This, if everyone kept just spreading out to smaller instances as suggested in the beginning, while still a sensible thing to do, no one would have noticed these performance issues. We need to think a few years out, assuming Lemmy succeeds and Reddit dies, and expect that "small instance" will mean 50k users.

load more comments (1 replies)
load more comments (3 replies)
[-] [email protected] 115 points 1 year ago

You guys had better quit it with all this amazing transparency or it's going to completely ruin every other service for me. Seriously though amazing work and amazing communication.

[-] [email protected] 99 points 1 year ago

My upvote can go through fast now

Good work

[-] [email protected] 98 points 1 year ago

I love the smell of updates in the morning.

[-] [email protected] 92 points 1 year ago

Thank you guys for your awesome work!

Also to other people: DONATE TO FOSS PROJECTS. If 50.000 people donate only 0.5€, we have 25.000€ for funding the servers, coding, motivating/ people etc. Just don't take a cup of coffee for 1 day. We are already 2 millions in Lemmy instances. We can build a decentralized world together!!

load more comments (6 replies)
[-] [email protected] 81 points 1 year ago

Boy does it feel good to have those reports and understand the work you guys do. It's really inspiring! Thanks for your hard work, everything has been silk smooth! This instance is really great, Lemmy and its devs are really amazing and I feel at home in a nice, cozy community.

[-] [email protected] 80 points 1 year ago

I'm not sure wtf you just said, but lemmy.world feels very smooth today, so thank you for your continued hard work!

[-] [email protected] 79 points 1 year ago

Am I getting this correct: the whole lemmy.world instance run in one single container on one single host?

[-] [email protected] 47 points 1 year ago

You'd be surprised at how much performance this kind of setup can squeeze off. Often the limitation is more on the DB/storage than network handling and processing power.

load more comments (1 replies)
load more comments (3 replies)
[-] [email protected] 69 points 1 year ago

So that's why it was so smooth today... Great work!

[-] [email protected] 25 points 1 year ago

Certainly feels fast today!

[-] [email protected] 63 points 1 year ago

This is better optimization than most enterprise devs will see in their lifetimes.

[-] [email protected] 24 points 1 year ago

Some managers of the devs are not that interested in significant optimizations... Depends on what incentives and company culture drives them

load more comments (2 replies)
[-] [email protected] 61 points 1 year ago

upvoting posts is so much more stable now, we might actually see more bean posts as a result

[-] [email protected] 61 points 1 year ago* (last edited 1 year ago)

Submitting PRs is literally the most effective response that helps everyone who uses Lemmy. Thanks to you all.

[-] [email protected] 55 points 1 year ago* (last edited 1 year ago)

Upvotes are still getting rejected. Replies hang so I cancel out and it turns out they did post.

That said, browsing is pretty snappy and smooth. I know the kinks will get worked out eventually. Thanks for the update.

Edit: This now appears resolved minutes later. All smooth on my end.

[-] [email protected] 90 points 1 year ago

Hmm. Seems to work for me.. (Yes this is a test reply)

[-] [email protected] 28 points 1 year ago

Everyone it’s a test reply, deploy the upvotes

load more comments (2 replies)
load more comments (2 replies)
load more comments (3 replies)
[-] [email protected] 53 points 1 year ago

Literally a night and day difference in performance and stability! Thank you all for the hard work. To other users like me, consider reducing or replacing one of your lesser used subscriptions and directing that money to Lemmy. It’s much better served here if you ask me.

[-] [email protected] 53 points 1 year ago

This is why I love open source. The fact that a community can directly debug the code that's it's being hosted on and directly contribute the improvements back is just wild. Thanks for all the hard work @[email protected] and the rest of the lemmy.world team! The site already feels much more responsive.

[-] [email protected] 49 points 1 year ago

The server is absofuckinglutely flying today! It feels smooth and bug free!!! You guys are legends.

[-] [email protected] 45 points 1 year ago* (last edited 1 year ago)

As a data engineer, I'd be interested in hearing more about the SQL troubleshooting.

EDIT: It looks like [email protected] is a good place to subscribe to for more technical info on some of these performance improvements.

Also the Lemmy GitHub of course contains more information on bugs/enhancements/etc.

load more comments (5 replies)
[-] [email protected] 40 points 1 year ago* (last edited 1 year ago)

Good to see a heavy production server taking on the scaling issues. Thank you! To discuss Lemmy performance issues, there is a dedicated community: [email protected]

[-] [email protected] 38 points 1 year ago

Appreciate that these updates use the yyyy-mm-dd format :D

load more comments (4 replies)
[-] [email protected] 36 points 1 year ago

It now feels pretty good to browse and it now makes the experience of using Lemmy much more enjoyable. Having to spam the vote buttons was really annoying.

load more comments (1 replies)
[-] [email protected] 34 points 1 year ago

It's so smooth now; the speed difference is insane! You all are doing excellent work!

[-] [email protected] 34 points 1 year ago* (last edited 1 year ago)

Can we have an update on the status of Lemmy.world and how close ties we are going to have with Meta's threads? Threads is going to support ActivityPub, but time has shown that this is an attempt to try to kill this open platform and eventually replace it with theirs once they get everyone in their ecosystem. (Embrace, Extend...extinguish) Mastodon has said today that they don't mind sleeping with vipers when their demise / dissolution is in Meta's best interest.

Please tell me we are defederating from Meta....or let us know what to expect

EDIT: I originally stated that Mastodon told them to fuck off, but I got confused with Fosstodon (who did that). Mastodon doesn't mind being in bed with Meta

load more comments (7 replies)
[-] [email protected] 33 points 1 year ago

Even though i'm not from this instance, this is such a nice way of keeping the users posted about changes. I wish more companies (I know this is not a company) went straight to the point, instead of using vague terms like "improved stability, fixed few issues with an update" when things are changed. I hope all instance owners follow this trend.

load more comments (3 replies)
[-] [email protected] 33 points 1 year ago

Thanks to all involved across the board. Great work all around πŸ‘πŸ‘

[-] [email protected] 31 points 1 year ago

Thanks for this very nice report.

[-] [email protected] 30 points 1 year ago

Huge props to everyone working on the project. It's awesome seeing everyone work together and resolving issues so quickly!

[-] [email protected] 29 points 1 year ago

Thanks for the detailed update and all the hard work you guys are doing!

[-] [email protected] 28 points 1 year ago

Gadzooks! These are huge fixes. Compliments to the team, you guys pulled off a small miracle today.

[-] [email protected] 28 points 1 year ago
[-] [email protected] 26 points 1 year ago

Lemmy's devs and the .world admins have done in a month what Reddit hasn't done in it's whole existence: having a smooth and almost bug-free experience.

Jerboa feels so damn FRESH to use now!

load more comments (2 replies)
[-] [email protected] 25 points 1 year ago

The site is running so much better now, thanks to all. BTW: Love these updates!

[-] [email protected] 24 points 1 year ago

I'm very curious: does single Lemmy instance have the ability to horizontally scale to multiple machines? You can only get so big of a machine. You did mention a second container, so that would suggest that the Lemmy software is able to do so, but I'm curious if I'm reading that right.

load more comments (6 replies)
[-] [email protected] 24 points 1 year ago

I don't understand anything other than you worked diligently to make things smoother. Thanks to everyone for their wonderful work!

load more comments (1 replies)
load more comments
view more: next β€Ί
this post was submitted on 05 Jul 2023
3317 points (99.4% liked)

Lemmy.World Announcements

28679 readers
42 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

founded 1 year ago
MODERATORS