this post was submitted on 14 Jun 2023

43 points (100.0% liked)

Technology

37585 readers

340 users here now

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 2 years ago

MODERATORS

[email protected]

#lemmy /#kbin has a problem that #mastodon hasn't even attempted to solve; groups and what happens when they get popular. (mas.to)

submitted 1 year ago by [email protected] to c/[email protected]

50 comments fedilink hide all child comments

#lemmy/#kbin has a problem that #mastodon hasn't even attempted to solve; groups and what happens when they get popular.

#Communities, #groups, #magazines, whatever they are called are implemented as #Actors in #ActivityPub. They are basically just *very* popular users who boost a *lot*.

You can't just distribute them across instances the way normal actors do. Whichever server hosts @[email protected] or @[email protected] is going to get HOSED on the regular.

top 40 comments

sorted by: hot top controversial new old

[–] [email protected] 30 points 1 year ago* (last edited 1 year ago) (1 children)

While it's true that the hosts of popular communities will get more traffic, it's actually not as bad as it first seems.

Every Lemmy instance with at least one subscriber in that popular community will act as a mirror. That means that users who are just reading posts and comments will not cause any additional load on the home-instance of the popular community, because they are consuming local copies of the posts and comments.

This will actually help scaling a lot, and is in fact exactly how many centralized platforms scale (by creating a bunch of read-only copies of content).

[–] [email protected] 3 points 1 year ago (1 children)

@sunaurus what about when those users like/boost/reply?

[–] [email protected] 7 points 1 year ago (1 children)

In those cases, the action will need to propagate back to the home server (that's where the "hosts of popular communities will get more traffic" comes from), but keep in mind - people usually read at least one or two orders of magnitude more than they write.

[–] [email protected] 2 points 1 year ago (1 children)

@sunaurus there's a lot of upvoting happening on popular subreddits

[–] [email protected] 9 points 1 year ago* (last edited 1 year ago) (1 children)

Absolutely, but a user will only upvote a post once, while they will read it on every reload of their page. (By "read" I mean "fetch it from their local mirror")

[–] [email protected] 9 points 1 year ago

Hmm, you could probably extend the protocol to do eventual consistency across instances if that ever becomes a problem, remote instances could keep their own counts and only send aggregated updates.

[–] [email protected] 5 points 1 year ago (3 children)

Did you post this from Mastodon? I wish I could tell where this came from.

Basically if I understand this right, if you have an instance with a very popular community on it. It is likely that it will need some massive infrastructure scaling if it wants to handle the enormous amount of world wide traffic?

[–] [email protected] 7 points 1 year ago (1 children)

Yes. If you run the server, then you are the source of truth of that community. All other servers that federate your community query your server to access the community and show it to their users.

So if you run a server and a community explodes there, you might only have 500 users on your instance, but you might have 50k users reading that community and interacting with it from other Lemmy instances, thus your server needs to scale to 50k users worth.

And ever more essential, your server is the source of truth of that community. So if your server is hacked or corrupted or deleted, that community is gone. Other instances don't mirror it (except for temporary caching), so the Lemmy network essentially is a trust network of other people maintaining servers long term (and each inventing a monetary system to pay for it). I still think the network might be better than a centralized system like reddit, but it definitely has a lot of growing and policies that need to be sorted out very soon

[–] [email protected] 3 points 1 year ago (2 children)

So are these other servers just routing requests from their users to your server's community? Or are they actually copying everything over every so often (caching) and serving up the requests themselves? How real time is it, I guess is what I'm asking?

[–] [email protected] 3 points 1 year ago

Yeah, apparently I was wrong about this (still learning lemmy and fediverse stuff...). Text content of posts and comments are "synced" to your server and stored in your database there. Then future requests for that content are served from your instance. So its not as bad as I thought it was (the network load should be lower since you aren't acting as entirely a proxy, more like a cache), but database bloat will be a huge problem (its already a big problem in other federated things like mastodon and matrix, where every server ends up saving everything they want into their own database).

I'm not sure what happens when the original server goes down, does the federated servers discard that data? Or do we each maintain a forever copy until we want to get rid of it ourselves? There's also some notes I've seen about how servers only incrementally cache federated content (only posts and comments that are viewed by someone are fetched, and new comments may not be fetched until someone wants to see it)... so not everybody has a "pure and full" copy of posts necessarily.

But overall I wonder how all the various sysadmins hosting these lemmy instances will deal with the expotential growth they're going to see, or if smaller instances will start defederating to save on hardware costs (no reason for my tiny instance that only talks about blue shiny rocks to federate with lemmy.world and store all that content)

[–] [email protected] 1 points 1 year ago

Information about post changes on the originating instance come in pretty much in real time, and get saved in the local database.

If the local instance is configured with pictrs support images are also cached locally.

[–] [email protected] 6 points 1 year ago

I wish I could tell where this came from.

Isn't that what this colourful icon in Lemmy is for? It appears to link to the original source of the post or comment:

[–] [email protected] 1 points 1 year ago

@Nymphioxetine posted from mastodon (because Lemmy is slow right now😊)

[–] [email protected] 5 points 1 year ago* (last edited 1 year ago) (1 children)

[–] [email protected] 8 points 1 year ago (1 children)

The network can actually scale quite well thanks to the fact that other instances will act as mirrors of communities!

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (2 children)

But what happens when the instance hosting the community goes down? Are all external instances still able to participate in that community? I get that they are mirrored but will everyone still be connected?

[–] [email protected] 8 points 1 year ago (1 children)

No. The "single source of truth" is the instance hosting the community. If it goes down the community itself goes down with the ship. The only way to prevent it is to have a IT infrastructure that can provide redundancy

[–] [email protected] 3 points 1 year ago (1 children)

Yeah, that isn't good.

[–] [email protected] 0 points 1 year ago (1 children)

having a redundant system is feasible (I'm just a dev, not an architect so don't take my words for granted) but it have to be designed and putted together ... and prices are gonna skyrocket

[–] [email protected] 1 points 1 year ago

Lemmy / the fediverse isn't designed this way, but it could be. There are certainly systems that share diskspace and are multimaster and keep stuff as long as someone is interested in it(i.e. accessing the data). I really start to think added to the lemmy / fediverse servers should be something like what freenet used to do in terms of hosting content.

[–] [email protected] 4 points 1 year ago (1 children)

it may be worth putting a bug request on the activitypub github, because I agree that could become a huge problem, but its also alot of work to implement because most instances will need to update to the newest activitypub standard once they approve a new version of the standard.

[–] [email protected] 5 points 1 year ago

Definitely a problem that needs solved sooner rather than later, but I assume with a lot more eyes on ActivityPub now, people much smarter than I will have a solution.

[–] [email protected] 3 points 1 year ago (3 children)

What does hosed mean, technically?

[–] [email protected] 12 points 1 year ago* (last edited 1 year ago) (1 children)

Lots of traffic, lots of posts, lots of comments, ... That's going to need more storage, more bandwidth, more CPU power, higher running costs. The original instance hosting the community bears a higher load than the instances that duplicate it.

Ideally, there would be a way to more evenly distribute this load across instances according to their resources, but from my (currently limited) knowledge, I don't think Lemmy/ActivityPub is really geared for that kind of distributed computing, and currently I don't believe that there's a way to move subs between instances to offload them (although I believe some people may be working on that).

Perhaps the Lemmy back-end could use a distributed architecture for serving requests and storage, such that anyone could run a backend server to donate resources without necessarily hosting an instance.

For example, I currently have access to a fairly powerful spare server. I'm reluctant to host a Lemmy instance on it as I can't guarantee its availability in the long term (so any communities/user accounts would be lost when it goes down), but while it's available I'd happily donate CPU/storage/bandwidth to a Lemmy cloud, if such a thing existed.

There are pros and cons to this approach, but it might be worth considering as Lemmy grows in popularity.

[–] [email protected] 1 points 1 year ago (1 children)

I don’t think it’s a problem. If you weren’t using activity pub and just something like reddit then if you were reddit (the sysadmin) you’d also deal with having to scale if your community gets really popular

Stuff that gets linked to also has the same problem

https://www.jwz.org/blog/2022/11/mastodon-stampede/

(Btw I don’t like jwz but he mentions it here)

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (1 children)

Funny how you say it's not a problem, then go on to describe the problem that needs to be dealt with. Dealing with scaling is a problem, and it's a problem that costs money.

Posts like this: https://lemm.ee/post/58472 suggest it is a problem. The rise in traffic seen by Lemmy in the last few days is absolutely tiny compared to a site like reddit, and already instances are struggling to cope. The recent growth in user registrations represents only about 0.007% of reddit's active user base. (~60K new Lemmy users vs 861,000,000 active monthly reddit users). A site like reddit costs millions to run.

There are 190+ Lemmy instances last time I checked, yet almost all the brunt of this load has been borne by a handful of servers, which see an inordinate amount of traffic while 100+ other servers sit around idle. Why should a handful of "lucky" servers have to pay all the hosting costs? What if a volunteer-run instance explodes to reddit-like levels of popularity? It will simply fold, unless the volunteer has serious money to throw at the problem.

[–] [email protected] 1 points 1 year ago (1 children)

Lemmy in the last few days is absolutely tiny compared to a site like reddit, and already instances are struggling to cope.

While this is true, 5 days ago lemmy.ml, the biggest instance, was on a 67 EUR server which is very small. https://news.ycombinator.com/item?id=36270094

Posts like this: https://lemm.ee/post/58472 suggest it is a problem

This is a scaling problem (having more users means you need more mods) but I disagree with how they handled it and it isn't a money related thing. My thoughts on this are in an older post when this was first announced https://partizle.com/comment/64178

Why should a handful of “lucky” servers have to pay all the hosting costs?

My initial idea is to use the something awful model of paying a one time fee to register an acount. The problem is that people would just sign up on another instance that doesn't charge a fee but still add load to the lucky instance. Another approach could be to participate in communities on one of those lucky servers then you need to pay a one time fee to that server (comments would need to be removed by a bot if they're not made by an approved user). I'm not saying that's perfect, but it's an idea. Adsense is another idea.

[–] [email protected] 1 points 1 year ago

Again, you say it's not a money problem.... then go on to describe a money problem! 🤦‍♂️

Also, did you read the link included in the post I linked to? ( https://beehaw.org/post/520044?scrollToComments=true )

That's a money problem and a time problem. (And time problems are money problems.)

High traffic sites need lots of money and resources to run. That's just a fact.

We can solve this in many ways as Lemmy grows (and I think we will), but to just pretend there aren't any problems to be solved is naive, IMO.

If Lemmy grows to any significant percentage of reddit traffic, the Lemmy of tomorrow will (necessarily) look quite different to the Lemmy of today.

[–] [email protected] 5 points 1 year ago

I think they meant like "overloaded", like a hose spraying water, but the water being users from all around the fediverse

[–] [email protected] 2 points 1 year ago

It means the F word

[–] [email protected] 3 points 1 year ago

I’d love to see feedback from admins on the scaling problems they are having. Hopefully that scales per server and not per user per server.

[–] [email protected] 3 points 1 year ago

@schizanon @[email protected] @[email protected] Yep. I expect that long-term a lot of instances will limit the ability to create groups (as Beehaw does) or place restrictions on the size a group is allowed to grow before asking that group to move to another/their own instance or ask for finances to help with the costs.

load more comments