this post was submitted on 03 Jul 2023
263 points (97.8% liked)

Selfhosted

40394 readers
189 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

Just started self hosting this instance. Nothing on the docs mentioned anything about storage considerations.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 271 points 1 year ago (7 children)

This is lemmy.world after 4 weeks:

58G	pictrs
34G	postgres
[–] [email protected] 81 points 1 year ago (2 children)

Considering this is going to be around a 5 user instance at most I think I'll be good for awhile. Thanks!

[–] [email protected] 57 points 1 year ago* (last edited 1 year ago) (1 children)

im running 50 users right now, subbed to A LOT of communities, seeing db growth of about 100mb per day.

[–] [email protected] 18 points 1 year ago (2 children)

That seems high when you extrapolate that to 10000 users, like a larger instance might have.

[–] [email protected] 69 points 1 year ago* (last edited 1 year ago) (2 children)

It's all about how many communities your user(s) subscribe to since your instance basically acts as a mirror for those.

My instance has been running for 23 days, and I am pretty much the only active local user:

7.3G    pictrs
5.3G    postgres

edit: I may have a slight ~~Reddit~~ Lemmy problem

[–] [email protected] 9 points 1 year ago (1 children)

So if you're the only user (let's assume for ease) then, that represents all the updates (posts, comments, votes) from each community that you are subscribed to?

[–] [email protected] 16 points 1 year ago (2 children)

Yeah, and I purposely subscribe to (or sometimes have a dedicated "federation helper bot" account I run subscribe to) most of the most popular communities on the most popular instances so I can get a decent sampling of what's going on in the fediverse on the "All" feed. So I assume my storage usage is maybe a bit higher than what an "average" single-user instance may be...

[–] [email protected] 4 points 1 year ago

lmao same here. I have a spare account that I use to sub to everything worth subbing to. I haven't automated it yet though.

[–] [email protected] 2 points 1 year ago (1 children)

Ooh, that's a really good idea, I need a federation helper bot/account when I start self-hosting a Lemmy instance!

[–] [email protected] 2 points 1 year ago

Yeah it's not automated or anything, I just pop an incognito window and use it when there is a communitI think is worth seeing sometimes in "All" (or just for archiving purposes) but don't want to clutter "Subscribed". I may make something to auto-subscribe to communities meeting some criteria or something at some point in the future...

[–] [email protected] 3 points 1 year ago (2 children)

Do you also post stuff? I mean my instance is only about an hour old, but I've subscribed to some communities, yet I don't see the picture service consuming the S3 storage I've configured

[–] [email protected] 13 points 1 year ago

Lemmy caches every thumbnail of every post for like a month or something using Pictrs, so that storage will eventually hit a sort of equilibrium and start growing much more slowly (only reflecting post/thumbnail volume during the cache time).

Between profile images, community banners/icons, post images etc. there are probably a few dozen images that will be sticking around for the long haul at the moment.

[–] [email protected] 5 points 1 year ago

Your instance only caches thumbnails, so it won't take much space. The full images are served from the remote instance. So you basically only store whatever your users upload.

[–] [email protected] 21 points 1 year ago (1 children)

It won't scale linearly. A lot of those users will be subscribed to subs the instance is already replicating. It would only be new subs that would add to the growth.

[–] [email protected] 3 points 1 year ago

And only active subs. And even then, it's just text and tiny thumbnails.

[–] [email protected] 15 points 1 year ago (1 children)

Question if you know: does a lemmy instance have to be publically accessable to work? Like, if I make an instance on my homelab can the instance "fetch" content and serve it faster locally? Could I reply to a post and have others see it? Etc

[–] [email protected] 1 points 1 year ago

wondering this also! wouldnt it require a domain for your account though?

[–] [email protected] 18 points 1 year ago (1 children)

Now I wonder how viable it would be to support video hosting. The answer is almost certainly "God no!"

[–] [email protected] 1 points 1 year ago

It is viable through other hostings

[–] [email protected] 17 points 1 year ago

Honestly, Less than I thought!

[–] [email protected] 13 points 1 year ago (1 children)

Interesting, I thought it would be waaayyy more

[–] [email protected] 16 points 1 year ago

At the end of the day the vast majority of what needs to be saved is text. If media content is embedded, the the server just has to save the path to the file not the file itself.

[–] [email protected] 11 points 1 year ago* (last edited 1 year ago)

Yeah lemmy seems to use just about nothing for data storage.

[–] [email protected] 11 points 1 year ago

Wow, that is surprisingly not bad given the size of the instance!

[–] [email protected] 4 points 1 year ago* (last edited 1 year ago)

Feels like this will benefit from some sort of fuzzy deduplication in the pictrs storage. I bet there are a lot of similar pics in there. E.g. if one pic or a gif is very similar to another, say just different quality or size, or compression, it should keep only one copy. It might already do this for the same files uploaded by different people as those can be compared trivially via hashing, but I doubt it does similarity based deduplication.