this post was submitted on 20 Aug 2024
21 points (100.0% liked)
Self Hosted - Self-hosting your services.
11197 readers
1 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules
- No harassment
- crossposts from c/Open Source & c/docker & related may be allowed, depending on context
- Video Promoting is allowed if is within the topic.
- No spamming.
- Stay friendly.
- Follow the lemmy.ml instance rules.
- Tag your post. (Read under)
Important
Beginning of January 1st 2024 this rule WILL be enforced. Posts that are not tagged will be warned and if not fixed within 24h then removed!
- Lemmy doesn't have tags yet, so mark it with [Question], [Help], [Project], [Other], [Promoting] or other you may think is appropriate.
Cross-posting
- [email protected] is allowed!
- [email protected] is allowed!
- [email protected] is allowed!
- [email protected] is allowed if topic has to do with selfhosting.
- [email protected] is allowed!
If you see a rule-breaker please DM the mods!
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Files won't change and are hundreds of GBs
ok, database it is then
Are they roughly 55GB compressed?
👀
🕵️
Spill the beans!
Could use Polars, afaik it supports streaming from CSVs too, and frankly the syntax is so much nicer than pandas coming from spark land.
Do you need to persist? What are you doing with them? A really common pattern for analytics is landing those in something like Parquet, Delta, less frequently seen Avro or ORC and then working right off that. If they don't change, it's an option. 100 gigs of CSVs will take some time to write to a database depending on resources, tools, db flavour, tbf writing into a compressed format takes time too, but saves you managing databases (unless you want to, just presenting some alternates)
Could look at a document db, again, will take time to ingest and index, but definitely another tool, I've touched elastic and stood up mongo before, but Solr is around and built on top of lucene which I knew elastic was but apparently so is mongo.
Edit: searchable? I'd look into a document db, it's quite literally what they're meant for, all of those I mentioned are used for enterprise search.