this post was submitted on 04 Jul 2023
91 points (97.9% liked)

Selfhosted

40132 readers
793 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

I'm thinking about starting a self hosting setup, and my first thought was to install k8s (k3s probably) and containerise everything.

But I see most people on here seem to recommend virtualizing everything with proxmox.

What are the benefits of using VMs/proxmox over containers/k8s?

Or really I'm more interested in the reverse, are there reasons not to just run everything with k8s as the base layer? Since it's more relevant to my actual job, I'd lean towards ramping up on k8s unless there's a compelling reason not to.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 6 points 1 year ago* (last edited 1 year ago) (1 children)

If everything you want to run makes sense to do within k8s it is perfectly reasonable to run k8s on some bare-metal OS. Some things lend themselves to certain ways of running them better than others. E.g. Home Assistant really does not like to run anywhere but a dedicated machine/VM (at least last time I looked into it).

Regardless of k8s it may make sense to run some sort of virtualization layer just to make management easier. One panel you can use to access all of the machines in you k8s cluster from a console level can be pretty nice, and a Proxmox cluster would give you this. You can make a VM on a host that takes up basically all of the available RAM/CPU on it. Proxmox specifically has some built-in niceties with gluster (which I've never use, I manage gluster myself on bare metal) which could even be useful inside a k8s cluster for PVCs and the like.

If you are willing to get weird (and experimental) look into Rancher's Harvester it's an HCI platform (similar to Proxmox or vSphere) that uses k8s as its base layer and even manages VMs through k8s APIs... I played with it a bit and it was really neat, but opted for bare metal Ubuntu for my lab install (and actually moved from rke2 to k3s to Nomad to docker compose with some custom management/clustering over the course of a few years).

[–] [email protected] 2 points 1 year ago (1 children)

Fwiw I've been running home assistant in a docker container for a couple years without any issues

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago) (2 children)

Yeah, I think the problem comes if you don't want to manually configure "Add-ons". Using this feature is only supported on their OS or using "Supervised". "Supervised" can't itself be in a container AFAIK, only supports Debian 12, requires the use of network manager, "The operating system is dedicated to running Home Assistant Supervised", etc, etc.

My point is they heavily push you to use a dedicated machine for HASS.

[–] [email protected] 1 points 1 year ago

Yeah, I haven't missed the HASS add-ons but that's a good point

[–] [email protected] 0 points 1 year ago

Yea I've been running "core" in docker-compose and not the "supervised" or whatever that's called.
It's been pretty flawless tbh.
It's running in docker-compose in a VM in proxmox.
At first, it was mostly because I wanted to avoid their implementation of DNS, which was breaking my split-horizon DNS.

Honestly, once you figure out docker-compose, it's much easier to manage than the supervised add-on thing. Although the learning curve is different.
Just the fact that your add-ons don't need to go down when you upgrade hass makes this much easier.

I could technically run non-hass related containers in that docker, but the other important stuff is already in lxc containers in proxmox.
Not everything works in containers, so having the option to spin a VM is neat.

I'm also using PCI passthrough so my home theater/gaming VM has access to the GPU and I need a VM for that.

Even if they only want to use k8s or dockers for now, having the option to create a VM is really convenient.