My apologies for the long post.
I have a single server running Unraid with about 12 services (Pihole, Wordpress, Heimdall, Jellyfin, etc.) all running on Docker. This server is also acting as my home lab NAS. Everything runs fine for my use case (at least for right now) but I’ve been thinking about creating some type of compute cluster for my services instead of a single server.
Recently, I saw a discussion about Proxmox, Docker, LXD and Incus where a user felt that Incus was a better option to all the others. Curious, I started reading up on Incus and playing around with it and contemplating switching all my services from Docker in Unraid to an Incus cluster (I’m thinking around 3 nodes) and leaving the Unraid server to serve as a NAS only.
In a nutshell Incus/LXD appear to be (to me) a lightweight version of a VM in which case I would have to manually install and configure each service I have running. Based on the services I have running, that seems like a ton of work to switch to Incus when I could just do 3 physical servers (Debian) in docker swarm mode or a Proxmox cluster with 3 Debian VMs with docker in swarm mode. I’d all possible, I would like to keep my services containerized rather then actual VMs.
What has me thinking that a switch to Incus may be worth it is performance. If the performance of my services is significantly better in Incus/LXDs as compared to Docker, then that’s worth it to me. I have not been able to find any type of performance comparison between Incus/LXD and Docker. I don’t know if there are other reasons as to “Incus over Proxmox and Docker” which is why I’m asking the greater community.
Here’s my question:
Based on your experience and taking into consideration my use case (home lab/home use), do the pros and cons of Incus outweigh accomplishing my goal by creating standalone hosts cluster or Proxmox cluster?
Strictly from a container perspective, wouldn’t this workflow create more overhead? For example, an incus cluster for me it would be Debian hosts (layer 1), incus (layer 2), lxd container (layer 3), docker (layer 4), app/service (layer 5). A Docker Swarm cluster (for me) would be Debian hosts (layer 1), docker (layer 2), app/service (layer 3).
Granted a docker swarm cluster would negate the possibility of VMs without having to install something else on the hosts but asking since I’m trying to keep my services in containers.
The setup I'm describing would just be incus running on the Debian host. You'd then spin up a VM with incus and run Docker Swarm inside the VM. Yes, there's a little bit of overhead with running the VM, but it's pretty minimal and makes it a lot easier to backup your whole server (since it's a VM) verses trying to backup a physical server.
You can run all of your containers in this incus VM. Yes, you could just run containers on the host but this setup makes it easier to keep everything self-contained IMO. Also, maybe you'll have a need to do something in the future that can't run in a Linux container (e.g. a FreeBSD server), so having the ability to spin up VMs and not only containers is useful